[
https://issues.apache.org/jira/browse/SPARK-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5687:
Summary: in TaskResultGetter need to catch OutOfMemoryError. (was: in
TaskResultGetter need to
Lianhui Wang created SPARK-5653:
---
Summary: in ApplicationMaster rename isDriver to isClusterMode
Key: SPARK-5653
URL: https://issues.apache.org/jira/browse/SPARK-5653
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5653:
Component/s: YARN
in ApplicationMaster rename isDriver to isClusterMode
[
https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14305396#comment-14305396
]
Lianhui Wang commented on SPARK-5594:
-
@John Sandiford, now you provide logs of
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5529:
Comment: was deleted
(was: OK, I will address a PR later and then please help to review code.)
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304810#comment-14304810
]
Lianhui Wang commented on SPARK-5529:
-
OK, I will address a PR later and then please
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304811#comment-14304811
]
Lianhui Wang commented on SPARK-5529:
-
OK, I will address a PR later and then please
[
https://issues.apache.org/jira/browse/SPARK-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5593:
Description:
More strictly, in ExecutorAllocationListener, we need to replace
onBlockManagerAdded,
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14306378#comment-14306378
]
Lianhui Wang commented on SPARK-5529:
-
OK, [~sandyr] thanks.
Executor is still hold
Lianhui Wang created SPARK-5593:
---
Summary: Replace onBlockManagerAdded, onBlockManagerRemoved with
SparkListenerExecutorRemoved,SparkListenerExecutorAdded in
ExecutorAllocationListener
Key: SPARK-5593
URL:
[
https://issues.apache.org/jira/browse/SPARK-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5593:
Summary: Replace onBlockManager listener with Executor listener in
ExecutorAllocationListener
[
https://issues.apache.org/jira/browse/SPARK-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5593:
Summary: Replace BlockManager listener with Executor listener in
ExecutorAllocationListener (was:
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304509#comment-14304509
]
Lianhui Wang commented on SPARK-5529:
-
the phenomenon is:
blockManagerSlave is timeout
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304509#comment-14304509
]
Lianhui Wang edited comment on SPARK-5529 at 2/4/15 2:39 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304509#comment-14304509
]
Lianhui Wang edited comment on SPARK-5529 at 2/4/15 2:40 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304509#comment-14304509
]
Lianhui Wang edited comment on SPARK-5529 at 2/4/15 2:37 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304509#comment-14304509
]
Lianhui Wang edited comment on SPARK-5529 at 2/4/15 2:27 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14301236#comment-14301236
]
Lianhui Wang commented on SPARK-5529:
-
i think when akka timeout time
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5529:
Comment: was deleted
(was: if set spark.akka.timeout=100, why executor is lost after half a hour. i
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302771#comment-14302771
]
Lianhui Wang commented on SPARK-5529:
-
if set spark.akka.timeout=100, why executor is
[
https://issues.apache.org/jira/browse/SPARK-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302772#comment-14302772
]
Lianhui Wang commented on SPARK-5529:
-
if set spark.akka.timeout=100, why executor is
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296651#comment-14296651
]
Lianhui Wang commented on SPARK-5162:
-
[~vgrigor] I have created JIRE:SPARK-5479 for
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296652#comment-14296652
]
Lianhui Wang edited comment on SPARK-5162 at 1/29/15 11:20 AM:
[
https://issues.apache.org/jira/browse/SPARK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5479:
Description:
In SPARK-5162 [~vgrigor] report it:
Now following code cannot work:
aws emr add-steps
[
https://issues.apache.org/jira/browse/SPARK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5479:
Description:
In SPARK-5162 [~vgrigor] reports this:
Now following code cannot work:
aws emr
[
https://issues.apache.org/jira/browse/SPARK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5479:
Description:
In SPARK-5162 [~vgrigor] reports it:
Now following code cannot work:
aws emr
[
https://issues.apache.org/jira/browse/SPARK-5479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5479:
Description:
In SPARK-5162 [~vgrigor] reports this:
Now following code cannot work:
aws emr
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296652#comment-14296652
]
Lianhui Wang edited comment on SPARK-5162 at 1/29/15 10:03 AM:
Lianhui Wang created SPARK-5479:
---
Summary: PySpark on yarn mode need to support non-local python
files
Key: SPARK-5479
URL: https://issues.apache.org/jira/browse/SPARK-5479
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296652#comment-14296652
]
Lianhui Wang edited comment on SPARK-5162 at 1/29/15 11:24 AM:
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5162:
Comment: was deleted
(was: [~vgrigor] I have created JIRE:SPARK-5479 for support non-local files.
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296652#comment-14296652
]
Lianhui Wang commented on SPARK-5162:
-
Vladimir Grigor I have created JIRE:SPARK-5479
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296375#comment-14296375
]
Lianhui Wang commented on SPARK-5162:
-
The previous problem is that packages.egg is
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5162:
Comment: was deleted
(was: The previous problem is that packages.egg is not found. but from your
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14296376#comment-14296376
]
Lianhui Wang commented on SPARK-5162:
-
The previous problem is that packages.egg is
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293271#comment-14293271
]
Lianhui Wang commented on SPARK-5162:
-
[~vgrigor] thank you for asking some questions.
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14294567#comment-14294567
]
Lianhui Wang commented on SPARK-5162:
-
[~vgrigor] i think that a quick way to resolve
Lianhui Wang created SPARK-5369:
---
Summary: remove allocatedHostToContainersMap.synchronized in
YarnAllocator
Key: SPARK-5369
URL: https://issues.apache.org/jira/browse/SPARK-5369
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283809#comment-14283809
]
Lianhui Wang edited comment on SPARK-4630 at 1/20/15 1:28 PM:
--
[
https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283809#comment-14283809
]
Lianhui Wang commented on SPARK-4630:
-
I think it is better that we use stage's output
Lianhui Wang created SPARK-5266:
---
Summary: numExecutorsFailed should exclude number of killExecutor
in yarn mode
Key: SPARK-5266
URL: https://issues.apache.org/jira/browse/SPARK-5266
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang closed SPARK-5266.
---
Resolution: Fixed
numExecutorsFailed should exclude number of killExecutor in yarn mode
[
https://issues.apache.org/jira/browse/SPARK-5162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272943#comment-14272943
]
Lianhui Wang commented on SPARK-5162:
-
[~dklassen] i submit a PR for this
[
https://issues.apache.org/jira/browse/SPARK-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5173:
Description: now when we run python application on yarn cluster mode
through spark-submit,
[
https://issues.apache.org/jira/browse/SPARK-5173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-5173:
Summary: support python application running on yarn cluster mode (was:
support python running on
Lianhui Wang created SPARK-5173:
---
Summary: support python running on yarn cluster mode
Key: SPARK-5173
URL: https://issues.apache.org/jira/browse/SPARK-5173
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270444#comment-14270444
]
Lianhui Wang commented on SPARK-2387:
-
[~xuefuz] [~sandyr] [~lirui]
yes, i think
[
https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270444#comment-14270444
]
Lianhui Wang edited comment on SPARK-2387 at 1/9/15 2:44 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270444#comment-14270444
]
Lianhui Wang edited comment on SPARK-2387 at 1/9/15 2:44 AM:
-
[
https://issues.apache.org/jira/browse/SPARK-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14269275#comment-14269275
]
Lianhui Wang commented on SPARK-4955:
-
yes, because YarnSchedulerActor cannot connect
[
https://issues.apache.org/jira/browse/SPARK-4585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14265915#comment-14265915
]
Lianhui Wang commented on SPARK-4585:
-
yes, i think initial executors number can be
Lianhui Wang created SPARK-4994:
---
Summary: Cleanup removed executors' ShuffleInfo in yarn shuffle
service
Key: SPARK-4994
URL: https://issues.apache.org/jira/browse/SPARK-4994
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-4994:
Description: when the application is completed, yarn's nodemanager can
remove application's
[
https://issues.apache.org/jira/browse/SPARK-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231161#comment-14231161
]
Lianhui Wang commented on SPARK-4644:
-
Shixiong Zhu yes, i agree with you. i will take
[
https://issues.apache.org/jira/browse/SPARK-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231161#comment-14231161
]
Lianhui Wang edited comment on SPARK-4644 at 12/2/14 8:48 AM:
--
[
https://issues.apache.org/jira/browse/SPARK-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231161#comment-14231161
]
Lianhui Wang edited comment on SPARK-4644 at 12/2/14 8:48 AM:
--
[
https://issues.apache.org/jira/browse/SPARK-4644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14228279#comment-14228279
]
Lianhui Wang commented on SPARK-4644:
-
hi @Shixiong Zhu, with skew data, can we use
Lianhui Wang created SPARK-4621:
---
Summary: when sort- based shuffle, Cache recently used shuffle
index can reduce indexFile's io
Key: SPARK-4621
URL: https://issues.apache.org/jira/browse/SPARK-4621
[
https://issues.apache.org/jira/browse/SPARK-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-4621:
Summary: when sort- based shuffle, Cache recently finished shuffle index
can reduce indexFile's io
[
https://issues.apache.org/jira/browse/SPARK-4621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-4621:
Priority: Minor (was: Critical)
when sort- based shuffle, Cache recently finished shuffle index
[
https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227280#comment-14227280
]
Lianhui Wang commented on SPARK-4630:
-
i am interesting in this. and we can learn some
[
https://issues.apache.org/jira/browse/SPARK-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227280#comment-14227280
]
Lianhui Wang edited comment on SPARK-4630 at 11/27/14 5:34 AM:
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225773#comment-14225773
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] that 's very great. With patch #3465, I
Lianhui Wang created SPARK-4534:
---
Summary: With YARN, JavaSparkContext provide to add
preferredNodeLocalityData to SparkContext
Key: SPARK-4534
URL: https://issues.apache.org/jira/browse/SPARK-4534
[
https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-4534:
Description:
example:
SparkConf sparkConf=new SparkConf();
Configuration conf = new
[
https://issues.apache.org/jira/browse/SPARK-4534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-4534:
Description:
example:
SparkConf sparkConf=new SparkConf();
Configuration conf = new
[
https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14220423#comment-14220423
]
Lianhui Wang commented on SPARK-2089:
-
yes, i think we need to add a Spark property
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203316#comment-14203316
]
Lianhui Wang commented on SPARK-2468:
-
ok, thanks.[~adav] i will try to do as you say.
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201778#comment-14201778
]
Lianhui Wang commented on SPARK-2468:
-
[~zzcclp] in am's log, you can find this log:
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202056#comment-14202056
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] yes,with
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202056#comment-14202056
]
Lianhui Wang edited comment on SPARK-2468 at 11/7/14 2:01 PM:
--
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200096#comment-14200096
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] i use your branch and memory overhead
[
https://issues.apache.org/jira/browse/SPARK-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201754#comment-14201754
]
Lianhui Wang commented on SPARK-2468:
-
[~adav] when i set the spark config
Lianhui Wang created SPARK-2687:
---
Summary: after receving allocated containers,amClient should
remove ContainerRequest.
Key: SPARK-2687
URL: https://issues.apache.org/jira/browse/SPARK-2687
Project:
Lianhui Wang created SPARK-2666:
---
Summary: when task is FetchFailed cancel running tasks of
failedStage
Key: SPARK-2666
URL: https://issues.apache.org/jira/browse/SPARK-2666
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-2618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianhui Wang updated SPARK-2618:
Description:
we use shark server to do interative query. every sql run with a job. sometimes
we
Lianhui Wang created SPARK-2648:
---
Summary: through shuffling blocksByAddress avoid much reducers to
fetch data from a executor at a time
Key: SPARK-2648
URL: https://issues.apache.org/jira/browse/SPARK-2648
Lianhui Wang created SPARK-2618:
---
Summary: use config spark.scheduler.priority for specifying
TaskSet's priority on DAGScheduler
Key: SPARK-2618
URL: https://issues.apache.org/jira/browse/SPARK-2618
Lianhui Wang created SPARK-2301:
---
Summary: add ability to submit multiple jars for Driver
Key: SPARK-2301
URL: https://issues.apache.org/jira/browse/SPARK-2301
Project: Spark
Issue Type:
Lianhui Wang created SPARK-2302:
---
Summary: master should discard exceeded completedDrivers
Key: SPARK-2302
URL: https://issues.apache.org/jira/browse/SPARK-2302
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-1399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971415#comment-13971415
]
Lianhui Wang commented on SPARK-1399:
-
i think the user defined accumulators of every
101 - 181 of 181 matches
Mail list logo
