[jira] [Created] (SPARK-19450) Replace askWithRetry with askSync.

jin xing created SPARK-19450: Summary: Replace askWithRetry with askSync. Key: SPARK-19450 URL: https://issues.apache.org/jira/browse/SPARK-19450 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19450) Replace askWithRetry with askSync.

[ https://issues.apache.org/jira/browse/SPARK-19450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19450: - Description: *askSync* is already added in *RpcEndpointRef* (see SPARK-19347 and

[jira] [Updated] (SPARK-19263) DAGScheduler should avoid sending conflicting task set.

[ https://issues.apache.org/jira/browse/SPARK-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19263: - Description: In current *DAGScheduler handleTaskCompletion* code, when *event.reason* is *Success*, it

[jira] [Updated] (SPARK-19263) DAGScheduler should avoid sending conflicting task set.

[ https://issues.apache.org/jira/browse/SPARK-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19263: - Description: In current *DAGScheduler handleTaskCompletion* code, when *event.reason* is *Success*, it

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15859015#comment-15859015 ] jin xing commented on SPARK-18113: -- [~xukun] Can you reproduce the bug with steps above? When

[jira] [Created] (SPARK-19437) ExecutorId in HearbeatReceiverSuite is incorrect.

jin xing created SPARK-19437: Summary: ExecutorId in HearbeatReceiverSuite is incorrect. Key: SPARK-19437 URL: https://issues.apache.org/jira/browse/SPARK-19437 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-19438) executorDataMap should be guarded by CoarseGrainedSchedulerBackend.this.synchronized

jin xing created SPARK-19438: Summary: executorDataMap should be guarded by CoarseGrainedSchedulerBackend.this.synchronized Key: SPARK-19438 URL: https://issues.apache.org/jira/browse/SPARK-19438

[jira] [Updated] (SPARK-19438) executorDataMap should be guarded by CoarseGrainedSchedulerBackend.this.synchronized

[ https://issues.apache.org/jira/browse/SPARK-19438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19438: - Description: Currently when handle *RegisterExecutor* in *CoarseGrainedSchedulerBackend*,

[jira] [Created] (SPARK-19461) Remove some unused imports.

jin xing created SPARK-19461: Summary: Remove some unused imports. Key: SPARK-19461 URL: https://issues.apache.org/jira/browse/SPARK-19461 Project: Spark Issue Type: Bug Components:

[jira] (SPARK-19398) Log in TaskSetManager is not correct

Title: Message Title jin xing created an issue

[jira] (SPARK-17022) Potential deadlock in driver handling message

Title: Message Title jin xing commented on SPARK-17022

[jira] [Comment Edited] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875599#comment-15875599 ] jin xing edited comment on SPARK-18113 at 2/21/17 9:00 AM: --- [~xukun] I got the

[jira] [Updated] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19659: - Description: Currently the whole block is fetched into memory(offheap by default) when shuffle-read. A

[jira] [Created] (SPARK-19659) Fetch big blocks to disk when shuffle-read

jin xing created SPARK-19659: Summary: Fetch big blocks to disk when shuffle-read Key: SPARK-19659 URL: https://issues.apache.org/jira/browse/SPARK-19659 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875599#comment-15875599 ] jin xing commented on SPARK-18113: -- [~xukun] I got the scenario you described. ??After driver retry

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15878389#comment-15878389 ] jin xing commented on SPARK-19659: -- [~irashid] Thanks a lot for your comments. I will file a design pdf

[jira] [Updated] (SPARK-19565) After fetching failed, success of old attempt of stage should be taken as valid.

[ https://issues.apache.org/jira/browse/SPARK-19565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19565: - Description: This is related to SPARK-19263. When fetch failed, stage will be resubmitted. There can be

[jira] [Created] (SPARK-19565) After fetching failed, success of old attempt of stage should be taken as valid.

jin xing created SPARK-19565: Summary: After fetching failed, success of old attempt of stage should be taken as valid. Key: SPARK-19565 URL: https://issues.apache.org/jira/browse/SPARK-19565 Project:

[jira] [Updated] (SPARK-19263) DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion.

[ https://issues.apache.org/jira/browse/SPARK-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19263: - Description: In current *DAGScheduler handleTaskCompletion* code, when *event.reason* is *Success*, it

[jira] [Created] (SPARK-19263) DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion.

jin xing created SPARK-19263: Summary: DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion. Key: SPARK-19263 URL: https://issues.apache.org/jira/browse/SPARK-19263

[jira] [Created] (SPARK-19262) DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion.

jin xing created SPARK-19262: Summary: DAGScheduler should handle stage's pendingPartitions properly in handleTaskCompletion. Key: SPARK-19262 URL: https://issues.apache.org/jira/browse/SPARK-19262

[jira] [Updated] (SPARK-19263) DAGScheduler should avoid sending conflicting task set.

[ https://issues.apache.org/jira/browse/SPARK-19263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19263: - Summary: DAGScheduler should avoid sending conflicting task set. (was: DAGScheduler should handle

[jira] [Updated] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19659: - Attachment: SPARK-19659-design-v1.pdf > Fetch big blocks to disk when shuffle-read >

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884789#comment-15884789 ] jin xing commented on SPARK-19659: -- [~irashid] I've uploaded a design doc, please take a look and give

[jira] [Created] (SPARK-19793) Use clock.getTimeMillis when mark task as finished in TaskSetManager.

jin xing created SPARK-19793: Summary: Use clock.getTimeMillis when mark task as finished in TaskSetManager. Key: SPARK-19793 URL: https://issues.apache.org/jira/browse/SPARK-19793 Project: Spark

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891999#comment-15891999 ] jin xing commented on SPARK-19659: -- [~rxin] [~davies] [~andrewor14] [~joshrosen] I've uploaded a design

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884789#comment-15884789 ] jin xing edited comment on SPARK-19659 at 3/2/17 10:21 AM: --- [~irashid] I've

[jira] [Updated] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19659: - Attachment: SPARK-19659-design-v1.pdf > Fetch big blocks to disk when shuffle-read >

[jira] [Updated] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19659: - Attachment: (was: SPARK-19659-design-v1.pdf) > Fetch big blocks to disk when shuffle-read >

[jira] [Created] (SPARK-19777) Scan runningTasksSet when check speculatable tasks in TaskSetManager.

jin xing created SPARK-19777: Summary: Scan runningTasksSet when check speculatable tasks in TaskSetManager. Key: SPARK-19777 URL: https://issues.apache.org/jira/browse/SPARK-19777 Project: Spark

[jira] [Commented] (SPARK-18820) Driver may send "LaunchTask" before executor receive "RegisteredExecutor"

[ https://issues.apache.org/jira/browse/SPARK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740754#comment-15740754 ] jin xing commented on SPARK-18820: -- [~lins05] Thanks a lot for your comment : ) In our company's

[jira] [Updated] (SPARK-18820) Driver may send "LaunchTask" before executor receive "RegisteredExecutor"

[ https://issues.apache.org/jira/browse/SPARK-18820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-18820: - Description: CoarseGrainedSchedulerBackend will update executorDataMap after receiving

[jira] [Created] (SPARK-18820) Driver may send "LaunchTask" before executor receive "RegisteredExecutor"

jin xing created SPARK-18820: Summary: Driver may send "LaunchTask" before executor receive "RegisteredExecutor" Key: SPARK-18820 URL: https://issues.apache.org/jira/browse/SPARK-18820 Project: Spark

[jira] [Updated] (SPARK-19176) Change bin.xml to be compatible with groupId "org.spark-project.hive"

[ https://issues.apache.org/jira/browse/SPARK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19176: - Description: Running "mvn clean package -DskipTests -Phadoop-2,dist" agains

[jira] [Updated] (SPARK-19176) Change bin.xml to be compatible with groupId "org.spark-project.hive"

[ https://issues.apache.org/jira/browse/SPARK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19176: - Description: Running "mvn clean package -DskipTests -Phadoop-2,dist" agains

[jira] [Commented] (SPARK-19176) Change bin.xml to be compatible with groupId "org.spark-project.hive"

[ https://issues.apache.org/jira/browse/SPARK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817753#comment-15817753 ] jin xing commented on SPARK-19176: -- [~joshrosen] How do you think about this? I can make a pr if

[jira] [Updated] (SPARK-19176) Change bin.xml to be compatible with groupId "org.spark-project.hive"

[ https://issues.apache.org/jira/browse/SPARK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19176: - Attachment: bin.patch > Change bin.xml to be compatible with groupId "org.spark-project.hive" >

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15817067#comment-15817067 ] jin xing commented on SPARK-18113: -- Thanks a lot for your reply~ [~aash]. Your comment is very helpful;

[jira] [Created] (SPARK-19176) Change bin.xml to be compatible with groupId "org.spark-project.hive"

jin xing created SPARK-19176: Summary: Change bin.xml to be compatible with groupId "org.spark-project.hive" Key: SPARK-19176 URL: https://issues.apache.org/jira/browse/SPARK-19176 Project: Spark

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15812002#comment-15812002 ] jin xing commented on SPARK-18113: -- [~aash] Thanks a lot for your reply! I think the ideas we have for

[jira] [Commented] (SPARK-18113) Sending AskPermissionToCommitOutput failed, driver enter into task deadloop

[ https://issues.apache.org/jira/browse/SPARK-18113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15806897#comment-15806897 ] jin xing commented on SPARK-18113: -- [~xq2005], [~aash] I am seeing this issue in my cluster some times.

[jira] [Commented] (SPARK-14660) Executors show up active tasks indefinitely after stage is killed

[ https://issues.apache.org/jira/browse/SPARK-14660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15814479#comment-15814479 ] jin xing commented on SPARK-14660: -- I think this is duplicate with SPARK-13054? > Executors show up

[jira] [Comment Edited] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769384#comment-15769384 ] jin xing edited comment on SPARK-15725 at 12/22/16 7:54 AM:

[jira] [Comment Edited] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769384#comment-15769384 ] jin xing edited comment on SPARK-15725 at 12/22/16 8:01 AM:

[jira] [Comment Edited] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769384#comment-15769384 ] jin xing edited comment on SPARK-15725 at 12/22/16 8:03 AM:

[jira] [Commented] (SPARK-15725) Dynamic allocation hangs YARN app when executors time out

[ https://issues.apache.org/jira/browse/SPARK-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15769384#comment-15769384 ] jin xing commented on SPARK-15725: -- [~b...@cloudera.com] May I ask two questions? 1. "a large stage will

[jira] [Commented] (SPARK-19565) After fetching failed, success of old attempt of stage should be taken as valid.

[ https://issues.apache.org/jira/browse/SPARK-19565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15931735#comment-15931735 ] jin xing commented on SPARK-19565: -- [~kayousterhout] Thanks a lot for notification :) I think this

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15891999#comment-15891999 ] jin xing edited comment on SPARK-19659 at 3/19/17 3:16 PM: --- [~rxin] [~davies]

[jira] [Updated] (SPARK-19973) StagePage should display the number of executors.

[ https://issues.apache.org/jira/browse/SPARK-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19973: - Description: In StagePage the total num of executors are not displayed. Since executorId may not be

[jira] [Updated] (SPARK-19973) StagePage should display the number of executors.

[ https://issues.apache.org/jira/browse/SPARK-19973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19973: - Summary: StagePage should display the number of executors. (was: StagePage should display number of

[jira] [Created] (SPARK-19973) StagePage should display number of executors.

jin xing created SPARK-19973: Summary: StagePage should display number of executors. Key: SPARK-19973 URL: https://issues.apache.org/jira/browse/SPARK-19973 Project: Spark Issue Type:

[jira] [Created] (SPARK-20219) Schedule tasks based on size of input from ScheduledRDD

jin xing created SPARK-20219: Summary: Schedule tasks based on size of input from ScheduledRDD Key: SPARK-20219 URL: https://issues.apache.org/jira/browse/SPARK-20219 Project: Spark Issue Type:

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964508#comment-15964508 ] jin xing commented on SPARK-19659: -- [~irashid] Tracking memory used by Netty by swapping in our own

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587 ] jin xing edited comment on SPARK-19659 at 4/11/17 4:11 PM: --- [~cloud_fan] Thanks

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587 ] jin xing commented on SPARK-19659: -- [~cloud_fan] Thanks a lot for taking look into this and sorry for

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587 ] jin xing edited comment on SPARK-19659 at 4/11/17 4:13 PM: --- [~cloud_fan] Thanks

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964596#comment-15964596 ] jin xing commented on SPARK-19659: -- *bytesShuffleToMemory* is different from *bytesInFlight*.

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964587#comment-15964587 ] jin xing edited comment on SPARK-19659 at 4/11/17 4:11 PM: --- [~cloud_fan] Thanks

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965452#comment-15965452 ] jin xing commented on SPARK-19659: -- Yes, I think it's a good idea to leverage memory manager instead of

[jira] [Commented] (SPARK-20219) Schedule tasks based on size of input from ScheduledRDD

[ https://issues.apache.org/jira/browse/SPARK-20219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960608#comment-15960608 ] jin xing commented on SPARK-20219: -- [~kayousterhout] [~irashid] Thanks a lot for taking look at this :)

[jira] [Updated] (SPARK-20219) Schedule tasks based on size of input from ScheduledRDD

[ https://issues.apache.org/jira/browse/SPARK-20219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20219: - Attachment: screenshot-1.png > Schedule tasks based on size of input from ScheduledRDD >

[jira] [Created] (SPARK-20288) Improve BasicSchedulerIntegrationSuite "multi-stage job"

jin xing created SPARK-20288: Summary: Improve BasicSchedulerIntegrationSuite "multi-stage job" Key: SPARK-20288 URL: https://issues.apache.org/jira/browse/SPARK-20288 Project: Spark Issue Type:

[jira] [Updated] (SPARK-20288) Improve BasicSchedulerIntegrationSuite "multi-stage job"

[ https://issues.apache.org/jira/browse/SPARK-20288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20288: - Description: ShuffleId is determined before job submitted. But it's hard to predict stageId by

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971025#comment-15971025 ] jin xing commented on SPARK-19659: -- [~cloud_fan] I refined the the pr. In current change, I'd propose:

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978394#comment-15978394 ] jin xing commented on SPARK-20426: -- Currently in the code, shuffle-read process is like below: 1.

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20426: - Description: Spark jobs are running on yarn cluster in my warehouse. We enabled the external shuffle

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20426: - Attachment: screenshot-1.png > OneForOneStreamManager occupies too much memory. >

[jira] [Comment Edited] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978345#comment-15978345 ] jin xing edited comment on SPARK-20426 at 4/21/17 9:00 AM: --- That's inside

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978409#comment-15978409 ] jin xing commented on SPARK-20426: -- [~jerryshao] {quote} Brain storm: The problem here is that too many

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978278#comment-15978278 ] jin xing commented on SPARK-20426: -- [~srowen] Thanks a lot for quick reply :) With

[jira] [Comment Edited] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978314#comment-15978314 ] jin xing edited comment on SPARK-20426 at 4/21/17 8:35 AM: --- I posted 2

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978314#comment-15978314 ] jin xing commented on SPARK-20426: -- I posted 2 screenshots. External shuffle service of spark is running

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20426: - Description: Spark jobs are running on yarn cluster in my warehouse. We enabled the external shuffle

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978398#comment-15978398 ] jin xing commented on SPARK-20426: -- [~jerryshao] Thanks a lot for looking into this jira :) My largest

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20426: - Attachment: screenshot-2.png > OneForOneStreamManager occupies too much memory. >

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978345#comment-15978345 ] jin xing commented on SPARK-20426: -- That's inside NodeManager(not application memory). With

[jira] [Commented] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15978431#comment-15978431 ] jin xing commented on SPARK-20426: -- Yes, the applications are requesting too many shuffle blocks and NM

[jira] [Created] (SPARK-20426) OneForOneStreamManager occupies too much memory.

jin xing created SPARK-20426: Summary: OneForOneStreamManager occupies too much memory. Key: SPARK-20426 URL: https://issues.apache.org/jira/browse/SPARK-20426 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.

[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20426: - Issue Type: Improvement (was: Bug) > OneForOneStreamManager occupies too much memory. >

[jira] [Updated] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20333: - Description: In test "don't submit stage until its dependencies map outputs are registered

[jira] [Created] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

jin xing created SPARK-20333: Summary: Fix HashPartitioner in DAGSchedulerSuite Key: SPARK-20333 URL: https://issues.apache.org/jira/browse/SPARK-20333 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20333: - Description: In test "don't submit stage until its dependencies map outputs are registered

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895411#comment-15895411 ] jin xing commented on SPARK-19659: -- [~rxin] Thanks a lot for comment. Tracking average size and also the

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895411#comment-15895411 ] jin xing edited comment on SPARK-19659 at 3/4/17 2:11 AM: -- [~rxin] Thanks a lot

[jira] [Created] (SPARK-19937) Collect metrics of block sizes when shuffle.

jin xing created SPARK-19937: Summary: Collect metrics of block sizes when shuffle. Key: SPARK-19937 URL: https://issues.apache.org/jira/browse/SPARK-19937 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15897244#comment-15897244 ] jin xing commented on SPARK-19659: -- [~irashid] Thanks a lot for your comments, really helpful. I got

[jira] [Updated] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-19659: - Attachment: SPARK-19659-design-v2.pdf > Fetch big blocks to disk when shuffle-read >

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900741#comment-15900741 ] jin xing commented on SPARK-19659: -- [~irashid] [~rxin] I uploaded SPARK-19659-design-v2.pdf, please take

[jira] [Comment Edited] (SPARK-19659) Fetch big blocks to disk when shuffle-read

[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900741#comment-15900741 ] jin xing edited comment on SPARK-19659 at 3/8/17 5:47 AM: -- [~irashid] [~rxin] I

[jira] [Created] (SPARK-21342) Fix DownloadCallback to work well with RetryingBlockFetcher

jin xing created SPARK-21342: Summary: Fix DownloadCallback to work well with RetryingBlockFetcher Key: SPARK-21342 URL: https://issues.apache.org/jira/browse/SPARK-21342 Project: Spark Issue

[jira] [Created] (SPARK-21343) Refine the document for spark.reducer.maxReqSizeShuffleToMem

jin xing created SPARK-21343: Summary: Refine the document for spark.reducer.maxReqSizeShuffleToMem Key: SPARK-21343 URL: https://issues.apache.org/jira/browse/SPARK-21343 Project: Spark Issue

[jira] [Created] (SPARK-21414) Buffer in SlidingWindowFunctionFrame could be big though window is small

jin xing created SPARK-21414: Summary: Buffer in SlidingWindowFunctionFrame could be big though window is small Key: SPARK-21414 URL: https://issues.apache.org/jira/browse/SPARK-21414 Project: Spark

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099916#comment-16099916 ] jin xing commented on SPARK-21445: -- I'm not sure how to reproduce, I will try. >

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16099722#comment-16099722 ] jin xing commented on SPARK-21445: -- With this change, I'm still seeing exception below:

[jira] [Commented] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

[ https://issues.apache.org/jira/browse/SPARK-21530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100236#comment-16100236 ] jin xing commented on SPARK-21530: -- I will send follow-up PR soon. Thanks [~tgraves] > Update

[jira] [Commented] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101224#comment-16101224 ] jin xing commented on SPARK-21445: -- Sorry, I report the exception by mistake. With the change, it works

[jira] [Created] (SPARK-21509) Add a config to enable adaptive query execution only for the last query execution.

jin xing created SPARK-21509: Summary: Add a config to enable adaptive query execution only for the last query execution. Key: SPARK-21509 URL: https://issues.apache.org/jira/browse/SPARK-21509 Project:

[jira] [Created] (SPARK-21262) Stop sending 'stream request' when shuffle blocks.

jin xing created SPARK-21262: Summary: Stop sending 'stream request' when shuffle blocks. Key: SPARK-21262 URL: https://issues.apache.org/jira/browse/SPARK-21262 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21270) Improvement for memory config.

[ https://issues.apache.org/jira/browse/SPARK-21270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16070264#comment-16070264 ] jin xing commented on SPARK-21270: -- cc [~rxin] [~cloud_fan] [~joshrosen] > Improvement for memory

[jira] [Created] (SPARK-21270) Improvement for memory config.

jin xing created SPARK-21270: Summary: Improvement for memory config. Key: SPARK-21270 URL: https://issues.apache.org/jira/browse/SPARK-21270 Project: Spark Issue Type: Bug Components:

  1   2   >