[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16603 Merged, thanks for working on this @viirya ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 Thanks! @vanzin @mridulm --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16603 > what is reported by a consumer as used need not match what it releases as freed Right, but if even the sum of the reported used memory is not enough for the new consumer, spilling existin

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16603 @vanzin IIRC what is reported by a consumer as used need not match what it releases as freed memory in spill() right ? That seems to be the assumption made in the code atleast from what I saw. I

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user vanzin commented on the issue: https://github.com/apache/spark/pull/16603 > The consumer which asks more memory will decide it fails or not after this memory acquirement Yeah, but at that point, you've already spilled a bunch of other consumers just to find out yo

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72234/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72234 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72234/testReport)** for PR 16603 at commit [`e41c6bd`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72234 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72234/testReport)** for PR 16603 at commit [`e41c6bd`](https://github.com/apache/spark/commit/e4

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16603 Thanks for the review @vanzin , I will leave this open for a day in case someone else also wants to review; and will commit tomorrow. --- If your project is set up for it, you can reply to this ema

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72221/ Test FAILed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72221/testReport)** for PR 16603 at commit [`e41c6bd`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72221/testReport)** for PR 16603 at commit [`e41c6bd`](https://github.com/apache/spark/commit/e4

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 @vazin Thanks for the review. For "fast fail", looks like we don't check if we obtain enough memory after spilling all consumers including itself. The consumer which asks more memory will de

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 @mridulm Ok. Thanks for the review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user mridulm commented on the issue: https://github.com/apache/spark/pull/16603 LGTM, will wait for @vanzin's comments before commiting in case he has any. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72179/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72179/testReport)** for PR 16603 at commit [`3176756`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72179 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72179/testReport)** for PR 16603 at commit [`3176756`](https://github.com/apache/spark/commit/31

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72154/ Test FAILed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72154/testReport)** for PR 16603 at commit [`3176756`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72154 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72154/testReport)** for PR 16603 at commit [`3176756`](https://github.com/apache/spark/commit/31

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 @mridulm Thanks! I've addressed those comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 ping @vanzin @mridulm Do you have more comments on the latest change? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72009/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72009/testReport)** for PR 16603 at commit [`9f7c8ca`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72009/testReport)** for PR 16603 at commit [`9f7c8ca`](https://github.com/apache/spark/commit/9f

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, o

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72006/ Test FAILed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72006/testReport)** for PR 16603 at commit [`9f7c8ca`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #72006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72006/testReport)** for PR 16603 at commit [`9f7c8ca`](https://github.com/apache/spark/commit/9f

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71989/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71989 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71989/testReport)** for PR 16603 at commit [`083cfb3`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 ping @vanzin I think this is in better shape now. Can you also take a look? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71989 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71989/testReport)** for PR 16603 at commit [`083cfb3`](https://github.com/apache/spark/commit/08

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user samkum commented on the issue: https://github.com/apache/spark/pull/16603 Combined with 16387. On Jan 18, 2017 3:54 PM, "Liang-Chi Hsieh" wrote: > @samkum Just this fix or combined with #16387 >

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 @samkum Just this fix or combined with #16387? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user samkum commented on the issue: https://github.com/apache/spark/pull/16603 This fix also has GC issues. Please find the screenshot attached.  --- If your pro

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user viirya commented on the issue: https://github.com/apache/spark/pull/16603 also cc @vanzin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or i

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71471/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71471/testReport)** for PR 16603 at commit [`070ec51`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71471/testReport)** for PR 16603 at commit [`070ec51`](https://github.com/apache/spark/commit/07

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71446/ Test PASSed. ---

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71446 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71446/testReport)** for PR 16603 at commit [`4c2b7b0`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #16603: [SPARK-19244][Core] Sort MemoryConsumers according to th...

Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16603 **[Test build #71446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71446/testReport)** for PR 16603 at commit [`4c2b7b0`](https://github.com/apache/spark/commit/4c