[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-213232245 @davies Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/10024 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-213017267 Merging this into master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212928123 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212928119 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212927052 **[Test build #56532 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56532/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212880975 **[Test build #56532 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56532/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212840033 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212840035 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212839853 **[Test build #56513 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56513/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212799855 **[Test build #56513 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56513/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212798899 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212798080 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212798081 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212797906 **[Test build #56496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56496/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60540340 --- Diff: core/src/main/scala/org/apache/spark/util/collection/Spillable.scala --- @@ -100,6 +103,27 @@ private[spark] trait Spillable[C] extends

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212793811 @davies I also have run unit tests with big number N. How about add a config "spark.shuffle.spill.reservedMemory"? its default is true. It does not force to

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread zzcclp
Github user zzcclp commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212790206 sorry, I mistakenly deleted my comment. what a pity, i can only merge this pr manually. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212786217 This is a big change, maybe not. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread zzcclp
Github user zzcclp commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212781140 @davies , will this pr be backported to branch-1.6? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212778562 @lianhuiwang Have you run some stress tests with latest change? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212777903 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60533610 --- Diff: core/src/main/scala/org/apache/spark/util/collection/Spillable.scala --- @@ -100,6 +103,27 @@ private[spark] trait Spillable[C] extends Logging {

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-21 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212768557 **[Test build #56496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56496/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212754891 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212754896 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212754203 **[Test build #56475 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56475/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212748052 @davies Now all tests have been passed. So Could you take a look again? Thanks. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212747490 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212747494 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212747343 **[Test build #56466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56466/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212746211 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212746210 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212746062 **[Test build #56462 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56462/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212739055 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212739052 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212738764 **[Test build #56459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56459/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212733142 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212733139 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212732804 **[Test build #56456 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56456/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212727564 **[Test build #56475 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56475/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212727414 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212717830 test it please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212713464 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212713463 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212713437 **[Test build #56465 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56465/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212711564 **[Test build #56466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56466/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212709653 **[Test build #56465 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56465/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212708444 **[Test build #56462 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56462/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212707902 @davies Yes, I have update it using object.lock. I will rebased to master. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212702375 Could you also rebased to master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212702206 @lianhuiwang I don't think this is enough. If it's not easy to make it thread safe, one option could be not do forced spilling if spill() will called from

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212698434 **[Test build #56459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56459/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212697127 @davies Now i make some variables volatile to avoid thread safe. Can you take a look? Thanks. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212695755 **[Test build #56456 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56456/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60499732 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -727,4 +779,64 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212632443 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212632448 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212631560 **[Test build #56397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56397/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212629142 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212629144 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212628881 **[Test build #56398 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56398/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212579832 **[Test build #56398 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56398/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212578347 **[Test build #56397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56397/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212496143 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212496147 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212495728 **[Test build #56365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56365/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-20 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212447578 **[Test build #56365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56365/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212059825 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212059821 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212059742 **[Test build #56246 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56246/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-212054901 **[Test build #56246 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56246/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211953148 @squito Yes, I think your understanding is correct. this PR only support that a Spillables will be called once. The code 'val sort = new Spillable()

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211937421 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211937417 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211937366 **[Test build #56229 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56229/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211936269 @davies Thanks. I have added a SpillableIterator that can make consumer and spill thread safe. I think you can take a look at it. --- If your project is set up

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211934985 **[Test build #56229 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56229/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211500937 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211500934 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211500764 **[Test build #56099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56099/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211488135 **[Test build #56099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/56099/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211480633 @lianhuiwang Thanks for working on this, I think it's in the good direction. Two things left: 1) thread safety. For example, you will have two threads for

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60093931 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -689,11 +760,18 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60093876 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -235,6 +237,52 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/10024#discussion_r60092037 --- Diff: core/src/main/java/org/apache/spark/memory/MemoryConsumer.java --- @@ -130,4 +130,22 @@ protected void freePage(MemoryBlock page) { used -=

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-04-18 Thread squito
Github user squito commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-211424963 Hi @lianhuiwang thanks for submitting this patch. I just have a really high-level question first. If I understand how this works correctly, the idea is that:

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174271874 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174271875 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174271872 **[Test build #49948 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49948/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174274054 **[Test build #49951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49951/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174274237 **[Test build #49951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49951/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174274240 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174274239 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2016-01-24 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-174271789 **[Test build #49948 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/49948/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160438090 **[Test build #46852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46852/consoleFull)** for PR 10024 at commit

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160438161 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160438158 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-29 Thread lianhuiwang
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160419872 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-4452][Core]Shuffle data structures can ...

2015-11-29 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/10024#issuecomment-160420970 **[Test build #46852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46852/consoleFull)** for PR 10024 at commit

  1   2   >