[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-25 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30963780 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-25 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30964306 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-25 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30964120 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-25 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30966102 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -306,14 +297,10 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-24 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30963426 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -306,14 +297,10 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-24 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r30963215 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -306,14 +297,10 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29561399 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -54,9 +54,14 @@ private[spark] abstract class BlockObjectWriter(val

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29561144 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -203,12 +208,32 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29559968 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-03 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29561207 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -54,9 +54,14 @@ private[spark] abstract class BlockObjectWriter(val

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4450 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98054293 Alright let's do it. Thanks for the work Sandy. Let's leave this on for now and we can decide to de-flag it later in the QA cycle if we want. --- If your project is

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98053275 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98053228 [Test build #31512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31512/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98053271 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98235396 Thanks for reviewing and merging @pwendell ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-05-01 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29533251 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -124,11 +125,22 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98038092 LGTM pending a final test run. I want to get this in since there may be some other changes to shuffle interfaces due to some of the binary management stuff. --- If

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98038047 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29486588 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -124,11 +125,22 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98038572 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98038737 [Test build #31512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31512/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-30 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-98038501 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97233359 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97233350 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97233223 [Test build #31160 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31160/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96979314 [Test build #31136 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31136/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96992154 [Test build #31136 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31136/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97178148 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97178070 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-28 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-97178429 [Test build #31160 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31160/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29190908 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29192947 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29191148 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -740,15 +723,29 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29191682 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29193245 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29206748 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96855945 [Test build #31080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31080/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29206723 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -740,15 +723,29 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96871624 [Test build #31080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31080/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96849231 [Test build #31074 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31074/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29171158 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96844031 [Test build #31074 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31074/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-27 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96844045 Updated patch rebases on master and incorporates @pwendell 's review comments. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113464 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113581 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113605 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113602 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112442 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112530 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113435 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113570 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29113616 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112203 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112308 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112469 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -53,9 +53,14 @@ private[spark] abstract class BlockObjectWriter(val

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112559 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112322 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112320 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112547 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112553 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112635 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112697 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112713 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -53,9 +53,14 @@ private[spark] abstract class BlockObjectWriter(val

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29112286 --- Diff: core/src/main/scala/org/apache/spark/util/collection/WritablePartitionedPairCollection.scala --- @@ -0,0 +1,117 @@ +/* + * Licensed to

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29117710 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118071 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -740,15 +723,29 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118289 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -740,15 +723,29 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118405 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118412 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118576 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r29118645 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-26 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-96485873 Hey Sandy, I've now taken a pretty thorough look at this patch. There are a lot of low level comments and it would be nice if you could do a pass to bring this

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-95237844 Hey @sryza - two higher level questions as I'm doing a deeper review of this. 1. This seems predicated on the idea that serialization streams can safely be

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r28884412 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r28882363 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r28892668 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala --- @@ -113,11 +114,21 @@ private[spark] class ExternalSorter[K, V, C](

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r28892612 --- Diff: core/src/main/scala/org/apache/spark/util/collection/PartitionedSerializedPairBuffer.scala --- @@ -0,0 +1,254 @@ +/* + * Licensed to the

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-04-22 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-95272806 Thanks for the review Patrick. Regarding Kryo: I would be really really surprised if Kryo were to change its serialization format in such a drastic way without

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-31 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-87956979 @sryza don't bother with my comments yet, still just taking a tour through this part of the code. --- If your project is set up for it, you can reply to this email and

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-30 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r27423831 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-30 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r27424659 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-30 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r27424016 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-09 Thread sryza
Github user sryza commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r26017595 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-03-02 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/4450#discussion_r25627160 --- Diff: core/src/main/scala/org/apache/spark/util/collection/ChainedBuffer.scala --- @@ -0,0 +1,134 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73627450 [Test build #27152 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27152/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-09 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73627859 Thanks - I'm pretty swamped right now. Will take a look once I free up more (maybe next week) --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-09 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73634417 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-09 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73634412 [Test build #27152 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27152/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73432683 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73432678 [Test build #27051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27051/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73432682 [Test build #27051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27051/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73433482 [Test build #27052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27052/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73434673 [Test build #27050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27050/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73434678 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73436094 [Test build #27054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27054/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73436490 [Test build #27052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27052/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73436493 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73438918 [Test build #27054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27054/consoleFull) for PR 4450 at commit

[GitHub] spark pull request: SPARK-4550. In sort-based shuffle, store map o...

2015-02-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4450#issuecomment-73438921 Test FAILed. Refer to this link for build results (access rights to CI server needed):

  1   2   >