[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15683958 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15684100 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15701250 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50921018 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50921522 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17689/consoleFull ---

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50927468 QA results for PR 1678:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-08-01 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50930691 @aarondav can you update the name of this PR to reference this sub-task JIRA instead: https://issues.apache.org/jira/browse/SPARK-2791? --- If your project is set up for

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50718100 QA results for PR 1678:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50718425 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50718818 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17565/consoleFull ---

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50724029 QA results for PR 1678:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50785966 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50785949 Both prior failures are due to being unable to bind to ports, which sounds like leftover state from other jenkins runs (especially since they were two entirely disjoint

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50786593 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17585/consoleFull ---

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50793796 QA results for PR 1678:br- This patch PASSES unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682389 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala --- @@ -120,8 +121,7 @@ private[spark] class HashShuffleWriter[K, V](

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682412 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682457 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682590 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682605 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15682607 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala --- @@ -120,8 +121,7 @@ private[spark] class HashShuffleWriter[K, V](

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15683205 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-31 Thread mridulm
Github user mridulm commented on a diff in the pull request: https://github.com/apache/spark/pull/1678#discussion_r15683224 --- Diff: core/src/main/scala/org/apache/spark/storage/BlockObjectWriter.scala --- @@ -147,28 +147,36 @@ private[spark] class DiskBlockObjectWriter(

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread aarondav
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/1678 SPARK-2532: Minimal shuffle consolidation fixes All changes from this PR are by @mridulm and are drawn from his work in #1609. This patch is intended to fix all major issues related to shuffle

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50704721 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17535/consoleFull ---

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50704906 QA results for PR 1678:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50705494 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17537/consoleFull ---

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50705532 @mridulm Please take a look if possible. @mateiz This now interacts with the ExternalSorter stuff, and it's possible it partially helps fix a serialization bug

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread witgo
Github user witgo commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50706018 We can also merge it to 1.0 branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-2532: Minimal shuffle consolidation fixe...

2014-07-30 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1678#issuecomment-50714957 QA tests have started for PR 1678. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17553/consoleFull ---