[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46816812 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46818315 Hi Matei, I just updated the code according to your comments. For OrderedRDDFunctions, I only set `KeyOrding` into the shuffle, but not move the code path, so what's

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46825394 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46825395 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16021/ --- If your

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46928371 Thanks for the update! I've merged this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-23 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1064 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-22 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46798422 Hi Matei, thanks for your review, I will update the code soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-21 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1064#discussion_r14051999 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala --- @@ -31,10 +31,24 @@ class HashShuffleReader[K, C](

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-21 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1064#discussion_r14052024 --- Diff: core/src/test/scala/org/apache/spark/ShuffleSuite.scala --- @@ -78,8 +81,11 @@ class ShuffleSuite extends FunSuite with Matchers with

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-21 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1064#discussion_r14052020 --- Diff: core/src/test/scala/org/apache/spark/ShuffleSuite.scala --- @@ -56,8 +56,11 @@ class ShuffleSuite extends FunSuite with Matchers with

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-21 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46767244 Hey Saisai, I noticed one thing that got lost in the move, which is the use of InterruptibleIterator. We need to bring that back to allow cancellation of reduce tasks.

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-19 Thread mateiz
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/1064#discussion_r13955604 --- Diff: core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala --- @@ -31,10 +31,24 @@ class HashShuffleReader[K, C](

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-19 Thread mateiz
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46529414 Hey, sorry, been a bit busy lately but I will take a look soon. At a quick glance it looks pretty good. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-18 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-46511981 Hi @mateiz, would you mind taking a look at this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread jerryshao
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/1064 [SPARK-2124] Move aggregation into shuffle implementations This PR is a sub-task of SPARK-2044 to move the execution of aggregation into shuffle implementations. You can merge this pull request

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45837351 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843613 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: [SPARK-2124] Move aggregation into shuffle imp...

2014-06-12 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1064#issuecomment-45843616 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15705/ --- If your