Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13690317
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
par
Github user colorant commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13690254
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13688732
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
par
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13688708
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -42,10 +42,11 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45829206
Thanks Reynold. I've now moved the other TODOs into sub-tasks on
https://issues.apache.org/jira/browse/SPARK-2044.
---
If your project is set up for it, you can reply to
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1009
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enab
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45827229
Ok merging this in master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45821959
Merged build finished. All automated tests passed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. I
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45821961
All automated tests passed.
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15699/
---
If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45819661
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45819651
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45819540
Thanks Andrew for the comments. @pwendell @rxin I think this is good to
merge now if it passes tests because other people want to contribute some other
things (e.g. moving
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13683531
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -60,9 +61,10 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
}
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13629622
--- Diff: core/src/main/scala/org/apache/spark/rdd/ShuffledRDD.scala ---
@@ -60,9 +61,10 @@ class ShuffledRDD[K, V, P <: Product2[K, V] : ClassTag](
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13629580
--- Diff: core/src/main/scala/org/apache/spark/rdd/CoGroupedRDD.scala ---
@@ -88,7 +90,7 @@ class CoGroupedRDD[K](@transient var rdds: Seq[RDD[_ <:
Product
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13628183
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software F
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13628104
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleWriter.scala ---
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software F
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45662409
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15631/
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45662407
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45656932
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45656917
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13612262
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (AS
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13608317
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/BlockStoreShuffleFetcher.scala
---
@@ -15,26 +15,26 @@
* limitations under the License.
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13573936
--- Diff: core/src/main/scala/org/apache/spark/SparkEnv.scala ---
@@ -163,13 +164,20 @@ object SparkEnv extends Logging {
def instantiateClass[T](p
Github user colorant commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13531291
--- Diff: core/src/main/scala/org/apache/spark/shuffle/ShuffleManager.scala
---
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13529575
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleManager.scala ---
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundat
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13525201
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Founda
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13525195
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Founda
Github user aarondav commented on a diff in the pull request:
https://github.com/apache/spark/pull/1009#discussion_r13523026
--- Diff:
core/src/main/scala/org/apache/spark/shuffle/hash/HashShuffleReader.scala ---
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foun
Github user jerryshao commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45430013
Hi Matei, since we already have `ShuffleWriter` and `ShuffleManager`, do we
still need to keep `ShuffleBlockManager`, I think two functionalities of
`ShuffleBlockManage
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429980
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429981
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15537/
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429424
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429427
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429416
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have t
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429386
Merged build finished.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not hav
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429387
Refer to this link for build results:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/15536/
---
If your project is set up for it, you can
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429378
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1009#issuecomment-45429375
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/1009
[SPARK-2044] Pluggable interface for shuffles
This is a first cut at moving shuffle logic behind a pluggable interface,
as described at https://issues.apache.org/jira/browse/SPARK-2044, to let us
mo
40 matches
Mail list logo
