GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/21428
[SPARK-24235][SS] Implement continuous shuffle write RDD for single reader
partition.
## What changes were proposed in this pull request?
Implement continuous shuffle write RDD for a single reader partition. (I
don't believe any implementation changes are actually required for multiple
reader partitions, but this PR is already very large, so I want to exclude
those for now to keep the size down.)
## How was this patch tested?
new unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jose-torres/spark writerTask
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21428.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21428
----
commit 1d6b71898e2a640e3c0809695d2b83f3f84eaa38
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-15T18:07:54Z
continuous shuffle read RDD
commit b5d100875932bdfcb645c8f6b2cdb7b815d84c80
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:11:11Z
docs
commit af407694a5f13c18568da4a63848f82374a44377
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:19:37Z
Merge remote-tracking branch 'apache/master' into readerRddMaster
commit 46456dc75a6aec9659b18523c421999debd060eb
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:22:49Z
fix ctor
commit 2ea8a6f94216e8b184e5780ec3e6ffb2838de382
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:43:10Z
multiple partition test
commit 955ac79eb05dc389e632d1aaa6c59396835c6ed5
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T13:33:51Z
unset task context after test
commit 8cefb724512b51f2aa1fdd81fa8a2d4560e60ce3
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:00:05Z
conf from RDD
commit f91bfe7e3fc174202d7d5c7cde5a8fb7ce86bfd3
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:00:44Z
endpoint name
commit 259029298fc42a65e8ebb4d2effe49b7fafa96f1
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:02:08Z
testing bool
commit 859e6e4dd4dd90ffd70fc9cbd243c94090d72506
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:22:10Z
tests
commit b23b7bb17abe3cbc873a3144c56d08c88bc0c963
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:40:55Z
take instead of poll
commit 97f7e8ff865e6054d0d70914ce9bb51880b161f6
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:58:44Z
add interface
commit de21b1c25a333d44c0521fe151b468e51f0bdc47
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T01:02:37Z
clarify comment
commit 7dcf51a13e92a0bb2998e2a12e67d351e1c1a4fc
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:39:28Z
multiple
commit ad0b5aab320413891f7c21ea6115b6da8d49ccf9
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-25T00:06:15Z
writer with 1 reader partition
commit c9adee5423c2e8a030911008d2e6942045d484bb
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-25T00:15:39Z
docs and iface
commit 63d38d849107eed226449cec8d24c2241cd583c9
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-25T00:27:26Z
Merge remote-tracking branch 'apache/master' into writerTask
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
