GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/21385
[SPARK-24234][SS] Support multiple row writers in continuous processing
shuffle reader.
## What changes were proposed in this pull request?
https://docs.google.com/document/d/1IL4kJoKrZWeyIhklKUJqsW-yEN7V7aL05MmM65AYOfE/edit#heading=h.8t3ci57f7uii
Support multiple different row writers in continuous processing shuffle
reader.
Note that having multiple read-side buffers ended up being the natural way
to do this. Otherwise it's hard to express the constraint of sending an epoch
marker only when all writers have sent one.
## How was this patch tested?
new unit tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jose-torres/spark multipleWrite
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21385.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21385
----
commit 1d6b71898e2a640e3c0809695d2b83f3f84eaa38
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-15T18:07:54Z
continuous shuffle read RDD
commit b5d100875932bdfcb645c8f6b2cdb7b815d84c80
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:11:11Z
docs
commit af407694a5f13c18568da4a63848f82374a44377
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:19:37Z
Merge remote-tracking branch 'apache/master' into readerRddMaster
commit 46456dc75a6aec9659b18523c421999debd060eb
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:22:49Z
fix ctor
commit 2ea8a6f94216e8b184e5780ec3e6ffb2838de382
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T03:43:10Z
multiple partition test
commit 955ac79eb05dc389e632d1aaa6c59396835c6ed5
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-17T13:33:51Z
unset task context after test
commit 8cefb724512b51f2aa1fdd81fa8a2d4560e60ce3
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:00:05Z
conf from RDD
commit f91bfe7e3fc174202d7d5c7cde5a8fb7ce86bfd3
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:00:44Z
endpoint name
commit 259029298fc42a65e8ebb4d2effe49b7fafa96f1
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:02:08Z
testing bool
commit 859e6e4dd4dd90ffd70fc9cbd243c94090d72506
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:22:10Z
tests
commit b23b7bb17abe3cbc873a3144c56d08c88bc0c963
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:40:55Z
take instead of poll
commit 97f7e8ff865e6054d0d70914ce9bb51880b161f6
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T00:58:44Z
add interface
commit de21b1c25a333d44c0521fe151b468e51f0bdc47
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T01:02:37Z
clarify comment
commit 7dcf51a13e92a0bb2998e2a12e67d351e1c1a4fc
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:39:28Z
multiple
commit 154843d799683c5cdfc035033475f223f85f0d66
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:41:18Z
don't use spark conf for the sql conf
commit f0262d0a9d3539bcf8fbdbb248968fd704d1e690
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:54:27Z
end thread
commit 3e7a6f9d31967d9efc618c4d319a9dabd22ae4e5
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:54:54Z
name thread
commit 0a38ced23b7e1a6dfe9588ef0ebf7c071a08055d
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T22:55:14Z
no toString
commit ef34e6e9817274df9378341bfb52105c591a5507
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T23:00:01Z
send method
commit 00f910ea39b76a24e1e21acdf3d6a20fd7784fa9
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-18T23:02:10Z
fix
commit aa463ddfd6dcc860f3a9119afe5db32db06945d8
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-21T16:27:23Z
Merge branch 'readerRddMaster' into multipleWrite
commit 504bf7426acf16cce21c549e494b8149dbaa3774
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-21T17:13:33Z
add test
commit 79c3158fc4bb70eb145503699eb6007a029e0c6c
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-21T20:10:06Z
Merge remote-tracking branch 'apache/master' into multipleWrite
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
