Jose Torres created SPARK-24386:
-----------------------------------
Summary: implement continuous processing coalesce(1)
Key: SPARK-24386
URL: https://issues.apache.org/jira/browse/SPARK-24386
Project: Spark
Issue Type: Sub-task
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Jose Torres
[~marmbrus] suggested this as a good implementation checkpoint. If we do the
shuffle reader and writer correctly, it should be easy to make a custom
coalesce(1) execution for continuous processing using them, without having to
implement the logic for shuffle writers finding out where shuffle readers are
located. (The coalesce(1) can just get the RpcEndpointRef directly from the
reader and pass it to the writers.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
