Eric Marnadi created SPARK-55122:
------------------------------------
Summary: Allow seamless and sequential source switching for
streaming queries
Key: SPARK-55122
URL: https://issues.apache.org/jira/browse/SPARK-55122
Project: Spark
Issue Type: Epic
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Eric Marnadi
Users may have multiple sources of data, for example a backfill and live
source, that they want to process one after the other. We propose the
`.followedBy` API where the user can specify multiple stateless streaming DFs
that will execute one after the other, which will hydrate the same downstream
stateful operators with no need for manual checkpoint intervention.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
