Boyang Jerry Peng created SPARK-39585:
-----------------------------------------
Summary: Multiple Stateful Operators
Key: SPARK-39585
URL: https://issues.apache.org/jira/browse/SPARK-39585
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.3.0
Reporter: Boyang Jerry Peng
Currently, Structured Streaming only supports one stateful operator per
pipeline. This constraint excludes Structured Streaming to be used for many
use cases such ones involving chained time window aggregations, chained
stream-stream outer equality join, stream-stream time interval join followed by
time window aggregation. Enabling multiple stateful operators within a
pipeline in Structured Streaming will open up many additional use cases for its
usage as well as be on par with other stream processing engines.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
