Bo Zhang created SPARK-36533:
--------------------------------
Summary: Allow streaming queries with Trigger.Once run in multiple
batches
Key: SPARK-36533
URL: https://issues.apache.org/jira/browse/SPARK-36533
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: Bo Zhang
Currently streaming queries with Trigger.Once will always load all of the
available data in a single batch. Because of this, the amount of data the
queries can process is limited, or Spark driver will be out of memory.
We should allow streaming queries with Trigger.Once run in multiple batches.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
