Jungtaek Lim created SPARK-53941:
------------------------------------
Summary: Support AQE in stateless streaming workloads
Key: SPARK-53941
URL: https://issues.apache.org/jira/browse/SPARK-53941
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 4.1.0
Reporter: Jungtaek Lim
We have been disabling AQE for streaming workloads since stateful operators do
not work with AQE. But applying AQE in stateless workloads is still beneficial
and we have been blocking the case too aggressively.
We have been observed streaming workloads which can enjoy the benefits of AQE
e.g. stream-static join with large static table, MERGE INTO in ForeachBatch
sink, etc. To accelerate the workloads, we would want to support AQE in
stateless streaming workloads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
