Jungtaek Lim created SPARK-53942:
------------------------------------
Summary: Support changing stateless shuffle partitions upon
restart of streaming query
Key: SPARK-53942
URL: https://issues.apache.org/jira/browse/SPARK-53942
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 4.1.0
Reporter: Jungtaek Lim
We have been having a huge restriction on the number of shuffle partitions in
streaming - once the streaming query runs, there is no way but discard the
checkpoint to change the number of shuffle partitions. There has been
consistent requests for unblocking this.
The main reason of the limitation is due to the fact the stateful operator has
fixed partitions and we should make sure it is unchanged. While the invariant
is not changed, there is no technical reason to also disallow changing of the
number of shuffle partitions in stateless shuffle e.g. stream-static join,
MERGE INTO, etcetc.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
