L. C. Hsieh created SPARK-33814:
-----------------------------------
Summary: Provide preferred locations for stateful operations
without reported state store locations
Key: SPARK-33814
URL: https://issues.apache.org/jira/browse/SPARK-33814
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 3.2.0
Reporter: L. C. Hsieh
Assignee: L. C. Hsieh
Stateful operators in SS provides preferred locations on the previous batches
if any. However, if there is no previous batch to follow, Spark possibly
schedules stateful tasks in inefficient distribution. As stateful operations
probably need to maintain large state stores, it is better we schedule stateful
tasks across all executors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
