L. C. Hsieh created SPARK-49070:
-----------------------------------
Summary: TransformWithStateExec.initialState is rewritten
incorrectly to produce invalid query plan
Key: SPARK-49070
URL: https://issues.apache.org/jira/browse/SPARK-49070
Project: Spark
Issue Type: Bug
Components: SQL, Structured Streaming
Affects Versions: 4.0.0
Reporter: L. C. Hsieh
SPARK-47363 added the support for users to provide initial state for streaming
query. Such query operators like `TransformWithStateExec` might have
`hasInitialState` as false which means the initial state related parameters are
not used. But when query planner applies rules on the query, it will still
apply on the initial state query plan. When `hasInitialState` is false, some
related parameters like `initialStateGroupingAttrs` are invalid and some rules
will use these invalid parameters to transform the initial state query plan.
For example, `EnsureRequirements` may apply invalid Sort and Exchange on the
initial query plan. We encountered these invalid query plan in our extension
rules.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
