Jungtaek Lim created SPARK-46062:
------------------------------------
Summary: CTE reference node does not inherit the flag
`isStreaming` from CTE definition node
Key: SPARK-46062
URL: https://issues.apache.org/jira/browse/SPARK-46062
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.0, 3.4.1, 3.3.2, 4.0.0
Reporter: Jungtaek Lim
Looks like this is a long standing bug.
We figured out that CTE reference node would never set the isStreaming flag to
true, regardless of the value for flag in CTE definition. The node cannot
determine the right value of isStreaming flag by itself (likewise it cannot
determine about resolution by itself) but it has no parameter in constructor,
hence always takes the default (no children, so batch one).
This may impact some rules which behaves differently depending on isStreaming
flag. It would no longer be a problem once CTE reference is replaced with CTE
definition at some point in "optimization phase", but all rules in analyzer and
optimizer being triggered before the rule takes effect may be impacted.
We probably couldn't sync the flag in real time, but we should sync the flag
when we mark CTE reference to be "resolved". The rule `ResolveWithCTE` will be
a good place to do that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
