Jungtaek Lim created SPARK-41198:
------------------------------------
Summary: Streaming query metrics is broken with CTE
Key: SPARK-41198
URL: https://issues.apache.org/jira/browse/SPARK-41198
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 3.3.1, 3.2.2, 3.4.0
Reporter: Jungtaek Lim
We have observed a case the metrics are not available for the streaming query
which contains CTE.
Looks like CTE was inlined in analysis phase in Spark 3.1.x and it was changed
to be inlined in optimization phase in Spark 3.2.x. ProgressReporter depends on
analyzed plan, hence the change made ProgressReporter to see CTE nodes, which
ends up with having different number of leaf nodes between analyzed plan and
executed plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
