Jungtaek Lim created SPARK-48597:
------------------------------------
Summary: Distinguish the streaming nodes from the text
representation of logical plan
Key: SPARK-48597
URL: https://issues.apache.org/jira/browse/SPARK-48597
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.0.0
Reporter: Jungtaek Lim
We had a hard time to figure out whether the nodes are streaming or not, when
we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 .
Plan text for logical plan does not show the property of isStreaming, hence we
had to speculate the value based on other context. In addition, even though the
type of leaf node is explicitly meant to be streaming which enables us to track
down the isStreaming for certain subtree, the plan could be very long and it’s
a non-trivial effort to trace down to the leaf nodes. Also, if the leaf nodes
are skipped on the representation due to the size, there is no way to get the
information of isStreaming.
We propose to introduce a marker of the representation for streaming, which
will be shown in the text logical plan. There is no concept of "isStreaming" in
physical plan, so the change only needs to happen in logical plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
