[
https://issues.apache.org/jira/browse/SPARK-48597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17854291#comment-17854291
]
Jungtaek Lim commented on SPARK-48597:
--------------------------------------
Will submit a PR sooner.
> Distinguish the streaming nodes from the text representation of logical plan
> ----------------------------------------------------------------------------
>
> Key: SPARK-48597
> URL: https://issues.apache.org/jira/browse/SPARK-48597
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> We had a hard time to figure out whether the nodes are streaming or not, when
> we debugged the issue https://issues.apache.org/jira/browse/SPARK-47305 .
> Plan text for logical plan does not show the property of isStreaming, hence
> we had to speculate the value based on other context. In addition, even
> though the type of leaf node is explicitly meant to be streaming which
> enables us to track down the isStreaming for certain subtree, the plan could
> be very long and it’s a non-trivial effort to trace down to the leaf nodes.
> Also, if the leaf nodes are skipped on the representation due to the size,
> there is no way to get the information of isStreaming.
> We propose to introduce a marker of the representation for streaming, which
> will be shown in the text logical plan. There is no concept of "isStreaming"
> in physical plan, so the change only needs to happen in logical plan.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
