HeartSaVioR commented on code in PR #52642:
URL: https://github.com/apache/spark/pull/52642#discussion_r2462482710
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/ProgressReporter.scala:
##########
@@ -544,7 +546,10 @@ abstract class ProgressContext(
val finalLogicalPlan = unrollCTE(lastExecution.logical)
val allLogicalPlanLeaves = finalLogicalPlan.collectLeaves() // includes
non-streaming
- val allExecPlanLeaves = lastExecution.executedPlan.collectLeaves()
+ val allExecPlanLeaves = StreamingQueryPlanTraverseHelper
+ .collectFromUnfoldedPlan(lastExecution.executedPlan) {
Review Comment:
That should be OK, as here we only add unwrapping AQE node. It won't be a
noticeable overhead.
(That said, if we could have a new mechanism which does not traverse the
executed plan to collect the accumulators but know accumulators to read after
the execution, that would remove out the cost of traversing the plan. That
would be non-small change.)
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/runtime/ProgressReporter.scala:
##########
@@ -544,7 +546,10 @@ abstract class ProgressContext(
val finalLogicalPlan = unrollCTE(lastExecution.logical)
val allLogicalPlanLeaves = finalLogicalPlan.collectLeaves() // includes
non-streaming
- val allExecPlanLeaves = lastExecution.executedPlan.collectLeaves()
+ val allExecPlanLeaves = StreamingQueryPlanTraverseHelper
+ .collectFromUnfoldedPlan(lastExecution.executedPlan) {
Review Comment:
That should be OK, as here we only add unwrapping AQE node. The change here
won't be a noticeable overhead.
(That said, if we could have a new mechanism which does not traverse the
executed plan to collect the accumulators but know accumulators to read after
the execution, that would remove out the cost of traversing the plan. That
would be non-small change.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
