pengbo commented on issue #24389: [SPARK-27482][SQL][Web UI]: Show BroadcastHashJoinExec numOutputRows statistics info on SparkSQL UI page URL: https://github.com/apache/spark/pull/24389#issuecomment-484111600 > I agree with the use case, but the implementation is too hacky. We need a general approach to propagate the statistics from logical plan to physical plan. How about: 1. Add `stats` field and getter/setter in `SparkPlanStats` (trait of `SparkPlan`) 2. Create a `StatisticsPropagation` `SparkPlan` strategy as the first strategy after `extraPlanningStrategies` that creates a `PlanLater(loginPlan)` with `stats` copied from `LogicalPlan`. To avoid infinity loop, one variable `isStatsPropagated` in `LogicalPlan` may be needed to indicate whether the `LogicalPlan` has been applied the strategy. 3. `SparkPlan.makeCopy` copy the `stats` as well Your feedback will be appreciated. @cloud-fan
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
