Rich-T-kid opened a new issue, #25757: URL: https://github.com/apache/datafusion/issues/25757
## Summary When debugging the single-node → distributed planner pipeline in downstream projects like [datafusion-distributed](https://github.com/apache/datafusion-distributed), it is useful to inspect the original physical plan before any transformations are applied. Currently there is no built-in way to observe or log the plan at that stage without adding ad-hoc instrumentation downstream. ## Motivation In datafusion-distributed, adding a local log of the pre-transformation plan was proposed as a way to capture reproducers and understand what the planner receives. However, logging the full plan on every query may be expensive and is arguably not the right place to address this — the concern is really about observability of DataFusion's own plan-rewriting pipeline. Addressing this upstream would let any project that wraps DataFusion's physical planning observe the plan at well-defined checkpoints without duplicating logging logic or paying the cost unconditionally. ## Proposal Consider one of the following approaches: - **A structured tracing span** (e.g. using the `tracing` crate) that includes the serialized or display-formatted plan at the entry point of physical plan optimization/rewriting, gated behind a low-overhead level (e.g. `TRACE`). - **An observer / hook API** on `SessionContext` or `QueryPlanner` that downstream implementors can register to receive the plan before and/or after each rewrite pass. - **A `EXPLAIN`-style plan-snapshot facility** that can be enabled per-session for debugging without affecting production workloads. Any approach should ensure the logging/inspection is opt-in and has negligible overhead when disabled. ## Related - Raised during review of datafusion-distributed (internal discussion on pre-transformation plan logging for reproducer capture). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
