andygrove opened a new pull request, #2042: URL: https://github.com/apache/datafusion-ballista/pull/2042
# Which issue does this PR close? No tracking issue was filed for this change. Happy to open one if preferred. # Rationale for this change After running a distributed query, there is no convenient way for a client to see the physical plan that actually executed on the cluster. `DistributedQueryExec` only holds the logical plan, and until now its display showed a single `scheduler_url=...` line. The distributed physical plan (the per-stage DAG with shuffle boundaries) is built and retained on the scheduler and was only surfaced through `EXPLAIN ANALYZE`, always bundled with runtime metrics. This PR lets a client fetch that executed plan on demand (with or without metrics) and makes the operator's default display more informative. # What changes are included in this PR? - Add `DistributedQueryExec::explain_executed_plan(&self, session_config, with_metrics)`, which fetches the executed per-stage plan for an already-run query via the existing `get_job_metrics` gRPC and returns it as a printable string. `with_metrics = true` matches the `EXPLAIN ANALYZE` output; `false` omits the `metrics=[...]` suffixes. It returns an error if called before the query has executed. - Extract the per-stage plan formatting out of the explain-analyze `RecordBatch` builder into a reusable `format_job_plan(job_metrics, with_metrics)`, and widen `fetch_job_metrics` to `pub(crate)`. Existing `EXPLAIN ANALYZE` output is unchanged (the record-batch builder now delegates to `format_job_plan(.., true)`). - Extend `DistributedQueryExec`'s `DisplayAs` (Default/Verbose) to also render the submitted logical plan via `LogicalPlan::display_indent()`. This is synchronous and makes no scheduler round-trip; the `TreeRender` arm is unchanged. No proto, planner, or scheduler changes. Covered by unit tests (formatter without metrics, display includes logical plan, error-before-execution) and an end-to-end integration test in `context_checks.rs` that runs a query and asserts the executed plan renders with and without metrics. # Are there any user-facing changes? Yes, additive and non-breaking: - New public method `DistributedQueryExec::explain_executed_plan`. - `DistributedQueryExec`'s default display now includes the submitted logical plan in addition to the scheduler URL. No breaking changes to public APIs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
