andygrove opened a new issue, #1958: URL: https://github.com/apache/datafusion-ballista/issues/1958
**Is your feature request related to a problem or challenge?** When diagnosing query performance (e.g. the Ballista-vs-Comet SF100 TPC-H gap), the executor's `LoggingMetricsCollector` logs each completed stage-partition's plan with metrics — but only the **shuffle-writer** wrapper metrics are actually populated: `write_time`, `repart_time`, `spill_time` / `spill_count` / `spill_bytes`, `input_rows`, `output_rows`. There is no visibility into the two costs that dominate a shuffle-heavy query: 1. **Shuffle-read / fetch cost.** `ShuffleReaderExec` carries only `BaselineMetrics` (`elapsed_compute` / `output_rows`). There are no fetch-specific metrics: time spent fetching remote partitions, bytes fetched, number of fetch requests, time blocked acquiring the reduce-side in-flight governor permits (in-flight-bytes / in-flight-requests / per-address, added in #1951), or retry counts. The reader's fetch path is exactly where the shuffle *transport* cost lives, and it is currently unmeasurable. 2. **Inner-operator compute.** `SortExec` / `SortMergeJoinExec` / `DataSourceExec` `elapsed_compute` (and spill) is not surfaced in the executor metrics display, so per-operator time cannot be attributed — only the shuffle-writer wrapper shows up. **Concrete example.** SF100 TPC-H Q9 runs in ~82.7 s and shuffles ~963 M rows across ~12 stages (2 executors × 8 slots). The writer metrics account for only ~72 slot-seconds (~6 % of slot capacity). The remaining ~90 % — shuffle read/fetch and sort/join compute — is invisible in the logs, so it is impossible to tell whether the bottleneck is the shuffle transport, sorting ~963 M rows for the sort-merge joins, or something else. **Describe the solution you'd like** 1. Add fetch metrics to `ShuffleReaderExec` (`send_fetch_partitions` / the fetch path): fetch wall-time, bytes fetched, request count, time blocked acquiring governor permits (in-flight-bytes / in-flight-requests / per-address), and retry count. 2. Surface inner-operator `MetricsSet` (`elapsed_compute`, spill, output rows) in the executor `LoggingMetricsCollector` display, so each operator's contribution is visible — not just the shuffle-writer wrapper. **Describe alternatives you've considered** External profiling (perf, flamegraphs) can attribute CPU but not the fetch/backpressure wait time, which is application-level and only the reader knows. `EXPLAIN ANALYZE` is affected by the same missing metrics. **Additional context** Needed to observe the effect of the reduce-side in-flight governor (#1943 / #1951) — that work bounds the shuffle-read path, but there is currently no metric to measure how much time is spent there. Related to #1319 (EPIC: shuffle file execs improvement). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
