peterxcli opened a new pull request, #5497: URL: https://github.com/apache/datafusion-comet/pull/5497
## Which issue does this PR close? Closes #5447. ## Rationale for this change #5370 and #5445 bridge native spill metrics into Spark task metrics (`diskBytesSpilled` / `memoryBytesSpilled`), but only for native shuffle write tasks. A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage executed via `CometExecRDD`) still reports `spilled_bytes` only as a SQL operator metric, so the Spark Stages/task view shows zero spill for the task — the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans. ## What changes are included in this PR? - `CometExecRDD.compute` now registers `CometMetricNode.reportSpillMetrics` for the stage's metric tree, before input producers are resolved and before the native iterator is created, so reverse-ordered task completion listeners publish final native metrics first, including for failed attempts (matching the ordering established in #5445). Disk and memory totals stay separate and no memory value is inferred from disk bytes. - Several native executions can run inside one Spark task with overlapping metric trees: `CometMetricNode.fromCometPlan` descends through union/coalesce input boundaries into nested native blocks, and a coalesced partition computes the same tree once per parent partition. All `reportSpillMetrics` registrations in a task (including the existing native shuffle writer path) now claim accumulators from a shared per-task identity seen-set keyed by task attempt id (one set per metric name, cleaned up on task completion), so each spill accumulator is counted exactly once per task while disjoint trees still all report. ## How are these changes tested? New tests in `CometTaskMetricsSuite`: - A unit test registering overlapping, repeated, and disjoint metric trees on one `TaskContext` (for both successful and failed attempts), with final metric values published by a completion listener to verify listener ordering, and the per-task seen-set discarded after completion. - An end-to-end test that a native sort spilling in a shuffle-free result stage reports its `spilled_bytes` as the stage's `diskBytesSpilled`, with `memoryBytesSpilled` staying zero. - An end-to-end test that a failed result-stage attempt (late-row ANSI division failure above a spilling sort) still preserves the disk spill metrics. Existing spill-metric tests in `CometTaskMetricsSuite` and `CometNativeShuffleInputRDDSuite` continue to pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
