peterxcli opened a new issue, #5447: URL: https://github.com/apache/datafusion-comet/issues/5447
## What / Why #5370 and #5445 bridge native spill metrics into Spark task metrics (`diskBytesSpilled` / `memoryBytesSpilled`), but only for native shuffle write tasks: `CometMetricNode.reportSpillMetrics` is registered solely on the `CometNativeShuffleInputRDD` path. A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via `CometExecRDD` in `operators.scala`) still reports `spilled_bytes` only as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans. ## Proposed scope - Register a task-level spill reporting callback for native stages executed via `CometExecRDD`, aggregating over the stage's `CometMetricNode` tree as `sumMetricValues` does for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes). - Register it before the native iterator so final metrics are published first, including for failed attempts, matching the ordering established in #5445. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
