sunchao opened a new issue, #5382: URL: https://github.com/apache/datafusion-comet/issues/5382
## What / Why Native shuffle execution can inline a spill-capable operator, such as a native sort or sort-merge join, beneath `ShuffleWriterExec` in one native plan. The writer and its child operators have separate `CometMetricNode`s and separate SQL spill metrics, but the shuffle writer's task-metric bridge currently publishes only spill counters from the root writer node. As a result, the Spark SQL operator view can show a child operator spilling while the Spark Stages/task view reports only the shuffle writer's own spill volume. This discrepancy already exists independently of #5370, which deliberately limits its fix to native shuffle writer metrics. For example, suppose the shuffle writer spills **32 MiB** to disk and a native sort running beneath it spills another **128 MiB**. The Spark task should report **160 MiB** of disk spill, but currently reports only the writer's **32 MiB**. In-memory spill bytes must also be included when a child operator exposes a trustworthy in-memory counter; compressed disk bytes must not be reused as a substitute for memory spill bytes. ## Proposed scope - Account for spill-capable descendant operators in the native metric tree when reporting task-level spill metrics for unified shuffle plans. - Preserve separate on-disk and in-memory spill semantics. If a child lacks an in-memory spill metric, provide an accurate source rather than copying compressed disk bytes. - Avoid counting the same writer, child metric, or shared/reused accumulator more than once. - Preserve completion-listener ordering so final native metrics are available for successful, failed, and canceled task attempts. - Add focused native sort or sort-merge-join coverage that compares Spark SQL operator spill metrics with Spark task metrics on Spark 3.5 and 4.0. Related: #3996 and #5370. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
