andygrove opened a new issue, #5307: URL: https://github.com/apache/datafusion-comet/issues/5307
### What is the problem the feature request solves? `WriteTaskStatsTracker.newRow(filePath: String, row: InternalRow)` is a per-row callback. Comet's native write path has columnar batches, not `InternalRow`s, so `CometWriteFilesExec.recordRows` calls it `n` times with `InternalRow.empty` rather than materializing every row just to hand it straight back. That is exactly right for `BasicWriteTaskStatsTracker`, the only implementation Spark ships, which ignores the row argument and just increments a counter (`BasicWriteStatsTracker.scala`). But a third-party tracker that inspects row contents — a Delta or custom-catalog stats collector, say — would silently compute its statistics over empty rows. Today this logs a warning per non-`BasicWriteTaskStatsTracker` instance. A warning is the honest minimum, but it is not a guarantee. ### Describe the potential solution The obstacle is that a plan-time guard is not possible: `WriteJobDescription.statsTrackers` only exists at execution time, by which point `getSupportLevel` has already accepted the write and there is no way to fall back. Options: 1. Materialize rows only when a non-basic tracker is present — correct, and pays the row-conversion cost solely in the case that needs it. 2. Fail the write with a clear message instead of warning, so nobody gets wrong statistics silently. 3. Find a plan-time signal for the tracker set so the write can fall back to Spark gracefully. Option 1 is the most useful; option 2 is a smaller step if the row-conversion path is not worth building yet. ### Additional context Introduced by #5293, which moved native writes onto Spark's `WriteFilesExec` seam and therefore onto Spark's stats-tracker contract. The old path bypassed the trackers entirely and reported its own metrics, so this is a new obligation rather than a regression. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
