andygrove opened a new issue, #5124: URL: https://github.com/apache/datafusion-comet/issues/5124
### What is the problem the feature request solves? `CollectMetricsExec` (`df.observe(...)`) is not supported, and because it sits in the middle of a plan the fallback is disproportionately expensive: it splits the surrounding native stage in two, adding two extra columnar/row transitions on top of running the observation itself in Spark. `df.observe` is the standard hook for data-quality instrumentation in ETL pipelines (row counts, null counts, constraint checks), so pipelines that add observability today pay for it twice. ### Describe the potential solution A native pass-through operator that forwards batches unchanged while computing the observed aggregate expressions, reporting results back through the existing native metrics / accumulator plumbing so `QueryExecutionListener` observation events still fire with the same values. The aggregate expressions allowed in `observe` are restricted (no distinct, no non-deterministic), which keeps the surface small. The main win is not the aggregation cost, it is restoring stage fusion so the operators above and below stay in one native stage. ### Additional context Found while completing the operator coverage table in #5111. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
