malinjawi opened a new pull request, #13165: URL: https://github.com/apache/gluten/pull/13165
## What changes are proposed in this pull request? Stacked on #13158 (its three commits are included; review the last one here). #13158 keeps Delta's conditional `IncrementMetric` counters on Spark, which is exact but leaves MERGE's write projection on the JVM. This PR evaluates the counter in Velox with exact counts, so that projection is offloaded again and MERGE's write stage is back to its pre-#13158 shape. How it counts: Velox already tracks the rows every expression is evaluated on (`numProcessedRows`), `FilterProject` exports those counts per function name (`OperatorStats.expressionStats`, behind `operator_track_expression_stats`), and `switch` evaluates a branch only on the rows that take it. So: - Each Delta metric gets its own pass-through Velox function, `increment_metric_<metric display name>`, registered on first sight during plan conversion (`IncrementMetricFunction`). Non-deterministic, so Velox never constant-folds, shares or dictionary-peels it; no default null behavior, so null inputs still count. - `WholeStageResultIterator` turns the config on and serializes `expressionStats` next to `customStats`; `ProjectMetricsUpdater` credits each metric from its own name. - On the JVM, `IncrementMetricCall` (gluten-substrait) is a `Transformable` pass-through; the Delta project transformer rewrites every `IncrementMetric` into it when the backend reports `supportNativeIncrementMetric`, and `OffloadDeltaProject` then accepts every projection shape. A metric without a usable display name, or two different metrics sharing one name in a projection, falls back. Filters keep #13158's root-stack strip (exact already, and Velox fuses a filter into the project above it, which moves its expression stats). Backends without the capability keep #13158's behavior. No Velox change. ## How was this patch tested? - `IncrementMetricOffloadSuite`: counter naming and the unnamed-metric fallback. - `DeltaSuite` exact-metric tests (MERGE with delete, update and insert clauses; no-op MERGE with CDF; DELETE/UPDATE with and without DVs) now assert that MERGE's `CASE WHEN` projection runs as `DeltaProjectExecTransformer` on Velox. - JVM compiled on the spark-3.4, spark-3.5 and spark-4.0 Delta profiles; the C++ compiles in CI (this machine cannot rebuild the native library). - `delta_spark_ut` against the #13158 baseline: expected zero regressions and zero now-passing. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
