andygrove opened a new pull request, #5031: URL: https://github.com/apache/datafusion-comet/pull/5031
## Which issue does this PR close? There is no dedicated issue. This adds a developer utility for auditing expression coverage. Related to the expression coverage epic #240. ## Rationale for this change Finding Spark expressions that Comet does not accelerate is currently a manual, error-prone exercise. Searching the source is unreliable because Comet dispatches expressions through several mechanisms: the serde maps, the `StaticInvoke` method map, `RuntimeReplaceable` rewrites to other handled expressions, and the Spark 4.x expression shims. A grep-based audit misses all but the first and produces false positives (for example flagging `from_xml`, `aes_encrypt`, or `array_prepend` as unsupported when they are handled via a shim or a rewrite). This adds a runtime audit that determines coverage by actually planning each function and inspecting the physical plan, which accounts for every dispatch path. It doubles as a per-Spark-version coverage report. ## What changes are included in this PR? - `CometExpressionCoverageSuite`: for every function in Spark's registry it runs the function's own documented example (`ExpressionInfo.getExamples`) over a Comet-scanned table, then uses `findFirstNonCometOperator` and `ExtendedExplainInfo.getFallbackReasons` to classify the result. Categories: `NATIVE`, `INCOMPATIBLE` (supported but opt-in), `LITERAL_PROBE` (supported; the all-literal example tripped an all-constant guard that would not fire for column inputs), `GAP` (genuine fallback, with the fallback reason and probed query recorded), `SKIP`, and `ERROR`. The report is written to `spark/target/expression-coverage-report-spark-<version>.md`. - A note in the contributor guide (`adding_a_new_expression.md`) on how to run it and read the report, including running it under each Spark profile to compare versions. On Spark 4.1 it currently reports roughly 330 native, a handful incompatible/opt-in, and a `GAP` list dominated by families already documented as not-planned (variant, geospatial, sketches, reflection) plus a few genuine candidates. ## How are these changes tested? The suite is itself a test and asserts that it probes a non-empty set of functions. It was run against the Spark 4.1 profile to produce the coverage report and validate the classification (including confirming that expressions handled via shims and rewrites are correctly reported as native). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
