andygrove opened a new pull request, #5201:
URL: https://github.com/apache/datafusion-comet/pull/5201
## Which issue does this PR close?
N/A
## Rationale for this change
The extended explain summary reports operator coverage and the number of
Spark/Comet transitions, but says nothing about expressions. There was no
way to
see how much of a plan's expression evaluation ran in native DataFusion
kernels
versus Spark's own generated code inside the JVM codegen dispatcher.
## What changes are included in this PR?
Expression coverage is tracked on the expression tree with a new
`NATIVE_EXPRS`
tag, the counterpart of the existing `CODEGEN_DISPATCH_EXPRS`, and both are
rolled up per operator onto the converted Comet plan node alongside the info
messages. The summary line now ends with the counts:
```
Comet accelerated 55 out of 58 eligible operators (94%). Final plan contains
3 transitions between Spark and Comet. Comet accelerated 14 expressions (14
native, 1 codegen dispatch).
```
Two new accessors on `ExtendedExplainInfo` give the same information as data:
```scala
val native: Seq[String] = info.getNativeExpressions(plan)
val dispatched: Seq[String] = info.getCodegenDispatchExpressions(plan)
```
Notes on the counting:
- Counts are **distinct expression names per plan**. A name can appear in
both
buckets when the same function is lowered natively for one set of
arguments and
dispatched for another, so the total is the union rather than the sum.
`cast` in
the TPC-DS goldens is a real example.
- Structural nodes (attribute references, literals, aliases, bound
references) are
excluded. They appear in nearly every expression tree and would swamp the
names
that matter, and excluding literals also keeps the dispatcher's
closure-serialized payload out of the native count.
- Names are the expression's `prettyName` lowercased (the UDF name for a
`ScalaUDF`), matching the existing `[COMET-INFO: JVM codegen dispatcher:
...]`
naming.
Dispatch tagging is no longer gated on `spark.comet.explain.codegen.enabled`.
That config now controls only whether the `[COMET-INFO: ...]` segment is
rendered, so the counts are available by default. The config is now enabled
in
`CometPlanStabilitySuite` so the goldens record *which* expressions took the
dispatcher path, not just how many — across TPC-DS that is `cast` and
`upper`.
## How are these changes tested?
Two new tests in `CometCodegenSuite` cover the split counts and both
accessors
(including that they are independent of `explain.codegen.enabled`), and the
all-fallback case.
TPC-DS plan stability goldens were regenerated for Spark 3.4, 3.5, 4.0, 4.1
and
4.2. All 158 changed `extended.txt` files differ only in summary lines and
`COMET-INFO` segments — no plan shapes changed. The suites were re-run
against
the regenerated goldens for 3.4, 3.5, 4.0 and the default profile.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
