andygrove opened a new issue, #5575:
URL: https://github.com/apache/datafusion-comet/issues/5575
Part of #5572.
`CometStaticInvoke` dispatches on an allowlist of `(functionName,
staticObject)` pairs and falls the whole operator back to Spark for anything
not in it:
```scala
case None =>
withFallbackReason(
expr,
s"Static invoke expression: ${expr.functionName} is not supported")
None
```
`spark/src/main/scala/org/apache/comet/serde/statics.scala:61-65`
`StaticInvoke` is a codegen-friendly node — `doGenCode` emits a static
method call — and `CometStaticInvokeCodegenDispatch` (`statics.scala:120`)
already exists and is wired for `aesEncrypt`, `aesDecrypt` and
`StringDecode.decode`. Making it the `case None` default turns a hard fallback
into in-pipeline execution for every lowering nobody has gotten to yet.
The compatibility guide already records the symptom.
`docs/source/user-guide/latest/expressions.md` lists `encode` as "🔜 Lowers to
`StaticInvoke(encode)` (not allowlisted); falls back", and `to_binary` as "Hex
form accelerated; other formats fall back". Both would be covered by the
default.
The same applies to `Invoke`, which has no entry in the serde map at all.
`Spark4xCometExprShim.sparkVersionSpecificExprToProtoInternal` recognizes
`StructsToJsonEvaluator` and `ParseUrlEvaluator` and returns `None` for
everything else
(`spark/src/main/spark-4.x/org/apache/comet/shims/Spark4xCometExprShim.scala:94`),
after which the generic path reports "not supported". Spark 4.x lowers a
growing number of `RuntimeReplaceable` expressions to evaluator-backed `Invoke`
nodes, so the same catch-all is worth having there.
This is safe by construction for the encoder and deserializer trees that
make up most `StaticInvoke` usage in typed Dataset operations: their arguments
and return types are `ObjectType`, which is outside
`CometBatchKernelCodegen.isSupportedDataType`, so `canHandle` rejects them and
they fall back exactly as they do now.
Depends on the closure-serialize guard, since this is the change that starts
handing the dispatcher arbitrary trees.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
