andygrove opened a new issue, #5585:
URL: https://github.com/apache/datafusion-comet/issues/5585
Part of #5572.
`CometStringTranslate` reports `Incompatible` unconditionally:
```scala
override def getSupportLevel(expr: StringTranslate): SupportLevel =
Incompatible(
Some(incompatReason))
```
`spark/src/main/scala/org/apache/comet/serde/strings.scala:119-120`
The reason is real — DataFusion's `translate` iterates Unicode graphemes
where Spark uses code points, and substitutes U+0000 rather than treating it as
a deletion sentinel. But with no `CodegenDispatchFallback` mixin, the
consequence is that `translate` falls the whole projection back to Spark unless
the user sets `spark.comet.expression.StringTranslate.allowIncompatible=true`
and accepts wrong answers.
Every comparable string function in the same file already has the better
shape: compatible by default through the dispatcher, native as an explicit
opt-in. See `CometInitCap` (`:146`), `CometStringReplace` (`:177`),
`CometCaseConversionBase` for `upper` / `lower` (`:50`), and the regex family
at `:368`, `:470`, `:508`, `:545`, `:598`.
Fix: mix `CodegenDispatchFallback` into `CometStringTranslate`.
`StringTranslate` is a plain codegen expression over `StringType`, so
`canHandle` admits it.
Once that lands, `docs/source/user-guide/latest/expressions.md` should
report `translate` as Hybrid, and the existing note about the
grapheme/code-point difference becomes an opt-in caveat rather than a
description of default behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
