andygrove opened a new issue, #5591: URL: https://github.com/apache/datafusion-comet/issues/5591
Part of #5572. Two datetime and string serdes decline non-default collations and fall the whole projection back to Spark: - `CometNextDay` returns `Incompatible(collationReason)` (`spark/src/main/scala/org/apache/comet/serde/datetime.scala:451`) - `CometLevenshtein` returns `Unsupported` (`spark/src/main/scala/org/apache/comet/serde/strings.scala:128-130`) Neither mixes in `CodegenDispatchFallback`. Both are plain codegen expressions over `StringType`, which `CometBatchKernelCodegen.isSupportedDataType` accepts regardless of collation ID, so the dispatcher can run Spark's own collation-aware code inside the pipeline. This is the same treatment the collation-declining serdes elsewhere already get — `CometFromUTCTimestamp` (`datetime.scala:366`), `CometToUTCTimestamp` (`:386`), `CometConvertTimezone` (`:406`), `CometTruncDate` (`:533`), `CometTruncTimestamp` (`:587`), `CometDateFormat` (`:691`) and `CometConcat` (`strings.scala:256`) all carry the mixin. `CometUnixTimestamp` has the same collation arm at `:314` but is covered by its own issue, since its bigger problem is string input. Fix: mix `CodegenDispatchFallback` into `CometNextDay` and `CometLevenshtein`. Worth a sanity check first that collated string columns actually reach Comet operators in the first place, rather than being rejected earlier at the scan or schema-support layer — if they never get that far, these two are dead code and the issue should be closed rather than fixed. `CometCollationSuite` is the place to confirm. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
