andygrove opened a new issue, #5577: URL: https://github.com/apache/datafusion-comet/issues/5577
Part of #5572. `CometUnixTimestamp` only accepts date and timestamp input; anything else, which in practice means **string**, falls the whole projection back to Spark: ```scala Unsupported(Some(s"unix_timestamp does not support input type: $inputType")) ``` `spark/src/main/scala/org/apache/comet/serde/datetime.scala:319` `unix_timestamp(str, fmt)` on a string column is the common spelling of this function, so the supported case is arguably the rarer one. `UnixTimestamp` is a normal codegen expression, and `StringType` is in `CometBatchKernelCodegen.isSupportedDataType`, so the dispatcher handles it. `to_unix_timestamp` is already Hybrid — `CometToUnixTimestamp` is a `CometCodegenDispatch[ToUnixTimestamp]` (`serde/datetime.scala:929`) — so the two functions, which differ only in argument order, behave completely differently today. The same serde also returns a bare `Incompatible(collationReason)` for collated input (`:314`), which the same mixin would route through the dispatcher rather than falling back. Two things to watch: - The input-type check is duplicated in `convert` (`:329`). Only the `getSupportLevel` copy is reachable by the dispatcher, so the `convert` copy should be removed as part of this change or it will mask the fix. See the prerequisite issue on `convert`-side declines. - Parser-policy behavior. `spark.sql.legacy.timeParserPolicy` is honored by Spark's generated code, so routing through the dispatcher preserves whatever policy the session selected — see the note in `docs/source/contributor-guide/spark_configs_support.md`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
