andygrove opened a new issue, #5588: URL: https://github.com/apache/datafusion-comet/issues/5588
Part of #5572. `timestamp_seconds` accepts only int, long, float and double, and falls the whole projection back to Spark for anything else: ```scala case IntegerType | LongType | FloatType | DoubleType => Compatible() case dt => Unsupported(Some(s"timestamp_seconds does not support input type $dt")) ``` `spark/src/main/scala/org/apache/comet/serde/datetime.scala:495-496` The serde's own `getUnsupportedReasons` spells out what that leaves behind: "`DecimalType`, `ByteType`, and `ShortType` fall back to Spark." The decimal case is the one that matters. `timestamp_seconds(decimal_col)` is how you get sub-microsecond-safe epoch conversion in Spark, and `SecondsToTimestamp.doGenCode` has a dedicated decimal branch that does the rounding Spark's semantics require. `DecimalType`, `ByteType` and `ShortType` are all in `CometBatchKernelCodegen.isSupportedDataType`. Fix: mix `CodegenDispatchFallback` into `CometSecondsToTimestamp`. It currently extends `CometScalarFunction[SecondsToTimestamp]`, which is fine — the mixin composes with it, as `CometConcat` and `CometMapFromEntries` already demonstrate. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
