sunchao commented on code in PR #5595:
URL: https://github.com/apache/datafusion-comet/pull/5595#discussion_r3898426494
##########
spark/src/main/scala/org/apache/comet/serde/datetime.scala:
##########
@@ -324,12 +323,7 @@ object CometUnixTimestamp extends
CometExpressionSerde[UnixTimestamp] {
expr: UnixTimestamp,
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
- if (!isSupportedInputType(expr)) {
- val inputType = expr.children.head.dataType
- withFallbackReason(expr, s"unix_timestamp does not support input type:
$inputType")
- return None
- }
-
+ // getSupportLevel reports an unsupported input type before reaching here,
so no re-check.
Review Comment:
[P2] Keep string-input rejection ahead of the collation opt-in
On Spark 4.0, `unix_timestamp(s, collate('yyyy-MM-dd', 'UTF8_LCASE'))` with
a non-foldable plain `STRING` column `s` makes `getSupportLevel` return
`Incompatible` before it checks `isSupportedInputType`. With
`spark.comet.expression.UnixTimestamp.allowIncompatible=true`, `exprToProto`
therefore calls this method. The deleted guard returned `None` for that input.
This now serializes the string child into the native UnixTimestamp expression.
The format is not serialized, so no collated scan or Collate serializer is
required. Native `SparkUnixTimestamp` only accepts date/timestamp inputs, and
its unsupported-string error propagates instead of falling back to Spark. Could
we retain the input-type rejection even when collation selects `Incompatible`,
and cover this live-string/collated-format case in a regression test? This path
is source-traced. The SQL witness was not executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
