sunchao commented on code in PR #6826:
URL: https://github.com/apache/datafusion-comet/pull/6826#discussion_r4235368916
##########
spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:
##########
@@ -119,10 +121,10 @@ object CometCast
inputs: Seq[Attribute],
binding: Boolean): Option[ExprOuterClass.Expr] = {
val cometEvalMode = evalMode(cast)
- cast.child match {
- case _: Literal =>
- exprToProtoInternal(Literal.create(cast.eval(), cast.dataType),
inputs, binding)
- case _ =>
+ foldLiteralCast(cast) match {
+ case Some(folded) =>
+ exprToProtoInternal(folded, inputs, binding)
+ case None =>
Review Comment:
[P2] Preserve Spark's rejection of malformed floating-point literals on the
failed-fold path. With ANSI enabled and a Parquet table `t(id BIGINT)`
containing `(0), (1), (2)`, run `SELECT id, CASE WHEN id = 1 THEN CAST('NaNF'
AS DOUBLE) ELSE 0D END FROM t`. Spark leaves the cast unfolded and raises
`CAST_INVALID_INPUT` when `id = 1`. This new `None` branch serializes it
natively because string-to-double is `Compatible()`, but
`parse_string_to_float` strips the `F` and accepts `NaN`, so the selected
branch returns a value instead of failing. `'InfinityD'` similarly becomes
infinity. The base rejected these literal casts through Spark's `cast.eval()`,
so this PR newly exposes literal queries to silent acceptance. Please correct
native rejection for these inputs or retain Spark execution for affected failed
folds, and add a conditional regression test.
Evidence: The local Spark 4.1.3 Catalyst probe confirmed that ANSI casts of
`NaNF` to DOUBLE and `InfinityD` to FLOAT throw `CAST_INVALID_INPUT`, survive
ConstantFolding inside CASE/IF, and throw when selected. The PR's unchanged
extracted `foldLiteralCast` returned None for both. An offline Rust probe
compiled the exact `parse_string_to_float` function from this head and printed
`NaNF: f64=Some(NaN), f32=Some(NaN)` and `InfinityD: f64=Some(inf),
f32=Some(inf)`. Source tracing confirms `CometCast.isSupported` admits both
targets and native `spark_cast` sends scalar strings through
`cast_string_to_float_impl`, which accepts those Some values even in ANSI mode.
The relevant Spark rejection logic is present in all five supported-version
sources inspected. Full end-to-end Comet execution was not run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
