coderfender commented on code in PR #7457:
URL: https://github.com/apache/datafusion-comet/pull/7457#discussion_r4571919828
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1394,6 +1395,72 @@ object QueryPlanSerde extends Logging with CometExprShim
with CometTypeShim {
}
}
+ /**
+ * Whether `expr` cannot raise an error for any row, so native execution may
evaluate it for
+ * rows that Spark skips. It answers from Spark's semantics, independently
of the native
+ * `cannot_fail` in `case_when.rs`, which judges the physical expressions it
is given. It is
+ * conservative: an allowed expression whose children also cannot fail.
Anything not listed is
+ * treated as able to fail.
+ */
+ def cannotFail(expr: Expression): Boolean = {
+ def numeric: Boolean = expr.dataType.isInstanceOf[NumericType]
+ def nonAnsiNumeric(evalMode: EvalMode.Value): Boolean = evalMode !=
EvalMode.ANSI && numeric
+ // Floating-point arithmetic never overflows into an error, even in ANSI
mode
+ def floating: Boolean = expr.dataType == FloatType || expr.dataType ==
DoubleType
+ val nodeCannotFail = expr match {
+ case _: Attribute | _: BoundReference | _: Literal => true
+ case _: GetStructField => true
+ case _: BinaryComparison | _: And | _: Or | _: Not | _: IsNull | _:
IsNotNull => true
+ case _: In | _: InSet | _: Greatest | _: Least => true
+ case _: If | _: CaseWhen | _: Coalesce | _: IsNaN | _: NaNvl => true
+ case _: KnownFloatingPointNormalized | _: NormalizeNaNAndZero => true
+ case _: Substring | _: Upper | _: Lower | _: Length | _: OctetLength |
_: BitLength => true
+ case _: StringTrim | _: StringTrimLeft | _: StringTrimRight => true
+ case _: StringReplace | _: InitCap => true
+ // These fail only when the result would exceed 2GB, more than a row can
hold
+ case _: ConcatWs => true
+ case c: Concat => c.children.nonEmpty && c.dataType == StringType
+ case _: Reverse => expr.dataType == StringType
+ case _: Year | _: Quarter | _: Month | _: DayOfMonth | _: DayOfWeek | _:
WeekDay |
+ _: DayOfYear | _: Hour | _: Minute | _: Second =>
+ true
+ case _: Murmur3Hash | _: XxHash64 | _: Md5 | _: Sha1 | _: Sha2 | _:
Crc32 => true
+ // Invalid input gives NaN or NULL, and a floating-point floor or ceil
saturates to a long
+ case _: Floor | _: Ceil | _: Sqrt | _: Exp | _: Log | _: Signum => true
+ // Rounding an integer fails on overflow in ANSI mode; a decimal is
never treated as safe
+ case r: Round => roundCannotFail(r.dataType, r.ansiEnabled)
+ case r: BRound => roundCannotFail(r.dataType, r.ansiEnabled)
+ case c: Cast => CometCast.cannotFail(c)
+ case _: UnaryPositive => true
+ case u: UnaryMinus => floating || (!u.failOnError && numeric)
+ case a: Abs => floating || (!a.failOnError && numeric)
+ case a: Add => floating || nonAnsiNumeric(a.evalMode)
+ case s: Subtract => floating || nonAnsiNumeric(s.evalMode)
+ case m: Multiply => floating || nonAnsiNumeric(m.evalMode)
+ case d: Divide => nonAnsiNumeric(d.evalMode)
+ case d: IntegralDivide => nonAnsiNumeric(d.evalMode)
+ case r: Remainder => nonAnsiNumeric(r.evalMode)
+ case p: Pmod => nonAnsiNumeric(p.evalMode)
+ case _ => false
+ }
+ nodeCannotFail && expr.children.forall(cannotFail)
+ }
+
+ private def roundCannotFail(dataType: DataType, ansiEnabled: Boolean):
Boolean =
+ dataType match {
+ case FloatType | DoubleType => true
+ case ByteType | ShortType | IntegerType | LongType => !ansiEnabled
+ case _ => false
+ }
+
+ /**
+ * The first sort key after the first that can fail. Spark's ordering
evaluates a sort key only
+ * for the rows whose earlier keys tie, but native execution evaluates every
key over the whole
+ * batch, so such a key may raise on a row that Spark skips (#7246).
+ */
+ def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder] =
+ sortOrder.drop(1).find(s => !cannotFail(s.child))
Review Comment:
Andy's point about ANSI off is right, and I missed it when I approved. With
ANSI off, which is the default on Spark 3.5, a later key like `date_add(d, 1)`,
`lpad(s, 5)`, `array_contains(a, x)` or `to_date(s)` now falls back because
`cannotFail` treats anything it doesn't list as able to fail. All of those ran
natively in 1.2, and with ANSI off they return NULL rather than raising, so
that's a performance regression against the release.
One shape that can't be worse than 1.2 is to keep the allow-list under ANSI,
and outside it only fall back for expressions that raise whatever the mode. 1.2
ran every top-K natively, so a list like that only adds safety on top of the
release, even if it misses something. Roughly:
```suggestion
def laterSortKeyThatCanFail(sortOrder: Seq[SortOrder]): Option[SortOrder]
= {
val canFail: Expression => Boolean =
if (SQLConf.get.ansiEnabled) e => !cannotFail(e) else raisesInAnyMode
sortOrder.drop(1).find(s => canFail(s.child))
}
/**
* With ANSI off, whether `expr` can still raise on a row that Spark's
ordering skips. 1.2 ran
* every top-K natively, so this only adds safety over the release even if
it is incomplete.
*/
private def raisesInAnyMode(expr: Expression): Boolean = expr.find {
// assert_true is replaced by If(_, null, RaiseError) before planning
case _: RaiseError | _: UserDefinedExpression => true
// element_at(array, 0) raises INVALID_INDEX_OF_ZERO in every mode
case e: ElementAt =>
e.left.dataType.isInstanceOf[ArrayType] && !(e.right.foldable &&
e.right.eval() != 0)
// an invalid pattern raises in every mode
case r: RLike => !r.right.foldable
// the native timestamp and date to string kernels raise even for TRY
(#7471)
case c: Cast => CometCast.failsInAnyMode(c)
case _ => false
}.isDefined
```
`CometCast.failsInAnyMode` doesn't exist yet. It would be the `hasTimestamp
|| dateToString` part of `cannotFail` pulled out, so a LEGACY `CAST(arr AS
STRING)` stays native the way it did in 1.2. Reading the session flag is a
simplification, and you may prefer to look at each expression's eval mode as
the arithmetic arms do. Could you also add a test with ANSI off where keys like
`date_add(d, 1)` and `lpad(s, 5)` stay native, and one where `element_at(arr,
0)` still falls back?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]