andygrove commented on code in PR #7529:
URL: https://github.com/apache/datafusion-comet/pull/7529#discussion_r4588126044
##########
spark/src/main/scala/org/apache/comet/serde/QueryPlanSerde.scala:
##########
@@ -1339,22 +1339,56 @@ object QueryPlanSerde extends Logging with
CometExprShim with CometTypeShim {
* a filter or a sort, are treated like a projection, which at worst keeps
them evaluated for
* every row where Spark does not. They are found on the first
`canShortCircuitNulls` call for
* the operator, so an operator without an argument to skip never runs
`EquivalentExpressions`.
- * `None` outside `withSharedSubexpressions`, where they are unknown.
+ * `None` outside `withSharedSubexpressions`, where they are unknown. Only
those that can raise
+ * are kept, see `cannotRaise`.
*/
private val sharedSubexpressions = new
DynamicVariable[Option[SharedSubexpressions]](None)
private final class SharedSubexpressions(expressions: Seq[Expression],
alwaysShared: Boolean) {
- // Find shared nodes in Spark's original trees, then compare them after
decimal promotion.
- lazy val common: Seq[Expression] =
+ // The shared subexpressions that can raise. Find shared nodes in Spark's
original trees, then
+ // compare them after decimal promotion. Promotion wraps decimal
arithmetic in CheckOverflow,
+ // which `cannotRaise` does not list, so that arithmetic is always kept.
+ lazy val raising: Seq[Expression] =
if (alwaysShared || SQLConf.get.subexpressionEliminationEnabled) {
val equivalence = new EquivalentExpressions
expressions.foreach(equivalence.addExprTree(_))
- equivalence.getCommonSubexpressions.map(DecimalPrecision.promote)
+
equivalence.getCommonSubexpressions.map(DecimalPrecision.promote).filterNot(cannotRaise)
} else {
Nil
}
}
+ /**
+ * Whether evaluating `expr` on a row cannot raise, whatever the row holds.
This fails closed:
+ * an expression is assumed to raise unless it is listed. Casts and numeric
arithmetic raise
+ * only in the ANSI eval mode they carry, but some expressions raise in any
mode, such as
+ * `element_at` with the index 0 and the arithmetic of intervals, so those
stay out.
+ */
+ private def cannotRaise(expr: Expression): Boolean =
+ !expr.exists(!nodeCannotRaise(_))
+
+ private def nodeCannotRaise(node: Expression): Boolean = {
+ def plainArithmetic(a: BinaryExpression, mode: EvalMode.Value): Boolean =
+ mode != EvalMode.ANSI && a.dataType.isInstanceOf[NumericType]
+ node match {
+ case _: Literal | _: AttributeReference | _: BoundReference | _:
GetStructField => true
+ case _: BinaryComparison | _: IsNull | _: IsNotNull | _: Not | _: And |
_: Or | _: In |
+ _: InSet | _: IsNaN | _: Coalesce | _: If | _: CaseWhen =>
+ true
+ case _: Length | _: Upper | _: Lower | _: Substring | _: StartsWith | _:
EndsWith |
+ _: Contains =>
+ true
+ case c: Cast => evalMode(c) != CometEvalMode.ANSI
Review Comment:
Good catch, you're right. In 3b0c8f297 the `Cast` arm treats a non-ANSI cast
as unable to raise only when both of its types are numeric, string, boolean,
binary or null, or arrays and structs of those. This is an allowlist rather
than an exclusion list, so it fails closed like the rest. Dates, timestamps
(including NTZ) and intervals are left out, along with maps, since a cast key
can become NULL, and the TIME type. I checked Spark 4.1.1 and 3.4.3: the
date-to-timestamp path goes through `daysToMicros` and `instantToMicros` and
`Math.multiplyExact` with no catch outside TRY. `QueryPlanSerdeSuite` now
asserts that a shared LEGACY string-to-int cast is still skippable, while date
to `TIMESTAMP`, date to `TIMESTAMP_NTZ` and array of date to array of timestamp
are not. That suite and the `AtLeastNNonNulls` tests in `CometExpressionSuite`
pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]