comphead commented on code in PR #7322:
URL: https://github.com/apache/datafusion-comet/pull/7322#discussion_r4607023073


##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -1026,7 +1029,149 @@ object CometFlatten extends 
CometExpressionSerde[Flatten] with ArraysBase {
   }
 }
 
-object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {
+/**
+ * What a serde needs to run the lambda of a higher-order function natively, 
under a DataFusion
+ * higher-order function: the gate on the lambda and its body, and the 
serialization.
+ */
+trait NativeLambdaSerde extends CometTypeShim with CometExprShim {
+
+  // The native lambda has the same semantics as Spark's `filter` for a 
one-parameter lambda whose
+  // body is deterministic, contains no nested lambda, and whose parameter is 
a type the native
+  // side can type-check against the list's element field. `exists` adds a 
condition of its own,
+  // since Spark stops at the first match. It needs bound references: the 
native side re-indexes
+  // the columns the body captures, which it cannot do for unbound ones.
+  protected def nativeLambdaSupported(argument: Expression, body: Expression): 
Boolean =
+    CometConf.COMET_NATIVE_LAMBDA_ENABLED.get() &&
+      body.deterministic &&
+      !body.exists(_.isInstanceOf[HigherOrderFunction]) &&
+      capturesOnlyFixedWidthColumns(body) &&
+      shortCircuitSafe(body) &&
+      !canProduceNullMapKey(argument) &&
+      !canProduceNullMapKey(body) &&
+      !convertsDateTime(body) &&
+      (argument.dataType match {
+        case ArrayType(elementType, _) => isLambdaElementType(elementType)
+        case _ => false
+      })
+
+  /**
+   * Whether `e` has a TRY cast whose map key cast can fail, which makes a 
null key that Arrow
+   * cannot hold (#6172). The native side would have to build that map, so an 
argument or a body
+   * with one stays on the dispatcher, which runs it in Spark's own code, 
where the map never
+   * reaches Arrow.
+   */
+  private def canProduceNullMapKey(e: Expression): Boolean =
+    e.exists {
+      case c: Cast => CometCast.canProduceNullMapKey(c)
+      case _ => false
+    }
+
+  /**
+   * Whether `body` casts between a date or timestamp and a string, date or 
timestamp. The native
+   * cast fails for years beyond chrono's range (about 262,143), which Spark 
supports up to year
+   * 294,247 (#7471), so such a body stays on the dispatcher.
+   */
+  private def convertsDateTime(body: Expression): Boolean = {
+    def isDateTime(dt: DataType): Boolean = dt match {
+      case DateType | TimestampType | TimestampNTZType => true
+      case _ => false
+    }
+    def isTextOrDateTime(dt: DataType): Boolean = dt.isInstanceOf[StringType] 
|| isDateTime(dt)
+    body.exists {
+      case c: Cast if c.child.dataType != c.dataType =>
+        (isDateTime(c.child.dataType) && isTextOrDateTime(c.dataType)) ||
+        (isDateTime(c.dataType) && isTextOrDateTime(c.child.dataType))
+      case _ => false
+    }
+  }
+
+  /**
+   * DataFusion copies each column the body captures once per element of its 
row's array before it
+   * evaluates the body, so a captured string, binary or nested value is 
copied as many times as
+   * the array has elements, and those copies are not reserved with the memory 
pool.
+   */
+  private def capturesOnlyFixedWidthColumns(body: Expression): Boolean =
+    body
+      .collect {
+        case a: AttributeReference => a.dataType
+        case b: BoundReference => b.dataType
+      }
+      .forall(isFixedWidth)
+
+  /**
+   * Spark evaluates `a AND b`, `a OR b` and `x IN (a, b)` lazily, so `x != 0 
AND 100 / x > 1`
+   * never divides by zero. The native side evaluates every operand over the 
whole batch, so the
+   * same lambda raises on the element Spark skips. `CASE WHEN`, `IF` and 
`COALESCE` are not
+   * affected. A body that short-circuits stays on the dispatcher unless none 
of its expressions
+   * can raise.
+   */
+  private def shortCircuitSafe(body: Expression): Boolean =
+    !body.exists {
+      case _: And | _: Or | _: In | _: InSet => true
+      case _ => false
+    } || !body.exists(mayRaise)
+
+  /**
+   * Whether `e` can raise on an input value. This fails closed: an expression 
is assumed to raise
+   * unless it is listed here, since some raise in any mode (`element_at` with 
the index 0), and
+   * arithmetic and casts raise only in the ANSI eval mode they carry. 
`infallible` in
+   * `case_when.rs` is the native counterpart.
+   */
+  protected def mayRaise(e: Expression): Boolean = e match {
+    case _: Literal | _: NamedLambdaVariable | _: AttributeReference | _: 
BoundReference |
+        _: BinaryComparison | _: IsNull | _: IsNotNull | _: Not | _: And | _: 
Or | _: In |
+        _: InSet | _: IsNaN | _: Coalesce | _: If | _: CaseWhen | _: Length | 
_: Upper |
+        _: Lower | _: Substring | _: StartsWith | _: EndsWith | _: Contains =>
+      false
+    case c: Cast => evalMode(c) == CometEvalMode.ANSI

Review Comment:
   `mayRaise` here and `nodeCannotRaise` in #7529 list the same nodes but have 
drifted apart. #7529 counts a non-ANSI `Cast` as safe only between the types 
`plainCastType` lists, because a date cast to a timestamp overflows 
`Math.multiplyExact` in every eval mode (`daysToMicros`, `instantToMicros`). It 
also lists `GetStructField`. Here every non-ANSI cast counts as safe. That 
holds today only because `convertsDateTime` keeps those casts out of the body, 
for a different reason (#7471). Once #7471 is fixed and that guard goes, 
`exists(dates, x -> CAST(x AS TIMESTAMP) > ts)` over an array with a date past 
about year 294247 after the first match would raise natively where Spark 
returns true. I have not run it.
   
   Since both PRs are approved, could whichever lands second switch to one 
classifier, for example #7529's `cannotRaise` made `private[comet]` with 
`NamedLambdaVariable` added? Then a fix to one reaches both.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to