comphead commented on code in PR #7322:
URL: https://github.com/apache/datafusion-comet/pull/7322#discussion_r4607214128


##########
native/spark-expr/src/conditional_funcs/case_when.rs:
##########
@@ -451,7 +451,7 @@ pub(super) fn cannot_fail(expr: &Arc<dyn PhysicalExpr>, 
input_schema: &Schema) -
 }
 
 fn infallible(expr: &Arc<dyn PhysicalExpr>, input_schema: &Schema, parsing: 
bool) -> bool {
-    if expr.is::<Column>() || expr.is::<Literal>() {
+    if expr.is::<Column>() || expr.is::<Literal>() || 
expr.is::<LambdaVariable>() {

Review Comment:
   `is_deterministic` in `lazy_logical.rs` needs this `LambdaVariable` arm too. 
Without it, `speculate` starts false whenever the right operand reads the 
lambda variable, so a `LazyLogical` in a lambda body always selects rows and 
never tries the whole batch, which its doc says costs less when most rows need 
the right operand. The `coalesce(x, '') <> '' AND length(x) > 1` body in 
`array_filter_native_lambda.sql` is one such case. I have not measured the 
difference.



##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -1026,7 +1029,149 @@ object CometFlatten extends 
CometExpressionSerde[Flatten] with ArraysBase {
   }
 }
 
-object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {
+/**
+ * What a serde needs to run the lambda of a higher-order function natively, 
under a DataFusion
+ * higher-order function: the gate on the lambda and its body, and the 
serialization.
+ */
+trait NativeLambdaSerde extends CometTypeShim with CometExprShim {
+
+  // The native lambda has the same semantics as Spark's `filter` for a 
one-parameter lambda whose
+  // body is deterministic, contains no nested lambda, and whose parameter is 
a type the native
+  // side can type-check against the list's element field. `exists` adds a 
condition of its own,
+  // since Spark stops at the first match. It needs bound references: the 
native side re-indexes
+  // the columns the body captures, which it cannot do for unbound ones.
+  protected def nativeLambdaSupported(argument: Expression, body: Expression): 
Boolean =
+    CometConf.COMET_NATIVE_LAMBDA_ENABLED.get() &&
+      body.deterministic &&
+      !body.exists(_.isInstanceOf[HigherOrderFunction]) &&
+      capturesOnlyFixedWidthColumns(body) &&
+      shortCircuitSafe(body) &&
+      !canProduceNullMapKey(argument) &&
+      !canProduceNullMapKey(body) &&
+      !convertsDateTime(body) &&
+      (argument.dataType match {
+        case ArrayType(elementType, _) => isLambdaElementType(elementType)
+        case _ => false
+      })
+
+  /**
+   * Whether `e` has a TRY cast whose map key cast can fail, which makes a 
null key that Arrow
+   * cannot hold (#6172). The native side would have to build that map, so an 
argument or a body
+   * with one stays on the dispatcher, which runs it in Spark's own code, 
where the map never
+   * reaches Arrow.
+   */
+  private def canProduceNullMapKey(e: Expression): Boolean =
+    e.exists {
+      case c: Cast => CometCast.canProduceNullMapKey(c)
+      case _ => false
+    }
+
+  /**
+   * Whether `body` casts between a date or timestamp and a string, date or 
timestamp. The native
+   * cast fails for years beyond chrono's range (about 262,143), which Spark 
supports up to year
+   * 294,247 (#7471), so such a body stays on the dispatcher.
+   */
+  private def convertsDateTime(body: Expression): Boolean = {

Review Comment:
   The chrono range limit reaches more than casts. #6843 lists native `year`, 
`month`, `dayofmonth`, `dayofyear`, `quarter`, `weekofyear`, `trunc` and 
`next_day` returning NULL, and `last_day` failing, for dates outside chrono's 
range, where Spark returns a value. `filter(dates, x -> year(x) > 2000)` has no 
cast, so it runs natively. Over this fixture's `date_from_unix_date(100000000)` 
it would drop the element that Spark keeps, because arrow's `date_part` maps 
each `Date32` through `date32_to_datetime`, which returns `None` there. Before 
this PR that lambda ran in Spark's own code.
   
   Could this also keep the #6843 functions on the dispatcher, or more simply 
any body that applies a function to a date or timestamp until #6843 and #7471 
are fixed, with that query in `array_filter_native_lambda_unsupported_body.sql` 
as `expect_dispatch(filter)`? I have not run it.



##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -1026,7 +1029,149 @@ object CometFlatten extends 
CometExpressionSerde[Flatten] with ArraysBase {
   }
 }
 
-object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {
+/**
+ * What a serde needs to run the lambda of a higher-order function natively, 
under a DataFusion
+ * higher-order function: the gate on the lambda and its body, and the 
serialization.
+ */
+trait NativeLambdaSerde extends CometTypeShim with CometExprShim {
+
+  // The native lambda has the same semantics as Spark's `filter` for a 
one-parameter lambda whose
+  // body is deterministic, contains no nested lambda, and whose parameter is 
a type the native
+  // side can type-check against the list's element field. `exists` adds a 
condition of its own,
+  // since Spark stops at the first match. It needs bound references: the 
native side re-indexes
+  // the columns the body captures, which it cannot do for unbound ones.
+  protected def nativeLambdaSupported(argument: Expression, body: Expression): 
Boolean =
+    CometConf.COMET_NATIVE_LAMBDA_ENABLED.get() &&
+      body.deterministic &&
+      !body.exists(_.isInstanceOf[HigherOrderFunction]) &&
+      capturesOnlyFixedWidthColumns(body) &&
+      shortCircuitSafe(body) &&
+      !canProduceNullMapKey(argument) &&
+      !canProduceNullMapKey(body) &&
+      !convertsDateTime(body) &&
+      (argument.dataType match {
+        case ArrayType(elementType, _) => isLambdaElementType(elementType)
+        case _ => false
+      })
+
+  /**
+   * Whether `e` has a TRY cast whose map key cast can fail, which makes a 
null key that Arrow
+   * cannot hold (#6172). The native side would have to build that map, so an 
argument or a body
+   * with one stays on the dispatcher, which runs it in Spark's own code, 
where the map never
+   * reaches Arrow.
+   */
+  private def canProduceNullMapKey(e: Expression): Boolean =
+    e.exists {
+      case c: Cast => CometCast.canProduceNullMapKey(c)
+      case _ => false
+    }
+
+  /**
+   * Whether `body` casts between a date or timestamp and a string, date or 
timestamp. The native
+   * cast fails for years beyond chrono's range (about 262,143), which Spark 
supports up to year
+   * 294,247 (#7471), so such a body stays on the dispatcher.
+   */
+  private def convertsDateTime(body: Expression): Boolean = {
+    def isDateTime(dt: DataType): Boolean = dt match {
+      case DateType | TimestampType | TimestampNTZType => true
+      case _ => false
+    }
+    def isTextOrDateTime(dt: DataType): Boolean = dt.isInstanceOf[StringType] 
|| isDateTime(dt)
+    body.exists {
+      case c: Cast if c.child.dataType != c.dataType =>
+        (isDateTime(c.child.dataType) && isTextOrDateTime(c.dataType)) ||
+        (isDateTime(c.dataType) && isTextOrDateTime(c.child.dataType))
+      case _ => false
+    }
+  }
+
+  /**
+   * DataFusion copies each column the body captures once per element of its 
row's array before it
+   * evaluates the body, so a captured string, binary or nested value is 
copied as many times as
+   * the array has elements, and those copies are not reserved with the memory 
pool.
+   */
+  private def capturesOnlyFixedWidthColumns(body: Expression): Boolean =
+    body
+      .collect {
+        case a: AttributeReference => a.dataType
+        case b: BoundReference => b.dataType
+      }
+      .forall(isFixedWidth)
+
+  /**
+   * Spark evaluates `a AND b`, `a OR b` and `x IN (a, b)` lazily, so `x != 0 
AND 100 / x > 1`
+   * never divides by zero. The native side evaluates every operand over the 
whole batch, so the
+   * same lambda raises on the element Spark skips. `CASE WHEN`, `IF` and 
`COALESCE` are not
+   * affected. A body that short-circuits stays on the dispatcher unless none 
of its expressions
+   * can raise.
+   */
+  private def shortCircuitSafe(body: Expression): Boolean =
+    !body.exists {
+      case _: And | _: Or | _: In | _: InSet => true

Review Comment:
   Since #7326, native `AND` and `OR` are built as a `LazyLogical` 
(`spark_comparison` wraps them), which evaluates a right operand that can fail 
only on the rows the left one leaves open, as Spark's `And` and `Or` do. That 
carries into a lambda body: DataFusion's `LambdaExpr::try_new` re-indexes the 
body with `transform_down`, and `LazyLogical::with_new_children` calls 
`try_new`, which re-projects the right operand with this PR's lambda-aware 
`project()`. `filter(strs, x -> coalesce(x, '') <> '' AND length(x) > 1)` in 
`array_filter_native_lambda.sql` already runs a `LazyLogical` over the lambda 
variable and matches Spark.
   
   So for `AND` and `OR` this check only sends bodies to the dispatcher. 
Because `mayRaise` fails closed, that is any body that pairs them with ANSI 
arithmetic, the Spark 4 default, or with an unlisted function in any mode, such 
as `x > 0 AND x + 1 < 10` or `x IS NOT NULL AND abs(x) < 5`. `IN` still needs 
the check: `lazy_in_list` stays eager when the value can fail, and on a row 
whose value is NULL it still evaluates a later candidate, where Spark's 
`In.eval` returns NULL without evaluating any.
   
   Could `shortCircuitSafe` match only `In`, with the `AND` and `OR` queries in 
`array_filter_native_lambda_ansi.sql` moved to `expect_native(filter)` and the 
tuning page bullet updated? Those queries compare with Spark, so they would 
check the native laziness. I have not run them.



##########
native/core/src/execution/planner.rs:
##########
@@ -1142,6 +1147,62 @@ impl PhysicalPlanner {
                 ));
                 Ok(expr)
             }
+            ExprStruct::HigherOrderFunc(hof) => {
+                let array =
+                    self.create_expr(hof.array.as_ref().unwrap(), 
Arc::clone(&input_schema))?;
+                // The lambda parameter takes the element field of the array 
as planned, since
+                // DataFusion checks it against the field of the list it 
evaluates.
+                let array_field = array.return_field(input_schema.as_ref())?;
+                let element_field = match array_field.data_type() {
+                    DataType::List(f) | DataType::LargeList(f) => f,
+                    other => {
+                        return Err(GeneralError(format!(
+                            "{} expects an array argument, got {other}",
+                            hof.func
+                        )))
+                    }
+                };
+                // The lambda body sees the enclosing schema with the 
parameter appended, so
+                // `BoundReference`s keep their indexes and a `LambdaVariable` 
is found by name.
+                let mut fields: Vec<FieldRef> = input_schema.fields().to_vec();
+                fields.push(Arc::new(
+                    element_field.as_ref().clone().with_name(&hof.param_name),
+                ));
+                let lambda_schema = Arc::new(Schema::new_with_metadata(
+                    fields,
+                    input_schema.metadata().clone(),
+                ));
+                let body = self.create_expr(hof.body.as_ref().unwrap(), 
lambda_schema)?;
+                let lambda: Arc<dyn PhysicalExpr> =
+                    Arc::new(LambdaExpr::try_new(vec![hof.param_name.clone()], 
body)?);
+                let func = match hof.func.as_str() {

Review Comment:
   Comet's session already registers DataFusion's higher-order functions 
(`functions_nested::register_all` in `jni_api.rs` includes 
`all_default_higher_order_functions`), so this can resolve the name the way the 
scalar function path does with `self.session_ctx.udf(fun_name)`: 
`self.session_ctx.higher_order_function(&hof.func)?`. That drops this second 
list of names and the two `functions_nested` imports, and leaves the Scala 
serde as the one place that decides which functions run natively.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to