comphead commented on code in PR #7322:
URL: https://github.com/apache/datafusion-comet/pull/7322#discussion_r4607214128
##########
native/spark-expr/src/conditional_funcs/case_when.rs:
##########
@@ -451,7 +451,7 @@ pub(super) fn cannot_fail(expr: &Arc<dyn PhysicalExpr>,
input_schema: &Schema) -
}
fn infallible(expr: &Arc<dyn PhysicalExpr>, input_schema: &Schema, parsing:
bool) -> bool {
- if expr.is::<Column>() || expr.is::<Literal>() {
+ if expr.is::<Column>() || expr.is::<Literal>() ||
expr.is::<LambdaVariable>() {
Review Comment:
`is_deterministic` in `lazy_logical.rs` needs this `LambdaVariable` arm too.
Without it, `speculate` starts false whenever the right operand reads the
lambda variable, so a `LazyLogical` in a lambda body always selects rows and
never tries the whole batch, which its doc says costs less when most rows need
the right operand. The `coalesce(x, '') <> '' AND length(x) > 1` body in
`array_filter_native_lambda.sql` is one such case. I have not measured the
difference.
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -1026,7 +1029,149 @@ object CometFlatten extends
CometExpressionSerde[Flatten] with ArraysBase {
}
}
-object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {
+/**
+ * What a serde needs to run the lambda of a higher-order function natively,
under a DataFusion
+ * higher-order function: the gate on the lambda and its body, and the
serialization.
+ */
+trait NativeLambdaSerde extends CometTypeShim with CometExprShim {
+
+ // The native lambda has the same semantics as Spark's `filter` for a
one-parameter lambda whose
+ // body is deterministic, contains no nested lambda, and whose parameter is
a type the native
+ // side can type-check against the list's element field. `exists` adds a
condition of its own,
+ // since Spark stops at the first match. It needs bound references: the
native side re-indexes
+ // the columns the body captures, which it cannot do for unbound ones.
+ protected def nativeLambdaSupported(argument: Expression, body: Expression):
Boolean =
+ CometConf.COMET_NATIVE_LAMBDA_ENABLED.get() &&
+ body.deterministic &&
+ !body.exists(_.isInstanceOf[HigherOrderFunction]) &&
+ capturesOnlyFixedWidthColumns(body) &&
+ shortCircuitSafe(body) &&
+ !canProduceNullMapKey(argument) &&
+ !canProduceNullMapKey(body) &&
+ !convertsDateTime(body) &&
+ (argument.dataType match {
+ case ArrayType(elementType, _) => isLambdaElementType(elementType)
+ case _ => false
+ })
+
+ /**
+ * Whether `e` has a TRY cast whose map key cast can fail, which makes a
null key that Arrow
+ * cannot hold (#6172). The native side would have to build that map, so an
argument or a body
+ * with one stays on the dispatcher, which runs it in Spark's own code,
where the map never
+ * reaches Arrow.
+ */
+ private def canProduceNullMapKey(e: Expression): Boolean =
+ e.exists {
+ case c: Cast => CometCast.canProduceNullMapKey(c)
+ case _ => false
+ }
+
+ /**
+ * Whether `body` casts between a date or timestamp and a string, date or
timestamp. The native
+ * cast fails for years beyond chrono's range (about 262,143), which Spark
supports up to year
+ * 294,247 (#7471), so such a body stays on the dispatcher.
+ */
+ private def convertsDateTime(body: Expression): Boolean = {
Review Comment:
The chrono range limit reaches more than casts. #6843 lists native `year`,
`month`, `dayofmonth`, `dayofyear`, `quarter`, `weekofyear`, `trunc` and
`next_day` returning NULL, and `last_day` failing, for dates outside chrono's
range, where Spark returns a value. `filter(dates, x -> year(x) > 2000)` has no
cast, so it runs natively. Over this fixture's `date_from_unix_date(100000000)`
it would drop the element that Spark keeps, because arrow's `date_part` maps
each `Date32` through `date32_to_datetime`, which returns `None` there. Before
this PR that lambda ran in Spark's own code.
Could this also keep the #6843 functions on the dispatcher, or more simply
any body that applies a function to a date or timestamp until #6843 and #7471
are fixed, with that query in `array_filter_native_lambda_unsupported_body.sql`
as `expect_dispatch(filter)`? I have not run it.
##########
spark/src/main/scala/org/apache/comet/serde/arrays.scala:
##########
@@ -1026,7 +1029,149 @@ object CometFlatten extends
CometExpressionSerde[Flatten] with ArraysBase {
}
}
-object CometArrayFilter extends CometExpressionSerde[ArrayFilter] {
+/**
+ * What a serde needs to run the lambda of a higher-order function natively,
under a DataFusion
+ * higher-order function: the gate on the lambda and its body, and the
serialization.
+ */
+trait NativeLambdaSerde extends CometTypeShim with CometExprShim {
+
+ // The native lambda has the same semantics as Spark's `filter` for a
one-parameter lambda whose
+ // body is deterministic, contains no nested lambda, and whose parameter is
a type the native
+ // side can type-check against the list's element field. `exists` adds a
condition of its own,
+ // since Spark stops at the first match. It needs bound references: the
native side re-indexes
+ // the columns the body captures, which it cannot do for unbound ones.
+ protected def nativeLambdaSupported(argument: Expression, body: Expression):
Boolean =
+ CometConf.COMET_NATIVE_LAMBDA_ENABLED.get() &&
+ body.deterministic &&
+ !body.exists(_.isInstanceOf[HigherOrderFunction]) &&
+ capturesOnlyFixedWidthColumns(body) &&
+ shortCircuitSafe(body) &&
+ !canProduceNullMapKey(argument) &&
+ !canProduceNullMapKey(body) &&
+ !convertsDateTime(body) &&
+ (argument.dataType match {
+ case ArrayType(elementType, _) => isLambdaElementType(elementType)
+ case _ => false
+ })
+
+ /**
+ * Whether `e` has a TRY cast whose map key cast can fail, which makes a
null key that Arrow
+ * cannot hold (#6172). The native side would have to build that map, so an
argument or a body
+ * with one stays on the dispatcher, which runs it in Spark's own code,
where the map never
+ * reaches Arrow.
+ */
+ private def canProduceNullMapKey(e: Expression): Boolean =
+ e.exists {
+ case c: Cast => CometCast.canProduceNullMapKey(c)
+ case _ => false
+ }
+
+ /**
+ * Whether `body` casts between a date or timestamp and a string, date or
timestamp. The native
+ * cast fails for years beyond chrono's range (about 262,143), which Spark
supports up to year
+ * 294,247 (#7471), so such a body stays on the dispatcher.
+ */
+ private def convertsDateTime(body: Expression): Boolean = {
+ def isDateTime(dt: DataType): Boolean = dt match {
+ case DateType | TimestampType | TimestampNTZType => true
+ case _ => false
+ }
+ def isTextOrDateTime(dt: DataType): Boolean = dt.isInstanceOf[StringType]
|| isDateTime(dt)
+ body.exists {
+ case c: Cast if c.child.dataType != c.dataType =>
+ (isDateTime(c.child.dataType) && isTextOrDateTime(c.dataType)) ||
+ (isDateTime(c.dataType) && isTextOrDateTime(c.child.dataType))
+ case _ => false
+ }
+ }
+
+ /**
+ * DataFusion copies each column the body captures once per element of its
row's array before it
+ * evaluates the body, so a captured string, binary or nested value is
copied as many times as
+ * the array has elements, and those copies are not reserved with the memory
pool.
+ */
+ private def capturesOnlyFixedWidthColumns(body: Expression): Boolean =
+ body
+ .collect {
+ case a: AttributeReference => a.dataType
+ case b: BoundReference => b.dataType
+ }
+ .forall(isFixedWidth)
+
+ /**
+ * Spark evaluates `a AND b`, `a OR b` and `x IN (a, b)` lazily, so `x != 0
AND 100 / x > 1`
+ * never divides by zero. The native side evaluates every operand over the
whole batch, so the
+ * same lambda raises on the element Spark skips. `CASE WHEN`, `IF` and
`COALESCE` are not
+ * affected. A body that short-circuits stays on the dispatcher unless none
of its expressions
+ * can raise.
+ */
+ private def shortCircuitSafe(body: Expression): Boolean =
+ !body.exists {
+ case _: And | _: Or | _: In | _: InSet => true
Review Comment:
Since #7326, native `AND` and `OR` are built as a `LazyLogical`
(`spark_comparison` wraps them), which evaluates a right operand that can fail
only on the rows the left one leaves open, as Spark's `And` and `Or` do. That
carries into a lambda body: DataFusion's `LambdaExpr::try_new` re-indexes the
body with `transform_down`, and `LazyLogical::with_new_children` calls
`try_new`, which re-projects the right operand with this PR's lambda-aware
`project()`. `filter(strs, x -> coalesce(x, '') <> '' AND length(x) > 1)` in
`array_filter_native_lambda.sql` already runs a `LazyLogical` over the lambda
variable and matches Spark.
So for `AND` and `OR` this check only sends bodies to the dispatcher.
Because `mayRaise` fails closed, that is any body that pairs them with ANSI
arithmetic, the Spark 4 default, or with an unlisted function in any mode, such
as `x > 0 AND x + 1 < 10` or `x IS NOT NULL AND abs(x) < 5`. `IN` still needs
the check: `lazy_in_list` stays eager when the value can fail, and on a row
whose value is NULL it still evaluates a later candidate, where Spark's
`In.eval` returns NULL without evaluating any.
Could `shortCircuitSafe` match only `In`, with the `AND` and `OR` queries in
`array_filter_native_lambda_ansi.sql` moved to `expect_native(filter)` and the
tuning page bullet updated? Those queries compare with Spark, so they would
check the native laziness. I have not run them.
##########
native/core/src/execution/planner.rs:
##########
@@ -1142,6 +1147,62 @@ impl PhysicalPlanner {
));
Ok(expr)
}
+ ExprStruct::HigherOrderFunc(hof) => {
+ let array =
+ self.create_expr(hof.array.as_ref().unwrap(),
Arc::clone(&input_schema))?;
+ // The lambda parameter takes the element field of the array
as planned, since
+ // DataFusion checks it against the field of the list it
evaluates.
+ let array_field = array.return_field(input_schema.as_ref())?;
+ let element_field = match array_field.data_type() {
+ DataType::List(f) | DataType::LargeList(f) => f,
+ other => {
+ return Err(GeneralError(format!(
+ "{} expects an array argument, got {other}",
+ hof.func
+ )))
+ }
+ };
+ // The lambda body sees the enclosing schema with the
parameter appended, so
+ // `BoundReference`s keep their indexes and a `LambdaVariable`
is found by name.
+ let mut fields: Vec<FieldRef> = input_schema.fields().to_vec();
+ fields.push(Arc::new(
+ element_field.as_ref().clone().with_name(&hof.param_name),
+ ));
+ let lambda_schema = Arc::new(Schema::new_with_metadata(
+ fields,
+ input_schema.metadata().clone(),
+ ));
+ let body = self.create_expr(hof.body.as_ref().unwrap(),
lambda_schema)?;
+ let lambda: Arc<dyn PhysicalExpr> =
+ Arc::new(LambdaExpr::try_new(vec![hof.param_name.clone()],
body)?);
+ let func = match hof.func.as_str() {
Review Comment:
Comet's session already registers DataFusion's higher-order functions
(`functions_nested::register_all` in `jni_api.rs` includes
`all_default_higher_order_functions`), so this can resolve the name the way the
scalar function path does with `self.session_ctx.udf(fun_name)`:
`self.session_ctx.higher_order_function(&hof.func)?`. That drops this second
list of names and the two `functions_nested` imports, and leaves the Scala
serde as the one place that decides which functions run natively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]