[
https://issues.apache.org/jira/browse/SPARK-59987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uroš Bojanić resolved SPARK-59987.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59227
[https://github.com/apache/spark/pull/59227]
> AssertionError on NOT CAST to BOOLEAN during V2 predicate pushdown
> ------------------------------------------------------------------
>
> Key: SPARK-59987
> URL: https://issues.apache.org/jira/browse/SPARK-59987
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Daniil Grbic
> Assignee: Daniil Grbic
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> With ANSI mode enabled, CAST(c AS BOOLEAN) becomes a scalar V2Cast
> ([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L152-L154]).
> When NOT/OR/AND wraps that cast inside COALESCE, the builder recursively
> translates NOT's child, then asserts that the result is already a V2Predicate
> - the child is a V2Cast, so the assertion fails before predicate translation
> can return None
> ([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L241-L245]).
> A top-level WHERE NOT CAST(c AS BOOLEAN) follows a different path and does
> not hit that assertion. DataSourceV2Strategy handles the outer NOT separately
> and translates CAST(c AS BOOLEAN) as a leaf predicate
> ([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala#L936-L940]).
> With the default alwaysCreateV2Predicate setting, buildPredicate wraps the
> scalar V2Cast as BOOLEAN_EXPRESSION(V2Cast(...)), then the outer path
> produces NOT(BOOLEAN_EXPRESSION(V2Cast(...)))
> ([pointer|https://github.com/apache/spark/blob/2f4102c8117a087aba1324d52b268f7966670136/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L41-L50]).
> Issue repro:
> {code:sql}
> SET spark.sql.ansi.enabled=true;
> CREATE NAMESPACE h2.test;
> CREATE TABLE h2.test.boolean_cast_repro (c INT);
> INSERT INTO h2.test.boolean_cast_repro VALUES (0), (1);
> SELECT c FROM h2.test.boolean_cast_repro
> WHERE coalesce(NOT CAST(c AS BOOLEAN), false);
> {code}
> Expected result: one row containing 0. If the filter cannot be pushed down,
> Spark should evaluate it after the scan.
> Actual result: optimization fails with org.apache.spark.SparkException:
> [INTERNAL_ERROR], caused by java.lang.AssertionError: assertion failed at
> V2ExpressionBuilder.scala:243.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
