[
https://issues.apache.org/jira/browse/SPARK-59019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59019:
-----------------------------------
Labels: optimizer pull-request-available sql (was: optimizer sql)
> Excluding RewriteWithExpression causes an INTERNAL_ERROR for a legal BETWEEN
> query
> ----------------------------------------------------------------------------------
>
> Key: SPARK-59019
> URL: https://issues.apache.org/jira/browse/SPARK-59019
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: {{Spark 5.0.0-SNAPSHOT, commit
> 9da9f8d673914d1648514f59d85e3adafb300d1a; macOS arm64; Java 17.0.17}}
> Reporter: Yibo Dong
> Priority: Major
> Labels: optimizer, pull-request-available, sql
>
> h2. What happened
>
> A legal `BETWEEN` query succeeds with `RewriteWithExpression` enabled but
> fails with a stable internal error when the rule is excluded.
> h2. How to reproduce
> Save the following SQL as `repro.sql`:
> {code:java}
> SELECT x BETWEEN 1 AND 2 AS in_range
> FROM (VALUES (1)) AS t(x);{code}
> {{Run it once normally and once with `RewriteWithExpression` excluded:}}
> {code:java}
> spark-sql --master 'local[2]' --conf spark.ui.enabled=false --conf
> spark.sql.adaptive.enabled=false --conf spark.sql.shuffle.partitions=1 -f
> repro.sql
> spark-sql --master 'local[2]' --conf spark.ui.enabled=false --conf
> spark.sql.adaptive.enabled=false --conf spark.sql.shuffle.partitions=1 --conf
> spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.RewriteWithExpression
> -f repro.sql{code}
>
> h2. Expected Result
>
> Both configurations should return the same result.
>
> |Configuration|in_range|
> |—|—|
> |`RewriteWithExpression` enabled|`true`|
> |`RewriteWithExpression` excluded|`true`|
> h2. Actual Result
>
> The enabled configuration returns the row, while excluding the rule causes
> Spark to fail.
>
> |Configuration|Result|
> |—|—|
> |`RewriteWithExpression` enabled|`true`|
> |`RewriteWithExpression` excluded|Internal error|
> The failing run reports:
>
> {code:java}
> Spark query failed: An error occurred while calling o41.showString.
> : org.apache.spark.SparkException: [INTERNAL_ERROR] Cannot generate code for
> expression: with(((commonexpressionref(CommonExpressionId(0,false),
> IntegerType, false) >= 1) AND
> (commonexpressionref(CommonExpressionId(0,false), IntegerType, false) <= 2)),
> commonexpressiondef(input[0, int, false], CommonExpressionId(0,false)))
> SQLSTATE: XX000
> at org.apache.spark.SparkException$.internalError(SparkException.scala:104)
> at
> org.apache.spark.SparkException$.internalError(SparkException.scala:108){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
