[
https://issues.apache.org/jira/browse/SPARK-58431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58431:
-----------------------------------
Labels: correctness pull-request-available (was: correctness)
> SimplifyExtractValueOps folds an out-of-bounds array index to NULL under
> ANSI, suppressing INVALID_ARRAY_INDEX and corrupting the result's nullability
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58431
> URL: https://issues.apache.org/jira/browse/SPARK-58431
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Josh Rosen
> Priority: Major
> Labels: correctness, pull-request-available
>
> This is a report of a correctness bug reachable under default configuration
> on Spark 4.x (ANSI mode is now enabled by default). A query that must raise
> {{INVALID_ARRAY_INDEX}} instead returns a fabricated value of {{{}0{}}},
> silently.
>
> {{SimplifyExtractValueOps}} rewrites {{array(...)[i]}} when the index is a
> literal ({{{}ComplexTypes.scala{}}}):
>
> {code:java}
> case ga @ GetArrayItem(CreateArray(elems, _), IntegerLiteral(idx), _) =>
> if (idx >= 0 && idx < elems.size) {
> elems(idx)
> } else {
> // out of bounds, mimic the runtime behavior and return null
> Literal(null, ga.dataType)
> } {code}
> The comment states the intent: mimic the runtime behavior. That was accurate
> when the rule was written (SPARK-18601, 2.2.0), but SPARK-33386 (3.1.1)
> changed the runtime behavior: under ANSI, an out-of-bounds array access
> _raises_ rather than returning null. The rule was never updated, and it does
> not consult {{GetArrayItem.failOnError}} (which defaults to
> {{{}SQLConf.get.ansiEnabled{}}}).
>
> This causes two problems:
> * The error is suppressed.
> * Under ANSI {{GetArrayItem}} declares itself *non-nullable* (it throws
> instead of returning null), so substituting a null literal places a NULL in a
> slot the schema marks non-nullable. Depending on the downstream consumer,
> this can return either NULL (incorrect under ANSI) or as the type's zero
> value (also incorrect).
>
> *Repro* (default configuration on 4.x; add {{SET
> spark.sql.ansi.enabled=true}} on older versions):
>
> {code:java}
> SET spark.sql.ansi.enabled=true;
> CREATE TABLE arr_src(c INT) USING parquet;
> INSERT INTO arr_src VALUES (1);
> SELECT array(c, 2, 3)[5] AS v FROM arr_src;
> -- prints: NULL (expected under ANSI: INVALID_ARRAY_INDEX error)
> SELECT count(*) FROM arr_src WHERE array(c, 2, 3)[5] IS NULL;
> -- prints: 0 (the value that just printed NULL matches no IS NULL
> filter)
> SELECT array(c, 2, 3)[5] IS NULL AS n FROM arr_src;
> -- prints: false
> SELECT sum(array(c,2,3)[5]) AS s, count(array(c,2,3)[5]) AS n FROM arr_src;
> -- prints: NULL, 1 (count treats it as one NON-null value; sum of it is
> null) {code}
> No consistent value can produce those four outputs together. {*}Which symptom
> the corrupted slot surfaces as is read-path-dependent{*}: the spark-sql CLI's
> row rendering honors the physical null bit and prints {{{}NULL{}}}, while
> {{Dataset.collect()}} (Scala/Java/Python APIs) trusts the declared
> {{{}nullable = false{}}}, elides the null check, and materializes a
> fabricated {{0.}}
>
> Affected versions: the fold dates from 2.2.0, but it only became incorrect
> when SPARK-33386 (3.1.1) made ANSI out-of-bounds access raise. So 3.1.1+ with
> ANSI enabled, and {*}4.x by default{*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
