[
https://issues.apache.org/jira/browse/SPARK-60860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60860:
-----------------------------------
Labels: pull-request-available (was: )
> Recursive CTEs inherit incorrect nested nullability from their anchors
> ----------------------------------------------------------------------
>
> Key: SPARK-60860
> URL: https://issues.apache.org/jira/browse/SPARK-60860
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Nick Young
> Priority: Major
> Labels: pull-request-available
>
> A non-null recursive CTE anchor does not guarantee non-null elements or
> fields in
> later iterations. Recursive references currently inherit nested nullability
> from
> the anchor. This can cause nested null predicates to be optimized to false,
> NullPointerException for null map values, and rejection of compatible
> recursive
> struct types.
> Reproduced on Apache Spark master at b8168834c8b18a4f16f3267ecd48f3afaaaf6937.
> Minimal reproduction:
> {code:sql}
> WITH RECURSIVE r(n, a, saw_null) AS (
> SELECT 0, array(1), false
> UNION ALL
> SELECT n + 1, array(CAST(NULL AS INT)), exists(a, x -> x IS NULL)
> FROM r WHERE n < 2
> )
> SELECT n, a, saw_null FROM r ORDER BY n;
> {code}
> Actual output:
> {noformat}
> 0 [1] false
> 1 [null] false
> 2 [null] false
> {noformat}
> Expected output:
> {noformat}
> 0 [1] false
> 1 [null] false
> 2 [null] true
> {noformat}
> Recursive references should permit nested nulls before recursive expressions
> are
> resolved. Coercion should preserve the anchor's element and field types while
> allowing wider nested nullability. Recursive element or field type widening
> should still be rejected.
> A candidate fix with deterministic synthetic coverage for arrays, structs,
> maps,
> struct map keys, and compatible/incompatible coercion is available at:
> https://github.com/apache/spark/pull/60074
> Validation at 2c632bfd3cfa12e541c94b0049b11a2a3e415800: 4 SQL
> execution/analyzer
> tests and 126 Catalyst tests passed; Catalyst scalastyle passed with zero
> errors
> and warnings. The focused SQL regression runs with ANSI mode enabled and
> disabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
