Nick Young created SPARK-60860:
----------------------------------
Summary: Recursive CTEs inherit incorrect nested nullability from
their anchors
Key: SPARK-60860
URL: https://issues.apache.org/jira/browse/SPARK-60860
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Nick Young
A non-null recursive CTE anchor does not guarantee non-null elements or fields
in
later iterations. Recursive references currently inherit nested nullability from
the anchor. This can cause nested null predicates to be optimized to false,
NullPointerException for null map values, and rejection of compatible recursive
struct types.
Reproduced on Apache Spark master at b8168834c8b18a4f16f3267ecd48f3afaaaf6937.
Minimal reproduction:
{code:sql}
WITH RECURSIVE r(n, a, saw_null) AS (
SELECT 0, array(1), false
UNION ALL
SELECT n + 1, array(CAST(NULL AS INT)), exists(a, x -> x IS NULL)
FROM r WHERE n < 2
)
SELECT n, a, saw_null FROM r ORDER BY n;
{code}
Actual output:
{noformat}
0 [1] false
1 [null] false
2 [null] false
{noformat}
Expected output:
{noformat}
0 [1] false
1 [null] false
2 [null] true
{noformat}
Recursive references should permit nested nulls before recursive expressions are
resolved. Coercion should preserve the anchor's element and field types while
allowing wider nested nullability. Recursive element or field type widening
should still be rejected.
A candidate fix with deterministic synthetic coverage for arrays, structs, maps,
struct map keys, and compatible/incompatible coercion is available at:
https://github.com/apache/spark/pull/60074
Validation at 2c632bfd3cfa12e541c94b0049b11a2a3e415800: 4 SQL execution/analyzer
tests and 126 Catalyst tests passed; Catalyst scalastyle passed with zero errors
and warnings. The focused SQL regression runs with ANSI mode enabled and
disabled.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
