Stevo Mitric created SPARK-60876:
------------------------------------
Summary: Keep the CHAR/VARCHAR metadata of an aliased outer
reference
Key: SPARK-60876
URL: https://issues.apache.org/jira/browse/SPARK-60876
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.4.0
Reporter: Stevo Mitric
When a LATERAL or correlated subquery returns a CHAR(n) or VARCHAR(n) column of
the outer query, the subquery output loses the CHAR/VARCHAR type and becomes a
plain STRING. Comparisons against it are then no longer padded, so a filter on
a CHAR value silently returns no rows. An inner CHAR column returned by the
same subquery is not affected.
{code:sql}
CREATE TABLE t (c CHAR(5), s STRING) USING parquet;
INSERT INTO t VALUES ('ab', 'x');
SELECT t.s FROM t, LATERAL (SELECT t.c AS k) l WHERE l.k = 'ab'; -- returns no
rows, expected x
SELECT t.s FROM t, LATERAL (SELECT t.c) l WHERE l.c = 'ab'; -- returns no
rows, expected x
SELECT t.s FROM t, LATERAL (SELECT t.*) l WHERE l.c = 'ab'; -- returns no
rows, expected x
{code}
The same happens when a correlated EXISTS, IN or scalar subquery selects an
outer CHAR column through a derived table, e.g. {{(SELECT t.c AS k FROM t2) x
WHERE x.k = 'ab'}}. CTAS and views over such a column also store STRING instead
of CHAR/VARCHAR.
Cause: the analyzer wraps an outer column returned by a subquery in an
{{Alias}}, which copies its child's metadata, where Spark keeps the raw
CHAR/VARCHAR type. {{OuterReference}} reported empty metadata, so the type was
lost. The explicit-alias form ({{t.c AS k}}) has always been affected. Outer
star expansion ({{t.*}}) is affected since SPARK-55794 (4.2.0), and the bare
form ({{t.c}}) since SPARK-58171 (4.3.0), which started aliasing those outer
references.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
