[
https://issues.apache.org/jira/browse/SPARK-61972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148135#comment-18148135
]
Shivadarshan commented on SPARK-61972:
--------------------------------------
The output of a `Union` takes the CHAR length of its first child, because
`Union.mergeChildOutputs` keeps the first child's metadata. With read-side
padding, the values of each child are padded to their own table's length, so
the literal is padded to the first child's length and can't match the values of
a child with a different length.
I opened a PR that pads the children to the widest CHAR length and gives the
output that CHAR type, as the SQL standard does for the union of CHAR(n) and
CHAR(m). With it, both queries in the description return 1 and 2. Children that
aren't all CHAR types at a position, such as CHAR and STRING, are left as they
are. If you'd prefer different semantics for this case, I'm happy to change it.
> UNION ALL of different CHAR lengths changes filtered rows when branches are
> reversed
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61972
> URL: https://issues.apache.org/jira/browse/SPARK-61972
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark 5.0.0-SNAPSHOT, upstream master at
> \{{dada2a6db5d73b152d6ba16e4fe21406f86abfe3}} (2026-11-15), local mode,
> default SQL settings
> Reporter: Yibo Dong
> Priority: Major
> Labels: pull-request-available
>
> h2. What happened
> A filter above {{UNION ALL}} drops a matching CHAR row, and reversing the
> branches changes which row is returned.
> h2. How to reproduce
> Run in a fresh Spark SQL session with a fresh warehouse. The two final
> queries differ only in branch order.
> {code:sql}
> CREATE TABLE t3 (id INT, c CHAR(3)) USING PARQUET;
> CREATE TABLE t5 (id INT, c CHAR(5)) USING PARQUET;
> INSERT INTO t3 VALUES (1, 'A');
> INSERT INTO t5 VALUES (2, 'A');
> SELECT id
> FROM (
> SELECT id, c FROM t3
> UNION ALL
> SELECT id, c FROM t5
> ) u
> WHERE c = 'A'
> ORDER BY id;
> SELECT id
> FROM (
> SELECT id, c FROM t5
> UNION ALL
> SELECT id, c FROM t3
> ) u
> WHERE c = 'A'
> ORDER BY id;
> {code}
> h2. Expected result
> Both queries should return the same rows regardless of branch order. Under
> the default CHAR padding semantics, both rows should match the short literal,
> returning IDs 1 and 2.
> h2. Actual result
> The CHAR(3)-first query returns only ID 1, while the CHAR(5)-first query
> returns only ID 2.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]