Yibo Dong created SPARK-61966:
---------------------------------
Summary: QUALIFY resolves a SELECT alias before a same-named input
column
Key: SPARK-61966
URL: https://issues.apache.org/jira/browse/SPARK-61966
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Environment: Spark 5.0.0-SNAPSHOT, upstream master at
\{{dada2a6db5d73b152d6ba16e4fe21406f86abfe3}} (2026-11-15), local mode, default
SQL settings
Reporter: Yibo Dong
h2. What happened
Spark resolves an unqualified name in {{QUALIFY}} to a SELECT window alias
instead of a same-named input column, contrary to the documented input-column
precedence. This changes which row survives the filter.
h2. How to reproduce
Run the following queries in a fresh Spark SQL session:
{code:sql}
SELECT id, ROW_NUMBER() OVER (ORDER BY score DESC) AS rn
FROM VALUES
(1, 30, 0),
(2, 20, 0),
(3, 10, 1)
AS t(id, score, rn)
QUALIFY rn = 1
ORDER BY id;
-- Control: explicitly reference the input column.
SELECT id, ROW_NUMBER() OVER (ORDER BY score DESC) AS rn
FROM VALUES
(1, 30, 0),
(2, 20, 0),
(3, 10, 1)
AS t(id, score, rn)
QUALIFY t.rn = 1
ORDER BY id;
{code}
h2. Expected result
Both queries should return {{(3, 3)}}. The input column {{rn}} takes precedence
over the SELECT alias, and {{QUALIFY}} filters after window evaluation.
h2. Actual result
The first query returns {{(1, 1)}}, while the qualified-column control returns
{{(3, 3)}}.
The analyzed plan for the first query binds {{rn}} to the window alias instead
of the input column.
h2. Additional information
Spark's QUALIFY documentation explicitly specifies that input columns take
precedence over same-named SELECT aliases:
https://spark.apache.org/docs/latest/sql-ref-syntax-qry-select-qualify.html
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]