Joel Robin created SPARK-59146:
----------------------------------
Summary: [SQL] Pipe SET do not retain qualified access to affected
source columns
Key: SPARK-59146
URL: https://issues.apache.org/jira/browse/SPARK-59146
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Joel Robin
h3. Problem
The pipe SQL SET operator does not retain a table alias for a source column
when that same column is assigned. A downstream qualified reference to the
original input column then fails during analysis.
This contradicts the documented SET behavior: after an assignment, top-level
column names are updated, but table aliases still refer to the original row
values.
This is an uncovered affected-column case related to SPARK-50772. The
regression scenario added there updates derived columns and later reads
untouched qualified source columns; it does not assign the qualified source
column itself.
h3. Reproduction
Reproduced on Apache Spark master at commit
cdab5402f9f5890117d6156b6f0e7c0ed8e1aca6.
{code:sql}
VALUES (1, 10) AS t(a, b)
|> SET a = a + 1
|> SELECT t.a;
{code}
h3. Actual behavior
The query fails during analysis:
{noformat}
UNRESOLVED_COLUMN: `t`.`a`
{noformat}
h3. Expected behavior
The query should return 1. The qualified reference t.a should continue to
expose the original input value, while the unqualified a should expose the
assigned value 2.
A control query shows that the updated top-level value and an untouched
qualified column are both available:
{code:sql}
VALUES (1, 10) AS t(a, b)
|> SET a = a + 1
|> SELECT a, t.b;
{code}
This returns (2, 10).
h3. Customer impact
Pipe SQL transformations cannot use the documented alias-retention behavior to
access both an updated value and its original source value. Queries that use
qualified references for before/after calculations, auditing, or join
disambiguation fail with an unresolved-column error.
h3. Related issues
* SPARK-50772 / PR #49420 introduced alias retention for SET, EXTEND, and DROP,
but did not cover a source column affected by SET.
* SPARK-54914 / PR #53691 concerns qualified or nested DROP targets and does
not cover this downstream SET reference.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
