[
https://issues.apache.org/jira/browse/SPARK-59146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114295#comment-18114295
]
Anmol Mishra commented on SPARK-59146:
--------------------------------------
Thanks! Does this cover the DROP statements as well?
Since I'm new to this, could you also share the link to the PR?
> [SQL] Pipe SET do not retain qualified access to affected source columns
> ------------------------------------------------------------------------
>
> Key: SPARK-59146
> URL: https://issues.apache.org/jira/browse/SPARK-59146
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Joel Robin
> Priority: Major
>
> h3. Problem
> The pipe SQL SET operator does not retain a table alias for a source column
> when that same column is assigned. A downstream qualified reference to the
> original input column then fails during analysis.
> This contradicts the documented SET behavior: after an assignment, top-level
> column names are updated, but table aliases still refer to the original row
> values.
> This is an uncovered affected-column case related to SPARK-50772. The
> regression scenario added there updates derived columns and later reads
> untouched qualified source columns; it does not assign the qualified source
> column itself.
> h3. Reproduction
> Reproduced on Apache Spark master at commit
> cdab5402f9f5890117d6156b6f0e7c0ed8e1aca6.
> {code:sql}
> VALUES (1, 10) AS t(a, b)
> |> SET a = a + 1
> |> SELECT t.a;
> {code}
> h3. Actual behavior
> The query fails during analysis:
> {noformat}
> UNRESOLVED_COLUMN: `t`.`a`
> {noformat}
> h3. Expected behavior
> The query should return 1. The qualified reference t.a should continue to
> expose the original input value, while the unqualified a should expose the
> assigned value 2.
> A control query shows that the updated top-level value and an untouched
> qualified column are both available:
> {code:sql}
> VALUES (1, 10) AS t(a, b)
> |> SET a = a + 1
> |> SELECT a, t.b;
> {code}
> This returns (2, 10).
> h3. Customer impact
> Pipe SQL transformations cannot use the documented alias-retention behavior
> to access both an updated value and its original source value. Queries that
> use qualified references for before/after calculations, auditing, or join
> disambiguation fail with an unresolved-column error.
> h3. Related issues
> * SPARK-50772 / PR #49420 introduced alias retention for SET, EXTEND, and
> DROP, but did not cover a source column affected by SET.
> * SPARK-54914 / PR #53691 concerns qualified or nested DROP targets and does
> not cover this downstream SET reference.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
