[
https://issues.apache.org/jira/browse/SPARK-59146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114285#comment-18114285
]
Anmol Mishra commented on SPARK-59146:
--------------------------------------
I would like to work on this.
Reproduced on master at 9a8c7cdb200. The assigned column is excluded from the
project
list in AstBuilder.visitOperatorPipeSet, so the qualified reference has nothing
left to
resolve to. I have a fix locally that retains it as hidden output via the
existing
Project.hiddenOutputTag mechanism, with golden file tests passing; details will
be in
the PR.
One scoping question first: the same query shape with DROP instead of SET, i.e.
"|> DROP a" followed by "|> SELECT t.a", fails the same way and from the same
cause,
and the docs make the same alias-retention promise for DROP. I could not find an
existing ticket for it. Should the fix cover DROP as well, or would you prefer a
separate ticket?
> [SQL] Pipe SET do not retain qualified access to affected source columns
> ------------------------------------------------------------------------
>
> Key: SPARK-59146
> URL: https://issues.apache.org/jira/browse/SPARK-59146
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Joel Robin
> Priority: Major
>
> h3. Problem
> The pipe SQL SET operator does not retain a table alias for a source column
> when that same column is assigned. A downstream qualified reference to the
> original input column then fails during analysis.
> This contradicts the documented SET behavior: after an assignment, top-level
> column names are updated, but table aliases still refer to the original row
> values.
> This is an uncovered affected-column case related to SPARK-50772. The
> regression scenario added there updates derived columns and later reads
> untouched qualified source columns; it does not assign the qualified source
> column itself.
> h3. Reproduction
> Reproduced on Apache Spark master at commit
> cdab5402f9f5890117d6156b6f0e7c0ed8e1aca6.
> {code:sql}
> VALUES (1, 10) AS t(a, b)
> |> SET a = a + 1
> |> SELECT t.a;
> {code}
> h3. Actual behavior
> The query fails during analysis:
> {noformat}
> UNRESOLVED_COLUMN: `t`.`a`
> {noformat}
> h3. Expected behavior
> The query should return 1. The qualified reference t.a should continue to
> expose the original input value, while the unqualified a should expose the
> assigned value 2.
> A control query shows that the updated top-level value and an untouched
> qualified column are both available:
> {code:sql}
> VALUES (1, 10) AS t(a, b)
> |> SET a = a + 1
> |> SELECT a, t.b;
> {code}
> This returns (2, 10).
> h3. Customer impact
> Pipe SQL transformations cannot use the documented alias-retention behavior
> to access both an updated value and its original source value. Queries that
> use qualified references for before/after calculations, auditing, or join
> disambiguation fail with an unresolved-column error.
> h3. Related issues
> * SPARK-50772 / PR #49420 introduced alias retention for SET, EXTEND, and
> DROP, but did not cover a source column affected by SET.
> * SPARK-54914 / PR #53691 concerns qualified or nested DROP targets and does
> not cover this downstream SET reference.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
