joelrobin18 opened a new pull request, #58987: URL: https://github.com/apache/spark/pull/58987
### What changes were proposed in this pull request? This PR preserves the qualified input row of each SQL pipe `SET` assignment. It adds a transparent `PipeSetInput` logical-plan marker whose original qualified output is exposed as qualified-access-only metadata and materialized only when a parent expression needs it. The fixed-point and single-pass analyzers both understand the marker. Occurrence-aware output merging preserves source order and repeated attributes, while metadata pruning keeps retained columns out of the visible `SET` schema and unqualified stars. The marker is removed after analysis. The change also handles Spark Connect plan-ID resolution, qualified-star expansion, metadata-column overlap, and struct-field/table-alias name collisions without changing ordinary metadata or USING-join behavior. ### Why are the changes needed? Pipe `SET` replaces an assigned column in its projection. The original attribute is consequently dropped, even though Spark documents that table aliases continue to refer to original row values after an assignment. For example, `t.a` currently fails to resolve here: ```sql VALUES (1, 10) AS t(a, b) |> SET a = a + 1 |> SELECT a, t.a; ``` The assigned value must remain available as `a`, while `t.a` must resolve to the original value. ### Does this PR introduce _any_ user-facing change? Yes. Qualified references to columns affected by pipe `SET` now resolve to the original input value instead of failing analysis. In the example above, the result is `(2, 1)`. The visible `SET` schema and unqualified `SELECT *` output are unchanged. ### How was this patch tested? Added Catalyst and SQL regression coverage for scalar qualified access, qualified and unqualified stars, sequential assignments, repeated source attributes, intervening operators, USING joins, struct/table-alias collisions, metadata columns, plan IDs, and fixed-point/single-pass analyzer parity. The following checks passed: ```text build/sbt catalyst/Test/compile build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.analysis.AnalysisSuite -- -z "SPARK-59146"' build/sbt 'sql/testOnly org.apache.spark.sql.analysis.resolver.ResolverGuardSuite -- -z "SPARK-59146"' build/sbt 'sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z pipe-operators.sql' build/sbt catalyst/scalastyle catalyst/Test/scalastyle sql/Test/scalastyle ``` `git diff --check`, changed-source line-length checks, and non-ASCII checks also passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
