This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new c30a6890c132 [SPARK-60462][4.3][SQL][CONNECT] Keep re-selected hidden 
join keys regular in DataFrame column resolution
c30a6890c132 is described below

commit c30a6890c132e0eb1e0d85ef55d4cc88846acfc0
Author: Dejan Krakovic <[email protected]>
AuthorDate: Fri Oct 23 09:23:39 2026 +0000

    [SPARK-60462][4.3][SQL][CONNECT] Keep re-selected hidden join keys regular 
in DataFrame column resolution
    
    Backport of #59694 to branch-4.3.
    
    The cherry-pick had a conflict in 
`python/pyspark/sql/tests/connect/test_parity_column.py`: master has a block of 
Connect-only `test_resolve_after_*` tests next to the insertion point that 
branch-4.3 does not have. The resolution adds only this change's two tests; the 
source change and the other test file apply unchanged.
    
    ### What changes were proposed in this pull request?
    `resolveDataFrameColumnRecursively` tags each plan-id candidate as `hidden` 
or regular, and `resolveDataFrameColumn` prefers regular candidates 
(SPARK-56632). A candidate was tagged `hidden` when its references were a 
subset of `p.metadataOutput`. But an attribute can be in both `p.output` and 
`p.metadataOutput`: a Project that re-selects the hidden right key of a USING 
join inherits that key in its `metadataOutput` (`Project.metadataOutput` falls 
back to `child.metadataOutput`). So a [...]
    
    This PR tags a candidate `hidden` at a node `p` when its references are not 
all visible at `p`, and keeps it hidden further up, as before. An attribute is 
visible at `p` if it is in `p.output` and `p` does not hide it itself, i.e. it 
is not in the part of `p.metadataOutput` that `p` introduces rather than 
inherits from its children:
    
    ```scala
    val introducedHidden = AttributeSet(p.metadataOutput) --
      p.children.flatMap(_.metadataOutput)
    val visible = p.outputSet -- introducedHidden
    ...
    val hidden = c.hidden || !c.expr.references.subsetOf(visible)
    ```
    
    So a Project that re-selects a USING join's hidden right key, and a USING 
join wrapper above it, only inherit the key in their `metadataOutput`, and a 
candidate there stays regular. The USING join wrapper that hides the key keeps 
it hidden, also after `AddMetadataColumns` adds it to the wrapper's output for 
a select above, and so does a relation for a metadata column it exposes the 
same way. A rule based on `p.outputSet` alone would make those widened nodes 
count as visible, which cha [...]
    
    Which candidates survive is unchanged (still `p.output ++ 
p.metadataOutput`), and so is the SPARK-56632 behavior of preferring a direct 
candidate over one hidden under a natural/USING join wrapper.
    
    ### Why are the changes needed?
    With Spark Connect, this fails since SPARK-56632 (4.2.0):
    
    ```python
    fact = spark.createDataFrame([(1, 10), (2, 20)], ["fid", "dk"])
    dim = spark.createDataFrame([(10, "a"), (20, "b")], ["dk", "name"])
    other = spark.createDataFrame([(1, 10, 7), (2, 30, 8)], ["fid", "dk", 
"qty"])
    sel = fact.join(dim, "dk", "left").select(fact["fid"], dim["dk"])  # 
re-selects the hidden key
    q = sel.join(other.select("fid", "qty"), "fid")
    r = other.select("dk", "qty").join(sel, "dk")                       # sel's 
dk is hidden here
    q.join(r, "fid", "left").sort("fid").select(sel["dk"]).collect()
    # [AMBIGUOUS_COLUMN_REFERENCE] Column "dk" is ambiguous. ...
    ```
    
    Before SPARK-56632, `sel["dk"]` resolved to the `q`-side `dk`.
    
    ### Does this PR introduce _any_ user-facing change?
    Yes, a bug fix. In Spark Connect, the query above fails with 
`AMBIGUOUS_COLUMN_REFERENCE` in 4.2.0; with this change it resolves `sel["dk"]` 
to the visible candidate and returns `[10, 20]`.
    
    ### How was this patch tested?
    Added 
`ColumnTestsMixin.test_select_reselected_hidden_join_key_with_reused_dataframe`,
 which runs in classic and Spark Connect. It asserts the exact rows `[10, 20]`; 
resolving to the hidden candidate would give `[10, NULL]`. Without the fix, the 
Spark Connect test fails with `AMBIGUOUS_COLUMN_REFERENCE`; the classic one 
passes either way, since classic resolves `df["dk"]` eagerly.
    
    ```
    python/run-tests --testnames 
'pyspark.sql.tests.test_column,pyspark.sql.tests.connect.test_parity_column'
    ```
    
    This includes the SPARK-56632 regression test 
`test_select_regular_column_with_reused_dataframe_hidden_in_natural_join`. Also 
added two Spark Connect tests, in strict and non-strict DataFrame column 
resolution, for shapes where `AddMetadataColumns` widens the USING join wrapper 
of a re-selected hidden key: they resolve as before SPARK-56632 rather than to 
a different candidate or with `AMBIGUOUS_COLUMN_REFERENCE`, which a rule based 
on `p.outputSet` alone would give. In classic mode t [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #59718 from dejankrak-db/SPARK-60462-branch-4.3.
    
    Authored-by: Dejan Krakovic <[email protected]>
    Signed-off-by: Uros Bojanic <[email protected]>
    (cherry picked from commit bb2b30a7ce275f4161dff6dc32ab6737da93c8fd)
    Signed-off-by: Uros Bojanic <[email protected]>
---
 .../sql/tests/connect/test_parity_column.py        | 37 ++++++++++++++++++++++
 python/pyspark/sql/tests/test_column.py            | 23 ++++++++++++++
 .../catalyst/analysis/ColumnResolutionHelper.scala | 26 ++++++++++++---
 3 files changed, 81 insertions(+), 5 deletions(-)

diff --git a/python/pyspark/sql/tests/connect/test_parity_column.py 
b/python/pyspark/sql/tests/connect/test_parity_column.py
index a2b00d7955ee..cf8fb496a185 100644
--- a/python/pyspark/sql/tests/connect/test_parity_column.py
+++ b/python/pyspark/sql/tests/connect/test_parity_column.py
@@ -40,6 +40,43 @@ class ColumnParityTests(ColumnTestsMixin, 
ReusedConnectTestCase):
     def test_validate_column_types(self):
         super().test_validate_column_types()
 
+    # Plan-id resolution of hidden USING join keys that `AddMetadataColumns` 
adds to
+    # the output of the wrapper that hides them. Classic rejects these 
self-joins
+    # through its own ambiguity check.
+
+    def test_select_hidden_join_key_widened_by_reuse(self):
+        # A hidden USING join key that a later select re-selects is added to 
the
+        # output of the USING join wrapper that hides it. Such a candidate 
must stay
+        # hidden, so the direct candidate is still preferred.
+        a = self.spark.createDataFrame([(1, 1), (2, 1)], ["k", "x"])
+        b = self.spark.createDataFrame([(1, "p"), (2, "q")], ["k", "y"])
+        other = self.spark.createDataFrame([(1, "z")], ["x", "w"])
+        w = a.join(b.alias("b"), "k")
+        s = w.select(w["x"], w["b.k"]).join(other, "x")
+        # `w["b.k"]` resolves to the direct `w`, so both `s` rows join its row 
with k 1.
+        # Resolving it to the `k` in `s` would instead keep only the `s` row 
with k 1
+        # and join it to both `w` rows.
+        result = s.join(w, s["x"] == w["b.k"])
+        self.assertEqual(
+            sorted(tuple(r) for r in result.collect()),
+            [(1, 1, "z", 1, 1, "p"), (1, 2, "z", 1, 1, "p")],
+        )
+
+    def 
test_select_reused_dataframe_hidden_key_reselected_in_natural_join(self):
+        # The SPARK-56632 shape, where `enriched` re-selects the hidden 
`df2["id"]`,
+        # which adds it to the output of the USING join wrapper that hides it.
+        df1 = self.spark.createDataFrame([(10, "T1"), (20, "T2")], ["key", 
"val"])
+        df2 = self.spark.createDataFrame([(10,), (20,), (30,)], ["id"])
+        df3 = self.spark.createDataFrame([(10, "T1"), (99, "T2")], ["id", 
"val"])
+        enriched = df3.join(df2, "id", "left").select(df3["val"], df2["id"])
+        result = (
+            df1.join(df2, df1["key"] == df2["id"], "left")
+            .join(enriched, "val", "full_outer")
+            .sort("val")
+            .select(df2["id"])
+        )
+        self.assertEqual([r["id"] for r in result.collect()], [10, 20])
+
     def test_resolve_after_union(self):
         # Connect diverges from Classic here: Union is treated as a leaf when
         # walking the plan tree for plan-id resolution, so the left-side plan
diff --git a/python/pyspark/sql/tests/test_column.py 
b/python/pyspark/sql/tests/test_column.py
index 6a99c7de1a52..82b172f2437f 100644
--- a/python/pyspark/sql/tests/test_column.py
+++ b/python/pyspark/sql/tests/test_column.py
@@ -587,6 +587,29 @@ class ColumnTestsMixin:
             [10, 20],
         )
 
+    def test_select_reselected_hidden_join_key_with_reused_dataframe(self):
+        # `sel` re-selects `dim["dk"]`, the hidden right key of a USING join, 
so in
+        # `sel` that attribute is in both `p.output` and the `p.metadataOutput`
+        # inherited from the USING-join wrapper. `sel` is then reused: once 
directly
+        # (in `q`) and once as the right side of another USING join on `dk` 
(in `r`),
+        # where its `dk` is hidden. Resolving `sel["dk"]` must treat the `q` 
candidate
+        # as regular and prefer it over the hidden `r` one, not throw
+        # AMBIGUOUS_COLUMN_REFERENCE.
+        fact = self.spark.createDataFrame([(1, 10), (2, 20)], ["fid", "dk"])
+        dim = self.spark.createDataFrame([(10, "a"), (20, "b")], ["dk", 
"name"])
+        # No `other` row has dk 20, so `r` drops fid 2 and the left join below 
yields
+        # NULL on the `r` side for it. If `sel["dk"]` were resolved to the 
hidden `r`
+        # candidate, the second row would be NULL instead of 20.
+        other = self.spark.createDataFrame([(1, 10, 7), (2, 30, 8)], ["fid", 
"dk", "qty"])
+        sel = fact.join(dim, "dk", "left").select(fact["fid"], dim["dk"])
+        q = sel.join(other.select("fid", "qty"), "fid")
+        r = other.select("dk", "qty").join(sel, "dk")
+        result = q.join(r, "fid", "left").sort("fid").select(sel["dk"])
+        self.assertEqual(
+            [row["dk"] for row in result.collect()],
+            [10, 20],
+        )
+
     def test_drop_notexistent_col(self):
         df1 = self.spark.createDataFrame(
             [("a", "b", "c")],
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
index 55488e8eed95..bfc213dec665 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
@@ -588,9 +588,10 @@ trait ColumnResolutionHelper extends Logging with 
DataTypeErrorsBase {
   // Candidate threaded through the plan walk:
   //   - expr:   the resolved expression
   //   - depth:  the depth at which it was matched
-  //   - hidden: whether the candidate's references live in `p.metadataOutput`
-  //             at some plan node along the way (e.g. a natural/USING join
-  //             wrapper that hides a join key via `Project.hiddenOutputTag`).
+  //   - hidden: whether the candidate's references are not all visible at some
+  //             plan node along the way, e.g. at a natural/USING join wrapper 
that
+  //             hides a join key via `Project.hiddenOutputTag` (see the 
filter in
+  //             `resolveDataFrameColumnRecursively`).
   private case class Candidate(expr: NamedExpression, depth: Int, hidden: 
Boolean)
 
   private def resolveDataFrameColumnByPlanId(
@@ -624,7 +625,7 @@ trait ColumnResolutionHelper extends Logging with 
DataTypeErrorsBase {
       // case, return an empty Seq to delay the failure, so it is possible to 
be
       // resolved in the next iteration.
       // Always initialize `hidden = false` here; the filter below will set it
-      // correctly based on whether the references live in `p.metadataOutput`.
+      // correctly based on whether the references are all visible at `p`.
       (resolved.map(Candidate(_, currentDepth, hidden = false)).toSeq, true)
     } else {
       val children = p match {
@@ -672,8 +673,23 @@ trait ColumnResolutionHelper extends Logging with 
DataTypeErrorsBase {
     // via `Project.hiddenOutputTag`). We accept that here but tag the 
candidate
     // as `hidden` so the top-level merge in `resolveDataFrameColumn` can 
prefer
     // a regular (p.output) match over hidden (p.metadataOutput) ones.
+    //
+    // A candidate becomes hidden at `p` when its references are not all 
visible at `p`,
+    // and once hidden it stays hidden further up (`c.hidden`). An attribute 
is visible
+    // at `p` if it is in `p.output` and `p` does not hide it itself, i.e. it 
is not in
+    // the part of `p.metadataOutput` that `p` introduces rather than inherits 
from its
+    // children. An attribute can be in both `p.output` and `p.metadataOutput`:
+    //   - A Project that re-selects a USING join's hidden right key inherits 
the key in
+    //     its `metadataOutput` (`Project.metadataOutput` falls back to the 
child's), and
+    //     so does a USING join wrapper above it. The key is visible at both.
+    //   - `AddMetadataColumns` adds a hidden key to the `output` of the USING 
join
+    //     wrapper that hides it, and a metadata column to the `output` of a 
relation.
+    //     That node introduces the attribute in its `metadataOutput`, so it 
stays hidden.
+    val introducedHidden = AttributeSet(p.metadataOutput) --
+      p.children.flatMap(_.metadataOutput)
+    val visible = p.outputSet -- introducedHidden
     val filtered = candidates.flatMap { c =>
-      val hidden = c.hidden || 
c.expr.references.subsetOf(AttributeSet(p.metadataOutput))
+      val hidden = c.hidden || !c.expr.references.subsetOf(visible)
       if (c.expr.references.subsetOf(AttributeSet(p.output ++ 
p.metadataOutput))) {
         Some(c.copy(hidden = hidden))
       } else {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to