This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new a424b71aa9ca [SPARK-56632][4.1][SQL][CONNECT] Fix
AMBIGUOUS_COLUMN_REFERENCE regression for reused DataFrame in natural join
a424b71aa9ca is described below
commit a424b71aa9cae9555fb7abf1ab8bd69b074333ea
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Fri Oct 23 08:26:12 2026 +0800
[SPARK-56632][4.1][SQL][CONNECT] Fix AMBIGUOUS_COLUMN_REFERENCE regression
for reused DataFrame in natural join
### What changes were proposed in this pull request?
Backport https://github.com/apache/spark/pull/55582 to `branch-4.1`
(upstream commit `266f7674b96a07958ef117194eae325565b53004`).
Prefer visible column candidates over hidden natural/USING-join candidates
and defer ambiguity checks until all matching candidates have been collected.
Include the original PySpark regression test, shared by classic and Connect
suites.
The Scala fix applies unchanged. Resolve the Python context conflict by
adding only the original regression test, with branch-compatible spacing.
### Why are the changes needed?
SPARK-55070 is present in 4.1 and can cause `AMBIGUOUS_COLUMN_REFERENCE`
when a DataFrame is used directly in a join and also nested under a
natural/USING join in the same plan.
### Does this PR introduce _any_ user-facing change?
Yes. Affected queries resolve the visible direct-join column instead of
raising `AMBIGUOUS_COLUMN_REFERENCE`, while preserving resolution when only
hidden candidates are available.
### How was this patch tested?
- Carry over
`test_select_regular_column_with_reused_dataframe_hidden_in_natural_join` from
#55582; it checks the selected values as well as successful resolution.
- Verify that the Scala additions/removals and regression test match
upstream.
- Pass Black 23.12.1 with the branch's configuration and `git diff
--check`; check added lines for ASCII and the 100-character limit.
- Local compilation and runtime tests were not completed: Catalyst
compilation could not download the required SBT 1.11.7 launcher. The Maven
Central hostnames were unavailable from this environment. CI will provide build
and test validation.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-6), for backport preparation and conflict
resolution.
Original patch: Generated-by: Claude Code (model: claude-opus-4-7).
Closes #59705 from zhengruifeng/backport-55582-branch-4.1.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/sql/tests/test_column.py | 27 ++++++++
.../catalyst/analysis/ColumnResolutionHelper.scala | 73 +++++++++++++++-------
2 files changed, 78 insertions(+), 22 deletions(-)
diff --git a/python/pyspark/sql/tests/test_column.py
b/python/pyspark/sql/tests/test_column.py
index 290be7a8dce5..66e9700e0cf8 100644
--- a/python/pyspark/sql/tests/test_column.py
+++ b/python/pyspark/sql/tests/test_column.py
@@ -501,6 +501,33 @@ class ColumnTestsMixin:
self.assertEqual(result[1][0], 40)
self.assertEqual(result[2][0], 60)
+ def
test_select_regular_column_with_reused_dataframe_hidden_in_natural_join(self):
+ # A DataFrame appears both as a direct join side and inside a
natural/USING
+ # join that hides one of its columns into `metadataOutput`. When
resolving
+ # `df2["id"]`, two candidates match the plan id: one from `p.output`
(the
+ # direct join side) and one only visible via `p.metadataOutput` (the
reused
+ # `df2` nested under the USING-join wrapper). We should prefer the
regular
+ # candidate and not throw AMBIGUOUS_COLUMN_REFERENCE.
+ df1 = self.spark.createDataFrame([(10, "T1"), (20, "T2")], ["key",
"val"])
+ df2 = self.spark.createDataFrame([(10,), (20,), (30,)], ["id"])
+ # The second row's id (99) does not match any df2 row, so the USING
+ # left-join in `enriched` produces NULL on the df2 side for val "T2".
+ # If `df2["id"]` were resolved to the hidden (USING-wrapper) candidate,
+ # the second row would yield NULL instead of 20, and the assertion
below
+ # would fail. This pins resolution to the direct-side `id`.
+ df3 = self.spark.createDataFrame([(10, "T1"), (99, "T2")], ["id",
"val"])
+ enriched = df3.join(df2, "id", "left")
+ result = (
+ df1.join(df2, df1["key"] == df2["id"], "left")
+ .join(enriched, "val", "full_outer")
+ .sort("val")
+ .select(df2["id"])
+ )
+ self.assertEqual(
+ [r["id"] for r in result.collect()],
+ [10, 20],
+ )
+
class ColumnTests(ColumnTestsMixin, ReusedSQLTestCase):
pass
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
index 1172ecee7223..3f043ac97b07 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ColumnResolutionHelper.scala
@@ -501,7 +501,10 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
// 5, resolve the expression against the target node, the resolved
attribute will be
// filtered by the output attributes of nodes in the path (from
matching to root node);
// 6. if more than one resolved attributes are found in the above
recursive process,
- // fails with 'AMBIGUOUS_COLUMN_REFERENCE'.
+ // disambiguate by preferring regular candidates (visible via
`output`) over
+ // hidden ones (only via `metadataOutput`), then preferring depth-0
(direct)
+ // matches over deeper ones; fails with 'AMBIGUOUS_COLUMN_REFERENCE'
if neither
+ // tiebreaker yields a single winner.
// 7. if all the resolved attributes are filtered out, return the
original expression
// as it is.
private def tryResolveDataFrameColumns(
@@ -526,7 +529,7 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
val isMetadataAccess = u.getTagValue(LogicalPlan.IS_METADATA_COL).nonEmpty
- val (resolved, matched) = resolveDataFrameColumnByPlanId(
+ val (candidates, matched) = resolveDataFrameColumnByPlanId(
u, planId, isMetadataAccess, q, 0)
if (!matched) {
// Can not find the target plan node with plan id, e.g.
@@ -535,27 +538,41 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
// df1.select(df2.a) <- illegal reference df2.a
throw QueryCompilationErrors.cannotResolveDataFrameColumn(u)
}
- resolved.map(_._1)
+
+ // If there is at least one regular (`hidden = false`) candidate, run the
+ // merge over regular candidates only and ignore hidden ones (e.g. a
+ // natural/USING join hidden key). Otherwise run the merge over hidden
+ // candidates.
+ val (regular, hidden) = candidates.partition(!_.hidden)
+ val finalists = if (regular.nonEmpty) regular else hidden
+ finalists.sortBy(_.depth).foldLeft(Option.empty[Candidate]) {
+ case (None, c) => Some(c)
+ // If the current winner is a direct match (depth 0) and a further
+ // candidate is nested deeper, prefer the direct one.
+ case (Some(c1), c2) if c1.depth == 0 && c2.depth != 0 => Some(c1)
+ case _ => throw QueryCompilationErrors.ambiguousColumnReferences(u)
+ }.map(_.expr)
}
+ // Candidate threaded through the plan walk:
+ // - expr: the resolved expression
+ // - depth: the depth at which it was matched
+ // - hidden: whether the candidate's references live in `p.metadataOutput`
+ // at some plan node along the way (e.g. a natural/USING join
+ // wrapper that hides a join key via `Project.hiddenOutputTag`).
+ private case class Candidate(expr: NamedExpression, depth: Int, hidden:
Boolean)
+
private def resolveDataFrameColumnByPlanId(
u: UnresolvedAttribute,
id: Long,
isMetadataAccess: Boolean,
q: Seq[LogicalPlan],
- currentDepth: Int): (Option[(NamedExpression, Int)], Boolean) = {
+ currentDepth: Int): (Seq[Candidate], Boolean) = {
val resolved = q.map(resolveDataFrameColumnRecursively(
u, id, isMetadataAccess, _, currentDepth))
- val merged = resolved
- .flatMap(_._1)
- .sortBy(_._2) // sort by depth
- .foldLeft(Option.empty[(NamedExpression, Int)]) {
- case (None, (r2, d2)) => Some((r2, d2))
- case (Some((r1, 0)), (r2, d2)) if d2 != 0 => Some((r1, 0))
- case _ => throw QueryCompilationErrors.ambiguousColumnReferences(u)
- }
+ val candidates = resolved.flatMap(_._1)
val matched = resolved.exists(_._2)
- (merged, matched)
+ (candidates, matched)
}
private def resolveDataFrameColumnRecursively(
@@ -563,8 +580,8 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
id: Long,
isMetadataAccess: Boolean,
p: LogicalPlan,
- currentDepth: Int): (Option[(NamedExpression, Int)], Boolean) = {
- val (resolved, matched) = if
(p.getTagValue(LogicalPlan.PLAN_ID_TAG).contains(id)) {
+ currentDepth: Int): (Seq[Candidate], Boolean) = {
+ val (candidates, matched) = if
(p.getTagValue(LogicalPlan.PLAN_ID_TAG).contains(id)) {
val resolved = if (!isMetadataAccess) {
p.resolve(u.nameParts, conf.resolver)
} else if (u.nameParts.size == 1) {
@@ -572,10 +589,12 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
} else {
None
}
- // The targe plan node is found, but might still fail to resolve.
- // In this case, return None to delay the failure, so it is possible to
be
+ // The target plan node is found, but might still fail to resolve. In
this
+ // case, return an empty Seq to delay the failure, so it is possible to
be
// resolved in the next iteration.
- (resolved.map(r => (r, currentDepth)), true)
+ // Always initialize `hidden = false` here; the filter below will set it
+ // correctly based on whether the references live in `p.metadataOutput`.
+ (resolved.map(Candidate(_, currentDepth, hidden = false)).toSeq, true)
} else {
val children = p match {
// treat Union node as the leaf node
@@ -613,12 +632,22 @@ trait ColumnResolutionHelper extends Logging with
DataTypeErrorsBase {
// df = spark.range(10).withColumn("v", sf.col("id") + 1)
// df.select(df.v).sort(df.id)
//
- // In this case, resolveDataFrameColumnByPlanId returns None,
+ // In this case, resolveDataFrameColumnByPlanId returns an empty Seq,
// the dataframe column 'df.id' will remain unresolved, and the analyzer
// will try to resolve 'id' without plan id later.
- val filtered = resolved.filter { r =>
- // A DataFrame column can be resolved as a metadata column, we should
keep it.
- r._1.references.subsetOf(AttributeSet(p.output ++ p.metadataOutput))
+ //
+ // A DataFrame column can also be resolved as a metadata column at some
+ // ancestor plan (e.g. a natural/USING join wrapper that hides a join key
+ // via `Project.hiddenOutputTag`). We accept that here but tag the
candidate
+ // as `hidden` so the top-level merge in `resolveDataFrameColumn` can
prefer
+ // a regular (p.output) match over hidden (p.metadataOutput) ones.
+ val filtered = candidates.flatMap { c =>
+ val hidden = c.hidden ||
c.expr.references.subsetOf(AttributeSet(p.metadataOutput))
+ if (c.expr.references.subsetOf(AttributeSet(p.output ++
p.metadataOutput))) {
+ Some(c.copy(hidden = hidden))
+ } else {
+ None
+ }
}
(filtered, matched)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]