dejankrak-db opened a new pull request, #59694:
URL: https://github.com/apache/spark/pull/59694
### What changes were proposed in this pull request?
`resolveDataFrameColumnRecursively` tags each plan-id candidate as `hidden`
or regular, and `resolveDataFrameColumn` prefers regular candidates
(SPARK-56632). A candidate was tagged `hidden` when its references were a
subset of `p.metadataOutput`. But an attribute can be in both `p.output` and
`p.metadataOutput`: a Project that re-selects the hidden right key of a USING
join inherits that key in its `metadataOutput` (`Project.metadataOutput` falls
back to `child.metadataOutput`). So a visible candidate was tagged `hidden`.
When the same DataFrame is also reused where that key really is hidden, both
candidates are `hidden`, neither is a depth-0 match, and resolution fails with
`AMBIGUOUS_COLUMN_REFERENCE`.
This PR tags a candidate `hidden` only when its references are not all in
`p.outputSet`:
```scala
- val hidden = c.hidden ||
c.expr.references.subsetOf(AttributeSet(p.metadataOutput))
+ val hidden = c.hidden || !c.expr.references.subsetOf(p.outputSet)
```
Which candidates survive is unchanged (still `p.output ++
p.metadataOutput`), and so is the SPARK-56632 behavior of preferring a direct
candidate over one hidden under a natural/USING join wrapper.
### Why are the changes needed?
With Spark Connect, this fails since SPARK-56632 (4.2.0):
```python
fact = spark.createDataFrame([(1, 10), (2, 20)], ["fid", "dk"])
dim = spark.createDataFrame([(10, "a"), (20, "b")], ["dk", "name"])
other = spark.createDataFrame([(1, 10, 7), (2, 30, 8)], ["fid", "dk", "qty"])
sel = fact.join(dim, "dk", "left").select(fact["fid"], dim["dk"]) #
re-selects the hidden key
q = sel.join(other.select("fid", "qty"), "fid")
r = other.select("dk", "qty").join(sel, "dk") # sel's
dk is hidden here
q.join(r, "fid", "left").sort("fid").select(sel["dk"]).collect()
# [AMBIGUOUS_COLUMN_REFERENCE] Column "dk" is ambiguous. ...
```
Before SPARK-56632, `sel["dk"]` resolved to the `q`-side `dk`.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix. In Spark Connect, the query above fails with
`AMBIGUOUS_COLUMN_REFERENCE` in 4.2.0; with this change it resolves `sel["dk"]`
to the visible candidate and returns `[10, 20]`.
### How was this patch tested?
Added
`ColumnTestsMixin.test_select_reselected_hidden_join_key_with_reused_dataframe`,
which runs in classic and Spark Connect. It asserts the exact rows `[10, 20]`;
resolving to the hidden candidate would give `[10, NULL]`. Without the fix, the
Spark Connect test fails with `AMBIGUOUS_COLUMN_REFERENCE`; the classic one
passes either way, since classic resolves `df["dk"]` eagerly.
```
python/run-tests --testnames
'pyspark.sql.tests.test_column,pyspark.sql.tests.connect.test_parity_column'
```
This includes the SPARK-56632 regression test
`test_select_regular_column_with_reused_dataframe_hidden_in_natural_join`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
