HyukjinKwon commented on code in PR #59267:
URL: https://github.com/apache/spark/pull/59267#discussion_r4271050826
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AsOfJoinResolver.scala:
##########
@@ -124,11 +132,62 @@ class AsOfJoinResolver(
)
cteRegistry.currentScope.tryPutWithCTE(
unresolvedOperator = unresolvedAsOfJoin,
- resolvedOperator = resolvedJoin
+ resolvedOperator = resolvedJoinWithOriginalOutput
)
}
}
+ /**
+ * If the join uses a hidden column, wraps it in a [[Project]] that also
keeps the hidden columns.
+ */
+ private def retainOriginalOutputIfHiddenColumnIsReferenced(
+ unresolvedAsOfJoin: AsOfJoin,
+ resolvedJoin: AsOfJoin,
+ leftNameScope: NameScope,
+ rightNameScope: NameScope): LogicalPlan = {
+ val originalOutput = AsOfJoin.computeOutput(
+ joinType = resolvedJoin.joinType,
+ leftOutput = leftNameScope.output,
+ rightOutput = rightNameScope.output
+ )
+ val originalOutputSet = AttributeSet(originalOutput)
+
+ if (resolvedJoin.references.subsetOf(originalOutputSet)) {
+ resolvedJoin
+ } else {
+ val qualifiedAccessOnlyColumns = AsOfJoin.computeOutput(
Review Comment:
**Nit (P3):** These appended hidden columns now go through
`AsOfJoin.computeOutput(resolvedJoin.joinType, ...)`, as does `originalOutput`
above. Under `LEFT ASOF JOIN` that makes the right side's visible and hidden
columns nullable, which matches `AsOfJoin.output`. Fixed-point reaches the same
result differently: `Project(node.output)` from `AddMetadataColumns`, plus
`UpdateAttributeNullability` for appended references. Every new golden
(FVT-ASOF-2-027..040) and every new `AsOfJoinSQLSuite` case uses an inner ASOF
JOIN, so dual-run never compares the LeftOuter wrapper. If this branch
regressed, for example by dropping the join-type widening, no test would fail.
A LEFT ASOF golden with a hidden `MATCH_CONDITION` operand would cover it.
Keep the parent from referencing the hidden key, so the test isolates this
branch from the existing single-pass handling of nested hidden columns under
outer joins. Also include a left row with no match. For example:
```sql
SELECT l.lts, r.rts
FROM using_left l LEFT ASOF JOIN (using_right r LEFT JOIN using_keys d USING
(k))
MATCH_CONDITION (l.k <= d.k)
```
Here `l.k = 8` has no `d.k >= 8`, so it should come back with a NULL `rts`.
**Verification:**
- **Regression:** A LEFT ASOF JOIN with a hidden MATCH_CONDITION operand
resolves with matching single-pass and fixed-point plans under dual-run, so a
nullability or shape regression in the LeftOuter wrapper branch would fail the
golden run.
- **Behavior:** The LEFT ASOF JOIN case returns the expected rows, including
NULL right-side values for left rows without a match.
- **Compatibility:** The previously generated ASOF JOIN goldens stay
identical after regeneration.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AsOfJoinResolver.scala:
##########
@@ -124,11 +132,43 @@ class AsOfJoinResolver(
)
cteRegistry.currentScope.tryPutWithCTE(
unresolvedOperator = unresolvedAsOfJoin,
- resolvedOperator = resolvedJoin
+ resolvedOperator = resolvedJoinWithOriginalOutput
)
}
}
+ /**
+ * If the join uses a hidden column, wraps it in a [[Project]] that also
keeps the hidden columns.
+ */
+ private def retainOriginalOutputIfHiddenColumnIsReferenced(
+ unresolvedAsOfJoin: AsOfJoin,
+ resolvedJoin: AsOfJoin,
+ leftNameScope: NameScope,
+ rightNameScope: NameScope): LogicalPlan = {
+ val originalOutput = AsOfJoin.computeOutput(
+ joinType = resolvedJoin.joinType,
+ leftOutput = leftNameScope.output,
+ rightOutput = rightNameScope.output
+ )
+ val originalOutputSet = AttributeSet(originalOutput)
+
+ if (resolvedJoin.references.subsetOf(originalOutputSet)) {
+ resolvedJoin
+ } else {
+ val hiddenColumns =
+ if (unresolvedAsOfJoin.containsTag(ResolverTag.TOP_LEVEL_OPERATOR)) {
Review Comment:
Thanks! FVT-ASOF-2-040 covers it: the bare FROM golden keeps the wrap
Project at [k, lts, k, rts] and the result schema leaves out d.k, so a
regression that appended hidden columns at the top level would now show up in
the goldens.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:4261554266","thread_id":"inline:4261554266","verdict_sha256":"851d55e3a512a10698033079680860d217e8ba21dcc47c4bc3bd4dfa384ce99f"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
