spark-test commented on code in PR #59843:
URL: https://github.com/apache/spark/pull/59843#discussion_r4395597720
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala:
##########
@@ -275,14 +281,47 @@ class JoinResolver(
hiddenOutput = Some(newHiddenOutput)
)
- val resolvedJoin = partiallyResolvedJoin.copy(condition =
resolvedCondition)
+ val resolvedJoin = retainTopLevelJoinOutput(
+ unresolvedJoin = unresolvedJoin,
+ resolvedJoin = partiallyResolvedJoin.copy(condition = resolvedCondition),
+ conditionReferencedAttributes = conditionReferencedAttributes
+ )
cteRegistry.currentScope.tryPutWithCTE(
unresolvedOperator = unresolvedJoin,
resolvedOperator = resolvedJoin
)
}
+ /**
+ * Wraps a top-level [[Join]] in a [[Project]] with the join's output when
its condition
+ * references a column found only in the hidden output, such as the hidden
key of a USING join
+ * below, which is added to a child's output. For example:
+ *
+ * {{{
+ * FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k
+ *
+ * Project [k#1, v1#2, v2#3, k#5, v2#6]
+ * +- Join Inner, (k#4 = k#5)
+ * :- Project [k#1, v1#2, v2#3, k#4]
+ * : +- Join Inner, (k#1 = k#4)
+ * +- ...
+ * }}}
+ */
+ private def retainTopLevelJoinOutput(
+ unresolvedJoin: Join,
+ resolvedJoin: Join,
+ conditionReferencedAttributes: HashMap[ExprId, Attribute]): LogicalPlan
= {
+ def referencesHiddenOutput: Boolean =
+
scopes.current.resolveMissingAttributesByHiddenOutput(conditionReferencedAttributes).nonEmpty
Review Comment:
**Non-blocking (P2):** This check counts any condition-referenced attribute
that is in the join scope's hidden output but not in its main output. The
hidden output built by `computeHiddenOutputForRegularJoin` keeps the
qualified-access-only keys of both children regardless of join type, while
`Join.computeOutput` returns only the left output for LEFT SEMI and LEFT ANTI.
Take a top-level semi or anti join whose condition references a right-side
hidden USING or NATURAL key:
```sql
FROM nt2 x LEFT SEMI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
```
The single-pass resolver now returns `Project [k, v2] +- Join LeftSemi`.
Fixed-point `AddMetadataColumns` appends the key to the right child's Project
but keeps the bare join, because `sameOutput` still holds: the semi join's
output did not change. `NormalizePlan` does not strip this Project, so under
dual-run (the test default) the query now fails with
`HYBRID_ANALYZER_EXCEPTION.LOGICAL_PLAN_COMPARISON_MISMATCH`. Before this PR,
both plans were the bare join. When the single-pass result is used directly,
the extra Project is a no-op and rows are unchanged, and the default
fixed-point path is unaffected. The fixed-point rule wraps only when the added
hidden column becomes part of the join's own output, and this check does not
take that into account.
The new `using-join.sql` cases cover only an inner join whose hidden key
comes from the left child. Neither this case nor any other join-type or
side-dependent mistake in the check would fail a golden. (Traced from source,
not executed.)
**Verification:**
- **Regression:** Under dual-run, a top-level FROM-first LEFT SEMI join and
a LEFT ANTI join over a right-side USING join, whose condition references the
right child's hidden key, analyze without a plan-comparison mismatch and keep
the bare join in the analyzed plan.
- **Behavior:** Under dual-run, top-level joins whose referenced hidden key
comes from the left child, including a LEFT SEMI join, and inner or left outer
joins whose key comes from the right child, still analyze to a Project that
restores the join output, and return the expected rows and schema.
- **Compatibility:** The existing using-join and natural-join golden
queries, including the two queries this PR adds, keep their current analyzed
plans and results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
