cloud-fan commented on code in PR #58760:
URL: https://github.com/apache/spark/pull/58760#discussion_r4255700366
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala:
##########
@@ -626,6 +629,34 @@ trait UnresolvedStarBase extends Star with Unevaluable {
override def toString: String = target.map(_.mkString("", ".",
".")).getOrElse("") + "*"
}
+object UnresolvedStarBase {
+
+ /**
+ * Merges the hidden output of a plan with its visible output, in the order
a qualified star
+ * expands them.
+ *
+ * Hidden output usually holds attributes that are not visible at all, such
as the duplicated
+ * join keys of a USING join. It may also repeat an attribute that is
visible, to pin that
+ * attribute's position in the expansion, as the SQL pipe SET operator does
to keep the original
+ * column order of the row its table alias refers to (SPARK-59146).
+ *
+ * Every occurrence of the source row is therefore preserved: a hidden
attribute consumes at
+ * most one visible occurrence, which is emitted in the hidden attribute's
place, because a
+ * projection may repeat the same attribute. Visible attributes that no
hidden attribute
+ * consumed keep their relative order, after the hidden output.
+ */
+ def mergeHiddenAndVisibleOutput(
+ hiddenOutput: Seq[Attribute],
+ visibleOutput: Seq[Attribute]): Seq[Attribute] = {
+ val unconsumed = visibleOutput.to(mutable.ArrayBuffer)
+ val merged = hiddenOutput.map { hidden =>
+ val visible = unconsumed.indexWhere(_.exprId == hidden.exprId)
Review Comment:
**Non-blocking (P2):** Each hidden attribute scans the remaining visible
buffer, and every overlap is removed from the middle. With wide USING/NATURAL
joins this performs `H * V` searches; with pipe SET's full-width overlap, the
removals repeatedly shift the remaining suffix. Qualified-star analysis
therefore becomes quadratic before planning for both existing join queries and
the new SET path.
**Verification:**
- **Behavior:** Compare merged output for disjoint, overlapping,
repeated-ExprId, and repeated-SET inputs and assert exact order, multiplicity,
qualifiers, and expression identities.
- **Regression:** Measure analysis across increasing schema widths for pipe
SET plus a qualified star and for a wide USING/NATURAL join plus a qualified
star, demonstrating linear rather than quadratic growth.
- **Inspection:** Confirm the merge performs one visible indexing pass,
constant-time expected occurrence consumption, and one final visible-order
emission pass.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -2085,6 +2086,47 @@ class Analyzer(
}.map(_.asInstanceOf[NamedExpression])
}
+ /**
+ * The SQL pipe SET operator is implemented as a star expansion that
excludes the assigned
+ * column and appends a replacement of the same name. That drops the
original attribute from
+ * the project list, which would also make it unreachable through its
table alias, contradicting
+ * the documented behavior that table aliases keep referring to the
original row values after an
+ * assignment. Retain the excluded attributes as hidden output instead,
the same way USING joins
+ * hide their duplicated join keys (SPARK-59146).
+ *
+ * The hidden output holds the whole qualified source row, with every
position in its original
+ * order, not just the excluded attributes, because a qualified star
expands hidden output
+ * ahead of the visible output. Retaining only the excluded attributes
would move them to the
+ * front, so that `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT
t.*` returned
Review Comment:
**Nit (P3):** This sentence starts a counterfactual with `would move` but
completes it with `returned`, which makes the hypothetical `(b, a, c)` result
read like observed current behavior. Please use `would return` (or make both
clauses past-counterfactual) so the ordering rationale is unambiguous.
**Verification:**
- **Inspection:** Read the complete ordering comment and confirm its
condition and consequence use consistent counterfactual grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
