AMC-hawk commented on code in PR #58760:
URL: https://github.com/apache/spark/pull/58760#discussion_r4328777580
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala:
##########
@@ -626,6 +629,34 @@ trait UnresolvedStarBase extends Star with Unevaluable {
override def toString: String = target.map(_.mkString("", ".",
".")).getOrElse("") + "*"
}
+object UnresolvedStarBase {
+
+ /**
+ * Merges the hidden output of a plan with its visible output, in the order
a qualified star
+ * expands them.
+ *
+ * Hidden output usually holds attributes that are not visible at all, such
as the duplicated
+ * join keys of a USING join. It may also repeat an attribute that is
visible, to pin that
+ * attribute's position in the expansion, as the SQL pipe SET operator does
to keep the original
+ * column order of the row its table alias refers to (SPARK-59146).
+ *
+ * Every occurrence of the source row is therefore preserved: a hidden
attribute consumes at
+ * most one visible occurrence, which is emitted in the hidden attribute's
place, because a
+ * projection may repeat the same attribute. Visible attributes that no
hidden attribute
+ * consumed keep their relative order, after the hidden output.
+ */
+ def mergeHiddenAndVisibleOutput(
+ hiddenOutput: Seq[Attribute],
+ visibleOutput: Seq[Attribute]): Seq[Attribute] = {
+ val unconsumed = visibleOutput.to(mutable.ArrayBuffer)
+ val merged = hiddenOutput.map { hidden =>
+ val visible = unconsumed.indexWhere(_.exprId == hidden.exprId)
Review Comment:
Fixed in 5545d0d.
1. Inspection: the merge now indexes visible positions by ExprId in one
pass, consumes each in O(1) expected time, and emits the leftovers in one final
pass.
2. Behavior: new AnalysisSuite cases cover disjoint, overlapping,
repeated-ExprId and repeated-SET inputs, asserting exact instances. The new
merge also matched the old one on 20k random inputs.
5. Regression: this also turned up AddMetadataColumns scanning the
children's metadataOutput once per attribute, which made any wide SET
quadratic. It now looks attributes up by ExprId. Two SETs + t.* at 16k columns
went from 11.3 s to 0.33 s, now growing linearly. A qualified star over a wide
USING/NATURAL join also grows linearly now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
