AMC-hawk commented on code in PR #58760:
URL: https://github.com/apache/spark/pull/58760#discussion_r4057441224
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -2085,6 +2086,45 @@ class Analyzer(
}.map(_.asInstanceOf[NamedExpression])
}
+ /**
+ * The SQL pipe SET operator is implemented as a star expansion that
excludes the assigned
+ * column and appends a replacement of the same name. That drops the
original attribute from
+ * the project list, which would also make it unreachable through its
table alias, contradicting
+ * the documented behavior that table aliases keep referring to the
original row values after an
+ * assignment. Retain the excluded attributes as hidden output instead,
the same way USING joins
+ * hide their duplicated join keys (SPARK-59146).
+ *
+ * The hidden output holds the whole qualified source row in its original
order, not just the
+ * excluded attributes, because a qualified star expands hidden output
ahead of the visible
+ * output. Retaining only the excluded attributes would move them to the
front, so that
+ * `VALUES (1, 2, 3) AS t(a, b, c) |> SET b = 20 |> SELECT t.*` returned
`(b, a, c)`. Attributes
+ * that are also visible are emitted once by the star expansion, in their
hidden position.
+ */
+ private def retainExceptedColumnsAsHiddenOutput(original: Project,
expanded: Project): Unit = {
+ val retain = original.projectList.exists {
+ case s: UnresolvedStarExceptOrReplace =>
s.retainExceptedColumnsAsHidden
+ case _ => false
+ }
+ if (retain) {
+ val child = expanded.child
+ if (child.output.exists(!expanded.outputSet.contains(_))) {
+ // The row a qualified star sees on the child: its qualified-only
hidden output first,
+ // then its output. Only qualified attributes are reachable through
a qualified star.
+ val sourceRow = (child.metadataOutput.filter(_.qualifiedAccessOnly)
++ child.output)
+ .filter(_.qualifier.nonEmpty)
+ .distinctBy(_.exprId)
+ val sourceRowIds = sourceRow.map(_.exprId).toSet
+ // The rule forwards the original tags, such as the Spark Connect
plan id, only onto a
+ // node without tags, so copy them here before adding the hidden
output tag.
+ expanded.copyTagsFrom(original)
+ expanded.setTagValue(
Review Comment:
The preference would be to retain `hiddenOutputTag` as the carrier,
consistent with the existing analyzer lifecycle, added golden coverage across
the relevant projection-rebuilding rules in `262b18bbe7d`, and happy to add a
case if any rule does not forward the tag correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
