cloud-fan commented on code in PR #48649:
URL: https://github.com/apache/spark/pull/48649#discussion_r1836075681
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1260,8 +1264,32 @@ class AstBuilder extends DataTypeAstBuilder
def createProject() = if (namedExpressions.nonEmpty) {
val newProjectList: Seq[NamedExpression] = if (isPipeOperatorSelect) {
- // If this is a pipe operator |> SELECT clause, add a [[PipeSelect]]
expression wrapping
- // each alias in the project list, so the analyzer can check
invariants later.
+ // If this is a pipe operator |> SELECT clause,
+ // (1) validate all the window references after OVER are valid, and
+ val windowDefs = Option(windowClause)
+ .map(_.namedWindow.asScala.map(_.name.getText).toSet)
+ .getOrElse(collection.immutable.Set.empty[String])
+ // Collect all window names from UnresolvedWindowExpressions
+ val unresolvedWindowNames = namedExpressions.collect {
+ case Alias(wExpr: UnresolvedWindowExpression, _) =>
wExpr.windowSpec.name
+ case UnresolvedAlias(wExpr: UnresolvedWindowExpression, _) =>
wExpr.windowSpec.name
+ }
+ if (unresolvedWindowNames.nonEmpty) {
+ if (windowDefs.isEmpty) {
+ // No window definitions provided, throw error for the first
unresolved window
+ throw
QueryParsingErrors.cannotFindWindowReferenceError(unresolvedWindowNames.head,
ctx)
+ } else {
+ // Find any unresolved window names not defined in windowDefs
+ unresolvedWindowNames.find(!windowDefs.contains(_)) match {
Review Comment:
It's better to not do analysis work in the parser, as we may miss something
when re-implementing the analysis work. E.g. shall we consider case sensitivity
here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
