cloud-fan commented on code in PR #48649:
URL: https://github.com/apache/spark/pull/48649#discussion_r1836079778
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -1260,8 +1264,32 @@ class AstBuilder extends DataTypeAstBuilder
def createProject() = if (namedExpressions.nonEmpty) {
val newProjectList: Seq[NamedExpression] = if (isPipeOperatorSelect) {
- // If this is a pipe operator |> SELECT clause, add a [[PipeSelect]]
expression wrapping
- // each alias in the project list, so the analyzer can check
invariants later.
+ // If this is a pipe operator |> SELECT clause,
+ // (1) validate all the window references after OVER are valid, and
+ val windowDefs = Option(windowClause)
+ .map(_.namedWindow.asScala.map(_.name.getText).toSet)
+ .getOrElse(collection.immutable.Set.empty[String])
+ // Collect all window names from UnresolvedWindowExpressions
+ val unresolvedWindowNames = namedExpressions.collect {
+ case Alias(wExpr: UnresolvedWindowExpression, _) =>
wExpr.windowSpec.name
+ case UnresolvedAlias(wExpr: UnresolvedWindowExpression, _) =>
wExpr.windowSpec.name
+ }
+ if (unresolvedWindowNames.nonEmpty) {
+ if (windowDefs.isEmpty) {
+ // No window definitions provided, throw error for the first
unresolved window
+ throw
QueryParsingErrors.cannotFindWindowReferenceError(unresolvedWindowNames.head,
ctx)
+ } else {
+ // Find any unresolved window names not defined in windowDefs
+ unresolvedWindowNames.find(!windowDefs.contains(_)) match {
Review Comment:
@Angryrou can you highlight the logical plan differences between classic and
pipe SQL syntaxes? I'd like to understand why we can reference previously
defined window frame in the pipe syntax and see how we can fix it in the
analyzer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
