allisonwang-db commented on a change in pull request #32503:
URL: https://github.com/apache/spark/pull/32503#discussion_r632198811
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -1445,4 +1445,10 @@ private[spark] object QueryCompilationErrors {
def operationNotSupportPartitioningError(operation: String): Throwable = {
new AnalysisException(s"'$operation' does not support partitioning")
}
+
+ def mixedRefInAggFunc(a: Expression): Throwable = {
Review comment:
We can tighten the type here:`a: Expression` -> `a: AggregateExpression`
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala
##########
@@ -189,16 +189,19 @@ object SubExprUtils extends PredicateHelper {
*/
def getOuterReferences(expr: Expression): Seq[Expression] = {
val outerExpressions = ArrayBuffer.empty[Expression]
- expr transformDown {
Review comment:
This is tricky. I tried updating this function as well and discovered
that `transformDown` is actually needed.
https://github.com/apache/spark/blob/8fa739fb9d3bd49efa7df7525df18b7111b0131e/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/subquery.scala#L193-L198
Notice the `newExpr` here has outer references removed. Say we have an
expression `min(outer(a))` and if we use `foreach`, we will collect both
`min(a)` and `a`, but if we use `transformDown`, the expression tree
`min(outer(a))` will be transformed into `min(a)` so no more outer references
will be collected.
(I added the comment but it's probably not very clear.)
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
##########
@@ -1445,4 +1445,10 @@ private[spark] object QueryCompilationErrors {
def operationNotSupportPartitioningError(operation: String): Throwable = {
new AnalysisException(s"'$operation' does not support partitioning")
}
+
+ def mixedRefInAggFunc(a: Expression): Throwable = {
+ val msg = "Found an aggregate function in a correlated predicate that has
both " +
+ "outer and local references, which is not supported: " + a.sql
Review comment:
`aggregate fucntion` -> `aggregate expression`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
