Github user nsyca commented on a diff in the pull request:
https://github.com/apache/spark/pull/16246#discussion_r92041145
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
---
@@ -1225,15 +1225,34 @@ class Analyzer(
}
/**
- * Rewrite the subquery in a safe way by preventing that the subquery
and the outer use the same
- * attributes.
+ * Rewrite the subquery in a safe way by preventing that the subquery
and
+ * the outer use the same attributes.
+ *
+ * If this is a scalar subquery, check that GROUP BY columns are a
subset
+ * of the columns used in the correlated predicate(s). Otherwise,
pulling up
+ * correlated predicates could cause incorrect results.
*/
private def rewriteSubQuery(
sub: LogicalPlan,
- outer: Seq[LogicalPlan]): (LogicalPlan, Seq[Expression]) = {
+ outer: Seq[LogicalPlan],
+ isScalarSubq: Boolean): (LogicalPlan, Seq[Expression]) = {
// Pull out the tagged predicates and rewrite the subquery in the
process.
val (basePlan, baseConditions) = pullOutCorrelatedPredicates(sub)
+ // SPARK-18504/SPARK-18814:
+ // Block cases where GROUP BY columns are not part of the correlated
columns
+ // of a scalar subquery.
+ sub collect {
+ case Aggregate(grouping, _, _) if isScalarSubq =>
--- End diff --
Right. Thank you. It seems we need to have this part from `CheckAnalysis`
included to walk the subquery plan:
````scala
153 // Skip projects and subquery aliases added by the Analyzer
and the SQLBuilder.
154 def cleanQuery(p: LogicalPlan): LogicalPlan = p match {
155 case s: SubqueryAlias => cleanQuery(s.child)
156 case p: Project => cleanQuery(p.child)
157 case child => child
158 }
159
160 cleanQuery(query) match {
161 case a: Aggregate => checkAggregate(a)
162 case Filter(_, a: Aggregate) => checkAggregate(a)
163 case fail => failAnalysis(s"Correlated scalar subqueries
must be Aggregated: $fail")
164 }
````
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
