This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 32274cda3af8 [SPARK-57353][SQL] Fix single-pass resolver crash on
GROUPING SETS with HAVING/ORDER BY
32274cda3af8 is described below
commit 32274cda3af8e1e26eadc13c8d7eeef7d6f63634
Author: Anupam Yadav <[email protected]>
AuthorDate: Tue Aug 11 15:41:33 2026 +0800
[SPARK-57353][SQL] Fix single-pass resolver crash on GROUPING SETS with
HAVING/ORDER BY
### What changes were proposed in this pull request?
Fix a crash in the single-pass resolver (Analyzer++) when GROUP BY
CUBE/ROLLUP/GROUPING SETS is used with HAVING, ORDER BY, or lateral column
aliases (LCA) containing aggregate functions.
The fix wires `GroupingAnalyticsResolver` into `AggregateResolver` to
expand `BaseGroupingSets` into an `Expand` operator before
`AggregationValidator` runs, and guards `ExprUtils.checkValidGroupingExprs`
against calling `.dataType` on unresolved grouping expressions. Post-expansion
validation is applied in both the LCA and non-LCA paths.
HAVING/ORDER BY/LCA combined with grouping analytics are not yet fully
supported in the single-pass resolver, so this PR converts the crash into a
clean unsupported-feature signal (`ExplicitlyUnsupportedResolverFeature`)
scoped to the direct aggregate lineage. Grouping analytics confined to a scalar
subquery or a derived table (SubqueryAlias, including CTEs and views) do not
affect unrelated outer Sort/Having in the enclosing query. Full support inside
the single-pass resolver is trac [...]
### Why are the changes needed?
With `spark.sql.analyzer.singlePassResolver.enabled=true`, queries like:
```sql
SELECT a, b, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
GROUP BY CUBE(a, b) ORDER BY SUM(b);
```
crash with:
```
org.apache.spark.SparkUnsupportedOperationException:
[UNSUPPORTED_CALL.WITHOUT_SUGGESTION]
Cannot call the method "dataType$" of the class
"org.apache.spark.sql.catalyst.expressions.BaseGroupingSets".
```
The single-pass resolver called `assertValidAggregation` while
`BaseGroupingSets` nodes were still present in the grouping expressions (the
legacy analyzer expands them via `ResolveGroupingAnalytics` before validation).
`BaseGroupingSets.dataType` throws by design because these nodes must be
expanded first.
### Does this PR introduce _any_ user-facing change?
Yes. GROUPING SETS/CUBE/ROLLUP queries with HAVING, ORDER BY, or lateral
column aliases no longer crash under the single-pass resolver.
- In tentative mode (`spark.sql.analyzer.singlePassResolver.enabled=true`
with fallback), the resolver signals these combinations as unsupported and
falls back to the fixed-point analyzer, returning correct results.
- In pure single-pass mode, they surface as an
`ExplicitlyUnsupportedResolverFeature` analysis error naming the specific
unsupported combination, rather than as an internal crash.
- Queries whose grouping analytics live entirely inside a scalar subquery
or a derived table are unaffected in the outer scope, so an outer ORDER BY or
HAVING that does not itself use grouping analytics is not rejected.
### How was this patch tested?
Added tests in `SQLQuerySuite` and
`ExplicitlyUnsupportedResolverFeatureSuite` covering:
- CUBE/ROLLUP/GROUPING SETS with ORDER BY and HAVING (tentative mode
returns results, matches the legacy analyzer baseline)
- HAVING/ORDER BY/LCA + grouping analytics rejected with
`ExplicitlyUnsupportedResolverFeature` in pure single-pass mode
- Grouping analytics confined to a scalar subquery or a derived table:
outer Sort and HAVING are not rejected (query-block-boundary regression)
- NULL grouping columns, empty results, multiple aggregates
- Negative test: column not in GROUP BY still errors with
`MISSING_AGGREGATION`
Note: a separate wrong-results issue (SPARK-57346) exists for full
single-pass support of these combinations and is out of scope for this crash
fix.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #56417 from yadavay-amzn/fix/SPARK-57353-grouping-sets-crash.
Authored-by: Anupam Yadav <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit e0a820234c95346d71d47d3f622ec2a83496db26)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../analysis/resolver/AggregateResolver.scala | 28 +++-
.../analysis/resolver/HavingResolver.scala | 8 ++
.../resolver/LateralColumnAliasResolver.scala | 3 +
.../resolver/OperatorResolutionContext.scala | 17 ++-
.../catalyst/analysis/resolver/SortResolver.scala | 8 ++
.../scala/org/apache/spark/sql/SQLQuerySuite.scala | 157 +++++++++++++++++++++
...ExplicitlyUnsupportedResolverFeatureSuite.scala | 71 ++++++++++
7 files changed, 282 insertions(+), 10 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala
index cd51f17c9766..da9083f99fd2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/AggregateResolver.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.expressions.{
Alias,
AliasHelper,
AttributeReference,
+ BaseGroupingSets,
Expression,
ExprId,
IntegerLiteral,
@@ -51,6 +52,8 @@ class AggregateResolver(
private val operatorResolutionContextStack =
operatorResolver.getOperatorResolutionContextStack
private val lcaResolver = expressionResolver.getLcaResolver
private val ordinalResolver = expressionResolver.getOrdinalResolver
+ private val groupingAnalyticsResolver =
+ new GroupingAnalyticsResolver(operatorResolver, expressionResolver)
/**
* Resolve [[Aggregate]] operator.
@@ -144,25 +147,36 @@ class AggregateResolver(
)
if (resolvedAggregateExpressions.hasLateralColumnAlias) {
+ // LCA + grouping analytics (CUBE/ROLLUP/GROUPING SETS) is not yet
supported in the
+ // single-pass resolver because Expand mints new attribute IDs that
get out of sync
+ // with the Project operator. Fall back to legacy analyzer for correct
results.
+ if
(finalAggregate.groupingExpressions.exists(_.isInstanceOf[BaseGroupingSets])) {
+ throw new ExplicitlyUnsupportedResolverFeature(
+ "lateral column alias with grouping analytics")
+ }
val aggregateWithLcaResolutionResult =
lcaResolver.handleLcaInAggregate(finalAggregate)
+ val lcaBaseAggregate = aggregateWithLcaResolutionResult.baseAggregate
AggregateResolutionResult(
operator = aggregateWithLcaResolutionResult.resolvedOperator,
outputList = aggregateWithLcaResolutionResult.outputList,
groupingAttributeIds =
-
getGroupingAttributeIds(aggregateWithLcaResolutionResult.baseAggregate),
+ getGroupingAttributeIds(lcaBaseAggregate),
aggregateListAliases =
aggregateWithLcaResolutionResult.aggregateListAliases,
- baseAggregate = aggregateWithLcaResolutionResult.baseAggregate
+ baseAggregate = lcaBaseAggregate
)
} else {
- AggregationValidator(finalAggregate)
+ // Grouping analytics (CUBE/ROLLUP/GROUPING SETS) are expanded
unconditionally.
+ // groupingAnalyticsResolver.resolve no-ops when no BaseGroupingSets
are present.
+ val expandedAggregate =
groupingAnalyticsResolver.resolve(finalAggregate)
+ AggregationValidator(expandedAggregate)
AggregateResolutionResult(
- operator = finalAggregate,
- outputList = finalAggregate.aggregateExpressions,
- groupingAttributeIds = getGroupingAttributeIds(finalAggregate),
+ operator = expandedAggregate,
+ outputList = expandedAggregate.aggregateExpressions,
+ groupingAttributeIds = getGroupingAttributeIds(expandedAggregate),
aggregateListAliases =
scopes.current.getTopAggregateExpressionAliases,
- baseAggregate = finalAggregate
+ baseAggregate = expandedAggregate
)
}
} finally {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HavingResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HavingResolver.scala
index 93c0eabb9487..ddf930ac7b1e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HavingResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/HavingResolver.scala
@@ -49,6 +49,14 @@ class HavingResolver(resolver: Resolver, expressionResolver:
ExpressionResolver)
override def resolve(unresolvedHaving: UnresolvedHaving): LogicalPlan = {
val resolvedChild = resolver.resolve(unresolvedHaving.child)
+ // HAVING over grouping analytics (CUBE/ROLLUP/GROUPING SETS) is not yet
supported in
+ // single-pass because the expanded Aggregate's ExprIds get out of sync
when HAVING
+ // inserts missing expressions. Fall back to legacy for correct results
(SPARK-57346).
+ if (operatorResolutionContextStack.current.hasGroupingAnalytics) {
+ throw new ExplicitlyUnsupportedResolverFeature(
+ "HAVING with grouping analytics (SPARK-57346)")
+ }
+
resolvedChild match {
case window: Window if scopes.current.baseAggregate.isDefined =>
resolveHavingAboveWindow(
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/LateralColumnAliasResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/LateralColumnAliasResolver.scala
index 3d19c4cbde8c..6859c98f1f99 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/LateralColumnAliasResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/LateralColumnAliasResolver.scala
@@ -70,6 +70,9 @@ class LateralColumnAliasResolver(expressionResolver:
ExpressionResolver, operato
case _ @Project(projectList: Seq[_], aggregate: Aggregate) =>
operatorResolutionContextStack.current.baseOperator = Some(aggregate)
+ // Note: LCA + grouping analytics (CUBE/ROLLUP/GROUPING SETS) is
intercepted in
+ // AggregateResolver which throws ExplicitlyUnsupportedResolverFeature
before reaching
+ // here. This validation is retained as defense-in-depth in case the
call path changes.
AggregationValidator(aggregate)
val remappedAliases = new HashMap[ExprId, Alias](projectList.size)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/OperatorResolutionContext.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/OperatorResolutionContext.scala
index 5df31a9c09fd..880fce67ba67 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/OperatorResolutionContext.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/OperatorResolutionContext.scala
@@ -51,7 +51,8 @@ class OperatorResolutionContext(
var baseOperator: Option[LogicalPlan] = None,
val isResolvingTreeUnderHaving: Boolean = false,
var parameterNamesToValues: Option[LinkedHashMap[String, Expression]] =
None,
- var hasGroupingAnalytics: Boolean = false) {
+ var hasGroupingAnalytics: Boolean = false,
+ val isSubqueryRoot: Boolean = false) {
/**
* @param subqueryExpressionsToValidate List of [[SubqueryExpression]]s in
the context of current
@@ -130,7 +131,8 @@ class OperatorResolutionContextStack {
new OperatorResolutionContext(
unresolvedPlan = Some(unresolvedPlan),
isResolvingTreeUnderHaving = isResolvingTreeUnderHaving,
- parameterNamesToValues = current.parameterNamesToValues
+ parameterNamesToValues = current.parameterNamesToValues,
+ isSubqueryRoot = isSubqueryRoot
)
)
}
@@ -138,11 +140,20 @@ class OperatorResolutionContextStack {
/**
* Pops the top resolution context from the stack. Before popping,
propagates the
* `hasGroupingAnalytics` from the child context to the parent context if it
was set.
+ * The flag is NOT propagated across subquery boundaries (isSubqueryRoot) or
derived-table
+ * boundaries (SubqueryAlias), because grouping analytics inside a subquery
or derived table
+ * are unrelated to operators in the outer query.
*/
def pop(): Unit = {
val childContext = current
stack.pop()
- if (childContext.hasGroupingAnalytics) {
+ val isDerivedTableBoundary = childContext.unresolvedPlan match {
+ case Some(_: SubqueryAlias) => true
+ case _ => false
+ }
+ if (childContext.hasGroupingAnalytics &&
+ !childContext.isSubqueryRoot &&
+ !isDerivedTableBoundary) {
current.hasGroupingAnalytics = childContext.hasGroupingAnalytics
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SortResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SortResolver.scala
index ed0edf90969f..1219d16e4c83 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SortResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SortResolver.scala
@@ -135,6 +135,14 @@ class SortResolver(operatorResolver: Resolver,
expressionResolver: ExpressionRes
val resolvedChild = operatorResolver.resolve(unresolvedSort.child)
+ // ORDER BY over grouping analytics (CUBE/ROLLUP/GROUPING SETS) is not yet
supported in
+ // single-pass because the expanded Aggregate's ExprIds get out of sync
when ORDER BY
+ // inserts missing expressions. Fall back to legacy for correct results
(SPARK-57346).
+ if (operatorResolutionContextStack.current.hasGroupingAnalytics) {
+ throw new ExplicitlyUnsupportedResolverFeature(
+ "ORDER BY with grouping analytics (SPARK-57346)")
+ }
+
operatorResolutionContextStack.current.ordinalReplacementExpressions =
Some(
OrdinalReplacementSortOrderExpressions(
expressions = scopes.current.output.toIndexedSeq,
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 03d3a122da16..8075a8a0776b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -5158,6 +5158,163 @@ class SQLQuerySuite extends SharedSparkSession with
AdaptiveSparkPlanHelper
sql("SELECT col1 - rand() FROM VALUES(1) GROUP BY ALL")
}
}
+
+ test("SPARK-57353: CUBE with ORDER BY - single pass resolver (tentative
fallback)") {
+ // ORDER BY + grouping analytics throws
ExplicitlyUnsupportedResolverFeature in pure
+ // single-pass (SPARK-57346). In tentative mode, the HybridAnalyzer falls
back to legacy.
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) as s FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a) ORDER BY s""".stripMargin),
+ Row(1, 30) :: Row(2, 30) :: Row(null, 60) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: ROLLUP with HAVING - single pass resolver (tentative
fallback)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a) HAVING SUM(b) > 30""".stripMargin),
+ Row(null, 60) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: GROUPING SETS with ORDER BY - single pass resolver
(tentative fallback)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, b, SUM(b) as s FROM VALUES (1,10),(1,20),(2,30) AS
t(a,b)
+ |GROUP BY a, b GROUPING SETS ((a, b), (a)) ORDER BY
s""".stripMargin),
+ Row(1, 10, 10) :: Row(1, 20, 20) :: Row(1, null, 30) ::
+ Row(2, 30, 30) :: Row(2, null, 30) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: CUBE with NULL grouping columns - single pass resolver
(tentative fallback)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) as s FROM VALUES (1,10),(null,20),(2,30) AS
t(a,b)
+ |GROUP BY CUBE(a) ORDER BY s""".stripMargin),
+ Row(1, 10) :: Row(null, 20) :: Row(2, 30) :: Row(null, 60) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: ROLLUP with HAVING filtering all rows - single pass
resolver" +
+ " (tentative fallback)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a) HAVING SUM(b) > 100""".stripMargin),
+ Nil)
+ }
+ }
+
+ test("SPARK-57353: CUBE with multiple aggregates in ORDER BY - single pass
resolver" +
+ " (tentative fallback)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) as s, COUNT(b) as c
+ |FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a) ORDER BY COUNT(b), SUM(b)""".stripMargin),
+ Row(2, 30, 1) :: Row(1, 30, 2) :: Row(null, 60, 3) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: multi-column ROLLUP with HAVING - single pass resolver
(tentative fallback" +
+ ", SPARK-57346)") {
+ // SPARK-57346: multi-column ROLLUP with HAVING previously produced wrong
results (1 row
+ // instead of 4) in pure single-pass. Now throws
ExplicitlyUnsupportedResolverFeature so
+ // tentative mode falls back to legacy for correct results.
+ val query =
+ """SELECT a, b, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a, b) HAVING SUM(b) > 25""".stripMargin
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ val legacyResult = withSQLConf(
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key ->
"false") {
+ sql(query).collect().toSeq
+ }
+ checkAnswer(sql(query), legacyResult)
+ }
+ }
+
+ test("SPARK-57353: missing-aggregation still detected after GROUPING SETS
expansion") {
+ // Confirms aggregate-expression validation still fires: column 'b' is not
in GROUP BY
+ // ROLLUP(a) and is not aggregated, so MISSING_AGGREGATION should be
raised.
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ val ex = intercept[AnalysisException] {
+ sql(
+ """SELECT a, b, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a) HAVING SUM(b) > 10""".stripMargin).collect()
+ }
+ assert(ex.getCondition === "MISSING_AGGREGATION")
+ }
+ }
+
+ test("SPARK-57353: LCA with GROUPING SETS - single pass resolver") {
+ // LCA + grouping analytics throws ExplicitlyUnsupportedResolverFeature in
single-pass mode.
+ // In tentative mode, the HybridAnalyzer catches it and falls back to
legacy for correct
+ // results.
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false",
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED_TENTATIVELY.key -> "true")
{
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) as total, total + 1
+ |FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a) ORDER BY total""".stripMargin),
+ Row(1, 30, 31) :: Row(2, 30, 31) :: Row(null, 60, 61) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: legacy analyzer handles CUBE/ROLLUP/GROUPING SETS
correctly") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "false") {
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) as s FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a) ORDER BY s""".stripMargin),
+ Row(1, 30) :: Row(2, 30) :: Row(null, 60) :: Nil)
+ checkAnswer(
+ sql(
+ """SELECT a, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a) HAVING SUM(b) > 30""".stripMargin),
+ Row(null, 60) :: Nil)
+ }
+ }
+
+ test("SPARK-57353: nested aggregate with GROUPING SETS is rejected") {
+ // Confirms aggregate-expression validation (nested agg check) still fires
with grouping sets.
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ val ex = intercept[AnalysisException] {
+ sql(
+ """SELECT a, SUM(COUNT(b)) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a)""".stripMargin).collect()
+ }
+ assert(ex.getCondition === "NESTED_AGGREGATE_FUNCTION")
+ }
+ }
+
+ test("SPARK-57353: GROUPING SETS with empty grouping list - all rows") {
+ // GROUPING SETS (()) produces a single grand-total row.
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ QueryTest.checkAnswer(
+ sql(
+ """SELECT SUM(b) as s FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY a GROUPING SETS (())""".stripMargin),
+ Row(60) :: Nil,
+ checkToRDD = false)
+ }
+ }
}
case class Foo(bar: Option[String])
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ExplicitlyUnsupportedResolverFeatureSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ExplicitlyUnsupportedResolverFeatureSuite.scala
index 3cf2613cf21d..a8c1041509cc 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ExplicitlyUnsupportedResolverFeatureSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/analysis/resolver/ExplicitlyUnsupportedResolverFeatureSuite.scala
@@ -45,6 +45,34 @@ class ExplicitlyUnsupportedResolverFeatureSuite extends
SharedSparkSession {
}
}
+ test("SPARK-57353: HAVING with grouping analytics is unsupported
(SPARK-57346)") {
+ checkResolution(
+ """SELECT a, SUM(b) FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY ROLLUP(a) HAVING SUM(b) > 30""".stripMargin,
+ shouldPass = false,
+ expectedMessage = Some("HAVING with grouping analytics (SPARK-57346)")
+ )
+ }
+
+ test("SPARK-57353: ORDER BY with grouping analytics is unsupported
(SPARK-57346)") {
+ checkResolution(
+ """SELECT a, SUM(b) as s FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a) ORDER BY s""".stripMargin,
+ shouldPass = false,
+ expectedMessage = Some("ORDER BY with grouping analytics (SPARK-57346)")
+ )
+ }
+
+ test("SPARK-57353: LCA with grouping analytics is unsupported") {
+ checkResolution(
+ """SELECT a, SUM(b) as total, total + 1
+ |FROM VALUES (1,10),(1,20),(2,30) AS t(a,b)
+ |GROUP BY CUBE(a)""".stripMargin,
+ shouldPass = false,
+ expectedMessage = Some("lateral column alias with grouping analytics")
+ )
+ }
+
test("ASOF JOIN MATCH_CONDITION operands requiring element-wise ordering") {
withSQLConf(SQLConf.SQL_ASOF_JOIN_ENABLED.key -> "true") {
checkResolution(
@@ -66,6 +94,49 @@ class ExplicitlyUnsupportedResolverFeatureSuite extends
SharedSparkSession {
}
}
+ test("SPARK-57353: ORDER BY with grouping analytics in subquery does not
reject outer sort") {
+ // Grouping analytics inside a scalar subquery must not leak
hasGroupingAnalytics to the
+ // outer query. The outer ORDER BY is unrelated and should resolve
successfully.
+ checkResolution(
+ """SELECT (SELECT SUM(x) FROM VALUES (1),(2) t(x)
+ | GROUP BY GROUPING SETS (())) AS s ORDER BY s""".stripMargin,
+ shouldPass = true
+ )
+ }
+
+ test("SPARK-57353: HAVING with grouping analytics in subquery does not
reject outer having") {
+ // Same boundary isolation for HAVING: grouping analytics confined to a
subquery must not
+ // cause the outer HAVING to be rejected.
+ checkResolution(
+ """SELECT a, (SELECT SUM(x) FROM VALUES (1),(2) t(x)
+ | GROUP BY GROUPING SETS (())) AS s
+ |FROM VALUES (1),(2),(3) v(a)
+ |GROUP BY a
+ |HAVING a > 1""".stripMargin,
+ shouldPass = true
+ )
+ }
+
+ test("SPARK-57353: HAVING with grouping analytics in derived table does not
reject outer") {
+ // Grouping analytics inside a derived table (SubqueryAlias) must not leak
+ // hasGroupingAnalytics to the outer query. The outer HAVING is unrelated.
+ checkResolution(
+ """SELECT a FROM (SELECT a FROM VALUES (1) t(a) GROUP BY CUBE(a)) s
+ |GROUP BY a HAVING a > 0""".stripMargin,
+ shouldPass = true
+ )
+ }
+
+ test("SPARK-57353: ORDER BY with grouping analytics in derived table does
not reject outer") {
+ // Same boundary isolation for ORDER BY: grouping analytics confined to a
derived table
+ // must not cause the outer ORDER BY to be rejected.
+ checkResolution(
+ """SELECT a FROM (SELECT a FROM VALUES (1) t(a) GROUP BY CUBE(a)) s
+ |ORDER BY a""".stripMargin,
+ shouldPass = true
+ )
+ }
+
private def checkResolution(
sqlText: String,
shouldPass: Boolean = false,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
