maropu commented on a change in pull request #27233: [SPARK-29701][SQL] Correct behaviours of group analytical queries when empty input given URL: https://github.com/apache/spark/pull/27233#discussion_r370966784
########## File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteGroupingAnalyticsWithoutKeys.scala ########## @@ -0,0 +1,142 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.catalyst.optimizer + +import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute, AttributeReference, AttributeSet, Literal, NamedExpression, VirtualColumn} +import org.apache.spark.sql.catalyst.expressions.aggregate.AggregateExpression +import org.apache.spark.sql.catalyst.plans.logical._ +import org.apache.spark.sql.catalyst.rules.Rule + +/** + * In grouping analytics (e.g., `GROUPING SETS`) with an empty grouping set, we need to follow + * the non-key aggregate semantics; it generates a single row having an initial aggregated value + * (e.g., 0 for `COUNT`) from an empty input. + * For example, a query below must return a row (NULL, NULL, 0) from + * an empty input (`empty_input`): + * + * {{{ + * SELECT k1, k2, COUNT(v) + * FROM empty_input t(k1, k2, v) + * GROUP BY GROUPING SETS ((), (k1, k2)) + * }}} + * + * To comply with the semantics, this rule rewrites the following (pseudo) resolved logical plan + * as an union query of aggregates with/without keys as follows: + * + * {{{ + * Aggregate( + * groupingExprs = ['k1, 'k2, 'spark_grouping_id], + * aggregateExprs = ['k1, 'k2, COUNT('v)], + * output = ['k1, 'k2, 'cnt], + * child = Expand( + * projections = [('k1, 'k2, 'v, 0), (null, null, 'v, 3)], + * output = ['k1, 'k2, 'v, 'spark_grouping_id]), + * child = LocalTableScan [...])) + * }}} + * + * It is transformed into an union plan below: + * + * {{{ + * Union([ + * Aggregate( + * groupingExprs = ['k1, 'k2, 'spark_grouping_id], + * aggregateExprs = ['k1, 'k2, COUNT('v)], + * output = ['k1, 'k2, 'cnt], + * child = Expand( + * projections = [('k1, 'k2, 'v, 0)], + * output = ['k1, 'k2, 'v, 'spark_grouping_id]), + * child = LocalTableScan [...])), + * Aggregate( + * groupingExprs = [], + * aggregateExprs = [null, null, COUNT('v)], + * output = ['k1, 'k2, 'cnt], + * child = Project( + * projectList = ['v], + * child = LocalTableScan [...]))] Review comment: @viirya Yea, currently yes. > And, that could be a reason to drop this PR and JIRA completely. @dongjoon-hyun Do you think it is ok to keep the current behaviour? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
