maropu commented on a change in pull request #28490:
URL: https://github.com/apache/spark/pull/28490#discussion_r434366090
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
##########
@@ -506,31 +506,55 @@ class Analyzer(
aggregations: Seq[NamedExpression],
groupByAliases: Seq[Alias],
groupingAttrs: Seq[Expression],
- gid: Attribute): Seq[NamedExpression] = aggregations.map {
- // collect all the found AggregateExpression, so we can check an
expression is part of
- // any AggregateExpression or not.
- val aggsBuffer = ArrayBuffer[Expression]()
- // Returns whether the expression belongs to any expressions in
`aggsBuffer` or not.
- def isPartOfAggregation(e: Expression): Boolean = {
- aggsBuffer.exists(a => a.find(_ eq e).isDefined)
+ gid: Attribute): Seq[NamedExpression] = {
+ val resolvedGroupByAliases = groupByAliases.map(_.transformDown {
Review comment:
Probably, we should not fix this issue in `ResolveGroupingAnalytics`,
but in `ResolveRefences` just like this;
```
object ResolveReferences extends Rule[LogicalPlan] {
...
def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp {
case p: LogicalPlan if !p.childrenResolved => p
....
case a @ Aggregate(groupingExprs, aggExprs, _) if both `aggExprs` and
`aggExprs` have the same struct field =>
val newAgg = resolve expressions so that they have the same exprIds
newAgg
```
A root cause seems to be that `ResolveReferences` assigns different exprIds
to `each#30.json_string AS json_string`s (`#31` vs `#32`);
```
20/06/03 16:22:47 WARN HiveSessionStateBuilder$$anon$1:
=== Applying Rule
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences ===
!'Aggregate [cube('a, 'get_json_object('each.json_string, $.iType))], ['a,
'coalesce('get_json_object('each.json_string, $.iType), -127) AS iType#29,
unresolvedalias('sum('b), None)]
+- Generate explode(c#4), false, x, [each#30]
+- SubqueryAlias t
+- Project [x AS a#2, 1 AS b#3, array(named_struct(row_id, 1,
json_string, y)) AS c#4]
+- Range (0, 1, step=1, splits=Some(4))
'Aggregate [cube(a#2, 'get_json_object(each#30.json_string AS
json_string#31, $.iType))], [a#2,
'coalesce('get_json_object(each#30.json_string AS json_string#32, $.iType),
-127) AS iType#29, unresolvedalias('sum(b#3), None)]
+- Generate explode(c#4), false, x, [each#30]
+- SubqueryAlias t
+- Project [x AS a#2, 1 AS b#3, array(named_struct(row_id, 1,
json_string, y)) AS c#4]
+- Range (0, 1, step=1, splits=Some(4))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
