This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.1 by this push:
new aa22199 [SPARK-35665][SQL] Resolve UnresolvedAlias in CollectMetrics
aa22199 is described below
commit aa221991b3f0f1b7c91e87337f9eb5d164167f5a
Author: Wenchen Fan <[email protected]>
AuthorDate: Mon Jun 7 21:05:11 2021 +0900
[SPARK-35665][SQL] Resolve UnresolvedAlias in CollectMetrics
### What changes were proposed in this pull request?
It's a long-standing bug that we forgot to resolve `UnresolvedAlias` in
`CollectMetrics`. It's a bit hard to trigger this bug before 3.2 as most likely
people won't create `UnresolvedAlias` when calling `Dataset.observe`. However
things have been changed after https://github.com/apache/spark/pull/30974
This PR proposes to handle `CollectMetrics` in the rule `ResolveAliases`.
### Why are the changes needed?
bug fix
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
updated test
Closes #32803 from cloud-fan/minor.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit a70e66ecfa638cacc99b4e9a7c464e41ec92ad30)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 3 +++
.../test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala | 3 ++-
2 files changed, 5 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 600a5af..899e723 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -420,6 +420,9 @@ class Analyzer(override val catalogManager: CatalogManager)
case Project(projectList, child) if child.resolved &&
hasUnresolvedAlias(projectList) =>
Project(assignAliases(projectList), child)
+
+ case c: CollectMetrics if c.child.resolved &&
hasUnresolvedAlias(c.metrics) =>
+ c.copy(metrics = assignAliases(c.metrics))
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
index b17c935..e3fcf8b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/util/DataFrameCallbackSuite.scala
@@ -251,7 +251,8 @@ class DataFrameCallbackSuite extends QueryTest
name = "my_event",
min($"id").as("min_val"),
max($"id").as("max_val"),
- sum($"id").as("sum_val"),
+ // Test unresolved alias
+ sum($"id"),
count(when($"id" % 2 === 0, 1)).as("num_even"))
.observe(
name = "other_event",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
