This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a commit to branch branch-3.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.3 by this push:
new 0fdb6757946 [SPARK-38813][3.3][SQL][FOLLOWUP] Improve the analysis
check for TimestampNTZ output
0fdb6757946 is described below
commit 0fdb6757946e2a0991256a3b73c0c09d6e764eed
Author: Gengliang Wang <[email protected]>
AuthorDate: Fri Apr 22 13:35:15 2022 +0800
[SPARK-38813][3.3][SQL][FOLLOWUP] Improve the analysis check for
TimestampNTZ output
### What changes were proposed in this pull request?
In https://github.com/apache/spark/pull/36094, a check for failing
TimestampNTZ output is added.
However, if there is an unresolved attribute in the plan, even if it is
note related to TimestampNTZ, the error message becomes confusing
```
scala> val df = spark.range(2)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]
scala> df.select("i")
org.apache.spark.sql.AnalysisException: Invalid call to dataType on
unresolved object;
'Project ['i]
+- Range (0, 2, step=1, splits=Some(16))
at
org.apache.spark.sql.catalyst.analysis.UnresolvedAttribute.dataType(unresolved.scala:137)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$4(CheckAnalysis.scala:164)
...
```
Before changes it was
```
org.apache.spark.sql.AnalysisException: Column 'i' does not exist. Did you
mean one of the following? [id];
```
This PR is the improve the check for TimestampNTZ and restore the error
message for unresolved attributes.
### Why are the changes needed?
Fix a regression in analysis error message.
### Does this PR introduce _any_ user-facing change?
No, it is not released yet.
### How was this patch tested?
Manual test
Closes #36316 from gengliangwang/bugFix.
Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>
---
.../scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index b714c5f9ceb..ff40272682e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -161,7 +161,8 @@ trait CheckAnalysis extends PredicateHelper with
LookupCatalog {
throw QueryCompilationErrors.commandUnsupportedInV2TableError("SHOW
TABLE EXTENDED")
case operator: LogicalPlan
- if !Utils.isTesting &&
operator.output.exists(_.dataType.isInstanceOf[TimestampNTZType]) =>
+ if !Utils.isTesting && operator.output.exists(attr =>
+ attr.resolved && attr.dataType.isInstanceOf[TimestampNTZType]) =>
operator.failAnalysis("TimestampNTZ type is not supported in Spark
3.3.")
case operator: LogicalPlan =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
