This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new e107b9e22836 [SPARK-60880][SQL] Extract shared CREATE TABLE analysis
helpers
e107b9e22836 is described below
commit e107b9e2283608ca6f9b01bc5e84c77c5d9b1d04
Author: Vladan Vasić <[email protected]>
AuthorDate: Thu Nov 5 14:01:38 2026 +0000
[SPARK-60880][SQL] Extract shared CREATE TABLE analysis helpers
### What changes were proposed in this pull request?
Extract shared operations for CREATE TABLE and REPLACE TABLE analysis:
- `CreateTableHelper` owns V1 lowering and storage construction,
default-value folding, table preprocessing, and CHAR/VARCHAR replacement.
Existing rules delegate to it.
- `CreateTableValidation` owns the current V2 table checks in Catalyst.
- `CatalogIdentifierResolution.columnsToOutput` converts column definitions
into output attributes, and `ResolveTableSpec.resolveTableSpec` resolves an
individual table specification.
The implementation retains current catalog/provider routing, write
distribution and ordering, constraint validation, generated-column checks, and
collation behavior. V2 validation continues to use `StoredExpressionFunctions`
to reject user-defined functions in generated columns.
### Why are the changes needed?
Table operations are currently embedded in tree-traversing rules.
Separating per-table operations from traversal allows analyzer paths to reuse
the same lowering and validation logic without running an entire rule over a
plan.
Jira: https://issues.apache.org/jira/browse/SPARK-60880
### Does this PR introduce _any_ user-facing change?
No. This is a behavior-preserving refactoring.
### How was this patch tested?
Focused tests cover column metadata and CHAR/VARCHAR modes, table-spec
option evaluation and root subqueries, and preservation of table metadata
during normalization.
Validation after syncing with current master (head
`9e5794241d18d46d421903b97b248e788986ae9e`):
- `build/sbt 'catalyst/testOnly
org.apache.spark.sql.catalyst.analysis.CatalogSuite
org.apache.spark.sql.catalyst.analysis.AnalysisSuite'`: 97 passed.
- Catalyst and SQL main/test scalastyle: zero errors or warnings.
Whitespace checks passed.
- Compared 22 extracted method and case bodies with master, including
generated-column function validation; the logic is unchanged apart from the
explicit configuration parameter and local variable renames.
- Full CI: https://github.com/vladanvasi-db/spark/actions/runs/42017541949:
passed, with 28 successful jobs and three conditional skips.
The local SQL rerun was blocked before compilation by a dependency
download. Dependency versions match master; SQL validation completed
successfully in CI.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex CLI 0.166.0 (GPT-6)
Closes #60099 from
vladanvasi-db/vladanvasi-db/create-table-analysis-helpers.
Authored-by: Vladan Vasić <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 4b7c3d50331ac31ad65c242629ec62cfee019fe6)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../analysis/CatalogIdentifierResolution.scala | 19 +-
.../sql/catalyst/analysis/CheckAnalysis.scala | 62 +--
.../catalyst/analysis/CreateTableValidation.scala | 92 ++++
.../sql/catalyst/analysis/ResolveCatalogs.scala | 11 +-
.../sql/catalyst/analysis/ResolveTableSpec.scala | 14 +-
.../sql/catalyst/analysis/AnalysisSuite.scala | 56 +++
.../spark/sql/catalyst/analysis/CatalogSuite.scala | 31 +-
.../sql/catalyst/analysis/CreateTableHelper.scala | 488 +++++++++++++++++++++
.../catalyst/analysis/ReplaceCharWithVarchar.scala | 14 +-
.../catalyst/analysis/ResolveSessionCatalog.scala | 225 +---------
.../execution/datasources/DataSourceStrategy.scala | 17 +-
.../spark/sql/execution/datasources/rules.scala | 196 +--------
.../execution/command/PlanResolutionSuite.scala | 29 +-
13 files changed, 747 insertions(+), 507 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CatalogIdentifierResolution.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CatalogIdentifierResolution.scala
index f36db09eea66..fa237ea27563 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CatalogIdentifierResolution.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CatalogIdentifierResolution.scala
@@ -17,15 +17,18 @@
package org.apache.spark.sql.catalyst.analysis
-import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.SQLConfHelper
+import org.apache.spark.sql.catalyst.expressions.{Attribute,
AttributeReference}
+import org.apache.spark.sql.catalyst.plans.logical.ColumnDefinition
import org.apache.spark.sql.catalyst.trees.CurrentOrigin
+import
org.apache.spark.sql.catalyst.util.SparkCharVarcharUtils.replaceCharVarcharWithString
import org.apache.spark.sql.connector.catalog.{CatalogManager, Identifier,
LookupCatalog}
/**
* Resolves command target names to a [[ResolvedIdentifier]] for
[[ResolveCatalogs]].
*/
class CatalogIdentifierResolution(override val catalogManager: CatalogManager)
- extends LookupCatalog {
+ extends LookupCatalog with SQLConfHelper {
/** Resolves a command target without column output, preserving its origin
and tags. */
def resolveUnresolvedIdentifier(
@@ -53,4 +56,16 @@ class CatalogIdentifierResolution(override val
catalogManager: CatalogManager)
ResolvedIdentifier(catalog, identifier, columnOutput)
}
}
+
+ /** Converts column definitions to output attributes, honoring CHAR/VARCHAR
configuration. */
+ def columnsToOutput(columns: Seq[ColumnDefinition]): Seq[AttributeReference]
= {
+ columns.map { col =>
+ val dataType = if (conf.charVarcharFirstClassTypes) {
+ col.dataType
+ } else {
+ replaceCharVarcharWithString(col.dataType)
+ }
+ AttributeReference(col.name, dataType, col.nullable, col.metadata)()
+ }
+ }
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index 5efe85190edb..e7373ba735a0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -28,7 +28,7 @@ import
org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression,
import org.apache.spark.sql.catalyst.optimizer.InlineCTE
import org.apache.spark.sql.catalyst.plans.logical._
import
org.apache.spark.sql.catalyst.trees.TreePattern.{LATERAL_COLUMN_ALIAS_REFERENCE,
PLAN_EXPRESSION, UNRESOLVED_WINDOW_EXPRESSION}
-import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, StringUtils,
TypeUtils, WriteDistributionAndOrdering}
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, StringUtils,
TypeUtils}
import org.apache.spark.sql.connector.catalog.{CatalogManager, LookupCatalog,
SupportsPartitionManagement}
import org.apache.spark.sql.errors.{QueryCompilationErrors, QueryErrorsBase}
import org.apache.spark.sql.internal.SQLConf
@@ -987,65 +987,7 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
}
case create: V2CreateTablePlan =>
- val references = create.partitioning.flatMap(_.references).toSet
- val badReferences = references.map(_.fieldNames).flatMap { column
=>
- create.tableSchema.findNestedField(column.toImmutableArraySeq)
match {
- case Some(_) =>
- None
- case _ =>
- Some(column.quoted)
- }
- }
-
- if (badReferences.nonEmpty) {
- create.failAnalysis(
- errorClass =
"UNSUPPORTED_FEATURE.PARTITION_WITH_NESTED_COLUMN_IS_UNSUPPORTED",
- messageParameters = Map(
- "cols" -> badReferences.map(r => toSQLId(r)).mkString(", ")))
- }
-
- // PreprocessTableCreation keeps a reference to a missing column
as is, so this is the
- // only check that rejects it, also for analyzers that do not run
that rule.
- val badOrderingReferences =
WriteDistributionAndOrdering.unknownReferences(
- create.tableSchema, create.writeOrdering)
-
- if (badOrderingReferences.nonEmpty) {
- create.failAnalysis(
- errorClass = "WRITE_ORDERING_WITH_UNKNOWN_COLUMN",
- messageParameters = Map(
- "cols" -> badOrderingReferences.map(parts =>
toSQLId(parts)).mkString(", ")))
- }
-
- create.tableSchema.foreach(f =>
TypeUtils.failWithIntervalType(f.dataType))
- TypeUtils.failUnsupportedDataType(create.tableSchema, SQLConf.get)
- SchemaUtils.checkIndeterminateCollationInSchema(create.tableSchema)
-
- // Validate generated column expressions
- create.columns.foreach { col =>
- col.generationExpression.foreach { genExpr =>
- def unsupportedExpressionError(reason: String):
AnalysisException = {
- new AnalysisException(
- errorClass = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
- messageParameters = Map(
- "fieldName" -> col.name,
- "expressionStr" -> genExpr.originalSQL,
- "reason" -> reason
- )
- )
- }
- // Only built-in functions are allowed in generated columns.
Functions that are
- // not built-in already fail earlier in analysis: unknown
functions with
- // UNRESOLVED_ROUTINE and persistent SQL functions with
UNSUPPORTED_SQL_UDF_USAGE.
- // Here we only need to reject user-defined functions
(Scala/Python/Java/Hive UDFs
- // and v2 catalog functions), which resolve successfully.
-
StoredExpressionFunctions.findUserDefinedFunctionCall(genExpr.child).foreach {
- case (_, functionName) =>
- throw unsupportedExpressionError(
- s"failed to resolve $functionName to a built-in
function")
- }
- genExpr.validate(col.name, col.dataType, create.columns)
- }
- }
+ CreateTableValidation.validateV2CreateTablePlan(create,
SQLConf.get)
case write: V2WriteCommand if write.resolved =>
write.query.schema.foreach(f =>
TypeUtils.failWithIntervalType(f.dataType))
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableValidation.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableValidation.scala
new file mode 100644
index 000000000000..4fd52da14f2b
--- /dev/null
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableValidation.scala
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.V2CreateTablePlan
+import org.apache.spark.sql.catalyst.util.{TypeUtils,
WriteDistributionAndOrdering}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.DataTypeErrors.toSQLId
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.util.SchemaUtils
+import org.apache.spark.util.ArrayImplicits._
+
+/** Validates partition references, column types, write ordering and generated
columns. */
+private[sql] object CreateTableValidation {
+ def validateV2CreateTablePlan(create: V2CreateTablePlan, conf: SQLConf):
Unit = {
+ val references = create.partitioning.flatMap(_.references).toSet
+ val badReferences = references.map(_.fieldNames).flatMap { column =>
+ create.tableSchema.findNestedField(column.toImmutableArraySeq) match {
+ case Some(_) =>
+ None
+ case _ =>
+ Some(column.quoted)
+ }
+ }
+
+ if (badReferences.nonEmpty) {
+ create.failAnalysis(
+ errorClass =
"UNSUPPORTED_FEATURE.PARTITION_WITH_NESTED_COLUMN_IS_UNSUPPORTED",
+ messageParameters = Map(
+ "cols" -> badReferences.map(r => toSQLId(r)).mkString(", ")))
+ }
+
+ // PreprocessTableCreation keeps a reference to a missing column as is, so
this is the
+ // only check that rejects it, also for analyzers that do not run that
rule.
+ val badOrderingReferences = WriteDistributionAndOrdering.unknownReferences(
+ create.tableSchema, create.writeOrdering)
+
+ if (badOrderingReferences.nonEmpty) {
+ create.failAnalysis(
+ errorClass = "WRITE_ORDERING_WITH_UNKNOWN_COLUMN",
+ messageParameters = Map(
+ "cols" -> badOrderingReferences.map(parts =>
toSQLId(parts)).mkString(", ")))
+ }
+
+ create.tableSchema.foreach(f => TypeUtils.failWithIntervalType(f.dataType))
+ TypeUtils.failUnsupportedDataType(create.tableSchema, conf)
+ SchemaUtils.checkIndeterminateCollationInSchema(create.tableSchema)
+
+ // Validate generated column expressions
+ create.columns.foreach { col =>
+ col.generationExpression.foreach { genExpr =>
+ def unsupportedExpressionError(reason: String): AnalysisException = {
+ new AnalysisException(
+ errorClass = "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN",
+ messageParameters = Map(
+ "fieldName" -> col.name,
+ "expressionStr" -> genExpr.originalSQL,
+ "reason" -> reason
+ )
+ )
+ }
+ // Only built-in functions are allowed in generated columns. Functions
that are
+ // not built-in already fail earlier in analysis: unknown functions
with
+ // UNRESOLVED_ROUTINE and persistent SQL functions with
UNSUPPORTED_SQL_UDF_USAGE.
+ // Here we only need to reject user-defined functions
(Scala/Python/Java/Hive UDFs
+ // and v2 catalog functions), which resolve successfully.
+
StoredExpressionFunctions.findUserDefinedFunctionCall(genExpr.child).foreach {
+ case (_, functionName) =>
+ throw unsupportedExpressionError(
+ s"failed to resolve $functionName to a built-in function")
+ }
+ genExpr.validate(col.name, col.dataType, create.columns)
+ }
+ }
+ }
+}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala
index b8d4cf42ca31..225fa8e7df4b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveCatalogs.scala
@@ -22,11 +22,9 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.SparkException
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.{FunctionIdentifier,
SqlScriptingContextManager}
-import org.apache.spark.sql.catalyst.expressions.AttributeReference
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.trees.{CurrentOrigin, Origin}
-import
org.apache.spark.sql.catalyst.util.SparkCharVarcharUtils.replaceCharVarcharWithString
import org.apache.spark.sql.connector.catalog._
import org.apache.spark.sql.errors.DataTypeErrors.toSQLId
import org.apache.spark.sql.errors.QueryCompilationErrors
@@ -159,14 +157,7 @@ class ResolveCatalogs(val catalogManager: CatalogManager)
nameParts: Seq[String],
allowTemp: Boolean,
columns: Seq[ColumnDefinition]): ResolvedIdentifier = {
- val columnOutput = columns.map { col =>
- val dataType = if (conf.charVarcharFirstClassTypes) {
- col.dataType
- } else {
- replaceCharVarcharWithString(col.dataType)
- }
- AttributeReference(col.name, dataType, col.nullable, col.metadata)()
- }
+ val columnOutput = catalogIdentifierResolution.columnsToOutput(columns)
catalogIdentifierResolution.resolveIdentifier(nameParts, allowTemp,
columnOutput)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index 7f6b1d34e23c..6684a3ac2e2f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -61,7 +61,13 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
private def resolveTableSpec(
input: LogicalPlan,
tableSpec: TableSpecBase,
- withNewSpec: TableSpecBase => LogicalPlan): LogicalPlan = tableSpec
match {
+ withNewSpec: TableSpecBase => LogicalPlan): LogicalPlan = {
+ val resolvedSpec = resolveTableSpec(tableSpec)
+ if (resolvedSpec eq tableSpec) input else withNewSpec(resolvedSpec)
+ }
+
+ /** Resolves a table specification after its expressions have been analyzed.
*/
+ def resolveTableSpec(tableSpec: TableSpecBase): TableSpecBase = tableSpec
match {
// Check this before `childrenResolved`: `ResolveSubquery` skips these
commands, so a subquery
// may never resolve. One that needs no resolution would be dropped by
`toV2Constraint` or
// evaluated as an option value below.
@@ -75,7 +81,7 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
throw QueryCompilationErrors.optionMustBeConstant(key)
case _ =>
}
- input
+ tableSpec
case u: UnresolvedTableSpec if u.childrenResolved =>
val newOptions: Seq[(String, String)] = u.optionExpression.options.map {
case (key: String, null) =>
@@ -139,8 +145,8 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
serde = u.serde,
external = u.external,
constraints = u.constraints.map(_.toV2Constraint))
- withNewSpec(newTableSpec)
+ newTableSpec
case _ =>
- input
+ tableSpec
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
index 37101372260a..70e154543d79 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
@@ -53,6 +53,62 @@ import org.apache.spark.sql.util.CaseInsensitiveStringMap
class AnalysisSuite extends AnalysisTest with Matchers {
import org.apache.spark.sql.catalyst.analysis.TestRelations._
+ test("resolve a table specification without changing option evaluation or
other fields") {
+ val specification = UnresolvedTableSpec(
+ properties = Map("property" -> "value"),
+ provider = Some("parquet"),
+ optionExpression = OptionList(Seq(
+ "answer" -> Literal(1),
+ "answer" -> Add(Literal(40), Literal(2)),
+ "nullable" -> Literal(null))),
+ location = Some("/table"),
+ comment = Some("table comment"),
+ collation = Some("UTF8_BINARY"),
+ serde = None,
+ external = true,
+ constraints = Nil)
+
+ val expected = TableSpec(
+ properties = specification.properties,
+ provider = specification.provider,
+ options = Map("answer" -> "42", "nullable" -> null),
+ location = specification.location,
+ comment = specification.comment,
+ collation = specification.collation,
+ serde = specification.serde,
+ external = specification.external,
+ constraints = Nil)
+ assert(ResolveTableSpec.resolveTableSpec(specification) == expected)
+ assert(ResolveTableSpec.resolveTableSpec(expected) eq expected)
+ val unresolved = specification.copy(
+ optionExpression = OptionList(Seq("answer" ->
UnresolvedAttribute("missing"))))
+ assert(ResolveTableSpec.resolveTableSpec(unresolved) eq unresolved)
+ }
+
+ gridTest("table specification rejects root subquery options before
evaluation")(
+ Seq(false, true)) { resolved =>
+ val subqueryPlan = if (resolved) OneRowRelation() else
UnresolvedRelation(Seq("missing"))
+ val specification = UnresolvedTableSpec(
+ properties = Map.empty,
+ provider = Some("parquet"),
+ optionExpression = OptionList(Seq("answer" ->
ScalarSubquery(subqueryPlan))),
+ location = None,
+ comment = None,
+ collation = None,
+ serde = None,
+ external = false,
+ constraints = Nil)
+
+ val error = intercept[AnalysisException] {
+ ResolveTableSpec.resolveTableSpec(specification)
+ }
+ checkError(
+ exception = error,
+ condition = "INVALID_SQL_SYNTAX.OPTION_IS_INVALID",
+ parameters = Map("key" -> "answer", "supported" -> "constant
expressions"))
+ assert(error.getCause == null)
+ }
+
test("fail for unresolved plan") {
intercept[AnalysisException] {
// `testRelation` does not have column `b`.
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CatalogSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CatalogSuite.scala
index 7786fd742944..9004c7697b4d 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CatalogSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CatalogSuite.scala
@@ -20,8 +20,10 @@ package org.apache.spark.sql.catalyst.analysis
import org.apache.spark.sql.catalyst.CurrentUserContext.CURRENT_USER
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat,
CatalogTable, CatalogTableType}
+import org.apache.spark.sql.catalyst.plans.logical.ColumnDefinition
import org.apache.spark.sql.catalyst.trees.{CurrentOrigin, Origin, TreeNodeTag}
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{CharType, MetadataBuilder, StringType,
StructType, VarcharType}
class CatalogSuite extends AnalysisTest {
@@ -59,6 +61,33 @@ class CatalogSuite extends AnalysisTest {
}
}
+ test("column output preserves metadata, nullability and CHAR/VARCHAR
configuration") {
+ val resolution = new
CatalogIdentifierResolution(getAnalyzer.catalogManager)
+ val metadata = new MetadataBuilder().putString("source", "column").build()
+ val columns = Seq(
+ ColumnDefinition("char_col", CharType(4), nullable = false, metadata =
metadata),
+ ColumnDefinition("nested", new StructType().add("varchar_col",
VarcharType(8))))
+
+ for (preserve <- Seq(false, true); standard <- Seq(false, true)) {
+ withSQLConf(
+ SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> preserve.toString,
+ SQLConf.CHAR_VARCHAR_STANDARD_SEMANTICS.key -> standard.toString) {
+ val output = resolution.columnsToOutput(columns)
+ val expectedTypes = if (preserve || standard) {
+ columns.map(_.dataType)
+ } else {
+ Seq(StringType, new StructType().add("varchar_col", StringType))
+ }
+ assert(output.map(_.name) == columns.map(_.name))
+ assert(output.map(_.dataType) == expectedTypes)
+ assert(output.map(_.nullable) == columns.map(_.nullable))
+ assert(output.map(_.metadata) == columns.map(_.metadata))
+ assert(output.map(_.exprId).distinct.size == columns.size)
+ }
+ }
+ assert(resolution.columnsToOutput(Nil).isEmpty)
+ }
+
test("desc table when owner is set to null") {
val table = CatalogTable(
identifier = TableIdentifier("tbl", Some("db1")),
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableHelper.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableHelper.scala
new file mode 100644
index 000000000000..f2cb818a1848
--- /dev/null
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/CreateTableHelper.scala
@@ -0,0 +1,488 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import scala.collection.mutable.{HashMap, HashSet}
+
+import org.apache.spark.SparkException
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys.CONFIG
+import org.apache.spark.sql.{AnalysisException, SaveMode}
+import org.apache.spark.sql.catalyst.{SQLConfHelper, TableIdentifier}
+import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType, CatalogUtils,
ClusterBySpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, AliasHelper,
RowOrdering}
+import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, GeneratedColumn,
IdentityColumn, ResolveDefaultColumns, WriteDistributionAndOrdering}
+import org.apache.spark.sql.catalyst.util.{ResolveDefaultColumns =>
DefaultCols}
+import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns._
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.connector.catalog.WriteDistributionMode
+import org.apache.spark.sql.connector.expressions.{FieldReference,
LogicalExpressions, NamedReference, RewritableTransform, SortOrder =>
V2SortOrder, Transform}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.execution.command.{CreateDataSourceTableCommand,
DDLUtils}
+import org.apache.spark.sql.execution.datasources.{BucketingUtils, CreateTable
=> CreateTableV1, PartitioningUtils}
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils
+import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
+import org.apache.spark.sql.types.{DataType, MetadataBuilder, StructType}
+import org.apache.spark.sql.util.SchemaUtils
+import org.apache.spark.util.ArrayImplicits._
+
+/** Lowering, preprocessing and validation of CREATE TABLE and REPLACE TABLE
plans. */
+private[sql] object CreateTableHelper extends SQLConfHelper with Logging with
AliasHelper {
+
+ /** Lowers a resolved session-catalog CREATE TABLE when its provider uses V1
commands. */
+ private[analysis] def lowerCreateTable(
+ createTable: CreateTable): LogicalPlan = createTable match {
+ case c @ CreateTable(ResolvedV1CommandIdentifier(ident), _, _, tableSpec:
TableSpec, _, _, _)
+ if c.resolved && c.columns.forall(_.isDefaultValueTypeCoerced) =>
+ val (storageFormat, provider) = getStorageFormatAndProvider(
+ c.tableSpec.provider, tableSpec.options, c.tableSpec.location,
c.tableSpec.serde,
+ ctas = false)
+ if (!isV2Provider(provider)) {
+ if (tableSpec.constraints.nonEmpty) {
+ throw QueryCompilationErrors.unsupportedTableOperationError(
+ ident, "CONSTRAINT")
+ }
+ failIfWriteDistributionOrOrdering(
+ ident, "CREATE TABLE", c.writeDistributionMode, c.writeOrdering)
+ // For V1 CREATE TABLE command, the default value expression is hidden
in the
+ // StructField metadata, and won't be constant folded by the
optimizer. Here we
+ // manually constant fold it, as exist default needs to be a constant.
+ val fields = c.columns.map { col =>
+ col.defaultValue.map { defaultValue =>
+ val existsDefaultSQL = constantFoldDefaultValue(
+ defaultValue, col.name, col.dataType, "CREATE TABLE")
+ // toV1Column already sets CURRENT_DEFAULT, so just update
EXISTS_DEFAULT
+ val field = col.toV1Column
+ val newMetadata = new
MetadataBuilder().withMetadata(field.metadata)
+ .putString(DefaultCols.EXISTS_DEFAULT_COLUMN_METADATA_KEY,
existsDefaultSQL).build()
+ field.copy(metadata = newMetadata)
+ }.getOrElse(col.toV1Column)
+ }
+ constructV1TableCmd(None, c.tableSpec, ident, StructType(fields),
c.partitioning,
+ c.ignoreIfExists, storageFormat, provider)
+ } else {
+ c
+ }
+
+ case _ => createTable
+ }
+
+ /** Rejects REPLACE TABLE for a session-catalog provider that only supports
V1 commands. */
+ private[analysis] def lowerReplaceTable(
+ replaceTable: ReplaceTable): LogicalPlan = replaceTable match {
+ case c @ ReplaceTable(ResolvedV1CommandIdentifier(ident), _, _, _, _, _,
_) if c.resolved =>
+ val provider = c.tableSpec.provider.getOrElse(conf.defaultDataSourceName)
+ if (!isV2Provider(provider)) {
+ throw QueryCompilationErrors.unsupportedTableOperationError(
+ ident, "REPLACE TABLE")
+ } else {
+ c
+ }
+
+ case _ => replaceTable
+ }
+
+ private[analysis] def constructV1TableCmd(
+ query: Option[LogicalPlan],
+ tableSpec: TableSpecBase,
+ ident: TableIdentifier,
+ tableSchema: StructType,
+ partitioning: Seq[Transform],
+ ignoreIfExists: Boolean,
+ storageFormat: CatalogStorageFormat,
+ provider: String): CreateTableV1 = {
+ val tableDesc = buildCatalogTable(
+ ident, tableSchema, partitioning, tableSpec.properties, provider,
tableSpec.location,
+ tableSpec.comment, tableSpec.collation, storageFormat,
tableSpec.external)
+ val mode = if (ignoreIfExists) SaveMode.Ignore else SaveMode.ErrorIfExists
+ CreateTableV1(tableDesc, mode, query)
+ }
+
+ private[analysis] def buildStorageFormatFromSerdeInfo(
+ location: Option[String],
+ maybeSerdeInfo: Option[SerdeInfo]): CatalogStorageFormat = {
+ HiveSerDe.buildStorageFormat(
+ location,
+ maybeSerdeInfo,
+ si => QueryCompilationErrors.invalidFileFormatForStoredAsError(si))
+ }
+
+ private[analysis] def getStorageFormatAndProvider(
+ provider: Option[String],
+ options: Map[String, String],
+ location: Option[String],
+ maybeSerdeInfo: Option[SerdeInfo],
+ ctas: Boolean): (CatalogStorageFormat, String) = {
+ val nonHiveStorageFormat = CatalogStorageFormat.empty.copy(
+ locationUri = location.map(CatalogUtils.stringToURI),
+ properties = options)
+ val defaultHiveStorage = HiveSerDe.getDefaultStorage(conf).copy(
+ locationUri = location.map(CatalogUtils.stringToURI),
+ properties = options)
+ val tableProvider = DataSourceV2Utils.createTableProvider(
+ provider, maybeSerdeInfo.isDefined, ctas, conf)
+
+ if (provider.isDefined) {
+ // The parser guarantees that USING and STORED AS/ROW FORMAT won't
co-exist.
+ if (maybeSerdeInfo.isDefined) {
+ throw SparkException.internalError(
+ s"Cannot create table with both USING ${provider.get} and " +
+ s"${maybeSerdeInfo.get.describe}")
+ }
+ (nonHiveStorageFormat, tableProvider)
+ } else if (maybeSerdeInfo.isDefined) {
+ val serdeInfo = maybeSerdeInfo.get
+ SerdeInfo.checkSerdePropMerging(serdeInfo.serdeProperties,
defaultHiveStorage.properties)
+ val storageFormat = if (serdeInfo.storedAs.isDefined) {
+ // If `STORED AS fileFormat` is used, infer inputFormat, outputFormat
and serde from it.
+ HiveSerDe.sourceToSerDe(serdeInfo.storedAs.get) match {
+ case Some(hiveSerde) =>
+ defaultHiveStorage.copy(
+ inputFormat =
hiveSerde.inputFormat.orElse(defaultHiveStorage.inputFormat),
+ outputFormat =
hiveSerde.outputFormat.orElse(defaultHiveStorage.outputFormat),
+ // User specified serde takes precedence over the one inferred
from file format.
+ serde =
serdeInfo.serde.orElse(hiveSerde.serde).orElse(defaultHiveStorage.serde),
+ properties = serdeInfo.serdeProperties ++
defaultHiveStorage.properties)
+ case _ => throw
QueryCompilationErrors.invalidFileFormatForStoredAsError(serdeInfo)
+ }
+ } else {
+ defaultHiveStorage.copy(
+ inputFormat =
+
serdeInfo.formatClasses.map(_.input).orElse(defaultHiveStorage.inputFormat),
+ outputFormat =
+
serdeInfo.formatClasses.map(_.output).orElse(defaultHiveStorage.outputFormat),
+ serde = serdeInfo.serde.orElse(defaultHiveStorage.serde),
+ properties = serdeInfo.serdeProperties ++
defaultHiveStorage.properties)
+ }
+ (storageFormat, tableProvider)
+ } else {
+ // If neither USING nor STORED AS/ROW FORMAT is specified, we create
native data source
+ // tables if:
+ // 1. `LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT` is false, or
+ // 2. It's a CTAS and `conf.convertCTAS` is true.
+ if (!DataSourceV2Utils.createsHiveTableByDefault(ctas, conf)) {
+ (nonHiveStorageFormat, tableProvider)
+ } else {
+ logWarning(log"A Hive serde table will be created as there is no table
provider " +
+ log"specified. You can set " +
+ log"${MDC(CONFIG, SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key)}
to false so that " +
+ log"native data source table will be created instead.")
+ (defaultHiveStorage, tableProvider)
+ }
+ }
+ }
+
+ private def buildCatalogTable(
+ table: TableIdentifier,
+ schema: StructType,
+ partitioning: Seq[Transform],
+ properties: Map[String, String],
+ provider: String,
+ location: Option[String],
+ comment: Option[String],
+ collation: Option[String],
+ storageFormat: CatalogStorageFormat,
+ external: Boolean): CatalogTable = {
+ val tableType = if (external || location.isDefined) {
+ CatalogTableType.EXTERNAL
+ } else {
+ CatalogTableType.MANAGED
+ }
+ val (partitionColumns, maybeBucketSpec, maybeClusterBySpec) =
partitioning.convertTransforms
+
+ CatalogTable(
+ identifier = table,
+ tableType = tableType,
+ storage = storageFormat,
+ schema = schema,
+ provider = Some(provider),
+ partitionColumnNames = partitionColumns,
+ bucketSpec = maybeBucketSpec,
+ properties = properties ++
+ maybeClusterBySpec.map(
+ clusterBySpec => ClusterBySpec.toProperty(schema, clusterBySpec,
conf.resolver)),
+ comment = comment,
+ collation = collation
+ )
+ }
+
+ /** Rejects a write distribution or ordering, which a v1 table cannot
record. */
+ private[analysis] def failIfWriteDistributionOrOrdering(
+ ident: TableIdentifier,
+ operation: String,
+ writeDistributionMode: WriteDistributionMode,
+ writeOrdering: Seq[V2SortOrder]): Unit = {
+ if (WriteDistributionAndOrdering.isRequested(writeDistributionMode,
writeOrdering)) {
+ throw QueryCompilationErrors.unsupportedTableOperationError(
+ ident, s"$operation ... ${WriteDistributionAndOrdering.CLAUSES}")
+ }
+ }
+
+ private[analysis] def isV2Provider(provider: String): Boolean = {
+ DataSourceV2Utils.isV2Provider(provider, conf)
+ }
+
+ /** Validates and constant-folds a V1 column default for EXISTS_DEFAULT
metadata. */
+ private[analysis] def constantFoldDefaultValue(
+ defaultValue: DefaultValueExpression,
+ colName: String,
+ colType: DataType,
+ statementType: String): String = {
+ validateDefaultValueExpr(defaultValue, statementType, colName,
Some(colType))
+ val fakePlan = Project(Seq(Alias(defaultValue.child, colName)()),
OneRowRelation())
+ val optimizedPlan =
ConstantFolding(DefaultColumnOptimizer.FinishAnalysis(fakePlan))
+ val constantFolded = optimizedPlan.collectFirst {
+ case Project(Seq(expr), OneRowRelation()) => trimAliases(expr)
+ }.get
+ if (!constantFolded.foldable) {
+ throw QueryCompilationErrors.defaultValueNotConstantError(statementType,
colName,
+ defaultValue.originalSQL)
+ }
+ constantFolded.sql
+ }
+
+ /** Checks partitioning and bucketing when a V1 CREATE TABLE has no schema.
*/
+ def checkCreateTableWithoutSchema(createTable: CreateTableV1): CreateTableV1
= {
+ val tableDesc = createTable.tableDesc
+ if (tableDesc.bucketSpec.isDefined) {
+ throw new AnalysisException(
+ errorClass = "SPECIFY_BUCKETING_IS_NOT_ALLOWED",
+ messageParameters = Map.empty
+ )
+ }
+ if (tableDesc.partitionColumnNames.nonEmpty) {
+ throw new AnalysisException(
+ errorClass = "SPECIFY_PARTITION_IS_NOT_ALLOWED",
+ messageParameters = Map.empty)
+ }
+ createTable
+ }
+
+ /** Normalizes a V1 CREATE TABLE schema and places partition columns last. */
+ def normalizeCreateTableWithoutQuery(createTable: CreateTableV1):
CreateTableV1 = {
+ val tableDesc = createTable.tableDesc
+ DDLUtils.checkTableColumns(tableDesc)
+
+ val normalizedTable = normalizeCatalogTable(tableDesc.schema, tableDesc)
+
+ val normalizedSchemaByName = HashMap(normalizedTable.schema.map(s =>
s.name -> s): _*)
+ val partitionSchema = normalizedTable.partitionColumnNames.map { partCol =>
+ normalizedSchemaByName(partCol)
+ }
+ val partitionSchemaSet = HashSet(partitionSchema: _*)
+ val reorderedSchema = StructType(
+ normalizedTable.schema.filterNot(partitionSchemaSet.contains) ++
partitionSchema
+ )
+
+ createTable.copy(tableDesc = normalizedTable.copy(schema =
reorderedSchema))
+ }
+
+ /** Validates and normalizes a V2 table's columns, partitioning and write
ordering. */
+ def preprocessV2CreateTablePlan(create: V2CreateTablePlan): LogicalPlan = {
+ val schema = create.tableSchema
+ val partitioning = create.partitioning
+ val isCaseSensitive = conf.caseSensitiveAnalysis
+ // Check that columns are not duplicated in the schema
+ val flattenedSchema = SchemaUtils.explodeNestedFieldNames(schema)
+ SchemaUtils.checkColumnNameDuplication(
+ flattenedSchema,
+ isCaseSensitive)
+
+ // Check that columns are not duplicated in the partitioning statement
+ SchemaUtils.checkTransformDuplication(
+ partitioning, "in the partitioning", isCaseSensitive)
+
+ // The write ordering has no duplicate check: a repeated sort key is
redundant, and
+ // `bucket(4, id), bucket(8, id)` are distinct keys.
+
+ if (schema.isEmpty) {
+ if (partitioning.nonEmpty) {
+ throw
QueryCompilationErrors.specifyPartitionNotAllowedWhenTableSchemaNotDefinedError()
+ }
+ if (create.writeOrdering.nonEmpty) {
+ throw QueryCompilationErrors
+
.specifyWriteOrderingNotAllowedWhenTableSchemaNotDefinedError(create.origin)
+ }
+
+ create
+ } else {
+ // Resolve and normalize partition columns as necessary
+ val resolver = conf.resolver
+ val normalizedPartitions = partitioning.map {
+ case transform: RewritableTransform =>
+ val rewritten = transform.references().map { ref =>
+ // Throws an exception if the reference cannot be resolved
+ val position = SchemaUtils
+ .findColumnPosition(ref.fieldNames().toImmutableArraySeq,
schema, resolver)
+ FieldReference(SchemaUtils.getColumnName(position, schema))
+ }
+ transform.withReferences(rewritten.toImmutableArraySeq)
+ case other => other
+ }
+
+ // Unlike the partitioning, an ordering reference to a missing column,
including a path
+ // through a non-struct field, is kept as is for CheckAnalysis to
report, and each
+ // reference is normalized independently.
+ def normalizeResolvable(ref: NamedReference): NamedReference = {
+ WriteDistributionAndOrdering
+ .findColumn(schema, ref.fieldNames().toImmutableArraySeq, resolver)
+ .map { case (path, field) => FieldReference(path :+ field.name) }
+ .getOrElse(ref)
+ }
+
+ val normalizedOrdering = create.writeOrdering.map { sortOrder =>
+ val key = sortOrder.expression() match {
+ case ref: NamedReference => normalizeResolvable(ref)
+ case transform: RewritableTransform =>
+ transform.withReferences(
+
transform.references().map(normalizeResolvable).toImmutableArraySeq)
+ case other => other
+ }
+ LogicalExpressions.sort(key, sortOrder.direction(),
sortOrder.nullOrdering())
+ }
+
+
create.withPartitioning(normalizedPartitions).withWriteOrdering(normalizedOrdering)
+ }
+ }
+
+ def normalizeCatalogTable(schema: StructType, table: CatalogTable):
CatalogTable = {
+ SchemaUtils.checkSchemaColumnNameDuplication(
+ schema,
+ conf.caseSensitiveAnalysis)
+ if (!conf.allowCollationsInMapKeys) {
+ SchemaUtils.checkNoCollationsInMapKeys(schema)
+ }
+
+ val normalizedPartCols = normalizePartitionColumns(schema, table)
+ val normalizedBucketSpec = normalizeBucketSpec(schema, table)
+
+ normalizedBucketSpec.foreach { spec =>
+ for (bucketCol <- spec.bucketColumnNames if
normalizedPartCols.contains(bucketCol)) {
+ throw
QueryCompilationErrors.bucketingColumnCannotBePartOfPartitionColumnsError(
+ bucketCol, normalizedPartCols)
+ }
+ for (sortCol <- spec.sortColumnNames if
normalizedPartCols.contains(sortCol)) {
+ throw
QueryCompilationErrors.bucketSortingColumnCannotBePartOfPartitionColumnsError(
+ sortCol, normalizedPartCols)
+ }
+ }
+
+ val normalizedProperties = table.properties ++ table.clusterBySpec.map {
spec =>
+ ClusterBySpec.toProperty(schema, spec, conf.resolver)
+ }
+
+ table.copy(partitionColumnNames = normalizedPartCols, bucketSpec =
normalizedBucketSpec,
+ properties = normalizedProperties)
+ }
+
+ private def normalizePartitionColumns(schema: StructType, table:
CatalogTable): Seq[String] = {
+ val normalizedPartitionCols = CatalogUtils.normalizePartCols(
+ tableName = table.identifier.unquotedString,
+ tableCols = schema.map(_.name),
+ partCols = table.partitionColumnNames,
+ resolver = conf.resolver)
+
+ SchemaUtils.checkColumnNameDuplication(normalizedPartitionCols,
conf.resolver)
+
+ if (schema.nonEmpty && normalizedPartitionCols.length == schema.length) {
+ throw new AnalysisException(
+ errorClass = "ALL_PARTITION_COLUMNS_NOT_ALLOWED",
+ messageParameters = Map.empty)
+ }
+
+ val normalizedPartitionColsSet = HashSet(normalizedPartitionCols: _*)
+ schema
+ .filter(f => normalizedPartitionColsSet.contains(f.name))
+ .foreach { field =>
+ if (!PartitioningUtils.canPartitionOn(field.dataType)) {
+ throw
QueryCompilationErrors.invalidPartitionColumnDataTypeError(field)
+ }
+ }
+
+ normalizedPartitionCols
+ }
+
+ private def normalizeBucketSpec(schema: StructType, table: CatalogTable):
Option[BucketSpec] = {
+ table.bucketSpec match {
+ case Some(bucketSpec) =>
+ val normalizedBucketSpec = CatalogUtils.normalizeBucketSpec(
+ tableName = table.identifier.unquotedString,
+ tableCols = schema.map(_.name),
+ bucketSpec = bucketSpec,
+ resolver = conf.resolver)
+
+ SchemaUtils.checkColumnNameDuplication(
+ normalizedBucketSpec.bucketColumnNames,
+ conf.resolver)
+ SchemaUtils.checkColumnNameDuplication(
+ normalizedBucketSpec.sortColumnNames,
+ conf.resolver)
+
+
normalizedBucketSpec.sortColumnNames.map(schema(_)).map(_.dataType).foreach {
+ case dt if RowOrdering.isOrderable(dt) => // OK
+ case other => failAnalysis(s"Cannot use ${other.catalogString} for
sorting column")
+ }
+
+ schema.filter(f =>
normalizedBucketSpec.bucketColumnNames.contains(f.name))
+ .foreach { field =>
+ if (!BucketingUtils.canBucketOn(field.dataType)) {
+ throw
QueryCompilationErrors.invalidBucketColumnDataTypeError(field.dataType)
+ }
+ }
+
+ Some(normalizedBucketSpec)
+
+ case None => None
+ }
+ }
+
+ private def failAnalysis(msg: String) = {
+ throw new AnalysisException(
+ errorClass = "_LEGACY_ERROR_TEMP_3072", messageParameters = Map("msg" ->
msg))
+ }
+
+ /** Validates a V1 datasource table and builds its CREATE TABLE command. */
+ def createDataSourceTableCommand(
+ tableDesc: CatalogTable,
+ mode: SaveMode): CreateDataSourceTableCommand = {
+ ResolveDefaultColumns.validateTableProviderForDefaultValue(
+ tableDesc.schema, tableDesc.provider, "CREATE TABLE", false)
+
+ if (GeneratedColumn.hasGeneratedColumns(tableDesc.schema)) {
+ throw QueryCompilationErrors.unsupportedTableOperationError(
+ tableDesc.identifier, "generated columns")
+ }
+
+ if (IdentityColumn.hasIdentityColumns(tableDesc.schema)) {
+ throw QueryCompilationErrors.unsupportedTableOperationError(
+ tableDesc.identifier, "identity columns")
+ }
+
+ CreateDataSourceTableCommand(tableDesc, ignoreIfExists = mode ==
SaveMode.Ignore)
+ }
+
+ private[analysis] def replaceCharWithVarcharInColumn(col: ColumnDefinition):
ColumnDefinition = {
+ col.copy(dataType = CharVarcharUtils.replaceCharWithVarchar(col.dataType))
+ }
+
+ private[analysis] def replaceCharWithVarcharInTableMeta(tbl: CatalogTable):
CatalogTable = {
+ tbl.copy(schema =
CharVarcharUtils.replaceCharWithVarchar(tbl.schema).asInstanceOf[StructType])
+ }
+}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ReplaceCharWithVarchar.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ReplaceCharWithVarchar.scala
index 408d0e3e54cb..8918b3fbb271 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ReplaceCharWithVarchar.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ReplaceCharWithVarchar.scala
@@ -17,15 +17,15 @@
package org.apache.spark.sql.catalyst.analysis
-import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.catalyst.plans.logical.{AddColumns, AlterColumns,
ColumnDefinition, CreateTable, LogicalPlan, ReplaceColumns, ReplaceTable}
+import org.apache.spark.sql.catalyst.plans.logical.{AddColumns, AlterColumns,
CreateTable, LogicalPlan, ReplaceColumns, ReplaceTable}
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.execution.command.{AlterTableAddColumnsCommand,
AlterTableChangeColumnCommand, CreateDataSourceTableCommand, CreateTableCommand}
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.StructType
object ReplaceCharWithVarchar extends Rule[LogicalPlan] {
+ import CreateTableHelper._
+
override def apply(plan: LogicalPlan): LogicalPlan = {
if (!conf.getConf(SQLConf.CHAR_AS_VARCHAR)) return plan
@@ -62,12 +62,4 @@ object ReplaceCharWithVarchar extends Rule[LogicalPlan] {
dataType =
CharVarcharUtils.replaceCharWithVarchar(cmd.newColumn.dataType)))
}
}
-
- private def replaceCharWithVarcharInColumn(col: ColumnDefinition):
ColumnDefinition = {
- col.copy(dataType = CharVarcharUtils.replaceCharWithVarchar(col.dataType))
- }
-
- private def replaceCharWithVarcharInTableMeta(tbl: CatalogTable):
CatalogTable = {
- tbl.copy(schema =
CharVarcharUtils.replaceCharWithVarchar(tbl.schema).asInstanceOf[StructType])
- }
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
index 2413607af28c..198fddd46d64 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
@@ -18,24 +18,19 @@
package org.apache.spark.sql.catalyst.analysis
import org.apache.spark.SparkException
-import org.apache.spark.internal.LogKeys.CONFIG
-import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.V1CommandUtils.supportsV1Command
-import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat,
CatalogTable, CatalogTableType, CatalogUtils, ClusterBySpec, HiveTableRelation}
-import org.apache.spark.sql.catalyst.expressions.{Alias, AliasHelper,
Attribute}
-import org.apache.spark.sql.catalyst.optimizer.ConstantFolding
+import org.apache.spark.sql.catalyst.catalog.{ClusterBySpec, HiveTableRelation}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
-import org.apache.spark.sql.catalyst.util.{quoteIfNeeded, toPrettySQL,
CharVarcharUtils, ResolveDefaultColumns => DefaultCols,
WriteDistributionAndOrdering}
+import org.apache.spark.sql.catalyst.util.{quoteIfNeeded, toPrettySQL,
CharVarcharUtils, ResolveDefaultColumns => DefaultCols}
import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns._
-import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util,
LookupCatalog, SupportsNamespaces, V1Table, ViewCatalog, WriteDistributionMode}
-import org.apache.spark.sql.connector.expressions.{SortOrder => V2SortOrder,
Transform}
+import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util,
LookupCatalog, SupportsNamespaces, V1Table, ViewCatalog}
import org.apache.spark.sql.errors.{QueryCompilationErrors,
QueryExecutionErrors}
import org.apache.spark.sql.execution.command._
-import org.apache.spark.sql.execution.datasources.{CreateTable =>
CreateTableV1, LogicalRelation}
-import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils
-import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.connector.V1Function
import org.apache.spark.sql.metricview.logical.CreateMetricView
import org.apache.spark.sql.types.{DataType, MetadataBuilder, StringType,
StructField, StructType}
@@ -48,7 +43,8 @@ import org.apache.spark.util.SparkStringUtils
* again, which may lead to inconsistent behavior if the current database is
changed in the middle.
*/
class ResolveSessionCatalog(val catalogManager: CatalogManager)
- extends Rule[LogicalPlan] with LookupCatalog with AliasHelper {
+ extends Rule[LogicalPlan] with LookupCatalog {
+ import CreateTableHelper._
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
import org.apache.spark.sql.connector.catalog.CatalogV2Util._
import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Implicits._
@@ -269,37 +265,8 @@ class ResolveSessionCatalog(val catalogManager:
CatalogManager)
// For CREATE TABLE [AS SELECT], we should use the v1 command if the
catalog is resolved to the
// session catalog and the table provider is not v2.
- case c @ CreateTable(ResolvedV1CommandIdentifier(ident), _, _, tableSpec:
TableSpec, _, _, _)
- if c.resolved && c.columns.forall(_.isDefaultValueTypeCoerced) =>
- val (storageFormat, provider) = getStorageFormatAndProvider(
- c.tableSpec.provider, tableSpec.options, c.tableSpec.location,
c.tableSpec.serde,
- ctas = false)
- if (!isV2Provider(provider)) {
- if (tableSpec.constraints.nonEmpty) {
- throw QueryCompilationErrors.unsupportedTableOperationError(
- ident, "CONSTRAINT")
- }
- failIfWriteDistributionOrOrdering(
- ident, "CREATE TABLE", c.writeDistributionMode, c.writeOrdering)
- // For V1 CREATE TABLE command, the default value expression is hidden
in the
- // StructField metadata, and won't be constant folded by the
optimizer. Here we
- // manually constant fold it, as exist default needs to be a constant.
- val fields = c.columns.map { col =>
- col.defaultValue.map { defaultValue =>
- val existsDefaultSQL = constantFoldDefaultValue(
- defaultValue, col.name, col.dataType, "CREATE TABLE")
- // toV1Column already sets CURRENT_DEFAULT, so just update
EXISTS_DEFAULT
- val field = col.toV1Column
- val newMetadata = new
MetadataBuilder().withMetadata(field.metadata)
- .putString(DefaultCols.EXISTS_DEFAULT_COLUMN_METADATA_KEY,
existsDefaultSQL).build()
- field.copy(metadata = newMetadata)
- }.getOrElse(col.toV1Column)
- }
- constructV1TableCmd(None, c.tableSpec, ident, StructType(fields),
c.partitioning,
- c.ignoreIfExists, storageFormat, provider)
- } else {
- c
- }
+ case c: CreateTable =>
+ CreateTableHelper.lowerCreateTable(c)
case c @ CreateTableAsSelect(
ResolvedV1CommandIdentifier(ident), _, _, tableSpec: TableSpec,
writeOptions, _, _, _, _) =>
@@ -328,14 +295,8 @@ class ResolveSessionCatalog(val catalogManager:
CatalogManager)
// For REPLACE TABLE [AS SELECT], we should fail if the catalog is
resolved to the
// session catalog and the table provider is not v2.
- case c @ ReplaceTable(ResolvedV1CommandIdentifier(ident), _, _, _, _, _,
_) if c.resolved =>
- val provider = c.tableSpec.provider.getOrElse(conf.defaultDataSourceName)
- if (!isV2Provider(provider)) {
- throw QueryCompilationErrors.unsupportedTableOperationError(
- ident, "REPLACE TABLE")
- } else {
- c
- }
+ case c: ReplaceTable =>
+ CreateTableHelper.lowerReplaceTable(c)
case c @ ReplaceTableAsSelect(ResolvedV1CommandIdentifier(ident), _, _, _,
_, _, _, _, _) =>
val provider = c.tableSpec.provider.getOrElse(conf.defaultDataSourceName)
@@ -690,149 +651,9 @@ class ResolveSessionCatalog(val catalogManager:
CatalogManager)
throw
QueryCompilationErrors.missingCatalogCreateFunctionAbilityError(catalog)
}
- /** Rejects a write distribution or ordering, which a v1 table cannot
record. */
- private def failIfWriteDistributionOrOrdering(
- ident: TableIdentifier,
- operation: String,
- writeDistributionMode: WriteDistributionMode,
- writeOrdering: Seq[V2SortOrder]): Unit = {
- if (WriteDistributionAndOrdering.isRequested(writeDistributionMode,
writeOrdering)) {
- throw QueryCompilationErrors.unsupportedTableOperationError(
- ident, s"$operation ... ${WriteDistributionAndOrdering.CLAUSES}")
- }
- }
-
- private def constructV1TableCmd(
- query: Option[LogicalPlan],
- tableSpec: TableSpecBase,
- ident: TableIdentifier,
- tableSchema: StructType,
- partitioning: Seq[Transform],
- ignoreIfExists: Boolean,
- storageFormat: CatalogStorageFormat,
- provider: String): CreateTableV1 = {
- val tableDesc = buildCatalogTable(
- ident, tableSchema, partitioning, tableSpec.properties, provider,
tableSpec.location,
- tableSpec.comment, tableSpec.collation, storageFormat,
tableSpec.external)
- val mode = if (ignoreIfExists) SaveMode.Ignore else SaveMode.ErrorIfExists
- CreateTableV1(tableDesc, mode, query)
- }
-
- private def buildStorageFormatFromSerdeInfo(
- location: Option[String],
- maybeSerdeInfo: Option[SerdeInfo]): CatalogStorageFormat = {
- HiveSerDe.buildStorageFormat(
- location,
- maybeSerdeInfo,
- si => QueryCompilationErrors.invalidFileFormatForStoredAsError(si))
- }
-
- private def getStorageFormatAndProvider(
- provider: Option[String],
- options: Map[String, String],
- location: Option[String],
- maybeSerdeInfo: Option[SerdeInfo],
- ctas: Boolean): (CatalogStorageFormat, String) = {
- val nonHiveStorageFormat = CatalogStorageFormat.empty.copy(
- locationUri = location.map(CatalogUtils.stringToURI),
- properties = options)
- val defaultHiveStorage = HiveSerDe.getDefaultStorage(conf).copy(
- locationUri = location.map(CatalogUtils.stringToURI),
- properties = options)
- val tableProvider = DataSourceV2Utils.createTableProvider(
- provider, maybeSerdeInfo.isDefined, ctas, conf)
-
- if (provider.isDefined) {
- // The parser guarantees that USING and STORED AS/ROW FORMAT won't
co-exist.
- if (maybeSerdeInfo.isDefined) {
- throw SparkException.internalError(
- s"Cannot create table with both USING ${provider.get} and " +
- s"${maybeSerdeInfo.get.describe}")
- }
- (nonHiveStorageFormat, tableProvider)
- } else if (maybeSerdeInfo.isDefined) {
- val serdeInfo = maybeSerdeInfo.get
- SerdeInfo.checkSerdePropMerging(serdeInfo.serdeProperties,
defaultHiveStorage.properties)
- val storageFormat = if (serdeInfo.storedAs.isDefined) {
- // If `STORED AS fileFormat` is used, infer inputFormat, outputFormat
and serde from it.
- HiveSerDe.sourceToSerDe(serdeInfo.storedAs.get) match {
- case Some(hiveSerde) =>
- defaultHiveStorage.copy(
- inputFormat =
hiveSerde.inputFormat.orElse(defaultHiveStorage.inputFormat),
- outputFormat =
hiveSerde.outputFormat.orElse(defaultHiveStorage.outputFormat),
- // User specified serde takes precedence over the one inferred
from file format.
- serde =
serdeInfo.serde.orElse(hiveSerde.serde).orElse(defaultHiveStorage.serde),
- properties = serdeInfo.serdeProperties ++
defaultHiveStorage.properties)
- case _ => throw
QueryCompilationErrors.invalidFileFormatForStoredAsError(serdeInfo)
- }
- } else {
- defaultHiveStorage.copy(
- inputFormat =
-
serdeInfo.formatClasses.map(_.input).orElse(defaultHiveStorage.inputFormat),
- outputFormat =
-
serdeInfo.formatClasses.map(_.output).orElse(defaultHiveStorage.outputFormat),
- serde = serdeInfo.serde.orElse(defaultHiveStorage.serde),
- properties = serdeInfo.serdeProperties ++
defaultHiveStorage.properties)
- }
- (storageFormat, tableProvider)
- } else {
- // If neither USING nor STORED AS/ROW FORMAT is specified, we create
native data source
- // tables if:
- // 1. `LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT` is false, or
- // 2. It's a CTAS and `conf.convertCTAS` is true.
- if (!DataSourceV2Utils.createsHiveTableByDefault(ctas, conf)) {
- (nonHiveStorageFormat, tableProvider)
- } else {
- logWarning(log"A Hive serde table will be created as there is no table
provider " +
- log"specified. You can set " +
- log"${MDC(CONFIG, SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key)}
to false so that " +
- log"native data source table will be created instead.")
- (defaultHiveStorage, tableProvider)
- }
- }
- }
-
- private def buildCatalogTable(
- table: TableIdentifier,
- schema: StructType,
- partitioning: Seq[Transform],
- properties: Map[String, String],
- provider: String,
- location: Option[String],
- comment: Option[String],
- collation: Option[String],
- storageFormat: CatalogStorageFormat,
- external: Boolean): CatalogTable = {
- val tableType = if (external || location.isDefined) {
- CatalogTableType.EXTERNAL
- } else {
- CatalogTableType.MANAGED
- }
- val (partitionColumns, maybeBucketSpec, maybeClusterBySpec) =
partitioning.convertTransforms
-
- CatalogTable(
- identifier = table,
- tableType = tableType,
- storage = storageFormat,
- schema = schema,
- provider = Some(provider),
- partitionColumnNames = partitionColumns,
- bucketSpec = maybeBucketSpec,
- properties = properties ++
- maybeClusterBySpec.map(
- clusterBySpec => ClusterBySpec.toProperty(schema, clusterBySpec,
conf.resolver)),
- comment = comment,
- collation = collation
- )
- }
-
private object CreateFunctionInSessionCatalog
extends ResolvedIdentifierInSessionCatalog("CREATE", "FUNCTION")
- private def isV2Provider(provider: String): Boolean = {
- DataSourceV2Utils.isV2Provider(provider, conf)
- }
-
/**
* Validates and constant-folds a column default value expression for V1
tables.
* Populates both CURRENT_DEFAULT and EXISTS_DEFAULT in the metadata builder.
@@ -850,28 +671,6 @@ class ResolveSessionCatalog(val catalogManager:
CatalogManager)
builder.putString(DefaultCols.EXISTS_DEFAULT_COLUMN_METADATA_KEY,
existsDefaultSQL)
}
- /**
- * Validates and constant-folds a column default value expression for V1
tables.
- * Returns the SQL string of the constant-folded expression for
EXISTS_DEFAULT metadata.
- */
- private def constantFoldDefaultValue(
- defaultValue: DefaultValueExpression,
- colName: String,
- colType: DataType,
- statementType: String): String = {
- validateDefaultValueExpr(defaultValue, statementType, colName,
Some(colType))
- val fakePlan = Project(Seq(Alias(defaultValue.child, colName)()),
OneRowRelation())
- val optimizedPlan =
ConstantFolding(DefaultColumnOptimizer.FinishAnalysis(fakePlan))
- val constantFolded = optimizedPlan.collectFirst {
- case Project(Seq(expr), OneRowRelation()) => trimAliases(expr)
- }.get
- if (!constantFolded.foldable) {
- throw QueryCompilationErrors.defaultValueNotConstantError(statementType,
colName,
- defaultValue.originalSQL)
- }
- constantFolded.sql
- }
-
// True when the ALTER COLUMN specs only change comments (set or drop, no
type / nullability /
// position / default changes) and, taken together, touch multiple columns
or a nested field.
// Such changes come from `COMMENT ON TABLE ... COLUMN (...)` and cannot be
expressed by the
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
index 61b8d5c1bd5d..45ca57ad6116 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala
@@ -43,7 +43,7 @@ import
org.apache.spark.sql.catalyst.plans.logical.{AppendData, InsertIntoDir, I
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.streaming.{StreamingRelationV2,
StreamingSourceIdentifyingName, Unassigned}
import org.apache.spark.sql.catalyst.types.DataTypeUtils
-import org.apache.spark.sql.catalyst.util.{GeneratedColumn, IdentityColumn,
PushableExpression, ResolveDefaultColumns}
+import org.apache.spark.sql.catalyst.util.PushableExpression
import org.apache.spark.sql.classic.{SparkSession, Strategy}
import org.apache.spark.sql.connector.catalog.{SupportsRead, V1Table}
import org.apache.spark.sql.connector.catalog.TableCapability._
@@ -143,20 +143,7 @@ object DataSourceAnalysis extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case CreateTable(tableDesc, mode, None) if
DDLUtils.isDatasourceTable(tableDesc) =>
- ResolveDefaultColumns.validateTableProviderForDefaultValue(
- tableDesc.schema, tableDesc.provider, "CREATE TABLE", false)
-
- if (GeneratedColumn.hasGeneratedColumns(tableDesc.schema)) {
- throw QueryCompilationErrors.unsupportedTableOperationError(
- tableDesc.identifier, "generated columns")
- }
-
- if (IdentityColumn.hasIdentityColumns(tableDesc.schema)) {
- throw QueryCompilationErrors.unsupportedTableOperationError(
- tableDesc.identifier, "identity columns")
- }
-
- CreateDataSourceTableCommand(tableDesc, ignoreIfExists = mode ==
SaveMode.Ignore)
+ CreateTableHelper.createDataSourceTableCommand(tableDesc, mode)
case CreateTable(tableDesc, mode, Some(query))
if query.resolved && DDLUtils.isDatasourceTable(tableDesc) =>
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
index 4e23db80d017..46899a3272a4 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/rules.scala
@@ -27,14 +27,12 @@ import org.apache.spark.{SparkException,
SparkUnsupportedOperationException}
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.catalog._
-import org.apache.spark.sql.catalyst.expressions.{Expression,
InputFileBlockLength, InputFileBlockStart, InputFileName, RowOrdering}
+import org.apache.spark.sql.catalyst.expressions.{Expression,
InputFileBlockLength, InputFileBlockStart, InputFileName}
import org.apache.spark.sql.catalyst.plans.logical._
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes
import org.apache.spark.sql.catalyst.util.TypeUtils._
-import org.apache.spark.sql.catalyst.util.WriteDistributionAndOrdering
import org.apache.spark.sql.classic.SparkSession
-import org.apache.spark.sql.connector.expressions.{FieldReference,
LogicalExpressions, NamedReference, RewritableTransform}
import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.command.DDLUtils
import org.apache.spark.sql.execution.datasources.{CreateTable =>
CreateTableV1}
@@ -153,18 +151,7 @@ case class PreprocessTableCreation(catalog:
SessionCatalog) extends Rule[Logical
// Since the runtime inferred partition columns could be different from
what user specified,
// we fail the query if the partitioning information is specified.
case c @ CreateTableV1(tableDesc, _, None) if tableDesc.schema.isEmpty =>
- if (tableDesc.bucketSpec.isDefined) {
- throw new AnalysisException(
- errorClass = "SPECIFY_BUCKETING_IS_NOT_ALLOWED",
- messageParameters = Map.empty
- )
- }
- if (tableDesc.partitionColumnNames.nonEmpty) {
- throw new AnalysisException(
- errorClass = "SPECIFY_PARTITION_IS_NOT_ALLOWED",
- messageParameters = Map.empty)
- }
- c
+ CreateTableHelper.checkCreateTableWithoutSchema(c)
// When we append data to an existing table, check if the given provider,
partition columns,
// bucket spec, etc. match the existing table, and adjust the columns
order of the given query
@@ -280,7 +267,8 @@ case class PreprocessTableCreation(catalog: SessionCatalog)
extends Rule[Logical
"Schema may not be specified in a Create Table As Select (CTAS)
statement")
val analyzedQuery = query.get
- val normalizedTable = normalizeCatalogTable(analyzedQuery.schema,
tableDesc)
+ val normalizedTable =
+ CreateTableHelper.normalizeCatalogTable(analyzedQuery.schema,
tableDesc)
DDLUtils.checkTableColumns(tableDesc.copy(schema =
analyzedQuery.schema))
SchemaUtils.checkIndeterminateCollationInSchema(analyzedQuery.schema)
@@ -302,87 +290,11 @@ case class PreprocessTableCreation(catalog:
SessionCatalog) extends Rule[Logical
c.copy(tableDesc = normalizedTable, query = Some(reorderedQuery))
} else {
- DDLUtils.checkTableColumns(tableDesc)
-
- val normalizedTable = normalizeCatalogTable(tableDesc.schema,
tableDesc)
-
- val normalizedSchemaByName = HashMap(normalizedTable.schema.map(s =>
s.name -> s): _*)
- val partitionSchema = normalizedTable.partitionColumnNames.map {
partCol =>
- normalizedSchemaByName(partCol)
- }
- val partitionSchemaSet = HashSet(partitionSchema: _*)
- val reorderedSchema = StructType(
- normalizedTable.schema.filterNot(partitionSchemaSet.contains) ++
partitionSchema
- )
-
- c.copy(tableDesc = normalizedTable.copy(schema = reorderedSchema))
+ CreateTableHelper.normalizeCreateTableWithoutQuery(c)
}
case create: V2CreateTablePlan if create.childrenResolved =>
- val schema = create.tableSchema
- val partitioning = create.partitioning
- val isCaseSensitive = conf.caseSensitiveAnalysis
- // Check that columns are not duplicated in the schema
- val flattenedSchema = SchemaUtils.explodeNestedFieldNames(schema)
- SchemaUtils.checkColumnNameDuplication(
- flattenedSchema,
- isCaseSensitive)
-
- // Check that columns are not duplicated in the partitioning statement
- SchemaUtils.checkTransformDuplication(
- partitioning, "in the partitioning", isCaseSensitive)
-
- // The write ordering has no duplicate check: a repeated sort key is
redundant, and
- // `bucket(4, id), bucket(8, id)` are distinct keys.
-
- if (schema.isEmpty) {
- if (partitioning.nonEmpty) {
- throw
QueryCompilationErrors.specifyPartitionNotAllowedWhenTableSchemaNotDefinedError()
- }
- if (create.writeOrdering.nonEmpty) {
- throw QueryCompilationErrors
-
.specifyWriteOrderingNotAllowedWhenTableSchemaNotDefinedError(create.origin)
- }
-
- create
- } else {
- // Resolve and normalize partition columns as necessary
- val resolver = conf.resolver
- val normalizedPartitions = partitioning.map {
- case transform: RewritableTransform =>
- val rewritten = transform.references().map { ref =>
- // Throws an exception if the reference cannot be resolved
- val position = SchemaUtils
- .findColumnPosition(ref.fieldNames().toImmutableArraySeq,
schema, resolver)
- FieldReference(SchemaUtils.getColumnName(position, schema))
- }
- transform.withReferences(rewritten.toImmutableArraySeq)
- case other => other
- }
-
- // Unlike the partitioning, an ordering reference to a missing column,
including a path
- // through a non-struct field, is kept as is for CheckAnalysis to
report, and each
- // reference is normalized independently.
- def normalizeResolvable(ref: NamedReference): NamedReference = {
- WriteDistributionAndOrdering
- .findColumn(schema, ref.fieldNames().toImmutableArraySeq, resolver)
- .map { case (path, field) => FieldReference(path :+ field.name) }
- .getOrElse(ref)
- }
-
- val normalizedOrdering = create.writeOrdering.map { sortOrder =>
- val key = sortOrder.expression() match {
- case ref: NamedReference => normalizeResolvable(ref)
- case transform: RewritableTransform =>
- transform.withReferences(
-
transform.references().map(normalizeResolvable).toImmutableArraySeq)
- case other => other
- }
- LogicalExpressions.sort(key, sortOrder.direction(),
sortOrder.nullOrdering())
- }
-
-
create.withPartitioning(normalizedPartitions).withWriteOrdering(normalizedOrdering)
- }
+ CreateTableHelper.preprocessV2CreateTablePlan(create)
}
private def fallBackV2ToV1(cls: Class[_]): Class[_] =
@@ -390,102 +302,6 @@ case class PreprocessTableCreation(catalog:
SessionCatalog) extends Rule[Logical
case f: FileDataSourceV2 => f.fallbackFileFormat
case _ => cls
}
-
- private def normalizeCatalogTable(schema: StructType, table: CatalogTable):
CatalogTable = {
- SchemaUtils.checkSchemaColumnNameDuplication(
- schema,
- conf.caseSensitiveAnalysis)
- if (!conf.allowCollationsInMapKeys) {
- SchemaUtils.checkNoCollationsInMapKeys(schema)
- }
-
- val normalizedPartCols = normalizePartitionColumns(schema, table)
- val normalizedBucketSpec = normalizeBucketSpec(schema, table)
-
- normalizedBucketSpec.foreach { spec =>
- for (bucketCol <- spec.bucketColumnNames if
normalizedPartCols.contains(bucketCol)) {
- throw
QueryCompilationErrors.bucketingColumnCannotBePartOfPartitionColumnsError(
- bucketCol, normalizedPartCols)
- }
- for (sortCol <- spec.sortColumnNames if
normalizedPartCols.contains(sortCol)) {
- throw
QueryCompilationErrors.bucketSortingColumnCannotBePartOfPartitionColumnsError(
- sortCol, normalizedPartCols)
- }
- }
-
- val normalizedProperties = table.properties ++ table.clusterBySpec.map {
spec =>
- ClusterBySpec.toProperty(schema, spec, conf.resolver)
- }
-
- table.copy(partitionColumnNames = normalizedPartCols, bucketSpec =
normalizedBucketSpec,
- properties = normalizedProperties)
- }
-
- private def normalizePartitionColumns(schema: StructType, table:
CatalogTable): Seq[String] = {
- val normalizedPartitionCols = CatalogUtils.normalizePartCols(
- tableName = table.identifier.unquotedString,
- tableCols = schema.map(_.name),
- partCols = table.partitionColumnNames,
- resolver = conf.resolver)
-
- SchemaUtils.checkColumnNameDuplication(normalizedPartitionCols,
conf.resolver)
-
- if (schema.nonEmpty && normalizedPartitionCols.length == schema.length) {
- throw new AnalysisException(
- errorClass = "ALL_PARTITION_COLUMNS_NOT_ALLOWED",
- messageParameters = Map.empty)
- }
-
- val normalizedPartitionColsSet = HashSet(normalizedPartitionCols: _*)
- schema
- .filter(f => normalizedPartitionColsSet.contains(f.name))
- .foreach { field =>
- if (!PartitioningUtils.canPartitionOn(field.dataType)) {
- throw
QueryCompilationErrors.invalidPartitionColumnDataTypeError(field)
- }
- }
-
- normalizedPartitionCols
- }
-
- private def normalizeBucketSpec(schema: StructType, table: CatalogTable):
Option[BucketSpec] = {
- table.bucketSpec match {
- case Some(bucketSpec) =>
- val normalizedBucketSpec = CatalogUtils.normalizeBucketSpec(
- tableName = table.identifier.unquotedString,
- tableCols = schema.map(_.name),
- bucketSpec = bucketSpec,
- resolver = conf.resolver)
-
- SchemaUtils.checkColumnNameDuplication(
- normalizedBucketSpec.bucketColumnNames,
- conf.resolver)
- SchemaUtils.checkColumnNameDuplication(
- normalizedBucketSpec.sortColumnNames,
- conf.resolver)
-
-
normalizedBucketSpec.sortColumnNames.map(schema(_)).map(_.dataType).foreach {
- case dt if RowOrdering.isOrderable(dt) => // OK
- case other => failAnalysis(s"Cannot use ${other.catalogString} for
sorting column")
- }
-
- schema.filter(f =>
normalizedBucketSpec.bucketColumnNames.contains(f.name))
- .foreach { field =>
- if (!BucketingUtils.canBucketOn(field.dataType)) {
- throw
QueryCompilationErrors.invalidBucketColumnDataTypeError(field.dataType)
- }
- }
-
- Some(normalizedBucketSpec)
-
- case None => None
- }
- }
-
- private def failAnalysis(msg: String) = {
- throw new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_3072", messageParameters = Map("msg" ->
msg))
- }
}
/**
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
index 06284ca95f81..ab0b073322a9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
@@ -27,7 +27,7 @@ import org.mockito.invocation.InvocationOnMock
import org.apache.spark.SparkUnsupportedOperationException
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest,
Analyzer, AsOfVersion, EmptyFunctionRegistry, NoSuchTableException,
RelationCache, RelationResolution, ResolvedFieldName, ResolvedFieldPosition,
ResolvedIdentifier, ResolvedTable, ResolveSessionCatalog, TimeTravelSpec,
UnresolvedAttribute, UnresolvedFieldPosition, UnresolvedInlineTable,
UnresolvedPartitionSpec, UnresolvedRelation, UnresolvedSubqueryColumnAliases,
UnresolvedTable}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest,
Analyzer, AsOfVersion, CreateTableHelper, EmptyFunctionRegistry,
NoSuchTableException, RelationCache, RelationResolution, ResolvedFieldName,
ResolvedFieldPosition, ResolvedIdentifier, ResolvedTable,
ResolveSessionCatalog, TimeTravelSpec, UnresolvedAttribute,
UnresolvedFieldPosition, UnresolvedInlineTable, UnresolvedPartitionSpec,
UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
import org.apache.spark.sql.catalyst.catalog.{BucketSpec,
CatalogStorageFormat, CatalogTable, CatalogTableType, InMemoryCatalog,
SessionCatalog, TempVariableManager}
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Cast,
EqualTo, Expression, InSubquery, IntegerLiteral, ListQuery, Literal,
StringLiteral}
import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
@@ -333,6 +333,33 @@ class PlanResolutionSuite extends SharedSparkSession with
AnalysisTest {
)
}
+ test("create table preprocessing preserves table metadata while normalizing
column names") {
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+ val partition = StructField("Part", IntegerType)
+ val value = StructField("Value", IntegerType)
+ val table = CatalogTable(
+ identifier = TableIdentifier("table", Some("default")),
+ tableType = CatalogTableType.EXTERNAL,
+ storage = CatalogStorageFormat.empty.copy(locationUri = Some(new
URI("file:/table"))),
+ schema = StructType(Seq(partition, value)),
+ provider = Some("parquet"),
+ partitionColumnNames = Seq("part"),
+ bucketSpec = Some(BucketSpec(4, Seq("VALUE"), Seq("value"))),
+ properties = Map("createdBy" -> "user"),
+ comment = Some("table comment"),
+ createTime = 123L,
+ lastAccessTime = 456L)
+ val create = CreateTableV1(table, SaveMode.Ignore, None)
+
+ val normalized =
CreateTableHelper.normalizeCreateTableWithoutQuery(create)
+
+ assert(normalized == create.copy(tableDesc = table.copy(
+ schema = StructType(Seq(value, partition)),
+ partitionColumnNames = Seq("Part"),
+ bucketSpec = Some(BucketSpec(4, Seq("Value"), Seq("Value"))))))
+ }
+ }
+
test("create table - with partitioned by") {
val query = "CREATE TABLE my_tab(a INT comment 'test', b STRING) " +
"USING parquet PARTITIONED BY (a)"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]