This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 544639d70202 [SPARK-58933][SQL][FOLLOWUP] Keep InsertIntoStatement as
the parsed plan for static INSERT targets
544639d70202 is described below
commit 544639d70202f8be4eb4d27d1585e578bedaea52
Author: Wenchen Fan <[email protected]>
AuthorDate: Sat Oct 24 09:26:12 2026 +0000
[SPARK-58933][SQL][FOLLOWUP] Keep InsertIntoStatement as the parsed plan
for static INSERT targets
### What changes were proposed in this pull request?
This is a follow-up to #58378 that partially reverts it. The parser again
produces `InsertIntoStatement` for static and literal `IDENTIFIER('...')`
INSERT targets. The temporary `UnresolvedInsert` node is used only for a
non-literal `IDENTIFIER(expr)` target, as introduced by #58204. Parser-level
consumers (SQL statement classification, `parse_sql`, and pipeline flow
registration) handle both shapes again.
These parts of #58378 are kept:
* Early resolution of a dynamic INSERT target runs inside the analysis
planning tracker, and its failures are reported through
`QueryPlanningTracker.setAnalysisFailed`.
* The `parse_sql` CTE-shadow exemption stays limited to INSERT targets, so
CTE-shadowed DELETE, UPDATE, and MERGE targets are not reported as catalog
target tables. The exemption now applies to the `InsertIntoStatement` target,
so `WITH t AS (...) INSERT INTO t ...` still reports `t` as a target.
In addition, `CheckAnalysis` raises an internal error if an
`UnresolvedInsert` reaches it without being lowered to `InsertIntoStatement`.
This should be merged to master and branch-4.x, where #58378 landed.
branch-4.3 has #58204 but not #58378, so it already has the restored behavior.
### Why are the changes needed?
`InsertIntoStatement` is the parsed INSERT plan that existing parser-level
consumers, extensions, and plan-based tooling match on. Only a dynamic
`IDENTIFIER` target needs analyzer traversal of the target before the INSERT
can be lowered. Parsing every INSERT as `UnresolvedInsert` forces all of these
consumers to handle a new intermediate node for the common static case, without
any benefit for that case. Scoping the temporary node to dynamic targets keeps
the plan-shape change limite [...]
### Does this PR introduce _any_ user-facing change?
No change relative to a released Spark version. Within unreleased master
and branch-4.x, the parsed plan of a static-target INSERT (for example, in
`EXPLAIN EXTENDED`) shows `InsertIntoStatement` again, as in branch-4.3.
### How was this patch tested?
Updated the existing tests and golden files to the restored plan shape, and
added tests for the parsed shape of a literal `IDENTIFIER('...')` target and
for the `CheckAnalysis` invariant check. Then ran:
```bash
build/sbt -Phive \
"catalyst/testOnly org.apache.spark.sql.catalyst.parser.PlanParserSuite
org.apache.spark.sql.catalyst.parser.DDLParserSuite
org.apache.spark.sql.catalyst.parser.IdentifierClauseParserSuite
org.apache.spark.sql.catalyst.analysis.AnalysisSuite" \
"sql/testOnly org.apache.spark.sql.ParametersSuite
org.apache.spark.sql.catalyst.parser.ParseSqlResultSuite
org.apache.spark.sql.execution.QueryExecutionSuite
org.apache.spark.sql.execution.command.v2.CreateFlowCommandSuite
org.apache.spark.sql.connector.AppendDataTransactionSuite" \
"sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z explain.sql -z
explain-aqe.sql -z identifier-clause.sql" \
"pipelines/testOnly org.apache.spark.sql.pipelines.graph.SqlPipelineSuite
org.apache.spark.sql.pipelines.graph.SqlQueryOriginSuite"
```
All tests passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59750 from cloud-fan/wenchen/spark-58933-narrow-parsed-insert.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit cecbd5d846a2cdd891a4d9f0bda302b58b02092a)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../spark/sql/catalyst/analysis/Analyzer.scala | 17 ++---
.../sql/catalyst/analysis/CheckAnalysis.scala | 6 ++
.../spark/sql/catalyst/analysis/unresolved.scala | 13 +++-
.../spark/sql/catalyst/parser/AstBuilder.scala | 39 +++++++-----
.../sql/catalyst/plans/logical/statements.scala | 9 ++-
.../sql/catalyst/analysis/AnalysisSuite.scala | 17 +++++
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 72 ++++++++++++----------
.../parser/IdentifierClauseParserSuite.scala | 6 +-
.../sql/catalyst/parser/PlanParserSuite.scala | 30 ++++-----
.../spark/sql/catalyst/parser/ParseSqlResult.scala | 8 ++-
.../sql/catalyst/parser/SqlStatementCodes.scala | 2 +-
.../sql-tests/analyzer-results/explain-aqe.sql.out | 2 +-
.../sql-tests/analyzer-results/explain.sql.out | 2 +-
.../sql-tests/results/explain-aqe.sql.out | 3 +-
.../resources/sql-tests/results/explain.sql.out | 3 +-
.../org/apache/spark/sql/ParametersSuite.scala | 19 +++---
.../command/v2/CreateFlowCommandSuite.scala | 12 ++--
.../graph/SqlGraphRegistrationContext.scala | 24 ++++----
18 files changed, 165 insertions(+), 119 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 8c5237adba9b..4f295db0a547 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -427,16 +427,7 @@ class Analyzer(
def getRelationResolution: RelationResolution = relationResolution
- private def toUnresolvedRelation(target: UnresolvedInsertTarget):
UnresolvedRelation = {
- val relation = withOrigin(target.origin) {
- UnresolvedRelation(target.multipartIdentifier, target.options)
- .requireWritePrivileges(target.writePrivileges)
- }
- relation.copyTagsFrom(target)
- relation
- }
-
- /** Lowers a parsed INSERT enough for transaction detection to inspect its
target. */
+ /** Resolves only the dynamic identifier of an INSERT so transaction
detection can inspect it. */
private[sql] def resolveUnresolvedInsert(insert: UnresolvedInsert):
LogicalPlan = {
runWithSessionConf {
val identifierResolvedTarget = insert.table match {
@@ -446,7 +437,7 @@ class Analyzer(
withOrigin(insert.origin) {
identifierResolvedTarget match {
case target: UnresolvedInsertTarget =>
- insert.toInsertIntoStatement(toUnresolvedRelation(target))
+ insert.toInsertIntoStatement(target.toUnresolvedRelation)
case target if !target.fastEquals(insert.table) =>
val updated = insert.copy(table = target)
updated.copyTagsFrom(insert)
@@ -1466,12 +1457,12 @@ class Analyzer(
}
}
- /** Lower a parsed INSERT as soon as its target identifier expression has
been evaluated. */
+ /** Lower an INSERT as soon as its target identifier expression has been
evaluated. */
object ResolveUnresolvedInsert extends Rule[LogicalPlan] {
override def apply(plan: LogicalPlan): LogicalPlan =
plan.resolveOperatorsUpWithPruning(
_.containsPattern(UNRESOLVED_INSERT), ruleId) {
case insert @ UnresolvedInsert(target: UnresolvedInsertTarget, _, _, _,
_, _, _, _, _) =>
- insert.toInsertIntoStatement(toUnresolvedRelation(target))
+ insert.toInsertIntoStatement(target.toUnresolvedRelation)
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index a6b3185beda6..99440b42a508 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -525,6 +525,12 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
messageParameters = Map("name" -> "IDENTIFIER", "expr" ->
p.identifierExpr.sql)
)
+ // Invariant check: the temporary UnresolvedInsert is lowered to
InsertIntoStatement once
+ // its target identifier is evaluated, so it must never survive analysis.
+ case insert: UnresolvedInsert =>
+ throw SparkException.internalError(
+ s"UnresolvedInsert target was not lowered during analysis:
${insert.table.nodeName}")
+
case c: CacheTableAsSelect if c.tempViewName.resolved =>
// The parser builds `tempViewName` as either a `Literal[StringType]`
(for direct
// identifiers and `IDENTIFIER('literal')`) or an
`ExpressionWithUnresolvedIdentifier`
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
index f3cb4653040c..189db237262e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
@@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
ExprCode}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.catalyst.plans.logical.{DeduplicateKeySpec,
LeafNode, LogicalPlan, SupportsSubquery, UnaryNode}
+import org.apache.spark.sql.catalyst.trees.CurrentOrigin
import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
@@ -131,7 +132,17 @@ case class ExpressionWithUnresolvedIdentifier(
case class UnresolvedInsertTarget(
multipartIdentifier: Seq[String],
options: CaseInsensitiveStringMap,
- writePrivileges: Set[TableWritePrivilege]) extends UnresolvedLeafNode
+ writePrivileges: Set[TableWritePrivilege]) extends UnresolvedLeafNode {
+
+ /** Converts this target to an [[UnresolvedRelation]] that keeps its origin
and tags. */
+ def toUnresolvedRelation: UnresolvedRelation = {
+ val relation = CurrentOrigin.withOrigin(origin) {
+ UnresolvedRelation(multipartIdentifier,
options).requireWritePrivileges(writePrivileges)
+ }
+ relation.copyTagsFrom(this)
+ relation
+ }
+}
/**
* Holds the name of a relation that has yet to be looked up in a catalog.
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 573ec8ede4d1..721403a91df3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -942,7 +942,7 @@ class AstBuilder extends DataTypeAstBuilder
ctx match {
case table: InsertIntoTableContext =>
val insertParams = visitInsertIntoTable(table)
- createUnresolvedInsert(
+ createInsertIntoStatement(
insertParams = insertParams,
query = query,
overwrite = false,
@@ -950,7 +950,7 @@ class AstBuilder extends DataTypeAstBuilder
writePrivileges = Set(TableWritePrivilege.INSERT))
case table: InsertOverwriteTableContext =>
val insertParams = visitInsertOverwriteTable(table)
- createUnresolvedInsert(
+ createInsertIntoStatement(
insertParams = insertParams,
query = query,
overwrite = true,
@@ -966,7 +966,7 @@ class AstBuilder extends DataTypeAstBuilder
val isInsertReplaceWhere = ctx.WHERE() != null
if (isInsertReplaceWhere) {
val insertParams = visitInsertIntoReplaceWhere(ctx)
- createUnresolvedInsert(
+ createInsertIntoStatement(
insertParams = insertParams,
query = query,
overwrite = true,
@@ -983,7 +983,7 @@ class AstBuilder extends DataTypeAstBuilder
}
}.getOrElse(query)
}
- createUnresolvedInsert(
+ createInsertIntoStatement(
insertParams = insertParams,
query = finalQuery,
overwrite = true,
@@ -992,7 +992,7 @@ class AstBuilder extends DataTypeAstBuilder
}
case ctx: InsertIntoReplaceUsingContext =>
val insertParams = visitInsertIntoReplaceUsing(ctx)
- createUnresolvedInsert(
+ createInsertIntoStatement(
insertParams = insertParams,
query = query,
overwrite = true,
@@ -1157,16 +1157,21 @@ class AstBuilder extends DataTypeAstBuilder
replaceCriteriaOpt = replaceCriteriaOpt)
}
- /** Creates an unresolved INSERT plan from [[InsertTableParams]]. */
- private def createUnresolvedInsert(
+ /**
+ * Creates an INSERT plan from [[InsertTableParams]]. A dynamic target is
exposed only by the
+ * temporary `UnresolvedInsert` plan. Once its identifier expression has
been evaluated, analysis
+ * continues with `InsertIntoStatement`, which a static target produces
directly.
+ */
+ private def createInsertIntoStatement(
insertParams: InsertTableParams,
query: LogicalPlan,
overwrite: Boolean,
withSchemaEvolution: Boolean,
- writePrivileges: Set[TableWritePrivilege]): UnresolvedInsert = {
- UnresolvedInsert(
- table = buildWriteTableSlot(
- insertParams.relationCtx, insertParams.options, writePrivileges),
+ writePrivileges: Set[TableWritePrivilege]): LogicalPlan = {
+ val tableSlot = buildWriteTableSlot(
+ insertParams.relationCtx, insertParams.options, writePrivileges)
+ val unresolvedInsert = UnresolvedInsert(
+ table = tableSlot,
partitionSpec = insertParams.partitionSpec,
userSpecifiedCols = insertParams.userSpecifiedCols,
query = query,
@@ -1175,12 +1180,18 @@ class AstBuilder extends DataTypeAstBuilder
byName = insertParams.byName,
replaceCriteriaOpt = insertParams.replaceCriteriaOpt,
withSchemaEvolution = withSchemaEvolution)
+
+ tableSlot match {
+ case target: UnresolvedInsertTarget =>
+ unresolvedInsert.toInsertIntoStatement(target.toUnresolvedRelation)
+ case _ => unresolvedInsert
+ }
}
/**
- * Build the target of an INSERT. An `IDENTIFIER` expression receives normal
analyzer traversal.
- * A resolved identifier is kept in `UnresolvedInsertTarget` rather than
`UnresolvedRelation` so
- * CTE substitution cannot interpret the target as a source.
+ * Build the target of an INSERT. A dynamic IDENTIFIER remains inside
`UnresolvedInsert` so its
+ * expression receives normal analyzer traversal. The builder produces
`UnresolvedInsertTarget`
+ * rather than `UnresolvedRelation` so CTE substitution cannot interpret the
target as a source.
*/
private def buildWriteTableSlot(
ctx: IdentifierReferenceContext,
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
index 85327eccae22..75a06a39b841 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statements.scala
@@ -158,12 +158,11 @@ case class QualifiedColType(
}
/**
- * An INSERT statement as parsed from SQL.
+ * A temporary INSERT plan whose target identifier expression still needs
analyzer traversal.
*
- * The target is exposed as a child until the analyzer converts this node to
the unary
- * [[InsertIntoStatement]]. A static target is represented by
`UnresolvedInsertTarget`, while a
- * dynamic target keeps its identifier expression in
`PlanWithUnresolvedIdentifier` until that
- * expression is resolved.
+ * The target is exposed as a child only until it becomes an
`UnresolvedInsertTarget`. The analyzer
+ * then converts this node to the unary [[InsertIntoStatement]], before
resolving the target as a
+ * relation.
*/
case class UnresolvedInsert(
table: LogicalPlan,
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
index 40696da40cc5..0ddee98ea441 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/AnalysisSuite.scala
@@ -970,6 +970,23 @@ class AnalysisSuite extends AnalysisTest with Matchers {
Array(ExpectedContext("test", 12, 15)))
}
+ test("UnresolvedInsert that is not lowered during analysis is an internal
error") {
+ // The parser never builds a resolved target here. It cannot be lowered to
+ // InsertIntoStatement, so the temporary INSERT plan would remain after
analysis.
+ val insert = UnresolvedInsert(
+ table = testRelation,
+ partitionSpec = Map.empty,
+ userSpecifiedCols = Nil,
+ query = testRelation2,
+ overwrite = false,
+ ifPartitionNotExists = false)
+ checkError(
+ exception = intercept[SparkException](insert.analyze),
+ condition = "INTERNAL_ERROR",
+ parameters = Map(
+ "message" -> "UnresolvedInsert target was not lowered during analysis:
LocalRelation"))
+ }
+
test("check CollectMetrics resolved") {
val a = testRelation.output.head
val sum = Sum(a).toAggregateExpression().as("sum")
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
index d32d5d45a091..432f05ebb971 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@@ -46,7 +46,11 @@ class DDLParserSuite extends AnalysisTest {
// We don't care the write privileges in this suite.
val parsed = parsePlan(sql).transform {
case u: UnresolvedRelation => u.clearWritePrivileges
- case u: UnresolvedInsertTarget =>
UnresolvedRelation(u.multipartIdentifier, u.options)
+ case i: InsertIntoStatement =>
+ i.table match {
+ case u: UnresolvedRelation => i.copy(table = u.clearWritePrivileges)
+ case _ => i
+ }
case o: OverwriteByExpression =>
o.table match {
case u: UnresolvedRelation => o.copy(table = u.clearWritePrivileges)
@@ -1625,7 +1629,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT INTO testcat.ns1.ns2.tbl SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Nil,
@@ -1640,7 +1644,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT INTO testcat.ns1.ns2.tbl (a, b) SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Seq("a", "b"),
@@ -1651,7 +1655,7 @@ class DDLParserSuite extends AnalysisTest {
test("insert table: append from another catalog") {
parseCompare("INSERT INTO TABLE testcat.ns1.ns2.tbl SELECT * FROM
testcat2.db.tbl",
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Nil,
@@ -1666,7 +1670,7 @@ class DDLParserSuite extends AnalysisTest {
|PARTITION (p1 = 3, p2)
|SELECT * FROM source
""".stripMargin,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map("p1" -> Some("3"), "p2" -> None),
Nil,
@@ -1681,7 +1685,7 @@ class DDLParserSuite extends AnalysisTest {
|PARTITION (p1 = 3, p2) (a, b)
|SELECT * FROM source
""".stripMargin,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map("p1" -> Some("3"), "p2" -> None),
Seq("a", "b"),
@@ -1695,7 +1699,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT OVERWRITE testcat.ns1.ns2.tbl SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Nil,
@@ -1710,7 +1714,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT OVERWRITE testcat.ns1.ns2.tbl (a, b) SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Seq("a", "b"),
@@ -1726,7 +1730,7 @@ class DDLParserSuite extends AnalysisTest {
|PARTITION (p1 = 3, p2)
|SELECT * FROM source
""".stripMargin,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map("p1" -> Some("3"), "p2" -> None),
Nil,
@@ -1741,7 +1745,7 @@ class DDLParserSuite extends AnalysisTest {
|PARTITION (p1 = 3, p2) (a, b)
|SELECT * FROM source
""".stripMargin,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map("p1" -> Some("3"), "p2" -> None),
Seq("a", "b"),
@@ -1756,7 +1760,7 @@ class DDLParserSuite extends AnalysisTest {
|PARTITION (p1 = 3) IF NOT EXISTS
|SELECT * FROM source
""".stripMargin,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map("p1" -> Some("3")),
Nil,
@@ -1806,7 +1810,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT INTO testcat.ns1.ns2.tbl BY NAME SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Nil,
@@ -1819,7 +1823,7 @@ class DDLParserSuite extends AnalysisTest {
"INSERT OVERWRITE testcat.ns1.ns2.tbl BY NAME SELECT * FROM source"
).foreach { sql =>
parseCompare(sql,
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
Map.empty,
Nil,
@@ -1842,7 +1846,7 @@ class DDLParserSuite extends AnalysisTest {
test("insert table: REPLACE WHERE with BY NAME") {
parseCompare(
"INSERT INTO testcat.ns1.ns2.tbl BY NAME REPLACE WHERE a > 5 SELECT *
FROM source",
- UnresolvedInsert(
+ InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Nil,
@@ -1857,7 +1861,7 @@ class DDLParserSuite extends AnalysisTest {
test("insert table: REPLACE WHERE without BY NAME") {
parseCompare(
"INSERT INTO testcat.ns1.ns2.tbl REPLACE WHERE a > 5 SELECT * FROM
source",
- UnresolvedInsert(
+ InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Nil,
@@ -1906,7 +1910,7 @@ class DDLParserSuite extends AnalysisTest {
test("insert table: REPLACE WHERE with column list") {
parseCompare(
"INSERT INTO testcat.ns1.ns2.tbl (a, b) REPLACE WHERE a > 5 SELECT *
FROM source",
- UnresolvedInsert(
+ InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq("a", "b"),
@@ -1957,7 +1961,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
"INSERT WITH SCHEMA EVOLUTION INTO testcat.ns1.ns2.tbl (a, b) " +
"REPLACE WHERE a > 5 SELECT * FROM source",
- UnresolvedInsert(
+ InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq("a", "b"),
@@ -1974,7 +1978,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
"INSERT INTO testcat.ns1.ns2.tbl WITH (key = 'value') (a, b) " +
"REPLACE WHERE a > 5 SELECT * FROM source",
- UnresolvedInsert(
+ InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl"), opts),
partitionSpec = Map.empty,
userSpecifiedCols = Seq("a", "b"),
@@ -2006,7 +2010,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = userSpecifiedCols,
@@ -2027,7 +2031,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = userSpecifiedCols,
@@ -2055,7 +2059,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2081,7 +2085,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2101,7 +2105,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2123,7 +2127,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2143,7 +2147,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = insertSQLStmt,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2163,7 +2167,7 @@ class DDLParserSuite extends AnalysisTest {
val table = "testcat.ns1.ns2.tbl"
parseCompare(
sql = s"INSERT INTO $table REPLACE ON col1 = col2 SELECT * FROM source",
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2197,7 +2201,7 @@ class DDLParserSuite extends AnalysisTest {
s"""INSERT INTO $table AS t
|REPLACE ON (t.a = s.b)
|(SELECT * FROM source) AS s""".stripMargin,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2224,7 +2228,7 @@ class DDLParserSuite extends AnalysisTest {
s"""INSERT INTO $table AS t
|REPLACE USING (col1, col2)
|(SELECT * FROM source) AS s""".stripMargin,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2244,7 +2248,7 @@ class DDLParserSuite extends AnalysisTest {
"""INSERT INTO testcat.ns1.ns2.tbl
|REPLACE WHERE a > 5
|(SELECT * FROM source) AS s""".stripMargin,
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Nil,
@@ -2260,7 +2264,7 @@ class DDLParserSuite extends AnalysisTest {
parseCompare(
sql = s"INSERT INTO $table AS t REPLACE ON t.a = s.a AND t.b = s.b " +
"(SELECT * FROM source) AS s",
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -2285,7 +2289,7 @@ class DDLParserSuite extends AnalysisTest {
val table = "testcat.ns1.ns2.tbl"
parseCompare(
sql = s"INSERT INTO $table AS t REPLACE USING (id) SELECT * FROM source",
- expected = UnresolvedInsert(
+ expected = InsertIntoStatement(
table = UnresolvedRelation(Seq("testcat", "ns1", "ns2", "tbl")),
partitionSpec = Map.empty,
userSpecifiedCols = Seq.empty,
@@ -3380,8 +3384,8 @@ class DDLParserSuite extends AnalysisTest {
}
test("SPARK-33474: Support typed literals as partition spec values") {
- def insertPartitionPlan(part: String, optimizeInsertIntoCmds: Boolean):
UnresolvedInsert = {
- UnresolvedInsert(
+ def insertPartitionPlan(part: String, optimizeInsertIntoCmds: Boolean):
InsertIntoStatement = {
+ InsertIntoStatement(
UnresolvedRelation(Seq("t")),
Map("part" -> Some(part)),
Seq.empty[String],
@@ -3532,7 +3536,7 @@ class DDLParserSuite extends AnalysisTest {
UnresolvedAttribute("DEFAULT"))))))
parseCompare(
"INSERT INTO t PARTITION(part = date'2019-01-02') VALUES ('a', DEFAULT)",
- UnresolvedInsert(
+ InsertIntoStatement(
UnresolvedRelation(Seq("t")),
Map("part" -> Some("2019-01-02")),
userSpecifiedCols = Seq.empty[String],
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/IdentifierClauseParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/IdentifierClauseParserSuite.scala
index fd0f88cf32e1..546065770fdc 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/IdentifierClauseParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/IdentifierClauseParserSuite.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql.catalyst.parser
import org.apache.spark.sql.catalyst.analysis.{AnalysisTest,
ExpressionWithUnresolvedIdentifier, UnresolvedAttribute,
UnresolvedExtractValue, UnresolvedFunction, UnresolvedInlineTable,
UnresolvedStar}
import org.apache.spark.sql.catalyst.expressions.{Alias, Cast, LambdaFunction,
Literal, UnresolvedNamedLambdaVariable}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
-import org.apache.spark.sql.catalyst.plans.logical.{OneRowRelation, Pivot,
Project, SubqueryAlias, Unpivot, UnresolvedInsert}
+import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement,
OneRowRelation, Pivot, Project, SubqueryAlias, Unpivot}
import org.apache.spark.sql.catalyst.util.EvaluateUnresolvedInlineTable
import org.apache.spark.sql.types.{IntegerType, StringType, StructField,
StructType}
@@ -179,13 +179,13 @@ class IdentifierClauseParserSuite extends AnalysisTest {
test("Partition spec with IDENTIFIER() for partition column name") {
val plan = parsePlan(
"INSERT INTO partition_spec_test PARTITION (IDENTIFIER('c2') = 'value1')
VALUES (1)")
- .asInstanceOf[UnresolvedInsert]
+ .asInstanceOf[InsertIntoStatement]
val values = EvaluateUnresolvedInlineTable.evaluate(
UnresolvedInlineTable(Seq("col1"), Seq(Seq(Literal(1)))))
comparePlans(
plan,
- UnresolvedInsert(
+ InsertIntoStatement(
plan.table,
Map("c2" -> Some("value1")),
Nil,
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
index b9104a9af403..7be554b89c85 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/PlanParserSuite.scala
@@ -22,7 +22,7 @@ import scala.annotation.nowarn
import org.apache.spark.SparkThrowable
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, RelationChanges,
RelationTimeTravel, UnresolvedAlias, UnresolvedAttribute, UnresolvedFunction,
UnresolvedGenerator, UnresolvedInlineTable, UnresolvedInsertTarget,
UnresolvedOrdinal, UnresolvedPipeAggregateOrdinal, UnresolvedRelation,
UnresolvedStar, UnresolvedSubqueryColumnAliases, UnresolvedTableValuedFunction,
UnresolvedTVFAliases}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, RelationChanges,
RelationTimeTravel, UnresolvedAlias, UnresolvedAttribute, UnresolvedFunction,
UnresolvedGenerator, UnresolvedInlineTable, UnresolvedOrdinal,
UnresolvedPipeAggregateOrdinal, UnresolvedRelation, UnresolvedStar,
UnresolvedSubqueryColumnAliases, UnresolvedTableValuedFunction,
UnresolvedTVFAliases}
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
@@ -47,7 +47,11 @@ class PlanParserSuite extends AnalysisTest {
// We don't care the write privileges in this suite.
val parsed = parsePlan(sqlCommand).transform {
case u: UnresolvedRelation => u.clearWritePrivileges
- case u: UnresolvedInsertTarget =>
UnresolvedRelation(u.multipartIdentifier, u.options)
+ case i: InsertIntoStatement =>
+ i.table match {
+ case u: UnresolvedRelation => i.copy(table = u.clearWritePrivileges)
+ case _ => i
+ }
}
comparePlans(parsed, plan, checkAnalysis = false)
}
@@ -56,11 +60,6 @@ class PlanParserSuite extends AnalysisTest {
super.parseException(parsePlan)(sqlText)
}
- private def unresolvedInsertInto(tableName: String, query: LogicalPlan):
LogicalPlan = {
- UnresolvedInsert(
- table(tableName), Map.empty, Nil, query, overwrite = false,
ifPartitionNotExists = false)
- }
-
private def cte(
plan: LogicalPlan,
allowRecursion: Boolean,
@@ -386,8 +385,8 @@ class PlanParserSuite extends AnalysisTest {
parameters = Map("error" -> "'from'", "hint" -> ""))
assertEqual(
"from a insert into tbl1 select * insert into tbl2 select * where s <
10",
- unresolvedInsertInto("tbl1", table("a").select(star())).union(
- unresolvedInsertInto("tbl2", table("a").where($"s" <
10).select(star()))))
+ table("a").select(star()).insertInto("tbl1").union(
+ table("a").where($"s" < 10).select(star()).insertInto("tbl2")))
assertEqual(
"select * from (from a select * select *)",
table("a").select(star())
@@ -485,7 +484,7 @@ class PlanParserSuite extends AnalysisTest {
partition: Map[String, Option[String]],
overwrite: Boolean = false,
ifPartitionNotExists: Boolean = false): LogicalPlan =
- UnresolvedInsert(table("s"), partition, Nil, plan, overwrite,
ifPartitionNotExists)
+ InsertIntoStatement(table("s"), partition, Nil, plan, overwrite,
ifPartitionNotExists)
// Single inserts
assertEqual(s"insert overwrite table s $sql",
@@ -500,7 +499,7 @@ class PlanParserSuite extends AnalysisTest {
// Multi insert
val plan2 = table("t").where($"x" > 5).select(star())
assertEqual("from t insert into s select * limit 1 insert into u select *
where x > 5",
- unresolvedInsertInto("s", plan.limit(1)).union(unresolvedInsertInto("u",
plan2)))
+ plan.limit(1).insertInto("s").union(plan2.insertInto("u")))
}
test("aggregation") {
@@ -678,10 +677,11 @@ class PlanParserSuite extends AnalysisTest {
|select *
|where s < 10
""".stripMargin,
- Union(unresolvedInsertInto("t2", from
+ Union(from
.generate(jsonTuple, alias = Some("jtup"), outputNames = Seq("q", "z"))
- .select(star())),
- unresolvedInsertInto("t3", from.where($"s" < 10).select(star()))))
+ .select(star())
+ .insertInto("t2"),
+ from.where($"s" < 10).select(star()).insertInto("t3")))
// Unresolved generator.
val expected = table("t")
@@ -1934,7 +1934,7 @@ class PlanParserSuite extends AnalysisTest {
assertEqual(
"INSERT INTO s SELECT /*+ REPARTITION(100), COALESCE(500), COALESCE(10)
*/ * FROM t",
- UnresolvedInsert(table("s"), Map.empty, Nil,
+ InsertIntoStatement(table("s"), Map.empty, Nil,
UnresolvedHint("REPARTITION", Seq(Literal(100)),
UnresolvedHint("COALESCE", Seq(Literal(500)),
UnresolvedHint("COALESCE", Seq(Literal(10)),
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
index ab1cf6c55570..3155fca0fd50 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/ParseSqlResult.scala
@@ -245,6 +245,8 @@ object ParseSqlResult {
case i: UnresolvedInsert =>
visitPlan(i.table, scope, TableRefRole.Target)(f)
visitPlan(i.query, nextScope, TableRefRole.Source)(f)
+ case i: InsertIntoStatement =>
+ visitPlan(i.query, nextScope, TableRefRole.Source)(f)
case c: CreateTableAsSelect =>
visitPlan(c.name, scope, TableRefRole.Target)(f)
visitPlan(c.query, nextScope, TableRefRole.Source)(f)
@@ -288,6 +290,9 @@ object ParseSqlResult {
visitPlan(cteRelation.plan, bodyScope, TableRefRole.Source)(f)
definitionScope += normalized
}
+ // An INSERT target is never resolved against a CTE, so no alias shadows
it.
+ case i: InsertIntoStatement =>
+ visitPlan(i.table, Set.empty, TableRefRole.Target)(f)
case c: CacheTable if c.multipartIdentifier.isEmpty =>
visitPlan(c.table, scope, TableRefRole.Target)(f)
case c: CompoundBody =>
@@ -360,7 +365,6 @@ object ParseSqlResult {
foreachExpressionDeep(p)(visitExpr)
def add(parts: Seq[String]): Unit = addTable(parts, scope, role)
p match {
- case u: UnresolvedInsertTarget => addTarget(u.multipartIdentifier)
case u: UnresolvedRelation => add(u.multipartIdentifier)
case u: UnresolvedTable => add(u.multipartIdentifier)
case u: UnresolvedView => add(u.multipartIdentifier)
@@ -407,6 +411,8 @@ object ParseSqlResult {
case UnresolvedWith(child, _, _) => primaryQueryPlan(child)
case UnresolvedInsert(_, _, _, query, _, _, _, _, _) =>
primaryQueryPlan(query)
+ case InsertIntoStatement(_, _, _, query, _, _, _, _, _) =>
+ primaryQueryPlan(query)
case c: CreateTableAsSelect => primaryQueryPlan(c.query)
case r: ReplaceTableAsSelect => primaryQueryPlan(r.query)
case c: CreateView => primaryQueryPlan(c.query)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementCodes.scala
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementCodes.scala
index 1a350baa60b4..24cfdfcd2eb4 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementCodes.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/parser/SqlStatementCodes.scala
@@ -132,7 +132,7 @@ object SqlStatementCodes {
def classify(plan: LogicalPlan): SqlStatementClassification = plan match {
case UnresolvedWith(child, _, _) => classify(child)
case _: CompoundBody => BeginEnd
- case _: UnresolvedInsert => Insert
+ case _: InsertIntoStatement | _: UnresolvedInsert => Insert
case _: DeleteFromTable | _: DeleteFromTableWithFilters => DeleteWhere
case _: UpdateTable => UpdateWhere
case _: MergeIntoTable => Merge
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out
index d5295294cdcb..d686b659fdb5 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/explain-aqe.sql.out
@@ -223,7 +223,7 @@ ExplainCommand 'Aggregate ['key], ['key,
unresolvedalias('MIN('val))], Formatted
-- !query
EXPLAIN EXTENDED INSERT INTO TABLE explain_temp5 SELECT * FROM explain_temp4
-- !query analysis
-ExplainCommand 'UnresolvedInsert false, false, false, false, ExtendedMode
+ExplainCommand 'InsertIntoStatement 'UnresolvedRelation [explain_temp5],
[__required_write_privileges__=INSERT], false, false, false, false, false,
ExtendedMode
-- !query
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out
index d5295294cdcb..d686b659fdb5 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/explain.sql.out
@@ -223,7 +223,7 @@ ExplainCommand 'Aggregate ['key], ['key,
unresolvedalias('MIN('val))], Formatted
-- !query
EXPLAIN EXTENDED INSERT INTO TABLE explain_temp5 SELECT * FROM explain_temp4
-- !query analysis
-ExplainCommand 'UnresolvedInsert false, false, false, false, ExtendedMode
+ExplainCommand 'InsertIntoStatement 'UnresolvedRelation [explain_temp5],
[__required_write_privileges__=INSERT], false, false, false, false, false,
ExtendedMode
-- !query
diff --git a/sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
b/sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
index ef54219c18d3..e33e2df78443 100644
--- a/sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out
@@ -1302,8 +1302,7 @@ EXPLAIN EXTENDED INSERT INTO TABLE explain_temp5 SELECT *
FROM explain_temp4
struct<plan:string>
-- !query output
== Parsed Logical Plan ==
-'UnresolvedInsert false, false, false, false
-:- 'UnresolvedInsertTarget [explain_temp5], [], {INSERT}
+'InsertIntoStatement 'UnresolvedRelation [explain_temp5],
[__required_write_privileges__=INSERT], false, false, false, false, false
+- 'Project [*]
+- 'UnresolvedRelation [explain_temp4], [], false
diff --git a/sql/core/src/test/resources/sql-tests/results/explain.sql.out
b/sql/core/src/test/resources/sql-tests/results/explain.sql.out
index 753d662dcdd1..a7b59e6ad85d 100644
--- a/sql/core/src/test/resources/sql-tests/results/explain.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/explain.sql.out
@@ -1158,8 +1158,7 @@ EXPLAIN EXTENDED INSERT INTO TABLE explain_temp5 SELECT *
FROM explain_temp4
struct<plan:string>
-- !query output
== Parsed Logical Plan ==
-'UnresolvedInsert false, false, false, false
-:- 'UnresolvedInsertTarget [explain_temp5], [], {INSERT}
+'InsertIntoStatement 'UnresolvedRelation [explain_temp5],
[__required_write_privileges__=INSERT], false, false, false, false, false
+- 'Project [*]
+- 'UnresolvedRelation [explain_temp4], [], false
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
index ffe685343375..a9fff5581974 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
@@ -20,7 +20,7 @@ package org.apache.spark.sql
import java.time.{Instant, LocalDate, LocalDateTime, ZoneId}
import org.apache.spark.sql.catalyst.ExtendedAnalysisException
-import org.apache.spark.sql.catalyst.analysis.{BindParameters,
CTESubstitution, ExpressionWithUnresolvedIdentifier, NameParameterizedQuery,
PlanWithUnresolvedIdentifier, UnresolvedInsertTarget, UnresolvedRelation}
+import org.apache.spark.sql.catalyst.analysis.{BindParameters,
CTESubstitution, ExpressionWithUnresolvedIdentifier, NameParameterizedQuery,
PlanWithUnresolvedIdentifier, UnresolvedRelation}
import org.apache.spark.sql.catalyst.expressions.Literal
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.catalyst.plans.logical.{CacheTableAsSelect,
CTEInChildren, InsertIntoStatement, Limit, ReplaceTableAsSelect,
UnresolvedInsert, WithCTE}
@@ -2566,11 +2566,11 @@ class ParametersSuite extends SharedSparkSession {
}
}
- // The unresolved binary node exposes the target to analyzer traversal and
keeps CTE definitions
- // on the input query.
+ // The temporary binary node exposes only a dynamic target to analyzer
traversal and keeps CTE
+ // definitions on the input query.
test("SPARK-46625: WITH ... INSERT INTO IDENTIFIER(:p) REPLACE WHERE ...
parser") {
// Use a non-literal-string expression so `withIdentClause` produces
- // `PlanWithUnresolvedIdentifier` rather than short-circuiting to
`UnresolvedInsertTarget`.
+ // `PlanWithUnresolvedIdentifier` rather than short-circuiting to a static
target.
val parsedPlan = spark.sessionState.sqlParser.parsePlan(
"""WITH transformation AS (SELECT 99 AS a)
|INSERT INTO IDENTIFIER('some' || '_table') REPLACE WHERE a = 10
@@ -2594,12 +2594,15 @@ class ParametersSuite extends SharedSparkSession {
}
}
- test("parsed INSERT targets use an unresolved binary plan") {
+ test("dynamic INSERT targets use a temporary binary plan") {
val regularInsert = spark.sessionState.sqlParser
.parsePlan("INSERT INTO target_table SELECT 1")
- .asInstanceOf[UnresolvedInsert]
- assert(regularInsert.table.isInstanceOf[UnresolvedInsertTarget])
- assert(regularInsert.children === Seq(regularInsert.table,
regularInsert.query))
+ assert(regularInsert.isInstanceOf[InsertIntoStatement])
+
+ // A literal IDENTIFIER is a static target, so it parses to the same plan
as a bare name.
+ val literalIdentifierInsert = spark.sessionState.sqlParser
+ .parsePlan("INSERT INTO IDENTIFIER('target_table') SELECT 1")
+ assert(literalIdentifierInsert === regularInsert)
val identifierInsert = spark.sessionState.sqlParser
.parsePlan("INSERT INTO IDENTIFIER(lower('TESTCAT.NS.TARGET_TABLE'))
SELECT 1")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CreateFlowCommandSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CreateFlowCommandSuite.scala
index cf079565b0c1..5298434cfe6f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CreateFlowCommandSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/CreateFlowCommandSuite.scala
@@ -17,15 +17,14 @@
package org.apache.spark.sql.execution.command.v2
-import org.apache.spark.sql.catalyst.analysis.{AnalysisTest,
UnresolvedIdentifier, UnresolvedInsertTarget}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest,
UnresolvedIdentifier, UnresolvedRelation}
import org.apache.spark.sql.catalyst.plans.logical.{
CreateFlowCommand,
- UnresolvedInsert
+ InsertIntoStatement
}
import org.apache.spark.sql.connector.catalog.TableWritePrivilege
import org.apache.spark.sql.execution.SparkSqlParser
import org.apache.spark.sql.execution.command.v1.CommandSuiteBase
-import org.apache.spark.sql.util.CaseInsensitiveStringMap
/**
* The class contains tests for the `CREATE FLOW ...` command
@@ -60,10 +59,9 @@ class CreateFlowCommandSuite extends CommandSuiteBase with
AnalysisTest {
val plan = parser.parsePlan("CREATE FLOW f AS INSERT OVERWRITE a PARTITION
(col1) BY NAME " +
"SELECT col1, col2 FROM b")
val cmd = plan.asInstanceOf[CreateFlowCommand]
- assert(cmd.right == UnresolvedInsert(
- table = UnresolvedInsertTarget(
- Seq("a"), CaseInsensitiveStringMap.empty(),
- Set(TableWritePrivilege.INSERT, TableWritePrivilege.DELETE)),
+ assert(cmd.right == InsertIntoStatement(
+ table = UnresolvedRelation(Seq("a"))
+ .requireWritePrivileges(Set(TableWritePrivilege.INSERT,
TableWritePrivilege.DELETE)),
partitionSpec = Map("col1" -> None),
userSpecifiedCols = Seq.empty,
query = parser.parsePlan("SELECT col1, col2 FROM b"),
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
index dbc58bd78d76..288cf9a0eb0e 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/SqlGraphRegistrationContext.scala
@@ -22,9 +22,9 @@ import org.apache.spark.{SparkException,
SparkRuntimeException}
import org.apache.spark.sql.{AnalysisException, SparkSession}
import org.apache.spark.sql.Column
import org.apache.spark.sql.catalyst.{QueryPlanningTracker, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute,
UnresolvedInsertTarget}
+import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute,
UnresolvedRelation}
import org.apache.spark.sql.catalyst.expressions.Expression
-import org.apache.spark.sql.catalyst.plans.logical.{AutoCdcInto,
CreateFlowCommand, CreateMaterializedViewAsSelect, CreateStreamingTable,
CreateStreamingTableAsSelect, CreateStreamingTableAutoCdc, CreateView,
LogicalPlan, UnresolvedInsert}
+import org.apache.spark.sql.catalyst.plans.logical.{AutoCdcInto,
CreateFlowCommand, CreateMaterializedViewAsSelect, CreateStreamingTable,
CreateStreamingTableAsSelect, CreateStreamingTableAutoCdc, CreateView,
InsertIntoStatement, LogicalPlan, UnresolvedInsert}
import org.apache.spark.sql.catalyst.util.StringUtils
import org.apache.spark.sql.classic.ClassicConversions._
import org.apache.spark.sql.execution.command.{CreateViewCommand,
SetCatalogCommand, SetCommand, SetNamespaceCommand}
@@ -624,8 +624,10 @@ class SqlGraphRegistrationContext(
.identifier
cf.flowOperation match {
- case i: UnresolvedInsert =>
+ case i: InsertIntoStatement =>
handleInsert(i, flowIdentifier, queryOrigin)
+ case i: UnresolvedInsert =>
+ handleInsert(i.toInsertIntoStatement(i.table), flowIdentifier,
queryOrigin)
case a: AutoCdcInto =>
val flowTargetDatasetName =
IdentifierHelper.toTableIdentifier(a.targetTable)
graphRegistrationContext.registerFlow(
@@ -664,12 +666,12 @@ class SqlGraphRegistrationContext(
}
private def handleInsert(
- insert: UnresolvedInsert,
+ insert: InsertIntoStatement,
flowIdentifier: TableIdentifier,
queryOrigin: QueryOrigin): Unit = {
validateInsertIntoFlow(insert, queryOrigin)
val flowTargetDatasetName = insert.table match {
- case u: UnresolvedInsertTarget =>
+ case u: UnresolvedRelation =>
IdentifierHelper.toTableIdentifier(u.multipartIdentifier)
case _ =>
throw SqlGraphElementRegistrationException(
@@ -705,34 +707,34 @@ class SqlGraphRegistrationContext(
.identifier
private def validateInsertIntoFlow(
- insert: UnresolvedInsert,
+ insertIntoStatement: InsertIntoStatement,
queryOrigin: QueryOrigin
): Unit = {
- if (insert.partitionSpec.nonEmpty) {
+ if (insertIntoStatement.partitionSpec.nonEmpty) {
throw SqlGraphElementRegistrationException(
msg = "Partition spec may not be specified for flow target.",
queryOrigin = queryOrigin
)
}
- if (insert.userSpecifiedCols.nonEmpty) {
+ if (insertIntoStatement.userSpecifiedCols.nonEmpty) {
throw SqlGraphElementRegistrationException(
msg = "Column schema may not be specified for flow target.",
queryOrigin = queryOrigin
)
}
- if (insert.overwrite) {
+ if (insertIntoStatement.overwrite) {
throw SqlGraphElementRegistrationException(
msg = "INSERT OVERWRITE flows not supported.",
queryOrigin = queryOrigin
)
}
- if (insert.ifPartitionNotExists) {
+ if (insertIntoStatement.ifPartitionNotExists) {
throw SqlGraphElementRegistrationException(
msg = "IF NOT EXISTS not supported for flows.",
queryOrigin = queryOrigin
)
}
- if (!insert.byName) {
+ if (!insertIntoStatement.byName) {
throw SqlGraphElementRegistrationException(
msg = "Only INSERT INTO by name flows supported.",
queryOrigin = queryOrigin
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]