This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 7f6ca2d017c2 [SPARK-58330][SQL] Prevent silent dropping of dynamic
options on same table references.
7f6ca2d017c2 is described below
commit 7f6ca2d017c23e50604847b74be597ebaff3a97a
Author: Anurag Mantripragada <[email protected]>
AuthorDate: Wed Jul 29 08:47:58 2026 -0700
[SPARK-58330][SQL] Prevent silent dropping of dynamic options on same table
references.
## What changes were proposed in this pull request?
This PR fixes dynamic table options being silently dropped when the same
table is referenced more than once in a single statement. Options are now
re-applied on a per-query relation cache hit, so each reference keeps its own
options.
## Why are the changes needed?
The per-query relation cache in RelationResolution is keyed by catalog,
namespace, name, and time travel spec, but not by options. When the same table
is referenced twice in one statement with different options, the second
reference gets a cache hit and reuses the first reference's options, so the
options the user wrote on the second reference are silently ignored. This
affects all queries that self reference a table.
## Does this PR introduce any user-facing change?
Yes. Options that were previously silently dropped on a second reference to
the same table now take effect, and options that previously leaked from one
reference onto another no longer do. This affects any statement with two or
more references to one table.
Added a note to `docs/sql-migration-guide.md` scoped to SELECT/INSERT since
those are the only two of the four statements that have shipped in a released
version. UPDATE/MERGE self-reference support hasn't reached a release yet, so
there's no upgrade path to document for those two.
## How was this patch tested?
Added end-to-end tests, each asserting that both references keep their own
options:
- `DataSourceV2OptionSuite`: INSERT selecting from the same table, a
self-join, and a streaming self-join and streaming CTE (both analysis-only, no
writeStream.start())
- `MergeIntoTableSuiteBase`: self-merge (target and source referencing the
same table)
- `UpdateTableSuiteBase`: a subquery and a CTE referencing the same table
as the update target
- `DDLSuite`: the same three shapes (self-join, INSERT self-select, CTE)
for v1 (session-catalog/Hive) tables
## Was this patch authored or co-authored using generative AI tooling?
I used Claude Code (Claude Opus 4.8) to generate the code and tests and
verified manually.
Closes #57508 from anuragmantri/self-ref-options.
Lead-authored-by: Anurag Mantripragada <[email protected]>
Co-authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit f27705db432c76119a09c7ef822927c2822eedc8)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../sql/catalyst/analysis/RelationResolution.scala | 21 ++++-
.../spark/sql/catalyst/parser/AstBuilder.scala | 4 -
.../sql/connector/DataSourceV2OptionSuite.scala | 104 ++++++++++++++++++++-
.../sql/connector/MergeIntoTableSuiteBase.scala | 43 +++++++++
.../spark/sql/connector/UpdateTableSuiteBase.scala | 85 +++++++++++++++++
.../spark/sql/execution/command/DDLSuite.scala | 52 +++++++++++
7 files changed, 304 insertions(+), 6 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index f6bc4380dbfd..709a895e0ecd 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -37,6 +37,7 @@ license: |
- Since Spark 4.3, the new `COMMENT ON COLUMN ... IS NULL` syntax removes a
column comment by passing a `null` comment to
`TableChange.updateColumnComment(String[], String)`, so a `UpdateColumnComment`
table change may now carry a `null` `newComment()`. Previously `newComment()`
was always non-null. DataSource V2 catalogs that handle `UpdateColumnComment`
should null-check `newComment()` and treat `null` as "remove the column
comment" (mirroring how `UpdateColumnDefaultValue` already car [...]
- Since Spark 4.3, `unix_seconds`, `unix_millis`, and `unix_micros` accept
`TIMESTAMP_NTZ` and the nanosecond-precision timestamp types directly, reading
them with no time-zone shift. Previously these functions accepted only
`TIMESTAMP_LTZ`; a `TIMESTAMP_NTZ` or nanosecond-timestamp argument was
rejected with a `DATATYPE_MISMATCH` error. This is a new capability and does
not change the result of any query that previously succeeded.
- Since Spark 4.3, `hash()` and `xxhash64()` include the `days` field of
`CalendarInterval` when computing the hash, so their output for interval values
differs from earlier releases. Previously the codegen path dropped `days`,
disagreeing with interpreted evaluation.
+- Since Spark 4.3, when a `SELECT` or `INSERT` statement references the same
table more than once with different `WITH (...)` options (for example a
self-join, or `INSERT INTO t WITH (...) SELECT * FROM t WITH (...)`), each
reference now uses its own options instead of the second reference silently
inheriting the first reference's options via the analyzer's relation cache.
## Upgrading from Spark SQL 4.1 to 4.2
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
index 5bd1256b631f..0a085fcc2971 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala
@@ -239,6 +239,7 @@ class RelationResolution(
val key = toCacheKey(catalog, ident, finalTimeTravelSpec)
val planId = u.getTagValue(LogicalPlan.PLAN_ID_TAG)
val writePrivileges =
u.options.get(UnresolvedRelation.REQUIRED_WRITE_PRIVILEGES)
+ val finalOptions = u.clearWritePrivileges.options
// A reference that requires write privileges is never served from the
per-query relation
// cache. The catalog authorizes the write in `loadTable(ident,
writePrivileges)` below, and
// a cache hit would skip that call entirely. The hit happens whenever
the write target is
@@ -247,9 +248,14 @@ class RelationResolution(
// for `INSERT INTO t SELECT * FROM t`.
val cached = if (writePrivileges == null) relationCache.get(key) else
None
cached
+ // The per-query relation cache is not keyed by options. When the
same table is referenced
+ // more than once in a single statement with different dynamic
options (e.g. a self-join,
+ // or a second reference sharing the target's cache entry), a cache
hit would otherwise
+ // reuse the first reference's options and silently drop this
reference's. Re-apply this
+ // reference's options to the cached relation so each reference
honors its own bag.
+ .map(applyOptions(_, finalOptions))
.map(adaptCachedRelation(_, planId))
.orElse {
- val finalOptions = u.clearWritePrivileges.options
// For a `RelationCatalog` with no time-travel / write privileges,
the single-RPC
// `loadRelation` answers both "is there a table?" and "is there a
view?" in one
// call. Time-travel and write privileges apply to tables only, so
for those the
@@ -364,6 +370,19 @@ class RelationResolution(
CatalogV2Util.lookupCachedRelation(sharedRelationCache, catalog, ident,
table, conf)
}
+ /**
+ * Re-applies `options` to the relation in a cached plan. Every
`relationCache` entry holds a
+ * single relation for its own identifier (a view's body is still unresolved
when it is cached),
+ * so this cannot reach another table's relation.
+ */
+ private def applyOptions(
+ cached: LogicalPlan,
+ options: CaseInsensitiveStringMap): LogicalPlan = cached transform {
+ case r: DataSourceV2Relation => r.copy(options = options)
+ case r: UnresolvedCatalogRelation => r.copy(options = options)
+ case r: StreamingRelationV2 => r.copy(extraOptions = options)
+ }
+
private def adaptCachedRelation(cached: LogicalPlan, planId: Option[Long]):
LogicalPlan = {
val plan = cached transform {
case multi: MultiInstanceRelation =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
index 2faf07b5379c..10c27ad8200c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala
@@ -1284,10 +1284,6 @@ class AstBuilder extends DataTypeAstBuilder
val withSchemaEvolution = ctx.EVOLUTION() != null
// The target and source may each carry their own dynamic table options
via `WITH (...)`.
- // Known limitation: if the same table is used as both the target and the
source
- // (e.g. `MERGE INTO t WITH (a) USING t WITH (b) s`), the analyzer's
relation cache is keyed
- // without options and reuses the first resolved relation, so the target's
options win and the
- // source's are silently dropped.
val sourceTableOrQuery = if (ctx.source != null) {
createUnresolvedRelation(ctx.source, Option(ctx.sourceOptions))
} else if (ctx.sourceQuery != null) {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
index 803dd35513f4..f785cac9c012 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2OptionSuite.scala
@@ -20,7 +20,8 @@ package org.apache.spark.sql.connector
import org.apache.spark.sql.{AnalysisException, Row}
import org.apache.spark.sql.QueryTest.withQueryExecutionsCaptured
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.connector.catalog.InMemoryBaseTable
+import org.apache.spark.sql.catalyst.streaming.StreamingRelationV2
+import org.apache.spark.sql.connector.catalog.{InMemoryBaseTable,
InMemoryRowLevelOperationTableCatalog}
import org.apache.spark.sql.execution.CommandResultExec
import org.apache.spark.sql.execution.datasources.v2._
import org.apache.spark.sql.functions.lit
@@ -121,6 +122,107 @@ class DataSourceV2OptionSuite extends DatasourceV2SQLBase
{
}
}
+ test("SPARK-58330: dynamic options are not lost when INSERT selects from the
same table") {
+ val t1 = s"${catalogAndNamespace}table"
+ withTable(t1) {
+ sql(s"CREATE TABLE $t1 (id bigint, data string)")
+ sql(s"INSERT INTO $t1 VALUES (1, 'a'), (2, 'b')")
+
+ // The target and the source are the same table, so they share one
per-query relation-cache
+ // entry. Each reference must keep its own options: the target's write
options must not be
+ // dropped by the source reference, and the source's read options must
not leak to the target.
+ val df = sql(s"INSERT INTO $t1 WITH (`write.split-size` = 10) " +
+ s"SELECT id, data FROM $t1 WITH (`split-size` = 5)")
+
+ val appendData = df.queryExecution.optimizedPlan.collectFirst {
+ case CommandResult(_, a: AppendData, _, _) => a
+ }.getOrElse(fail("expected an AppendData in the optimized plan"))
+
+ // target: the write relation keeps its own option and does not pick up
the source's
+ val targetOptions =
appendData.table.asInstanceOf[DataSourceV2Relation].options
+ assert(targetOptions.get("write.split-size") === "10", "target write
option")
+ assert(!targetOptions.containsKey("split-size"), "target must not see
source option")
+
+ // source: the read relation under the query keeps its own option and
does not pick up
+ // the target's
+ val sourceOptions = appendData.query.collect {
+ case r: DataSourceV2Relation => r.options
+ case s: DataSourceV2ScanRelation => s.relation.options
+ }.filter(_.containsKey("split-size"))
+ assert(sourceOptions.nonEmpty, "source relation carrying the option was
not found")
+ sourceOptions.foreach { opts =>
+ assert(opts.get("split-size") === "5", "source read option")
+ assert(!opts.containsKey("write.split-size"), "source must not see
target option")
+ }
+
+ checkAnswer(sql(s"SELECT * FROM $t1"),
+ Seq(Row(1, "a"), Row(2, "b"), Row(1, "a"), Row(2, "b")))
+ }
+ }
+
+ test("SPARK-58330: each reference in a self-join keeps its own dynamic
options") {
+ val t1 = s"${catalogAndNamespace}table"
+ withTable(t1) {
+ sql(s"CREATE TABLE $t1 (id bigint, data string)")
+ sql(s"INSERT INTO $t1 VALUES (1, 'a'), (2, 'b')")
+
+ // Both references are reads of the same table with different options,
so they share one
+ // per-query relation-cache entry. Neither scan should inherit the
other's options.
+ val df = sql(s"SELECT a.id FROM $t1 WITH (`split-size` = 5) a " +
+ s"JOIN $t1 WITH (`split-size` = 9) b ON a.id = b.id")
+
+ val splitSizes = df.queryExecution.optimizedPlan.collect {
+ case s: DataSourceV2ScanRelation =>
s.relation.options.get("split-size")
+ }
+ assert(splitSizes.sorted === Seq("5", "9"))
+
+ checkAnswer(df, Seq(Row(1), Row(2)))
+ }
+ }
+
+ test("SPARK-58330: each reference in a streaming self-join keeps its own
dynamic options") {
+ withSQLConf(
+ "spark.sql.catalog.streamcat" ->
classOf[InMemoryRowLevelOperationTableCatalog].getName) {
+ val t1 = "streamcat.ns.table"
+ withTable(t1) {
+ sql(s"CREATE TABLE $t1 (id bigint, data string)")
+
+ // Both references are streaming reads of the same table with
different options, so they
+ // share one per-query relation-cache entry. Neither should inherit
the other's options.
+ // The plan is only analyzed here, never executed (no
writeStream.start()), so no actual
+ // streaming query runs.
+ val df = sql(s"SELECT a.id FROM STREAM $t1 WITH (`split-size` = 5) a "
+
+ s"JOIN STREAM $t1 WITH (`split-size` = 9) b ON a.id = b.id")
+
+ val splitSizes = df.queryExecution.analyzed.collect {
+ case r: StreamingRelationV2 => r.extraOptions.get("split-size")
+ }
+ assert(splitSizes.sorted === Seq("5", "9"))
+ }
+ }
+ }
+
+ test("SPARK-58330: a streaming CTE referencing the same table keeps its own
dynamic options") {
+ withSQLConf(
+ "spark.sql.catalog.streamcat" ->
classOf[InMemoryRowLevelOperationTableCatalog].getName) {
+ val t1 = "streamcat.ns.table"
+ withTable(t1) {
+ sql(s"CREATE TABLE $t1 (id bigint, data string)")
+
+ // The CTE's inner scan of `STREAM t WITH (...)` and the outer `STREAM
t WITH (...)`
+ // reference resolve to the same per-query relation-cache entry; CTE
substitution must
+ // not let one leak into the other. Analysis-only, as above.
+ val df = sql(s"WITH x AS (SELECT id FROM STREAM $t1 WITH (`split-size`
= 5)) " +
+ s"SELECT a.id FROM STREAM $t1 WITH (`split-size` = 9) a CROSS JOIN
x")
+
+ val splitSizes = df.queryExecution.analyzed.collect {
+ case r: StreamingRelationV2 => r.extraOptions.get("split-size")
+ }
+ assert(splitSizes.sorted === Seq("5", "9"))
+ }
+ }
+ }
+
test("SPARK-50286: Propagate options for DataFrameWriter Append") {
val t1 = s"${catalogAndNamespace}table"
withTable(t1) {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
index 9b625c1b81d2..de6c45006911 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/MergeIntoTableSuiteBase.scala
@@ -2960,6 +2960,49 @@ abstract class MergeIntoTableSuiteBase extends
RowLevelOperationSuiteBase
}
}
+ test("SPARK-58330: self-merge keeps each reference's own dynamic options") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |""".stripMargin)
+
+ // Target and source are the same table, so they share one per-query
relation-cache entry.
+ // Each reference must keep its own options: the target keeps its write
options and the source
+ // keeps its read options; neither is dropped or inherited from the other.
+ val Seq(qe) = withQueryExecutionsCaptured(spark) {
+ sql(
+ s"""MERGE INTO $tableNameAsString t WITH (`write.split-size` = 10)
+ |USING $tableNameAsString WITH (`split-size` = 5) s
+ |ON t.pk = s.pk
+ |WHEN MATCHED THEN UPDATE SET t.salary = s.salary
+ |""".stripMargin)
+ }
+
+ // target write relation carries only its own option, not the source's
+ val writeRelation = qe.optimizedPlan.collectFirst {
+ case rd: ReplaceData => rd.table
+ case wd: WriteDelta => wd.table
+ }.getOrElse(fail("couldn't find row-level operation in optimized plan"))
+ .asInstanceOf[DataSourceV2Relation]
+ assert(writeRelation.options.get("write.split-size") === "10", "target
relation option")
+ assert(!writeRelation.options.containsKey("split-size"), "target must not
see source option")
+
+ // source scan carries only its own option, not the target's
+ val sourceOptions = qe.optimizedPlan.collect {
+ case r: DataSourceV2Relation => r.options
+ case s: DataSourceV2ScanRelation => s.relation.options
+ }.filter(_.containsKey("split-size"))
+ assert(sourceOptions.nonEmpty, "source relation carrying the option was
not found")
+ sourceOptions.foreach { opts =>
+ assert(opts.get("split-size") === "5", "source relation option")
+ assert(!opts.containsKey("write.split-size"), "source must not see
target option")
+ }
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, 100, "hr") :: Row(2, 200, "software") :: Nil)
+ }
+
test("SPARK-58007: merge with dynamic options on the target, insert-only
fast path") {
withTable(sourceNameAsString) {
createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
index adaa3be4510f..b54445466d7e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
@@ -20,9 +20,12 @@ package org.apache.spark.sql.connector
import org.apache.spark.SparkRuntimeException
import org.apache.spark.internal.config
import org.apache.spark.sql.{sources, AnalysisException, Row}
+import org.apache.spark.sql.QueryTest.withQueryExecutionsCaptured
+import org.apache.spark.sql.catalyst.plans.logical.{ReplaceData, WriteDelta}
import org.apache.spark.sql.connector.catalog.{Aborted, Column,
ColumnDefaultValue, Committed, InMemoryTable, TableChange, TableInfo}
import org.apache.spark.sql.connector.expressions.{GeneralScalarExpression,
LiteralValue}
import org.apache.spark.sql.connector.write.UpdateSummary
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation,
DataSourceV2ScanRelation}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{IntegerType, StringType}
@@ -1265,6 +1268,46 @@ abstract class UpdateTableSuiteBase extends
RowLevelOperationSuiteBase {
Row(1, -1, "hr") :: Row(2, 200, "software") :: Row(3, -1, "hr") :: Nil)
}
+ test("SPARK-58330: update keeps its own options when a subquery references
the same table " +
+ "with different options") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 300, "dep": "hr" }
+ |""".stripMargin)
+
+ // The target and the WHERE subquery reference the same table with
different options; they
+ // share one per-query relation-cache entry. Each reference must keep its
own options.
+ val Seq(qe) = withQueryExecutionsCaptured(spark) {
+ sql(s"UPDATE $tableNameAsString WITH (`write.split-size` = 10) SET
salary = -1 " +
+ s"WHERE pk IN (SELECT pk FROM $tableNameAsString WITH (`split-size` =
5) WHERE dep = 'hr')")
+ }
+
+ // target write relation carries only its own option
+ val writeRelation = qe.optimizedPlan.collectFirst {
+ case rd: ReplaceData => rd.table
+ case wd: WriteDelta => wd.table
+ }.getOrElse(fail("couldn't find row-level operation in optimized plan"))
+ .asInstanceOf[DataSourceV2Relation]
+ assert(writeRelation.options.get("write.split-size") === "10", "target
relation option")
+ assert(!writeRelation.options.containsKey("split-size"), "target must not
see subquery option")
+
+ // the subquery scan carries only its own option
+ val subqueryOptions = qe.optimizedPlan.collect {
+ case r: DataSourceV2Relation => r.options
+ case s: DataSourceV2ScanRelation => s.relation.options
+ }.filter(_.containsKey("split-size"))
+ assert(subqueryOptions.nonEmpty, "subquery relation carrying the option
was not found")
+ subqueryOptions.foreach { opts =>
+ assert(opts.get("split-size") === "5", "subquery relation option")
+ assert(!opts.containsKey("write.split-size"), "subquery must not see
target option")
+ }
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, -1, "hr") :: Row(2, 200, "software") :: Row(3, -1, "hr") :: Nil)
+ }
+
test("SPARK-57681: update with dynamic options and a CTE on the same table")
{
createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
"""{ "pk": 1, "salary": 100, "dep": "hr" }
@@ -1282,4 +1325,46 @@ abstract class UpdateTableSuiteBase extends
RowLevelOperationSuiteBase {
sql(s"SELECT * FROM $tableNameAsString"),
Row(1, -1, "hr") :: Row(2, 200, "software") :: Row(3, -1, "hr") :: Nil)
}
+
+ test("SPARK-58330: update keeps its own options when a CTE references the
same table " +
+ "with different options") {
+ createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
+ """{ "pk": 1, "salary": 100, "dep": "hr" }
+ |{ "pk": 2, "salary": 200, "dep": "software" }
+ |{ "pk": 3, "salary": 300, "dep": "hr" }
+ |""".stripMargin)
+
+ // The target and the CTE reference the same table with different options;
they share one
+ // per-query relation-cache entry. Each reference must keep its own
options.
+ val Seq(qe) = withQueryExecutionsCaptured(spark) {
+ sql(s"WITH hr_rows AS (SELECT pk FROM $tableNameAsString WITH
(`split-size` = 5) " +
+ s"WHERE dep = 'hr') " +
+ s"UPDATE $tableNameAsString WITH (`write.split-size` = 10) SET salary
= -1 " +
+ s"WHERE pk IN (SELECT pk FROM hr_rows)")
+ }
+
+ // target write relation carries only its own option
+ val writeRelation = qe.optimizedPlan.collectFirst {
+ case rd: ReplaceData => rd.table
+ case wd: WriteDelta => wd.table
+ }.getOrElse(fail("couldn't find row-level operation in optimized plan"))
+ .asInstanceOf[DataSourceV2Relation]
+ assert(writeRelation.options.get("write.split-size") === "10", "target
relation option")
+ assert(!writeRelation.options.containsKey("split-size"), "target must not
see CTE option")
+
+ // the CTE scan carries only its own option
+ val cteOptions = qe.optimizedPlan.collect {
+ case r: DataSourceV2Relation => r.options
+ case s: DataSourceV2ScanRelation => s.relation.options
+ }.filter(_.containsKey("split-size"))
+ assert(cteOptions.nonEmpty, "CTE relation carrying the option was not
found")
+ cteOptions.foreach { opts =>
+ assert(opts.get("split-size") === "5", "CTE relation option")
+ assert(!opts.containsKey("write.split-size"), "CTE must not see target
option")
+ }
+
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(1, -1, "hr") :: Row(2, 200, "software") :: Row(3, -1, "hr") :: Nil)
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
index a52baaeac42b..2bcbe4d375c9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala
@@ -27,6 +27,7 @@ import org.apache.hadoop.fs.permission.{AclEntry, AclStatus}
import org.apache.spark.{SparkClassNotFoundException, SparkException,
SparkFiles, SparkRuntimeException, SparkUnsupportedOperationException}
import org.apache.spark.internal.config
import org.apache.spark.sql.{AnalysisException, QueryTest, Row, SaveMode}
+import org.apache.spark.sql.QueryTest.withQueryExecutionsCaptured
import org.apache.spark.sql.catalyst.{FunctionIdentifier, QualifiedTableName,
TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.TempTableAlreadyExistsException
import org.apache.spark.sql.catalyst.catalog._
@@ -1472,6 +1473,57 @@ abstract class DDLSuite extends QueryTest with
DDLSuiteBase {
}
}
+ test("SPARK-58330: self-reference to a v1 table keeps each reference's own
dynamic options") {
+ withTable("t") {
+ spark.sql("CREATE TABLE t(a string, b string) USING CSV")
+ spark.sql("INSERT INTO TABLE t VALUES ('a;b', 'c')")
+
+ checkAnswer(
+ sql("SELECT x.a, x.b, y.a, y.b FROM t x CROSS JOIN t WITH ('delimiter'
= ';') y"),
+ Row("a;b", "c", "a", "b,c") :: Nil
+ )
+ }
+ }
+
+ test("SPARK-58330: INSERT keeps its own options when selecting from the same
v1 table") {
+ withTable("t") {
+ spark.sql("CREATE TABLE t(a string, b string) USING CSV")
+ spark.sql("INSERT INTO TABLE t VALUES ('a;b', 'c')")
+
+ // The target and the source are the same session-catalog table, so they
share one
+ // per-query relation-cache entry. The target's write option must not be
dropped by the
+ // source reference, and the source's read option must not leak to the
target.
+ val Seq(qe) = withQueryExecutionsCaptured(spark) {
+ sql("INSERT INTO t WITH ('sep' = '|') SELECT a, b FROM t WITH ('sep' =
';')")
+ }
+
+ val targetOptions = qe.analyzed.collectFirst {
+ case cmd: InsertIntoHadoopFsRelationCommand => cmd.options
+ }.getOrElse(fail("target InsertIntoHadoopFsRelationCommand not found"))
+ val sourceOptions = qe.analyzed.collect {
+ case LogicalRelation(fsRelation: HadoopFsRelation, _, _, _, _) =>
fsRelation.options
+ }.find(_.get("sep").contains(";")).getOrElse(fail("source relation with
sep=; not found"))
+ assert(targetOptions.get("sep").contains("|"), "target write option")
+ assert(sourceOptions("sep") === ";", "source read option")
+ }
+ }
+
+ test("SPARK-58330: a CTE referencing the same v1 table keeps its own dynamic
options") {
+ withTable("t") {
+ spark.sql("CREATE TABLE t(a string, b string) USING CSV")
+ spark.sql("INSERT INTO TABLE t VALUES ('a;b', 'c')")
+
+ // `t` (no options) and the CTE's inner scan of `t WITH ('delimiter' =
';')` resolve to
+ // the same per-query relation-cache entry; CTE substitution must not
let one leak into
+ // the other.
+ checkAnswer(
+ sql("WITH x AS (SELECT a, b FROM t WITH ('delimiter' = ';')) " +
+ "SELECT t.a, t.b, x.a, x.b FROM t CROSS JOIN x"),
+ Row("a;b", "c", "a", "b,c") :: Nil
+ )
+ }
+ }
+
test("SPARK-18009 calling toLocalIterator on commands") {
import scala.jdk.CollectionConverters._
val df = sql("show databases")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]