This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ba5bb0ba7764 [SPARK-60780][SQL] Fix internal error when table OPTIONS
contain a subquery
ba5bb0ba7764 is described below
commit ba5bb0ba77641c1c718d996e4cff86054449508f
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 12:31:33 2026 -0800
[SPARK-60780][SQL] Fix internal error when table OPTIONS contain a subquery
### What changes were proposed in this pull request?
This PR fixes the errors for option values that cannot be resolved in the
`OPTIONS` clause of `CREATE TABLE`, `REPLACE TABLE`, `CREATE TABLE AS SELECT`
(CTAS), and `REPLACE TABLE AS SELECT` (RTAS).
1. `ResolveTableSpec` now fails with the existing
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` error when an option value contains a
subquery. It checks this before evaluating option values, together with the
existing check for subqueries in CHECK constraints.
2. `V2CreateTableAsSelectPlan.resolved` now also requires its expressions
to be resolved, so CTAS and RTAS are no longer treated as resolved while their
table spec has unresolved option values.
### Why are the changes needed?
An option value must be a constant expression. A subquery is not one, but
`ResolveTableSpec` reported this only when the subquery plan is resolved at
parsing time, such as `(SELECT 1 AS x)`, and only by evaluating it, with
`[INTERNAL_ERROR] Cannot evaluate expression` as the cause. Other subqueries
get an internal or misleading error.
```sql
CREATE TABLE testcat.ns.t (id INT) USING _ OPTIONS (a = (SELECT 1));
-- [INTERNAL_ERROR] Found the unresolved operator: 'Project
[unresolvedalias(1)] SQLSTATE: XX000
```
CTAS and RTAS also fail with `INTERNAL_ERROR` instead of
`UNRESOLVED_COLUMN` for an unresolved option value without a subquery, such as
the unquoted value in `OPTIONS ('compression' = snappy)`.
There are two causes.
- `ResolveSubquery` does not handle these commands, so a subquery plan that
is unresolved after parsing stays unresolved. `ResolveTableSpec` keeps waiting
for `childrenResolved`, and `CheckAnalysis` then fails on the subquery plan.
For the same reason, a subquery that reads an existing table fails with
`TABLE_OR_VIEW_NOT_FOUND`.
- `V2CreateTableAsSelectPlan.resolved` ignores the table spec, so analyzer
code that only handles resolved plans reaches unresolved option values.
- `UpdateAttributeNullability` calls `exprId` on an
`UnresolvedAttribute`, which fails with `INTERNAL_ERROR` for `snappy` or `1 +
missing_col`.
- When `spark.sql.planChangeValidation` is enabled, which is the default
in tests, `LogicalPlanIntegrity` asks for the schema of the unresolved subquery
plan before `ResolveTableSpec` runs. This is why the first change alone is not
enough.
### Does this PR introduce _any_ user-facing change?
Yes. These statements failed before and still fail, but with a different
error. The results below are from a V2 catalog.
| Option value | Statements | Before | After |
|---|---|---|---|
| `(SELECT 1)`, `1 + (SELECT 1)`, `EXISTS (SELECT 1)`, `1 IN (SELECT 1)` |
`CREATE TABLE`, `REPLACE TABLE`, CTAS, RTAS | `INTERNAL_ERROR` |
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
| `(SELECT max(id) FROM <existing table>)` | `CREATE TABLE`, `REPLACE
TABLE`, CTAS, RTAS | `TABLE_OR_VIEW_NOT_FOUND` |
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
| `snappy` (unquoted), `1 + missing_col` | CTAS, RTAS | `INTERNAL_ERROR` |
`UNRESOLVED_COLUMN.WITH_SUGGESTION` |
`(SELECT 1 AS x)` already failed with
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID`. It no longer has `[INTERNAL_ERROR]
Cannot evaluate expression` as its cause.
The session catalog behaves the same, except that `REPLACE TABLE AS SELECT`
is not supported there: `1 + missing_col` keeps failing with
`UNSUPPORTED_FEATURE.TABLE_OPERATION`, and `EXISTS (SELECT 1)` and `1 IN
(SELECT 1)` change from that error to `INVALID_SQL_SYNTAX.OPTION_IS_INVALID`.
### How was this patch tested?
Added two test cases to `TableOptionsConstantFoldingSuite`. Both fail
without this change. With only the `ResolveTableSpec` change both still fail,
and with only the `V2CreateTableAsSelectPlan` change the subquery test still
fails.
```
build/sbt 'sql/testOnly *TableOptionsConstantFoldingSuite'
```
The tests use the session catalog because `ResolveTableSpec` does not
depend on the catalog. The statements in the table above were also run manually
against a V2 catalog before and after this change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60007 from dongjoon-hyun/SPARK-60780.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit ddecacaaf970b990022f297cb2ff075d228506d7)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/catalyst/analysis/ResolveTableSpec.scala | 12 +++++++---
.../sql/catalyst/plans/logical/v2Commands.scala | 2 +-
.../sql/TableOptionsConstantFoldingSuite.scala | 27 ++++++++++++++++++++++
3 files changed, 37 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index 59c9f40ebe8e..d36bcdecfc12 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -62,13 +62,19 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
input: LogicalPlan,
tableSpec: TableSpecBase,
withNewSpec: TableSpecBase => LogicalPlan): LogicalPlan = tableSpec
match {
- // Check this before `childrenResolved`: `ResolveSubquery` skips these
commands, so the subquery
- // may never resolve, and `toV2Constraint` below would drop one that needs
no resolution.
- case u: UnresolvedTableSpec if
u.constraints.exists(SubqueryExpression.hasSubquery) =>
+ // Check this before `childrenResolved`: `ResolveSubquery` skips these
commands, so a subquery
+ // may never resolve. One that needs no resolution would be dropped by
`toV2Constraint` or
+ // evaluated as an option value below.
+ case u: UnresolvedTableSpec if SubqueryExpression.hasSubquery(u) =>
u.constraints.foreach {
case check: CheckConstraint => check.failIfContainsSubquery()
case _ =>
}
+ u.optionExpression.options.foreach {
+ case (key, value) if SubqueryExpression.hasSubquery(value) =>
+ throw QueryCompilationErrors.optionMustBeConstant(key)
+ case _ =>
+ }
input
case u: UnresolvedTableSpec if u.childrenResolved =>
val newOptions: Seq[(String, String)] = u.optionExpression.options.map {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
index 5984d0269b6a..a13002687e04 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
@@ -577,7 +577,7 @@ trait V2CreateTableAsSelectPlan
withNameAndQuery(newName = name, newQuery = WithCTE(query, cteDefs))
}
- override lazy val resolved: Boolean = childrenResolved && {
+ override lazy val resolved: Boolean = childrenResolved &&
expressions.forall(_.resolved) && {
// the table schema is created from the query schema, so the only
resolution needed is to check
// that the columns referenced by the table's partitioning exist in the
query schema
val references = partitioning.flatMap(_.references).toSet
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
index 67159a05d22a..6d35c2db38ae 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
@@ -112,4 +112,31 @@ class TableOptionsConstantFoldingSuite extends
SharedSparkSession {
"key" -> "k",
"supported" -> "constant expressions"))
}
+
+ test("SPARK-60780: Subquery in CREATE/REPLACE TABLE OPTIONS is not a
constant expression") {
+ Seq(
+ s"$prefix ('k' = (select 1))",
+ "replace table t (col int) using json options ('k' = (select 1))",
+ "create table t using json options ('k' = (select 1)) as select 1 as
col",
+ "replace table t using json options ('k' = (select 1)) as select 1 as
col"
+ ).foreach { statement =>
+ checkError(
+ exception = intercept[AnalysisException](sql(statement)),
+ condition = "INVALID_SQL_SYNTAX.OPTION_IS_INVALID",
+ parameters = Map(
+ "key" -> "k",
+ "supported" -> "constant expressions"))
+ }
+ }
+
+ test("SPARK-60780: Unresolved column in CREATE TABLE AS SELECT OPTIONS") {
+ checkError(
+ exception = intercept[AnalysisException](
+ sql("create table t using json options ('k' = unresolvedAttribute) as
select 1 as col")),
+ condition = "UNRESOLVED_COLUMN.WITH_SUGGESTION",
+ parameters = Map(
+ "objectName" -> "`unresolvedAttribute`",
+ "proposal" -> "`col`"),
+ queryContext = Array(ExpectedContext("", "", 41, 59,
"unresolvedAttribute")))
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]