This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new ddecacaaf970 [SPARK-60780][SQL] Fix internal error when table OPTIONS 
contain a subquery
ddecacaaf970 is described below

commit ddecacaaf970b990022f297cb2ff075d228506d7
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Mon Nov 2 12:31:33 2026 -0800

    [SPARK-60780][SQL] Fix internal error when table OPTIONS contain a subquery
    
    ### What changes were proposed in this pull request?
    
    This PR fixes the errors for option values that cannot be resolved in the 
`OPTIONS` clause of `CREATE TABLE`, `REPLACE TABLE`, `CREATE TABLE AS SELECT` 
(CTAS), and `REPLACE TABLE AS SELECT` (RTAS).
    
    1. `ResolveTableSpec` now fails with the existing 
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` error when an option value contains a 
subquery. It checks this before evaluating option values, together with the 
existing check for subqueries in CHECK constraints.
    2. `V2CreateTableAsSelectPlan.resolved` now also requires its expressions 
to be resolved, so CTAS and RTAS are no longer treated as resolved while their 
table spec has unresolved option values.
    
    ### Why are the changes needed?
    
    An option value must be a constant expression. A subquery is not one, but 
`ResolveTableSpec` reported this only when the subquery plan is resolved at 
parsing time, such as `(SELECT 1 AS x)`, and only by evaluating it, with 
`[INTERNAL_ERROR] Cannot evaluate expression` as the cause. Other subqueries 
get an internal or misleading error.
    
    ```sql
    CREATE TABLE testcat.ns.t (id INT) USING _ OPTIONS (a = (SELECT 1));
    -- [INTERNAL_ERROR] Found the unresolved operator: 'Project 
[unresolvedalias(1)] SQLSTATE: XX000
    ```
    
    CTAS and RTAS also fail with `INTERNAL_ERROR` instead of 
`UNRESOLVED_COLUMN` for an unresolved option value without a subquery, such as 
the unquoted value in `OPTIONS ('compression' = snappy)`.
    
    There are two causes.
    
    - `ResolveSubquery` does not handle these commands, so a subquery plan that 
is unresolved after parsing stays unresolved. `ResolveTableSpec` keeps waiting 
for `childrenResolved`, and `CheckAnalysis` then fails on the subquery plan. 
For the same reason, a subquery that reads an existing table fails with 
`TABLE_OR_VIEW_NOT_FOUND`.
    - `V2CreateTableAsSelectPlan.resolved` ignores the table spec, so analyzer 
code that only handles resolved plans reaches unresolved option values.
      - `UpdateAttributeNullability` calls `exprId` on an 
`UnresolvedAttribute`, which fails with `INTERNAL_ERROR` for `snappy` or `1 + 
missing_col`.
      - When `spark.sql.planChangeValidation` is enabled, which is the default 
in tests, `LogicalPlanIntegrity` asks for the schema of the unresolved subquery 
plan before `ResolveTableSpec` runs. This is why the first change alone is not 
enough.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. These statements failed before and still fail, but with a different 
error. The results below are from a V2 catalog.
    
    | Option value | Statements | Before | After |
    |---|---|---|---|
    | `(SELECT 1)`, `1 + (SELECT 1)`, `EXISTS (SELECT 1)`, `1 IN (SELECT 1)` | 
`CREATE TABLE`, `REPLACE TABLE`, CTAS, RTAS | `INTERNAL_ERROR` | 
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
    | `(SELECT max(id) FROM <existing table>)` | `CREATE TABLE`, `REPLACE 
TABLE`, CTAS, RTAS | `TABLE_OR_VIEW_NOT_FOUND` | 
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
    | `snappy` (unquoted), `1 + missing_col` | CTAS, RTAS | `INTERNAL_ERROR` | 
`UNRESOLVED_COLUMN.WITH_SUGGESTION` |
    
    `(SELECT 1 AS x)` already failed with 
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID`. It no longer has `[INTERNAL_ERROR] 
Cannot evaluate expression` as its cause.
    
    The session catalog behaves the same, except that `REPLACE TABLE AS SELECT` 
is not supported there: `1 + missing_col` keeps failing with 
`UNSUPPORTED_FEATURE.TABLE_OPERATION`, and `EXISTS (SELECT 1)` and `1 IN 
(SELECT 1)` change from that error to `INVALID_SQL_SYNTAX.OPTION_IS_INVALID`.
    
    ### How was this patch tested?
    
    Added two test cases to `TableOptionsConstantFoldingSuite`. Both fail 
without this change. With only the `ResolveTableSpec` change both still fail, 
and with only the `V2CreateTableAsSelectPlan` change the subquery test still 
fails.
    
    ```
    build/sbt 'sql/testOnly *TableOptionsConstantFoldingSuite'
    ```
    
    The tests use the session catalog because `ResolveTableSpec` does not 
depend on the catalog. The statements in the table above were also run manually 
against a V2 catalog before and after this change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60007 from dongjoon-hyun/SPARK-60780.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/catalyst/analysis/ResolveTableSpec.scala   | 12 +++++++---
 .../sql/catalyst/plans/logical/v2Commands.scala    |  2 +-
 .../sql/TableOptionsConstantFoldingSuite.scala     | 27 ++++++++++++++++++++++
 3 files changed, 37 insertions(+), 4 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
index 59c9f40ebe8e..d36bcdecfc12 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveTableSpec.scala
@@ -62,13 +62,19 @@ object ResolveTableSpec extends Rule[LogicalPlan] {
       input: LogicalPlan,
       tableSpec: TableSpecBase,
       withNewSpec: TableSpecBase => LogicalPlan): LogicalPlan = tableSpec 
match {
-    // Check this before `childrenResolved`: `ResolveSubquery` skips these 
commands, so the subquery
-    // may never resolve, and `toV2Constraint` below would drop one that needs 
no resolution.
-    case u: UnresolvedTableSpec if 
u.constraints.exists(SubqueryExpression.hasSubquery) =>
+    // Check this before `childrenResolved`: `ResolveSubquery` skips these 
commands, so a subquery
+    // may never resolve. One that needs no resolution would be dropped by 
`toV2Constraint` or
+    // evaluated as an option value below.
+    case u: UnresolvedTableSpec if SubqueryExpression.hasSubquery(u) =>
       u.constraints.foreach {
         case check: CheckConstraint => check.failIfContainsSubquery()
         case _ =>
       }
+      u.optionExpression.options.foreach {
+        case (key, value) if SubqueryExpression.hasSubquery(value) =>
+          throw QueryCompilationErrors.optionMustBeConstant(key)
+        case _ =>
+      }
       input
     case u: UnresolvedTableSpec if u.childrenResolved =>
       val newOptions: Seq[(String, String)] = u.optionExpression.options.map {
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
index 5984d0269b6a..a13002687e04 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
@@ -577,7 +577,7 @@ trait V2CreateTableAsSelectPlan
     withNameAndQuery(newName = name, newQuery = WithCTE(query, cteDefs))
   }
 
-  override lazy val resolved: Boolean = childrenResolved && {
+  override lazy val resolved: Boolean = childrenResolved && 
expressions.forall(_.resolved) && {
     // the table schema is created from the query schema, so the only 
resolution needed is to check
     // that the columns referenced by the table's partitioning exist in the 
query schema
     val references = partitioning.flatMap(_.references).toSet
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
index 67159a05d22a..6d35c2db38ae 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/TableOptionsConstantFoldingSuite.scala
@@ -112,4 +112,31 @@ class TableOptionsConstantFoldingSuite extends 
SharedSparkSession {
         "key" -> "k",
         "supported" -> "constant expressions"))
   }
+
+  test("SPARK-60780: Subquery in CREATE/REPLACE TABLE OPTIONS is not a 
constant expression") {
+    Seq(
+      s"$prefix ('k' = (select 1))",
+      "replace table t (col int) using json options ('k' = (select 1))",
+      "create table t using json options ('k' = (select 1)) as select 1 as 
col",
+      "replace table t using json options ('k' = (select 1)) as select 1 as 
col"
+    ).foreach { statement =>
+      checkError(
+        exception = intercept[AnalysisException](sql(statement)),
+        condition = "INVALID_SQL_SYNTAX.OPTION_IS_INVALID",
+        parameters = Map(
+          "key" -> "k",
+          "supported" -> "constant expressions"))
+    }
+  }
+
+  test("SPARK-60780: Unresolved column in CREATE TABLE AS SELECT OPTIONS") {
+    checkError(
+      exception = intercept[AnalysisException](
+        sql("create table t using json options ('k' = unresolvedAttribute) as 
select 1 as col")),
+      condition = "UNRESOLVED_COLUMN.WITH_SUGGESTION",
+      parameters = Map(
+        "objectName" -> "`unresolvedAttribute`",
+        "proposal" -> "`col`"),
+      queryContext = Array(ExpectedContext("", "", 41, 59, 
"unresolvedAttribute")))
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to