dongjoon-hyun opened a new pull request, #60007:
URL: https://github.com/apache/spark/pull/60007
### What changes were proposed in this pull request?
This PR fixes the errors for option values that cannot be resolved in the
`OPTIONS` clause of `CREATE TABLE`, `REPLACE TABLE`, `CREATE TABLE AS SELECT`
(CTAS), and `REPLACE TABLE AS SELECT` (RTAS).
1. `ResolveTableSpec` now fails with the existing
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` error when an option value of an
unresolved table spec contains a subquery.
2. `V2CreateTableAsSelectPlan.resolved` now also requires its expressions to
be resolved, so CTAS and RTAS are no longer treated as resolved while their
table spec has unresolved option values.
### Why are the changes needed?
An option value must be a constant expression. A subquery is not one, and
`ResolveTableSpec` already reports this when the subquery plan is resolved at
parsing time, such as `(SELECT 1 AS x)`. Other subqueries get an internal or
misleading error.
```sql
CREATE TABLE testcat.ns.t (id INT) USING _ OPTIONS (a = (SELECT 1));
-- [INTERNAL_ERROR] Found the unresolved operator: 'Project
[unresolvedalias(1)] SQLSTATE: XX000
```
There are two causes.
- `ResolveSubquery` does not handle these commands, so a subquery plan that
is unresolved after parsing stays unresolved. `ResolveTableSpec` keeps waiting
for `childrenResolved`, and `CheckAnalysis` then fails on the subquery plan.
For the same reason, a subquery that reads an existing table fails with
`TABLE_OR_VIEW_NOT_FOUND`.
- `V2CreateTableAsSelectPlan.resolved` ignores the table spec, so analyzer
code that only handles resolved plans reaches unresolved option values.
- `UpdateAttributeNullability` calls `exprId` on an `UnresolvedAttribute`,
which fails with `INTERNAL_ERROR` for `1 + missing_col`.
- When `spark.sql.planChangeValidation` is enabled, which is the default
in tests, `LogicalPlanIntegrity` asks for the schema of the unresolved subquery
plan before `ResolveTableSpec` runs. This is why the first change alone is not
enough.
### Does this PR introduce _any_ user-facing change?
Yes. These statements failed before and still fail, but with a different
error. The results below are from a V2 catalog.
| Option value | Statements | Before | After |
|---|---|---|---|
| `(SELECT 1)`, `1 + (SELECT 1)`, `EXISTS (SELECT 1)`, `1 IN (SELECT 1)` |
`CREATE TABLE`, `REPLACE TABLE`, CTAS, RTAS | `INTERNAL_ERROR` |
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
| `(SELECT max(id) FROM <existing table>)` | `CREATE TABLE`, `REPLACE
TABLE`, CTAS, RTAS | `TABLE_OR_VIEW_NOT_FOUND` |
`INVALID_SQL_SYNTAX.OPTION_IS_INVALID` |
| `1 + missing_col` | CTAS, RTAS | `INTERNAL_ERROR` |
`UNRESOLVED_COLUMN.WITH_SUGGESTION` |
The session catalog behaves the same, except that `REPLACE TABLE AS SELECT`
is not supported there: `1 + missing_col` keeps failing with
`UNSUPPORTED_FEATURE.TABLE_OPERATION`, and `EXISTS (SELECT 1)` and `1 IN
(SELECT 1)` change from that error to `INVALID_SQL_SYNTAX.OPTION_IS_INVALID`.
### How was this patch tested?
Added two test cases to `TableOptionsConstantFoldingSuite`. Both fail
without this change. With only the `ResolveTableSpec` change both still fail,
and with only the `V2CreateTableAsSelectPlan` change the subquery test still
fails.
```
build/sbt 'sql/testOnly *TableOptionsConstantFoldingSuite'
```
The tests use the session catalog because `ResolveTableSpec` does not depend
on the catalog. The statements in the table above were also run manually
against a V2 catalog, with and without `spark.testing`, before and after this
change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
