[
https://issues.apache.org/jira/browse/SPARK-60033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-60033:
-----------------------------
Description:
ResolveTableConstraints builds the write-time check from the stored V2
predicate() when it converts to Catalyst, and only falls back to parsing
predicateSql when it does not. V2ExpressionUtils.toCatalyst turns a column
reference into an UnresolvedAttribute without checking it against the table, so
a stored predicate that names a column the table does not have makes every
later INSERT, UPDATE and MERGE fail with UNRESOLVED_COLUMN, and the
predicateSql fallback is never reached.
Such predicates exist: before SPARK-59962, ALTER TABLE ... ADD CONSTRAINT ...
CHECK (nullif(a + b, 3) > 0) stored CASE WHEN _common_expr_1 = 3 THEN NULL ELSE
_common_expr_1 > 0 END. A catalog that persists the V2 predicate keeps
returning it after an upgrade, so the table stays unwritable until the
constraint is dropped and added again.
Falling back to predicateSql when the converted predicate refers to a column
outside the table would keep such tables writable.
Raised in https://github.com/apache/spark/pull/59217#discussion_r4199939327
> Fall back to predicateSql when a stored CHECK predicate refers to a column
> the table does not have
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-60033
> URL: https://issues.apache.org/jira/browse/SPARK-60033
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
>
> ResolveTableConstraints builds the write-time check from the stored V2
> predicate() when it converts to Catalyst, and only falls back to parsing
> predicateSql when it does not. V2ExpressionUtils.toCatalyst turns a column
> reference into an UnresolvedAttribute without checking it against the table,
> so a stored predicate that names a column the table does not have makes every
> later INSERT, UPDATE and MERGE fail with UNRESOLVED_COLUMN, and the
> predicateSql fallback is never reached.
> Such predicates exist: before SPARK-59962, ALTER TABLE ... ADD CONSTRAINT ...
> CHECK (nullif(a + b, 3) > 0) stored CASE WHEN _common_expr_1 = 3 THEN NULL
> ELSE _common_expr_1 > 0 END. A catalog that persists the V2 predicate keeps
> returning it after an upgrade, so the table stays unwritable until the
> constraint is dropped and added again.
> Falling back to predicateSql when the converted predicate refers to a column
> outside the table would keep such tables writable.
> Raised in https://github.com/apache/spark/pull/59217#discussion_r4199939327
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
