Anurag Mantripragada created SPARK-58330:
--------------------------------------------
Summary: Dynamic table options are dropped when the same table is
referenced twice in a DML statement
Key: SPARK-58330
URL: https://issues.apache.org/jira/browse/SPARK-58330
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Anurag Mantripragada
Spark supports a per-statement `WITH (key = value)` dynamic table options
clause on `SELECT` (SPARK-36680), `INSERT` (SPARK-49098), `UPDATE`
(SPARK-57681), and `MERGE` (SPARK-58007). When a single statement references
the same table more than once with different options, only the first-resolved
reference's options are honored; the other reference's options are silently
dropped.
Examples (each drops one side's options):
```sql
-- INSERT self-select: the target's write options are dropped
INSERT INTO t WITH (`write.split-size` = 10) SELECT * FROM t
-- MERGE self-merge: the source's read options are dropped
MERGE INTO t WITH (`write.split-size` = 10)
USING t WITH (`split-size` = 5) s
ON t.id = s.id
WHEN MATCHED THEN UPDATE SET ...
-- UPDATE self-reference in a subquery/CTE: the subquery's options are dropped
UPDATE t WITH (`write.split-size` = 10) SET ...
WHERE id IN (SELECT id FROM t WITH (`split-size` = 5))
```
No error is raised; an option the user explicitly wrote is silently ignored.
Root cause: the per-query analyzer relation cache (`RelationResolution`) is
keyed by catalog + namespace + name + time-travel spec only -- options are not
part of the key. On a cache hit the cached relation is returned as-is, without
applying the current reference's options, so the second reference to the same
table inherits the first reference's options. Which reference "wins" is a
traversal-order accident. The shared (cross-query) relation cache already
re-applies the caller's options on a hit, so this is an inconsistency between
the two cache paths.
This affects INSERT, UPDATE, and MERGE and should be fixed once in the shared
resolution path so each reference honors exactly its own options. This subsumes
the separately-noted INSERT self-select option-drop.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
