zhuxiangyi opened a new pull request, #10061: URL: https://github.com/apache/paimon/pull/10061
### Purpose Bug fix: on any table that has a `CHAR(n)` column, a V1 `UPDATE` only works in one shape — assigning a literal to every `CHAR` column. Every other shape is rejected at analysis and never runs. #### Symptom ```sql CREATE TABLE t (id INT, c CHAR(3), b INT); -- same for append, deletion-vector, primary-key and data-evolution tables INSERT INTO t VALUES (1, 'a', 10), (2, 'b', 20); ``` | statement | master (Spark 3.5) | |---|---| | `UPDATE t SET b = 0 WHERE id = 1` — the `CHAR` column is not mentioned at all | `MISSING_ATTRIBUTES ... "c" missing from "id", "c", "b" in operator !Project [...]` | | `UPDATE t SET b = 1 WHERE c = 'a'` — read in `WHERE` | `MISSING_ATTRIBUTES ... in operator !Filter (c#78 = a )` | | `UPDATE t SET b = length(c) WHERE id = 2` — read in a `SET` value | `MISSING_ATTRIBUTES` | | `UPDATE t SET b = 5` — no condition | `MISSING_ATTRIBUTES` | | `UPDATE t SET c = 'z' WHERE id = 1` — literal assigned to the `CHAR` column | works | The first row is the important one: the statement does not reference `c` in any way and still fails. The mere presence of a `CHAR` column makes `UPDATE` unusable on the table, and the error is Spark's internal `MISSING_ATTRIBUTES`, which gives the user no hint that `CHAR` is involved. On data-evolution tables the operator in the message is `MergeRows[...]` instead; everything else is the same. #### Impact - **Every table kind.** Append, append with deletion vectors, primary-key and data-evolution tables all reproduce it. They share the `PaimonUpdateTable` rule; the commands differ, but they all receive the same unresolvable expressions. - **No workaround.** V1 is the default path (`write.use-v2-write` defaults to `false`), and the V2 row-level operations explicitly exclude tables with `CHAR` columns (`SparkTable.supportsV2CopyOnWriteOps` / `supportsV2DeltaOps` check `!SparkTypeUtils.containsCharType`), so they fall back to V1 too. There is no configuration under which these statements run. - **Default configuration triggers it.** It needs `spark.sql.readSideCharPadding`, which is `true` by default on Spark 3.4+ (older versions apply read-side padding unconditionally). - **Who hits it.** Tables migrated from Hive or a traditional warehouse routinely use `CHAR(n)` for fixed-width codes (country, status, currency, gender). Any `UPDATE` of some other column on such a table fails. - **Not affected.** `SELECT`, `INSERT`, `DELETE` and `MERGE INTO` (MERGE aligns assignments through a different path and showed no problem); `VARCHAR` columns (Spark does not pad them on read, so their exprIds are stable); `UPDATE`s that assign a literal to each `CHAR` column. - **Why the existing tests pass.** The only `CHAR` case in `UpdateTableTestBase` is `UPDATE T SET c = 'b' WHERE id = 1` — exactly the one shape that works. #### Root cause With read-side char padding, the analyzer places a padding `Project` on top of the relation and gives the `CHAR` attributes new exprIds: `u.table.output` carries `c#a` while the underlying `DataSourceV2Relation` carries `c#b`. Non-`CHAR` attributes pass through the `Project` unchanged, which is why tables without `CHAR` never see this. `PaimonUpdateTable` aligns the assignments against `u.table.output`. That is the right choice for the assignment *keys* (it is what #7976 introduced it for, so that a key on a `CHAR` column resolves), but the rule then hands the aligned *values* and the *condition* — both written in terms of `c#a` — to `UpdatePaimonTableCommand` / `UpdatePaimonDataEvolutionTableCommand`: ```scala val alignedExpressions = alignedAssignments.map(_.value).zip(relation.output) ... UpdatePaimonTableCommand(relation, paimonTable, condition.getOrElse(TrueLiteral), alignedExpressions) ``` Both commands plan on the bare `relation` (`c#b`), so every reference to `c#a` is unresolvable there. An untouched `CHAR` column is the worst case: its aligned value is `c#a` itself, which is why a statement that never mentions the column still fails. The data-evolution command suffers one more consequence: `isModifiedAssignment` sees key `c#b` ≠ value `c#a` and treats the `CHAR` column as an updated column. #### Fix After alignment, rewrite the aligned values and the condition from `u.table.output` attributes to the positionally matching `relation.output` attributes — the two line up 1:1, which the existing zip already relies on — including `OuterReference`s inside correlated subqueries, and build the V1 commands from the rewritten expressions. The V2 branch is untouched: it returns the aligned `UpdateTable` with `u.table` still in place. Padding semantics are preserved. Whenever the commands' plans are analyzed, the analyzer re-applies the read-side padding `Project` on top of `relation`, so a rewritten reference reads the padded value exactly as a `SELECT` does: `length(c)` is 3 for `CHAR(3) 'a'`, and `c = 'a'` compares against the padded literal. #### Benefit - `UPDATE` works on tables with `CHAR` columns for every table kind and every statement shape, not only when each `CHAR` column is assigned a literal. - On data-evolution tables an untouched `CHAR` column aligns to its own attribute again, so it is no longer treated as an updated column: only the assigned columns go into the new column-group file (`write_cols` is `[b]` in the test), and a `WHERE` on a `CHAR` column takes the self-merge shortcut from #10037 instead of the join path. ### Tests - `UpdateTableTestBase` — new `CHAR column that is read or left untouched`, run for an append table, a deletion-vector table and a primary-key table, covering: the `CHAR` column untouched, `WHERE c = 'a'`, `SET b = length(c)`, a correlated `EXISTS` subquery comparing `s.k = t.c`, and an unconditional `UPDATE`. Every one of these shapes fails on master with the `MISSING_ATTRIBUTES` error above. - `RowTrackingTestBase` — new `V1 update on a table with a CHAR column`: the untouched `CHAR` column is not written (`write_cols` of the new file is `[b]`), and `UPDATE t SET b = length(c) WHERE c = 'b'` takes the self-merge shortcut and returns the padded length. - The existing `Paimon update: update table with char type` and the overlong `CHAR` / `VARCHAR` tests keep passing. Run on Spark 3.5: `UpdateTableTest`, `RowTrackingTest`, `BlobUpdateTest`, `DataEvolutionDeletionTest` (120 tests) and `DataEvolutionUpdateSnapshotTest`. `spotless:check` and `checkstyle:check` pass on both modules. ### API and Format No changes. ### Documentation No changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
