Stephen0421 opened a new issue, #11064: URL: https://github.com/apache/paimon/issues/11064
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation A wide primary key table often needs to update a few columns, such as a historical backfill or a recomputed metric. Today that write stores another full primary-key record. `partial-update` can send only the changed fields, but the old values stay in older files until compaction and expiration. [PIP-44](https://cwiki.apache.org/confluence/display/PAIMON/PIP-44%3A+Support+Data+Evolution+for+Primary+Key+Tables) proposes physical column-level updates by reusing Data Evolution alignment, with `firstRowId` as an internal alignment id rather than row tracking. The [discussion](https://lists.apache.org/thread/7dpn89ndn0xsp2l9j186zr0br6kpx8c9) covers this approach, and the proposer agreed that the work can proceed. This issue tracks the implementation. The first pull request is the foundation only. ### Solution A primary key table can enable `data-evolution.enabled` without `row-tracking.enabled`. - Newly written full rows receive an alignment range (`firstRowId`, `rowCount`) at commit, from `Snapshot.nextRowId`. No `_ROW_ID` column is added. - A caller can commit a column file for one existing base file: same `(partition, bucket, firstRowId, rowCount)`, non-key and non-partition columns only, `level = 0`, via `compactIncrement` (`CommitKind.COMPACT`). - Reads stitch the column file onto the base file before the merge function runs. Sequence and value kind come from the base file, so a backfill does not become a newer record. A merge engine therefore sees an ordinary full row and needs no separate read path. For example, a null field in a newer `partial-update` record keeps the backfilled value and a non-null field replaces it; an aggregate such as `max` uses the backfilled value. - Predicates are applied to the whole alignment group. A split that contains a column file is not raw-convertible. Limit pushdown, at both the manifest scan and the split plan, keeps that bucket. - Compaction rewrites a base file together with its column files into full rows and assigns a new alignment range. Column files are not upgraded on their own. - Commit keeps primary-key conflict detection and adds alignment checks: one base file per group, equal row counts, no overlapping ranges, and one file per column. - An existing table can turn the option on (`false` to `true` only). Files written before that have no alignment range. They can be read until a column file appears in the same bucket; a full compaction of the bucket migrates them. - Column-file rows are not added to `totalRecordCount` or partition `recordCount`. Constraints for this version: fixed bucket (`bucket > 0`) or dynamic bucket (`bucket = -1`). Merge engines that do not require lookup are allowed; `deduplicate`, `partial-update`, and `aggregation` are covered by tests, and `first-row` is rejected because it requires lookup. A new merge engine is allowed unless it requires lookup. Changelog producer is `none` or `input`. No deletion vectors, no lookup, no postpone bucket (`bucket = -2`), no `changelog-producer=full-compaction`, no Iceberg metadata, and no row tracking. Dynamic bucket still uses the existing write API: the caller passes the bucket, and alignment stays per `(partition, bucket)`. The read-optimized table is rejected, because column files stay at level 0 and reading only the top level would return the pre-backfill values. There is no SQL entry point. The caller writes the column file through a Java API and must keep the base file's row order. The committer checks the row count only. A column file whose rows are in another order is committed and silently corrupts those columns. `PrimaryKeyDataEvolutionTableTest` covers this version: validation, alignment assignment, backfill and replacement, compaction including a discarded compaction, both limit layers, whole-group predicate pruning, legacy-file migration, `partial-update`, `aggregation`, and dynamic bucket. ### Anything else? Follow-ups: - `MERGE INTO` and locating `rowIndex` by primary key, so a backfill can be expressed without the caller managing row order. - Column families (`column-family.cols.*`) and writing one logical row as multiple physical files. - Compaction output that keeps column files instead of rewriting them into full rows. - Deletion vectors, lookup, `first-row`, postpone bucket, `changelog-producer=full-compaction`, and Iceberg metadata. - Python read, write, and validation. The first pull request should use `Part of #<this issue>`. `Fixes` would close this issue before those follow-ups. Please assign this issue before that pull request is opened. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
