Stephen0421 opened a new pull request, #11065: URL: https://github.com/apache/paimon/pull/11065
### Purpose Part of #11064 [PIP-44](https://cwiki.apache.org/confluence/display/PAIMON/PIP-44%3A+Support+Data+Evolution+for+Primary+Key+Tables) lets a primary key table update selected columns without rewriting the whole row. The [discussion](https://lists.apache.org/thread/7dpn89ndn0xsp2l9j186zr0br6kpx8c9) agreed to reuse Data Evolution alignment, with `firstRowId` as an alignment id rather than row tracking. This pull request is the foundation agreed there. A primary key table can enable `data-evolution.enabled` without `row-tracking.enabled`. - A newly written full row receives `firstRowId` and `rowCount` at commit. No `_ROW_ID` column is added. - `ColumnFileWriter` commits one column file for one existing base file through `compactIncrement` (`CommitKind.COMPACT`). The file keeps the base file's `(partition, bucket, firstRowId, rowCount)`, contains only non-key and non-partition columns, and stays at level 0. - Reads stitch that column file onto the base file before the merge function runs. Sequence and value kind come from the base file, so the column update does not become a newer record. `deduplicate`, `partial-update`, and `aggregation` see an ordinary full row. - Predicates apply to the whole alignment group. A split containing a column file is not raw-convertible. Limit pushdown keeps that bucket at both the manifest scan and the split plan. - Compaction rewrites the base file together with its column files and assigns a new alignment range. A discarded compaction keeps the column files. - Commit keeps primary-key conflict detection and checks alignment: one base file per group, equal row counts, no overlapping ranges, and one file per column. - An existing table can turn the option from `false` to `true`. Column-file rows are excluded from record counts. Fixed bucket (`bucket > 0`) and dynamic bucket (`bucket = -1`) are allowed. Postpone bucket, lookup, `first-row`, deletion vectors, `changelog-producer=full-compaction`, Iceberg metadata, and row tracking are rejected. The read-optimized table is rejected because column files stay at level 0. The caller must preserve the base file's row order. Commit checks the row count only, so a column file in another order is accepted and silently corrupts those columns. This pull request has no SQL entry point. ### Tests `PrimaryKeyDataEvolutionTableTest` covers validation, alignment assignment, column update and replacement, compaction including a discarded compaction, both limit layers, whole-group predicate pruning, legacy-file migration, `partial-update`, `aggregation`, and dynamic bucket. `ConflictDetectionTest` covers routing to primary-key conflict detection when row tracking is off. - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dtest=PrimaryKeyDataEvolutionTableTest test` - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -DfailIfNoTests=false -Dtest=ConflictDetectionTest test` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
