Stephen0421 opened a new pull request, #11065:
URL: https://github.com/apache/paimon/pull/11065

   ### Purpose
   
   Part of #11064 
   
   
[PIP-44](https://cwiki.apache.org/confluence/display/PAIMON/PIP-44%3A+Support+Data+Evolution+for+Primary+Key+Tables)
 lets a primary key table update selected columns without rewriting the whole 
row. The 
[discussion](https://lists.apache.org/thread/7dpn89ndn0xsp2l9j186zr0br6kpx8c9) 
agreed to reuse Data Evolution alignment, with `firstRowId` as an alignment id 
rather than row tracking. This pull request is the foundation agreed there.
   
   A primary key table can enable `data-evolution.enabled` without 
`row-tracking.enabled`.
   
   - A newly written full row receives `firstRowId` and `rowCount` at commit. 
No `_ROW_ID` column is added.
   - `ColumnFileWriter` commits one column file for one existing base file 
through `compactIncrement` (`CommitKind.COMPACT`). The file keeps the base 
file's `(partition, bucket, firstRowId, rowCount)`, contains only non-key and 
non-partition columns, and stays at level 0.
   - Reads stitch that column file onto the base file before the merge function 
runs. Sequence and value kind come from the base file, so the column update 
does not become a newer record. `deduplicate`, `partial-update`, and 
`aggregation` see an ordinary full row.
   - Predicates apply to the whole alignment group. A split containing a column 
file is not raw-convertible. Limit pushdown keeps that bucket at both the 
manifest scan and the split plan.
   - Compaction rewrites the base file together with its column files and 
assigns a new alignment range. A discarded compaction keeps the column files.
   - Commit keeps primary-key conflict detection and checks alignment: one base 
file per group, equal row counts, no overlapping ranges, and one file per 
column.
   - An existing table can turn the option from `false` to `true`. Column-file 
rows are excluded from record counts.
   
   Fixed bucket (`bucket > 0`) and dynamic bucket (`bucket = -1`) are allowed. 
Postpone bucket, lookup, `first-row`, deletion vectors, 
`changelog-producer=full-compaction`, Iceberg metadata, and row tracking are 
rejected. The read-optimized table is rejected because column files stay at 
level 0.
   
   The caller must preserve the base file's row order. Commit checks the row 
count only, so a column file in another order is accepted and silently corrupts 
those columns. This pull request has no SQL entry point.
   
   ### Tests
   
   `PrimaryKeyDataEvolutionTableTest` covers validation, alignment assignment, 
column update and replacement, compaction including a discarded compaction, 
both limit layers, whole-group predicate pruning, legacy-file migration, 
`partial-update`, `aggregation`, and dynamic bucket. `ConflictDetectionTest` 
covers routing to primary-key conflict detection when row tracking is off.
   
   - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none 
-DfailIfNoTests=false -Dtest=PrimaryKeyDataEvolutionTableTest test`
   - `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none 
-DfailIfNoTests=false -Dtest=ConflictDetectionTest test`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to