JingsongLi opened a new pull request, #11139:
URL: https://github.com/apache/paimon/pull/11139

   ### Purpose
   
   Repeated partial-column updates make Data Evolution compaction rewrite large 
unchanged normal base files. Add 
`data-evolution.compaction.incremental.enabled` (default `false`) to compact 
newer column-file versions while older files keep supplying untouched columns.
   
   - Select a continuous newest-first prefix using the existing file-count and 
size-ratio options. Keep files with the same sequence number together, and 
estimate redundant space from current column or leaf coverage before selecting 
a full-group rewrite.
   - Write only selected columns, resolved through historical field IDs. 
Support nested ROW leaves, column renames, explicit NULLs, and parent-struct 
nullness; ARRAY and MAP fields remain whole units.
   - Preserve complete row-ID ranges, retained deletion-vector anchors, and 
column sequence metadata. With 
`data-evolution.compaction.split-large-files=true`, eligible oversized files 
select every normal version of their range before splitting. Other groups 
continue incremental selection.
   - Add an incremental task type without changing the serializer version or 
existing task encoding. New binaries restore existing tasks; older binaries 
cannot restore an in-flight incremental task. Snapshot and data-file formats 
are unchanged.
   
   The existing nested-read limitations and separate BLOB/VECTOR compaction 
options still apply. Document the selection policy, splitting rules, and job 
downgrade constraint.
   
   ### Tests
   
   Validated on JDK 8 with standard Checkstyle, Spotless, and Maven Enforcer 
checks enabled.
   
   - Broad regression run: **653 core cases, zero failures/errors, one existing 
skip**, plus **24 common cases passed**.
   - Focused incremental compaction class: **21 cases passed**, covering 
default behavior, retained base files, adjacent groups, concurrent updates, 
historical snapshots, schema changes, nested NULL/projection semantics, nested 
collections, deletion vectors, dedicated-file boundaries, and oversized-file 
splitting with nested updates.
   
   ```sh
   mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
     
'-Dtest=*DataEvolution*Test,CompactCandidateRangeCollectorTest,VectorStoreTableTest,BlobTableTest,MultipleBlobTableTest,ConflictDetectionTest'
 test
   
   mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
     -Dtest=DataEvolutionIncrementalCompactionTest test
   ```
   
   Full Spark and Flink engine suites were not run.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to