JingsongLi opened a new pull request, #11139:
URL: https://github.com/apache/paimon/pull/11139
### Purpose
Repeated partial-column updates make Data Evolution compaction rewrite large
unchanged normal base files. Add
`data-evolution.compaction.incremental.enabled` (default `false`) to compact
newer column-file versions while older files keep supplying untouched columns.
- Select a continuous newest-first prefix using the existing file-count and
size-ratio options. Keep files with the same sequence number together, and
estimate redundant space from current column or leaf coverage before selecting
a full-group rewrite.
- Write only selected columns, resolved through historical field IDs.
Support nested ROW leaves, column renames, explicit NULLs, and parent-struct
nullness; ARRAY and MAP fields remain whole units.
- Preserve complete row-ID ranges, retained deletion-vector anchors, and
column sequence metadata. With
`data-evolution.compaction.split-large-files=true`, eligible oversized files
select every normal version of their range before splitting. Other groups
continue incremental selection.
- Add an incremental task type without changing the serializer version or
existing task encoding. New binaries restore existing tasks; older binaries
cannot restore an in-flight incremental task. Snapshot and data-file formats
are unchanged.
The existing nested-read limitations and separate BLOB/VECTOR compaction
options still apply. Document the selection policy, splitting rules, and job
downgrade constraint.
### Tests
Validated on JDK 8 with standard Checkstyle, Spotless, and Maven Enforcer
checks enabled.
- Broad regression run: **653 core cases, zero failures/errors, one existing
skip**, plus **24 common cases passed**.
- Focused incremental compaction class: **21 cases passed**, covering
default behavior, retained base files, adjacent groups, concurrent updates,
historical snapshots, schema changes, nested NULL/projection semantics, nested
collections, deletion vectors, dedicated-file boundaries, and oversized-file
splitting with nested updates.
```sh
mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
'-Dtest=*DataEvolution*Test,CompactCandidateRangeCollectorTest,VectorStoreTableTest,BlobTableTest,MultipleBlobTableTest,ConflictDetectionTest'
test
mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
-Dtest=DataEvolutionIncrementalCompactionTest test
```
Full Spark and Flink engine suites were not run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]