JingsongLi opened a new pull request, #9709:
URL: https://github.com/apache/paimon/pull/9709
### Purpose
Data-evolution compaction skips an oversized file when the file count is
below the compaction threshold, and its normal rewrite path produces only one
output file. This prevents resizing existing large files after lowering
`target-file-size`.
Add `data-evolution.compaction.split-large-files` (default: `false`). When
enabled, any normal file strictly larger than twice `target-file-size` can
trigger compaction independently of `compaction.min.file-num`, and normal
compaction output rolls near the target size. Existing small-file merging
remains available; the combined size of multiple versions alone does not bypass
the minimum file count.
Preserve row IDs, column sequence metadata, and logical deletions.
Redistribute deletion vectors across output files and rewrite associated
BLOB/VECTOR files within the new normal-file boundaries, resolving dedicated
columns by field ID. Document the option, its additional dedicated-file I/O,
and historical snapshot retention.
### Tests
84 targeted tests passed on JDK 17, including:
- Physical-size threshold boundaries, minimum-file-count behavior, and
overflow protection.
- Historical-file splitting with and without column updates, task
serialization, data preservation, and repeated planning.
- Multiple BLOB columns and a renamed VECTOR column with aligned output
ranges and row-by-row value checks.
- Both deletion-vector bitmap formats, deleted-row positions, row IDs, and
existing compaction regressions.
```sh
mvn -pl paimon-core -am \
-DwildcardSuites=none -DfailIfNoTests=false \
-Dtest=DataEvolutionCompactCoordinatorTest,CompactCandidateRangeCollectorTest,DataEvolutionCompactRangePlannerTest,DataEvolutionNormalCompactTaskTest,DataEvolutionDeletionVectorTest
\
test
```
Run without `fast-build`; Checkstyle, Spotless, and Maven Enforcer passed.
The existing Byte Buddy agent was preloaded for Mockito in the local sandbox.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
