leaves12138 opened a new pull request, #9257:
URL: https://github.com/apache/paimon/pull/9257
## What changed
- Keep fragmented manifests in the bounded run-merge path instead of falling
back to external sort after 25,000 entries.
- Continue to bound the merge by `MAX_STREAM_CURSORS` and the configured
maximum file handles.
- Remove the unused row-id sort-key parameter from run-merge plan and cursor
construction.
- Rename the large-fragmented-manifest tests to reflect that they are sorted
by run merge.
## Why
The entry-count limit creates a sharp performance cliff. A section is first
scanned to discover its natural runs, but when the accumulated fragmented
manifests exceed 25,000 entries, the whole section is scanned again and
processed by the spillable external sorter.
On production snapshot 11442, only one of 69 sections fell back, but that
section contained 38 of the 119 rewritten manifests. Five fragmented manifests
accumulated 25,829 entries, just exceeding the limit and forcing the large
section through external sort. Manifest files are capped at 8 MiB, while the
number of simultaneously retained cursors remains bounded.
## Results
Production snapshot 11442, without an `-Xmx` limit:
| | Before | After |
|---|---:|---:|
| Elapsed time | 78.29 s | 24.46 s |
| Peak RSS | 5.03 GiB | 1.25 GiB |
Replaying the optimized result against the online compact result produced
`PASS_EXACT` for 32,890,232 active records.
## Validation
- `~/bin/check`
- `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false
-DwildcardSuites=none -Dtest='ManifestFileMetaTest#*RunMerge*' test`
- 14 tests passed
- Production metadata replay for snapshot 11442: `PASS_EXACT`
- `git diff --check`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
