JingsongLi commented on code in PR #9802:
URL: https://github.com/apache/paimon/pull/9802#discussion_r4003543917
##########
paimon-core/src/main/java/org/apache/paimon/operation/ManifestFileSorter.java:
##########
@@ -862,18 +888,6 @@ private static void rewriteSections(
for (int i = 0; i < sections.size(); i++) {
Section section = sections.get(i);
- // A single-file section is always handled directly, regardless of
the budget.
- if (section.files.size() == 1) {
- rewriteSection(
- section.files,
- output,
- sortNewFiles,
- ctx,
- manifestFile,
- manifestReadParallelism);
- continue;
- }
-
// Phase 1: budget not yet exhausted -- perform aggressive sort
rewrite.
Review Comment:
[P2] Do not charge unchanged singleton sections against the rewrite budget
Removing this shortcut also changes ordinary commits (`fullSort=false`).
`rewriteSection` still returns a singleton unchanged when it does not require
delete cleanup, but `rewriteSections` now charges its size and can set
`budgetExhausted=true`. Subsequent overlapping sections without
default-compaction files are then skipped, even though no manifest has actually
been rewritten.
I reproduced this with six ADD-only manifests, each with `fileSize = 8 MiB`:
the first contains partition 0, and the other five have overlapping partition
ranges [1, 2]. With `bucket=-1`, manifest sorting enabled, and a 4 MiB rewrite
budget, the picker selects all five runs, but the unchanged first singleton
exhausts the budget and all six manifests are returned unchanged. Restoring the
original singleton shortcut for `!ctx.fullSort` makes the regression test pass
and allows the overlapping manifests to be merged.
Please preserve the existing singleton handling for ordinary compaction, or
only deduct budget for files that are actually rewritten, while retaining
forced singleton rewriting for explicit full sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
