leaves12138 opened a new pull request, #9963:
URL: https://github.com/apache/paimon/pull/9963
### Purpose
Avoid materializing data-file entries when committing an equivalent,
index-only compaction on a data-evolution table.
Currently, added global-index row ranges participate in `scanBaseDataFiles`.
A large index compaction can therefore read all data manifests, retain a full
`SimpleFileEntry` list, and run data-file merge/range checks even though its
data-file delta is empty. This can consume substantial heap in the commit
process.
This draft adds a conservative metadata-only fast path before the data-file
conflict scan. It applies only when:
- The commit is `COMPACT`, the data-file delta is empty, and no historical
row-ID conflict check is configured.
- All index changes are global-index entries without deletion-vector
metadata, with both removals and additions, distinct output names, and no
duplicate input/output names.
- The union of removed and added row ranges is identical for every
partition, bucket, index type, and ordered indexed-field set. Comparing unions
rather than bounding ranges prevents filling gaps.
- Every removed input is still present in the attempt's latest index
manifest with exactly matching partition, bucket, and `IndexFileMeta`,
including row ranges and source metadata.
The full metadata comparison matters because row-ID reassignment may retain
an index file's name while rewriting its row range. The decision is recomputed
against each commit attempt's latest snapshot; it is not cached across retries.
Ineligible or changed-input cases fall back to the existing detection path.
Index-manifest combination, retained-index overlap checks, and optimistic
snapshot commit remain unchanged.
The argument for skipping data-file coverage validation is that unchanged
live input indexes already refer to valid row ranges, and an equivalent rewrite
preserves those ranges without changing any data files. No table option,
snapshot format, or index-source metadata format is changed. Index
construction, mixed data/index commits, and deletion-vector changes are not
eligible.
### Tests
- Unit coverage for equivalent merges and key-sharded outputs; input changes
across snapshots; same-name row-ID reassignment; changes to partition, bucket,
index type, indexed fields, physical metadata, and source metadata;
output-scope mismatches; range gaps; duplicate/reused names;
build-only/delete-only changes; DV changes; data-file deltas; and historical
row-ID checks.
- An end-to-end compaction test rejects any attempt to open a current data
manifest, including after an intervening append. The new path passes.
Temporarily restoring the original call site makes the same test fail while
reading data manifests; the optimized call site was restored afterwards.
- Existing stale-index-compaction-after-reassignment coverage is retained
unchanged.
Validation with JDK 8:
```text
mvn -pl paimon-core -am -Pfast-build -DwildcardSuites=none \
-Dtest=ConflictDetectionTest,DataEvolutionDeletionVectorTest,TableCommitTest,FileStoreCommitTest,IndexManifestFileHandlerTest,DataEvolutionRowIdReassignerTest,DataEvolutionGlobalIndexRefreshPlannerTest
\
-DfailIfNoTests=false package
```
All 257 tests across these seven classes passed, with no failures, errors,
or skipped tests. `spotless:check`, `checkstyle:check`, and `git diff --check`
also passed. Build the codegen loader through the package phase; switching an
existing workspace between `test` and `package` may require cleaning core's
compiled classes to avoid mixing unshaded and shaded format signatures.
No heap-size or wall-clock benchmark is claimed. The integration test
verifies elimination of data-manifest reads for eligible commits, not a
particular percentage improvement. Reading and rewriting index manifests still
has its existing cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
