leaves12138 opened a new pull request, #9743:
URL: https://github.com/apache/paimon/pull/9743
### Purpose
Row-ID filters can retain manifests whose coarse min/max overlaps the query
even when only a few Avro blocks contain relevant entries. Add optional row-id
sidecars that select complete blocks before the existing manifest entry
filtering and ADD/DELETE merge.
- Store block byte ranges, entry ordinals, and bounded row-id interval
unions, covering ADD/DELETE entries and all column groups. Use block min/max
before exact interval matching.
- Record a nullable `indexFileName` in Java and PyPaimon manifest metadata.
Old metadata reads as null without probing for an index; legacy manifest-list
readers can ignore the added field.
- Validate the sidecar version, completeness, manifest identity, lengths and
checksum. Missing, invalid or over-budget indexes fall back to the original
read path. Partial reads bypass the full-manifest cache.
- Publish the reference after the sidecar closes, and follow explicit
references through rolling/rewrite, failed-commit cleanup,
snapshot/tag/changelog retention and orphan cleanup.
Both `manifest.row-id-index.write` and `manifest.row-id-index.read` default
to false. Java builds each index by projecting the final manifest after it
closes; existing immutable manifests are not backfilled.
### Tests
Java 8, with normal Maven checks enabled: 139 tests passed.
```sh
mvn -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false \
'-Dtest=CoreOptionsTest,ManifestFileMetaSerializerTest,ManifestListTest,ManifestFileTest,ManifestRowIdIndexTest,ManifestFileMergerTest,ManifestRewriteCleanupTest,ExpireSnapshotsTest#testSidecarsFollowSnapshotAndTagRetention,LocalOrphanFilesCleanTest#testOrphanCleanupProtectsReferencedSidecars'
test
```
Python: 44 tests passed.
```sh
PYTHONPATH=paimon-python python3 -m pytest \
paimon-python/pypaimon/tests/manifest/row_id_index_test.py \
paimon-python/pypaimon/tests/file_store_commit_test.py -q
```
Coverage includes shared Java/Python format fixtures, 64-bit interval
boundaries, block offsets and ordinals, selective body reads, ADD/DELETE
correctness, legacy metadata compatibility, explicit index names,
no-index/no-query behavior, conservative fallback, rolling/rewrite failures,
commit cleanup, snapshot/tag protection and orphan collection. The manifest
tests also retain the upstream cache-size and bucket-filter regression coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
