wangyong9999 opened a new pull request, #9460: URL: https://github.com/apache/paimon/pull/9460
### Purpose Close #9458. A source-backed primary-key sorted-index payload keeps one immutable, ordered source-file list as its row-ordinal namespace. After a later snapshot retires only part of that list, the current Java and Python readers compare the payload with the complete active file set at that level. The mismatch rejects the still-valid payload, so surviving indexed files fall back to raw scans. **Key code and data-flow changes** - Keep the complete payload source list for group-ordinal localization, while deriving snapshot coverage from its active intersection at the metadata-declared data level. - Preserve the existing one-group-per-level invariant; ambiguous payload candidates at one level still fall back instead of introducing a new lifecycle. - Bind scan-side source lookup to both the group data level and source identity so a file that moves to another level cannot inherit a stale group. - Leave newly active or otherwise uncovered files on the existing raw-scan path. - Apply the same lifecycle and scan-planning semantics to Java core and Python. The equivalent C++ reader correction is tracked by [apache/paimon-cpp#253](https://github.com/apache/paimon-cpp/pull/253). **Error and compatibility boundaries** - Reject malformed or misordered source metadata, active row-count mismatches, zero-active intersections, and multiple valid candidates for one level. - Preserve the payload's retired prefix or middle sources so surviving files retain their original group ordinal offsets. - No public API, storage format, index protocol, writer lifecycle, or concurrency behavior changes. ### Tests - Java: `mvn -pl paimon-core -Pfast-build -DwildcardSuites=none -Dtest=PkSortedBucketIndexStateTest,PrimaryKeySortedIndexScanTest test` — 20 tests passed. - Python: `pytest paimon-python/pypaimon/tests/primary_key_sorted_index_scan_test.py -q` — 7 tests passed, including five invalid-payload subcases. - Java Spotless, Python flake8 and py_compile, and `git diff --check` passed. - Added coverage for retired-source ordinal offsets, newly active raw fallback, row-count mismatch, misordered metadata, zero-active payloads, ambiguous candidates, and cross-level source movement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
