zjw1111 opened a new pull request, #368: URL: https://github.com/apache/paimon-cpp/pull/368
### Purpose Closes #367. Global Index score completion currently consumes scores sequentially after deletion-vector or predicate filtering has removed candidate rows. A surviving row can therefore receive another row's score; for example, filtering out row ID 102 can give row ID 104 the score 102.5 instead of 104.5. Match scores to surviving global row IDs. Add `_ROW_ID` to the internal read schema when needed and remove it before returning batches, while preserving explicit row-ID projections and keeping the shared read schema unchanged. Indexed splits without scores continue to return null `_INDEX_SCORE` values. Validate scored row ranges before constructing the lookup, rejecting invalid endpoints, overlaps and score-count mismatches. This change covers Global Index; File Index behavior is outside its scope. ### Tests - `CompleteIndexScoreBatchReaderTest.*`: sparse and non-monotonic row IDs, bitmap filtering, multiple batch sizes, both row-ID projections, unselected null scores, and invalid score/schema inputs. - `IndexedSplitTest.*`: range overlap, invalid endpoints, boundary values and score counts. - `DataEvolutionSplitReadTest.*`: existing split-reader coverage. - `DataEvolutionTableTest.TestGlobalIndexScoresWithSingleFileFiltering` and `TestGlobalIndexScoresWithMergedFileFiltering`: Global Index combined with DV, predicates and both together; with/without `_ROW_ID`; empty results; scored and unscored splits reused with the same `TableRead`. Parameterized over Parquet/ORC/Avro and manifest-cache settings. - Final local validation: 35 focused unit tests, 138 Global Index integration tests and 216 Data Evolution integration tests passed. The Data Evolution suite skipped 18 existing cases because of format capabilities. - Changed-file pre-commit checks and `git diff --check` passed. - Regression control using the original sequential-score path failed all 12 new integration cases with score mismatches. Debug builds used ASan with `ASAN_OPTIONS=detect_leaks=0` because LeakSanitizer cannot scan `/proc` in the sandbox. The complete repository test suite and optional Lumina/Lucene/Tantivy/Mosaic backend configurations were not run. Commands: ```bash ASAN_OPTIONS=detect_leaks=0 ./build/debug/score-alignment-test ASAN_OPTIONS=detect_leaks=0 ./build/debug/paimon-global-index-test ASAN_OPTIONS=detect_leaks=0 ./build/debug/paimon-data-evolution-table-test ``` The focused unit executable links the existing score-reader, indexed-split and data-evolution split-reader test translation units. ### API and Format No public API under `include/`, storage format or protocol changes. The internal `CompleteIndexScoreBatchReader` constructor now accepts scores keyed by global row ID. ### Documentation No new user-facing feature or configuration. Internal reader preconditions and score-alignment behavior are documented in the class header. ### Generative AI tooling Generated-by: OpenAI Codex CLI 0.154.0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
