JingsongLi opened a new pull request, #8672: URL: https://github.com/apache/paimon/pull/8672
## Purpose Align source-backed primary-key index maintenance with the data LSM lifecycle. The previous independent index LSM could build payloads for data files that were replaced by the next data compaction. In the target workload, compaction rewrites nearly every file in the affected levels, so maintaining separate index fanout and stale-ratio policies adds complexity without useful reuse. ## Changes - replace the independent primary-key index LSM planner with exact data-level reconciliation - persist and validate the data level in primary-key index source metadata - maintain one complete payload per non-zero data level for BTree, Bitmap, FullText, and Vector indexes - reject partial, duplicate, cross-level, and stale payload coverage during restore and publication - make scalar reads fall back to data files when a level is uncovered while preserving existing FullText and Vector FAST semantics - remove the unreleased index fanout and stale-ratio options, validation, and documentation - update tests and user documentation for data-level maintenance The source metadata format is changed directly because the previous format has not been released. Index payload size is intentionally not capped because the amount of data in one bucket is bounded. ## Verification - 217 focused primary-key index tests passed - 49 Vector tests passed after rebasing onto the latest upstream batch-search and deletion-vector changes - paimon-core and dependencies compiled successfully with Checkstyle, Spotless, and Enforcer enabled - git diff --check passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
