JingsongLi commented on PR #9402: URL: https://github.com/apache/paimon/pull/9402#issuecomment-5534880849
[P1] Make the rebuild path replace legacy/incompatible indexes This change makes every pre-upgrade global-index entry (`schemaId == null`) fail closed on reads, but the build planners still count those same entries as coverage. `GlobalIndexBuilderUtils.currentIndexEntries` returns them unchanged (lines 139-169), and both Generic/Sorted incremental scans feed them into `unindexedRowRanges`; the Python path does the same in `pypaimon/globalindex/build_plan.py:61-88` and returns early from `create_global_index.py:170-178`. Therefore a fully indexed table upgraded to this version has no usable index, while rerunning the normal create/build command silently schedules no work. With an appended tail, only the tail is rebuilt and the old ranges remain raw. Filtering those entries out of coverage alone is insufficient: the manifest combiner rejects adding overlapping global-index ranges while the old entries remain. Please partition current entries into compatible/incompatible sets, compute coverage only from compatible entries, add each incompatible range to `rangesToBuild`, and emit those incompatible entries as deletes in the same commit before adding replacements. The Python planner/commit path needs the same treatment. Please add upgrade tests for both Generic and Sorted planners (and Python): start with a legacy null-schema entry covering the whole table, assert an incremental build schedules the full range plus deletion, commit successfully, and verify the replacement carries the current schema ID. Also cover a non-null entry whose indexed type changed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
