JingsongLi commented on PR #10146: URL: https://github.com/apache/paimon/pull/10146#issuecomment-5808203774
Requirement fit: SUPPORTED. The reported hundred-billion-row PoC and billion-row index result give this feature concrete end-to-end value. Implementation: FINDINGS. [P1] Protect deferred index files (`paimon-core/src/main/java/org/apache/paimon/globalindex/GlobalIndexScanPlan.java:308`). The recorded index path is opened only when a reader starts. If an index is replaced and the planning snapshot expires while its data file stays current, a queued or checkpoint-restored split can start after cleanup deleted that index and fail. Read-protection tags are optional and unset by default; the tagged-snapshot test does not cover this path. Pin the index/snapshot through split completion, or safely fall back to a full split scan with residual filtering. [P2] Preserve bounded range planning (`GlobalIndexScanPlan.java:207`). Each side of an `AND` range is planned as a half-infinite predicate. Both can separately exceed the default 256 MiB BTree file budget even when their intersection fits. The eager evaluator pairs these bounds into `visitRange`; this mode falls back to a full data scan. Pair compatible bounds before applying the budget. Before production rollout, define the rollback procedure: an older binary cannot restore a checkpoint containing `LazyIndexedSplit`, and disabling the option does not convert pending splits. Drain them or restart from compatible state before rollback. Verification: CI succeeded on this head. Locally, 139 focused common tests, 37 `GlobalIndexScanPlanTest` cases, 27 other `LazyIndexedSplitTest` cases, and 3 `LazyIndexSourceTest` cases passed. Four partition-index cases could not complete because this local JVM failed to load the existing `CodeGenerator` service; this is not evidence of a PR failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
