JingsongLi commented on PR #9177: URL: https://github.com/apache/paimon/pull/9177#issuecomment-5254808634
Thanks for the detailed reproduction. Addressed in `129068954c`: 1. **Dedicated-file bridge correctness:** `CompactPlanner` now derives contiguous groups from normal files only, then associates BLOB/vector files with the normal file containing their first row ID. Dedicated files can no longer define normal-range connectivity. `DataEvolutionNormalCompactTask` also calls `checkContiguousRowRange` as a defensive fail-fast. The regression test follows the reported shape: two candidate normal ranges separated by an omitted oversized range, with BLOB files spanning across the gap. 2. **Rewrite-row-ids migration wording:** confirmed that there is currently no exposed Flink/Spark operation equivalent to the removed physical DV materialization path. `reassign_row_id` is metadata-only and is not a replacement. I removed the inaccurate “run a separate operation” guidance from the exception, option description, generated configuration, and user docs; they now explicitly state that data-evolution compaction preserves logical deletions and that no standalone materialization action is currently exposed. Keeping `true` unsupported is intentional here; restoring that capability would be a separate compatibility decision rather than pointing users at a nonexistent path. 3. **Manifest rescans / batching:** for manifests with row-id bounds, each `RangeBatch` now carries only manifests intersecting its candidate ranges. For legacy manifests without bounds, all candidate ranges are consolidated into one batch so the full manifest group is scanned once instead of once per candidate batch. The constant is renamed to `CANDIDATE_FILES_PER_BATCH` and documented as a soft target, since a logical range or its intersecting full metadata can exceed it. Added regression coverage for the bridge case, disjoint-task rejection, manifest pruning, and legacy one-scan behavior. The related 174 core tests, dependent Flink/Spark compile, Spotless, and `ConfigOptionsDocsCompletenessITCase` pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
