JunRuiLee opened a new pull request, #479:
URL: https://github.com/apache/paimon-rust/pull/479
### Purpose
Rust's `create_global_index` (and the lumina/vindex vector-index build
builders) diverged from Java: each planned shards over the **full** data
coverage and then errored on overlap with any existing index for the same
field. As a result a global index was effectively build-once — a second build
(e.g. after appending data) failed with `"Existing global index file '...'
overlaps requested row range"`, and the only way to refresh was drop + full
rebuild.
Java's `create_global_index` is incremental by construction for **both** the
sorted (btree) and the vector builders: it builds only the previously-unindexed
row ranges (`gap = [0, nextRowId-1] − indexedRowRanges(index_type, field)`),
appends new non-overlapping index files, and is a clean no-op when the gap is
empty (`SortedGlobalIndexBuilder.incrementalScan()` and
`GlobalIndexBuilderUtils.unindexedRowRanges` via `GenericIndexTopoBuilder`'s
`autoIncremental`, both keyed on the same `indexedRowRanges` helper). This PR
brings all three Rust build builders to that behavior.
### Brief change log
- **Shared range primitives** (`table/source.rs`,
`table/global_index_build_common.rs`):
- `exclude_row_ranges(base, subtract)` — inclusive `RowRange`
set-difference (`gap = coverage − indexed`), replacing a previously-untested
private copy in `global_index_scanner.rs` so the read-side `unindexed_ranges`
and the write-side build gap share one, now unit-tested, subtraction (handles
the `i64::MAX` segment-exhaustion edge).
- `indexed_row_ranges(table, manifest, index_type, field_id,
extra_field_ids)` — reads the index manifest and returns coverage for one
global-index identity, mirroring Java's single shared `indexedRowRanges`; used
by all three builders.
- **Incremental build** for `btree`, `lumina`, and `vindex` builders:
`execute()` computes the indexed coverage for its own index type, subtracts it
from the data coverage, clamps each coverage segment to the gap **before**
shard-splitting (keeps every shard's row-id range contiguous even when the gap
starts mid–grid cell), and returns `Ok(0)` when fully indexed. Build stays
**strictly append-only** — no existing index file is read, rewritten, or merged.
- **Overlap guards keyed on full identity**: the three duplicated builder
`validate_existing_index_overlap` copies are consolidated into one shared
guard, and both commit-layer guards
(`TableCommit::validate_global_index_overlap` and
`validate_added_global_index_overlap`) are narrowed, so overlap identity
everywhere is `(index_type, index_field_id, same_extra_field_ids)` + range —
matching the coverage key and Java's `GlobalIndexIdentifier`.
Different-identity global indexes on the same field now coexist (e.g. a
`lumina` and an `ivf-flat` index on the same embedding column); two files of
the **same** identity overlapping are still rejected.
- `exclude_row_ranges` is scoped `pub(crate)` (no external consumer).
Not in scope: full-text (`FullTextSearchBuilder` is a search/read path with
no build builder — nothing to make incremental); composite `extra_field_ids`
index *build* (unimplemented in both Rust and Java, though the guards are now
correct for it).
### Tests
All added tests drive the real build path (write → commit → build → read
back the index manifest); vindex/btree run fully in CI, lumina's native-lib
build is `#[ignore]` so its incremental behavior is pinned by planner-level
tests.
- `exclude_row_ranges`: 10 unit tests
(empty/prefix/mid-hole/full-cover/adjacent/multi-segment/`i64::MAX` boundary).
- Per builder (btree, lumina, vindex): second-build-without-new-data is a
no-op (returns 0, manifest unchanged); incremental build indexes only the
appended gap and retains prior index files (vindex asserts this end-to-end);
first build equals full coverage; planner-level exact-range tests for a
mid-coverage hole split and an indexed-prefix suffix (both bounds pinned, none
spans the hole, union == coverage − indexed).
- Overlap identity: a build over a field with a pre-existing
**different**-type index succeeds (no spurious overlap); commit guards allow
different-type / different-extra overlaps and still reject same-identity
overlaps; pre-existing same-type overlap-rejection tests unchanged.
- Full suite: `cargo test -p paimon --lib` 1270 passed / 1 ignored; `cargo
test -p paimon-datafusion --test procedures` 22 passed; `cargo fmt --all
--check` and `cargo clippy -p paimon --all-targets -- -D warnings` clean.
### API and Format
No storage-format change (same `GlobalIndexMeta` / `IndexFileMeta` layout,
same `global-index.row-count-per-shard` shard unit). No public API signature
change (`Table::new_*_index_build_builder` unchanged); the only behavioral
change is that a repeat `create` now appends the gap instead of erroring.
`exclude_row_ranges` is internal (`pub(crate)`).
### Documentation
No doc/SQL changes required — this aligns existing `create` behavior with
Java; no new option or syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
