JingsongLi opened a new pull request, #1015:
URL: https://github.com/apache/paimon-rust/pull/1015
### Purpose
Support building, dropping, and querying ordered multi-column BTree global
indexes from Rust. For an index on `(category, item_number)`, a predicate such
as `category IN ('a', 'b') AND item_number >= 10` can now use tuple bounds
while retaining the original predicates for data filtering.
Linked issue: None.
### Brief change log
- Encode tuple keys using Java's `RowCompactedSerializer` layout and compare
typed components directly from slices, with NULL components ordered first.
- Plan left-prefix equality, IN, NULL, and range conditions, including
deduplication, contradictory bounds, a 256-interval limit, and scan budgets per
source row-ID range group.
- Identify composite definitions by their complete ordered field IDs for
incremental builds, queries, and drops. Preserve independent scalar and
reversed-column definitions.
- Continue intersecting independent indexes for AND, combine OR branches
safely, and retain unindexed data for residual filtering in full/detail modes.
Prefer a fully covering scalar BTree or bitmap for leading-column-only
predicates.
- Normalize decimal literals exactly and decline index bounds that cannot
preserve decimal or timestamp precision.
### Tests
- `cargo test -p paimon --all-targets --features fulltext,vortex --locked`:
4,038 passed, 6 ignored, no failures.
- `cargo clippy --locked --all-targets --workspace --features
fulltext,vortex -- -D warnings`: passed.
- `cargo fmt --all -- --check` and `git diff --check`: passed.
- Regenerated nine fixtures from the current Java serializer and BTree
writer sources; all match the checked-in fixtures byte-for-byte. Rust
uncompressed V1/V2 output also matches Java byte-for-byte.
- Added 650 query-boundary cases across Java V1/V2 and NONE/LZ4 fixtures,
checked against ordinary row filtering.
- End-to-end tests cover tuple NULLs, residual predicates, incremental
builds, exact-name resolution, drops, independent indexes, and differing index
coverage under AND/OR in fast/full/detail modes.
### API and Format
- Add `.with_index_columns(&[...])` to the sorted index build and drop
builders. The list API preserves exact names; `.with_index_column(...)`
resolves an exact name first, then supports a comma-separated definition.
- Reuse ordered `extra_field_ids` metadata and Java-compatible compacted
tuple keys with existing BTree V1/V2 files. Scalar index formats remain
unchanged.
- This change assumes indexed column types stay unchanged. Schema evolution
and signed-zero/NaN query semantics are outside this PR's scope.
### Documentation
Document composite create/drop APIs, query-prefix rules, interval and scan
limits, precision fallback, definition selection, and fixture regeneration in
`docs/src/sql.md` and the BTree fixture README.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
