JingsongLi opened a new pull request, #766: URL: https://github.com/apache/paimon-rust/pull/766
### Purpose Add an exact substring global index for character-string columns, with Rust reader/writer compatibility for the Java Paimon FM-index V1 format. Linked issue: N/A ### Brief change log - implement partitioned FM-index V1 writing, reading, suffix-array construction, compression, checksums, caching, and bounded I/O - integrate `fm` with global-index build, scan, create/drop procedures, predicate planning, coverage fallback, and coexistence with BTree indexes - stream FM source batches, move CPU-heavy partition encoding off Tokio workers, and clean partial/uncommitted files on failure - validate Java multi-file row coverage and compressed output sizes before allocation - add a Java production-writer golden generator and require byte-for-byte Rust compatibility ### Tests - `cargo test -p paimon --lib` — 2499 passed, 2 ignored - `cargo test -p paimon-datafusion --test procedures` — 32 passed - `cargo clippy -p paimon -p paimon-datafusion --all-targets -- -D warnings -A clippy::nonminimal-bool` - `cargo fmt --all -- --check` - `./scripts/generate-fm-index-v1-golden.sh --check` ### API and Format Adds `index_type => fm` for `create_global_index` and `drop_global_index`. The persisted FM-index format is byte-compatible with Java Paimon V1; the checked-in fixture is generated by the Java production writer and verified by the Rust reader and writer. ### Documentation Updated the SQL global-index guide with FM usage, supported predicates, configuration defaults, fallback behavior, and drop support. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
