JingsongLi opened a new pull request, #1457: URL: https://github.com/apache/paimon-rust/pull/1457
### Purpose Add a Rust reader and writer for the packed R-tree global index introduced in [apache/paimon#11238](https://github.com/apache/paimon/pull/11238). Tables using the native planar `GEOMETRY` type can build the same V1 files and use their envelopes for filtering. ### Brief change log - Stream WKB from snapshot-pinned table shards into Hilbert-packed files. Each physical file records its own row count and uses shard-relative row IDs. Manual and automatic builds handle multi-file shards, including cleanup after a partial build fails. - Use the manifest envelope for file pruning and read selected pages or row-ID blocks on demand. The reader bounds the page cache, coalesces adjacent reads and shares a file-concurrency limit across queries using the same read context. - Add the `bbox_intersects(shape, min_x, min_y, max_x, max_y)` DataFusion function and push constant bounds through the native predicate API. The rectangle includes its boundary. The index also answers `IS NULL` and `IS NOT NULL`, including the distinction between null and empty geometry. FAST searches indexed coverage. FULL and DETAIL also evaluate uncovered rows. ### Tests - Rust core library suite passed with 4,977 tests and 7 ignored. An initial concurrent run timed out in the existing disk-cache child-process test. The full suite passed on rerun with four test threads. - DataFusion library suite passed with 448 tests. The global-index schema-evolution test and both spatial integration tests passed, including the failed-build cleanup case. - Compared both Java V1 fixtures byte for byte. Both implementations read the fixtures and reproduce the mixed multilevel file. The Java R-tree suites passed with 54 tests. Rust also covers noncontiguous child pages, sparse 64-bit row IDs, invalid headers and checksums, bounded parallel reads, and partial index coverage. - Rustfmt and strict Clippy passed for the core and DataFusion crates. ### API and Format The Rust table builder and DataFusion's create/drop global index procedures accept `rtree`. The native filter is `PredicateBuilder::bbox_intersects`. The index operates on planar XY envelopes in the column's coordinate system. Both engines use the existing [R-tree V1 file format](https://github.com/apache/paimon/blob/79b8ec4788e2cd8ed87b1745d5ca54aa8db0e293/docs/docs/multimodal-table/global-index/rtree.mdx#file-format-version-1). ### Documentation Updated the SQL and automatic maintenance guides with usage, search modes, build settings, and read limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
