JingsongLi opened a new pull request, #1457:
URL: https://github.com/apache/paimon-rust/pull/1457

   ### Purpose
   
   Add a Rust reader and writer for the packed R-tree global index introduced 
in [apache/paimon#11238](https://github.com/apache/paimon/pull/11238). Tables 
using the native planar `GEOMETRY` type can build the same V1 files and use 
their envelopes for filtering.
   
   ### Brief change log
   
   - Stream WKB from snapshot-pinned table shards into Hilbert-packed files. 
Each physical file records its own row count and uses shard-relative row IDs. 
Manual and automatic builds handle multi-file shards, including cleanup after a 
partial build fails.
   - Use the manifest envelope for file pruning and read selected pages or 
row-ID blocks on demand. The reader bounds the page cache, coalesces adjacent 
reads and shares a file-concurrency limit across queries using the same read 
context.
   - Add the `bbox_intersects(shape, min_x, min_y, max_x, max_y)` DataFusion 
function and push constant bounds through the native predicate API. The 
rectangle includes its boundary. The index also answers `IS NULL` and `IS NOT 
NULL`, including the distinction between null and empty geometry. FAST searches 
indexed coverage. FULL and DETAIL also evaluate uncovered rows.
   
   ### Tests
   
   - Rust core library suite passed with 4,977 tests and 7 ignored. An initial 
concurrent run timed out in the existing disk-cache child-process test. The 
full suite passed on rerun with four test threads.
   - DataFusion library suite passed with 448 tests. The global-index 
schema-evolution test and both spatial integration tests passed, including the 
failed-build cleanup case.
   - Compared both Java V1 fixtures byte for byte. Both implementations read 
the fixtures and reproduce the mixed multilevel file. The Java R-tree suites 
passed with 54 tests. Rust also covers noncontiguous child pages, sparse 64-bit 
row IDs, invalid headers and checksums, bounded parallel reads, and partial 
index coverage.
   - Rustfmt and strict Clippy passed for the core and DataFusion crates.
   
   ### API and Format
   
   The Rust table builder and DataFusion's create/drop global index procedures 
accept `rtree`. The native filter is `PredicateBuilder::bbox_intersects`. The 
index operates on planar XY envelopes in the column's coordinate system. Both 
engines use the existing [R-tree V1 file 
format](https://github.com/apache/paimon/blob/79b8ec4788e2cd8ed87b1745d5ca54aa8db0e293/docs/docs/multimodal-table/global-index/rtree.mdx#file-format-version-1).
   
   ### Documentation
   
   Updated the SQL and automatic maintenance guides with usage, search modes, 
build settings, and read limits.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to