JingsongLi opened a new pull request, #11238:
URL: https://github.com/apache/paimon/pull/11238
### Purpose
Add a packed R-tree global index for bounding-box queries on a single
`GEOMETRY` column in row-tracked Data Evolution append tables. Java callers can
build an `rtree` index and use `PredicateBuilder.bboxIntersects` to find
envelopes that overlap or touch a query rectangle. Empty geometries remain
available to non-null queries.
The writer packs Hilbert-ordered envelopes into checksummed V1 pages. The
reader prunes files and subtrees, coalesces adjacent page reads and searches up
to four files concurrently by default. Queries and the byte-bounded cache share
the backing array of each contiguous read. The format documentation and golden
fixtures cover file layout, row IDs, nulls and empty geometries.
An indexed bbox can participate in a compound scalar filter for ranked
queries. The evaluator tracks whether all branches contributed so a second,
unsupported predicate on the same column still gets evaluated against the data.
### Tests
- Ran 430 focused Maven tests across common, format and core, including V1
validation, WKB parsing, mixed predicates, DE scans, vector/full-text
filtering, concurrency and cache eviction.
- Verified that the Java writer reproduced the frozen V1 fixture byte for
byte, and that Java and Rust could read each other's fixture.
- Ran the Java benchmark against JTS HPRtree and STRtree on uniform points,
clustered boxes and elongated boxes. Ran the Java and Rust file readers on the
same 1,048,576 uniform points. Each worker compared complete query result
bitmaps with a brute-force scan.
- Built the documentation site and ran the root Apache RAT check.
During development, at sixteen concurrent files with 20 ms added per
physical read, sharing page buffers reduced the median sampled Java query peak
from 435.1 to 364.6 MiB. Large-window warm queries took 105.47 and 105.66 ms
respectively. This workload used 16 files, an 8 MiB cache, one query client and
three independent Corretto 8 JVMs per configuration.
The Java/JTS benchmark can be reproduced from the repository root with JDK
8, Maven and Python 3.
```sh
python3 paimon-benchmark/paimon-micro-benchmarks/run-spatial-benchmark.py \
--engines paimon-file,jts-hpr,jts-str --rows 1048576 \
--distributions uniform-points --forks 3 --queries 12 --rounds 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]