YannByron opened a new pull request, #11264:
URL: https://github.com/apache/paimon/pull/11264

   ### Purpose
   
   Ray's ordinary join reads and shuffles both inputs even when a small source 
can select rows through a Paimon global index. Add the explicit 
`pypaimon.ray.indexed_lookup_join` API to query a data-evolution lookup table 
using batches of source keys.
   
   Actors reuse scalar-index readers at one fixed snapshot, read matching row 
IDs, apply residual predicates, and join the results back to the original 
source rows. Duplicate keys retain many-to-many semantics and NULL keys do not 
match. Ordinary `Dataset.join()` is unchanged.
   
   The initial API supports single integer or string keys and inner joins with 
a complete, fresh BTree or Bitmap index. It rejects missing, incomplete, stale, 
disabled, or query-authorized lookup indexes before scheduling work. 
Documentation describes snapshot lifetime and the memory implications of 
high-fanout keys.
   
   ### Tests
   
   - New indexed lookup suite: 20 passed, including Python and native readers, 
real Ray actors, index row-ID assertions, duplicate/NULL/empty input, residual 
filters, nested payloads, deletions, fixed and historical snapshots, and stale 
indexes left by a writer-only `IGNORE` update.
   - Existing `ray_bucket_join_test.py`, `ray_datasource_test.py`, 
`split_provider_test.py`, and `ray_integration_test.py`: 71 passed.
   - Project-configured flake8, all Python license headers, and `git diff 
--check`: passed.
   
   Validated locally with Python 3.13, Ray 2.59.1, PyArrow 19.0.1, and 
pypaimon-rust 0.5.0. No performance benchmark is claimed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to