YannByron opened a new pull request, #11264: URL: https://github.com/apache/paimon/pull/11264
### Purpose Ray's ordinary join reads and shuffles both inputs even when a small source can select rows through a Paimon global index. Add the explicit `pypaimon.ray.indexed_lookup_join` API to query a data-evolution lookup table using batches of source keys. Actors reuse scalar-index readers at one fixed snapshot, read matching row IDs, apply residual predicates, and join the results back to the original source rows. Duplicate keys retain many-to-many semantics and NULL keys do not match. Ordinary `Dataset.join()` is unchanged. The initial API supports single integer or string keys and inner joins with a complete, fresh BTree or Bitmap index. It rejects missing, incomplete, stale, disabled, or query-authorized lookup indexes before scheduling work. Documentation describes snapshot lifetime and the memory implications of high-fanout keys. ### Tests - New indexed lookup suite: 20 passed, including Python and native readers, real Ray actors, index row-ID assertions, duplicate/NULL/empty input, residual filters, nested payloads, deletions, fixed and historical snapshots, and stale indexes left by a writer-only `IGNORE` update. - Existing `ray_bucket_join_test.py`, `ray_datasource_test.py`, `split_provider_test.py`, and `ray_integration_test.py`: 71 passed. - Project-configured flake8, all Python license headers, and `git diff --check`: passed. Validated locally with Python 3.13, Ray 2.59.1, PyArrow 19.0.1, and pypaimon-rust 0.5.0. No performance benchmark is claimed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
