JingsongLi opened a new pull request, #452: URL: https://github.com/apache/paimon-rust/pull/452
## Summary Adds DataFusion support for `CROSS JOIN LATERAL vector_search(...)` when the query vector comes from the left input column, planning it as batched vector search per input batch. The existing literal JSON `vector_search` path remains unchanged. ## Changes - Registers a custom optimizer rule and query planner in `SQLContext` to rewrite supported lateral joins into `LateralVectorSearchExec`. - Extends the `vector_search` UDTF with a lateral marker provider for column query vector arguments. - Executes batch vector search per left record batch, reads target rows by `_ROW_ID`, and emits joined left/right rows. - Adds DataFusion coverage with per-row query vectors that produce different top-k matches. ## Testing - [x] `cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_lateral_join_uses_query_vectors -- --nocapture` - [x] `cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_java_vindex_table -- --nocapture` - [x] `cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_without_matching_index_returns_empty -- --nocapture` - [x] `cargo test -p paimon table::vector_search_builder -- --nocapture` - [x] `cargo fmt --all --check` - [x] `git diff --check` - [x] `cargo clippy -p paimon-datafusion --test read_tables -- -D warnings` ## Notes - First version supports `CROSS JOIN LATERAL vector_search(...)` / inner joins with no join predicate. Unsupported shapes continue to fail through the marker provider scan path. - The lateral query vector currently must be a column expression yielding `List<Float32>` or `FixedSizeList<Float32>`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
