JingsongLi commented on code in PR #760:
URL: https://github.com/apache/paimon-rust/pull/760#discussion_r3893379992
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -3046,7 +3313,7 @@ impl RawScoringPlan {
.collect();
for (query_index, vector_search) in vector_searches.iter().enumerate()
{
Review Comment:
[P1] Keep shared pre-filters compact in the raw fallback
For a lateral batch, every query can intentionally share the same
`Arc<Bitmap32>`, but this loop expands that bitmap into one stored query index
per `(row, query)` pair. A filter with 1M included rows and 1K queries
therefore retains roughly 1 billion `usize` entries (about 8 GiB before the
per-row `Vec` overhead) before scoring starts. The raw path also reads the full
unindexed/detail ranges before consulting this structure, so a selective
pre-filter does not reduce raw I/O.
Please preserve the shared case as one bitmap plus a query-index group (and
intersect the raw ranges with the shared/union filter before the Paimon read)
instead of materializing `O(Q×B)` associations. A regression test with many
queries sharing one sparse bitmap should assert both bounded planning memory
and raw-row pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
