shyjsarah commented on code in PR #760:
URL: https://github.com/apache/paimon-rust/pull/760#discussion_r3892309487
##########
crates/integrations/datafusion/src/lateral_vector_search.rs:
##########
@@ -398,17 +505,35 @@ impl LateralVectorSearchExec {
return empty_batch(self.output_schema.clone());
}
- let mut builder = self.target_table.new_batch_vector_search_builder();
- let results = builder
+ let (target_table, include_row_ids) = match &self.filter {
+ Some(filter) => {
+ let prepared = self
+ .prepared_filter
+ .get_or_try_init(|| {
+ self.target_table
+ .prepare_vector_search_filter(filter.clone())
+ })
+ .await
+ .map_err(to_datafusion_error)?;
+ (
+ prepared.table(),
Review Comment:
Fixed in 3b69116. The plan-lifetime OnceCell is replaced by an
execution-scoped weak cache: partitions and batches using the same live
TaskContext share one prepared filter, while dropping the execution streams
releases it so re-executing the physical plan prepares against a fresh
snapshot. Added a regression test that reuses the same physical plan after
appending and indexing a new target row.
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -1634,6 +1758,43 @@ async fn evaluate_batch_vector_search(
let index_search_limit = indexed_search_limit(max_limit, refine_factor)?;
let vector_entry_count = vector_entries.len();
+ let shared_include_row_ids =
+ vector_searches[0]
+ .effective_include_row_ids()
+ .filter(|include_row_ids| {
+ vector_searches
+ .iter()
+ .all(|search| search.effective_include_row_ids() ==
Some(*include_row_ids))
Review Comment:
Fixed in dfd45ec. Shared-filter detection now inspects
shared_include_row_ids and requires Arc::ptr_eq with the first query Arc. Owned
bitmaps, equal-but-distinct Arcs, and mixed inputs fall back to per-query
localization. Added unit coverage for all three cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
