shyjsarah commented on code in PR #760:
URL: https://github.com/apache/paimon-rust/pull/760#discussion_r3900700272
##########
crates/paimon/src/table/vector_search_builder.rs:
##########
@@ -196,6 +197,33 @@ pub struct BatchVectorSearchBuilder<'a> {
options: HashMap<String, String>,
projection: Option<Vec<String>>,
filter: Option<Predicate>,
+ include_row_ids: Option<Arc<RoaringTreemap>>,
+ prepared_filter: Option<PreparedVectorSearchFilter>,
+}
+
+/// A scalar vector pre-filter resolved once against one pinned snapshot.
+///
+/// Reusing this value avoids repeating the same scalar-index/table read for
+/// every input batch of a lateral vector query.
+#[derive(Debug, Clone)]
+pub struct PreparedVectorSearchFilter {
+ table: Table,
+ include_row_ids: Arc<RoaringTreemap>,
+}
+
+impl PreparedVectorSearchFilter {
+ pub fn table(&self) -> &Table {
+ &self.table
+ }
+
+ pub fn include_row_ids(&self) -> &Arc<RoaringTreemap> {
+ &self.include_row_ids
+ }
+}
+
+fn same_vector_search_table(left: &Table, right: &Table) -> bool {
Review Comment:
Fixed in `45da2d0`. Each `FileIOBuilder::build` now creates a fresh private
storage-lineage token, while `FileIO::clone` and all table copy paths preserve
it automatically. Prepared-filter validation now requires matching FileIO
lineage in addition to normalized location and branch, so equal URI text cannot
retarget a builder across isolated memory/custom operators. Added a public
prepare-path regression using two independent memory FileIOs at the same
location, plus a unit test proving only FileIO clones share lineage. `paimon`
lib tests (2482 passed, 2 ignored), DataFusion lateral tests, formatting, and
clippy all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
