JingsongLi commented on code in PR #760:
URL: https://github.com/apache/paimon-rust/pull/760#discussion_r3893811978
##########
crates/paimon/src/vindex/reader.rs:
##########
@@ -421,16 +421,37 @@ fn search_vindex(
Ok(Some(id_to_scores))
}
-#[derive(Clone, PartialEq, Eq)]
+#[derive(Clone)]
struct PreparedSearch {
params: VectorSearchParams,
- filter_bytes: Option<Vec<u8>>,
+ filter_bytes: Option<Arc<[u8]>>,
}
+impl PreparedSearch {
+ fn same_batch_group(&self, other: &Self) -> bool {
+ self.params == other.params
+ && match (&self.filter_bytes, &other.filter_bytes) {
+ (Some(left), Some(right)) => Arc::ptr_eq(left, right),
Review Comment:
[P2] Keep batching equal filters supplied through the public API
`Arc::ptr_eq` only groups the new crate-internal shared-filter path.
External callers of `VindexVectorGlobalIndexReader::visit_batch_vector_search`
can only build filters with public `VectorSearch::with_include_row_ids`; two
equal filters (or clones made through that API) are serialized into distinct
`Arc<[u8]>` values, so this comparison puts every query in a singleton group.
Those groups take `execute_scalar_search`, repeating native setup and
probed-list I/O Q times. Before this commit, `PreparedSearch` equality compared
the serialized bytes and these queries used the filtered native batch. The
updated `filtered_batch_matches_scalar_searches_and_reuses_probed_lists` test
switches from two owned public filters to the crate-private shared setter, so
it no longer covers that public path. Please retain a pointer fast path but
fall back to content fingerprint/equality (or expose a public shared-filter
API), and restore coverage using separate `with_include_row_ids` filters while
ass
erting native batching/I/O reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
