JingsongLi opened a new pull request, #11227:
URL: https://github.com/apache/paimon/pull/11227

   ### Purpose
   
   When a Data Evolution vector index is available but scalar indexes cannot 
evaluate the filter, run ANN once per query/shard and check its candidates 
against the pinned data. Remove the immediate raw-scoring fallback for indexed 
ranges.
   
   Add `vector-index.post-filter.candidates-factor`, a positive integer 
defaulting to `10`. Candidate count is `max(original Top-K * factor, required 
reranking limit)`, capped by shard size and backend bounds. Filtered results 
may contain fewer than Top-K rows, including zero; they are returned without 
retries or automatic raw recovery.
   
   Scalar candidate checks run on the execution worker. Large batches are 
processed in bounded query chunks, with one ANN request per input query and 
compact retained scores. The reader is reused across chunks; asynchronous 
continuations avoid single-permit executor deadlocks and preserve cleanup on 
errors/cancellation. Exact scalar prefilters, snapshot/DV handling, reranking 
and explicit Raw sources remain supported; automatically planned scalar gaps do 
not duplicate indexed work.
   
   This aligns with [the Rust 
implementation](https://github.com/apache/paimon-rust/pull/1439). There are no 
persisted table/index format changes. Raw split provenance is an additive 
serialization field; older splits default to authoritative Raw work. Updated 
vector filtering documentation describes the default and short-result semantics.
   
   ### Tests
   
   - `mvn -pl paimon-core -am -Dflink.forkCount=2 -DfailIfNoTests=false 
-DwildcardSuites=none '-Dtest=*Vector*Test,*GlobalIndex*Test,*Hybrid*Test' 
test`: passed (522 core tests, plus dependency-module tests).
   - Final focused 
`VectorSearchPostFilterTest,VectorSearchRowFilterExactnessTest`: 34 passed, 
including the additional historical-snapshot/DV regression.
   - Standard Maven checks, including Spotless/checkstyle, passed without the 
fast-build profile. The fresh worktree's codegen loader was packaged before the 
broad run.
   - Tests cover small/large factors and short results, mixed indexed/unindexed 
OR predicates, query-option precedence, scalar search modes, split 
serialization, global row IDs/allow-lists, retained scores, single-permit 
execution, query-once chunking, failure propagation and 
cancellation/reader-close ordering.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to