JingsongLi opened a new pull request, #11227: URL: https://github.com/apache/paimon/pull/11227
### Purpose When a Data Evolution vector index is available but scalar indexes cannot evaluate the filter, run ANN once per query/shard and check its candidates against the pinned data. Remove the immediate raw-scoring fallback for indexed ranges. Add `vector-index.post-filter.candidates-factor`, a positive integer defaulting to `10`. Candidate count is `max(original Top-K * factor, required reranking limit)`, capped by shard size and backend bounds. Filtered results may contain fewer than Top-K rows, including zero; they are returned without retries or automatic raw recovery. Scalar candidate checks run on the execution worker. Large batches are processed in bounded query chunks, with one ANN request per input query and compact retained scores. The reader is reused across chunks; asynchronous continuations avoid single-permit executor deadlocks and preserve cleanup on errors/cancellation. Exact scalar prefilters, snapshot/DV handling, reranking and explicit Raw sources remain supported; automatically planned scalar gaps do not duplicate indexed work. This aligns with [the Rust implementation](https://github.com/apache/paimon-rust/pull/1439). There are no persisted table/index format changes. Raw split provenance is an additive serialization field; older splits default to authoritative Raw work. Updated vector filtering documentation describes the default and short-result semantics. ### Tests - `mvn -pl paimon-core -am -Dflink.forkCount=2 -DfailIfNoTests=false -DwildcardSuites=none '-Dtest=*Vector*Test,*GlobalIndex*Test,*Hybrid*Test' test`: passed (522 core tests, plus dependency-module tests). - Final focused `VectorSearchPostFilterTest,VectorSearchRowFilterExactnessTest`: 34 passed, including the additional historical-snapshot/DV regression. - Standard Maven checks, including Spotless/checkstyle, passed without the fast-build profile. The fresh worktree's codegen loader was packaged before the broad run. - Tests cover small/large factors and short results, mixed indexed/unindexed OR predicates, query-option precedence, scalar search modes, split serialization, global row IDs/allow-lists, retained scores, single-permit execution, query-once chunking, failure propagation and cancellation/reader-close ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
