LuciferYang opened a new pull request, #9909: URL: https://github.com/apache/paimon/pull/9909
### Purpose close #9883 Vector search with a WHERE filter delegates pre-filtering to the scalar global index. When the index cannot evaluate the predicate at all (no scalar index files, or an unsupported function such as `IS NOT NULL` on a multivalue-indexed array column), `scalarMatchedRows` returned an empty bitmap. That bitmap was AND-ed into every index split, so all index-covered rows silently vanished from the result. This makes `scalarMatchedRows` return `null`, meaning "cannot decide", in that case. `demoteUncoverableIndexSplits` then routes the ranges no raw split already covers through the raw search, where the exact final-read filter decides; ranges an existing raw split covers keep the current dedup. The computed scalar pre-filter is cached for `preFilters` to reuse. Every reader entry point (local, batch, Spark, Flink) demotes right after splitting its splits. Scope: this keeps the index fast path for a partially-evaluable predicate, for example `a = 1 AND tags IS NOT NULL` where the array conjunct is unsupported. Routing those to the raw search would regress performance, so only the fully-unevaluable case is demoted. This is the minimal-scope version of the closed #9884. ### Tests `VectorSearchBuilderTest#testVectorSearchFilterIndexCannotEvaluate` covers a filter the scalar index cannot evaluate (`IS NOT NULL` on a multivalue-indexed array). Against the old empty-bitmap handling the result is empty; with the fallback it returns the closest rows that match the filter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
