LuciferYang opened a new pull request, #9884: URL: https://github.com/apache/paimon/pull/9884
### Purpose close #9883 Vector search with a WHERE filter delegates pre-filtering to the scalar global index. The index returns all-or-nothing bitmaps, and the index read path has no final-read filter to correct them, so it trusts each bitmap as the exact match set. Two cases broke that assumption: - The index could not evaluate the predicate at all: no scalar index files, or an unsupported function such as `IS NOT NULL` on a multivalue-indexed array column. `scalarMatchedRows` returned an empty bitmap, which was AND-ed into every index split, so every index-covered row silently vanished from the result. - The index could evaluate only some of the filter's fields, for example `a = 1 AND tags IS NOT NULL` where the array conjunct is unsupported. The evaluator drops that conjunct and returns a superset matching `a = 1` alone, which polluted the vector top-K with non-matching rows and displaced matching ones. `scalarMatchedRows` now returns `null`, meaning "cannot decide exactly", in both cases: when `scanWithCoverage` finds nothing, and when the contributing fields do not cover every field the filter references. `demoteUncoverableIndexSplits` then routes the ranges no raw split already covers through the raw search, where the exact final-read filter decides; ranges an existing raw split covers keep the current dedup. The computed scalar pre-filter is cached for `preFilters` to reuse. Every reader entry point (local, batch, Spark, Flink) demotes right after splitting its splits. ### Tests Added two cases to `VectorSearchBuilderTest`, both asserting the exact row ids the search returns: - `testVectorSearchFilterIndexCannotEvaluate` uses a filter the index cannot evaluate at all (`IS NOT NULL` on a multivalue-indexed array). The old empty-bitmap handling returns an empty result; the fallback returns the closest rows that match the filter. - `testVectorSearchFilterPartiallyEvaluableAnd` uses a partially-evaluable `AND` (`id >= 0 AND tags IS NOT NULL`). The old superset handling puts a row that fails the filter into the top-K; the fallback returns only matching rows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
