LuciferYang opened a new issue, #9883: URL: https://github.com/apache/paimon/issues/9883
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master ### Compute Engine Any (data-evolution vector search: local / Spark / Flink) ### Minimal reproduce step On a data-evolution table with a vector index, run a vector search whose WHERE filter the scalar global index cannot evaluate. Two shapes trigger it: 1. The filter uses a function the scalar index does not support, e.g. `IS NOT NULL` on a multivalue-indexed array column, or references a column that has no scalar index. 2. The filter is a partially-evaluable `AND`, e.g. `a = 1 AND tags IS NOT NULL` where `tags IS NOT NULL` is unsupported. ### What doesn't meet your expectations? Rows that match the filter go missing from the result (shape 1), or the top-K is polluted by rows that do not match the filter and matching rows are displaced out of it (shape 2). Root cause: vector search delegates pre-filtering to the scalar global index. `scalarMatchedRows` returned an empty bitmap when the index could not evaluate the predicate, and that bitmap was AND-ed into every index split, so every index-covered row was dropped. For a partially-evaluable `AND`, the evaluator drops the unsupported conjunct and returns a superset that the index path trusts as exact. The index read path has no final-read filter, so an inexact bitmap can neither be corrected nor ignored. ### Anything else? No response ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
