LuciferYang opened a new issue, #9931: URL: https://github.com/apache/paimon/issues/9931
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master ### Compute Engine Spark (primary-key / data-evolution vector search via the `vector_search` TVF) ### Minimal reproduce step Run a Spark `vector_search(...)` with a `LIMIT k`, and a WHERE conjunct that `SparkV2FilterConverter` cannot convert to a Paimon predicate: a UDF call, a column-to-column comparison (`a > b`), or an unresolvable cast. For example a filter like `udf(col) = 1` combined with the vector search, `limit = k`. ### What doesn't meet your expectations? The result can miss qualifying rows. The vector search returns exactly `k` rows, the top-`k` by vector distance, computed over the pushed-down (convertible) predicates only. A non-convertible conjunct is not pushed into Paimon (`PaimonBaseScanBuilder.pushPredicates` keeps it in `postScan`), so Spark applies it as a `FilterExec` above the already-truncated top-`k`. Post-filtering the top-`k` can only drop rows, never refill the ones displaced out of it, so rows that satisfy the residual but ranked just outside the returned `k` are lost, and the result is short. This is distinct from #9883, which is about a predicate that reaches Paimon but the scalar index cannot evaluate. Here the predicate never reaches Paimon at all, so the evaluator-level fix in #9883 does not cover it. The Flink `vector_search` procedure does not have this problem: it parses the whole `where` string into the Paimon predicate, so an inexpressible conjunct fails the call rather than silently becoming a residual. ### Anything else? Found while tracing the vector-search filter pushdown for #9883. Fix direction: when a vector search has a non-convertible residual, either refuse the pushdown / fail fast, or over-retrieve and re-apply the residual to refill the top-`k`. The lateral / dynamic-query-vector path (`PushDownLateralVectorSearchFilter`) has the same exposure. ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
