LuciferYang opened a new pull request, #9964: URL: https://github.com/apache/paimon/pull/9964
### Purpose close #9931 A `vector_search` WHERE conjunct that `SparkV2FilterConverter` cannot convert to a Paimon predicate (a UDF, a column-to-column comparison, an unresolvable cast) stays a Spark residual above the search. The search result is already truncated to the top-K, so that residual `FilterExec` can only drop rows and never refill the ones displaced out of the top-K, so the result silently comes back short. This rejects such a query with a clear error instead of returning wrong results, matching the Flink `vector_search` procedure, which fails on an inexpressible predicate rather than dropping rows. `CheckUnpushableSearchFilter` handles the static form and `PushDownLateralVectorSearchFilter` the lateral form; hybrid and full-text search are covered too. A convertible residual such as `payload = 'keep'` is unaffected: it is pushed into the search and pre-filters the top-K, so it stays correct. ### Tests `PrimaryKeyVectorSearchTest`: a `vector_search(..., 2)` with `WHERE id > threshold` (a column-to-column comparison Spark cannot push down) now fails with a "cannot be pushed down" error. Before this change the search returned the two nearest rows, both of which fail the filter, so the query silently came back empty instead of returning the two nearest rows that satisfy it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
