JingsongLi commented on PR #9964: URL: https://github.com/apache/paimon/pull/9964#issuecomment-5770947890
Thanks for the work on this. After looking at how other search systems handle filters, I don't think we should add a session-wide `fail/post-filter` option here. Post-filtering an already limited search result can return fewer than K rows; #9855 already establishes and documents that behavior for Paimon. Elasticsearch and OpenSearch document the same trade-off. Systems that expose a filter-mode choice, such as Azure AI Search and LanceDB, use it to choose when filtering runs for a query. I did not find a comparable session-level switch that fails a query when a residual cannot be pushed down. This PR keeps `post-filter` as the default, so the new behavior helps only callers that explicitly opt into `fail`. We do not have a concrete production use case for that contract, while the public configuration and optimizer checks add ongoing complexity. I don't think that trade-off is justified now, so I'm closing this PR. If a production caller needs strict filtered top-K results, please bring the specific query and expected semantics so we can design that behavior directly. References: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-knn-query ; https://docs.opensearch.org/latest/vector-search/filter-search-knn/index/ ; https://learn.microsoft.com/en-us/azure/search/vector-search-filters ; https://github.com/lancedb/docs/blob/main/docs/search/full-text-search.mdx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
