LuciferYang commented on PR #9964: URL: https://github.com/apache/paimon/pull/9964#issuecomment-5772880633
Agreed on closing this. The session-wide fail switch was the wrong shape for it, and the branch predates #9855, which documents the post-filter "may be short" semantics this PR ended up conflicting with. One thing worth putting on the record about the original concern: #9931 wasn't really about post-filter returning fewer than K rows. The issue is that applying the residual after the top-K limit hurts recall. For a selective predicate, matching rows that are near can be dropped because they never entered the unfiltered top-K, so the result isn't the true nearest-K among matching rows. The systems you cite that expose a filter mode use it to run the filter during the search, which is what addresses this. A fail switch doesn't. I don't have a concrete production caller that needs this today, so leaving it closed is fine by me. If one comes up, I'll bring the specific query and expected semantics and we can design pre-filter support directly. Thanks for the detailed writeup and the references. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
