zhuxiangyi commented on issue #9931: URL: https://github.com/apache/paimon/issues/9931#issuecomment-5732806438
@LuciferYang @JingsongLi In #9855, keeping predicates that Spark cannot push down into Paimon as post-filters was intentional. The documentation and tests explicitly cover the possibility of returning fewer than K rows: these predicates filter the already-truncated search results, without refilling them. For this case, I lean toward **`post-filter` as the default, with `fail` available as an opt-in**, consistently across the three search TVFs. This would preserve the behavior documented in #9855 while giving callers an explicit way to reject post-filter semantics. Keeping the existing full-text test under `post-filter` and adding rejection coverage under `fail` sounds reasonable. The check should distinguish genuinely non-pushable predicates from Spark-side rechecks of predicates already pushed into Paimon. I’m open to aligning the behavior across the search APIs, and would also like to hear Jingsong’s thoughts before settling on the default. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
