+1 for this FLIP. BTW, I think we can add an option for projection push down too.
Yes, we can do all things in the connector, but a common implementation helps a lot! And can introduce an unify option! Best, Jingsong On Wed, Oct 25, 2023 at 10:07 AM Jark Wu <imj...@gmail.com> wrote: > > Thank you for updating Jiabao, > > The FLIP looks good to me. > > Best, > Jark > > On Wed, 25 Oct 2023 at 00:42, Jiabao Sun <jiabao....@xtransfer.cn.invalid> > wrote: > > > Thanks Jane for the feedback. > > > > The default value of "table.optimizer.source.predicate" is true that means > > by default, > > allowing predicate pushdown to all sources is permitted. > > > > Therefore, disabling the pushdown filter for individual sources can take > > effect. > > > > > > Best, > > Jiabao > > > > > > > 2023年10月24日 23:52，Jane Chan <qingyue....@gmail.com> 写道： > > > > > >> > > >> I believe that the configuration "table.optimizer.source.predicate" has > > a > > >> higher priority at the planner level than the configuration at the > > source > > >> level, > > >> and it seems easy to implement now. > > >> > > > > > > Correct me if I'm wrong, but I think the fine-grained configuration > > > "scan.filter-push-down.enabled" should have a higher priority because the > > > default value of "table.optimizer.source.predicate" is true. As a result, > > > turning off filter push-down for a specific source will not take effect > > > unless the default value of "table.optimizer.source.predicate" is changed > > > to false, or, alternatively, let users manually set > > > "table.optimizer.source.predicate" to false first and then selectively > > > enable filter push-down for the desired sources, which is less intuitive. > > > WDYT? > > > > > > Best, > > > Jane > > > > > > On Tue, Oct 24, 2023 at 6:05 PM Jiabao Sun <jiabao....@xtransfer.cn > > .invalid> > > > wrote: > > > > > >> Thanks Jane, > > >> > > >> I believe that the configuration "table.optimizer.source.predicate" has > > a > > >> higher priority at the planner level than the configuration at the > > source > > >> level, > > >> and it seems easy to implement now. > > >> > > >> Best, > > >> Jiabao > > >> > > >> > > >>> 2023年10月24日 17:36，Jane Chan <qingyue....@gmail.com> 写道： > > >>> > > >>> Hi Jiabao, > > >>> > > >>> Thanks for driving this discussion. I have a small question that will > > >>> "scan.filter-push-down.enabled" take precedence over > > >>> "table.optimizer.source.predicate" when the two parameters might > > conflict > > >>> each other? > > >>> > > >>> Best, > > >>> Jane > > >>> > > >>> On Tue, Oct 24, 2023 at 5:05 PM Jiabao Sun <jiabao....@xtransfer.cn > > >> .invalid> > > >>> wrote: > > >>> > > >>>> Thanks Jark, > > >>>> > > >>>> If we only add configuration without adding the enableFilterPushDown > > >>>> method in the SupportsFilterPushDown interface, > > >>>> each connector would have to handle the same logic in the applyFilters > > >>>> method to determine whether filter pushdown is needed. > > >>>> This would increase complexity and violate the original behavior of > > the > > >>>> applyFilters method. > > >>>> > > >>>> On the contrary, we only need to pass the configuration parameter in > > the > > >>>> newly added enableFilterPushDown method > > >>>> to decide whether to perform predicate pushdown. > > >>>> > > >>>> I think this approach would be clearer and simpler. > > >>>> WDYT? > > >>>> > > >>>> Best, > > >>>> Jiabao > > >>>> > > >>>> > > >>>>> 2023年10月24日 16:58，Jark Wu <imj...@gmail.com> 写道： > > >>>>> > > >>>>> Hi JIabao, > > >>>>> > > >>>>> I think the current interface can already satisfy your requirements. > > >>>>> The connector can reject all the filters by returning the input > > filters > > >>>>> as `Result#remainingFilters`. > > >>>>> > > >>>>> So maybe we don't need to introduce a new method to disable > > >>>>> pushdown, but just introduce an option for the specific connector. > > >>>>> > > >>>>> Best, > > >>>>> Jark > > >>>>> > > >>>>> On Tue, 24 Oct 2023 at 16:38, Leonard Xu <xbjt...@gmail.com> wrote: > > >>>>> > > >>>>>> Thanks @Jiabao for kicking off this discussion. > > >>>>>> > > >>>>>> Could you add a section to explain the difference between proposed > > >>>>>> connector level config `scan.filter-push-down.enabled` and existing > > >>>> query > > >>>>>> level config `table.optimizer.source.predicate-pushdown-enabled` ? > > >>>>>> > > >>>>>> Best, > > >>>>>> Leonard > > >>>>>> > > >>>>>>> 2023年10月24日 下午4:18，Jiabao Sun <jiabao....@xtransfer.cn.INVALID> > > 写道： > > >>>>>>> > > >>>>>>> Hi Devs, > > >>>>>>> > > >>>>>>> I would like to start a discussion on FLIP-377: support > > configuration > > >>>> to > > >>>>>> disable filter pushdown for Table/SQL Sources[1]. > > >>>>>>> > > >>>>>>> Currently, Flink Table/SQL does not expose fine-grained control for > > >>>>>> users to enable or disable filter pushdown. > > >>>>>>> However, filter pushdown has some side effects, such as additional > > >>>>>> computational pressure on external systems. > > >>>>>>> Moreover, Improper queries can lead to issues such as full table > > >> scans, > > >>>>>> which in turn can impact the stability of external systems. > > >>>>>>> > > >>>>>>> Suppose we have an SQL query with two sources: Kafka and a > > database. > > >>>>>>> The database is sensitive to pressure, and we want to configure it > > to > > >>>>>> not perform filter pushdown to the database source. > > >>>>>>> However, we still want to perform filter pushdown to the Kafka > > source > > >>>> to > > >>>>>> decrease network IO. > > >>>>>>> > > >>>>>>> I propose to support configuration to disable filter push down for > > >>>>>> Table/SQL sources to let user decide whether to perform filter > > >> pushdown. > > >>>>>>> > > >>>>>>> Looking forward to your feedback. > > >>>>>>> > > >>>>>>> [1] > > >>>>>> > > >>>> > > >> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=276105768 > > >>>>>>> > > >>>>>>> Best, > > >>>>>>> Jiabao > > >>>>>> > > >>>>>> > > >>>> > > >>>> > > >> > > >> > > > >
