Thank you very much for the clarification Seth. Best Regards Krzysztof Chmielewski
śr., 15 gru 2021, 16:12 użytkownik Seth Wiesman <sjwies...@gmail.com> napisał: > Hi Krzysztof, > > There is a difference in semantics here between yourself and Caizhi. SQL > UDFs can be used statefully - see AggregateFunction and > TableAggregateFunction for examples. You even have access to ListView and > MapView which are backed by ListState and MapState accordingly. These > functions contain aggregates which do participate in checkpointing and are > strongly consistent. > > What is not supported are more low level process function type operations > (custom state registration, user access to timers, broadcast state as > you've discovered). There have been some discussions about how to add this > sort of functionality in a SQL compliant manner but nothing concrete. > > In the meantime, Flink SQL has strong interop with the DataStream API. You > can always transform a Table into a DataStream, do some low level > processing, and then transform it back into a table to run further SQL. > > Seth > > On Wed, Dec 15, 2021 at 3:52 AM Krzysztof Chmielewski < > krzysiek.chmielew...@gmail.com> wrote: > >> Thank you, >> yes I was thinking about simply running my own thread in UDF and consume >> some queue something like that. >> Having some background with DataStreamAPI i was hoping that I can reuse >> same mechanisms (like Broadcast State Pattern or CoProcessFunction) in >> Flink SQL. >> However it seems there is a quite noticeable gap between what you can do >> with SQL and what you can control comparing to DataStreamAPI. >> >> Regarding UDF being stateless. I assume you mean that UDF does not >> participate in checkpoint mechanism and I cannot initialize a Flink state >> in UDF right? >> I'm wondering why it is not possible. Seems like an "obligatory" feature >> for Statefull stream processing platform that supports SQL. In a time when >> there is a huge interest with Flink SQL and everyone is talking about it, >> thing like state support is not available in SQL UDF is suprissing. >> >> Are there any plans, maybe FLIP to change it? >> >> Regards, >> Krzysztof Chmielewski >> >> śr., 15 gru 2021 o 02:36 Caizhi Weng <tsreape...@gmail.com> napisał(a): >> >>> Hi! >>> >>> Currently you can't use broadcast state in Flink SQL UDF because UDFs >>> are all stateless. >>> >>> However you mentioned your use case that you want to control the logic >>> in UDF with some information. If that is the case, you can just run a >>> thread in your UDF to read that information and change the behavior of the >>> eval method accordingly. >>> >>> Krzysztof Chmielewski <krzysiek.chmielew...@gmail.com> 于2021年12月15日周三 >>> 05:47写道： >>> >>>> Hi, >>>> Is there a way to build an UDF [1] for FLink SQL that can be used with >>>> Broadcast State Pattern [2]? >>>> >>>> I have a use case, where I would like to be able to use broadcast >>>> control stream to change logic in UDF. >>>> >>>> Regards, >>>> Krzysztof Chmielewski >>>> >>>> [1] >>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/#user-defined-functions >>>> [2] >>>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/#the-broadcast-state-pattern >>>> >>>
