Hi, I can use redis but I’m still having hard time figuring out how I can eliminate duplicate data. Today without broadcast state in 1.4 I’m using cache to lazy load the data. I thought the broadcast state will be similar to that of kafka streams where I have read access to the state across the pipeline. That will indeed solve a lot of problems. Is there some way I can do the same with flink?
Thanks! On Mon, Sep 30, 2019 at 10:36 PM Congxian Qiu <qcx978132...@gmail.com> wrote: > Hi, > > Could you use some cache system such as HBase or Reids to storage this > data, and query from the cache if needed? > > Best, > Congxian > > > Navneeth Krishnan <reachnavnee...@gmail.com> 于2019年10月1日周二 上午10:15写道： > >> Thanks Oytun. The problem with doing that is the same data will be have >> to be stored multiple times wasting memory. In my case there will around >> million entries which needs to be used by at least two operators for now. >> >> Thanks >> >> On Mon, Sep 30, 2019 at 5:42 PM Oytun Tez <oy...@motaword.com> wrote: >> >>> This is how we currently use broadcast state. Our states are re-usable >>> (code-wise), every operator that wants to consume basically keeps the same >>> descriptor state locally by processBroadcastElement'ing into a local state. >>> >>> I am open to suggestions. I see this as a hard drawback of dataflow >>> programming or Flink framework? >>> >>> >>> >>> --- >>> Oytun Tez >>> >>> *M O T A W O R D* >>> The World's Fastest Human Translation Platform. >>> oy...@motaword.com — www.motaword.com >>> >>> >>> On Mon, Sep 30, 2019 at 8:40 PM Oytun Tez <oy...@motaword.com> wrote: >>> >>>> You can re-use the broadcasted state (along with its descriptor) that >>>> comes into your KeyedBroadcastProcessFunction, in another operator >>>> downstream. that's basically duplicating the broadcasted state whichever >>>> operator you want to use, every time. >>>> >>>> >>>> >>>> --- >>>> Oytun Tez >>>> >>>> *M O T A W O R D* >>>> The World's Fastest Human Translation Platform. >>>> oy...@motaword.com — www.motaword.com >>>> >>>> >>>> On Mon, Sep 30, 2019 at 8:29 PM Navneeth Krishnan < >>>> reachnavnee...@gmail.com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Is it possible to access a broadcast state across the pipeline? For >>>>> example, say I have a KeyedBroadcastProcessFunction which adds the >>>>> incoming >>>>> data to state and I have downstream operator where I need the same state >>>>> as >>>>> well, would I be able to just read the broadcast state with a readonly >>>>> view. I know this is possible in kafka streams. >>>>> >>>>> Thanks >>>>> >>>>
