About 2gb and it should be distributed בתאריך יום א׳, 15 במאי 2022, 19:50, מאת Reuven Lax <[email protected]>:
> How large is this state? Is it distributed? > > On Sun, May 15, 2022 at 8:12 AM Sigalit Eliazov <[email protected]> > wrote: > >> Thanks for your response. >> The use case is 2 pipelines: >> A. receives information from kafka and "holds" that info >> B. a pipeline which is triggered by a scheduler and every x minutes >> should send the info i received in pipeline A to another kafka topic >> >> Currently these pipelines are implemented by BEAM and I use DB in order >> to save the output of pipeline A, so when B is triggered I select the >> relevant info from DB. >> I would prefer to drop the DB and use internal state if possible. >> I also tried to merge those pipelines and use the spec state (@stateId) >> but it did not work. the state was not available for the operator which >> reads from the scheduler . >> >> Thanks >> Sigalit >> >> On Sun, May 15, 2022 at 6:03 PM Reuven Lax <[email protected]> wrote: >> >>> Beam supports side inputs, which might help you. Can you describe your >>> use case? >>> >>> On Sun, May 15, 2022 at 7:34 AM Sigalit Eliazov <[email protected]> >>> wrote: >>> >>>> Hello >>>> does beam have support for something similar to >>>> KeyedBroadcastProcessFunction which exists in flink? >>>> I am looking for an option to have broadcast state in beam so it can >>>> be shared between different operators >>>> >>>> Thanks >>>> Sigalit >>>> >>>
