Re: Broadcast state before events stream consumption

Hi Averell, BroadcastState is a special case of OperatorState. Operator state is always kept in-memory at runtime (must fit into memory), no matter what state backend you use. Nevertheless it is snapshotted and thus fault tolerant. Best, Dawid On 21/02/2019 11:50, Averell wrote: > Hi

Re: Broadcast state before events stream consumption

Hi Konstantin, The statement below is mentioned at the end of the page broadcast_state.html#important-considerations /"No RocksDB state backend: Broadcast state is kept

Re: Broadcast state before events stream consumption

Hi Chirag, Broadcast state is checkpointed, hence the savepoint would contain it. Best, Konstantin On Wed, Feb 13, 2019 at 4:04 PM Chirag Dewan wrote: > Hi Konstantin, > > For the second solution, would savepoint persist the Broadcast state in > State backend? Because I am aware that

Re: Broadcast state before events stream consumption

Hi Konstantin, For the second solution, would savepoint persist the Broadcast state in State backend? Because I am aware that Broadcast state is not checkpointed.  Is that correct? Thanks, Chirag Sent from Yahoo Mail on Android On Mon, 11 Feb 2019 at 2:39 PM, Konstantin Knauf wrote: Hi

Re: Broadcast state before events stream consumption

Hi Chirag, Hi Vadim, from the top of my head, I see two options here: * Buffer the "fast" stream inside the KeyedBroadcastProcessFunction until relevant (whatever this means in your use case) broadcast events have arrived. Advantage: operationally easy, events are emitted as early as possible.

Re: Broadcast state before events stream consumption

Hi Vadim, I would be interested in this too.  Presently, I have to read my lookup source in the open method and keep it in a cache. By doing that I cannot make use of the broadcast state until ofcourse the first emit comes on the Broadcast stream. The problem with waiting the event stream is

Broadcast state before events stream consumption

Hi all, I need to use the broadcast state mechanism (https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html) for the next scenario. I have a reference data stream (slow) and an events stream (fast running) and I want to do a kind of lookup in the