Re: Order of events in Broadcast State
Thank you David
From: David Anderson
Sent: Monday, December 6, 2021 1:36:20 AM
To: Alexey Trenikhun
Cc: Flink User Mail List
Subject: Re: Order of events in Broadcast State
Event ordering in Flink is only maintained between pairs of events that take
exactly the same path through the execution graph. So if you have multiple
instances of A (let's call them A1 and A2), each broadcasting a partition of
the total rule space, then one instance of B (B1) might receive rule1 from A1
before rule2 from A2, while B2 might receive rule2 before rule1.
If it fits your needs, one simple way to avoid having problems with this is to
broadcast from a task with a parallelism of 1. Then every downstream instance
will receive the broadcast stream in the same order.
David
On Sat, Dec 4, 2021 at 2:45 AM Alexey Trenikhun
mailto:yen...@msn.com>> wrote:
[1] -
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
The Broadcast State Pattern | Apache
Flink<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
The Broadcast State Pattern # In this section you will learn about how to use
broadcast state in practise. Please refer to Stateful Stream Processing to
learn about the concepts behind stateful stream processing. Provided APIs # To
show the provided APIs, we will start with an example before presenting their
full functionality. As our running example, we will use the case where we have
a ...
nightlies.apache.org<http://nightlies.apache.org>
From: Alexey Trenikhun mailto:yen...@msn.com>>
Sent: Friday, December 3, 2021 4:33 PM
To: Flink User Mail List mailto:user@flink.apache.org>>
Subject: Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may
differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules,
KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state,
like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A"
(same key) broadcasts "rule 2" with same name X, is there guarantee that
eventually broadcast state will contain "rule 2" or since there is no ordering,
B could receive "rule 2", then "rule 1" and broadcast state will end up with
{X="rule 1"} forever ?
Thanks,
Alexey
Re: Order of events in Broadcast State
Event ordering in Flink is only maintained between pairs of events that
take exactly the same path through the execution graph. So if you
have multiple instances of A (let's call them A1 and A2), each broadcasting
a partition of the total rule space, then one instance of B (B1) might
receive rule1 from A1 before rule2 from A2, while B2 might receive rule2
before rule1.
If it fits your needs, one simple way to avoid having problems with this is
to broadcast from a task with a parallelism of 1. Then every downstream
instance will receive the broadcast stream in the same order.
David
On Sat, Dec 4, 2021 at 2:45 AM Alexey Trenikhun wrote:
> [1] -
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
> The Broadcast State Pattern | Apache Flink
> <https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
> The Broadcast State Pattern # In this section you will learn about how to
> use broadcast state in practise. Please refer to Stateful Stream Processing
> to learn about the concepts behind stateful stream processing. Provided
> APIs # To show the provided APIs, we will start with an example before
> presenting their full functionality. As our running example, we will use
> the case where we have a ...
> nightlies.apache.org
>
>
> --
> *From:* Alexey Trenikhun
> *Sent:* Friday, December 3, 2021 4:33 PM
> *To:* Flink User Mail List
> *Subject:* Order of events in Broadcast State
>
> Hello,
> Trying to understand what statement "Order of events in Broadcast State
> may differ across tasks" in [1] means.
> Let's say I have keyed function "A" which broadcasting stream of rules,
> KeyedBroadcastProcessFunction "B" receives rules and updates broadcast
> state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X,
> then "A" (same key) broadcasts "rule 2" with same name X, is there
> guarantee that eventually broadcast state will contain "rule 2" or since
> there is no ordering, B could receive "rule 2", then "rule 1" and broadcast
> state will end up with {X="rule 1"} forever ?
>
> Thanks,
> Alexey
>
Re: Order of events in Broadcast State
[1] -
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/
The Broadcast State Pattern | Apache
Flink<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/>
The Broadcast State Pattern # In this section you will learn about how to use
broadcast state in practise. Please refer to Stateful Stream Processing to
learn about the concepts behind stateful stream processing. Provided APIs # To
show the provided APIs, we will start with an example before presenting their
full functionality. As our running example, we will use the case where we have
a ...
nightlies.apache.org
From: Alexey Trenikhun
Sent: Friday, December 3, 2021 4:33 PM
To: Flink User Mail List
Subject: Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may
differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules,
KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state,
like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A"
(same key) broadcasts "rule 2" with same name X, is there guarantee that
eventually broadcast state will contain "rule 2" or since there is no ordering,
B could receive "rule 2", then "rule 1" and broadcast state will end up with
{X="rule 1"} forever ?
Thanks,
Alexey
Order of events in Broadcast State
Hello,
Trying to understand what statement "Order of events in Broadcast State may
differ across tasks" in [1] means.
Let's say I have keyed function "A" which broadcasting stream of rules,
KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state,
like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A"
(same key) broadcasts "rule 2" with same name X, is there guarantee that
eventually broadcast state will contain "rule 2" or since there is no ordering,
B could receive "rule 2", then "rule 1" and broadcast state will end up with
{X="rule 1"} forever ?
Thanks,
Alexey
