Thank you David ________________________________ From: David Anderson <dander...@apache.org> Sent: Monday, December 6, 2021 1:36:20 AM To: Alexey Trenikhun <yen...@msn.com> Cc: Flink User Mail List <user@flink.apache.org> Subject: Re: Order of events in Broadcast State
Event ordering in Flink is only maintained between pairs of events that take exactly the same path through the execution graph. So if you have multiple instances of A (let's call them A1 and A2), each broadcasting a partition of the total rule space, then one instance of B (B1) might receive rule1 from A1 before rule2 from A2, while B2 might receive rule2 before rule1. If it fits your needs, one simple way to avoid having problems with this is to broadcast from a task with a parallelism of 1. Then every downstream instance will receive the broadcast stream in the same order. David On Sat, Dec 4, 2021 at 2:45 AM Alexey Trenikhun <yen...@msn.com<mailto:yen...@msn.com>> wrote: [1] - https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/ The Broadcast State Pattern | Apache Flink<https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/broadcast_state/> The Broadcast State Pattern # In this section you will learn about how to use broadcast state in practise. Please refer to Stateful Stream Processing to learn about the concepts behind stateful stream processing. Provided APIs # To show the provided APIs, we will start with an example before presenting their full functionality. As our running example, we will use the case where we have a ... nightlies.apache.org<http://nightlies.apache.org> ________________________________ From: Alexey Trenikhun <yen...@msn.com<mailto:yen...@msn.com>> Sent: Friday, December 3, 2021 4:33 PM To: Flink User Mail List <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Order of events in Broadcast State Hello, Trying to understand what statement "Order of events in Broadcast State may differ across tasks" in [1] means. Let's say I have keyed function "A" which broadcasting stream of rules, KeyedBroadcastProcessFunction "B" receives rules and updates broadcast state, like example in [1]. Let's say "A" broadcasts "rule 1" with name X, then "A" (same key) broadcasts "rule 2" with same name X, is there guarantee that eventually broadcast state will contain "rule 2" or since there is no ordering, B could receive "rule 2", then "rule 1" and broadcast state will end up with {X="rule 1"} forever ? Thanks, Alexey
