Re: Flink solution for having shared variable between task managers
Thanks, I'll check it out. On Mon, Feb 3, 2020 at 10:05 AM Fabian Hueske wrote: > Hi, > > I think you are looking for BroadcastState [1]. > > Best, Fabian > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html > > Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani < > soheil.i...@gmail.com>: > >> Hi, >> >> According to the processing logic, I need to have a HashMap variable that >> should be shared between the taskmanagers. The scenario is the HashMap data >> will be continuously updated according to the incoming stream of data. >> >> What I observed is declaring the HashMap variable as a class attribute, >> it will be shared among a single taskmanagers slots, but in case I have >> multiple taskmanager, each will have a separate HashMap instance. >> >> What is the standard way to achieve this? Does Flink provide any utility >> for that? >> >
Re: Flink solution for having shared variable between task managers
Hi, I think you are looking for BroadcastState [1]. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/broadcast_state.html Am Fr., 17. Jan. 2020 um 14:50 Uhr schrieb Soheil Pourbafrani < soheil.i...@gmail.com>: > Hi, > > According to the processing logic, I need to have a HashMap variable that > should be shared between the taskmanagers. The scenario is the HashMap data > will be continuously updated according to the incoming stream of data. > > What I observed is declaring the HashMap variable as a class attribute, it > will be shared among a single taskmanagers slots, but in case I have > multiple taskmanager, each will have a separate HashMap instance. > > What is the standard way to achieve this? Does Flink provide any utility > for that? >
RE: EXT :Flink solution for having shared variable between task managers
I think you’re looking for Broadcast State. Here’s a detailed guide. https://flink.apache.org/2019/06/26/broadcast-state.html From: Soheil Pourbafrani [mailto:soheil.i...@gmail.com] Sent: Friday, January 17, 2020 6:50 AM To: user Subject: EXT :Flink solution for having shared variable between task managers Hi, According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data. What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance. What is the standard way to achieve this? Does Flink provide any utility for that?
Flink solution for having shared variable between task managers
Hi, According to the processing logic, I need to have a HashMap variable that should be shared between the taskmanagers. The scenario is the HashMap data will be continuously updated according to the incoming stream of data. What I observed is declaring the HashMap variable as a class attribute, it will be shared among a single taskmanagers slots, but in case I have multiple taskmanager, each will have a separate HashMap instance. What is the standard way to achieve this? Does Flink provide any utility for that?
