Re: Maintain heavy hitters in Flink application

Hi, I think it would be easier to implement a custom key selector and introduce some artifical key that spreads the load more evenly. This would also allow you to use keyed state. You could use a ProcessFunction and set timers to define the "every now and then". Keyed state would also ease

Re: Maintain heavy hitters in Flink application

Anyone, someone, somebody? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Maintain heavy hitters in Flink application

Hello everyone and Happy New Year! Regarding the Heavy Hitter tracking...I wanna do it in a distributed manner. Thus, 1 -- Round Robin the input stream to a number of parallel map instances (say p = env.parallelism) 2 -- Each one of the p mappers maintains approximately the HH of its

Re: Maintain heavy hitters in Flink application

Kostas and Fabian, Thanks for the advice. I guess I will find a workaround to do the state redistribution. I also read about side outputs in this thread, which might be also an option that I will consider.

Re: Maintain heavy hitters in Flink application

Hi Max, You are right that Queryable State is not designed to be used as a means for a job to query its own state. In fact, given that you do not know the jobId of your job from within the job itself, I do not think you can use queryable state in your scenario. What you can do is to have a

Re: Maintain heavy hitters in Flink application

Hi, I haven't done that before either. The query API will change with the next version (Flink 1.4.0) which is currently being prepared for releasing. Kostas (in CC) might be able to help you. Best, Fabian 2017-12-05 9:52 GMT+01:00 m@xi : > Hi Fabian, > > Thanks for your

Re: Maintain heavy hitters in Flink application

Hi Max, state (keyed or operator state) is always local to the task. By default it is not accessible (read or write) from the outside or other tasks of the application. You can expose keyed state as queryable state [1] to perform key look ups. This feature was designed for external application