Re: [DataStream API] Best practice to handle custom session window - time gap based but handles event for ending session

2019-08-04 Thread Jungtaek Lim
Thanks Dongwon to provide feedback and share your approach! I'm not sure it could be possible (not an expert), but if we could reset intermediate result (aggregated) after processing "fire event", I guess it would work as expected, as window would still expand even after "session end", but it will

Re: Execution environments for testing: local vs collection vs mini cluster

2019-08-04 Thread Juan Rodríguez Hortalá
Hi, Thanks for your answer. I hadn't noticed that the collection environment only works for the batch API. It's also nice to know that the mini cluster is more an internal tool. So that the local execution environments for batch and streaming are working very well for me, I was just curious, thank

Dynamically allocating right-sized task resources

2019-08-04 Thread Chad Dombrova
Hi all, First time poster, so go easy on me :) What is Flink's story for accommodating task workloads with vastly disparate resource requirements: e.g. some require very little CPU and RAM, while others require quite a lot. Our current strategy is to bundle resource-intensive tasks and send them

[ANNOUNCE] Weekly Community Update 2019/31

2019-08-04 Thread Konstantin Knauf
Dear community, happy to share this week's community update with news about a Flink on PyPi, code style proposals, Flink 1.9.0 RC1, Flame Graphs in the WebUI and a bit more. As always, please feel free to add additional updates and news to this thread! Flink Development === * [devel

Re: [DataStream API] Best practice to handle custom session window - time gap based but handles event for ending session

2019-08-04 Thread Dongwon Kim
Hi Jungtaek, I've faced a similar problem in the past; we need to calculate an aggregate upon receiving an end message from each user. While you're trying to solve problem by defining a custom window assigner, I took a different approach to the problem by implementing a custom trigger. You can s

[DataStream API] Best practice to handle custom session window - time gap based but handles event for ending session

2019-08-04 Thread Jungtaek Lim
Hi Flink users, I've been spending time to learn and play with Flink DataStream API, not an expert level but as a beginner. :) To play with custom window API, I just created a small example, session window based on fixed time gap, but indicate the type of event which may contain "end of session".