Jinhua, I thought about this a bit more, and found that incremental windowing operation may not be suitable (more clear, not effective) by nature of Storm Acker model. (JStorm recently dropped acker model which may be why you can't see tuple ack in their code.)
In acker model, we should store the tuple until the tuple is expired to ensure we send correct XOR value to Acker, (The tuple should be retained while emitting new tuples with anchoring the tuple.) While we could still have a room for optimization via dropping the tuple's values to reduce the size, distributed snapshotting looks better to live with window & stateful operations. I think we could take distributed snapshot into account because Acker has been pointed out to performance bottleneck for a long time since it requires calculating target task of Acker based on message ID, and sending metadata tuple to Acker (incurring network transfer) for every tuple. Thanks, Jungtaek Lim (HeartSaVioR) 2017년 12월 22일 (금) 오후 5:47, Jungtaek Lim <[email protected]>님이 작성: > Thanks again. > > I guess windowing implementation in JStorm is basically inspired by Storm > (meaning copying the codebase, which is allowed on Apache Software License. > Heron is doing the same thing.), but looks like they have been going > forward afterwards. By "technically", we could also migrate the code from > JStorm, but better to try to understand the code and implement our own, if > that's feasible. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2017년 12월 22일 (금) 오후 5:29, Jinhua Luo <[email protected]>님이 작성: > >> Yes, you could check the jstorm codes relation to this topic: >> >> >> https://github.com/alibaba/jstorm/blob/master/jstorm-core/src/main/java/com/alibaba/jstorm/window/WindowedBoltExecutor.java >> >> But it seems that it does not consider tuple ack. >> >> >> 2017-12-22 16:18 GMT+08:00 Jungtaek Lim <[email protected]>: >> > Thanks for the pointer. Looks like it can be achieved with stateful >> > windowing, but it's just based on skimming so there may be hard parts in >> > detail. >> > I'll spend time to read it and see how we can achieve it. (Both core >> API and >> > stream API which will be brought to Storm 2.0.0.) >> > >> > Thanks, >> > Jungtaek Lim (HeartSaVioR) >> > >> > 2017년 12월 22일 (금) 오후 5:08, Jinhua Luo <[email protected]>님이 작성: >> >> >> >> >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#reducefunction >> >> >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/windows.html#processwindowfunction-with-incremental-aggregation >> >> >> >> Jstorm also supports incremental aggregation upon the window. >> >> >> >> 2017-12-22 13:31 GMT+08:00 Jungtaek Lim <[email protected]>: >> >> > Hi Jinhua, >> >> > >> >> > could you refer the link of doc for Flink? I'm not exactly aware of >> >> > incremental aggregation upon the window, so let me take a look at. >> >> > >> >> > 2017년 12월 21일 (목) 오후 2:09, Jinhua Luo <[email protected]>님이 작성: >> >> >> >> >> >> Hi All, >> >> >> >> >> >> The window may buffer many tuples before evaluation, does storm >> >> >> support incremental aggregation upon the window, just like flink >> does? >> >
