As you already noted, tick tuples provides only seconds granularity so that was one of the reasons for having separate threads for triggering processing time windows. Also the tick tuples are published to the receive queue of the executor, so if there are messages in front of it, the delivery of the tick tuple will be delayed. So we may not be able to guarantee processing time windows (that processes say events in the last x secs).
Perhaps if we could deliver the tick tuples at a higher priority via a different queue it would work. Also the tick tuples itself requires one thread per executor so we would still need to have a thread. However this would simplify the processing of the windowing logic since the logic need not be thread safe (currently it needs to be due to the triggers). Regarding you other solution, I didn’t quite get how you would implement the callbacks without additional threads per executor. Thanks, Arun On 10/23/17, 7:42 AM, "Jerry Peng" <jerry.boyang.p...@gmail.com> wrote: >Hello all, > >I just realized that in the windowing implementation, we use an >additional thread as a timer for processing time triggered windows: > >https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/windowing/TimeTriggerPolicy.java#L71 > >We also do the same thing in: > >https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/windowing/WaterMarkEventGenerator.java#L71 > >to periodically generate watermarks. > >Perhaps, instead of additional threads we can just use tick tuples for >such triggering of events? Or we can allow registration of timers and >callback functions in topologies so that additional timer threads >would be not be needed? I know that the tick tuple frequency is in >seconds and windowing supports millisecond granularity but the tick >tuple frequency can easily be changed to support millisecond >frequency. > >On the other hand, a more eloquent solution would be having an API >that allows users to register arbitrary timer events in their >topologies. Something like this: > >conf.registerTimer(long frequency, Object callback) > >Anyone have any thoughts on this? > >Best, > >Jerry >
