Hi all, The iteration feature has been merged to the devel-3 branch in Apex Core.
We introduced a special kind of operator called DelayOperator, which takes incoming windows and increments the window ID by one for all outgoing ports, thus allowing loops in the topology. This feature will be included with Apex Core 3.3.0 release. *Please help test this feature by checking out the latest devel-3 branch in Apex Core and building your own iteration topology.* You can take a look at this pull request in Apex Malhar (not merged yet) for an example of the usage of the iteration feature: https://github.com/apache/incubator-apex-malhar/pull/177 This iteration feature opens up Apex to the world of iterative streaming algorithms, and will be especially useful for ML training algorithms. Known issues: - Input Operator: The topology still requires at least one regular InputOperator to generate the windows, even though in some cases, technically an InputOperator is not needed because the DelayOperator is capable of creating data for the first window. - Locality: for now THREAD_LOCAL is not allowed in the iteration loop, though it should be technically possible. Coming soon: - DelayByNOperator: A DelayOperator that simulates +N increment on top of the native +1 increment by the engine. Thank you! David
