The goal is: * to split data, random-uniformly, across N nodes, * window the data identically on each node, * transform the windows locally on each node, and * merge the N parallel windows into a global window stream, such that one window from each parallel process is merged into a "global window" aggregate
I've achieved all but the last bullet point, merging one window from each partition into a globally-aggregated window output stream. To be clear, a rolling reduce won't work because it would aggregate over all previous windows in all partitioned streams, and I only need to aggregate over one window from each partition at a time. Similarly for a fold. The closest I have found is ParallelMerge for ConnectedStreams, but I have not found a way to apply it to this problem. Can flink achieve this? If so, I'd greatly appreciate a point in the right direction. Cheers, -aj
