Hi, I don't understand why we need the same machnism twice in the code... Could checkpoing barrieres and low watermarks be unified (or one build on-top/by-using the other)
-Matthias On 05/12/2015 02:47 PM, Gyula Fóra wrote: > Hi, > > Checkpoint barriers are handled directly on top of the network layer and > you are right they work similarly, by blocking input channels until it gets > the barrier from all of them. > > A way of implementing this on the operator level would be by adding a way > to ask the inputreader the channel index of the last record. This way the > operator could keep track of the channels from which it has received > records and execute the watermark logic. The IndexedReaders have > implemented the necessarry funcionality but were patched away accidentally > buy some earlier changes (as they were not used anyway) > > Adding a union operator is probably an overkill and would pose the same > difficulties when implementing it. > > Cheers, > Gyula > > On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <[email protected]> > wrote: > >> Hi Folks, >> as I said in the subject. How will this work? I'm in the process about >> thinking how to implement low watermarks in Streaming. I'm thinking >> that the implementation should be quite similar to how the >> checkpointing barriers will be implemented since they also flush out >> stuff. >> >> Now I'm wondering how this will work with merged Streams and the >> output selectors (split streams). It seems to me that there are a lot >> of paths that elements can take to arrive at operators. The problem I >> have is that an operator can only emit a low watermark itself if it >> knows that all input operators have sent him a low watermark with that >> value (the low watermark is the minimum of the low watermarks of all >> upstream operators). I imagine that the checkpoint barriers exhibit >> the same behaviour. >> >> Do we maybe have to add an explicit union (merge) operator and change >> how split streams are implemented? >> >> What are your thoughts? >> >> Cheers, >> Aljoscha >> >
signature.asc
Description: OpenPGP digital signature
