Its actually a very different mechanism as watermarks will not block the computations
On Tue, May 12, 2015 at 2:48 PM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > Hi, > > I don't understand why we need the same machnism twice in the code... > Could checkpoing barrieres and low watermarks be unified (or one build > on-top/by-using the other) > > -Matthias > > > On 05/12/2015 02:47 PM, Gyula Fóra wrote: > > Hi, > > > > Checkpoint barriers are handled directly on top of the network layer and > > you are right they work similarly, by blocking input channels until it > gets > > the barrier from all of them. > > > > A way of implementing this on the operator level would be by adding a way > > to ask the inputreader the channel index of the last record. This way the > > operator could keep track of the channels from which it has received > > records and execute the watermark logic. The IndexedReaders have > > implemented the necessarry funcionality but were patched away > accidentally > > buy some earlier changes (as they were not used anyway) > > > > Adding a union operator is probably an overkill and would pose the same > > difficulties when implementing it. > > > > Cheers, > > Gyula > > > > On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljos...@apache.org> > > wrote: > > > >> Hi Folks, > >> as I said in the subject. How will this work? I'm in the process about > >> thinking how to implement low watermarks in Streaming. I'm thinking > >> that the implementation should be quite similar to how the > >> checkpointing barriers will be implemented since they also flush out > >> stuff. > >> > >> Now I'm wondering how this will work with merged Streams and the > >> output selectors (split streams). It seems to me that there are a lot > >> of paths that elements can take to arrive at operators. The problem I > >> have is that an operator can only emit a low watermark itself if it > >> knows that all input operators have sent him a low watermark with that > >> value (the low watermark is the minimum of the low watermarks of all > >> upstream operators). I imagine that the checkpoint barriers exhibit > >> the same behaviour. > >> > >> Do we maybe have to add an explicit union (merge) operator and change > >> how split streams are implemented? > >> > >> What are your thoughts? > >> > >> Cheers, > >> Aljoscha > >> > > > >
