Watermarks also don't need to flush buffers, they can actually simply queue in as special stream records, if we want to.
On Tue, May 12, 2015 at 2:53 PM, Gyula Fóra <gyula.f...@gmail.com> wrote: > Its actually a very different mechanism as watermarks will not block the > computations > > On Tue, May 12, 2015 at 2:48 PM, Matthias J. Sax < > mj...@informatik.hu-berlin.de> wrote: > > > Hi, > > > > I don't understand why we need the same machnism twice in the code... > > Could checkpoing barrieres and low watermarks be unified (or one build > > on-top/by-using the other) > > > > -Matthias > > > > > > On 05/12/2015 02:47 PM, Gyula Fóra wrote: > > > Hi, > > > > > > Checkpoint barriers are handled directly on top of the network layer > and > > > you are right they work similarly, by blocking input channels until it > > gets > > > the barrier from all of them. > > > > > > A way of implementing this on the operator level would be by adding a > way > > > to ask the inputreader the channel index of the last record. This way > the > > > operator could keep track of the channels from which it has received > > > records and execute the watermark logic. The IndexedReaders have > > > implemented the necessarry funcionality but were patched away > > accidentally > > > buy some earlier changes (as they were not used anyway) > > > > > > Adding a union operator is probably an overkill and would pose the same > > > difficulties when implementing it. > > > > > > Cheers, > > > Gyula > > > > > > On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljos...@apache.org > > > > > wrote: > > > > > >> Hi Folks, > > >> as I said in the subject. How will this work? I'm in the process about > > >> thinking how to implement low watermarks in Streaming. I'm thinking > > >> that the implementation should be quite similar to how the > > >> checkpointing barriers will be implemented since they also flush out > > >> stuff. > > >> > > >> Now I'm wondering how this will work with merged Streams and the > > >> output selectors (split streams). It seems to me that there are a lot > > >> of paths that elements can take to arrive at operators. The problem I > > >> have is that an operator can only emit a low watermark itself if it > > >> knows that all input operators have sent him a low watermark with that > > >> value (the low watermark is the minimum of the low watermarks of all > > >> upstream operators). I imagine that the checkpoint barriers exhibit > > >> the same behaviour. > > >> > > >> Do we maybe have to add an explicit union (merge) operator and change > > >> how split streams are implemented? > > >> > > >> What are your thoughts? > > >> > > >> Cheers, > > >> Aljoscha > > >> > > > > > > > >
