Aljoscha Krettek created FLINK-2577:
---------------------------------------
Summary: Watermarks Stall When a Source Finishes Prematurely
Key: FLINK-2577
URL: https://issues.apache.org/jira/browse/FLINK-2577
Project: Flink
Issue Type: Bug
Components: Streaming
Affects Versions: 0.10
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
Fix For: 0.10
The problem with a streaming source that closes is that downstream operations
never notice that it is not running anymore and keep waiting for watermarks
from all upstream operations (including the source). This has the effect that
watermarks just stop propagating through the topology.
I think an easy fix is to change sources to emit a last watermark of +Inf
before closing. Because watermarks are always the minimum of all watermarks on
the inputs this would have the effect of advancing only depending on the other
inputs.
The added benefit would be that once all sources emit a +Inf watermark the
operator also get's a last +Inf watermark which tells it that all sources are
done. Right now, streaming operators (and user code) have no way of telling if
there are going to come elements in the future. This is especially problematic
in Co-Map (Co-FlatMap) operations where you have one input that feeds a
hash-table and the other input is elements that you want to stream by this
hash-table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
