Hi Ken, Thanks! I would expect more folks to run into this and hence surprised to not find this in LocalStreamEnvironment. Is there a reason for that?
In the specific case, we have an unbounded source (Kinesis), but for testing we would like to make it bounded. Hence the earlier question whether it is possible to terminate a topology with a final watermark or different means from within the source, similar to how a bounded source in Beam would behave. Thanks, Thomas On Tue, Feb 6, 2018 at 5:16 PM, Ken Krugler <kkrugler_li...@transpac.com> wrote: > Hi Thomas, > > Normally the streaming job will terminate when the sources are exhausted > and all records have been processed. > > I assume you have some unbounded source(s), thus this doesn’t work for > your case. > > We’d run into a similar situation with a streaming job that has iterations. > > Our solution was your option #1 below, where we created a modified version > of LocalStreamEnvironment <https://raw.githubusercontent.com/ > ScaleUnlimited/flink-crawler/master/src/main/java/org/ > apache/flink/streaming/api/environment/LocalStreamEnvironmentWithAsyn > cExecution.java> that supports async execution. > > — Ken > > > > On Feb 6, 2018, at 4:21 PM, Thomas Weise <t...@apache.org> wrote: > > > > Hi, > > > > I'm looking for an example of an integration test that runs a streaming > job > > and terminates when the expected result becomes available. I could think > of > > 2 approaches: > > > > 1. Modified version of LocalStreamEnvironment that executes the job > > asynchronously and polls for the result or > > > > 2. Source that emits a final watermark that causes the topology to > > terminate after the watermark has traversed the topology. Is that > possible > > with Flink? > > > > But probably this is a rather common testing need that's already solved?! > > > > Thanks, > > Thomas > > -------------------------- > Ken Krugler > http://www.scaleunlimited.com > custom big data solutions & training > Hadoop, Cascading, Cassandra & Solr > >
