Hi Sonex,
When using readTextFile(...) with event time, only one watermark with the
value Long.MAX_VALUE is sent at the end of the stream, which explais why
the windows are stored until the whole file is processed. In order to have
periodic watermarks, you need to process the file continuousely as folows:
TextInputFormat inputFormat = new TextInputFormat(new Path("file/to/read.txt"));
env.readFile(inputFormat,"file/to/read.txt",
FileProcessingMode.PROCESS_CONTINUOUSLY,10000L,
TypeInformation.of(String.class))
.map(...)
Hope this helps.
Best,
Yassine
2017-03-23 9:47 GMT+01:00 Sonex <alfredjens...@gmail.com>:
> Hi everyone,
>
> I am using a simple window computation on a stream with event time. The
> code
> looks like this:
>
> streamData.readTextFile(...)
> .map(...)
> .assignAscendingTimestamps(_.timestamp)
> .keyBy(_.id)
> .timeWindow(Time.seconds(3600),Time.seconds(3600))
> .apply(new MyWindowFunction)
> .map(...)
>
> By monitoring the memory usage and the flink web dashboard, I noticed that
> flink applies the window function until the entire stream finishes (thus
> storing all aggregations in memory) and then continues to the map
> transformation. What I would expect is emission of window results to the
> map
> transformation as soon as results of the window are ready.
>
> Can anyone explain this behavior?
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Windows-emit-
> results-at-the-end-of-the-stream-tp12337.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
