Re: Flink not writing last few elements to disk

2018-02-05 Thread Piotr Nowojski
Hi, FileProcessMode.PROCESS_CONTINUOUSLY processes the file continuously - the stream will not end. Simple `writeAsCsv(…)` on the other hand only flushes the output file on a stream end (see `OutputFormatSinkFunction`). You can either use `PROCESS_ONCE` mode or use more advanced data sink: -

Flink not writing last few elements to disk

2018-02-02 Thread geoff halmo
Hi Flink community: I am testing Flink but can't write the final(18 or so elements out to disk) Setup: Using NYC yellow taxi from data 2017-09.csv, I sorted the data on pickup_datetime in bash. I am working in event time. Skeleton program: val ds = senv.readFile(input_format, input_path,