Re: Stopping of a streaming job empties state store on HDFS

2018-06-15 Thread Till Rohrmann
Hi Peter, this sounds very strange. I just tried to reproduce the issue locally but for me it worked without a problem. Could you maybe share the jobmanager logs on DEBUG log level with us? As a side note, enabling the asynchronous checkpointing mode for the FsStateBackend does not have an

Re: Stopping of a streaming job empties state store on HDFS

2018-06-15 Thread Peter Zende
Hi Stefan, Thanks for the answer. Fixing the uids solved the problem, that's not an issue anymore. The savepoint directory is there, but the RocksDB state is not restored after restarting the application because that state directory has been removed when I stopped the application. It looks like

Re: Stopping of a streaming job empties state store on HDFS

2018-06-11 Thread Stefan Richter
Hi, > Am 08.06.2018 um 01:16 schrieb Peter Zende : > > Hi all, > > We have a streaming pipeline (Flink 1.4.2) for which we implemented stoppable > sources to be able to gracefully exit from the job with Yarn state > "finished/succeeded". > This works fine, however after creating a savepoint,

Stopping of a streaming job empties state store on HDFS

2018-06-07 Thread Peter Zende
Hi all, We have a streaming pipeline (Flink 1.4.2) for which we implemented stoppable sources to be able to gracefully exit from the job with Yarn state "finished/succeeded". This works fine, however after creating a savepoint, stopping the job (stop event) and restarting it we remarked that the