Re: Flink job restart at checkpoint interval

2016-11-16 Thread Satish Chandra Gupta
Cheers, > Till > > On Tue, Nov 15, 2016 at 10:52 PM, Satish Chandra Gupta < > scgupt...@gmail.com> wrote: > >> Hi Ufuk and Till, >> >> Thanks a lot. Both these suggestions were useful. Older version of xerces >> was being loaded from one of the

Re: Flink job restart at checkpoint interval

2016-11-15 Thread Satish Chandra Gupta
361/issue-http- > apache-org-xml-features-xinclude-testing-log4j-2 > [3] http://dmitrypukhov.pro/apache-spark-feature-httpapache- > orgxmlfeaturesxinclude-is-not-recognized/ > > Cheers, > Till > > On Tue, Nov 15, 2016 at 3:24 AM, Satish Chandra Gupta > wrote: > >&

Re: Flink job restart at checkpoint interval

2016-11-14 Thread Satish Chandra Gupta
4 November 2016 at 10:51:06, Satish Chandra Gupta (scgupt...@gmail.com) > wrote: > > Hi, > > > > I am using Value State, backed by FsStateBackend on hdfs, as following: > > > > env.setStateBackend(new FsStateBackend(stateBackendPath)) > > env.enableCheckpointi

Flink job restart at checkpoint interval

2016-11-14 Thread Satish Chandra Gupta
Hi, I am using Value State, backed by FsStateBackend on hdfs, as following: env.setStateBackend(new FsStateBackend(stateBackendPath)) env.enableCheckpointing(checkpointInterval) It is non-iterative job running Flink/Yarn. The job restarts at checkpointInterval, I have tried interval varying fro

Re: Best way of doing some global initialization

2016-11-03 Thread Satish Chandra Gupta
gt; Cheers, > Aljoscha > > On Thu, 3 Nov 2016 at 09:26 Satish Chandra Gupta > wrote: > >> Hi, >> >> I need to do set/initialize some config of a framework/util that is used >> in my Flink stream processing app. Basically, a piece of code that needs to >>

Best way of doing some global initialization

2016-11-03 Thread Satish Chandra Gupta
Hi, I need to do set/initialize some config of a framework/util that is used in my Flink stream processing app. Basically, a piece of code that needs to be executed exactly once before anything else. Clearly doing it in the main flink processor function will not suffice, as apart from the client,

How to debug why Flink makes and executes only partial plan

2016-10-14 Thread Satish Chandra Gupta
Hi, In my Flink program, after a couple of map, union and connect, I have a final filter and a sink. Something like this (after abstracting out details): val filteredEvents: DataStream[NotificationEvent] = allThisStuffWorking .name("filtered_users") filteredEvents *.filter(x => check(x