Large state RocksDb backend increases app start time

Hi, I'm currently storing around 70GB of data in map sate backed by RocksDB backend . Once I restore an application from savepoint currently the application takes more than 4mins to start processing events. How can I speed this up or is there any other recommended approach. I'm using the

Externally load data into RocksDB backend

Hi, Is it possible to load data to RocksDB backend externally(outside Flink ) using common dbPath, so that it will be available to MapState inside ProcessFunction. I've external data available in JSON format which I want to load to RocksDb. One option is to make Stream from the JSON path and

Re: Is MapState tied to Operator Input & Output type?

points.html#assigning-operator-ids > > Best > Yun Tang > > -- > *From:* Arpith P > *Sent:* Tuesday, October 13, 2020 19:26 > *To:* user > *Subject:* Is MapState tied to Operator Input & Output type? > > > Hi, > > > I’ve a *ProcessFunction* whi

Is MapState tied to Operator Input & Output type?

Hi, I’ve a *ProcessFunction* which initially was receiving input & output type of String (1) & inside *processElement* I was updating MapState. Now I have changed the Input & Output type to be Map, String (2), but if I restore from the last checkpoint folder MapState is coming in as empty. I’ve

Re: App gets stuck in Created State

duled yet. It is > strange it a job gets stuck in this state. > Is it possible that you share the job manager log so we can check what is > happening there? > > Thanks, > Zhu > > Arpith P 于2020年9月21日周一 下午3:52写道： > >> Hi, >> >> We have Flink 1.8.0 cluster deployed

Re: Zookeeper connection loss causing checkpoint corruption

sounds like a critical priority issue to me. > > Thanks, > Timo > > On 22.09.20 06:25, Arpith P wrote: > > Hi Peter, > > > > I have recently had a similar issue where I could not load from the > > checkpoints path. I found that whenever a corrupt checkpoint hap

Re: Zookeeper connection loss causing checkpoint corruption

Hi Peter, I have recently had a similar issue where I could not load from the checkpoints path. I found that whenever a corrupt checkpoint happens the "_metadata" file will not be persisted, and I've a program which tracks if checkpoint location based on this strategy and updates DB with

App gets stuck in Created State

Hi, We have Flink 1.8.0 cluster deployed in Hadoop distributed mode, I often see even though Hadoop has enough resources Flink sits in Created state. We have 4 operators using 15 parallelism, 1 operator using 40 & 2 operators using 10. At time of submission I'm passing taskmanager memory as 4Gb

Re: Automatically restore from checkpoint

points, but the job manager and task managers must be able to read > (and write) them. > > Regards, > David > > On Fri, Sep 18, 2020 at 6:23 AM Arpith P wrote: > >> Hi, >> >> I'm running Flink job in distributed mode deployed in Yarn; I've enabled >> e

Automatically restore from checkpoint

Hi, I'm running Flink job in distributed mode deployed in Yarn; I've enabled externalized checkpoint to save in Hdfs, but I don't have access to read checkpoints folder. To restart Flink job from the last saved checkpoint is it possible to do without passing "-s :checkpointPath". If this is not