Understanding flink rocks db state and compaction

2024-08-08 Thread banu priya
Hi All, I have a flink job with Rabbitmq source,keyby, tumbling window and aggregator function, sink. I’m using rockDB state backend along with incremental checkpoints. Every 10s checkpointing is happening with 5s pause between checkpoints. Constant load with same keys I used for my test. I

Re: Restore rocksDB from savepoint exception

2024-08-08 Thread Bjarke Tornager
Hi Gabor, What kind of development work - is it the Disaggregated State Storage and Management FLIP ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=293046855) that you are referring to? Thanks for the suggestion. I am not familiar with the state processor api but I will have a

Re: Restore rocksDB from savepoint exception

2024-08-08 Thread Gabor Somogyi
Hi Bjarke, It's normal to see longer recovery time as the state size grows. There are developments in progress to mitigate this problem. > Any ideas as to what could be causing this? I think there is no easy way to tell it, I can give just some pointers. First I would take a look at the state fi