Re: Increasing Disk Read Throughput and IOPS

Hi, I just wanted to add that if you are using EBS you could consider to switch to IO provisioned type of it (io1: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html ) if it is ok from the cost

Re: Increasing Disk Read Throughput and IOPS

One more thing, I am aware of one older thread that might be interesting for you about RocksDB backend and EBS: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/checkpoint-stuck-with-rocksdb-statebackend-and-s3-filesystem-td18679.html

Re: Increasing Disk Read Throughput and IOPS

Hi, if the problem is seemingly from reads, I think incremental checkpoints are less likely to cause the problem. What Flink version are you using? Since you mentioned the use of map state, what comes to my mind as a potential cause is described in this issue

Re: Increasing Disk Read Throughput and IOPS

One thing that you can always to is disable fsync, because Flink does not rely on RocksDBs fsync for persistence. If you disable incremental checkpoints, does that help? If yes, it could be an issue with too many small SSTable files due to incremental checkpoints (an issue we have on the roadmap

Re: Increasing Disk Read Throughput and IOPS

Hi, This issue might have something to do with compaction. Problems with compaction can especially degrade reads performance (or just increase reads IO). Have you tried to further enforce more compactions or change CompactionStyle? Have you taken a look on

Increasing Disk Read Throughput and IOPS

Hi All, We are running flink in AWS and we are observing a strange behavior. We are using docker containers, EBS for storage and Rocks DB state backend. We have a few map and value states with checkpointing every 30 seconds and incremental checkpointing turned on. The issue we are noticing is the