Hi, > Now Majid stated that he uses "RBD" - Majid, any chance to specify > what that RBD really is ? What's the tech? What fs? Any ioping or fio > results? What's the blockdev --report /dev/XXX output ? (you stated > "high" latency and "high" bandwidth , but it is relative, for me 15ms+ > is high latency and >1000MB/s sequential, but it would help others in > future if you could specify it by the exact numbers please). Maybe > it's just a matter of enabling readahead (line in [1]) there and/or > using a higher WAL segment during initdb.
Unfortunately, I quit that company a month ago (I wish we could discuss this earlier) and don't have access to the environment anymore. I'll try to ask my teammates and see if they can find anything about the exact values of latency, bw, etc. Anyway, here is some description of the environment. Sadly, there are no numbers in this description, but I'll try to provide as much details as possible. There is a k8s cluster running over some VMs. Each postgres instance runs as a pod inside the k8s cluster. So, both the primary and standby servers are in the same DC, but might be on different baremetal nodes. There is an overlay network for the pods to see each other, and there's also another overlay network for the VMs to see each other. The storage servers are in the same DC, but we're sure they're on some racks other than the postgres pods. They run Ceph [1] project and provide Rados Block Devices (RBD) [2] interface. In order for k8s to use ceph, a Ceph-CSI [3] controller is running inside the k8s cluster. BTW, the FS type is ext4. [1] - https://ceph.io/en/ [2] - https://docs.ceph.com/en/latest/rbd/ [3] - https://github.com/ceph/ceph-csi
