Greetings, I am working on standing up a fresh Ceph object storage cluster and have some questions about what I should be seeing as far as inter-OSD connectivity. I have spun up my monitor and radosgw nodes as VMs, all running on a 192.168.10.0/24 network (all IP ranges have been changed to protect the innocent). I also have four physical servers to serve as my storage nodes. These physical servers have two bonded interfaces, one IP'd on the public network (192.168.10.0/24) and one IP'd on a cluster/private network (172.22.20.0/24). I have added the following lines to my ceph.conf:
public network = 192.168.10.0/24 cluster network = 172.22.20.0/24 So far, so good. Everything in the cluster starts up, all OSDs are up and in, and 'ceph -s' shows a happy, healthy cluster. I can create users and otherwise do all the normal things that one would expect. So why am I writing? When looking at my network connections on my storage servers, I was expecting to see a few OSD <-> Mon/RGW connections on the public network, then a much larger number of OSD <-> OSD connections on the cluster network. What I actually see is an equal number of connections between OSDs on both the public and cluster networks (in addition to OSD <-> Mon/RGW connections). My question - is this normal? If so, can someone explain what traffic is moving between OSDs on the public network? Based on some additional testing (read: bringing down the cluster interface), this causes all OSDs on that node to be marked down, so there's evidence to support all heartbeat traffic moving over the interface. I just want to ensure that what I'm seeing is normal and that I haven't otherwise botched the configuration. Many thanks, Brian Felton
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
