Hi,
1. In short, the OSD need to heartbeat with up to #PG x (#Replica -1 ),
but actually will be much less since most of the peers are redundant.
For example, An OSD (say OSD 1) is holding 100 PGs, especially for some PGs,
say PG 1, OSD1 is the primary OSD of PG1, then OSD1 need to peer with all other
OSDs in PG1's acting set and up set(basically you could think these two sets
are other replications for PG1).
So if the cluster with very simple(default) ruleset, it's possible that an OSD
need to peer with all other OSDs.
2. OSD will randomly select a mon when the OSD boot up , and talking to
the mon consistently. It's the monitor quorum's job to reach an agreement about
the OSD status. See paxos if you want to know more details in how to reach the
agreement.
3. OSD do ping with Mon, but in reality, the network between monitor and
osd likely not the networking between OSD's . As an instance, Mon <-> OSD is in
management network bug OSD<-> OSD is in 10Gb data network. So only ping with
Mon is not enough.
Actually there are heartbeat on both public and cluster network, just use to
ensure the network connectivity.
Xiaoxi
From: ceph-users [mailto:[email protected]] On Behalf Of Liu,
Ming (HPIT-GADSC)
Sent: Monday, April 13, 2015 12:08 PM
To: [email protected]
Subject: [ceph-users] question about OSD failure detection
Hi, all,
I am new to study Ceph. Trying to understand how it works and designs.
One basic question for me is about how Ceph OSD perform failure detection. I
did some searching but cannot get satisfied answer, so try to ask here and hope
someone can kindly help me.
The document said, OSD will send heartbeat to other OSD and report failure to
Monitor when it detect some other OSD is down.
My questions are
1. One particular OSD will heartbeat to how many other OSD? Is it possible for
one OSD to do heartbeat to most of other OSDs or even all other OSDs in the
cluster? In other words, how OSD decide the list of OSDs that it needs to check
for health?
2. If OSD detect a failure, to which Monitor it report to? This selection is
random or has some rule? Or it needs to report to multiple Monitors?
3. Why not each OSD directly do heartbeat with a monitor?
If I get question 1 answered that one OSD maybe need to checking all other OSDs
in the cluster, and if this is true for many OSDs in the cluster, it looks like
a lot of network traffic and redundant. Say, OSD-1 check OSD-2, OSD-3, OSD-4,
while OSD-2 also check OSD-3, OSD-4, OSD-5. Then both OSD1,2 do redundant
health checking for OSD-4,5.
If answer of question 1 is: one OSD only need to do heartbeat with very few
other OSDs, never has possibility to check most of other OSDs, then I am fine,
this will decentralize the monitor for health checking. But I want to know the
rule of how OSD decide which other OSD it needs to check to further understand
this.
I read almost all articles on internet I can find up to now, but still cannot
get very satisfied answer. I don't want to dive into source code yet, that may
take a long time for me. I want to first understand the principles. Then decide
if I really worth to spend huge time to read src code. So really want someone
can help me here.
Any help will be very appreciated!!
Thanks,
Ming
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
