On 12/3/19 3:07 PM, Aleksey Gutikov wrote:

That is true. When an OSD goes down it will take a few seconds for it's
Placement Groups to re-peer with the other OSDs. During that period
writes to those PGs will stall for a couple of seconds.

I wouldn't say it's 40s, but it can take ~10s.


According to my experience, in case of OSD crashes, killed -9 (any kind abnormat termination) OSD failure handling contains next steps: 1) Failed OSD's peers detect that it does not respond - it can take up to osd_heartbeat_grace + osd_heartbeat_interval seconds

If a 'Connection Refused' is detected the OSD will be marked as down right away.

2) Peers send reports to monitor
3) Monitor makes a decision according to (options from it's own config) mon_osd_adjust_heartbeat_grace, osd_heartbeat_grace, mon_osd_laggy_halflife, mon_osd_min_down_reporters, ... And finally mark OSD down in osdmap.


4) Monitor send updated OSDmap to OSDs and clients
5) OSDs starting peering
5.1) Peering itself is complicated process, for example we had experienced PGs stuck in inactive state due to osd_max_pg_per_osd_hard_ratio.

I would say that 5.1 isn't relevant for most cases. Yes, it can happen, but it's rare.

6) Peering finished (PGs' data continue moving) - clients can normally access affected PGs. Clients also have their own timeouts that can affect time to recover. >
Again, according to my experience, 40s with default settings is possible.

40s is possible in certain scenarios. But I wouldn't say that's the default for all cases.


ceph-users mailing list

Reply via email to