Faidon has submitted this change and it was merged. Change subject: Ceph: be more resilient to malfunctioning OSDs ......................................................................
Ceph: be more resilient to malfunctioning OSDs There's currently an issue where a single malfunctioning OSD (or multiple of them) can mark down all of the other OSDs. This is the behavior previously described by yours truly on http://tracker.ceph.com/issues/4552 and supposedly fixed in the next version (Cuttlefish). However, we did have an outage because of this an hour ago, so it seems at least prudent to increase the number of reporters for now. Set it to 14, i.e. the number of OSDs in a single box + 2, as to protect ourselves from a malfunctioning or otherwise hung SCSI controller as well, as was 4552's case. Change-Id: Ia7cd6618e0bf9983ab7c22b53de7eb9c0e35c998 --- M manifests/role/ceph.pp 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Faidon: Looks good to me, approved jenkins-bot: Verified diff --git a/manifests/role/ceph.pp b/manifests/role/ceph.pp index a46d597..d13c318 100644 --- a/manifests/role/ceph.pp +++ b/manifests/role/ceph.pp @@ -37,6 +37,8 @@ 'osd journal size' => '10240', 'osd op thread timeout' => '7200', 'osd recovery thread timeout' => '7200', + # be more resilient to malfunctioning OSDs; see Ceph #4552 et al + 'osd min down reporters' => '14', }, 'radosgw' => { 'rgw enable ops log' => 'false', -- To view, visit https://gerrit.wikimedia.org/r/61423 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia7cd6618e0bf9983ab7c22b53de7eb9c0e35c998 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Faidon <[email protected]> Gerrit-Reviewer: Faidon <[email protected]> Gerrit-Reviewer: jenkins-bot _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
