On Thu, 13 Oct 2016, Henrik Korkuc wrote:
> from status page it seems that Ceph didn't like networking problems. May we
> find out some details what happened? Underprovisioned servers (RAM upgrades
> were in there too)? Too much load on disks? Something else?
> 
> This situation may be not pleasant but I feel that others can learn from it to
> prevent such situations in the future.

Yep.

These VMs were backed by an old ceph cluster and the cluster fell over 
after a switch failed.  Because it's a beta cluster that's due to be 
decommissioned shortly it wasn't upgraded from firefly.  And because it's 
old the PGs were mistuned (way too many) and machines were 
underprovisioned on RAM (32GB for 12 OSDs; normally probably enough but 
not on a very large cluster with 1000+ OSDs and too many PGs).  It fell 
into the somewhat familiar pattern of OSDs OOMing because of large OSDMaps 
due to a degraded cluster.

The recovery was a bit tedious (tune osdmap caches way down, get all OSDs 
to catch up on maps and rejoin cluster) but it's a procedure that's been 
described on this list before.  Once the core issue was identified it came 
back pretty quickly.

Had the nodes had more RAM or had the PG counts been better tuned it would 
have been avoided, and had the cluster been upgraded it *might* have been 
avoided (hammer+ is more memory efficient, and newer versions have lower 
default map cache sizes).

This was one of the very first large-scale clusters we ever built, so 
we've learned quite a bit since then.  :)

sage


> 
> On 16-10-13 06:55, Dan Mick wrote:
> > Everything should have been back some time ago (0000 UTC or thereabouts)
> > 
> > On 10/11/2016 10:41 PM, Brian :: wrote:
> > > Looks like they are having major challenges getting that ceph cluster
> > > running again.. Still down.
> > > 
> > > On Tuesday, October 11, 2016, Ken Dreyer <kdre...@redhat.com
> > > <mailto:kdre...@redhat.com>> wrote:
> > > > I think this may be related:
> > > > 
> > > http://www.dreamhoststatus.com/2016/10/11/dreamcompute-us-east-1-cluster-service-disruption/
> > > > On Tue, Oct 11, 2016 at 5:57 AM, Sean Redmond <sean.redmo...@gmail.com
> > > <mailto:sean.redmo...@gmail.com>> wrote:
> > > > > Hi,
> > > > > 
> > > > > Looks like the ceph website and related sub domains are giving errors
> > > > > for
> > > > > the last few hours.
> > > > > 
> > > > > I noticed the below that I use are in scope.
> > > > > 
> > > > > http://ceph.com/
> > > > > http://docs.ceph.com/
> > > > > http://download.ceph.com/
> > > > > http://tracker.ceph.com/
> > > > > 
> > > > > Thanks
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
> 
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Reply via email to