Hi all,
It appears that OSD daemons only very slowly free RAM after an extended period
of an unhealthy cluster (shuffling PGs around).
Prior to a power outage (and recovery) around July 25th, the amount of RAM
used was fairly constant, at most 10GB (out of 24GB). You can see in the
attached PNG "osd6_stack2.png" (Week 30) that the amount of used RAM on
osd06.physics.wisc.edu was holding steady around 7GB.
Around July 25th our Ceph cluster rebooted after a power outage. Not all
nodes booted successfully, so Ceph proceeded to shuffle PGs to attempt to
return
to health with the renaming nodes. You can see in "osd6_stack2.png" two
purplish spikes showing that the node used around 10GB swap space during the
recovery period.
Finally the cluster recovered around July 31st. During that period some I had
to take some osd daemons out of the pool b/c their nodes ran out of swap space
and the daemons were killed by the out of memory (OOM) kernel feature. (The
recovery period was probably extended by me trying to add the daemons/drives
back. If I recall correctly that is what was occurring during the second swap
peak.)
This RAM usage pattern is in generalthe same for all the nodes in the cluster.
Almost three weeks later, the amount of RAM used on the node is still
decreasing, but it has not returned to pre-power outage levels. 15GB instead
of 7GB.
Why is Ceph using 2x more RAM than it used to in steady state?
Thanks,
Chad.
(P.S. It is really unfortunate that Ceph uses more RAM when recovering - can
lead to cascading failure!)
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
