This is probably LevelDB being slow. The monitor has some options to "compact" the store on startup and I thought the osd handled it automatically, but you could try looking for something like that and see if it helps. -Greg On Fri, Feb 27, 2015 at 5:02 AM Corin Langosch <corin.lango...@netskin.com> wrote:
> Hi guys, > > I'm using ceph for a long time now, since bobtail. I always upgraded every > few weeks/ months to the latest stable > release. Of course I also removed some osds and added new ones. Now during > the last few upgrades (I just upgraded from > 80.6 to 80.8) I noticed that old osds take much longer to startup than > equal newer osds (same amount of data/ disk > usage, same kind of storage+journal backing device (ssd), same weight, > same number of pgs, ...). I know I observed the > same behavior earlier but just didn't really care about it. Here are the > relevant log entries (host of osd.0 and osd.15 > has less cpu power than the others): > > old osds (average pgs load time: 1.5 minutes) > > 2015-02-27 13:44:23.134086 7ffbfdcbe780 0 osd.0 19323 load_pgs > 2015-02-27 13:49:21.453186 7ffbfdcbe780 0 osd.0 19323 load_pgs opened 824 > pgs > > 2015-02-27 13:41:32.219503 7f197b0dd780 0 osd.3 19317 load_pgs > 2015-02-27 13:42:56.310874 7f197b0dd780 0 osd.3 19317 load_pgs opened 776 > pgs > > 2015-02-27 13:38:43.909464 7f450ac90780 0 osd.6 19309 load_pgs > 2015-02-27 13:40:40.080390 7f450ac90780 0 osd.6 19309 load_pgs opened 806 > pgs > > 2015-02-27 13:36:14.451275 7f3c41d33780 0 osd.9 19301 load_pgs > 2015-02-27 13:37:22.446285 7f3c41d33780 0 osd.9 19301 load_pgs opened 795 > pgs > > new osds (average pgs load time: 3 seconds) > > 2015-02-27 13:44:25.529743 7f2004617780 0 osd.15 19325 load_pgs > 2015-02-27 13:44:36.197221 7f2004617780 0 osd.15 19325 load_pgs opened > 873 pgs > > 2015-02-27 13:41:29.176647 7fb147fb3780 0 osd.16 19315 load_pgs > 2015-02-27 13:41:31.681722 7fb147fb3780 0 osd.16 19315 load_pgs opened > 848 pgs > > 2015-02-27 13:38:41.470761 7f9c404be780 0 osd.17 19307 load_pgs > 2015-02-27 13:38:43.737473 7f9c404be780 0 osd.17 19307 load_pgs opened > 821 pgs > > 2015-02-27 13:36:10.997766 7f7315e99780 0 osd.18 19299 load_pgs > 2015-02-27 13:36:13.511898 7f7315e99780 0 osd.18 19299 load_pgs opened > 815 pgs > > The old osds also take more memory, here's an example: > > root 15700 22.8 0.7 1423816 485552 ? Ssl 13:36 4:55 > /usr/bin/ceph-osd -i 9 --pid-file > /var/run/ceph/osd.9.pid -c /etc/ceph/ceph.conf --cluster ceph > root 15270 15.4 0.4 1227140 297032 ? Ssl 13:36 3:20 > /usr/bin/ceph-osd -i 18 --pid-file > /var/run/ceph/osd.18.pid -c /etc/ceph/ceph.conf --cluster ceph > > > It seems to me there is still some old data around for the old osds which > was not properly migrated/ cleaned up during > the upgrades. The cluster is healthy, no problems at all the last few > weeks. Is there any way to clean this up? > > Thanks > Corin > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
