On 10/6/22 13:06, Frank Schilder wrote:
Hi all,
we are stuck with a really unpleasant situation and we would appreciate help.
Yesterday we completed the ceph deamon upgrade from mimic to octopus all he way
through with bluestore_fsck_quick_fix_on_mount = false and started the OSD OMAP
conversion today in the morning. Everything went well at the beginning. The
conversion went much faster than expected and OSDs came slowly back up.
Unfortunately, trouble was only around the corner.
That sucks. Not sure how far into the upgrade process you are based on
the info in this mail, but just to make sure you are not hit by RocksDB
degradation:
Have you done an offline compaction of the OSDs after the conversion? We
have seen that degraded RocksDB can severely impact the performance. So
make sure the OSDs are compacted, i.e.:
stop osd processes: systemctl stop ceph-osd.target
df|grep "/var/lib/ceph/osd"|awk '{print $6}'|cut -d '-' -f 2|sort
-n|xargs -n 1 -P 10 -I OSD ceph-kvstore-tool bluestore-kv
/var/lib/ceph/osd/ceph-OSD compact
There can be a ton of other things happening of course. In that case try
to gather debug logs.
Gr. Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
