On 8/30/18 10:28 AM, Dan van der Ster wrote:
> Hi,
>
> Is anyone else seeing rocksdb mon stores slowly growing to >15GB,
> eventually triggering the 'mon is using a lot of disk space' warning?
>
> Since upgrading to luminous, we've seen this happen at least twice.
> Each time, we restart all the mons and then stores slowly trim down to
> <500MB. We have 'mon compact on start = true', but it's not the
> compaction that's shrinking the rockdb's -- the space used seems to
> decrease over a few minutes only after *all* mons have been restarted.
>
> This reminds me of a hammer-era issue where references to trimmed maps
> were leaking -- I can't find that bug at the moment, though.
>
I just saw our message in the other thread and I thought I'd reply here.
I have seen this recently as well with Luminous 12.2.8 after a large
migration. Cluster grew from ~2000 OSDs to ~2500. Rebalance took about 4
days.
After this all the MONs were 15~16GB in size and were issuing a warning.
I stopped the MONs and compacted their MON stores using
ceph-monstore-tool and started them again, that worked.
I'm usually cautions with doing a online compaction as this sometimes
hits the MON performance.
Not sure yet why this is happening as the MONs should compact during
normal operations.
Wido
> Cheers, Dan
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
