Hi Eugen,

It is correct that I manually added the configuration, but not to the
unit.run but rather to each mon's config (i.e.
/var/lib/ceph/FSID/mon.*/config). I also added it to the cluster config
with "ceph config set mon mon_rocksdb_options", but it seems that this
option doesn't have any effect at all.

/Z

On Tue, 5 Mar 2024 at 09:58, Eugen Block <ebl...@nde.ag> wrote:

> Hi,
>
> > 1. RocksDB options, which I provided to each mon via their configuration
> > files, got overwritten during mon redeployment and I had to re-add
> > mon_rocksdb_options back.
>
> IIRC, you didn't use the extra_entrypoint_args for that option but
> added it directly to the container unit.run file. So it's expected
> that it's removed after an update. If you want it to persist a
> container update you should consider using the extra_entrypoint_args:
>
> cat mon.yaml
> service_type: mon
> service_name: mon
> placement:
>    hosts:
>    - host1
>    - host2
>    - host3
> extra_entrypoint_args:
>    -
>
> '--mon-rocksdb-options=write_buffer_size=33554432,compression=kLZ4Compression,level_compaction_dynamic_level_bytes=true,bottommost_compression=kLZ4HCCompression,max_background_jobs=4,max_subcompactions=2'
>
> Regards,
> Eugen
>
> Zitat von Zakhar Kirpichenko <zak...@gmail.com>:
>
> > Hi,
> >
> > I have upgraded my test and production cephadm-managed clusters from
> > 16.2.14 to 16.2.15. The upgrade was smooth and completed without issues.
> > There were a few things which I noticed after each upgrade:
> >
> > 1. RocksDB options, which I provided to each mon via their configuration
> > files, got overwritten during mon redeployment and I had to re-add
> > mon_rocksdb_options back.
> >
> > 2. Monitor debug_rocksdb option got silently reset back to the default
> 4/5,
> > I had to set it back to 1/5.
> >
> > 3. For roughly 2 hours after the upgrade, despite the clusters being
> > healthy and operating normally, all monitors would run manual compactions
> > very often and write to disks at very high rates. For example, production
> > monitors had their rocksdb:low0 thread write to store.db:
> >
> > monitors without RocksDB compression: ~8 GB/5 min, or ~96 GB/hour;
> > monitors with RocksDB compression: ~1.5 GB/5 min, or ~18 GB/hour.
> >
> > After roughly 2 hours with no changes to the cluster the write rates
> > dropped to ~0.4-0.6 GB/5 min and ~120 MB/5 min respectively. The reason
> for
> > frequent manual compactions and high write rates wasn't immediately
> > apparent.
> >
> > 4. Crash deployment broke ownership of /var/lib/ceph/FSID/crash
> > /var/lib/ceph/FSID/crash/posted, despite I already fixed it manually
> after
> > the upgrade to 16.2.14 which had broken it as well.
> >
> > 5. Mgr RAM usage appears to be increasing at a slower rate than it did
> with
> > 16.2.14, although it's too early to tell whether the issue with mgrs
> > randomly consuming all RAM and getting OOM-killed has been fixed - with
> > 16.2.14 this would normally take several days.
> >
> > Overall, things look good. Thanks to the Ceph team for this release!
> >
> > Zakhar
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-le...@ceph.io
>
>
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-le...@ceph.io
>
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io

Reply via email to