[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

___ > From: Zakhar Kirpichenko > Sent: Wednesday, October 18, 2023 6:14 AM > To: Eugen Block > Cc: ceph-users@ceph.io > Subject: [ceph-users] Re: Ceph 16.2.x mon compactions, disk writes > > Many thanks for this, Eugen! I very much appreciate yours and Mykola's > effor

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Sent: Wednesday, October 18, 2023 6:14 AM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph 16.2.x mon compactions, disk writes Many thanks for this, Eugen! I very much appreciate yours and Mykola's efforts and insight! Another thing I noticed was a reduction of RocksDB store

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Many thanks for this, Eugen! I very much appreciate yours and Mykola's efforts and insight! Another thing I noticed was a reduction of RocksDB store after the reduction of the total PG number by 30%, from 590-600 MB: 65M 3675511.sst 65M 3675512.sst 65M 3675513.sst 65M 3675514.sst

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Hi Zakhar, I took a closer look into what the MONs really do (again with Mykola's help) and why manual compaction is triggered so frequently. With debug_paxos=20 I noticed that paxosservice and paxos triggered manual compactions. So I played with these values: paxos_service_trim_max =

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

With the help of community members, I managed to enable RocksDB compression for a test monitor, and it seems to be working well. Monitor w/o compression writes about 750 MB to disk in 5 minutes: 4854 be/4 167 4.97 M755.02 M 0.00 % 0.24 % ceph-mon -n mon.ceph04 -f --setuser

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

The issue persists, although to a lesser extent. Any comments from the Ceph team please? /Z On Fri, 13 Oct 2023 at 20:51, Zakhar Kirpichenko wrote: > > Some of it is transferable to RocksDB on mons nonetheless. > > Please point me to relevant Ceph documentation, i.e. a description of how >

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

> Some of it is transferable to RocksDB on mons nonetheless. Please point me to relevant Ceph documentation, i.e. a description of how various Ceph monitor and RocksDB tunables affect the operations of monitors, I'll gladly look into it. > Please point me to such recommendations, if they're on

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Some of it is transferable to RocksDB on mons nonetheless. > but their specs exceed Ceph hardware recommendations by a good margin Please point me to such recommendations, if they're on docs.ceph.com I'll get them updated. > On Oct 13, 2023, at 13:34, Zakhar Kirpichenko

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Thank you, Anthony. As I explained to you earlier, the article you had sent is about RocksDB tuning for Bluestore OSDs, while the issue at hand is not with OSDs but rather monitors and their RocksDB store. Indeed, the drives are not enterprise-grade, but their specs exceed Ceph hardware

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

cf. Mark's article I sent you re RocksDB tuning. I suspect that with Reef you would experience fewer writes. Universal compaction might also help, but in the end this SSD is a client SKU and really not suited for enterprise use. If you had the 1TB SKU you'd get much longer life, or you could

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

I guess you need to look for write intensive >> drives with decent specs. If you do, also go for sufficient size. This will >> absorb temporary usage peaks that can be very large and also provide extra >> endurance with SSDs with good controllers. >> >> I also t

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

iles[0 0 0 0 0 0 9] max score 0.00, MB/sec: 175.8 rd, > 173.9 > >> >> wr, > >> >> > level 6, files in(1, 9) out(9) MB in(0.3, 572.9) out(566.9), > >> >> > read-write-amplify(3434.6) write-amplify(1707.7) OK, records in: > >> 35108, > >> >

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Wednesday, October 11, 2023 4:30 PM To: Eugen Block Cc: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: Ceph 16.2.x mon compactions, disk writes Eugen, Thanks for your response. May I ask what numbers you're referring to? I am not referring to monitor store.db sizes. I am specifically

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

nt": "compaction_finished", > >> >> > "compaction_time_micros": 3418822, "compaction_time_cpu_micros": > >> 785454, > >> >> > "output_level": 6, "num_output_files": 9, "total_output_size":

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

ceph fs clients and >> about >> >> 800 librbd clients. Per week our mon logs are about 70M, the cluster >> logs >> >> about 120M , the audit logs about 70M and I see between 100-200Kb/s >> writes >> >> to the mon store. That's in the lower-di

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

; > is normal and what can be done about it. > >> > > >> > /Z > >> > > >> > On Wed, 11 Oct 2023 at 13:55, Frank Schilder wrote: > >> > > >> >> I need to ask here: where exactly do you observe the hundreds of GB > >>

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

ogging to MON store" >> issue that was present starting with pacific but should have been addressed >> by now? >> >> @Eugen: Was there not an option to limit logging to the MON store? >> >> For information to readers, we followed old recommendations from a Dell >> white paper for building a

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

doing stuff? > >> Is backfill going on? > >> Is recovery going on? > >> Is your ceph version affected by the "excessive logging to MON store" > >> issue that was present starting with pacific but should have been > addressed > >> by now? > >&

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

cluster is 500M-1G, but we have seen this ballooning to 100+GB in degraded conditions. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Zakhar Kirpichenko Sent: Wednesday, October 11, 2023 12:00 PM To: Eugen Block C

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

ON store for a healthy cluster is 500M-1G, but > we have seen this ballooning to 100+GB in degraded conditions. > > Best regards, > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > > From: Zakhar Kirp

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

3 12:00 PM To: Eugen Block Cc: ceph-users@ceph.io Subject: [ceph-users] Re: Ceph 16.2.x mon compactions, disk writes Thank you, Eugen. I'm interested specifically to find out whether the huge amount of data written by monitors is expected. It is eating through the endurance of our system drives,

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Thank you, Eugen. I'm interested specifically to find out whether the huge amount of data written by monitors is expected. It is eating through the endurance of our system drives, which were not specced for high DWPD/TBW, as this is not a documented requirement, and monitors produce hundreds of

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Hi, what you report is the expected behaviour, at least I see the same on all clusters. I can't answer why the compaction is required that often, but you can control the log level of the rocksdb output: ceph config set mon debug_rocksdb 1/5 (default is 4/5) This reduces the log entries

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Any input from anyone, please? On Tue, 10 Oct 2023 at 09:44, Zakhar Kirpichenko wrote: > Any input from anyone, please? > > It's another thing that seems to be rather poorly documented: it's unclear > what to expect, what 'normal' behavior should be, and what can be done > about the huge amount

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Any input from anyone, please? It's another thing that seems to be rather poorly documented: it's unclear what to expect, what 'normal' behavior should be, and what can be done about the huge amount of writes by monitors. /Z On Mon, 9 Oct 2023 at 12:40, Zakhar Kirpichenko wrote: > Hi, > >