[ceph-users] Ceph 16.2.14: how to set mon_rocksdb_options to enable RocksDB compression?

Hi, I'm still trying to fight large Ceph monitor writes. One option I considered is enabling RocksDB compression, as our nodes have more than sufficient RAM and CPU. Unfortunately, monitors seem to completely ignore the compression setting: I tried: - setting ceph config set mon.ceph05

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

> Some of it is transferable to RocksDB on mons nonetheless. Please point me to relevant Ceph documentation, i.e. a description of how various Ceph monitor and RocksDB tunables affect the operations of monitors, I'll gladly look into it. > Please point me to such recommendations, if they're on

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

At home Im running a small cluster, Ceph v17.2.6, Debian 11 Bullseye. I have recently added a new server to the cluster but face the same problem as Patrick, I can't add any HDD. Ceph doesn't recognise them. I have run the same tests as Patrick, using Ceph v14-v18, and as Patrick showed the

[ceph-users] Re: Time Estimation for cephfs-data-scan scan_links

>> However, I've observed that the cephfs-data-scan scan_links step has >> been running for over 24 hours on 35 TB of data, which is replicated >> across 3 OSDs, resulting in more than 100 TB of raw data. What matters is the number of "inodes" (and secondarily their size), that is the number of

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Some of it is transferable to RocksDB on mons nonetheless. > but their specs exceed Ceph hardware recommendations by a good margin Please point me to such recommendations, if they're on docs.ceph.com I'll get them updated. > On Oct 13, 2023, at 13:34, Zakhar Kirpichenko

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

Thank you, Anthony. As I explained to you earlier, the article you had sent is about RocksDB tuning for Bluestore OSDs, while the issue at hand is not with OSDs but rather monitors and their RocksDB store. Indeed, the drives are not enterprise-grade, but their specs exceed Ceph hardware

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

cf. Mark's article I sent you re RocksDB tuning. I suspect that with Reef you would experience fewer writes. Universal compaction might also help, but in the end this SSD is a client SKU and really not suited for enterprise use. If you had the 1TB SKU you'd get much longer life, or you could

[ceph-users] Re: Time Estimation for cephfs-data-scan scan_links

>> However, I've observed that the cephfs-data-scan scan_links step has >> been running for over 24 hours on 35 TB of data, which is replicated >> across 3 OSDs, resulting in more than 100 TB of raw data. What matters is the number of "inodes" (and secondarily their size), that is the number of

[ceph-users] Re: Ceph 16.2.x mon compactions, disk writes

An interesting find: I reduced the number of PGs for some of the less utilized pools, which brought the total number of PGs in the cluster down from 2400 to 1664. The cluster is healthy, the only change was the 30% reduction of PGs, but mons now have a much smaller store.db, have much fewer

[ceph-users] Re: Is nfs-ganesha + kerberos actually a thing?

On 13-10-2023 16:57, John Mulligan wrote: On Friday, October 13, 2023 10:46:24 AM EDT Torkil Svensgaard wrote: On 13-10-2023 16:40, Torkil Svensgaard wrote: On 13-10-2023 14:00, John Mulligan wrote: On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote: Hi We have kerberos

[ceph-users] Re: Is nfs-ganesha + kerberos actually a thing?

On Friday, October 13, 2023 10:46:24 AM EDT Torkil Svensgaard wrote: > On 13-10-2023 16:40, Torkil Svensgaard wrote: > > On 13-10-2023 14:00, John Mulligan wrote: > >> On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote: > >>> Hi > >>> > >>> We have kerberos working with bare metal

[ceph-users] Re: Is nfs-ganesha + kerberos actually a thing?

On 13-10-2023 16:40, Torkil Svensgaard wrote: On 13-10-2023 14:00, John Mulligan wrote: On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote: Hi We have kerberos working with bare metal kernel NFS exporting RBDs. I can see in the ceph documentation[1] that nfs-ganesha should

[ceph-users] Re: Is nfs-ganesha + kerberos actually a thing?

On 13-10-2023 14:00, John Mulligan wrote: On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote: Hi We have kerberos working with bare metal kernel NFS exporting RBDs. I can see in the ceph documentation[1] that nfs-ganesha should work with kerberos but I'm having little luck

[ceph-users] Re: Please help collecting stats of Ceph monitor disk writes

ent time and count will give a correct average for the day starting at 00:00 failed in my situation. To get a correct count for today I would have had to combine 2 log files and send you the current time and count: # grep -i "manual compaction from" /var/log/ceph/ceph-mon.ceph-01.log-2

[ceph-users] Re: Please help collecting stats of Ceph monitor disk writes

Thank you, Frank. Tbh, I think it doesn't matter if the number of manual compactions is for 24h or for a smaller period, as long as it's over a reasonable period of time, so that an average number of compactions per hour can be calculated. /Z On Fri, 13 Oct 2023 at 16:01, Frank Schilder wrote:

[ceph-users] Re: Please help collecting stats of Ceph monitor disk writes

Hi Zakhar, I'm pretty sure you wanted the #manual compactions for an entire day, not from whenever the log starts to current time, which is most often not 23:59. You need to get the date from the previous day and make sure the log contains a full 00:00-23:59 window. 1) iotop results: TID

[ceph-users] Re: Is nfs-ganesha + kerberos actually a thing?

On Friday, October 13, 2023 6:11:18 AM EDT Torkil Svensgaard wrote: > Hi > > We have kerberos working with bare metal kernel NFS exporting RBDs. I > can see in the ceph documentation[1] that nfs-ganesha should work with > kerberos but I'm having little luck getting it to work. > Could you

[ceph-users] Re: Ceph 16.2.14: pgmap updated every few seconds for no apparent reason

I have just checked 2 quincy 17.2.6 clusters, and I see exactly the same. The pgmap version is bumping every two seconds (which ties in with the frequency you observed). Both clusters are healthy with nothing apart from client IO happening. On 13/10/2023 12:09, Zakhar Kirpichenko wrote: Hi,

[ceph-users] Ceph 16.2.14: pgmap updated every few seconds for no apparent reason

Hi, I am investigating excessive mon writes in our cluster and wondering whether excessive pgmap updates could be the culprit. Basically pgmap is updated every few seconds, sometimes over ten times per minute, in a healthy cluster with no OSD and/or PG changes: Oct 13 11:03:03 ceph03 bash[4019]:

[ceph-users] Re: Please help collecting stats of Ceph monitor disk writes

Here is some data from a small, very lightly loaded cluster. It is manually deployed on debian11, with the mon store on an SSD: 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND 1923 be/4 ceph 0.00 B104.00 K ?unavailable? ceph-mon -f

[ceph-users] Is nfs-ganesha + kerberos actually a thing?

Hi We have kerberos working with bare metal kernel NFS exporting RBDs. I can see in the ceph documentation[1] that nfs-ganesha should work with kerberos but I'm having little luck getting it to work. This bit from the container log seems to suggest that some plumbing is missing? "

[ceph-users] Re: Please help collecting stats of Ceph monitor disk writes

Hi, thanks for looking into this: our system disks also wear out too quickly! Here are the numbers on our small cluster. Best, 1) iotop results:     TID  PRIO  USER DISK READ  DISK WRITE  SWAPIN  IO COMMAND     TID  PRIO  USER DISK READ  DISK WRITE  SWAPIN  IO COMMAND    6426

[ceph-users] Re: [EXTERN] Please help collecting stats of Ceph monitor disk writes

Hi, this is on our nautilus cluster, not sure if it is relevant, however here are the results: 1) iotop results: TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND TID PRIO USER DISK READ DISK WRITE SWAPIN IOCOMMAND 1801 be/4 ceph 0.00 B

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

The server has enough available storage: [root@mostha1 log]# df -h Sys. de fichiers  Taille Utilisé Dispo Uti% Monté sur devtmpfs 24G   0   24G   0% /dev tmpfs    24G 84K   24G   1% /dev/shm tmpfs    24G   

[ceph-users] Re: [EXTERNAL] [Pacific] ceph orch device ls do not returns any HDD

Trying to resend with the attachment. I can't really find anything suspicious, ceph-volume (16.2.11) does recognize /dev/sdc though: [2023-10-12 08:58:14,135][ceph_volume.process][INFO ] stdout NAME="sdc" KNAME="sdc" PKNAME="" MAJ:MIN="8:32" FSTYPE="" MOUNTPOINT="" LABEL="" UUID=""

[ceph-users] Please help collecting stats of Ceph monitor disk writes

Hi! Further to my thread "Ceph 16.2.x mon compactions, disk writes" ( https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/XGCI2LFW5RH3GUOQFJ542ISCSZH3FRX2/) where we have established that Ceph monitors indeed write considerable amounts of data to disks, I would like to request fellow