[ceph-users] ceph/daemon stable tag

2023-01-30 Thread Jonas Nemeikšis
Hello, What's the status with the *-stable-* tags? https://quay.io/repository/ceph/daemon?tab=tags No longer build/support? What should we use until we'll migrate from ceph-ansible to cephadm? Thanks. -- Jonas ___ ceph-users mailing list --

[ceph-users] Re: Permanently ignore some warning classes

2023-01-30 Thread Reed Dier
You should be able to do `ceph health mute OSD_SLOW_PING_TIME_BACK --sticky` to mute the health warn/error state as it flaps. You can also set a TTL for the mute (1d, 1w, 1m) to have it roll off after a specific time. Code here is the warning error such as OSD_SLOW_PING_TIME_BACK or

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

2023-01-30 Thread Victor Rodriguez
On 1/30/23 15:15, Ana Aviles wrote: Hi, Josh already suggested, but I will one more time. We had similar behaviour upgrading from Nautilus to Pacific. In our case compacting the OSDs did the trick. Thanks for chimming in! Unfortunately, in my case neither an online compaction (ceph tell

[ceph-users] Re: Replacing OSD with containerized deployment

2023-01-30 Thread mailing-lists
oph wait, i might have been too impatient: 1/30/23 4:43:07 PM[INF]Deploying daemon osd.232 on ceph-a1-06 1/30/23 4:42:26 PM[INF]Found osd claims for drivegroup dashboard-admin-1661788934732 -> {'ceph-a1-06': ['232']} 1/30/23 4:42:26 PM[INF]Found osd claims -> {'ceph-a1-06': ['232']}

[ceph-users] Re: Replacing OSD with containerized deployment

2023-01-30 Thread mailing-lists
root@ceph-a2-01:/# ceph osd destroy 232 --yes-i-really-mean-it destroyed osd.232 OSD 232 shows now as destroyed and out in the dashboard. root@ceph-a1-06:/# ceph-volume lvm zap /dev/sdm --> Zapping: /dev/sdm --> --destroy was not specified, but zapping a whole device will remove the

[ceph-users] Re: Write amplification for CephFS?

2023-01-30 Thread Janne Johansson
I think you will have to do the science there, it will be dependent on thousands of factors like size of OSD, WAL, DB, placement of them, size of incoming data if it is many small ops or few large ones and so on. I don't think any one single answer there would work out for many cases. Most of us

[ceph-users] Re: Replacing OSD with containerized deployment

2023-01-30 Thread David Orman
The 'down' status is why it's not being replaced, vs. destroyed, which would allow the replacement. I'm not sure why --replace lead to that scenario, but you will probably need to mark it destroyed for it to be replaced.

[ceph-users] Re: Write amplification for CephFS?

2023-01-30 Thread Manuel Holtgrewe
OK. How much data will be written to the WAL and elsewhere? On Mon, Jan 30, 2023 at 3:17 PM Janne Johansson wrote: > > I'm concerned with the potential increased NVME wear. Assuming writes of > multiples of the block size, when I write 1GB of data to the CephFS, how > much data is written to

[ceph-users] Re: Write amplification for CephFS?

2023-01-30 Thread Janne Johansson
> I'm concerned with the potential increased NVME wear. Assuming writes of > multiples of the block size, when I write 1GB of data to the CephFS, how much > data is written to the disks? In that case, repl=3 will write 1GB to three PGs. EC 8+3 would write 125M (ie, 1/8 of a GB) to 11 (8+3)

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

2023-01-30 Thread Ana Aviles
Hi, Josh already suggested, but I will one more time. We had similar behaviour upgrading from Nautilus to Pacific. In our case compacting the OSDs did the trick. For us there was no performance impact running the compaction (ceph osd daemon osd.0 compact) although we run them in batches and

[ceph-users] Re: Write amplification for CephFS?

2023-01-30 Thread Manuel Holtgrewe
Dear Janne, I'm concerned with the potential increased NVME wear. Assuming writes of multiples of the block size, when I write 1GB of data to the CephFS, how much data is written to the disks? Best wishes, Manuel On Mon, Jan 30, 2023 at 2:25 PM Janne Johansson wrote: > > is there information

[ceph-users] Re: Write amplification for CephFS?

2023-01-30 Thread Janne Johansson
> is there information available anywhere about the write amplification for > CephFS? I found quite some material on write amplification of VMs using > journaled file system on top of RBD but nothing as it relates to CephFS? > > From my understanding I would expect the following: > > - for X-rep,

[ceph-users] Re: rbd online sparsify image

2023-01-30 Thread Ilya Dryomov
On Sun, Jan 29, 2023 at 1:12 PM Jiatong Shen wrote: > > Hello Ilya, > >Thank you very much for the clarification! Another question, for some > historical reasons, there are still some > Luminous clients existing. Is it dangerous to sparsify an image which is > still being used by a

[ceph-users] OSDs will not start

2023-01-30 Thread Geoffrey Rhodes
Hey all, I haven't managed to solve this issue yet. To simplify things, I'm looking to restart one OSD which crashes shortly after starting. As mentioned before I've ruled out this being related to hardware. Not a dev but looking at the log my error occurs at this

[ceph-users] Write amplification for CephFS?

2023-01-30 Thread Manuel Holtgrewe
Dear all, is there information available anywhere about the write amplification for CephFS? I found quite some material on write amplification of VMs using journaled file system on top of RBD but nothing as it relates to CephFS? >From my understanding I would expect the following: - for X-rep,

[ceph-users] Re: Replacing OSD with containerized deployment

2023-01-30 Thread mailing-lists
# ceph orch osd rm status No OSD remove/replace operations reported # ceph orch osd rm 232 --replace Unable to find OSDs: ['232'] It is not finding 232 anymore. It is still shown as down and out in the Ceph-Dashboard.     pgs: 3236 active+clean This is the new disk shown as locked

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

2023-01-30 Thread Frank Schilder
Glad you found it, its quite a long one. We could never confirm that a manual compaction after an upgrade with quick fix on mount = true would solve the problem with snap trim. You might want to try off-line fixing using ceph-bluestore-tool repair --path /var/lib/ceph/osd/OSD_ID maybe:

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

2023-01-30 Thread Victor Rodriguez
I'm reading that thread on spinics. Thanks for pointing that out. The cluster was upgraded from Nautilus to Octopus first week of December. No previous version has been used in this cluster. I just don't remember if we used snapshots after the upgrade. bluestore_fsck_quick_fix_on_mount is at

[ceph-users] Re: Very slow snaptrim operations blocking client I/O

2023-01-30 Thread Frank Schilder
Hi Victor, out of curiosity, did you upgrade the cluster recently to octopus? We and others observed this behaviour when following one out of two routes to upgrade OSDs. There was a thread "Octopus OSDs extremely slow during upgrade from mimic", which seems to have been lost with the recent

[ceph-users] Re: rbd-mirror ceph quincy Not able to find rbd_mirror_journal_max_fetch_bytes config in rbd mirror

2023-01-30 Thread Eugen Block
Hi, it's not the best idea to create multiple threads for the same topic. I don't really have an answer for you, but maybe you could try to add more rbd-mirror daemons and see if the replication speed increases. I haven't tried that myself yet, though. Regards, Eugen Zitat von ankit