[ceph-users] How to specify to only build ceph-radosgw package from source?

Hi, I usually install the SRPM and then build from ceph.spec like this: rpmbuild -bb /root/rpmbuild/SPECS/ceph.spec --without ceph_test_package But it take a long time and contain many packages that I don't need. So is there a way to optimize this build process for only needed package, for

[ceph-users] Re: Small RGW objects and RADOS 64KB minimun size

Hey Josh! Sorry for necroing this thread, but my team is currently running a Pacific cluster that was updated from Nautilus, and we are rebuilding hosts one by one to reclaim the space in the OSDs. We might have missed it, but was the automated rolling format with cephadm eventually

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi Dan, Sorry, I meant Pacific. The version number was correct, the name wasn’t. ;-) Yes, I have five active MDS and five hot standbys. Static pinning isn’t really an options for our directory structure, so we’re using ephemeral pins. Janek > On 31. May 2023, at 18:44, Dan van der Ster

[ceph-users] Re: reef v18.1.0 QE Validation status

Casey I will rerun rgw and we will see. Stay tuned. On Wed, May 31, 2023 at 10:27 AM Casey Bodley wrote: > > On Tue, May 30, 2023 at 12:54 PM Yuri Weinstein wrote: > > > > Details of this release are summarized here: > > > > https://tracker.ceph.com/issues/61515#note-1 > > Release Notes - TBD

[ceph-users] Re: reef v18.1.0 QE Validation status

Orch approved. The orch/cephadm tests looked good and the orch/rook tests are known to not work currently. On Tue, May 30, 2023 at 12:54 PM Yuri Weinstein wrote: > Details of this release are summarized here: > > https://tracker.ceph.com/issues/61515#note-1 > Release Notes - TBD > > Seeking

[ceph-users] Re: reef v18.1.0 QE Validation status

On Tue, May 30, 2023 at 12:54 PM Yuri Weinstein wrote: > > Details of this release are summarized here: > > https://tracker.ceph.com/issues/61515#note-1 > Release Notes - TBD > > Seeking approvals/reviews for: > > rados - Neha, Radek, Travis, Ernesto, Adam King (we still have to > merge

[ceph-users] bucket notification retries

Dear Community, I would like to collect your feedback on this issue. This is a followup from a discussion that started in the RGW refactoring meeting on 31-May-23 (thanks @Krunal Chheda for bringing up this topic!). Currently persistent notifications are retried indefinitely. The only limiting

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi Janek, A few questions and suggestions: - Do you have multi-active MDS? In my experience back in nautilus if something went wrong with mds export between mds's, the mds log/journal could grow unbounded like you observed until that export work was done. Static pinning could help if you are not

[ceph-users] Re: BlueStore fragmentation woes

On 5/31/23 16:15, Igor Fedotov wrote: On 31/05/2023 15:26, Stefan Kooman wrote: On 5/29/23 15:52, Igor Fedotov wrote: Hi Stefan, given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed. Although you omitted historic probes so it's

[ceph-users] Re: RGW versioned bucket index issues

I've proposed some new radosgw-admin commands for both identifying and fixing these leftover index entries in this open PR: https://github.com/ceph/ceph/pull/51700 Cory From: Mark Nelson Sent: Wednesday, May 31, 2023 10:42 AM To: ceph-users@ceph.io Subject:

[ceph-users] Re: slow mds requests with random read test

Thank you Patrick for help. The random write tests are performing well enough, though. Wonder why read test is so poor with the same configuration(resulting read bandwidth about 15MB/s vs 400MB/s of write). especially the logs of slow requests are irrelevant with testing ops. I am thinking it is

[ceph-users] Re: [EXTERNAL] [Pacific] Admin keys no longer works I get access denied URGENT!!!

Greetings, Try: ceph -n mon. --keyring /var/lib/ceph//mon//keyring get-or-create client.admin mon 'allow *' mds 'allow *' mgr 'allow *' osd 'allow *' Thank you, Josh Beaman From: wodel youchi Date: Wednesday, May 31, 2023 at 5:39 AM To: ceph-users@ceph.io Subject: [EXTERNAL] [ceph-users]

[ceph-users] Re: all buckets mtime = "0.000000" after upgrade to 17.2.6

thanks for the report. this regression was already fixed in https://tracker.ceph.com/issues/58932 and will be in the next quincy point release On Wed, May 31, 2023 at 10:46 AM wrote: > > I was running on 17.2.5 since October, and just upgraded to 17.2.6, and now > the "mtime" property on all my

[ceph-users] Re: MDS corrupt (also RADOS-level copy?)

Forgot so say: As for your corrupt rank 0, you should check the logs with a higher debug level. Looks like you were less lucky than we were. Your journal position may be incorrect. This could be fixed by editing the journal header. You might also try to tell your MDS to skip corrupt entries.

[ceph-users] Re: BlueStore fragmentation woes

On 5/31/23 09:15, Igor Fedotov wrote: On 31/05/2023 15:26, Stefan Kooman wrote: On 5/29/23 15:52, Igor Fedotov wrote: Hi Stefan, given that allocation probes include every allocation (including short 4K ones) your stats look pretty high indeed. Although you omitted historic probes so

[ceph-users] Re: RGW versioned bucket index issues

Thank you Cory for this excellent write up!  A quick question: Is there a simple method to find and more importantly fix the zombie index entries and OLH objects? I saw in https://tracker.ceph.com/issues/59663 that there was an example using radosgw-admin to examine the

[ceph-users] Re: MDS corrupt (also RADOS-level copy?)

Hi Jake, Very interesting. This sounds very much like what we have been experiencing the last two days. We also had a sudden fill-up of the metadata pool, which repeated last night. See my question here:

[ceph-users] PGs incomplete - Data loss

Hi guys, I'm awake since 36h and try to restore a broken ceph Pool (2 PGs incomplete) My vm are all broken. Some Boot, some Dont Boot... Also I have 5 removed disk with Data of that Pool "in my Hands" - Dont ask... So my question is it possible to restore Data of these other disks and "add"

[ceph-users] all buckets mtime = "0.000000" after upgrade to 17.2.6

I was running on 17.2.5 since October, and just upgraded to 17.2.6, and now the "mtime" property on all my buckets is 0.00. On all previous versions going back to Nautilus this wasn't an issue, and we do like to have that value present. radosgw-admin has no quick way to get the last

[ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process

Hello all, Thank you very much for your valuable feedback. I'd like to provide some context and clarify certain points as there seems to be some confusion regarding the objective of this discussion and how a cephadm initial bootstrap works. As you know, Ceph has the capability to run multiple

[ceph-users] Re: MDS corrupt (also RADOS-level copy?)

Dear All, My apologies, I forgot to state we are using Quincy 17.2.6 thanks again, Jake root@wilma-s1 15:22 [~]: ceph -v ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable) Dear All, we are trying to recover from what we suspect is a corrupt MDS :( and have

[ceph-users] MDS corrupt (also RADOS-level copy?)

Dear All, we are trying to recover from what we suspect is a corrupt MDS :( and have been following the guide here: Symptoms: MDS SSD pool (2TB) filled completely over the weekend, normally uses less than 400GB, resulting in

[ceph-users] how to use ctdb_mutex_ceph_rados_helper

Hey, I have a test setup with a 3-node samba cluster. This cluster consists of 3 vm's storing its locks on a replicated gluster volume. I want to switch to 2 physical smb-gateways for performance reasons (not enough money for 3), and since the 2-node cluster can't get quorum, I hope to switch to

[ceph-users] Re: Important: RGW multisite bug may silently corrupt encrypted objects on replication

On Wed, May 31, 2023 at 7:24 AM Tobias Urdin wrote: > > Hello Casey, > > Understood, thanks! > > That means that the original copy in the site that it was uploaded to is still > safe as long as that copy is not removed, and no underlying changes below > RadosGW in the Ceph storage could corrupt

[ceph-users] Re: Important: RGW multisite bug may silently corrupt encrypted objects on replication

Hello Casey, Understood, thanks! That means that the original copy in the site that it was uploaded to is still safe as long as that copy is not removed, and no underlying changes below RadosGW in the Ceph storage could corrupt the original copy? Best regards Tobias On 30 May 2023, at 14:48,

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

I checked our logs from yesterday, the PG scaling only started today, perhaps triggered by the snapshot trimming. I disabled it, but it didn't change anything. What did change something was restarting the MDS one by one, which had got far behind with trimming their caches and with a bunch of

[ceph-users] Re: CEPH Version choice

Hi Frank, Thanks! I have added this to my test environment todo > > I uploaded all scripts and a rudimentary readme to > https://github.com/frans42/cephfs-bench . I hope it is sufficient to get > started. I'm afraid its very much tailored to our deployment and I can't > make it fully

[ceph-users] [Pacific] Admin keys no longer works I get access denied URGENT!!!

Hi, After a wrong manipulation, the admin key no longer works, it seems it has been modified. My cluster is built using containers. When I execute ceph -s I get [root@controllera ceph]# ceph -s 2023-05-31T11:33:20.940+0100 7ff7b2d13700 -1 monclient(hunting): handle_auth_bad_method server

[ceph-users] Re: BlueStore fragmentation woes

Hi Kevin, according to the shared probes there were no fragmented allocations - cnt = frags for all the probes.  And average allocation request is pretty large - more than 1.5 MB for the probes I checked. So to me it looks like your disk fragmentation (at least for new allocations) is of

[ceph-users] RGW versioned bucket index issues

Hi all, I wanted to call attention to some RGW issues that we've observed on a Pacific cluster over the past several weeks. The problems relate to versioned buckets and index entries that can be left behind after transactions complete abnormally. The scenario is multi-faceted and we're still

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Another thing I just noticed is that the auto-scaler is trying to scale the pool down to 128 PGs. That could also result in large fluctuations, but this big?? In any case, it looks like a bug to me. Whatever is happening here, there should be safeguards with regard to the pool's capacity.

[ceph-users] Re: Seeking feedback on Improving cephadm bootstrap process

I'm a new ceph user and I have some trouble with boostraping with cephadm: using Pacific or Quincy  no hard drive are detected by Ceph. Using Octopus all the hard drives are detected. As I do not know how to really clean, even a successful  install but not functional, each test  requires me a

[ceph-users] Re: CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Forgot to add: We are still on Nautilus (16.2.12). On 31/05/2023 09:53, Janek Bevendorff wrote: Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of

[ceph-users] CephFS metadata pool grows by two orders of magnitude while trimming (?) snapshots

Hi, Perhaps this is a known issue and I was simply too dumb to find it, but we are having problems with our CephFS metadata pool filling up over night. Our cluster has a small SSD pool of around 15TB which hosts our CephFS metadata pool. Usually, that's more than enough. The normal size of