[ceph-users] Re: External RGW always down

Hi Euden, Yes the osds stay online when i start them manually. No pg recovery starts automatically when the osd starts. I'm using an erasure coded pool for rgw .In that rule we have k=11 m=4 total 15 hosts and the crush rule is host . I didn't find any error logs in the osds. First time I

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

Just a datapoint - we upgraded several large Mimic-born clusters straight to 15.2.12 with the quick fsck disabled in ceph.conf, then did require-osd-release, and finally did the omap conversion offline after the cluster was upgraded using the bluestore tool while the OSDs were down (all done in

[ceph-users] Re: Ceph configuration for rgw

You can always "config get" what was set by "config set", cause that's just write and read KV to and from configuration DB. To "config show" what was set by "config set" requires the support for mgr to connect to the service daemon to get running config. I see such support for mgr, mon and osd,

[ceph-users] Re: laggy OSDs and staling krbd IO after upgrade from nautilus to octopus

Hi Frank, Thank you very much for this! :) > > we just completed a third upgrade test. There are 2 ways to convert the > OSDs: > > A) convert along with the upgrade (quick-fix-on-start=true) > B) convert after setting require-osd-release=octopus (quick-fix-on- > start=false until

[ceph-users] Re: osds not bootstrapping: monclient: wait_auth_rotating timed out

Yes, we restarted the primary mon and mgr services. Still no luck. From: Dhairya Parmar Sent: Monday, September 26, 2022 3:44 PM To: Wyll Ingersoll Cc: ceph-users@ceph.io Subject: Re: [ceph-users] osds not bootstrapping: monclient: wait_auth_rotating timed

[ceph-users] Re: osds not bootstrapping: monclient: wait_auth_rotating timed out

Looking at the shared tracker, I can see people talking about restarting primary mon/mgr and getting this fixed at note-4 and note-8 . Did you try that out? On Tue, Sep 27, 2022 at 12:44 AM Wyll Ingersoll

[ceph-users] Re: Ceph Cluster clone

Can you provide some more information on this? Can you show exactly what error you get while trying to start the cluster? On Mon, Sep 26, 2022 at 7:19 PM Ahmed Bessaidi wrote: > Hello, > I am working on cloning an existent Ceph Cluster (VMware). > I fixed the IP/hostname part, but I cannot get

[ceph-users] Re: Newer linux kernel cephfs clients is more trouble?

Stefan Kooman schreef op 2022-05-11 18:06: Hi List, We have quite a few linux kernel clients for CephFS. One of our customers has been running mainline kernels (CentOS 7 elrepo) for the past two years. They started out with 3.x kernels (default CentOS 7), but upgraded to mainline when those

[ceph-users] osds not bootstrapping: monclient: wait_auth_rotating timed out

Ceph Pacific (16.2.9) on a large cluster. Approximately 60 (out of 700) osds fail to start and show an error: monclient: wait_auth_rotating timed out after 300 We modified the "rotating_keys_bootstrap_timeout" from 30 to 300, but they still fail. All nodes are time-synced with NTP and the

[ceph-users] Re: weird performance issue on ceph

Hi Zoltan and Mark, this observation of performance loss when a solid state drive gets full and/or exceeded a certain number of write OPS is very typical even for enterprise SSDs. This performance drop can be very dramatic. Therefore, I'm reluctant to add untested solid state drives (SSD/NVMe)

[ceph-users] Re: Cephadm credential support for private container repositories

On Monday, September 26, 2022 12:53:04 PM EDT Gary Molenkamp wrote: > I'm trying to determine whether cephadm can use credential based login > for container images from private repositories. I don't see anything > obvious on the official documentation for cephadm to specify the > credentials

[ceph-users] Re: External RGW always down

Yes, I have an inactive pgs when the osd goes down. Then I started the osds manually. But the rgw fails to start. But the OSDs stay online if you start them manually? Do the inactive PGs recover when you start them manually? By the way, you should check your crush rules, depending on how

[ceph-users] Re: PGImbalance

Is the autoscaler running [1]? You can see the status with: ceph osd pool autoscale-status If it's turned off you can enable warn mode first to see what it would do: ceph osd pool set pg_autoscale_mode warn If the autoscaler doesn't help you could increase the pg_num manually to 512 and

[ceph-users] Cephadm credential support for private container repositories

I'm trying to determine whether cephadm can use credential based login for container images from private repositories.  I don't see anything obvious on the official documentation for cephadm to specify the credentials to use.   Can someone confirm whether this is supported? The motivation for

[ceph-users] Re: Slow OSD startup and slow ops

Hello Stefan, Thank you for your answers. On Thu, Sep 22, 2022 at 5:54 PM Stefan Kooman wrote: > Hi, > > On 9/21/22 18:00, Gauvain Pocentek wrote: > > Hello all, > > > > We are running several Ceph clusters and are facing an issue on one of > > them, we would appreciate some input on the

[ceph-users] Re: weird performance issue on ceph

Hi Zoltan, Great investigation work!  I think in my tests the data set typically was smaller than 500GB/drive.  If you have a simple fio test that can be run against a bare NVMe drive I can try running it on one of our test nodes.  FWIW I kind of suspected that the issue I had to work around

[ceph-users] PGImbalance

Dear Ceph-Users, i've recently setup a 4.3P Ceph-Cluster with cephadm. I am seeing that the health is ok, as seen here: ceph -s   cluster:     id: 8038f0xxx     health: HEALTH_OK   services:     mon: 5 daemons, quorum ceph-a2-07,ceph-a1-01,ceph-a1-10,ceph-a2-01,ceph-a1-05 (age 3w)    

[ceph-users] Re: External RGW always down

Hi Eugen, Yes, I have an inactive pgs when the osd goes down. Then I started the osds manually. But the rgw fails to start. Only upgrading to a newer version is only for the issue and we faced this issue two times. I dont know why it is happening. But maybe the rgw are running in separate

[ceph-users] Ceph Cluster clone

Hello, I am working on cloning an existent Ceph Cluster (VMware). I fixed the IP/hostname part, but I cannot get the cloned cluster to start (Monitors issues). Any ideas ? Best Regards, Ahmed. ___ ceph-users mailing list -- ceph-users@ceph.io To

[ceph-users] Re: MDS crashes after evicting client session

You can find the upstream fix here https://github.com/ceph/ceph/pull/46833 Thanks, Kotresh HR On Mon, Sep 26, 2022 at 3:17 PM Dhairya Parmar wrote: > Patch for this has already been merged and backported to quincy as well. It > will be there in the next Quincy release. > > On Thu, Sep 22, 2022

[ceph-users] Re: weird performance issue on ceph

Hi Mark and the mailing list, we managed to figure something very weird out what I would like to share with you and ask if you have seen anything like this before. We started to investigate the drives one-by-one after Mark's suggestion that a few osd-s are holding back the ceph and we noticed

[ceph-users] Re: MDS crashes after evicting client session

Patch for this has already been merged and backported to quincy as well. It will be there in the next Quincy release. On Thu, Sep 22, 2022 at 5:12 PM E Taka <0eta...@gmail.com> wrote: > Ceph 17.2.3 (dockerized in Ubuntu 20.04) > > The subject says it. The MDS process always crashes after

[ceph-users] Re: HA cluster

You should give this doc https://docs.ceph.com/en/quincy/rados/configuration/mon-config-ref/#monitor-quorum a read. Will help you understand and set up the HA cluster much better. Long story short, you would need at least 3 MONs to achieve HA because of the monitor quoram. On Sun, Sep 25, 2022 at

[ceph-users] Re: HA cluster

We need at least 3 hosts to achieve HA with a shared storage. If one node is turned off/fails , the storage is stopped. On Mon, Sep 26, 2022 at 2:01 PM Neeraj Pratap Singh wrote: > We need at least 3 hosts to achieve HA with a shared storage. > If one node is turned off/fails , the storage is

[ceph-users] Re: HA cluster

Am 25.09.22 um 19:20 schrieb Murilo Morais: I set up two hosts with cephadm, You cannot have HA with only two hosts. You need at least three separate hosts for three MONs to keep your cluster running. Regards -- Robert Sander Heinlein Consulting GmbH Schwedter Str. 8/9b, 10119 Berlin

[ceph-users] Re: Ceph configuration for rgw

Just adding this: ses7-host1:~ # ceph config set client.rgw.ebl-rgw rgw_frontends "beast port=8080" This change is visible in the config get output: client.rgw.ebl-rgwbasic rgw_frontendsbeast port=8080 Zitat von Eugen Block : Hi, the docs [1] show how to specifiy the

[ceph-users] Re: Low read/write rate

Den lör 24 sep. 2022 kl 23:38 skrev Murilo Morais : > I'm relatively new to Ceph. I set up a small cluster with two hosts with 12 > disks each host, all 3 TB SAS 7500 RPM and two 10 Gigabit interfaces. I > created a pool in replicated mode and configured it to use two replicas. > > What I'm

[ceph-users] Re: Ceph configuration for rgw

Hi, the docs [1] show how to specifiy the rgw configuration via yaml file (similar to OSDs). If you applied it with ceph orch you should see your changes in the 'ceph config dump' output, or like this: ---snip--- ses7-host1:~ # ceph orch ls | grep rgw rgw.ebl-rgw?:80 2/2