[ceph-users] Re: Force processing of num_strays in mds

On Tue, May 18, 2021 at 5:40 AM Mark Schouten wrote: > I have a 12.2.13 I want to go and upgrade. However, there are a whole > bunch of stray files/inodes(?) which I would want to have processed. > Also because I get a lot of 'No space left on device' messages. I > started a 'find . -ls' in the

[ceph-users] One mds daemon damaged, filesystem is offline. How to recover?

Hi all An accidental power failure happened. That resulted CephFS offline and cannot be mounted. I have 3 MDS daemons but it complains "1 mds daemon damaged". It seems a PG of cephfs_metadata is inconsistent. I tried to repair, but doesn't get it repaired. How do I repair the damaged MDS and

[ceph-users] mgr+Prometheus, grafana, consul

I recently configured Prometheus to scrape mgr /metrics and add Grafana dashboards. All daemons currently at 15.2.11 I use Hashicorp consul to advertise the active mgr in DNS, and Prometheus points at a single DNS target. (Is anyone else using this method, or just statically pointing Prometheus

[ceph-users] Re: One mds daemon damaged, filesystem is offline. How to recover?

Sorry, forgot the links: [1] https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/4WJ5IUANKYO4AY333QWUGBK5CFBZW43V/ [2] https://tracker.ceph.com/issues/43068 Zitat von Eugen Block : Hi, I went through similar trouble just this week [1], but the root cause seems different so

[ceph-users] Re: One mds daemon damaged, filesystem is offline. How to recover?

Hi, I went through similar trouble just this week [1], but the root cause seems different so it probably won't apply to your case. Which version of ceph are you running? There are a couple of reports with similar error messages, e. g. [2], it may already been resolved. Can you share rados

[ceph-users] Re: ceph orch status hangs forever

Hi Eugen, Thank you very much for your help! In the end we found that running "sudo systemctl restart ceph.target" on each Ceph node, one by one, monitoring the health of the Cluster with "ceph status" on a separate terminal has been the solution. After restarting everything all commands are now

[ceph-users] Re: orch apply mon assigns wrong IP address?

The public network is defined, and it happens anyway (the temporary, unlisted interface address is in the public network, but not either forward or reverse resolveable to the host in dns or /etc/hosts). Okay, then I don't have a solution for that. Maybe someone else has dealt with this

[ceph-users] Re: orch apply mon assigns wrong IP address?

On 5/21/21 9:49 AM, Eugen Block wrote: > You can define the public_network [1]: > > ceph config set mon public_network ** > > For example: > > ceph config set mon public_network 10.1.2.0/24 > > Or is that already defined and it happens anyway? The public network is defined, and it happens anyway

[ceph-users] Re: orch apply mon assigns wrong IP address?

You can define the public_network [1]: ceph config set mon public_network ** For example: ceph config set mon public_network 10.1.2.0/24 Or is that already defined and it happens anyway? [1] https://docs.ceph.com/en/latest/cephadm/mon/#designating-a-particular-subnet-for-monitors Zitat

[ceph-users] orch apply mon assigns wrong IP address?

Is there a way to force '.. orch apply  *' to limit ip address selection to addresses matching the hostname in dns or /etc/hosts, or to a specific address given at 'host add' time?   I've hit a bothersome problem: On v15, 'ceph orch apply mon ...' appears not to use the dns ip or /etc/hosts when

[ceph-users] Re: OSD's still UP after power loss

Is there any better solution? Yes, add more nodes. ;-) Having only two OSD nodes is not the best idea, it's kind of a corner case and I've observed some weird behaviour with corner cases in the past, not even mentioning the 2 replicas. Is this a test environment? Zitat von by morphin :

[ceph-users] Re: OSD's still UP after power loss

I've figured out but I'm scared from the result. The solution is "mon_osd_min_down_reporters = 1" Due to "two node" cluster and "replicated 2" with "chooseleaf host" the reporter count should be set to 1 but on a malfunction this could be a serious problem. Is there any better solution? by

[ceph-users] question regarding markers in radosgw

Hello everybody, It seems that I have a metric ton of orphan objects in my s3 cluster. They look like this: $ rados -p eu-central-1.rgw.buckets.data stat ff7a8b0c-07e6-463a-861b-78f0adeba8ad.811806.9_1063978/features/2018-02-23.json

[ceph-users] upmap+assimilate-conf clarification

Hi, I've successfully updated my luminous lab environment to nautilus so next week I'll give a try to the prod env but 2 things same up in the description: 1. Upmap: I've never used this before don't know how I couldn't see it because it is quite cool feature. So ceph says let it in upmap,

[ceph-users] Re: ceph osd df size shows wrong, smaller number

> On 21 May 2021, at 12:17, Rok Jaklič wrote: > > There isnt any manual method for bluestore. Your block is 107374182400, so report is current For bluestore is better to use "ceph-volume lvm batch /dev/sdb" command k ___ ceph-users mailing list --

[ceph-users] Re: ceph df: pool stored vs bytes_used -- raw or not?

> On 20 May 2021, at 21:09, Igor Fedotov wrote: > > Perhaps you're facing a different issue, could you please share "ceph osd > tree" output? Here: https://pastebin.com/bic4v5Xy Thanks, k ___ ceph-users mailing

[ceph-users] Re: ceph orch status hangs forever

Hi, But we are not sure if we can enable some of them. Now all the logs we have from Ceph are not showing errors. Would it help to see more logs to enable some of those modules? I would not enable more modules, that could make it worse. Instead you could try to disable

[ceph-users] Re: ceph osd df size shows wrong, smaller number

Ahh, thx for noticing that. Its the same for 56. We've figured out that this maybe due to the fact that this manual method is for filestore https://docs.ceph.com/en/latest/install/manual-deployment/#long-form only. There isnt any manual method for bluestore. On Fri, May 21, 2021 at 11:00 AM

[ceph-users] Re: ceph osd df size shows wrong, smaller number

Den fre 21 maj 2021 kl 10:49 skrev Rok Jaklič : > It shows > sdb8:16 0 5.5T 0 disk /var/lib/ceph/osd/ceph-56 That one says osd-56, you asked about why osd 85 was small in ceph osd df >> Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič : >> > why would ceph osd df show in

[ceph-users] Re: ceph osd df size shows wrong, smaller number

It shows sdb8:16 0 5.5T 0 disk /var/lib/ceph/osd/ceph-56 when I do ll on that dir it says [root@ctplosd8 ~]# ll /var/lib/ceph/osd/ceph-56 total 552 -rw--- 1 root root9 May 21 10:40 bfm_blocks -rw--- 1 root root4 May 21 10:40

[ceph-users] Re: ceph orch status hangs forever

Hi Eugen, this is the output: ceph mgr module ls { "always_on_modules": [ "balancer", "crash", "devicehealth", "orchestrator", "pg_autoscaler", "progress", "rbd_support", "status", "telemetry", "volumes" ],

[ceph-users] Re: ceph osd df size shows wrong, smaller number

Den fre 21 maj 2021 kl 09:41 skrev Rok Jaklič : > why would ceph osd df show in SIZE field smaller number than there is: > 85hdd 0.8 1.0 100 GiB 96 GiB 95 GiB 289 KiB 952 > MiB 4.3 GiB 95.68 3.37 10 up > > instead of 100GiB there should be 5.5TiB. What does

[ceph-users] ceph osd df size shows wrong, smaller number

Hi, why would ceph osd df show in SIZE field smaller number than there is: 85hdd 0.8 1.0 100 GiB 96 GiB 95 GiB 289 KiB 952 MiB 4.3 GiB 95.68 3.37 10 up instead of 100GiB there should be 5.5TiB. Kind regards, Rok ___

[ceph-users] Re: ceph orch status hangs forever

Hi, if you check ceph mgr module ls | jq -r '.always_on_modules[]' you'll see that crash, orchestrator and other modules are always on and can't be disabled. Without the pipe to jq you can see the whole list which is a bit long to get just an overview. Anyway, comparing your enabled