[ceph-users] Re: Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0

; unable to `ls lost_found`, or otherwise interact with the directory's > contents. > > Best, > Justin lee > > On Fri, Aug 2, 2024 at 8:24 AM Dhairya Parmar wrote: > >> Hi Justin, >> >> You should able to delete inodes from the lost+found dirs just by sim

[ceph-users] Re: Ceph MDS failing because of corrupted dentries in lost+found after update from 17.2.7 to 18.2.0

Hi Justin, You should able to delete inodes from the lost+found dirs just by simply `sudo rm -rf lost+found/` What do you get when you try to delete? Do you get `EROFS`? On Fri, Aug 2, 2024 at 8:42 AM Justin Lee wrote: > After we updated our ceph cluster from 17.2.7 to 18.2.0 the MDS kept bein

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

as possibly not be able to view the solution? Would > it not be possible to discuss this more publicly for the benefit of the > other users on the mailing list? > Kindest regards, > > Ivan > On 09/07/2024 10:44, Dhairya Parmar wrote: > > CAUTION: This email originated from ou

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

y > MDSs don't seem to enter the replay stage when they crash (or at least > don't now after I've thrown the logs away) but those errors do crop up in > the prior logs I shared when the system first crashed. > > Kindest regards, > > Ivan > On 08/07/2024 14:08,

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

t; last_failure0 > last_failure_osd_epoch494515 > compatcompat={},rocompat={},incompat={1=base v0.20,2=client writeable > ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds > uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline > data,8=no a

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

switching the mount points to `wsync` and also set the MDS config `mds_client_delegate_inos_pct` to `0` so that you don't end up in this situation again (until the bug fix arrives :)). [0] https://github.com/ceph/ceph-client/commit/f7a67b463fb83a4b9b11ceaa8ec4950b8fb7f902 > Kindest regards, > > Ivan >

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

mport-export [1] https://docs.ceph.com/en/latest/man/8/ceph-post-file On Thu, Jun 27, 2024 at 5:09 PM Dhairya Parmar wrote: > Hi Ivan, > > The solution (which has been successful for us in the past) is to reset > the journal. This would bring the fs back online and return the MDSes to

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

x5592e532c7b7] > 8: (EUpdate::replay(MDSRank*)+0x61) [0x5592e5330bd1] > 9: (MDLog::_replay_thread()+0x7bb) [0x5592e52b754b] > 10: (MDLog::ReplayThread::entry()+0x11) [0x5592e4f6a041] > 11: /lib64/libpthread.so.0(+0x81ca) [0x7f18558a41ca] > 12: clone() > > We have a relativ

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

as we can easily > induce this crash. > Since you can easily induce the crash, can you share the reproducer please i.e. what all action you take in order to hit this? > Kindest regards, > > Ivan > On 25/06/2024 09:58, Dhairya Parmar wrote: > > CAUTION: This email originat

[ceph-users] Re: CephFS MDS crashing during replay with standby MDSes crashing afterwards

provide us more logs. CC @Rishabh Dave @Venky Shankar @Patrick Donnelly @Xiubo Li [0] https://tracker.ceph.com/issues/61009 [1] https://tracker.ceph.com/issues/66251 -- *Dhairya Parmar* Associate Software Engineer, CephFS <https://www.redhat.com/>IBM, Inc. On Mon, Jun 24, 2024 at 8

[ceph-users] Re: How to recover from an MDs rank in state 'failed'

wrong and can lead to an inaccessible file system, more info regarding the command can be found at [0] and [1]. [0] https://docs.ceph.com/en/quincy/man/8/ceph/ [1] https://docs.ceph.com/en/latest/cephfs/administration/#advanced -- *Dhairya Parmar* Associate Software Engineer, CephFS <https://w

[ceph-users] Re: Linux Laptop Losing CephFS mounts on Sleep/Hibernate

eveal better insights. *Dhairya Parmar* Associate Software Engineer, CephFS <https://www.redhat.com/>IBM, Inc. On Thu, Mar 28, 2024 at 4:42 PM Kotresh Hiremath Ravishankar < khire...@redhat.com> wrote: > I think the client should reconnect when it's out of sleep. Could you >

[ceph-users] Re: Clients failing to advance oldest client?

I think this bug has already been worked on in https://tracker.ceph.com/issues/63364, can you tell which version you're on? -- *Dhairya Parmar* Associate Software Engineer, CephFS IBM, Inc. On Tue, Mar 26, 2024 at 2:32 AM Erich Weiler wrote: > Hi Y'all, > > I'm seein

[ceph-users] Re: ambigous mds behind on trimming and slowops (ceph 17.2.5 and rook operator 1.10.8)

Hi, May I know which version is being used in the cluster? It was started after 2 hours of one of the active mds was crashed Do we know the reason for the crash? Please share more info, `ceph -s` and MDS logs should reveal more insights. -- *Dhairya Parmar* Associate Software Engineer

[ceph-users] Re: Ceph MDS randomly hangs when pg nums reduced

share MDS logs to see what's going on exactly and if there is something that needs attention. -- *Dhairya Parmar* Associate Software Engineer, CephFS IBM, Inc. On Fri, Feb 23, 2024 at 8:27 PM wrote: > Hi, > > I have a CephFS cluster > ``` > > ceph -s > > cluster

[ceph-users] Re: CephFS warning: clients laggy due to laggy OSDs

are being said laggy and you will see the lagginess go away. *Dhairya Parmar* Associate Software Engineer, CephFS Red Hat Inc. <https://www.redhat.com/> dpar...@redhat.com <https://www.redhat.com/> On Wed, Sep 20, 2023 at 3:25 PM Venky Shankar wrote: > Hey Janek, > > I t

[ceph-users] Re: Client failing to respond to capability release

Hi Frank, This usually happens when the client is buggy/unresponsive. This warning is triggered when the client fails to respond to MDS's request to release caps in time which is determined by session_timeout(defaults to 60 secs). Did you make any config changes? *Dhairya Parmar* Asso

[ceph-users] Re: what is the point of listing "auth: unable to find a keyring on /etc/ceph/ceph.client nfs-ganesha

Okay then I'd suggest adding keyring to the client section in ceph.conf, it is as simple as keyring = /keyring I hope the client(that the logs complain) is in the keyring file. Do let me know if that works for you, if not, some logs would be good to have to diagnose further. On Fri, Jul 21, 2023

[ceph-users] Re: what is the point of listing "auth: unable to find a keyring on /etc/ceph/ceph.client nfs-ganesha

Hi Marc, Can you confirm if the mon ip in ceph.conf is correct and is public; also the keyring path is specified correctly? *Dhairya Parmar* Associate Software Engineer, CephFS Red Hat Inc. <https://www.redhat.com/> dpar...@redhat.com <https://www.redhat.com/> On Thu, Jul 2

[ceph-users] Re: Delete or move files from lost+found in cephfs

Hi, These symptoms look relevant to [0] and its PR is already merged in main; backported to quincy but pacific and reef are pending. [0] https://tracker.ceph.com/issues/59569 - Dhairya On Tue, Jul 4, 2023 at 1:54 AM Thomas Widhalm wrote: > Hi, > > I had some trouble in the past with my CephF

[ceph-users] Re: what are the options for config a CephFS client session

Hi, There's just one option for `session config` (or `client config` both are same) as of now i.e. "timeout" #> ceph tell mds.0 session config timeout *Dhairya Parmar* Associate Software Engineer, CephFS On Mon, Jun 12, 2023 at 2:29 PM Denis Polom wrote: > Hi, >

[ceph-users] Re: MDS Upgrade from 17.2.5 to 17.2.6 not possible

On Wed, May 17, 2023 at 9:26 PM Henning Achterrath wrote: > Hi all, > > we did a major update from Pacific to Quincy (17.2.5) a month ago > without any problems. > > Now we have tried a minor update from 17.2.5 to 17.2.6 (ceph orch > upgrade). It stucks at mds upgrade phase. At this point the clu

[ceph-users] Re: Unable to restart mds - mds crashes almost immediately after finishing recovery

Apart from PR mentioned by xiubo, #49691 also contains a good fix for this issue. - Dhairya On Fri, May 5, 2023 at 6:32 AM Xiubo Li wrote: > Hi Emmanuel, > > This should be one known issue as https://tracker.ceph.com/issues/58392 > and there is one fix

[ceph-users] Re: Ceph v15.2.14 - Dirty Object issue

Did you try options from cache-sizing or other-tunables ? - Dhairya On Fri, Mar 3, 2023 at 5:39 AM wrote: > Hi, we have a cluster with this ceph df > > --- RAW STORAGE

[ceph-users] Re: Recent ceph.io Performance Blog Posts

If this is the same issue that affected a couple of PRs in the last few weeks then rebasing the PR with the latest fetch of the main branch and force pushing it should solve the problem. - Dhairya On Fri, Dec 23, 2022 at 7:12 PM Stefan Kooman wrote: > On 12/19/22 10:26, Stefan Kooman wrote: > >

[ceph-users] Re: MDS_DAMAGE dir_frag

Hi there, You might want to look at [1] for this, also I found a relevant thread [2] that could be helpful. [1] https://docs.ceph.com/en/latest/cephfs/disaster-recovery-experts/#disaster-recovery-experts [2] https://www.spinics.net/lists/ceph-users/msg53202.html - Dhairya On Mon, Dec 12, 2022

[ceph-users] Re: what happens if a server crashes with cephfs?

Hi Charles, There are many scenarios where the write/close operation can fail but generally failures/errors are logged (normally every time) to help debug the case. Therefore there are no silent failures as such except you encountered a very rare bug. - Dhairya On Wed, Dec 7, 2022 at 11:38 PM C

[ceph-users] Re: How to determine if a filesystem is allow_standby_replay = true

m yours. > > Respectfully, > > *Wes Dillingham* > w...@wesdillingham.com > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > > > On Thu, Oct 20, 2022 at 2:43 PM Dhairya Parmar wrote: > >> Hi Wesley, >> >> You can find if the `allow_stan

[ceph-users] Re: How to determine if a filesystem is allow_standby_replay = true

ingham.com > LinkedIn <http://www.linkedin.com/in/wesleydillingham> > ___________ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > >

[ceph-users] Re: why rgw generates large quantities orphan objects?

gt; > ceph verison: > > ``` > > [root@node01 /]# ceph versions > > { > >"mon": { > >"ceph version 16.2.10 (45fa1a083152e41a408d15505f594ec5f1b4fe17) > > pacific (stable)": 3 > >}, > >"mgr": { >

[ceph-users] Re: osds not bootstrapping: monclient: wait_auth_rotating timed out

er suggestions on how to get these OSDs to sync up with the cluster? > > > thanks! > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- *Dhairya Parmar* He/Him/His A

[ceph-users] Re: Ceph Cluster clone

annot get the cloned cluster to start > (Monitors issues). > Any ideas ? > > > > > Best Regards, > Ahmed. > > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@

[ceph-users] Re: MDS crashes after evicting client session

, >"process_name": "ceph-mds", >"stack_sig": > "b75e46941b5f6b7c05a037f9af5d42bb19d82ab7fc6a3c168533fc31a42b4de8", >"timestamp": "2022-09-22T11:26:24.013274Z", >"utsname_hostname": "ceph03&quo

[ceph-users] Re: HA cluster

I turn off the first, the second > is totally irresponsible. What could be causing this? > ___ > ceph-users mailing list -- ceph-users@ceph.io > To unsubscribe send an email to ceph-users-le...@ceph.io > > -- *Dhairya Parmar* He/Him/H

[ceph-users] Re: MDS crashes after evicting client session

on_id": "8", >"process_name": "ceph-mds", >"stack_sig": > "b75e46941b5f6b7c05a037f9af5d42bb19d82ab7fc6a3c168533fc31a42b4de8", >"timestamp": "2022-09-22T11:26:24.013274Z", >"utsname_hostname&

[ceph-users] Re: Multi-active MDS cache pressure

0003bff", > >>>> "entity_id": "cephfs_client", > >>>> > >>>> > >>>> # ceph fs status > >>>> > >>>> cephfs - 25 clients > >>>> == > >>>>

[ceph-users] Re: cephfs: num_stray growing without bounds (octopus)

s fail to my benchmark and report back how all of this > works under heavy load. > Good to hear it keeps the track. Yeah, that report will be very helpful. Thanks in advance! > > Best regards and thanks for our help! > = > Frank Schilder > AIT Risø Campus > Bygning

[ceph-users] Re: cephfs: num_stray growing without bounds (octopus)

> Thanks for your fast reply and best regards, > = > Frank Schilder > AIT Risø Campus > Bygning 109, rum S14 > > > From: Dhairya Parmar > Sent: 05 August 2022 16:10 > To: Frank Schilder > Subject: Re: [ceph-users] ceph

[ceph-users] Re: Troubleshooting cephadm - not deploying any daemons

Hi Zach, Try running `ceph orch apply mgr 2` or `ceph orch apply mgr --placement=" "`. Refer this doc for more information, hope it helps. Regards, Dhairya On Thu, Jun 9, 2022 at 1:59 AM Zach Heise (SSCC) wrote:

[ceph-users] Re: Maintenance mode?

Hi Jeremy, I think there is a maintenance mode for Ceph, maybe check this out or maybe this could help too. T

[ceph-users] Re: Recommendations on books

Hi Angelo, Publications and RPs: You can follow this link , it contains all the Ceph publications and research papers that will substantially help you understand Ceph and its umbrella(Ceph's components). Ceph Architecture: link

[ceph-users] Re: zap an osd and it appears again

Can you try using the --force option with your command? On Thu, Mar 31, 2022 at 1:25 AM Alfredo Rezinovsky wrote: > I want to create osds manually > > If I zap the osd 0 with: > > ceph orch osd rm 0 --zap > > as soon as the dev is available the orchestrator creates it again > > If I use: > > ce