[ceph-users] Re: MDS stuck in rejoin

2023-11-09 Thread Xiubo Li
Sent: Wednesday, November 8, 2023 1:38 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin Hi Frank, Recently I found a new possible case could cause this, please see https://github.com/ceph/ceph/pull/54259. This is just a ceph side fix, after this we need

[ceph-users] Re: MDS stuck in rejoin

2023-11-09 Thread Frank Schilder
regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Wednesday, November 8, 2023 1:38 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin Hi Frank, Recently I found

[ceph-users] Re: MDS stuck in rejoin

2023-11-07 Thread Xiubo Li
@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin On 8/7/23 21:54, Frank Schilder wrote: Dear Xiubo, I managed to collect some information. It looks like there is nothing in the dmesg log around the time the client failed to advance its TID. I collected short snippets around

[ceph-users] Re: MDS stuck in rejoin

2023-08-08 Thread Frank Schilder
To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin On 8/7/23 21:54, Frank Schilder wrote: > Dear Xiubo, > > I managed to collect some information. It looks like there is nothing in the > dmesg log around the time the client failed to advance its TID.

[ceph-users] Re: MDS stuck in rejoin

2023-08-07 Thread Xiubo Li
_ From: Xiubo Li Sent: Monday, July 31, 2023 12:14 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin On 7/31/23 16:50, Frank Schilder wrote: Hi Xiubo, its a kernel client. I actually made a mistake when trying to evict the client and my command

[ceph-users] Re: MDS stuck in rejoin

2023-08-07 Thread Frank Schilder
hilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Monday, July 31, 2023 12:14 PM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin On 7/31/23 16:50, Frank Schilder wrote: > Hi Xiubo, > > its

[ceph-users] Re: MDS stuck in rejoin

2023-07-31 Thread Xiubo Li
31, 2023 4:12 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin Hi Frank, On 7/30/23 16:52, Frank Schilder wrote: Hi Xiubo, it happened again. This time, we might be able to pull logs from the client node. Please take a look at my intermediate action

[ceph-users] Re: MDS stuck in rejoin

2023-07-31 Thread Frank Schilder
regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Monday, July 31, 2023 4:12 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin Hi Frank, On 7/30/23 16:52, Frank

[ceph-users] Re: MDS stuck in rejoin

2023-07-30 Thread Xiubo Li
. Thanks - Xiubo Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Friday, July 28, 2023 11:37 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck in rejoin On 7/26

[ceph-users] Re: MDS stuck in rejoin

2023-07-30 Thread Frank Schilder
it in this state until I return from holidays. Best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Friday, July 28, 2023 11:37 AM To: Frank Schilder; ceph-users@ceph.io Subject: Re: [ceph-users] Re: MDS stuck

[ceph-users] Re: MDS stuck in rejoin

2023-07-28 Thread Xiubo Li
On 7/26/23 22:13, Frank Schilder wrote: Hi Xiubo. ... I am more interested in the kclient side logs. Just want to know why that oldest request got stuck so long. I'm afraid I'm a bad admin in this case. I don't have logs from the host any more, I would have needed the output of dmesg and

[ceph-users] Re: MDS stuck in rejoin

2023-07-26 Thread Frank Schilder
Hi Xiubo. > ... I am more interested in the kclient side logs. Just want to > know why that oldest request got stuck so long. I'm afraid I'm a bad admin in this case. I don't have logs from the host any more, I would have needed the output of dmesg and this is gone. In case it happens again I

[ceph-users] Re: MDS stuck in rejoin

2023-07-25 Thread Xiubo Li
Xiubo Li Sent: Friday, July 21, 2023 1:32 PM To: ceph-users@ceph.io Subject: [ceph-users] Re: MDS stuck in rejoin On 7/20/23 22:09, Frank Schilder wrote: Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush

[ceph-users] Re: MDS stuck in rejoin

2023-07-24 Thread Frank Schilder
Subject: [ceph-users] Re: MDS stuck in rejoin On 7/20/23 22:09, Frank Schilder wrote: > Hi all, > > we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients > failing to advance oldest client/flush tid". I looked at the client and there > was nothin

[ceph-users] Re: MDS stuck in rejoin

2023-07-21 Thread Xiubo Li
On 7/20/23 22:09, Frank Schilder wrote: Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still

[ceph-users] Re: MDS stuck in rejoin

2023-07-20 Thread Xiubo Li
On 7/20/23 22:09, Frank Schilder wrote: Hi all, we had a client with the warning "[WRN] MDS_CLIENT_OLDEST_TID: 1 clients failing to advance oldest client/flush tid". I looked at the client and there was nothing going on, so I rebooted it. After the client was back, the message was still