[ceph-users] Re: ceph fs (meta) data inconsistent

2023-12-11 Thread Xiubo Li
-users] Re: ceph fs (meta) data inconsistent Frank, By using your script I still couldn't reproduce it. Locally my python version is 3.9.16, and I didn't have other VMs to test python other versions. Could you check the tracker to provide the debug logs ? Thanks - Xiubo On 12/1/23 21:08, Frank

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-12-08 Thread Frank Schilder
Campus Bygning 109, rum S14 From: Xiubo Li Sent: Tuesday, December 5, 2023 1:58 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent Frank, By using your script I still couldn't reproduce

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-12-04 Thread Xiubo Li
27, 2023 3:59 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent On 11/24/23 21:37, Frank Schilder wrote: Hi Xiubo, thanks for the update. I will test your scripts in our system next week. Something important: running both

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-12-03 Thread Xiubo Li
: [ceph-users] Re: ceph fs (meta) data inconsistent On 11/24/23 21:37, Frank Schilder wrote: Hi Xiubo, thanks for the update. I will test your scripts in our system next week. Something important: running both scripts on a single client will not produce a difference. You need 2 clients

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-12-01 Thread Frank Schilder
Sent: Monday, November 27, 2023 3:59 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent On 11/24/23 21:37, Frank Schilder wrote: > Hi Xiubo, > > thanks for the update. I will test your scripts in our system

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-26 Thread Xiubo Li
From: Xiubo Li Sent: Thursday, November 23, 2023 3:47 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent I just raised one tracker to follow this: https://tracker.ceph.com/issues/63510 Thanks - Xiubo

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-24 Thread Frank Schilder
Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent I just raised one tracker to follow this: https://tracker.ceph.com/issues/63510 Thanks - Xiubo ___ ceph-users mailing list -- ceph-users@ceph.io To unsubscribe send

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-22 Thread Xiubo Li
] Re: ceph fs (meta) data inconsistent On 11/10/23 00:18, Frank Schilder wrote: Hi Xiubo, I will try to answer questions from all your 3 e-mails here together with some new information we have. New: The problem occurs in newer python versions when using the shutil.copy function. There is also

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-22 Thread Xiubo Li
. = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Xiubo Li Sent: Friday, November 10, 2023 3:14 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent On 11/10/23 00

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-10 Thread Frank Schilder
ygning 109, rum S14 From: Xiubo Li Sent: Friday, November 10, 2023 3:14 AM To: Frank Schilder; Gregory Farnum Cc: ceph-users@ceph.io Subject: Re: [ceph-users] Re: ceph fs (meta) data inconsistent On 11/10/23 00:18, Frank Schilder wrote: > Hi Xiubo, >

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-09 Thread Xiubo Li
On 11/10/23 00:18, Frank Schilder wrote: Hi Xiubo, I will try to answer questions from all your 3 e-mails here together with some new information we have. New: The problem occurs in newer python versions when using the shutil.copy function. There is also a function shutil.copy2 for which

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-09 Thread Frank Schilder
Hi Xiubo, I will try to answer questions from all your 3 e-mails here together with some new information we have. New: The problem occurs in newer python versions when using the shutil.copy function. There is also a function shutil.copy2 for which the problem does not show up. Copy2 behaves a

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-06 Thread Xiubo Li
Thanks and best regards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 From: Frank Schilder Sent: Thursday, November 2, 2023 12:12 PM To: Gregory Farnum; Xiubo Li Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph fs (meta) data inconsisten

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-06 Thread Xiubo Li
On 11/1/23 23:57, Gregory Farnum wrote: We have seen issues like this a few times and they have all been kernel client bugs with CephFS’ internal “capability” file locking protocol. I’m not aware of any extant bugs like this in our code base, but kernel patches can take a long and winding

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-06 Thread Xiubo Li
On 11/1/23 22:14, Frank Schilder wrote: Dear fellow cephers, today we observed a somewhat worrisome inconsistency on our ceph fs. A file created on one host showed up as 0 length on all other hosts: [user1@host1 h2lib]$ ls -lh total 37M -rw-rw 1 user1 user1 12K Nov 1 11:59

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-03 Thread Frank Schilder
egards, = Frank Schilder AIT Risø Campus Bygning 109, rum S14 ________ From: Frank Schilder Sent: Thursday, November 2, 2023 12:12 PM To: Gregory Farnum; Xiubo Li Cc: ceph-users@ceph.io Subject: [ceph-users] Re: ceph fs (meta) data inconsistent Hi al

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-02 Thread Frank Schilder
Hi all, the problem re-appeared in the following way. After moving the problematic folder out and copying it back, all files showed the correct sizes. Today, we observe that the issue is back in the copy that was fine yesterday: [user1@host11 h2lib]$ ls -l total 37198 -rw-rw 1 user1 user1

[ceph-users] Re: ceph fs (meta) data inconsistent

2023-11-01 Thread Gregory Farnum
We have seen issues like this a few times and they have all been kernel client bugs with CephFS’ internal “capability” file locking protocol. I’m not aware of any extant bugs like this in our code base, but kernel patches can take a long and winding path before they end up on deployed systems.