Hi Xiubo and Dhariya,
The way I was able to reproduce the problem was to look at the error log
of my copy processes and take those files and check on them on the
various clients. Today, unfortunately, all those files seem to be
visible - and at the moment I don't have a way of reproducing the issue
from scratch. I'm trying to see what could be done to set up the
reproduction from scratch. If I manage to do that, I'll try to gather
some information on whether this is a client issue (i.e. flushing the
client caches changes anything) or whether it comes from the MDS.
Unfortunately the MDS in question is a very busy one, our main file
system MDS, so there are limits to what I can do to it.
Andras
On 7/22/26 5:37 AM, Xiubo Li via ceph-users wrote:
Hi Dhairya,
Yeah, makes sense.
Since I couldn't reproduce this locally, @Andras Pataki Could you help
reproduce it by enabling the debug logs as Dhairya mentioned? And if
possible please also try with the fuse client to see could you see
too.
Thanks
Xiubo
On Wed, 22 Jul 2026 at 17:11, Dhairya Parmar via dev <[email protected]> wrote:
Hi Xiubo,
I did look at the codepaths you shared when I responded but the thing that made
me put my money on kclient is in `Server::handle_client_readdir`, there is this
block
https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/squid/src/mds/Server.cc*L5029-L5041__;Iw!!DSb-azq1wVFtOg!Ww2_939ITEQ4zTmiwm-Dqm0Z0_U-4DS8xqNlCAhqm7seVWvRYliPPaiyH1Gx6VlvYqmpWSgDn6DwznFAoV7jNcLUPbA$
. Meaning say if the trim_dentry() didn't clear the `CDir::STATE_COMPLETE`
state then the readdir should've also served from memory and omit GSK, no? IMO
if this is easily reproducible, we should get the MDS logs to verify the exact
sequence. WDYT?
On Wed, Jul 22, 2026 at 12:38 PM Xiubo Li <[email protected]> wrote:
Hi Andras, Dhairya,
I recall hitting a similar issue with this and the workaround involves
running the 'ls' command, but I can't recall exactly which command or
the tracker reference. This happened years ago.
This morning I tried to reproduce this locally but failed. And I just
went through the mds and kclient code just found two suspect places
could cause this:
MDS side: in
https://urldefense.com/v3/__https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc*L8693-L8696__;Iw!!DSb-azq1wVFtOg!Ww2_939ITEQ4zTmiwm-Dqm0Z0_U-4DS8xqNlCAhqm7seVWvRYliPPaiyH1Gx6VlvYqmpWSgDn6DwznFAoV7jMegjVuQ$
MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist:
if (curdir->is_complete() || // (1)
(snapid == CEPH_NOSNAP && ...bloom...)) { // (2)
return -ENOENT;
is_complete() means all live entries are cached in memory. Shortcut
(2) correctly only applies to live lookups (snapid == CEPH_NOSNAP),
but shortcut (1) applies to all snapids — including snapshots.
When a snapshot exists but the file was later deleted from the live
tree, the MDS eventually evicts the deleted file's dentry from the
in-memory cache (a null dentry, which trim_dentry() removes without
clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits
is_complete()=true with the dentry absent (ENOENT), without ever
checking the OSD where the snapshot data actually lives.
READDIR takes a different code path that fetches from the OSD, which
is why ls -l fixes it.
Fix: move snapid == CEPH_NOSNAP to guard both shortcuts:
if (snapid == CEPH_NOSNAP &&
(curdir->is_complete() || ...bloom...)) {
@Andras
BTW, to verify the above could you try to reproduce this by using the
ceph-fuse mount ? Locally I couldn't reproduce it by trying many times
and I am planing to add some debug code in MDS to ammulated it to
reproduce it later.
====
Kclient side:
https://urldefense.com/v3/__https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c*L1984-L1987__;Iw!!DSb-azq1wVFtOg!Ww2_939ITEQ4zTmiwm-Dqm0Z0_U-4DS8xqNlCAhqm7seVWvRYliPPaiyH1Gx6VlvYqmpWSgDn6DwznFAoV7jVPfl0do$
Plus a kclient defense: ceph_d_revalidate() unconditionally trusts
negative dentries in snapshot directories, caching a transient ENOENT
forever. Change valid = 1 to valid = inode ? 1 : 0.
I just created on trakcer to follow this:
https://urldefense.com/v3/__https://tracker.ceph.com/issues/78529__;!!DSb-azq1wVFtOg!Ww2_939ITEQ4zTmiwm-Dqm0Z0_U-4DS8xqNlCAhqm7seVWvRYliPPaiyH1Gx6VlvYqmpWSgDn6DwznFAoV7jqXjfp5c$
Thanks
Xiubo
On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users
<[email protected]> wrote:
Hi Andras,
This sounds like the client served the lookups from its cached negative?
Can you check if you see the lookup calls being made to MDS and maybe set
debug_mds=20 to check if the client did send the request but it was MDS
which returned ENOENT but since the `ls -l` succeeded I'm wondering if this
is actually something from MDS. Maybe try a sync and drop_caches to
validate that?
*Dhairya Parmar*
Software Engineer, CephFS
On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users <
[email protected]> wrote:
Hi ceph users/devs,
We've been encountering a strange issue with snapshots as part of a
project to copy our large ceph file system to a backup in parallel
across nodes. We are taking a snapshot, and then doing a copy of the
snapshot, which we expect to be static (unchanging) - using several
clients in parallel - all kernel mounts. Some clients are unable to
open files and directories that are known to exist (were discovered
traversing the file system on perhaps another node).
After some playing around, I could reproduce the issue on a single node
with just 'ls' as follows:
# This file doesn't seems to exist - and some number of parent directories
are also "missing": [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot
access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':
No such file or directory # Until we get to this 'Projects' parent
directory, which seems to be there [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45
1961 1961 46 Jun 8 14:02
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it
is still not there ... [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot
access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':
No such file or directory # But ... let's see if the "tickle" the Projects
directory a bit ...
[root@cephmds603 ~]# ls -l
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null
# Now all subdirectories and the file in question immediately becomes
visible:
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK
drwxrwsr-x 5 1961 1961 3 May 7 17:18
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
drwxrwsr-x 12 1961 1961 10 May 11 16:08
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
drwxrwsr-x 5 1961 1961 3 May 11 16:18
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
-rw-rw-r-- 1 1961 1961 51538071 May 15 14:10
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
# This is the kernel client 6.18.32 (relatively recent LTS kernel) with
ceph squid 19.2.3:
[root@cephmds603 ~]# uname -a
Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64
x86_64 x86_64 GNU/Linux
[root@cephmds603 ~]# ceph --version
ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid
(stable)
[root@cephmds603 ~]#
What is most interesting is that in the live tree (non-snapshotted tree) -
the Projects directory exists, but everything under it has been removed!
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/
drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK
ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or
directory
Am I running into some known issue? This looks to be specific to some
clients - i.e. some clients see the directory and some don't at the same
point in time, i.e. this is perhaps a kclient problem?
Any ideas/suggestions would be appreciated,
Andras
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
