Hi Andras, Dhairya, I recall hitting a similar issue with this and the workaround involves running the 'ls' command, but I can't recall exactly which command or the tracker reference. This happened years ago.
This morning I tried to reproduce this locally but failed. And I just went through the mds and kclient code just found two suspect places could cause this: MDS side: in https://github.com/ceph/ceph/blob/main/src/mds/MDCache.cc#L8693-L8696 MDCache::path_traverse() uses two shortcuts to conclude a dentry doesn't exist: if (curdir->is_complete() || // (1) (snapid == CEPH_NOSNAP && ...bloom...)) { // (2) return -ENOENT; is_complete() means all live entries are cached in memory. Shortcut (2) correctly only applies to live lookups (snapid == CEPH_NOSNAP), but shortcut (1) applies to all snapids — including snapshots. When a snapshot exists but the file was later deleted from the live tree, the MDS eventually evicts the deleted file's dentry from the in-memory cache (a null dentry, which trim_dentry() removes without clearing the COMPLETE flag). A subsequent snapshot LOOKUP hits is_complete()=true with the dentry absent (ENOENT), without ever checking the OSD where the snapshot data actually lives. READDIR takes a different code path that fetches from the OSD, which is why ls -l fixes it. Fix: move snapid == CEPH_NOSNAP to guard both shortcuts: if (snapid == CEPH_NOSNAP && (curdir->is_complete() || ...bloom...)) { @Andras BTW, to verify the above could you try to reproduce this by using the ceph-fuse mount ? Locally I couldn't reproduce it by trying many times and I am planing to add some debug code in MDS to ammulated it to reproduce it later. ==== Kclient side: https://github.com/ceph/ceph-client/blob/for-linus/fs/ceph/dir.c#L1984-L1987 Plus a kclient defense: ceph_d_revalidate() unconditionally trusts negative dentries in snapshot directories, caching a transient ENOENT forever. Change valid = 1 to valid = inode ? 1 : 0. I just created on trakcer to follow this: https://tracker.ceph.com/issues/78529 Thanks Xiubo On Wed, 22 Jul 2026 at 13:49, Dhairya Parmar via ceph-users <[email protected]> wrote: > > Hi Andras, > > This sounds like the client served the lookups from its cached negative? > Can you check if you see the lookup calls being made to MDS and maybe set > debug_mds=20 to check if the client did send the request but it was MDS > which returned ENOENT but since the `ls -l` succeeded I'm wondering if this > is actually something from MDS. Maybe try a sync and drop_caches to > validate that? > > *Dhairya Parmar* > > Software Engineer, CephFS > > > On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < > [email protected]> wrote: > > > Hi ceph users/devs, > > > > We've been encountering a strange issue with snapshots as part of a > > project to copy our large ceph file system to a backup in parallel > > across nodes. We are taking a snapshot, and then doing a copy of the > > snapshot, which we expect to be static (unchanging) - using several > > clients in parallel - all kernel mounts. Some clients are unable to > > open files and directories that are known to exist (were discovered > > traversing the file system on perhaps another node). > > After some playing around, I could reproduce the issue on a single node > > with just 'ls' as follows: > > > > # This file doesn't seems to exist - and some number of parent directories > > are also "missing": [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > ls: cannot access > > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > > No such file or directory [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # Until we get to this 'Projects' parent > > directory, which seems to be there [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > > 1961 1961 46 Jun 8 14:02 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > > is still not there ... [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > > No such file or directory # But ... let's see if the "tickle" the Projects > > directory a bit ... > > [root@cephmds603 ~]# ls -l > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > > > # Now all subdirectories and the file in question immediately becomes > > visible: > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > [root@cephmds603 ~]# ls -ld > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > > ceph squid 19.2.3: > > [root@cephmds603 ~]# uname -a > > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > > x86_64 x86_64 GNU/Linux > > [root@cephmds603 ~]# ceph --version > > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > > (stable) > > [root@cephmds603 ~]# > > > > What is most interesting is that in the live tree (non-snapshotted tree) - > > the Projects directory exists, but everything under it has been removed! > > > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > > directory > > > > Am I running into some known issue? This looks to be specific to some > > clients - i.e. some clients see the directory and some don't at the same > > point in time, i.e. this is perhaps a kclient problem? > > Any ideas/suggestions would be appreciated, > > > > Andras > > _______________________________________________ > > ceph-users mailing list -- [email protected] > > To unsubscribe send an email to [email protected] > > > > > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
