Hi Andras, This sounds like the client served the lookups from its cached negative? Can you check if you see the lookup calls being made to MDS and maybe set debug_mds=20 to check if the client did send the request but it was MDS which returned ENOENT but since the `ls -l` succeeded I'm wondering if this is actually something from MDS. Maybe try a sync and drop_caches to validate that?
*Dhairya Parmar* Software Engineer, CephFS On Wed, Jul 22, 2026 at 3:46 AM Andras Pataki via ceph-users < [email protected]> wrote: > Hi ceph users/devs, > > We've been encountering a strange issue with snapshots as part of a > project to copy our large ceph file system to a backup in parallel > across nodes. We are taking a snapshot, and then doing a copy of the > snapshot, which we expect to be static (unchanging) - using several > clients in parallel - all kernel mounts. Some clients are unable to > open files and directories that are known to exist (were discovered > traversing the file system on perhaps another node). > After some playing around, I could reproduce the issue on a single node > with just 'ls' as follows: > > # This file doesn't seems to exist - and some number of parent directories > are also "missing": [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen': > > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > ls: cannot access > '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM': > No such file or directory [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # Until we get to this 'Projects' parent > directory, which seems to be there [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45 > 1961 1961 46 Jun 8 14:02 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it > is still not there ... [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot > access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK': > No such file or directory # But ... let's see if the "tickle" the Projects > directory a bit ... > [root@cephmds603 ~]# ls -l > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null > > # Now all subdirectories and the file in question immediately becomes > visible: > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > drwxrwsr-x 5 1961 1961 3 May 7 17:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > drwxrwsr-x 12 1961 1961 10 May 11 16:08 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > drwxrwsr-x 5 1961 1961 3 May 11 16:18 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen > [root@cephmds603 ~]# ls -ld > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > -rw-rw-r-- 1 1961 1961 51538071 May 15 14:10 > /mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log > > # This is the kernel client 6.18.32 (relatively recent LTS kernel) with > ceph squid 19.2.3: > [root@cephmds603 ~]# uname -a > Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 > x86_64 x86_64 GNU/Linux > [root@cephmds603 ~]# ceph --version > ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid > (stable) > [root@cephmds603 ~]# > > What is most interesting is that in the live tree (non-snapshotted tree) - > the Projects directory exists, but everything under it has been removed! > > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/ > drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/ > [root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK > ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or > directory > > Am I running into some known issue? This looks to be specific to some > clients - i.e. some clients see the directory and some don't at the same > point in time, i.e. this is perhaps a kclient problem? > Any ideas/suggestions would be appreciated, > > Andras > _______________________________________________ > ceph-users mailing list -- [email protected] > To unsubscribe send an email to [email protected] > > _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
