Hi ceph users/devs,
We've been encountering a strange issue with snapshots as part of a
project to copy our large ceph file system to a backup in parallel
across nodes. We are taking a snapshot, and then doing a copy of the
snapshot, which we expect to be static (unchanging) - using several
clients in parallel - all kernel mounts. Some clients are unable to
open files and directories that are known to exist (were discovered
traversing the file system on perhaps another node).
After some playing around, I could reproduce the issue on a single node
with just 'ls' as follows:
# This file doesn't seems to exist - and some number of parent directories are also "missing": [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
ls: cannot access
'/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM':
No such file or directory [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot
access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':
No such file or directory # Until we get to this 'Projects' parent
directory, which seems to be there [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects drwxr-sr-x 45
1961 1961 46 Jun 8 14:02
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects # GSK under it
is still not there ... [root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK ls: cannot
access '/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK':
No such file or directory # But ... let's see if the "tickle" the Projects directory a bit ...
[root@cephmds603 ~]# ls -l
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects > /dev/null
# Now all subdirectories and the file in question immediately becomes visible:
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK
drwxrwsr-x 5 1961 1961 3 May 7 17:18
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
drwxrwsr-x 12 1961 1961 10 May 11 16:08
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
drwxrwsr-x 5 1961 1961 3 May 11 16:18
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen
[root@cephmds603 ~]# ls -ld
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
-rw-rw-r-- 1 1961 1961 51538071 May 15 14:10
/mnt/ceph/users/.snap/sync-20260701-0904/bturzo/Projects/GSK/RosettaQM/af3_ensemble_gen/output/ddg_analysis/mutation_total_scores/C_32_S_to_W/gamess_io/gamess_qm_energy_thread0_2026_05_15_13_18_13.log
# This is the kernel client 6.18.32 (relatively recent LTS kernel) with ceph
squid 19.2.3:
[root@cephmds603 ~]# uname -a
Linux cephmds603 6.18.32.1.fi #4 SMP Mon May 18 11:45:58 EDT 2026 x86_64 x86_64
x86_64 GNU/Linux
[root@cephmds603 ~]# ceph --version
ceph version 19.2.3 (c92aebb279828e9c3c1f5d24613efca272649e62) squid (stable)
[root@cephmds603 ~]#
What is most interesting is that in the live tree (non-snapshotted tree) - the
Projects directory exists, but everything under it has been removed!
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/
drwxr-sr-x 43 1961 1961 45 Jul 17 11:56 /mnt/ceph/users/bturzo/Projects/
[root@cephmds600 ~]# ls -ld /mnt/ceph/users/bturzo/Projects/GSK
ls: cannot access '/mnt/ceph/users/bturzo/Projects/GSK': No such file or
directory
Am I running into some known issue? This looks to be specific to some clients
- i.e. some clients see the directory and some don't at the same point in time,
i.e. this is perhaps a kclient problem?
Any ideas/suggestions would be appreciated,
Andras
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
