Hi,
I'm afraid I can't really help any further. I don't have any CephFS
snapshot mirroring enabled in any of my test clusters at the moment.
The only thing I could quickly check is that on a Squid cluster, I was
able to create a snapshot via dashboard of a directory with these
permissions:
/mnt# ll volumes/_nogroup/
total 0
drwxr-sr-x 3 root root 1 May 8 12:43 ./
drwxr-xr-x 3 root root 2 May 8 12:43 ../
d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/
/mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive
[
{
"name": "2026-05-08T14:45:13.823+02:00"
}
This works a bit differently in Pacific, I created a subvolume in the
same way, changed directory permissions like above, but the Dashboard
can't show me the path with this error:
cephfs.OSError: opendir failed: Permission denied [Errno 13]
I can create a snapshot via command line though, so I'm not sure how
this error would come into play here. I would expect/hope that
upgrading your primary site could resolve this, but of course that's
just a guess.
Regards,
Eugen
Zitat von "Adiga, Anantha via ceph-users" <[email protected]>:
Hi Eugen,
Regarding this issue, we have 18TB of data successfully mirrored,
but the process is completely blocked by these permission
restrictions. CephFS mirroring daemon cannot access directories with
restrictive permissions, and I'm seeking guidance on the proper
solution.
We also find that this issue is not seen on Ceph Reef ( both backup
source and target )
Environment:
• Primary Cluster: Ceph Pacific 16.2.5
• Backup Cluster: Ceph Reef 18.2.7
• Data Size: ~18TB successfully mirrored before hitting permission issues
Problem Description:
CephFS mirroring fails when encountering directories with highly
restrictive permissions (d-----S---). The mirror daemon cannot
access these directories, causing snapshot synchronization to fail
entirely.
Error logs:
cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat
prev entry= <path/to/file>: (13) Permission denied
cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize
dir_root=/volumes/capi_subvolumegroup/capi_subvolume,
snapshot=scheduled-2026-04-29-22_00_00
Root Cause Analysis:
The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph
arguments, causing it to drop privileges to the ceph user. Even with
full Ceph capabilities, the MDS enforces POSIX permissions,
preventing access to directories with no read/execute permissions.
Directory permissions:
• Many directories with d-----S--- permissions (no access for anyone)
• These directories contain critical data that must be backed up
• Cannot modify source file permissions (production system)
Attempted solutions:
[client.cephfs-mirror.a001s017.aoolcx]
key = <key>
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
root@a001s016:~# ceph auth get client.mirror_gfs_src
exported keyring for client.mirror_gfs_src
[client.mirror_gfs_src]
key = <key>
caps mds = "allow *"
caps mgr = "allow *"
caps mon = "allow *"
caps osd = "allow *"
root@a001s016:~#
ceph config set client.cephfs-mirror client_mount_uid 0
ceph config set client.cephfs-mirror client_mount_gid 0
ceph config set client.cephfs-mirror setuser 0
ceph config set client.cephfs-mirror setgroup 0
Attempted to use extra_entrypoint_args in service spec, but Pacific returns:
Error EINVAL: ServiceSpec: __init__() got an unexpected keyword
argument 'extra_entrypoint_args'
Is there a supported way to run cephfs-mirror with root privileges
to bypass POSIX permission checks?
Are there service spec parameters in Pacific that allow overriding
the --setuser/--setgroup arguments?
Is this a known limitation of CephFS mirroring with restrictive file
permissions?
What is the recommended approach for mirroring filesystems
containing directories that require root access?
Request guidance on:
• Proper configuration for root-level CephFS mirroring access
• Whether this is a design limitation or configuration issue
• Recommended best practices for mirroring restricted filesystems
Regards,
Anantha
-----Original Message-----
From: Eugen Block <[email protected]>
Sent: Thursday, April 23, 2026 1:11 PM
To: Adiga, Anantha <[email protected]>
Cc: [email protected]
Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File
exists already an active peer" Error (EEXIST) Despite Empty
Awesome, glad I could help!
Zitat von "Adiga, Anantha" <[email protected]>:
Hi Eugen,
Thank you so much. Yes, that fixed the issue:
removed the peer uuid from the primary cluster. This removed the
attribute from the backup cluster as well. Then import peer bootstrap
token worked.
Regards,
Anantha
-----Original Message-----
From: Eugen Block via ceph-users <[email protected]>
Sent: Thursday, April 23, 2026 12:56 AM
To: [email protected]
Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists
already an active peer" Error (EEXIST) Despite Empty
Hi,
a quick look into the code [0] shows (note that I'm not a developer):
def set_mirror_info(local_cluster_id, local_fsid, remote_fs):
log.info(f'setting {local_cluster_id}::{local_fsid} on remote')
try:
remote_fs.setxattr('/', 'ceph.mirror.info',
f'cluster_id={local_cluster_id}
fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE)
except cephfs.Error as e:
if e.errno == errno.EEXIST:
My interpretation is that the setxattr command fails on the remote
site. Could this be an auth caps issue? Did you verify the keyrings
and their permissions?
This is the only lead I got.
Regards,
Eugen
[0]
https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20c
bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411
Zitat von "Adiga, Anantha via ceph-users" <[email protected]>:
Hi,
This is issue with CephFS snapshot mirroring setup between two
clusters and would appreciate your assistance.
Environment:
* Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions)
[cid:[email protected]]
* Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions)
[cid:[email protected]]
* Filesystem: cephfs
* Issue: Unable to import peer bootstrap token
Problem Description:
When attempting to import a peer bootstrap token using:
ceph fs snapshot mirror peer_bootstrap import cephfs <token> I
receive the error:
Error EEXIST: already an active peer mgr.server reply reply (17) File
exists already an active peer However, ceph fs snapshot mirror
peer_list cephfs returns an empty list [], indicating no peers are
configured.
Steps Performed:
Initial Setup:
1. Enabled mirroring module: ceph mgr module enable mirroring
2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror
3. Enabled mirroring on filesystem: ceph fs snapshot mirror
enable cephfs
4. Created remote user and bootstrap token on backup cluster
5. Verified mirroring daemons are running: ceph fs snapshot
mirror daemon status
Troubleshooting Attempts:
1. Reset filesystem mirroring:
ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror enable
cephfs
2. Module reset:
ceph mgr module disable mirroring
ceph mgr module enable mirroring
3. Service recreation:
ceph orch rm cephfs-mirror
ceph orch apply cephfs-mirror
4. Checked for stale configuration:
o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror
o ceph fs snapshot mirror dirmap cephfs returns empty
o ceph fs snapshot mirror status cephfs shows no active mirroring
Current Status:
* Mirroring daemons: Running and visible in daemon status
* Peer list: Empty []
* Filesystem mirroring: Enabled
* Error persists: EEXIST despite no visible peers
Diagnostic Information:
* MGR logs show: mgr.server reply reply (17) File exists already
an active peer
* No connectivity issues between clusters (ports 6789, 3300,
6800+ tested)
* Both clusters are healthy with no other issues
Regards,
Anantha
_______________________________________________
ceph-users mailing list -- [email protected] To unsubscribe send an
email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
