Thank you Eugen. Truly appreciate your quick response as always. Regards, Anantha
-----Original Message----- From: Eugen Block via ceph-users <[email protected]> Sent: Friday, May 8, 2026 6:15 AM To: [email protected] Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists already an active peer" Error (EEXIST) Despite Empty Hi, I'm afraid I can't really help any further. I don't have any CephFS snapshot mirroring enabled in any of my test clusters at the moment. The only thing I could quickly check is that on a Squid cluster, I was able to create a snapshot via dashboard of a directory with these permissions: /mnt# ll volumes/_nogroup/ total 0 drwxr-sr-x 3 root root 1 May 8 12:43 ./ drwxr-xr-x 3 root root 2 May 8 12:43 ../ d-----S--- 3 root root 2 May 8 12:43 test-sub-restrictive/ /mnt# ceph fs subvolume snapshot ls cephfs test-sub-restrictive [ { "name": "2026-05-08T14:45:13.823+02:00" } This works a bit differently in Pacific, I created a subvolume in the same way, changed directory permissions like above, but the Dashboard can't show me the path with this error: cephfs.OSError: opendir failed: Permission denied [Errno 13] I can create a snapshot via command line though, so I'm not sure how this error would come into play here. I would expect/hope that upgrading your primary site could resolve this, but of course that's just a guess. Regards, Eugen Zitat von "Adiga, Anantha via ceph-users" <[email protected]>: > Hi Eugen, > > Regarding this issue, we have 18TB of data successfully mirrored, but > the process is completely blocked by these permission restrictions. > CephFS mirroring daemon cannot access directories with restrictive > permissions, and I'm seeking guidance on the proper solution. > > We also find that this issue is not seen on Ceph Reef ( both backup > source and target ) > > Environment: > • Primary Cluster: Ceph Pacific 16.2.5 > • Backup Cluster: Ceph Reef 18.2.7 > • Data Size: ~18TB successfully mirrored before hitting permission issues > > Problem Description: > CephFS mirroring fails when encountering directories with highly > restrictive permissions (d-----S---). The mirror daemon cannot access > these directories, causing snapshot synchronization to fail entirely. > > Error logs: > cephfs::mirror::PeerReplayer() should_sync_entry: failed to stat prev > entry= <path/to/file>: (13) Permission denied > cephfs::mirror::PeerReplayer() do_sync_snaps: failed to synchronize > dir_root=/volumes/capi_subvolumegroup/capi_subvolume, > snapshot=scheduled-2026-04-29-22_00_00 > > Root Cause Analysis: > The cephfs-mirror daemon runs with --setuser ceph --setgroup ceph > arguments, causing it to drop privileges to the ceph user. Even with > full Ceph capabilities, the MDS enforces POSIX permissions, preventing > access to directories with no read/execute permissions. > > Directory permissions: > • Many directories with d-----S--- permissions (no access for anyone) > • These directories contain critical data that must be backed up > • Cannot modify source file permissions (production system) > > Attempted solutions: > > [client.cephfs-mirror.a001s017.aoolcx] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > > root@a001s016:~# ceph auth get client.mirror_gfs_src exported keyring > for client.mirror_gfs_src [client.mirror_gfs_src] > key = <key> > caps mds = "allow *" > caps mgr = "allow *" > caps mon = "allow *" > caps osd = "allow *" > root@a001s016:~# > > ceph config set client.cephfs-mirror client_mount_uid 0 ceph config > set client.cephfs-mirror client_mount_gid 0 > > ceph config set client.cephfs-mirror setuser 0 ceph config set > client.cephfs-mirror setgroup 0 > > Attempted to use extra_entrypoint_args in service spec, but Pacific returns: > Error EINVAL: ServiceSpec: __init__() got an unexpected keyword > argument 'extra_entrypoint_args' > > Is there a supported way to run cephfs-mirror with root privileges to > bypass POSIX permission checks? > Are there service spec parameters in Pacific that allow overriding the > --setuser/--setgroup arguments? > Is this a known limitation of CephFS mirroring with restrictive file > permissions? > What is the recommended approach for mirroring filesystems containing > directories that require root access? > > Request guidance on: > • Proper configuration for root-level CephFS mirroring access > • Whether this is a design limitation or configuration issue > • Recommended best practices for mirroring restricted filesystems > > Regards, > Anantha > > -----Original Message----- > From: Eugen Block <[email protected]> > Sent: Thursday, April 23, 2026 1:11 PM > To: Adiga, Anantha <[email protected]> > Cc: [email protected] > Subject: Re: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists > already an active peer" Error (EEXIST) Despite Empty > > Awesome, glad I could help! > > Zitat von "Adiga, Anantha" <[email protected]>: > >> Hi Eugen, >> >> Thank you so much. Yes, that fixed the issue: >> removed the peer uuid from the primary cluster. This removed the >> attribute from the backup cluster as well. Then import peer bootstrap >> token worked. >> >> Regards, >> Anantha >> >> -----Original Message----- >> From: Eugen Block via ceph-users <[email protected]> >> Sent: Thursday, April 23, 2026 12:56 AM >> To: [email protected] >> Subject: [ceph-users] Re: CephFS Snapshot Mirroring - "File exists >> already an active peer" Error (EEXIST) Despite Empty >> >> Hi, >> >> a quick look into the code [0] shows (note that I'm not a developer): >> >> def set_mirror_info(local_cluster_id, local_fsid, remote_fs): >> log.info(f'setting {local_cluster_id}::{local_fsid} on remote') >> try: >> remote_fs.setxattr('/', 'ceph.mirror.info', >> f'cluster_id={local_cluster_id} >> fs_id={local_fsid}'.encode('utf-8'), os.XATTR_CREATE) >> except cephfs.Error as e: >> if e.errno == errno.EEXIST: >> >> >> My interpretation is that the setxattr command fails on the remote >> site. Could this be an auth caps issue? Did you verify the keyrings >> and their permissions? >> >> This is the only lead I got. >> >> Regards, >> Eugen >> >> [0] >> https://github.com/ceph/ceph/blob/267bd6273e173c3e9e9ed8693aa73d91a20 >> c >> bd6a/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L411 >> >> Zitat von "Adiga, Anantha via ceph-users" <[email protected]>: >> >>> Hi, >>> >>> This is issue with CephFS snapshot mirroring setup between two >>> clusters and would appreciate your assistance. >>> >>> Environment: >>> >>> * Primary Cluster: Ceph 15.2.13/16.2.5 (mixed versions) >>> [cid:[email protected]] >>> * Backup Cluster: Ceph 17.2.7/18.2.7 (mixed versions) >>> [cid:[email protected]] >>> * Filesystem: cephfs >>> * Issue: Unable to import peer bootstrap token >>> >>> Problem Description: >>> When attempting to import a peer bootstrap token using: >>> ceph fs snapshot mirror peer_bootstrap import cephfs <token> I >>> receive the error: >>> Error EEXIST: already an active peer mgr.server reply reply (17) >>> File exists already an active peer However, ceph fs snapshot mirror >>> peer_list cephfs returns an empty list [], indicating no peers are >>> configured. >>> >>> Steps Performed: >>> Initial Setup: >>> 1. Enabled mirroring module: ceph mgr module enable mirroring >>> 2. Deployed cephfs-mirror service: ceph orch apply cephfs-mirror >>> 3. Enabled mirroring on filesystem: ceph fs snapshot mirror >>> enable cephfs >>> 4. Created remote user and bootstrap token on backup cluster >>> 5. Verified mirroring daemons are running: ceph fs snapshot >>> mirror daemon status >>> Troubleshooting Attempts: >>> 1. Reset filesystem mirroring: >>> ceph fs snapshot mirror disable cephfs ceph fs snapshot mirror >>> enable cephfs >>> 2. Module reset: >>> ceph mgr module disable mirroring >>> ceph mgr module enable mirroring >>> 3. Service recreation: >>> ceph orch rm cephfs-mirror >>> ceph orch apply cephfs-mirror >>> 4. Checked for stale configuration: >>> o ceph config-key ls | grep mirror shows: mgr/cephadm/spec.cephfs-mirror >>> o ceph fs snapshot mirror dirmap cephfs returns empty >>> o ceph fs snapshot mirror status cephfs shows no active mirroring >>> >>> Current Status: >>> >>> * Mirroring daemons: Running and visible in daemon status >>> * Peer list: Empty [] >>> * Filesystem mirroring: Enabled >>> * Error persists: EEXIST despite no visible peers >>> >>> Diagnostic Information: >>> >>> * MGR logs show: mgr.server reply reply (17) File exists already >>> an active peer >>> * No connectivity issues between clusters (ports 6789, 3300, >>> 6800+ tested) >>> * Both clusters are healthy with no other issues >>> >>> Regards, >>> Anantha >> >> >> _______________________________________________ >> ceph-users mailing list -- [email protected] To unsubscribe send an >> email to [email protected] > > > > _______________________________________________ > ceph-users mailing list -- [email protected] To unsubscribe send an > email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected] _______________________________________________ ceph-users mailing list -- [email protected] To unsubscribe send an email to [email protected]
