Hi Dominic,
I thought I explained it in the email quoted below where I mention that
yes they are unprivileged LXC containers. This means that when they are
viewed from the outside they have their UIDs and GIDs shifted by some
amount so root and other privileged users become ordinary.
Oh I suppose that some are wondering why not do the snapshots on the
machines you're backing up? Perhaps then the snapshots wouldn't be
shifted. Well I think it's probably unusual to have access to the
logical volume system that is used for the guest OSes, and it would make
more sense to access it from the host system. Let me know if there are
better ways to set that up. :-)
best,
Clif
On 2/9/22 12:46 PM, Mr. Clif wrote:
Howdy,
I've dug into this further and think I now know what's going on. Just
FYI this is more than an academic question for me because I have
several vms that I would like to take snapshots of and this first one
is by far the smallest.
These VMs are LXC containers, and when I started out a long time ago,
I would just manually create the filesystems and use some cli tools to
install a fresh distro. Eventually the linux kernal started supporting
namspaces to improve security and they were adopted by the
virtualization ecosystems.
I'm not sure when it happened because I just noticed it, maybe it was
when I switched to letting proxmox spin up the new VMs, but now the
UIDs and GIDs in the filesystems for the unprivileged containers have
all been shifted by adding 100000 to them. This is why rdiff-backup
updated all that metadata.
This is not just a mapping in ram, it's actually in the filesystem
image on disk. There are several ways of dealing with this, some tools
will update the UID/GIDs for you when you reboot the vm. Other tools
act like layer in a bind mount to mostly duplicate a filesystem
somewhere else, and they rewrite the UID/GIDs on the fly. Some
utilities like rdiff-backup and rsync have some ability to rewrite or
map the UID/GIDs as they copy. The last two seem most attractive to me.
rsync has --usermap, and --groupmap, and rdiff-backup has
--user-mapping-file, and --group-mapping-file. In the filesystem mount
utility area there are, shiftfs, idmapped mounts, and bindfs.
Shiftfs is deprecated in favor of idmapped mounts, though some of my
kernels don't have that yet. Bindfs is a FUSE based solution and so
might be slower, however it might be the only one that is really
workable for me at the moment. This is because it has the
--uid-offset, and --gid-offset options. Bye the way, you can put in
negative offsets too, good thing. :-)
It would be great if rdiff-backup would allow offsets like this or
even better the ability to specify a range like
100000-165535:0-65535
Or you could just have the starting UID after the colon.
In the man page under USERS AND GROUPS, it says:
"If you specify both --preserve-numerical-ids and one of the mapping
options, the behavior is undefined."
I think it would be better to allow both with the user-mapping-file
overriding the preserve-numerical-ids behavior when necessary. As in
my use case I never want user name mapping.
What do you think? I appreciator the discussion, and everyone's help.
Thanks,
Clif
On 2/14/22 10:55 PM, Dominic Raferd wrote:
On 15/02/2022 02:24, Mr. Clif wrote:
Hi Folks,
In case it's helpful, here's a shell function I came up with to
backup snapshots of VMs in a way that preserves the usual UIDs and
GIDs: ...
This is clever but I was puzzled why you experience this UID/GID shift
problem in the first place - I don't (I think). Containers maybe? See
https://forum.proxmox.com/threads/uid-mapping-container-backup.47287/
https://docs.docker.com/engine/security/userns-remap/
