Bug#995046: 3.2.3-7 fails with libc6-2.30
Hello Eugene,
> > > Yes, this issue may occure when libc6 vesion is outdated in normal
> > > upgrade process.
> >
> > I would disagree on the definition of a normal upgrade process, one
> > would never end up with a combination of incompatible rsync + libc,
> > that can only happen if somebody installs a package that is not
> > supported by Debian (not from the official repos) and that package
> > holds libc back (thus running an unsupported version)[4].
>
> I had several (at least 2 distinct) live systems with this post-upgrade
> problem. They have backups, but now it too late: now those backups are
> replaced by new ones.
>
> However, it requires some clarification what is "normal upgrade" process.
> I do not run full-upgrade ("aptitude safe-upgrade" in my taste) if some
> new package it to be installed. I simply check dependences to be sure
> that new package does not bring risk to break functionality of important
> services on production system. The full-upgrade is performed on a schedule,
> once a week or two, and requires additional monitoring of the system's
> behaviour. For example, I have an LXC containter with libc-2.31-17,
> which was not updated at least a month (file libc.so.6 dated 23-Aug).
You generally don't need to run full-upgrade on a stable Debian
release, that's true.
But if you are upgrading between major Debian releases, you are
required to perform a full-upgrade and install everything that it says
so there, if you fail to do it, you are running an unstable system
because you are mixing packages from different major releases.
This is where "normal upgrade" comes in, as soon as you upgrade to a
new Debian major release and don't perform full-upgrade, you're in
"unknown territory" (not officially supported/not normal).
We have lots of QA and CI tools that ensure things don't break in
multiple different scenarios, but this is one where we can't ensure
things will work (and don't test for it).
> Running "apt-get update ; aptitude -R install rsync" I got rsync-3.2.3-8
> without libc6 update. I consider this as a normal situation, because
> I have no obligation to do full-upgrade when I install some package.
This is true generally speaking, but if you break the upgrade process
when going to a new major release, then this is not the case anymore,
the system can now be considered unstable. Anything you install might
require full-upgrade because this should have been done at the time
when you switched major releases.
You are bound to keep hitting issues like this one because it's not a
supported system where we can predict what's gonna happen, our tooling
doesn't test for such scenarios.
The correct approach would be to stick to the previous release, and/or
use backports, or solve the issues blocking the system from performing
a full-upgrade (which are issues that only arises with third party
packages, since we do have automated tests that ensure systems with
official packages-only are able to fully upgrade).
> However, this combination (rsync-3.2.3-8 + libc6-2.31-17) is not
> functional: rsync replies "failed to set permissions [...] Function
> not implemented (38)".
I'd like to mention again, that even though I think you might be
risking too much with a system that bumped to a bigger major version
without a full-upgrade, and that this is not officially supported by
Debian, this is still a valid and real issue that you hit, and I will
solve it.
I also trust that you know what you're doing with those systems, and I
don't mean to say what you should or should not do. As long as you're
aware of the risks involved and understand that this is an unsupported
scenario with hidden issues, it should be fine.
> > I did try to reproduce the issue on Jessie, with libc 2.28, but I was
> > able to run "rsync -va /etc/ld.so.cache /tmp" with no issues.
>
> Did you try the last rsync binary with libc6-2.28?
> I tried and got an error (see below).
There was an issue on my reproduction steps, I'm now able to reproduce
the issue on Buster and Bullseye.
For the record and future references:
rsync 3.2.3-8 can be built on any Debian release as of now, the issue
only arises when a package built with libc6-dev >= 2.32 is installed
on a system that contains libc6 < 2.32.
Our dpkg-symbols solution is not able to identify this and naturally
choses a lower version requirement for the libc6 dependency.
Reproduction can be done on a Buster/Bullseye VM by installing an
rsync 3.2.3-8 that was compiled with libc6-dev >= 2.32 and running:
rsync -va /etc/ld.so.cache /tmp
I'm currently studying what's the best approach on solving this in a
clean manner, and I have updated upstream on this matter[0].
My previous plan still stands; I'd like to try to solve this outside
of the packaging of rsync, if possible, but if not I will bump rsync's
dependency to libc6 >=2.32 when it gets compiled with libc6-dev 2.32.
>> Just a small correction, I see you mentioned lchown, but the issue is
>> related to
Bug#995046: 3.2.3-7 fails with libc6-2.30
Hi Samuel.
On Tue, Sep 28, 2021 at 08:51:59PM +0100, Samuel Henrique wrote:
> > To build version dependency automatically script should have access
> > to the whole history of all versions of libc6, with list of symbols
> > and markers when each symbol become usable.
>
> We have that, you can read about it here[0][1][2], if you're interested.
Thank you for those references, I've got much helpful info from them.
> > Yes, this issue may occure when libc6 vesion is outdated in normal
> > upgrade process.
>
> I would disagree on the definition of a normal upgrade process, one
> would never end up with a combination of incompatible rsync + libc,
> that can only happen if somebody installs a package that is not
> supported by Debian (not from the official repos) and that package
> holds libc back (thus running an unsupported version)[4].
I had several (at least 2 distinct) live systems with this post-upgrade
problem. They have backups, but now it too late: now those backups are
replaced by new ones.
However, it requires some clarification what is "normal upgrade" process.
I do not run full-upgrade ("aptitude safe-upgrade" in my taste) if some
new package it to be installed. I simply check dependences to be sure
that new package does not bring risk to break functionality of important
services on production system. The full-upgrade is performed on a schedule,
once a week or two, and requires additional monitoring of the system's
behaviour. For example, I have an LXC containter with libc-2.31-17,
which was not updated at least a month (file libc.so.6 dated 23-Aug).
Running "apt-get update ; aptitude -R install rsync" I got rsync-3.2.3-8
without libc6 update. I consider this as a normal situation, because
I have no obligation to do full-upgrade when I install some package.
However, this combination (rsync-3.2.3-8 + libc6-2.31-17) is not
functional: rsync replies "failed to set permissions [...] Function
not implemented (38)".
> I did try to reproduce the issue on Jessie, with libc 2.28, but I was
> able to run "rsync -va /etc/ld.so.cache /tmp" with no issues.
Did you try the last rsync binary with libc6-2.28?
I tried and got an error (see below).
> It might be possible that the issue you're seeing only affects libc
> between 2.28 and 2.31, if that's the case, it will be very tricky for
> me to reproduce since Debian only supports 2.28, 2.31 and 2.32.
Many versions are saved here: http://snapshot.debian.org/binary/libc6/.
Using the previously mentioned LXC container and binary packages from
snapshot.debian.org, I tried rsync-3.2.3-8 with libc 2.31-17, 2.30-4,
2.29-10, 2.28-10, 2.28-1, 2.27-1, in all cases rsync fails with the same
"not implemented" error. I did not go below 2.27-1 because it requires
downgrading of base system packages...
> If you have an affected system at hand, I would kindly ask you to try
> to implement upstream's workaround[5] and see if it makes your issue
> go away.
I could take me too much time to set up building environment and compile
from sources... However, I can test binary on my setup, if you provide it
(for arch:amd64, please). I can provide access to my LXC environment:
just send me your ssh public key, and I'll create an entry for you.
> Just a small correction, I see you mentioned lchown, but the issue is
> related to lchmod.
Yes, it should be read "lchown" (lchmod was mentioned by mistake).
--
Eugene Berdnikov
Bug#995046: 3.2.3-7 fails with libc6-2.30
Hello Eugene, > On Sat, Sep 25, 2021 at 09:29:42PM +0100, Samuel Henrique wrote: > > > for lchown() is definitely a new feature, so the right way, IMHO, > > > is to correct dependences for rsync package: "libc6 (>= 2.31)". > > > > I don't think it's as simple as that, I understand your issue now, but > > I wouldn't just bump the dependency requirement without understanding > > it better. > > As it stands now, the version requirement is automatically set by > > debhelper, based on the symbols rsync uses. > > Are you sure about version? I doubt. The list of libraries is built > by debhelper, I agree. But the *version* dependency, IMHO, can't be built > on the list of referenced symbols. Yes, I'm not setting the required libc6 version anywhere, that's automatically done by dpkg. > To build version dependency automatically script should have access > to the whole history of all versions of libc6, with list of symbols > and markers when each symbol become usable. We have that, you can read about it here[0][1][2], if you're interested. > Note that presense of > symbol is not mean usability: for libc6-2.30 entry "lchown()" is > definitely present (ltrace confirms it), but attemptp to call it > return an error "not implemented", i.e. this version contains only > stub code (symbol is reserved for future use). Yes, that's why the version automatically picked doesn't solve this issue; the symbol was there before, it just changed behavior. The symbols are also provided by a human so there's always room for mistakes there, but that's not the case here. > I belive the version dependency have to be filled manually. Moreover, > with manual assignment buggy versions of library can be excluded, > which is very unlikely to be done automatically. We can also do it manually when needed, and that's my plan B[3]. > > The bug we are facing comes from libc6, as we can see in the upstream > > bug report, and rsync is trying to workaround that issue (the patch I > > introduced on 3.2.3-7). > > I did not look through the code, but probably it detects presence of > lchown() but don't cover the situation when this entry is not functional. Yes, I believe that could be what's happening with rsync's workaround. > > Having outdated packages is not a problem, but having only a subset of > > them is an indicator of something wrong. Rsync 3.2.3-7 migrated to > > testing after libc6 2.32, so this issue will not happen unless the > > person purposely holds libc back. I thought your system was running > > like that by accident. > > I'm not judging here, I just meant to explain that the trigger to the > > issue will not affect every Debian user. > > Yes, this issue may occure when libc6 vesion is outdated in normal > upgrade process. I would disagree on the definition of a normal upgrade process, one would never end up with a combination of incompatible rsync + libc, that can only happen if somebody installs a package that is not supported by Debian (not from the official repos) and that package holds libc back (thus running an unsupported version)[4]. This doesn't necessarily mean your bug report is not valid, this issue could mean I wouldn't be able to backport rsync, so I appreciate you reporting it. > OK, I expressed my thoughts, technical problem is clear, so > it's left for you and upstream to find the best and right solution. I did try to reproduce the issue on Jessie, with libc 2.28, but I was able to run "rsync -va /etc/ld.so.cache /tmp" with no issues. It might be possible that the issue you're seeing only affects libc between 2.28 and 2.31, if that's the case, it will be very tricky for me to reproduce since Debian only supports 2.28, 2.31 and 2.32. I want to try it again, but this time by trying to reproduce the issues I've seen in our CI system, let's see how it goes. If you have an affected system at hand, I would kindly ask you to try to implement upstream's workaround[5] and see if it makes your issue go away. Just a small correction, I see you mentioned lchown, but the issue is related to lchmod. I reckon you understand this, and it was just a typo (easy to mix both together). I'm only mentioning this to avoid confusion from someone else reading this. [0] https://www.debian.org/doc/debian-policy/ch-sharedlibs.html#s-sharedlibs-symbols [1] https://www.debian.org/doc/manuals/maint-guide/advanced.en.html#librarysymbols [2] https://wiki.debian.org/UsingSymbolsFiles [3] That is, if we confirm that this is a bug and upstream doesn't address it, though it's looking like upstream would be happy to fix it, removing the need for the version requirement. [4] It could also happen if a person does a partial upgrade between releases with official packages only, but then it's something we can't really say we support. [5] https://github.com/WayneD/rsync/issues/109#issuecomment-927481747 Thank you, -- Samuel Henrique
Bug#995046: 3.2.3-7 fails with libc6-2.30
Hi Samuel. On Sat, Sep 25, 2021 at 09:29:42PM +0100, Samuel Henrique wrote: > > for lchown() is definitely a new feature, so the right way, IMHO, > > is to correct dependences for rsync package: "libc6 (>= 2.31)". > > I don't think it's as simple as that, I understand your issue now, but > I wouldn't just bump the dependency requirement without understanding > it better. > As it stands now, the version requirement is automatically set by > debhelper, based on the symbols rsync uses. Are you sure about version? I doubt. The list of libraries is built by debhelper, I agree. But the *version* dependency, IMHO, can't be built on the list of referenced symbols. To build version dependency automatically script should have access to the whole history of all versions of libc6, with list of symbols and markers when each symbol become usable. Note that presense of symbol is not mean usability: for libc6-2.30 entry "lchown()" is definitely present (ltrace confirms it), but attemptp to call it return an error "not implemented", i.e. this version contains only stub code (symbol is reserved for future use). I belive the version dependency have to be filled manually. Moreover, with manual assignment buggy versions of library can be excluded, which is very unlikely to be done automatically. > The bug we are facing comes from libc6, as we can see in the upstream > bug report, and rsync is trying to workaround that issue (the patch I > introduced on 3.2.3-7). I did not look through the code, but probably it detects presence of lchown() but don't cover the situation when this entry is not functional. > Having outdated packages is not a problem, but having only a subset of > them is an indicator of something wrong. Rsync 3.2.3-7 migrated to > testing after libc6 2.32, so this issue will not happen unless the > person purposely holds libc back. I thought your system was running > like that by accident. > I'm not judging here, I just meant to explain that the trigger to the > issue will not affect every Debian user. Yes, this issue may occure when libc6 vesion is outdated in normal upgrade process. > Being able to keep the current version requirement will be important > in the long term as I want to be able to upload new releases to > backports. I also think that there's a chance rsync's upstream solves > the new regression soon (if confirmed), which would deem the version > bump unneeded. > > What do you think? OK, I expressed my thoughts, technical problem is clear, so it's left for you and upstream to find the best and right solution. Thank you for support, Samuel. -- Eugene Berdnikov
Bug#995046: 3.2.3-7 fails with libc6-2.30
Hello Eugene, > > 1) Do you know how is it possible that you were running Debian testing > > with libc6 2.30-4? Even Debian stable has a newer version, I believe > > you could have missed running apt full-upgrade at some point. > > I have several hosts with old kernels (like 4.17) and their headers, > and gcc-7, which are tied to libc6-2.30 by their dependences. > On "aptitude install libc6" first resolver's proposal was to remove > these packages. I agreed with this choice to install libc6-2.32. Interesting, so I will assume the system is in a consistent state. > I have a copy of one such system on backup, it may be studied for > more details if need... However, it looks pointless, because support > for lchown() is definitely a new feature, so the right way, IMHO, > is to correct dependences for rsync package: "libc6 (>= 2.31)". I don't think it's as simple as that, I understand your issue now, but I wouldn't just bump the dependency requirement without understanding it better. As it stands now, the version requirement is automatically set by debhelper, based on the symbols rsync uses. The bug we are facing comes from libc6, as we can see in the upstream bug report, and rsync is trying to workaround that issue (the patch I introduced on 3.2.3-7). > > 2) Is your system under a chroot and/or without /proc mounted? > > No, fault happens with physical hosts (not VMs or containers), > where /proc is mounted. However, they are all running SysV init. Cool, I think SysV init won't take a role in this, but good to know the details. > > > Upgrade libc6 to 2.32-4 solves the problem. > > That's the current version available in testing and unstable, so > > anybody running those will be fine. > > Right. I also have some similar hosts with libc-2.32, probably they were > upgraded to last version due to absence of old kernels, old gcc, etc. > Rsync runs fine on them. Good to know. > > If anything, 3.2.3-7 is fixing the very same issue you reported, which > > should have been happening in rsync 3.2.3 + libc6 2.32 (but you > > mention you had an older libc6). > > > > Overall I believe there might be something wrong in your system, > > related to libc6. > > Every system with periodic updates might have outdated packages, > generally it's not a problem (and should not be a problem) if binary API > is compatible and all package dependences are correct. Having outdated packages is not a problem, but having only a subset of them is an indicator of something wrong. Rsync 3.2.3-7 migrated to testing after libc6 2.32, so this issue will not happen unless the person purposely holds libc back. I thought your system was running like that by accident. I'm not judging here, I just meant to explain that the trigger to the issue will not affect every Debian user. My current guess is that rsync's workaround accidentally moved the regression from libc 2.32 to libc < 2.32. I've added a comment to rsync's issue, you can follow the discussion there as well: https://github.com/WayneD/rsync/issues/109 To be clear, I agree that as it stands right now, rsync should depend on libc >= 2.32, but I don't wanna make this change before confirming with upstream that this is not being caused by a regression from the workaround. Being able to keep the current version requirement will be important in the long term as I want to be able to upload new releases to backports. I also think that there's a chance rsync's upstream solves the new regression soon (if confirmed), which would deem the version bump unneeded. What do you think? Thank you, -- Samuel Henrique
Bug#995046: 3.2.3-7 fails with libc6-2.30
Hi Samuel. On Sat, Sep 25, 2021 at 03:08:47PM +0100, Samuel Henrique wrote: > 1) Do you know how is it possible that you were running Debian testing > with libc6 2.30-4? Even Debian stable has a newer version, I believe > you could have missed running apt full-upgrade at some point. I have several hosts with old kernels (like 4.17) and their headers, and gcc-7, which are tied to libc6-2.30 by their dependences. On "aptitude install libc6" first resolver's proposal was to remove these packages. I agreed with this choice to install libc6-2.32. I have a copy of one such system on backup, it may be studied for more details if need... However, it looks pointless, because support for lchown() is definitely a new feature, so the right way, IMHO, is to correct dependences for rsync package: "libc6 (>= 2.31)". > 2) Is your system under a chroot and/or without /proc mounted? No, fault happens with physical hosts (not VMs or containers), where /proc is mounted. However, they are all running SysV init. > > Upgrade libc6 to 2.32-4 solves the problem. > That's the current version available in testing and unstable, so > anybody running those will be fine. Right. I also have some similar hosts with libc-2.32, probably they were upgraded to last version due to absence of old kernels, old gcc, etc. Rsync runs fine on them. > If anything, 3.2.3-7 is fixing the very same issue you reported, which > should have been happening in rsync 3.2.3 + libc6 2.32 (but you > mention you had an older libc6). > > Overall I believe there might be something wrong in your system, > related to libc6. Every system with periodic updates might have outdated packages, generally it's not a problem (and should not be a problem) if binary API is compatible and all package dependences are correct. -- Eugene Berdnikov
Bug#995046: 3.2.3-7 fails with libc6-2.30
Control: severity -1 normal Hello Eugene, thanks for your bug report :) I think I still need to understand a few things before proceeding, seems like there's something weird going on in your system. 1) Do you know how is it possible that you were running Debian testing with libc6 2.30-4? Even Debian stable has a newer version, I believe you could have missed running apt full-upgrade at some point. 2) Is your system under a chroot and/or without /proc mounted? We've had this issue in the near past [0], but it should be patched for 3.2.3-7 [1], even with older libc6. > Upgrade libc6 to 2.32-4 solves the problem. That's the current version available in testing and unstable, so anybody running those will be fine. Stable currently has libc6 2.31, and I plan on having a backports of rsync at some point in the future, so this issue could affect that, though I don't see how the changes between 3.2.3-4 and 3.2.3-7 could trigger this bug. I'm also considering that upstream's bug reports says this issue only started with libc6 2.32 [0][2]. If anything, 3.2.3-7 is fixing the very same issue you reported, which should have been happening in rsync 3.2.3 + libc6 2.32 (but you mention you had an older libc6). Overall I believe there might be something wrong in your system, related to libc6. I will lower the priority to normal, considering all of these, and will increase it if needed once we understand your issue better. [0] https://github.com/WayneD/rsync/issues/109 [1] https://salsa.debian.org/debian/rsync/-/commit/3320a1c1b9e9fcf4b4b759a194a6059380c56b88 [2] https://sourceware.org/bugzilla/show_bug.cgi?id=26401 Thank you, -- Samuel Henrique
Bug#995046: 3.2.3-7 fails with libc6-2.30
Package: rsync
Version: 3.2.3-7
Severity: important
Hello. After last update of Debian/testing (bookworm/sid) with rsync 3.2.3-4
to rsync-3.2.3-7, new version can't change file attributes. Example:
# rsync -va /etc/ld.so.cache /tmp
sending incremental file list
ld.so.cache
rsync: [receiver] failed to set permissions on "/tmp/.ld.so.cache.hvkXg2":
Function not implemented (38)
Proper permissions for file are not set.
Rsync 3.2.3-4 on the same system works good.
Ltrace for rsync-3.2.3-7 shows call to non-implemented function "lchmod":
[pid 5535] __lxstat(1, ".ld.so.cache.Hhy5UI", 0x7ffe8558e790
[pid 5535] SYS_lstat(".ld.so.cache.Hhy5UI", 0x7ffe8558e790) = 0
[pid 5535] <... __lxstat resumed> ) = 0
[pid 5535] utimensat(0xff9c, 0x7ffe85590a00, 0x7ffe8558e700, 256
[pid 5535] SYS_utimensat(0xff9c, 0x7ffe85590a00, 0x7ffe8558e700, 256) = 0
[pid 5535] <... utimensat resumed> ) = 0
[pid 5535] lchmod(0x7ffe85590a00, 420, 0x7ffe8558e700, 0)=
0x
___^^___
[pid 5535] __errno_location()=
0x7fc937826480
[pid 5535] __asprintf_chk(0x55c1a1db93d8, 1, 0x55c1a1d8e0cf, 0x55c1a1db8360) =
29
[pid 5535] __errno_location()=
0x7fc937826480
[pid 5535] __snprintf_chk(0x7ffe8558d270, 5120, 1, 5120) = 18
[pid 5535] __vsnprintf_chk(0x7ffe8558d282, 5102, 1, -1) = 58
[pid 5535] strerror(38) =
"Function not implemented"
[pid 5535] __snprintf_chk(0x7ffe8558d2bc, 5044, 1, -1) = 32
[pid 5535] memcpy(0x55c1a3b57924, "rsync: [receiver] failed to set permissions
on "/rz/etc/.ld.so.cache.Hhy5UI": Function not implemented (38)\n", 108) =
0x55c1a3b57924
List of packages for rsync-3.2.3-7 on broken system are:
||/ Name Version Architecture Description
+++--===--===
ii libacl1:amd642.3.1-1 amd64access control list - shared
library
ii libc6:amd64 2.30-4 amd64GNU C Library: Shared
libraries
ii liblz4-1:amd64 1.9.3-2 amd64Fast LZ compression algorithm
library - runtime
ii libpopt0:amd64 1.18-3 amd64lib for parsing cmdline
parameters
ii libssl1.1:amd64 1.1.1l-1amd64Secure Sockets Layer toolkit
- shared libraries
ii libxxhash0:amd64 0.8.0-2 amd64shared library for xxhash
ii libzstd1:amd64 1.4.8+dfsg-2.1 amd64fast lossless compression
algorithm
ii lsb-base 11.1.0 all Linux Standard Base init
script functionality
ii zlib1g:amd64 1:1.2.11.dfsg-2 amd64compression library - runtime
Upgrade libc6 to 2.32-4 solves the problem.
I suspect that lchmod() function appears in libc6-2.31 (or in some other
library), but dependency list for rsync mentions "libc6 (>= 2.15)".
Probably it should be corrected.
--
Eugene Berdnikov
