[gentoo-portage-dev] [PATCH 1/1] bin/misc-function.sh: check scanelf return code
This is part of a series of fixes for the linked bug (failure
to preserve libraries in some situations).
We need to check if scanelf failed when calling it in the
installed-files QA check as we later use it to populate the VDB.
Silently continuing results in either blank e.g. PROVIDES,
NEEDED{,.ELF.2} or those files may be missing entirely,
resulting in a corrupt state both on the system and in
any generated binpkgs.
Adds an escape variable (PORTAGE_NO_SCANELF_CHECK) to allow
re-emerging pax-utils if it's broken.
Bug: https://bugs.gentoo.org/811462
Signed-off-by: Sam James
---
bin/misc-functions.sh | 64 +--
1 file changed, 50 insertions(+), 14 deletions(-)
diff --git a/bin/misc-functions.sh b/bin/misc-functions.sh
index bd1fb7553..e4defa550 100755
--- a/bin/misc-functions.sh
+++ b/bin/misc-functions.sh
@@ -177,25 +177,61 @@ install_qa_check() {
if type -P scanelf > /dev/null ; then
# Save NEEDED information after removing self-contained
providers
rm -f "$PORTAGE_BUILDDIR"/build-info/NEEDED{,.ELF.2}
+
# We don't use scanelf -q, since that would omit libraries like
# musl's /usr/lib/libc.so which do not have any DT_NEEDED or
# DT_SONAME settings. Since we don't use scanelf -q, we have to
# handle the special rpath value " - " below.
- scanelf -yRBF '%a;%p;%S;%r;%n' "${D%/}/" | { while IFS= read -r
l; do
- arch=${l%%;*}; l=${l#*;}
- obj="/${l%%;*}"; l=${l#*;}
- soname=${l%%;*}; l=${l#*;}
- rpath=${l%%;*}; l=${l#*;}; [ "${rpath}" = " - " ] &&
rpath=""
- needed=${l%%;*}; l=${l#*;}
-
- # Infer implicit soname from basename (bug 715162).
- if [[ -z ${soname} && $(file "${D%/}${obj}") == *"SB
shared object"* ]]; then
- soname=${obj##*/}
- fi
+ scanelf_output=$(scanelf -yRBF '%a;%p;%S;%r;%n' "${D%/}/")
+
+ case $? in
+ 0)
+ # Proceed
+ ;;
+ 159)
+ # Unknown syscall
+ eerror "Failed to run scanelf (unknown syscall)"
+
+ if [[ -z ${PORTAGE_NO_SCANELF_CHECK} ]]; then
+ # Abort only if the special recovery
variable isn't set
+ eerror "Please upgrade pax-utils with:"
+ eerror " PORTAGE_NO_SCANELF_CHECK=1
emerge -v1 app-misc/pax-utils"
+ eerror "Aborting to avoid corrupting
metadata"
+ die "${0##*/}: Failed to run scanelf!
Update pax-utils?"
+ fi
+ ;;
+ *)
+ # Failed in another way
+ eerror "Failed to run scanelf (returned: $?)!"
+
+ if [[ -z ${PORTAGE_NO_SCANELF_CHECK} ]]; then
+ # Abort only if the special recovery
variable isn't set
+ eerror "Please report this bug at
https://bugs.gentoo.org/!;
+ eerror "It may be possible to re-emerge
pax-utils with:"
+ eerror " PORTAGE_NO_SCANELF_CHECK=1
emerge -v1 app-misc/pax-utils"
+ eerror "Aborting to avoid corrupting
metadata"
+ die "${0##*/}: Failed to run scanelf!"
+ fi
+ ;;
+ esac
+
+ if [[ -n ${scanelf_output} ]]; then
+ while IFS= read -r l; do
+ arch=${l%%;*}; l=${l#*;}
+ obj="/${l%%;*}"; l=${l#*;}
+ soname=${l%%;*}; l=${l#*;}
+ rpath=${l%%;*}; l=${l#*;}; [ "${rpath}" = " -
" ] && rpath=""
+ needed=${l%%;*}; l=${l#*;}
+
+ # Infer implicit soname from basename (bug
715162).
+ if [[ -z ${soname} && $(file "${D%/}${obj}") ==
*"SB shared object"* ]]; then
+ soname=${obj##*/}
+ fi
- echo "${obj} ${needed}" >>
"${PORTAGE_BUILDDIR}"/build-info/NEEDED
- echo "${arch#EM_};${obj};${soname};${rpath};${needed}"
>> "${PORTAGE_BUILDDIR}"/build-info/NEEDED.ELF.2
- done }
+ echo "${obj} ${needed}" >>
[gentoo-portage-dev] [PATCH 0/1] Check for errors in scanelf
Posted at https://github.com/gentoo/portage/pull/750 originally and merged in 3.0.24, but posting here for posterity. Related to the general preserve-libs issues I've been working on. Sam James (1): bin/misc-function.sh: check scanelf return code bin/misc-functions.sh | 64 +-- 1 file changed, 50 insertions(+), 14 deletions(-) -- 2.33.0
[gentoo-portage-dev] [PATCH 2/2] lib/_emerge/resolver/output_helpers.py: explicitly state 'all satisfied'
This makes things a bit less confusing and tries to avoid
users focusing on (soft) blocks which aren't actually
the problem they're hitting.
Signed-off-by: Sam James
---
lib/_emerge/resolver/output_helpers.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/lib/_emerge/resolver/output_helpers.py
b/lib/_emerge/resolver/output_helpers.py
index f80b79ccf..6ce812189 100644
--- a/lib/_emerge/resolver/output_helpers.py
+++ b/lib/_emerge/resolver/output_helpers.py
@@ -163,6 +163,8 @@ class _PackageCounters:
myoutput.append(
bad(" (%s unsatisfied)") % (self.blocks -
self.blocks_satisfied)
)
+else:
+myoutput.append(" (all satisfied)")
return "".join(myoutput)
--
2.33.0
[gentoo-portage-dev] [PATCH 1/2] lib/_emerge/resolver/output.py: say 'soft blocking' explicitly
Before:
```
[blocks b ] >perl-core/Scalar-List-Utils-1.550.0-r999
(">perl-core/Scalar-List-Utils-1.550.0-r999" is blocking
virtual/perl-Scalar-List-Utils-1.550.0)
```
After:
```
[blocks b ] >perl-core/Scalar-List-Utils-1.550.0-r999
(">perl-core/Scalar-List-Utils-1.550.0-r999" is soft blocking
virtual/perl-Scalar-List-Utils-1.550.0)
```
This should make it a little bit clearer when a block matters,
especially given we already explicitly say 'hard blocking'
for the opposite case.
Signed-off-by: Sam James
---
lib/_emerge/resolver/output.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/_emerge/resolver/output.py b/lib/_emerge/resolver/output.py
index e891d84c0..7b5602a78 100644
--- a/lib/_emerge/resolver/output.py
+++ b/lib/_emerge/resolver/output.py
@@ -108,7 +108,7 @@ class Display:
if blocker.atom.blocker.overlap.forbid:
blocking_desc = "hard blocking"
else:
-blocking_desc = "blocking"
+blocking_desc = "soft blocking"
if self.resolved != blocker.atom:
addl += colorize(
self.blocker_style,
--
2.33.0
[gentoo-portage-dev] [PATCH 2/2] Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg
This is part of a series of fixes for the linked bug (failure to preserve libraries in some situations). When unpacking a binpkg to be installed, we should check for the existence of PROVIDES if we're installing any dynamic libraries. If PROVIDES does not exist in that case, this suggests that e.g. scanelf malfunctioned or some corruption occurred. Bug: https://bugs.gentoo.org/811462 Signed-off-by: Sam James --- lib/_emerge/Binpkg.py | 28 +++- 1 file changed, 27 insertions(+), 1 deletion(-) diff --git a/lib/_emerge/Binpkg.py b/lib/_emerge/Binpkg.py index c7dde69bd..9b876f354 100644 --- a/lib/_emerge/Binpkg.py +++ b/lib/_emerge/Binpkg.py @@ -2,7 +2,7 @@ # Distributed under the terms of the GNU General Public License v2 import functools - +import glob import _emerge.emergelog from _emerge.EbuildPhase import EbuildPhase from _emerge.BinpkgFetcher import BinpkgFetcher @@ -13,6 +13,7 @@ from _emerge.EbuildMerge import EbuildMerge from _emerge.EbuildBuildDir import EbuildBuildDir from _emerge.SpawnProcess import SpawnProcess from portage.eapi import eapi_exports_replace_vars +from portage.output import colorize from portage.util import ensure_dirs from portage.util._async.AsyncTaskFuture import AsyncTaskFuture import portage @@ -425,6 +426,31 @@ class Binpkg(CompositeTask): self._async_unlock_builddir(returncode=self.returncode) return +# Before anything else, let's do an integrity check. +(provides,) = self._bintree.dbapi.aux_get(self.pkg.cpv, ["PROVIDES"]) +if not provides: +# Let's check if we've got inconsistent results. +# If we're installing dynamic libraries (.so files), we should +# really have a PROVIDES. +# (This is a complementary check at the point of ingestion for the +# creation check in doebuild.py) +# Note: we could check a non-empty PROVIDES against the list of .sos, +# but this doesn't gain us anything. We're interested in failure +# to properly parse the installed files at all, which should really +# be a global problem (e.g. bug #811462) +installed_dynlibs = glob.glob( +self.settings["D"] + "/**/*.so", recursive=True +) +if installed_dynlibs: +self._writemsg_level( +colorize( +"BAD", +"!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!", +), +noiselevel=-1, +level=logging.ERROR, +) + try: with io.open( _unicode_encode( -- 2.33.0
[gentoo-portage-dev] [PATCH 1/2] doebuild.py: check for inconsistent PROVIDES/image post-src_install
This is part of a series of fixes for the linked bug (failure to preserve libraries in some situations). At the point of installation (even if not merging), we need to detect inconsistent metadata: PROVIDES should be populated if we're installing any dynamic libraries. This suggests that e.g. scanelf malfunctioned or some corruption occurred. Bug: https://bugs.gentoo.org/811462 Signed-off-by: Sam James --- lib/portage/package/ebuild/doebuild.py | 24 +++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/lib/portage/package/ebuild/doebuild.py b/lib/portage/package/ebuild/doebuild.py index 9650a8444..dc3fe3d97 100644 --- a/lib/portage/package/ebuild/doebuild.py +++ b/lib/portage/package/ebuild/doebuild.py @@ -3,6 +3,7 @@ __all__ = ["doebuild", "doebuild_environment", "spawn", "spawnebuild"] +import glob import grp import gzip import errno @@ -3079,7 +3080,7 @@ def _post_src_install_soname_symlinks(mysettings, out): ) as f: f.write(soname_deps.requires) -if soname_deps.provides is not None: +if soname_deps.provides: with io.open( _unicode_encode( os.path.join(build_info_dir, "PROVIDES"), @@ -3091,6 +3092,27 @@ def _post_src_install_soname_symlinks(mysettings, out): errors="strict", ) as f: f.write(soname_deps.provides) +else: +# Let's check if we've got inconsistent results. +# If we're installing dynamic libraries (.so files), we should +# really have a PROVIDES. +# (This is a complementary check at the point of creation for the +# ingestion check in Binpkg.py) +# Note: we could check a non-empty PROVIDES against the list of .sos, +# but this doesn't gain us anything. We're interested in failure +# to properly parse the installed files at all, which should really +# be a global problem (e.g. bug #811462) +installed_dynlibs = glob.glob(image_dir + "/**/*.so", recursive=True) + +if installed_dynlibs: +self._writemsg_level( +colorize( +"BAD", +"!!! Error! Installing dynamic libraries (.so) with blank PROVIDES!", +), +noiselevel=-1, +level=logging.ERROR, +) if unrecognized_elf_files: qa_msg = ["QA Notice: Unrecognized ELF file(s):"] -- 2.33.0
[gentoo-portage-dev] [PATCH 0/2] Detect broken VDB on merging/binpkg creation
Further fixes for when the VDB is corrupted by e.g. broken scanelf. Already posted on GH a while ago at https://github.com/gentoo/portage/pull/744. Sam James (2): doebuild.py: check for inconsistent PROVIDES/image post-src_install Binpkg.py: check for inconsistent PROVIDES/image when unpacking binpkg lib/_emerge/Binpkg.py | 28 +- lib/portage/package/ebuild/doebuild.py | 24 +- 2 files changed, 50 insertions(+), 2 deletions(-) -- 2.33.0
