Re: [gentoo-portage-dev] [PATCH] Add QA check for unresolved soname dependencies (bug 704320)

2020-01-06 Thread Zac Medico
On 1/5/20 10:48 PM, Michael 'veremitz' Everitt wrote:
> On 06/01/20 06:38, Zac Medico wrote:
>> Example output for maven-bin-3.6.2 (bug 704618):
>>
>>  * QA Notice: Unresolved soname dependencies:
>>  *
>>  *   /usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: 
>> libc.so.7 libutil.so.9
>>  *   /usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: 
>> libc.so.7 libutil.so.9
>>  *
>>
>> This warning comes up when a library or executable has one or
>> more soname dependencies (found in its NEEDED.ELF.2 metadata)
>> that could not be resolved by usual means. If you run ldd
>> on files like these then it will report a "not found" error
>> for each unresolved soname dependency. In order to correct
>> problems with soname dependency resolution, use one or more
>> of the approaches described in the following sections.
>>
>> Content of the NEEDED.ELF.2 metadata file may be useful
>> for debugging purposes. Find the NEEDED.ELF.2 file in the
>> ${D}/../build-info/ directory after the ebuild src_install
>> phase completes, or in the /var/db/pkg/*/*/ directory for an
>> installed package. Each line of the NEEDED.ELF.2 file contains
>> semicolon separated values for a single ELF file. The soname
>> dependencies are found in the DT_NEEDED column:
>>
>>  E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category
>>
>> External dependencies
>>
>> For packages that install pre-built binaries, it may be possible
>> to resolve soname dependencies simply by adding dependencies
>> for one or more other packages that are known to provide the
>> needed sonames.
>>
>> Removal of unecessary files
>>
>> For packages that install pre-built binaries, it may be possible
>> to resolve soname dependencies simply by removing unnecessary
>> files which have unresolved soname dependencies. For example,
>> some pre-built binary packages include binaries intended for
>> irrelevant architectures or operating systems, and these files
>> can simply be removed because they are unnecessary.
>>
>> Addition of DT_RUNPATH entries
>>
>> If the relevant dependencies are installed in a location that
>> is not included in the dynamic linker search path, then it's
>> necessary for files to include a DT_RUNPATH entry which refers
>> to the appropriate directory. The special $ORIGIN value can
>> be used to create a relative path reference in DT_RUNPATH,
>> where $ORIGIN is a placeholder for the directory where the
>> file having the DT_RUNPATH entry is located.
>>
>> For pre-built binaries, it may be necessary to fix up
>> DT_RUNPATH using patchelf --set-rpath. For example, use
>> patchelf --set-rpath '$ORIGIN' if a given binary should link
>> to libraries found in the same directory as the binary itself,
>> or use patchelf --set-rpath '$ORIGIN/libs' if a given binary
>> should link to libraries found in a subdirectory named libs
>> found in the same directory as the binary itself.
>>
>> For binaries built from source, a flag like
>> -Wl,-rpath,/path/of/directory/containing/libs will create
>> binaries with the desired DT_RUNPATH entry.
>>
>> Bug: https://bugs.gentoo.org/704320
>> Signed-off-by: Zac Medico 
>> ---
> 
> 
> Looks awesome!
> 
> One comment: would it be helpful for developers/maintainers to have some
> indication of what's going on here, and what attempts portage is making in
> this regard?

Effectively, it's as though portage runs ldd on all the files installed
by the ebuild, and reports any unresolved soname dependencies. If
portage doesn't agree with ldd then it's a portage bug. If this happens
then I'd expect people to simply file a bug. I would indicate a flaw in
the SonameDepsProcessor logic, an it's a fairly small amount of code:

https://gitweb.gentoo.org/proj/portage.git/tree/lib/portage/util/_dyn_libs/soname_deps.py?h=portage-2.3.84

> Something like a 'verbose' option that would spit out some
> debugging info so that /if/ something goes wrong, or the outcome wasn't
> what was intended, then this can be picked up and corrected somehow else?

Mope, just file a bug and include the exact ebuild that triggered the
unexpected results.

> I think automation is great for picking these issues up, and shining a
> light on where/why something goes wrong,

Yeah, issues like this will go completely unnoticed otherwise (until
someone uses emerge --ignore-soname-deps=n or triggers some undesirable
preserve-libs behavior like bug 265372 or bug 400979).

> but sometimes it needs a few
> minutes of human intervention to achieve the sane outcome in many
> circumstances.

Well, yeah, the whole point of the QA Notice is to solicit some human
intervention in order to resolve the issue. If the issue could be solved
without human intervention then we wouldn't need the QA Notice.

Maybe I should include "file a portage bug" as an approach to take if
none of the previously suggested resolution approaches appears to be
applicable. After all, the SonameDepsProcessor logic is almost certainly
not an entirely perfect model of 

Re: [gentoo-portage-dev] [PATCH] Add QA check for unresolved soname dependencies (bug 704320)

2020-01-05 Thread Michael 'veremitz' Everitt
On 06/01/20 06:38, Zac Medico wrote:
> Example output for maven-bin-3.6.2 (bug 704618):
>
>  * QA Notice: Unresolved soname dependencies:
>  *
>  */usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: 
> libc.so.7 libutil.so.9
>  */usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: 
> libc.so.7 libutil.so.9
>  *
>
> This warning comes up when a library or executable has one or
> more soname dependencies (found in its NEEDED.ELF.2 metadata)
> that could not be resolved by usual means. If you run ldd
> on files like these then it will report a "not found" error
> for each unresolved soname dependency. In order to correct
> problems with soname dependency resolution, use one or more
> of the approaches described in the following sections.
>
> Content of the NEEDED.ELF.2 metadata file may be useful
> for debugging purposes. Find the NEEDED.ELF.2 file in the
> ${D}/../build-info/ directory after the ebuild src_install
> phase completes, or in the /var/db/pkg/*/*/ directory for an
> installed package. Each line of the NEEDED.ELF.2 file contains
> semicolon separated values for a single ELF file. The soname
> dependencies are found in the DT_NEEDED column:
>
>  E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category
>
> External dependencies
>
> For packages that install pre-built binaries, it may be possible
> to resolve soname dependencies simply by adding dependencies
> for one or more other packages that are known to provide the
> needed sonames.
>
> Removal of unecessary files
>
> For packages that install pre-built binaries, it may be possible
> to resolve soname dependencies simply by removing unnecessary
> files which have unresolved soname dependencies. For example,
> some pre-built binary packages include binaries intended for
> irrelevant architectures or operating systems, and these files
> can simply be removed because they are unnecessary.
>
> Addition of DT_RUNPATH entries
>
> If the relevant dependencies are installed in a location that
> is not included in the dynamic linker search path, then it's
> necessary for files to include a DT_RUNPATH entry which refers
> to the appropriate directory. The special $ORIGIN value can
> be used to create a relative path reference in DT_RUNPATH,
> where $ORIGIN is a placeholder for the directory where the
> file having the DT_RUNPATH entry is located.
>
> For pre-built binaries, it may be necessary to fix up
> DT_RUNPATH using patchelf --set-rpath. For example, use
> patchelf --set-rpath '$ORIGIN' if a given binary should link
> to libraries found in the same directory as the binary itself,
> or use patchelf --set-rpath '$ORIGIN/libs' if a given binary
> should link to libraries found in a subdirectory named libs
> found in the same directory as the binary itself.
>
> For binaries built from source, a flag like
> -Wl,-rpath,/path/of/directory/containing/libs will create
> binaries with the desired DT_RUNPATH entry.
>
> Bug: https://bugs.gentoo.org/704320
> Signed-off-by: Zac Medico 
> ---


Looks awesome!

One comment: would it be helpful for developers/maintainers to have some
indication of what's going on here, and what attempts portage is making in
this regard? Something like a 'verbose' option that would spit out some
debugging info so that /if/ something goes wrong, or the outcome wasn't
what was intended, then this can be picked up and corrected somehow else?

I think automation is great for picking these issues up, and shining a
light on where/why something goes wrong, but sometimes it needs a few
minutes of human intervention to achieve the sane outcome in many
circumstances.



signature.asc
Description: OpenPGP digital signature


[gentoo-portage-dev] [PATCH] Add QA check for unresolved soname dependencies (bug 704320)

2020-01-05 Thread Zac Medico
Example output for maven-bin-3.6.2 (bug 704618):

 * QA Notice: Unresolved soname dependencies:
 *
 *  /usr/share/maven-bin-3.6/lib/jansi-native/freebsd32/libjansi.so: 
libc.so.7 libutil.so.9
 *  /usr/share/maven-bin-3.6/lib/jansi-native/freebsd64/libjansi.so: 
libc.so.7 libutil.so.9
 *

This warning comes up when a library or executable has one or
more soname dependencies (found in its NEEDED.ELF.2 metadata)
that could not be resolved by usual means. If you run ldd
on files like these then it will report a "not found" error
for each unresolved soname dependency. In order to correct
problems with soname dependency resolution, use one or more
of the approaches described in the following sections.

Content of the NEEDED.ELF.2 metadata file may be useful
for debugging purposes. Find the NEEDED.ELF.2 file in the
${D}/../build-info/ directory after the ebuild src_install
phase completes, or in the /var/db/pkg/*/*/ directory for an
installed package. Each line of the NEEDED.ELF.2 file contains
semicolon separated values for a single ELF file. The soname
dependencies are found in the DT_NEEDED column:

 E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category

External dependencies

For packages that install pre-built binaries, it may be possible
to resolve soname dependencies simply by adding dependencies
for one or more other packages that are known to provide the
needed sonames.

Removal of unecessary files

For packages that install pre-built binaries, it may be possible
to resolve soname dependencies simply by removing unnecessary
files which have unresolved soname dependencies. For example,
some pre-built binary packages include binaries intended for
irrelevant architectures or operating systems, and these files
can simply be removed because they are unnecessary.

Addition of DT_RUNPATH entries

If the relevant dependencies are installed in a location that
is not included in the dynamic linker search path, then it's
necessary for files to include a DT_RUNPATH entry which refers
to the appropriate directory. The special $ORIGIN value can
be used to create a relative path reference in DT_RUNPATH,
where $ORIGIN is a placeholder for the directory where the
file having the DT_RUNPATH entry is located.

For pre-built binaries, it may be necessary to fix up
DT_RUNPATH using patchelf --set-rpath. For example, use
patchelf --set-rpath '$ORIGIN' if a given binary should link
to libraries found in the same directory as the binary itself,
or use patchelf --set-rpath '$ORIGIN/libs' if a given binary
should link to libraries found in a subdirectory named libs
found in the same directory as the binary itself.

For binaries built from source, a flag like
-Wl,-rpath,/path/of/directory/containing/libs will create
binaries with the desired DT_RUNPATH entry.

Bug: https://bugs.gentoo.org/704320
Signed-off-by: Zac Medico 
---
 doc/qa.docbook   | 74 +++
 lib/_emerge/EbuildPhase.py   | 63 ++---
 lib/portage/dep/soname/SonameAtom.py | 12 ++-
 lib/portage/util/_dyn_libs/soname_deps_qa.py | 98 
 4 files changed, 232 insertions(+), 15 deletions(-)
 create mode 100644 lib/portage/util/_dyn_libs/soname_deps_qa.py

diff --git a/doc/qa.docbook b/doc/qa.docbook
index 28ff6cf8e..cf8a3bc8d 100644
--- a/doc/qa.docbook
+++ b/doc/qa.docbook
@@ -127,6 +127,80 @@
   
  
 
+ 
+  Unresolved soname dependencies
+  
+   
+QA Notice: Unresolved soname dependencies
+   
+  
+  
+   This warning comes up when a library or executable has one or more
+   soname dependencies (found in its NEEDED.ELF.2 metadata) that could
+   not be resolved by usual means. If you run ldd on
+   files like these then it will report a "not found" error for each
+   unresolved soname dependency. In order to correct problems with
+   soname dependency resolution, use one or more of the approaches
+   described in the following sections.
+   
+   
+   Content of the NEEDED.ELF.2 metadata file may be useful for
+   debugging purposes. Find the NEEDED.ELF.2 file in the
+   ${D}/../build-info/ directory after the ebuild src_install phase
+   completes, or in the /var/db/pkg/*/*/ directory for an installed
+   package. Each line of the NEEDED.ELF.2 file contains semicolon
+   separated values for a single ELF file. The soname dependencies are
+   found in the DT_NEEDED column:
+   
+   E_MACHINE;path;DT_SONAME;DT_RUNPATH;DT_NEEDED;multilib category
+  
+  
+  
+   External dependencies
+   
+   For packages that install pre-built binaries, it may be possible to
+   resolve soname dependencies simply by adding dependencies for one
+   or more other packages that are known to provide the needed sonames.
+   
+  
+  
+   Removal of unecessary files
+   
+   For packages that install pre-built binaries, it may be possible to
+   resolve soname dependencies simply by removing unnecessary files
+   which have unresolved soname dependencies. For example, some pre-built
+