bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Ludovic Courtès  skribis:

> $ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; 
> GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/IMB-MPI1'
> /tmp/fakechroot-test/gnu/store/i9ypns78lmy8xr0x4zvdwc4b80m0yl72-intel-mpi-benchmarks-2019.6/bin//IMB-MPI1:
>  error while loading shared libraries: libevent_core-2.1.so.7: cannot open 
> shared object file: No such file or directory
>
> Reason appears to be that ‘LOADER_AUDIT_RUNPATH’ only contains what’s in
> the RUNPATH of ‘IMB-MPI1’ and not its indirect dependencies—libevent is
> one of them.

Fixed (for good!) in d15dbdb79c617b8b710ef5a9502be0d5f640aaaf!

Ludo’.

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Reopening this old bug from this commit:

--8<---cut here---start->8---
$ guix describe
Generation 250  Mar 12 2023 23:58:03(current)
  guix 1ed227d
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 1ed227d7952af48efe50a2f6c9537e17c356daa1
--8<---cut here---end--->8---

> I pushed this as a workaround as commit
> 58abd5873985e0cd9a2926867bf697c5e7bc01f9.
>
> The ld.so issue is still open but there seems to be consensus that it’s
> a bug: .

Commit 58abd5873985e0cd9a2926867bf697c5e7bc01f9 appears to be
incomplete.

Let’s suppose this:

  (mkdir /tmp/fakechroot-test && cd ~/tmp/fakechroot-test/ && \
   tar xf $(guix pack --save-provenance -RR openmpi intel-mpi-benchmarks 
bash-minimal -S /bin=bin))

The motivating example at the beginning of this thread works fine:

--8<---cut here---start->8---
$ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; 
GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/orterun --version'
orterun (OpenRTE) 4.1.4

Report bugs to http://www.open-mpi.org/community/help/
--8<---cut here---end--->8---

… but this doesn’t:

--8<---cut here---start->8---
$ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; 
GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/IMB-MPI1'
/tmp/fakechroot-test/gnu/store/i9ypns78lmy8xr0x4zvdwc4b80m0yl72-intel-mpi-benchmarks-2019.6/bin//IMB-MPI1:
 error while loading shared libraries: libevent_core-2.1.so.7: cannot open 
shared object file: No such file or directory
--8<---cut here---end--->8---

Reason appears to be that ‘LOADER_AUDIT_RUNPATH’ only contains what’s in
the RUNPATH of ‘IMB-MPI1’ and not its indirect dependencies—libevent is
one of them.

Ludo’.

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Hi,

Ludovic Courtès  skribis:

> The patch below fixes the issue for this particular example by simply
> passing the whole RUNPATH of the wrapped executable as ‘--library-path’,
> as was the case in v1 of the patch set¹.  (This assumes that the RUNPATH
> of pack-audit.so is a subset of that of the program, which is usually
> the case.)

I pushed this as a workaround as commit
58abd5873985e0cd9a2926867bf697c5e7bc01f9.

The ld.so issue is still open but there seems to be consensus that it’s
a bug: .

Closing!

Ludo’.

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Ludovic Courtès  skribis:

> Ludovic Courtès  skribis:
>
>> Indeed, we can see ‘stat’ calls passed raw /gnu/store file names from
>> RUNPATH entries (instead of /tmp/fakechroot-test/gnu/store), suggesting
>> that ‘la_objsearch’ didn’t have a chance to rewrite them:
>
> This is probably an ld.so bug:
>
>   https://sourceware.org/bugzilla/show_bug.cgi?id=26634

The patch below provides a fix/workaround for glibc, confirming the
hypothesis above.  (I don’t think we should apply this patch though,
rather we’ll work around the issue in ‘guix pack’.)

Ludo’.

diff --git a/gnu/packages/base.scm b/gnu/packages/base.scm
index c83775d8ee..fa4da801af 100644
--- a/gnu/packages/base.scm
+++ b/gnu/packages/base.scm
@@ -685,6 +685,7 @@ the store.")
   (package
(name "glibc")
(version "2.31")
+   (replacement glibc-2.31/fixed)
(source (origin
 (method url-fetch)
 (uri (string-append "mirror://gnu/glibc/glibc-" version ".tar.xz"))
@@ -933,6 +934,15 @@ with the Linux kernel.")
(license lgpl2.0+)
(home-page "https://www.gnu.org/software/libc/;)))
 
+(define-public glibc-2.31/fixed
+  (package
+(inherit glibc)
+(version "2.3A")
+(source (origin
+  (inherit (package-source glibc))
+  (patches (append (search-patches "glibc-audit-stat.patch")
+   (origin-patches (package-source glibc
+
 ;; Below are old libc versions, which we use mostly to build locale data in
 ;; the old format (which the new libc cannot cope with.)
 
--- /dev/null
+++ b/gnu/packages/patches/glibc-audit-stat.patch
@@ -0,0 +1,126 @@
+diff --git a/elf/dl-load.c b/elf/dl-load.c
+index a6b80f9395..9daa32f76b 100644
+--- a/elf/dl-load.c
 b/elf/dl-load.c
+@@ -1461,11 +1461,15 @@ print_search_path (struct r_search_path_elem **list,
+user might want to know about this.
+ 
+If FD is not -1, then the file is already open and FD refers to it.
+-   In that case, FD is consumed for both successful and error returns.  */
++   In that case, FD is consumed for both successful and error returns.
++
++   Set *CHANGED_BY_AUDIT to true if the audit module provided a file name
++   different from NAME.  */
+ static int
+ open_verify (const char *name, int fd,
+  struct filebuf *fbp, struct link_map *loader,
+-	 int whatcode, int mode, bool *found_other_class, bool free_name)
++	 int whatcode, int mode, bool *found_other_class, bool free_name,
++	 bool *changed_by_audit)
+ {
+   /* This is the expected ELF header.  */
+ #define ELF32_CLASS ELFCLASS32
+@@ -1500,6 +1504,8 @@ open_verify (const char *name, int fd,
+   const char *errstring = NULL;
+   int errval = 0;
+ 
++  *changed_by_audit = false;
++
+ #ifdef SHARED
+   /* Give the auditing libraries a chance.  */
+   if (__glibc_unlikely (GLRO(dl_naudit) > 0) && whatcode != 0
+@@ -1521,12 +1527,19 @@ open_verify (const char *name, int fd,
+ 	  afct = afct->next;
+ 	}
+ 
+-  if (fd != -1 && name != original_name && strcmp (name, original_name))
++  if (name != original_name && strcmp (name, original_name))
+ {
+-  /* An audit library changed what we're supposed to open,
+- so FD no longer matches it.  */
+-  __close_nocancel (fd);
+-  fd = -1;
++	  /* Tell the caller we're looking at something different from
++	 ORIGINAL_NAME.  */
++	  *changed_by_audit = true;
++
++	  if (fd != -1)
++	{
++	  /* An audit library changed what we're supposed to open, so FD
++		 no longer matches it.  */
++	  __close_nocancel (fd);
++	  fd = -1;
++	}
+ }
+ }
+ #endif
+@@ -1782,6 +1795,7 @@ open_path (const char *name, size_t namelen, int mode,
+   char *edp;
+   int here_any = 0;
+   int err;
++  bool changed_by_audit;
+ 
+   /* If we are debugging the search for libraries print the path
+ 	 now if it hasn't happened now.  */
+@@ -1810,16 +1824,19 @@ open_path (const char *name, size_t namelen, int mode,
+ 	_dl_debug_printf ("  trying file=%s\n", buf);
+ 
+ 	  fd = open_verify (buf, -1, fbp, loader, whatcode, mode,
+-			found_other_class, false);
++			found_other_class, false,
++			_by_audit);
+ 	  if (this_dir->status[cnt] == unknown)
+ 	{
+ 	  if (fd != -1)
+ 		this_dir->status[cnt] = existing;
+ 	  /* Do not update the directory information when loading
+ 		 auditing code.  We must try to disturb the program as
+-		 little as possible.  */
+-	  else if (loader == NULL
+-		   || GL(dl_ns)[loader->l_ns]._ns_loaded->l_auditing == 0)
++		 little as possible.  Additionally, if the audit module
++		 change the file name, keep directory information as is.  */
++	  else if ((loader == NULL
++			|| GL(dl_ns)[loader->l_ns]._ns_loaded->l_auditing == 0)
++		   && !changed_by_audit)
+ 		{
+ 		  /* We failed to open machine dependent library.  Let's
+ 		 test whether there is any directory at all.  */
+@@ -2064,10 +2081,11 @@ _dl_map_object (struct link_map 

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Ludovic Courtès  skribis:

> Indeed, we can see ‘stat’ calls passed raw /gnu/store file names from
> RUNPATH entries (instead of /tmp/fakechroot-test/gnu/store), suggesting
> that ‘la_objsearch’ didn’t have a chance to rewrite them:

This is probably an ld.so bug:

  https://sourceware.org/bugzilla/show_bug.cgi?id=26634

Ludo’.

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Ludovic Courtès  skribis:

> $ guix describe
> Generacio 159 Sep 18 2020 08:55:27(nuna)
>   guix 98b89f4
> repository URL: https://git.savannah.gnu.org/git/guix.git
> branch: master
> commit: 98b89f432103b66efacee0bcba41a94148b8e870
> $ mkdir /tmp/fakechroot-test
> $ (cd /tmp/fakechroot-test; tar xf $(guix pack -RR openmpi -S /bin=bin -v0))
> $ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; echo /gnu/*; 
> GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/orterun'
> /gnu/*
> /tmp/fakechroot-test/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/bin//orterun:
>  error while loading shared libraries: libevent_core-2.1.so.7: cannot open 
> shared object file: No such file or directory

If we look at the LD_DEBUG=all output, we see that
/gnu/store/…libevent-2.11.1/lib has disappeared from the search path at
the point where ld.so looks for libevent_core.so:

--8<---cut here---start->8---
 15220: file=libevent_core-2.1.so.7 [0];  needed by 
/tmp/openmpi/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/bin//orterun
 [0]
 15220: find library=libevent_core-2.1.so.7 [0]; searching
 15220:  search 
path=/tmp/openmpi/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/tmp/openmpi/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib:/tmp/openmpi/gnu/store:/tmp/openmpi/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../..
  (LD_LIBRARY_PATH)
 15220:   trying 
file=/tmp/openmpi/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib/libevent_core-2.1.so.7
 15220:   trying 
file=/tmp/openmpi/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/libevent_core-2.1.so.7
 15220:   trying file=/tmp/openmpi/gnu/store/libevent_core-2.1.so.7
 15220:   trying 
file=/tmp/openmpi/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../libevent_core-2.1.so.7
 15220:  search 
path=/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/lib:/gnu/store/xk22b03jb0c0b6vkadqbkr9d4556rwxh-hwloc-2.2.0-lib/lib:/gnu/store/i1smdd4r44c5wwi03g40dvc6wfnqgdwq-libpciaccess-0.16/lib
(RUNPATH from file 
/tmp/openmpi/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/bin//orterun)
 15220:   trying 
file=/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/lib/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/xk22b03jb0c0b6vkadqbkr9d4556rwxh-hwloc-2.2.0-lib/lib/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/i1smdd4r44c5wwi03g40dvc6wfnqgdwq-libpciaccess-0.16/lib/libevent_core-2.1.so.7
 15220:  search path=   (system search path)
 15220:  search 
path=/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell/x86_64:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/x86_64:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../haswell/x86_64:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../haswell:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../x86_64:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../..
(RUNPATH from file 
/tmp/openmpi/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/bin//orterun)
 15220:   trying 
file=/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell/x86_64/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/haswell/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/x86_64/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../tls/libevent_core-2.1.so.7
 15220:   trying 
file=/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../../haswell/x86_64/libevent_core-2.1.so.7
 15220:   trying 

bug#43491: Fakechroot execution engine can fail to find libraries

[Ludovic Courtès]

Using the ‘fakechroot’ execution engine of ‘guix pack -RR’, I found a
case where an executable’s NEEDED library is not found.

--8<---cut here---start->8---
$ guix describe
Generacio 159   Sep 18 2020 08:55:27(nuna)
  guix 98b89f4
repository URL: https://git.savannah.gnu.org/git/guix.git
branch: master
commit: 98b89f432103b66efacee0bcba41a94148b8e870
$ mkdir /tmp/fakechroot-test
$ (cd /tmp/fakechroot-test; tar xf $(guix pack -RR openmpi -S /bin=bin -v0))
$ unshare -m -U -r -f sh -c 'mount -t tmpfs none /gnu; echo /gnu/*; 
GUIX_EXECUTION_ENGINE=fakechroot /tmp/fakechroot-test/bin/orterun'
/gnu/*
/tmp/fakechroot-test/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/bin//orterun:
 error while loading shared libraries: libevent_core-2.1.so.7: cannot open 
shared object file: No such file or directory
--8<---cut here---end--->8---

That library is in the RUNPATH of the original executable though:

--8<---cut here---start->8---
$ objdump -x $(guix build openmpi)/bin/orterun | grep PATH
objdump: Advertiment: 
'/gnu/store/3ign6b3i5ambjgqqiyman4mqw0wljlvl-openmpi-4.0.5-debug' és un 
directori
  RUNPATH  
/gnu/store/rm1d9z3z3iwsv779vqlg7q5z7pdk5k12-openmpi-4.0.5/lib:/gnu/store/xk22b03jb0c0b6vkadqbkr9d4556rwxh-hwloc-2.2.0-lib/lib:/gnu/store/i1smdd4r44c5wwi03g40dvc6wfnqgdwq-libpciaccess-0.16/lib:/gnu/store/c7wscymmk379v16invi8m68f6v5c8gsv-libevent-2.1.11/lib:/gnu/store/fa6wj5bxkj5ll1d7292a70knmyl7a0cr-glibc-2.31/lib:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib:/gnu/store/rykm237xkmq7rl1p0nwass01p090p88x-zlib-1.2.11/lib:/gnu/store/01b4w3m6mp55y531kyi1g8shh722kwqm-gcc-7.5.0-lib/lib/gcc/x86_64-unknown-linux-gnu/7.5.0/../../..
$ objdump -x $(guix build openmpi)/bin/orterun | grep NEED.*libevent
objdump: Advertiment: 
'/gnu/store/3ign6b3i5ambjgqqiyman4mqw0wljlvl-openmpi-4.0.5-debug' és un 
directori
  NEEDED   libevent_core-2.1.so.7
  NEEDED   libevent_pthreads-2.1.so.7
--8<---cut here---end--->8---

Ludo’.