Re: bug#48941: [powerpc64le-linux] libfaketime CLOCK_MONOTONIC test hangs

[Chris Marusich]

Thiago Jung Bauermann  writes:

> GDB uses the shell to launch the debugged program. That is probably where
> ‘/bin/sh’ is entering the picture here. I don’t know whether that has any 
> relation to your foreign distro’s libc being used.
>
> The output of `help run` in GDB mentions that the shell is specified by the 
> ‘$SHELL’ environment variable. Perhaps you have that set?
>
> One way to see if this is the problem is to use the GDB command
> `set startup-with-shell off` to make it launch the debugged program without 
> the shell.

Thank you for this suggestion.  When I ran "set startup-with-shell off",
gdb successfully launched the program.

It seems something in the Guix-built software is trying to run /bin/sh.
Perhaps there is stray hard-coded path in GDB or something that needs to
be fixed...  In any case, I'm glad I can continue debugging!

>> The x86-64 systems I have that run pure Guix don't have any
>> /lib*/ directories. You might try running gdb with
>> LD_LIBRARY_PATH=/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/l
>> ib to have the Guix libc.so picked up before the other one. HTH
>
> Another alternative worth trying is the ‘--container’ option to ‘guix 
> environment’, to completely isolate GDB from the foreign distro. You might 
> want to add the coreutils package to the ‘--ad-hoc’ list so that you can 
> get amenities such as an ls command. :-)

When I tried "--container", gdb successfully launched the program.  So
it's another viable work-around for the gdb issue.

As for the actual libfaketime bug, when it hung I pressed Control+C, and
here's the backtrace at that point:

--8<---cut here---start->8---
(gdb) backtrace
#0  0x77eccba0 in __futex_abstimed_wait_common64 
(futex_word=0x77aaf1c0, expected=, clockid=, 
abstime=0x0, private=, cancel=) at 
../sysdeps/nptl/futex-internal.c:74
#1  0x77eb9934 in __pthread_clockjoin_ex (threadid=140737348563184, 
thread_return=0x7fffe2d0, clockid=, abstime=, 
block=) at pthread_join_common.c:102
#2  0x77eb9684 in __pthread_join (threadid=, 
thread_return=) at pthread_join.c:24
#3  0x10001784 in main (argc=1, argv=0x7fffe718) at timetest.c:146
--8<---cut here---end--->8---

Here's the entire session in which I built the problematic test and
debugged it, including a full back trace with local variables:

--8<---cut here---start->8---
[0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0
$ ~/guix-core-updates/pre-inst-env guix environment libfaketime --ad-hoc 
gcc-toolchain@10.3.0:debug gdb
[0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0
$ cd source/
[0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0/source
$ make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix clean
make  -C src clean
make[1]: Entering directory '/tmp/guix-build-libfaketime-0.9.9.drv-0/source/src'
make[1]: Leaving directory '/tmp/guix-build-libfaketime-0.9.9.drv-0/source/src'
make  -C test clean
make[1]: Entering directory 
'/tmp/guix-build-libfaketime-0.9.9.drv-0/source/test'
make[1]: Leaving directory '/tmp/guix-build-libfaketime-0.9.9.drv-0/source/test'
[0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0/source
$ make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix
make  -C src all
make[1]: Entering directory '/tmp/guix-build-libfaketime-0.9.9.drv-0/source/src'
gcc -o libfaketime.o -c -std=gnu99 -Wall -Wextra -Werror -Wno-nonnull-compare 
-DFAKE_PTHREAD -DFAKE_STAT -DFAKE_UTIME -DFAKE_SLEEP -DFAKE_TIMERS 
-DFAKE_INTERNAL_CALLS -fPIC 
-DPREFIX='"'/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix'"' 
-DLIBDIRNAME='"'/lib/faketime'"' -DFORCE_MONOTONIC_FIX -g   libfaketime.c
gcc -o libfaketime.so.1 -Wl,-soname,libfaketime.so.1  -lpthread 
-Wl,--version-script=libfaketime.map -shared libfaketime.o -ldl -lm -lrt
gcc -o libfaketimeMT.o -c -std=gnu99 -Wall -Wextra -Werror -Wno-nonnull-compare 
-DFAKE_PTHREAD -DFAKE_STAT -DFAKE_UTIME -DFAKE_SLEEP -DFAKE_TIMERS 
-DFAKE_INTERNAL_CALLS -fPIC 
-DPREFIX='"'/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix'"' 
-DLIBDIRNAME='"'/lib/faketime'"' -DFORCE_MONOTONIC_FIX -g  
-DPTHREAD_SINGLETHREADED_TIME libfaketime.c
gcc -o libfaketimeMT.so.1 -Wl,-soname,libfaketimeMT.so.1  -lpthread 
-Wl,--version-script=libfaketime.map -shared libfaketimeMT.o -ldl -lm -lrt
gcc -o faketime -std=gnu99 -Wall -Wextra -Werror -Wno-nonnull-compare 
-DFAKE_PTHREAD -DFAKE_STAT -DFAKE_UTIME -DFAKE_SLEEP -DFAKE_TIMERS 
-DFAKE_INTERNAL_CALLS -fPIC 
-DPREFIX='"'/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix'"' 
-DLIBDIRNAME='"'/lib/faketime'"' -DFORCE_MONOTONIC_FIX -g   faketime.c  
-lpthread -Wl,--version-script=libfaketime.map -lrt
make[1]: Leaving directory '/tmp/guix-build-libfaketime-0.9.9.drv-0/source/src'
[0] [env] 

Re: bug#48941: [powerpc64le-linux] libfaketime CLOCK_MONOTONIC test hangs

[Chris Marusich]

Hi Kaelyn,

Thank you for the reply.

Kaelyn  writes:

> Are you using Guix on a foreign distro? This line looks like your
> distro's normal libc.so was being used and it was from glibc-2.31 or
> older. The x86-64 systems I have that run pure Guix don't have any
> /lib*/ directories. You might try running gdb with
> LD_LIBRARY_PATH=/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib
> to have the Guix libc.so picked up before the other one. HTH

Yes, I'm using Guix on a foreign distro.  It isn't clear to me what is
trying to access the /lib directories.  That's what I'm trying to figure
out.  In a --pure environment using only Guix-built tools, one would
expect that I would not have to set LD_LIBRARY_PATH.  However, if I
can't figure it out, that's another trick I can try.

-- 
Chris


signature.asc
Description: PGP signature

Re: bug#48941: [powerpc64le-linux] libfaketime CLOCK_MONOTONIC test hangs

[Thiago Jung Bauermann]

Hello,

Em quarta-feira, 21 de julho de 2021, às 11:31:25 -03, Kaelyn escreveu:
> Hi,
> 
> ‐‐‐ Original Message ‐‐‐
> 
> On Wednesday, July 21st, 2021 at 1:08 AM, Chris Marusich 
 wrote:
> > Now run it:
> > 
> > --8<---cut here---start->8---
> > 
> > (gdb) run
> > 
> > Starting program:
> > /tmp/guix-build-libfaketime-0.9.9.drv-0/source/test/timetest
> > 
> > /bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.33' not
> > found (required by ../src/libfaketime.so.1) /bin/sh:
> > /lib/powerpc64le-linux-gnu/libc.so.6: version` GLIBC_2.32' not found
> > (required by
> > /gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/libpthread.
> > so.0)
> Are you using Guix on a foreign distro? This line looks like your
> distro's normal libc.so was being used and it was from glibc-2.31 or
> older.

GDB uses the shell to launch the debugged program. That is probably where
‘/bin/sh’ is entering the picture here. I don’t know whether that has any 
relation to your foreign distro’s libc being used.

The output of `help run` in GDB mentions that the shell is specified by the 
‘$SHELL’ environment variable. Perhaps you have that set?

One way to see if this is the problem is to use the GDB command
`set startup-with-shell off` to make it launch the debugged program without 
the shell.

> The x86-64 systems I have that run pure Guix don't have any
> /lib*/ directories. You might try running gdb with
> LD_LIBRARY_PATH=/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/l
> ib to have the Guix libc.so picked up before the other one. HTH

Another alternative worth trying is the ‘--container’ option to ‘guix 
environment’, to completely isolate GDB from the foreign distro. You might 
want to add the coreutils package to the ‘--ad-hoc’ list so that you can 
get amenities such as an ls command. :-)

-- 
Thanks,
Thiago

Re: bug#48941: [powerpc64le-linux] libfaketime CLOCK_MONOTONIC test hangs

[Kaelyn]

Hi,

‐‐‐ Original Message ‐‐‐

On Wednesday, July 21st, 2021 at 1:08 AM, Chris Marusich  
wrote:

> Hi,
>
> I need a little help figuring out how to use gdb in Guix for bug 48941:
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48941
>
> Here's the situation. A libfaketime test hangs forever. Upstream
>
> suggested I debug it. I'm trying to, but gdb errors out. What am I
>
> doing wrong? It's probably something simple, but I can't see what.
>
> I'll describe what I've done. First, I started a build like so:
>
> ./pre-inst-env guix build --keep-failed libfaketime
>
> While the problematic test hung, I found the PID of the test and killed
>
> it. This caused the build to fail, leaving the build environment for me
>
> to play around in.
>
> I entered a pure environment that contains all the things I need to
>
> debug the test (gcc 10.3.0 is currently the default gcc on
>
> core-updates):
>
> ./pre-inst-env guix environment --pure libfaketime --ad-hoc 
> gcc-toolchain@10.3.0 gcc-toolchain@10.3.0:debug gdb
>
> In the pure environment, I confirmed I can build and run the hanging
>
> test via the following commands (I added -g in order to get debug
>
> symbols):
>
> make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
> PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix
>
> make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
> PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix test
>
> OK, so I can trigger the hang. Great! Next step, fire up GDB:
>
> --8<---cut here---start->8---
>
> [0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0/source/test
>
> $ gdb ./timetest
>
> GNU gdb (GDB) 10.2
>
> Copyright (C) 2021 Free Software Foundation, Inc.
>
> License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
>
> This is free software: you are free to change and redistribute it.
>
> There is NO WARRANTY, to the extent permitted by law.
>
> Type "show copying" and "show warranty" for details.
>
> This GDB was configured as "powerpc64le-unknown-linux-gnu".
>
> Type "show configuration" for configuration details.
>
> For bug reporting instructions, please see:
>
> https://www.gnu.org/software/gdb/bugs/.
>
> Find the GDB manual and other documentation resources online at:
>
> http://www.gnu.org/software/gdb/documentation/.
>
> For help, type "help".
>
> Type "apropos word" to search for commands related to "word"...
>
> Reading symbols from ./timetest...
>
> (gdb)
>
> --8<---cut here---end--->8---
>
> The debug symbols provided by gcc-toolchain@10.3.0:debug are under
>
> $GUIX_ENVIRONMENT/lib/debug. This is the value of GUIX_ENVIRONMENT:
>
> --8<---cut here---start->8---
>
> $ echo $GUIX_ENVIRONMENT
>
> /gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile
>
> --8<---cut here---end--->8---
>
> By the way, this directory corresponds to glibc 2.33:
>
> --8<---cut here---start->8---
>
> $ realpath /gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile/lib/debug
>
> /gnu/store/8akrlhc25d7xvi85gzvginw0vdi4zyg4-glibc-2.33-debug/lib/debug
>
> --8<---cut here---end--->8---
>
> Let's tell GDB where to find those debug symbols:
>
> (gdb) set debug-file-directory 
> /gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile/lib/debug
>
> Let's also tell GDB to set the environment variables that upstream
>
> recommended when running the test program:
>
> --8<---cut here---start->8---
>
> (gdb) set environment LD_PRELOAD=../src/libfaketime.so.1
>
> (gdb) set environment FAKETIME=-10d
>
> (gdb) set environment NO_FAKE_STAT=1
>
> --8<---cut here---end--->8---
>
> Now run it:
>
> --8<---cut here---start->8---
>
> (gdb) run
>
> Starting program: /tmp/guix-build-libfaketime-0.9.9.drv-0/source/test/timetest
>
> /bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.33' not found 
> (required by ../src/libfaketime.so.1) /bin/sh: 
> /lib/powerpc64le-linux-gnu/libc.so.6: version` GLIBC_2.32' not found 
> (required by 
> /gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/libpthread.so.0)

Are you using Guix on a foreign distro? This line looks like your distro's 
normal libc.so was being used and it was from glibc-2.31 or older. The x86-64 
systems I have that run pure Guix don't have any /lib*/ directories. You might 
try running gdb with 
LD_LIBRARY_PATH=/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib to 
have the Guix libc.so picked up before the other one. HTH

Cheers,
Kaelyn

>
> /bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.32' not found 
> (required by 
> /gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/libdl.so.2) 
> /bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version` GLIBC_2.32' not found 
> 

Re: bug#48941: [powerpc64le-linux] libfaketime CLOCK_MONOTONIC test hangs

[Chris Marusich]

Hi,

I need a little help figuring out how to use gdb in Guix for bug 48941:

  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=48941

Here's the situation.  A libfaketime test hangs forever.  Upstream
suggested I debug it.  I'm trying to, but gdb errors out.  What am I
doing wrong?  It's probably something simple, but I can't see what.

I'll describe what I've done.  First, I started a build like so:

  ./pre-inst-env guix build --keep-failed libfaketime

While the problematic test hung, I found the PID of the test and killed
it.  This caused the build to fail, leaving the build environment for me
to play around in.

I entered a pure environment that contains all the things I need to
debug the test (gcc 10.3.0 is currently the default gcc on
core-updates):

  ./pre-inst-env guix environment --pure libfaketime --ad-hoc 
gcc-toolchain@10.3.0 gcc-toolchain@10.3.0:debug gdb

In the pure environment, I confirmed I can build and run the hanging
test via the following commands (I added -g in order to get debug
symbols):

  make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix
  make FAKETIME_COMPILE_CFLAGS='-DFORCE_MONOTONIC_FIX -g' CC=gcc 
PREFIX=/tmp/guix-build-libfaketime-0.9.9.drv-0/myprefix test

OK, so I can trigger the hang.  Great!  Next step, fire up GDB:

--8<---cut here---start->8---
[0] [env] marusich@suzaku:/tmp/guix-build-libfaketime-0.9.9.drv-0/source/test
$ gdb ./timetest
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
.
Find the GDB manual and other documentation resources online at:
.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./timetest...
(gdb) 
--8<---cut here---end--->8---

The debug symbols provided by gcc-toolchain@10.3.0:debug are under
$GUIX_ENVIRONMENT/lib/debug.  This is the value of GUIX_ENVIRONMENT:

--8<---cut here---start->8---
$ echo $GUIX_ENVIRONMENT
/gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile
--8<---cut here---end--->8---

By the way, this directory corresponds to glibc 2.33:

--8<---cut here---start->8---
$ realpath /gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile/lib/debug
/gnu/store/8akrlhc25d7xvi85gzvginw0vdi4zyg4-glibc-2.33-debug/lib/debug
--8<---cut here---end--->8---

Let's tell GDB where to find those debug symbols:

  (gdb) set debug-file-directory 
/gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile/lib/debug

Let's also tell GDB to set the environment variables that upstream
recommended when running the test program:

--8<---cut here---start->8---
(gdb) set environment LD_PRELOAD=../src/libfaketime.so.1
(gdb) set environment FAKETIME=-10d
(gdb) set environment NO_FAKE_STAT=1
--8<---cut here---end--->8---

Now run it:

--8<---cut here---start->8---
(gdb) run
Starting program: /tmp/guix-build-libfaketime-0.9.9.drv-0/source/test/timetest 
/bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.33' not found 
(required by ../src/libfaketime.so.1)
/bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.32' not found 
(required by 
/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/libpthread.so.0)
/bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.32' not found 
(required by 
/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/libdl.so.2)
/bin/sh: /lib/powerpc64le-linux-gnu/libc.so.6: version `GLIBC_2.32' not found 
(required by 
/gnu/store/kmblbljiygayhlc5gb02an9imhy90ws9-glibc-2.33/lib/librt.so.1)
During startup program exited with code 1.
(gdb) 
--8<---cut here---end--->8---

Huh?  What happened?  I've double checked that I'm using gdb provided by
Guix:

--8<---cut here---start->8---
$ type -P gdb
/gnu/store/32fjhp30k34fh0g9f1gmgcj8pc5wldq6-profile/bin/gdb
--8<---cut here---end--->8---

I also tried running gdb by invoking it via that absolute file name, and
it still errored out in the same way.

I'm operating in a --pure environment.  All the tools are provided by
Guix.  I'm surprised that /bin/sh and /lib are even mentioned above.

If anyone can