Re: globus-net-manager FTBFS during python 3.10 transition
On 6/9/22 11:04, Mattias Ellert wrote: >>> Was this implemented? >>> Mainly I'm asking if there is any point in clicking the give-back >>> link >>> on the builds that failed with the old set-up. >> >> Sorry, not yet, I had a busy week. >> >> I’ll try to get it done this week and will let you know. >> > Any news? Not yet, sorry. It takes quite some time to get the filesystems converted and I haven't found the time for that yet. It's frustrating. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
mån 2022-05-23 klockan 09:56 +0200 skrev John Paul Adrian Glaubitz: > Hi Matthias! > > > On May 23, 2022, at 9:43 AM, Mattias Ellert > > wrote: > > > > mån 2022-05-16 klockan 10:28 +0200 skrev John Paul Adrian > > Glaubitz: > > > Hi Matthias! > > > > > > > On 5/16/22 10:24, Mattias Ellert wrote: > > > > I have noticed that in addition to m68k and sh4, the builds > > > > also > > > > fail > > > > in the same way on the hppa host known as pasta, but when given > > > > back > > > > and run on a different hppa host they succeed. > > > > > > > > E.g. > > > > https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa > > > > One failure on pasta - give back succeded on pacific > > > > > > > > https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa > > > > Two failures on pasta - third attempt on physik succeeded > > > > > > > > Is this related? > > > > > > pasta is using a similar setup like m68k and sh4. > > > > > > The underlying bug is an issue with glibc and ext4 [1] and can be > > > resolved > > > by switching the buildroot to btrfs. I have verified that this > > > fixes > > > the > > > issue. > > > > > > I will do that later this week when I find the time. > > > > > > Adrian > > > > > > > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 > > > > > > > Was this implemented? > > Mainly I'm asking if there is any point in clicking the give-back > > link > > on the builds that failed with the old set-up. > > Sorry, not yet, I had a busy week. > > I’ll try to get it done this week and will let you know. > > Adrian Hi. Any news? Mattias signature.asc Description: This is a digitally signed message part
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Matthias! > On May 23, 2022, at 9:43 AM, Mattias Ellert > wrote: > > mån 2022-05-16 klockan 10:28 +0200 skrev John Paul Adrian Glaubitz: >> Hi Matthias! >> >>> On 5/16/22 10:24, Mattias Ellert wrote: >>> I have noticed that in addition to m68k and sh4, the builds also >>> fail >>> in the same way on the hppa host known as pasta, but when given >>> back >>> and run on a different hppa host they succeed. >>> >>> E.g. >>> https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa >>> One failure on pasta - give back succeded on pacific >>> >>> https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa >>> Two failures on pasta - third attempt on physik succeeded >>> >>> Is this related? >> >> pasta is using a similar setup like m68k and sh4. >> >> The underlying bug is an issue with glibc and ext4 [1] and can be >> resolved >> by switching the buildroot to btrfs. I have verified that this fixes >> the >> issue. >> >> I will do that later this week when I find the time. >> >> Adrian >> >>> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 >> > > Was this implemented? > Mainly I'm asking if there is any point in clicking the give-back link > on the builds that failed with the old set-up. Sorry, not yet, I had a busy week. I’ll try to get it done this week and will let you know. Adrian
Re: globus-net-manager FTBFS during python 3.10 transition
mån 2022-05-16 klockan 10:28 +0200 skrev John Paul Adrian Glaubitz: > Hi Matthias! > > On 5/16/22 10:24, Mattias Ellert wrote: > > I have noticed that in addition to m68k and sh4, the builds also > > fail > > in the same way on the hppa host known as pasta, but when given > > back > > and run on a different hppa host they succeed. > > > > E.g. > > https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa > > One failure on pasta - give back succeded on pacific > > > > https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa > > Two failures on pasta - third attempt on physik succeeded > > > > Is this related? > > pasta is using a similar setup like m68k and sh4. > > The underlying bug is an issue with glibc and ext4 [1] and can be > resolved > by switching the buildroot to btrfs. I have verified that this fixes > the > issue. > > I will do that later this week when I find the time. > > Adrian > > > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 > Was this implemented? Mainly I'm asking if there is any point in clicking the give-back link on the builds that failed with the old set-up. Mattias signature.asc Description: This is a digitally signed message part
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Rob, On Mon, May 16, 2022 at 2:51 PM Rob Landley wrote: > QEMU's sh4 system emulation only has 64 megs ram, but you can add as much swap > space as you like and the compiler's been happy with it so far. (Even when it > gets a little thrashy, it stays in the host's page cache so it's all > memory-to-memory copying. I have a todo item to map some more memory into the > virtual board and kernel, but both sides involve a lot of digging.) The m68k > board I'm using has 256 megs ram, that's generally enough for a -j1 build. (Or > even -j3 with distcc calling out to the cross compiler running on the host's > loopback instead of running in the emulator.) I haven't tried to get an hppa > system working yet because musl doesn't support it. With Linux v5.19, you will be able to use the new virtual m68k machine with 3.2 GiB of RAM. https://lore.kernel.org/all/20220406201523.243733-1-laur...@vivier.eu Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- ge...@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds
Re: globus-net-manager FTBFS during python 3.10 transition
On 5/16/22 03:28, John Paul Adrian Glaubitz wrote: > Hi Matthias! > > On 5/16/22 10:24, Mattias Ellert wrote: >> I have noticed that in addition to m68k and sh4, the builds also fail >> in the same way on the hppa host known as pasta, but when given back >> and run on a different hppa host they succeed. >> >> E.g. >> https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa >> One failure on pasta - give back succeded on pacific >> >> https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa >> Two failures on pasta - third attempt on physik succeeded >> >> Is this related? > > pasta is using a similar setup like m68k and sh4. > > The underlying bug is an issue with glibc and ext4 [1] and can be resolved > by switching the buildroot to btrfs. I have verified that this fixes the > issue. I've generally found qemu's application emulation to be MUCH more brittle than its system emulation, so I usually create little system emulation versions to native compile under instead. (Currently https://landley.net/toybox/downloads/binaries/mkroot/latest/ ala https://landley.net/toybox/faq.html#mkroot previously https://landley.net/aboriginal/downloads/binaries/ ala https://landley.net/aboriginal/about.html .) QEMU's sh4 system emulation only has 64 megs ram, but you can add as much swap space as you like and the compiler's been happy with it so far. (Even when it gets a little thrashy, it stays in the host's page cache so it's all memory-to-memory copying. I have a todo item to map some more memory into the virtual board and kernel, but both sides involve a lot of digging.) The m68k board I'm using has 256 megs ram, that's generally enough for a -j1 build. (Or even -j3 with distcc calling out to the cross compiler running on the host's loopback instead of running in the emulator.) I haven't tried to get an hppa system working yet because musl doesn't support it. That said, I've never figured out how to do a fully from-source debootstrap. (I.E. build a Debian root filesystem from source under a Linux From Scratch system and use that to populate a binary package repository.) Everything always assumes "fetch binary packages from an existing repository to populate a chroot", which means the architecture and libc combination has to already exist. I'd really like to try doing a musl version of debian, but every documented process I can find starts with "have this script wget a bunch of glibc binaries off a server". Is there documentation on actually doing it entirely from source somewhere? Rob
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Matthias! On 5/16/22 10:24, Mattias Ellert wrote: > I have noticed that in addition to m68k and sh4, the builds also fail > in the same way on the hppa host known as pasta, but when given back > and run on a different hppa host they succeed. > > E.g. > https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa > One failure on pasta - give back succeded on pacific > > https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa > Two failures on pasta - third attempt on physik succeeded > > Is this related? pasta is using a similar setup like m68k and sh4. The underlying bug is an issue with glibc and ext4 [1] and can be resolved by switching the buildroot to btrfs. I have verified that this fixes the issue. I will do that later this week when I find the time. Adrian > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Hi! Now the openssl3 transition is underway, and many more of the globus packages fail in the same way. My original idea that it was failing due to doxygen using an input filter, seems to be incorrect. Only 4 of the globus packages uses an input filter, an many more of them are failing than those ones. I did not try all the failed builds on mitchy.debian.net (the m68k porterbox), but I did try 3 of them and the builds succeeded there. I have noticed that in addition to m68k and sh4, the builds also fail in the same way on the hppa host known as pasta, but when given back and run on a different hppa host they succeed. E.g. https://buildd.debian.org/status/logs.php?pkg=globus-common=hppa One failure on pasta - give back succeded on pacific https://buildd.debian.org/status/logs.php?pkg=globus-gass-transfer=hppa Two failures on pasta - third attempt on physik succeeded Is this related? The logs look OK, the configure scripts find /usr/bin/doxygen, and there in a call to /usr/bin/doxygen in the log. There are no errors reported in the log from the doxygen call, but the files that the call is supposed ro produce are not there later during the install phase. rm -rf doc /usr/bin/doxygen touch doc-stamp Mattias tor 2022-05-12 klockan 18:10 +0200 skrev Laurent Vivier: > Le 12/05/2022 à 17:51, John Paul Adrian Glaubitz a écrit : > > On 5/12/22 17:45, Laurent Vivier wrote: > > > No, the 'F' means 'fix-binary': > > > > > > The interpreter is loaded in memory once when the binfmt_misc is > > > configured (when > > > the configuration is written to > > > /proc/sys/fs/binfmt_misc/register) from the host > > > filesystem. So you don't need to put it in the chroot. > > > > So, the flags are correct then? Is there any way to properly test > > the argument issue? > > > > Adrian > > > > The flags are correct. > > This can be tested with: > > $ sudo chroot m68k-chroot sh -c 'echo $0' > sh > > if the argument is not properly managed, you would have > "/usr/bin/sh". > > Thanks, > Laurent smime.p7s Description: S/MIME cryptographic signature
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Laurent! On 5/12/22 18:22, Laurent Vivier wrote: >> I guess it's related to the glibc issue then [1]. > > You can try by mounting your chroot on a non ext4 partition. It works well > with btrfs. > > Ext4 stores a hash in a field that is normally an index, so a 64bit (host > kernel long) hash cannot be passed to the 32bit guest. > > btrfs uses it as a real index, so it will overflow only with 2³² entries. Yes, you are absolutely right. Using btrfs for the chroot fixes the problem. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Le 12/05/2022 à 18:16, John Paul Adrian Glaubitz a écrit : Hi! Hi :) On 5/12/22 18:10, Laurent Vivier wrote: The flags are correct. This can be tested with: $ sudo chroot m68k-chroot sh -c 'echo $0' sh if the argument is not properly managed, you would have "/usr/bin/sh". OK, this is working properly: root@z6:/srv/chroot> chroot unstable-m68k-sbuild/ sh -c 'echo $0' sh root@z6:/srv/chroot> I guess it's related to the glibc issue then [1]. You can try by mounting your chroot on a non ext4 partition. It works well with btrfs. Ext4 stores a hash in a field that is normally an index, so a 64bit (host kernel long) hash cannot be passed to the 32bit guest. btrfs uses it as a real index, so it will overflow only with 2³² entries. Thanks, Laurent Adrian [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960
Re: globus-net-manager FTBFS during python 3.10 transition
Hi! On 5/12/22 18:10, Laurent Vivier wrote: > The flags are correct. > > This can be tested with: > > $ sudo chroot m68k-chroot sh -c 'echo $0' > sh > > if the argument is not properly managed, you would have "/usr/bin/sh". OK, this is working properly: root@z6:/srv/chroot> chroot unstable-m68k-sbuild/ sh -c 'echo $0' sh root@z6:/srv/chroot> I guess it's related to the glibc issue then [1]. Adrian > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Le 12/05/2022 à 17:51, John Paul Adrian Glaubitz a écrit : On 5/12/22 17:45, Laurent Vivier wrote: No, the 'F' means 'fix-binary': The interpreter is loaded in memory once when the binfmt_misc is configured (when the configuration is written to /proc/sys/fs/binfmt_misc/register) from the host filesystem. So you don't need to put it in the chroot. So, the flags are correct then? Is there any way to properly test the argument issue? Adrian The flags are correct. This can be tested with: $ sudo chroot m68k-chroot sh -c 'echo $0' sh if the argument is not properly managed, you would have "/usr/bin/sh". Thanks, Laurent
Re: globus-net-manager FTBFS during python 3.10 transition
On 5/12/22 17:45, Laurent Vivier wrote: > No, the 'F' means 'fix-binary': > > The interpreter is loaded in memory once when the binfmt_misc is configured > (when > the configuration is written to /proc/sys/fs/binfmt_misc/register) from the > host > filesystem. So you don't need to put it in the chroot. So, the flags are correct then? Is there any way to properly test the argument issue? Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Le 12/05/2022 à 17:41, John Paul Adrian Glaubitz a écrit : Hi Laurent! The issue can be reproduced on a freshly installed Debian system. Is the problem the additional "F" flag? glaubitz@z6:/tmp/test> cat /proc/sys/fs/binfmt_misc/qemu-m68k enabled interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P flags: POCF offset 0 magic 7f454c46010201020004 mask ff00fffe glaubitz@z6:/tmp/test> Adrian No, the 'F' means 'fix-binary': The interpreter is loaded in memory once when the binfmt_misc is configured (when the configuration is written to /proc/sys/fs/binfmt_misc/register) from the host filesystem. So you don't need to put it in the chroot. Thanks, Laurent
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Laurent! The issue can be reproduced on a freshly installed Debian system. Is the problem the additional "F" flag? glaubitz@z6:/tmp/test> cat /proc/sys/fs/binfmt_misc/qemu-m68k enabled interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P flags: POCF offset 0 magic 7f454c46010201020004 mask ff00fffe glaubitz@z6:/tmp/test> Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Laurent! On 5/10/22 23:48, Laurent Vivier wrote: > Did you configure binfmt_misc to preserve argv0 [2] ? > > $ cat /proc/sys/fs/binfmt_misc/qemu-m68k > enabled > interpreter //qemu-m68k > flags: POC > offset 0 > magic 7f454c46010201020004 > mask fe00fffe Looks like this for me: root@nofan:~> cat /proc/sys/fs/binfmt_misc/qemu-m68k enabled interpreter /usr/libexec/qemu-binfmt/m68k-binfmt-P flags: POCF offset 0 magic 7f454c46010201020004 mask ff00fffe root@nofan:~> Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Le 10/05/2022 à 21:23, John Paul Adrian Glaubitz a écrit : Hi! On 4/27/22 11:19, John Paul Adrian Glaubitz wrote: On 4/27/22 11:04, Laurent Vivier wrote: Normally, since qemu v6.0.0 you can preserve argv[0]: (...) You need also kernel v5.12: OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU buildds now. Let's see if that helps. Tested with QEMU 7.0 and glibc patches from [1], still fails. Need to keep digging. Did you configure binfmt_misc to preserve argv0 [2] ? $ cat /proc/sys/fs/binfmt_misc/qemu-m68k enabled interpreter //qemu-m68k flags: POC offset 0 magic 7f454c46010201020004 mask fe00fffe Thanks, Laurent Adrian [1] https://github.com/zatrazz/glibc/tree/azanella/redir-refactor [2] ``P`` - preserve-argv[0] Legacy behavior of binfmt_misc is to overwrite the original argv[0] with the full path to the binary. When this flag is included, binfmt_misc will add an argument to the argument vector for this purpose, thus preserving the original ``argv[0]``. e.g. If your interp is set to ``/bin/foo`` and you run ``blah``
Re: globus-net-manager FTBFS during python 3.10 transition
Hi! On 4/27/22 11:19, John Paul Adrian Glaubitz wrote: > On 4/27/22 11:04, Laurent Vivier wrote: >> Normally, since qemu v6.0.0 you can preserve argv[0]: >> (...) >> You need also kernel v5.12: > > OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU > buildds now. Let's see if that helps. Tested with QEMU 7.0 and glibc patches from [1], still fails. Need to keep digging. Adrian > [1] https://github.com/zatrazz/glibc/tree/azanella/redir-refactor -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Hi Laurent! On 4/27/22 11:04, Laurent Vivier wrote: > Normally, since qemu v6.0.0 you can preserve argv[0]: > (...) > You need also kernel v5.12: OK, I'm installing QEMU 6.2 and kernel 5.16 from backports on all QEMU buildds now. Let's see if that helps. Adrian -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913
Re: globus-net-manager FTBFS during python 3.10 transition
Le 26/04/2022 à 21:24, John Paul Adrian Glaubitz a écrit : Hi! ... [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004 Normally, since qemu v6.0.0 you can preserve argv[0]: commit 6e1c0d7b951e19c53b8467e8bc4b71ee73a394ea Author: Laurent Vivier Date: Mon Feb 22 11:50:04 2021 +0100 linux-user: manage binfmt-misc preserve-arg[0] flag Add --preserve-argv0 in qemu-binfmt-conf.sh to configure the preserve-argv0 flag. This patch allows to use new flag in AT_FLAGS to detect if preserve-argv0 is configured for this interpreter: argv[0] (the full pathname provided by binfmt-misc) is removed and replaced by argv[1] (the original argv[0] provided by binfmt-misc when 'P'/preserve-arg[0] is set) For instance with this patch and kernel support for AT_FLAGS: $ sudo chroot m68k-chroot sh -c 'echo $0' sh without this patch: $ sudo chroot m68k-chroot sh -c 'echo $0' /usr/bin/sh The new flag is available in kernel (v5.12) since: 2347961b11d4 ("binfmt_misc: pass binfmt_misc flags to the interpreter") This can be tested with something like: # cp /qemu-ppc /chroot/powerpc/jessie # qemu-binfmt-conf.sh --qemu-path / --systemd ppc --credential yes \ --persistent no --preserve-argv0 yes # systemctl restart systemd-binfmt.service # cat /proc/sys/fs/binfmt_misc/qemu-ppc enabled interpreter //qemu-ppc flags: POC offset 0 magic 7f454c46010201020014 mask ff00fffe # chroot /chroot/powerpc/jessie sh -c 'echo $0' sh # qemu-binfmt-conf.sh --qemu-path / --systemd ppc --credential yes \ --persistent no --preserve-argv0 no # systemctl restart systemd-binfmt.service # cat /proc/sys/fs/binfmt_misc/qemu-ppc enabled interpreter //qemu-ppc flags: OC offset 0 magic 7f454c46010201020014 mask ff00fffe # chroot /chroot/powerpc/jessie sh -c 'echo $0' /bin/sh Signed-off-by: Laurent Vivier Message-Id: <20210222105004.1642234-1-laur...@vivier.eu> You need also kernel v5.12: commit 2347961b11d4079deace3c81dceed460c08a8fc1 Author: Laurent Vivier Date: Tue Jan 28 14:25:39 2020 +0100 binfmt_misc: pass binfmt_misc flags to the interpreter It can be useful to the interpreter to know which flags are in use. For instance, knowing if the preserve-argv[0] is in use would allow to skip the pathname argument. This patch uses an unused auxiliary vector, AT_FLAGS, to add a flag to inform interpreter if the preserve-argv[0] is enabled. Note by Helge Deller: The real-world user of this patch is qemu-user, which needs to know if it has to preserve the argv[0]. See Debian bug #970460. Signed-off-by: Laurent Vivier Reviewed-by: YunQiang Su URL: http://bugs.debian.org/970460 Signed-off-by: Helge Deller Thanks, Laurent
Re: globus-net-manager FTBFS during python 3.10 transition
Hi! On 4/21/22 09:25, Mattias Ellert wrote: > Doxygen uses popen to run filters. Is there an issue with using popen > to run perl scripts on the buildds for m68k and sh4? > > The reason I ask is that I found this report in BTS: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004 > which seems to be a similar issue. > Popen will fork and exec the perl script as: > > /bin/sh -c "./perlscript.pl arg" > > where ./perlscript.pl is a text file with a #!/usr/bin/perl shebang. It's most likely related to emulation issues with qmeu-user. Either because of the related glibc bug [1] or because of the bug you mentioned #974004 [2]. I will rebuild the glibc package with the preliminary fix for [1] that has not been merged upstream yet, unfortunately. Adrian > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=23960 > [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=974004 -- .''`. John Paul Adrian Glaubitz : :' : Debian Developer `. `' Physicist `-GPG: 62FF 8A75 84E0 2956 9546 0006 7426 3B37 F5B5 F913