bug#50151: Coreutils, aarch64 and chroot
On 8/26/21 11:14, Frans de Boer wrote: On 8/26/21 10:19, Andreas Schwab wrote: On Aug 26 2021, Frans de Boer wrote: I was not sure, but I tried the same also with x86_64 cross compiled programs using qemu-i386, which worked. Of course, they can be natively executed. Andreas. I understand now better how things are connected. Until some time ago I always used qemu-system-xxx, but have failed to understand that the qemu-xxx user emulation is only about the cpu. So, starting programs with a different coding will fail. The binfmt is the way to solve this issue. Now only find a decent explanation how to setup the /etc/binfmt.d/register for the aarch64, powerpc and maybe other architectures. At kernel.org there is an explanation, but fairly limited and thus not very helpful at this point in time. I keep on digging. Frans Ok, found the magic number list and (mostly) how to read it. Time to experiment... Frans. -- A: Yes, just like thatA: Ja, net zo Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant?
bug#50151: Coreutils, aarch64 and chroot
On 8/26/21 10:19, Andreas Schwab wrote: On Aug 26 2021, Frans de Boer wrote: I was not sure, but I tried the same also with x86_64 cross compiled programs using qemu-i386, which worked. Of course, they can be natively executed. Andreas. I understand now better how things are connected. Until some time ago I always used qemu-system-xxx, but have failed to understand that the qemu-xxx user emulation is only about the cpu. So, starting programs with a different coding will fail. The binfmt is the way to solve this issue. Now only find a decent explanation how to setup the /etc/binfmt.d/register for the aarch64, powerpc and maybe other architectures. At kernel.org there is an explanation, but fairly limited and thus not very helpful at this point in time. I keep on digging. Frans
bug#50151: Coreutils, aarch64 and chroot
On 8/26/21 01:25, Assaf Gordon wrote: tag 50151 notabug close 50151 stop On 2021-08-25 12:54 p.m., Frans de Boer wrote: On 8/25/21 10:16 AM, Assaf Gordon wrote: qemu-aarch64 -strace -L /newroot \ /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \ | tee log.txt @assaf: your suggestions no. 1 and 2, had the predicted results. Thus, suggestion no. 3 failed because of suggestion no.2. I followed then suggestion 4 and attached the strace output to this message. It seems that chroot is working as expected, only env seems to fail with an error. Not exactly: The 'chroot' system-call *seems* to succeed, followed by a failed "execve(2)" system call to execute another binary. That "execve" system fails - so it is not 'env' per-se, it is any program that will try to execute another aarch64 binary. Learning that, searching for "qemu-user", "chroot" and "architecture" leads to several web pages detailing similar errors (and few suggested solutions): https://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot https://newbedev.com/how-can-i-chroot-into-a-filesystem-with-a-different-architechture https://ownyourbits.com/2018/06/13/transparently-running-binaries-from-any-architecture-in-linux-with-qemu-and-binfmt_misc/ I hope you have some clue of what is going wrong. With the above information, we can conclude this is not a bug in coreutils - it is a limitation of the linux+qemu-user setup. So I'm closing this item and marking it as "not a bug", but discussion can continue by replying to this thread. regards, - assaf Okay, thanks anyhow for your effort. I will explore the suggested solutions next. I was not sure, but I tried the same also with x86_64 cross compiled programs using qemu-i386, which worked. So, using your conclusion too, it looks like qemu is not behaving as expected. Regards, Frans.
bug#50151: Coreutils, aarch64 and chroot
On 8/25/21 10:16 AM, Assaf Gordon wrote: Hello, On 2021-08-24 2:39 a.m., Paul Eggert wrote: However, I think it'll be a better use of our time for you to debug this one yourself. It doesn't sound like a Coreutils problem; it sounds like a problem in your virtual machine setup, and you're the best expert on that setup. Few suggestions to check, that might help you and us to troubleshoot: 1. ensure the binaries are indeed for aarch64: file /newroot/usr/sbin/chroot file /newroot/usr/bin/env file /newroot/usr/bin/bash it should say something like "ELF 64-bit LSB pie executable, ARM aarch64" for all of them. 2. ensure each binary works by itself: qemu-aarch64 -L /newroot /newroot/usr/sbin/chroot --version qemu-aarch64 -L /newroot /newroot/usr/bin/env --version qemu-aarch64 -L /newroot /newroot/usr/bin/bash --version (the actual version doesn't matter here, the main thing is that the qemu user-mode emulator was able to run the binaries.) On 2021-08-21 4:33 a.m., Frans de Boer wrote: Running 'qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /usr/bin/env> --help' does show the env help text. So, I guess chroot is to blame? Note that the above command runs your *host's* /usr/bin/env because chroot is not used - the binary under qemu (/newroot/usr/bin/bash) sees your host's file system. Observe with: qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /bin/uname -m qemu-aarch64 -L /newroot /newroot/usr/bin/env /bin/uname -m I'm guessing you will see "x86_64", not "aarch64". 3. What you should try is: qemu-aarch64 -L /newroot \ /newroot/usr/bin/bash -c /newroot/usr/bin/env --version and: qemu-aarch64 -L /newroot \ /newroot/usr/bin/env /newroot/usr/bin/bash --version In both cases, one aarch64 binary will try to execute another aach64 binary. Do these work for you, or are you seeing an error? 4. Use qemu's "-strace" to see the syscalls, hopefully that will help pinpoint the cause: qemu-aarch64 -strace -L /newroot \ /newroot/usr/sbin/chroot /newroot /usr/bin/env --version 2&1 \ | tee log.txt If the command results in an error, the "log.txt" file will show more details about what failed. If you're not familiar with 'strace' output, post it here as an email attachment. Hope this helps, - assaf P.S. On 2021-08-24 2:39 a.m., Paul Eggert wrote: A complete set of instructions for an outsider to reproduce the problem from scratch. Assume the outsider is running Fedora 34 x86-64 (since that's what I'm running :-). I'm not familiar with Fedora, but on Debian/x86_64 the following works: apt-get qemu-user apt-get install crossbuild-essential-arm64 libc6-arm64-cross cd coreutils ./configure --host=aarch64-linux-gnu make then: $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/uname -m aarch64 Somewhat related: $ qemu-aarch64 -L /usr/aarch64-linux-gnu/ ./src/env ./src/uname -m /lib/ld-linux-aarch64.so.1: No such file or directory This fails because once "inside" qemu, the aarch64 searches for "/lib/ld-linux-aarch64.so.1" but the file is in "/usr/aarch64-linux-gnu/lib/ld-linux-aarch64.so.1". One possible work-around is to build static binaries. I don't want to assume that is the culprit for Frans, so we'll wait for the logs... @assaf: your suggestions no. 1 and 2, had the predicted results. Thus, suggestion no. 3 failed because of suggestion no.2. I followed then suggestion 4 and attached the strace output to this message. It seems that chroot is working as expected, only env seems to fail with an error. I hope you have some clue of what is going wrong. --- Frans. -- A: Yes, just like thatA: Ja, net zo Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant? 1388 brk(NULL) = 0x0041b000 1388 uname(0x55007ff6b8) = 0 1388 faccessat(AT_FDCWD,"/etc/ld.so.preload",R_OK,AT_EACCESS|0x34cc8) = -1 errno=2 (No such file or directory) 1388 openat(AT_FDCWD,"/etc/ld.so.cache",O_RDONLY|O_CLOEXEC) = 3 1388 newfstatat(3,"",0x0055007fec90,0x1000) = 0 1388 mmap(NULL,78408,PROT_READ,MAP_PRIVATE,3,0) = 0x005500839000 1388 close(3) = 0 1388 openat(AT_FDCWD,"/usr/lib/tls/aarch64/atomics/libc.so.6",O_RDONLY|O_CLOEXEC) = -1 errno=2 (No such file or directory) 1388 newfstatat(AT_FDCWD,"/usr/lib/tls/aarch64/atomics",0x0055007fecc0,0) = -1 errno=2 (No such file or directory) 1388 openat(AT_FDCWD,"/usr/lib/tls/aarch64/libc.so.6",O_RDONLY|O_CLOEXEC) = -1 errno=2 (No such file or directory) 1388 newfstatat(AT_FDCWD,"/usr/lib/tls/aarch64",0x0055007fecc0,0) = -1 errno=2 (No such fi
bug#50151: Coreutils, aarch64 and chroot
On 8/21/21 6:43 PM, Paul Eggert wrote: On 8/21/21 3:33 AM, Frans de Boer wrote: So, I guess chroot is to blame? Seems unlikely that it'd be a coreutils bug. Possibly you simply have the wrong type of executable installed as /newroot/usr/sbin/chroot. If it is a coreutils bug, please give complete steps to let others reproduce the bug. If it is the wrong executable, would it ever startup and continue and (try to) load bash or env? It even displays the help message! As I said before, on its own it does function but, as opposed to bash, it does NOT load other programs. All programs are compiled the same with aarch64-cross-linux tools. No problem when operating under qemu-x86_64 or qemu-i386 - of course both compiled with their own cross compiler tools. I am currently trying this with other architectures (powerpc64) I can supply more, but what exactly would help you forward? --- Frans
bug#50151: Coreutils, aarch64 and chroot
LS, I have an issue with chroot in the next context: qemu-aarch64 -cpu cortex-a53 -L /newroot /newroot/usr/sbin/chroot /newroot \ /usr/bin/env -i \ HOME=/root \ TERM="$TERM" \ PS1='(arm chroot) \u:\w\$ ' \ PATH=/usr/bin:/usr/sbin \ /usr/bin/bash --login +h qemu-aarch64 is running on top of x86_64 linux and chroot, env and bash are compiled as aarch64 binaries. By themselves they all run under qemu-aarch64. But when using the above command, it always fails with 'Exec format error'. The same if running: qemu-aarch64 -L /newroot /newroot/usr/sbin/chroot /newroot It should run /bin/bash, but ends with the same error message. Running 'qemu-aarch64 -L /newroot /newroot/usr/bin/bash -c /usr/bin/env --help' does show the env help text. So, I guess chroot is to blame? Regards, Frans. -- A: Yes, just like thatA: Ja, net zo Q: Oh, Just like reading a book backwards Q: Oh, net als een boek achterstevoren lezen A: Because it upsets the natural flow of a story A: Omdat het de natuurlijke gang uit het verhaal haalt Q: Why is top-posting annoying? Q: Waarom is Top-posting zo irritant?
