Re: Cross-compile worked, cross-install not so much ...
On Apr 25, 2022, at 3:26 PM, Patrick M. Hausen wrote: > Hi, > >> Am 25.04.2022 um 21:18 schrieb Brooks Davis : >> Cross install is not supported. As you have seen, certain tools are >> bootstrapped on the build host and used during the install process. You >> might be able to get away with nuking >> /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then >> running `make toolchain` to build native versions of those tools. > > that comes as a big surprise and disappointment. What is the point of > cross-compiling, then? > How to update a small slow embedded platform? At least as far back as July 2016 I kept my FreeBSD/arm systems updated by cross-building on a FreeBSD/amd64 system and using PkgBase to update the OS on the FreeBSD/arm systems: https://lists.freebsd.org/pipermail/freebsd-arm/2016-July/01.html This worked well for me, at least until I stopped running FreeBSD on my ARM systems. I don't know whether it still works. Cheers, Paul.
Re: Cross-compile worked, cross-install not so much ...
On Mon, Apr 25, 2022 at 09:26:04PM +0200, Patrick M. Hausen wrote: > Hi, > > > Am 25.04.2022 um 21:18 schrieb Brooks Davis : > > Cross install is not supported. As you have seen, certain tools are > > bootstrapped on the build host and used during the install process. You > > might be able to get away with nuking > > /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then > > running `make toolchain` to build native versions of those tools. > > that comes as a big surprise and disappointment. What is the point of > cross-compiling, then? > How to update a small slow embedded platform? The traditional answer is: build a new image and install it as a whole-sale replacement for the device image or in a nanobsd system with two boot partitions. If someone was motivated they could presumably work things out so we could cross compile the boostrap tools, but that would be a fair bit of work. It would probably be easier to figure out how to process the METALOG file in a sysroot to extract the files you want to update and install them in the right order (the order mostly doesn't matter until it really does and everything explodes as you crossthread rtld/libc/libthr). If I had time, this is the path I would pursue. > I tried your suggestion - unfortunately no worky: > > cd /usr/src/tools/build; make DIRPRFX=tools/build/ > DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks > Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin > cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: > Operation not supported > *** Error code 1 You might try editing tools/build/Makefile and dropping -p from _COPY_HOST_TOOL or just tweaking the code to take the !FreeBSD branch. I'm not fully convinced that cp -p should fail if file flags can't be manipulated, but apparently it does. -- Brooks signature.asc Description: PGP signature
Re: Cross-compile worked, cross-install not so much ...
You can : - setenv DESTDIR /wherever make -jWhatever TARGET_ARCH=aarch64 -DNO_ROOT DESTDIR=$DESTDIR installworld make -jWhatever TARGET_ARCH=aarch64 -DNO_ROOT DESTDIR=$DESTDIR distribution make -jWhatever TARGET_ARCH=aarch64 -DNO_ROOT DESTDIR=$DESTDIR installkernel - Patrick M. Hausen wrote: > What are typical > build times on a CM3+? depends on how long you sleep at night if yo want it finished after wakeup ;-), .. on the Pi4 approx. 8 to 9 hours w/o fine-tuning but with powerd configured . you can immediately PXE-boot your -DNO_ROOT from a pxe-configured server(e.g. where your DESTDIR lives) if you’re not in the mood of waiting so long. K.
Re: Cross-compile worked, cross-install not so much ...
On Mon, Apr 25, 2022 at 1:26 PM Patrick M. Hausen wrote: > Hi, > > > Am 25.04.2022 um 21:18 schrieb Brooks Davis : > > Cross install is not supported. As you have seen, certain tools are > > bootstrapped on the build host and used during the install process. You > > might be able to get away with nuking > > /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then > > running `make toolchain` to build native versions of those tools. > > that comes as a big surprise and disappointment. What is the point of > cross-compiling, then? > How to update a small slow embedded platform? > Cross installing is supported. Native installing of a cross build world isn't. When I've had to do this in the past, I've just mounted the embedded system on my beefy server under /mumble (allowing root on the embedded system for NFS) and did a sudo make installworld ... DESTDIR=/mumble. Though I've forgotten the gymnastics to do etcupdate/mergemaster this way. Warner > I tried your suggestion - unfortunately no worky: > > cd /usr/src/tools/build; make DIRPRFX=tools/build/ > DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks > Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin > cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: > Operation not supported > *** Error code 1 > > So I will probably need to checkout and compile on the Pi. What are typical > build times on a CM3+? Plus I am going to wear down the builtin eMMC much > faster. > > Kind regards and thanks, > Patrick >
Re: Cross-compile worked, cross-install not so much ...
On Mon, Apr 25, 2022 at 10:26 PM Patrick M. Hausen wrote: > Hi, > > > Am 25.04.2022 um 21:18 schrieb Brooks Davis : > > Cross install is not supported. As you have seen, certain tools are > > bootstrapped on the build host and used during the install process. You > > might be able to get away with nuking > > /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then > > running `make toolchain` to build native versions of those tools. > > that comes as a big surprise and disappointment. What is the point of > cross-compiling, then? > How to update a small slow embedded platform? > > You can cross compile a program and then use it on a related system . Please think this is a contribution . In that way , piece by piece you may construct another system . You are right : Being able to construct an installable system is a good idea . When it is not available as a whole , having a partial capability is a good step . Mehmet Erol Sanliturk > I tried your suggestion - unfortunately no worky: > > cd /usr/src/tools/build; make DIRPRFX=tools/build/ > DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks > Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin > cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: > Operation not supported > *** Error code 1 > > So I will probably need to checkout and compile on the Pi. What are typical > build times on a CM3+? Plus I am going to wear down the builtin eMMC much > faster. > > Kind regards and thanks, > Patrick >
Re: Cross-compile worked, cross-install not so much ...
Hi, > Am 25.04.2022 um 21:18 schrieb Brooks Davis : > Cross install is not supported. As you have seen, certain tools are > bootstrapped on the build host and used during the install process. You > might be able to get away with nuking > /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then > running `make toolchain` to build native versions of those tools. that comes as a big surprise and disappointment. What is the point of cross-compiling, then? How to update a small slow embedded platform? I tried your suggestion - unfortunately no worky: cd /usr/src/tools/build; make DIRPRFX=tools/build/ DESTDIR=/usr/obj/usr/src/arm64.aarch64/tmp/legacy host-symlinks Linking host tools into /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin cp: chflags: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin/basename: Operation not supported *** Error code 1 So I will probably need to checkout and compile on the Pi. What are typical build times on a CM3+? Plus I am going to wear down the builtin eMMC much faster. Kind regards and thanks, Patrick
Re: Cross-compile worked, cross-install not so much ...
On Mon, Apr 25, 2022 at 09:07:47PM +0200, Patrick M. Hausen wrote: > Hi all, > > getting into FreeBSD ARM64 I tried to compile a current system on a fast > AMD64 VM and > now I am somewhat stuck - no help using search engines and the like. > > 1st step: > > # checkout main aka 14-CURRENT > cd /usr/src > make -j 8 TARGET=arm64 TARGET_ARCH=aarch64 buildworld buildkernel > > That worked like a charm and I seem to have a well populated > /usr/obj/usr/src/arm64.aarch64. > > 2nd step: > > I then mounted /usr/src and /usr/obj on my Raspberry Pi via NFS. The binaries > in /usr/obj are the correct architecture and run perfectly well: > > root@pi8:~ # file /usr/obj/usr/src/arm64.aarch64/bin/sh/sh > /usr/obj/usr/src/arm64.aarch64/bin/sh/sh: ELF 64-bit LSB pie > executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter > /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, not stripped > root@pi8:~ # /usr/obj/usr/src/arm64.aarch64/bin/sh/sh > root@pi8:~ # ^D > > 3rd step: > > root@pi8:/usr/src # make TARGET=arm64 TARGET_ARCH=aarch64 installkernel > -- > >>> Install check kernel > -- > -- > >>> Installing kernel GENERIC on Mon Apr 25 21:03:58 CEST 2022 > -- > cd /usr/obj/usr/src/arm64.aarch64/sys/GENERIC; MACHINE_ARCH=aarch64 > MACHINE=arm64 CPUTYPE= CC="cc -target aarch64-unknown-freebsd14.0 > --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp > -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CXX="c++ -target > aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp > -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CPP="cpp -target > aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp > -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" AS="as" AR="ar" > ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib > STRINGS= SIZE="size" STRIPBIN="strip" > PATH=/usr/obj/usr/src/arm64.aarch64/tmp/bin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin > make KERNEL=kernel install > /bin/sh: make: Exec format error > *** Error code 126 > > Stop. > make[1]: stopped in /usr/src > *** Error code 1 > > Stop. > make: stopped in /usr/src > > So via that complicated PATH setting it seems to run: > /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make > > Which surprisingly is an amd64 binary: > > root@pi8:/usr/src # file > /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make > /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make: ELF 64-bit LSB > pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter > /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, stripped > > > So what did I do wrong? Cross install is not supported. As you have seen, certain tools are bootstrapped on the build host and used during the install process. You might be able to get away with nuking /usr/obj/usr/src/arm64.aarch64/tmp/legacy (or maybe tmp) and then running `make toolchain` to build native versions of those tools. -- Brooks signature.asc Description: PGP signature
Cross-compile worked, cross-install not so much ...
Hi all, getting into FreeBSD ARM64 I tried to compile a current system on a fast AMD64 VM and now I am somewhat stuck - no help using search engines and the like. 1st step: # checkout main aka 14-CURRENT cd /usr/src make -j 8 TARGET=arm64 TARGET_ARCH=aarch64 buildworld buildkernel That worked like a charm and I seem to have a well populated /usr/obj/usr/src/arm64.aarch64. 2nd step: I then mounted /usr/src and /usr/obj on my Raspberry Pi via NFS. The binaries in /usr/obj are the correct architecture and run perfectly well: root@pi8:~ # file /usr/obj/usr/src/arm64.aarch64/bin/sh/sh /usr/obj/usr/src/arm64.aarch64/bin/sh/sh: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, not stripped root@pi8:~ # /usr/obj/usr/src/arm64.aarch64/bin/sh/sh root@pi8:~ # ^D 3rd step: root@pi8:/usr/src # make TARGET=arm64 TARGET_ARCH=aarch64 installkernel -- >>> Install check kernel -- -- >>> Installing kernel GENERIC on Mon Apr 25 21:03:58 CEST 2022 -- cd /usr/obj/usr/src/arm64.aarch64/sys/GENERIC; MACHINE_ARCH=aarch64 MACHINE=arm64 CPUTYPE= CC="cc -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CXX="c++ -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" CPP="cpp -target aarch64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/arm64.aarch64/tmp -B/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin" AS="as" AR="ar" ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" PATH=/usr/obj/usr/src/arm64.aarch64/tmp/bin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/sbin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/bin:/usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin make KERNEL=kernel install /bin/sh: make: Exec format error *** Error code 126 Stop. make[1]: stopped in /usr/src *** Error code 1 Stop. make: stopped in /usr/src So via that complicated PATH setting it seems to run: /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make Which surprisingly is an amd64 binary: root@pi8:/usr/src # file /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make /usr/obj/usr/src/arm64.aarch64/tmp/legacy/usr/bin/make: ELF 64-bit LSB pie executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 14.0 (1400057), FreeBSD-style, stripped So what did I do wrong? Thanks for all help, Patrick
Re: nullfs and ZFS issues
On Mon, 2022-04-25 at 15:27 +0200, Alexander Leidinger wrote: > Quoting Alexander Leidinger (from Sun, 24 > Apr 2022 19:58:17 +0200): > > > Quoting Alexander Leidinger (from Fri, 22 > > Apr 2022 09:04:39 +0200): > > > > > Quoting Doug Ambrisko (from Thu, 21 Apr > > > 2022 09:38:35 -0700): > > > > > > I've attached mount.patch that when doing mount -v should > > > > show the vnode usage per filesystem. Note that the problem I was > > > > running into was after some operations arc_prune and arc_evict would > > > > consume 100% of 2 cores and make ZFS really slow. If you are not > > > > running into that issue then nocache etc. shouldn't be needed. > > > > > > I don't run into this issue, but I have a huge perf difference when > > > using nocache in the nightly periodic runs. 4h instead of 12-24h > > > (22 jails on this system). > > > > > > > On my laptop I set ARC to 1G since I don't use swap and in the past > > > > ARC would consume to much memory and things would die. When the > > > > nullfs holds a bunch of vnodes then ZFS couldn't release them. > > > > > > > > FYI, on my laptop with nocache and limited vnodes I haven't run > > > > into this problem. I haven't tried the patch to let ZFS free > > > > it's and nullfs vnodes on my laptop. I have only tried it via > > > > > > I have this patch and your mount patch installed now, without > > > nocache and reduced arc reclaim settings (100, 1). I will check the > > > runtime for the next 2 days. > > > > 9-10h runtime with the above settings (compared to 4h with nocache > > and 12-24h without any patch and without nocache). > > I changed the sysctls back to the defaults and will see in the next > > run (in 7h) what the result is with just the patches. > > And again 9-10h runtime (I've seen a lot of the find processes in the > periodic daily run of those 22 jails in the state "*vnode"). Seems > nocache gives the best perf for me in this case. Sorry for jumping in here - I've got a couple of questions: - Will this also apply to nullfs read-only mounts? Or is it only in case of writing "through" a nullfs mount that these problems are seen? - Is it a problem also in 13, or is this "new" in -CURRENT? We're having weird and unexplained CPU spikes on several systems, even after tuning geli to not use gazillions of threads. So far our suspicion has been ZFS snapshot cleanups but this is an interesting contender - unless the whole "read only" part makes it moot. /Eirik
Re: nullfs and ZFS issues
Quoting Alexander Leidinger (from Sun, 24 Apr 2022 19:58:17 +0200): Quoting Alexander Leidinger (from Fri, 22 Apr 2022 09:04:39 +0200): Quoting Doug Ambrisko (from Thu, 21 Apr 2022 09:38:35 -0700): I've attached mount.patch that when doing mount -v should show the vnode usage per filesystem. Note that the problem I was running into was after some operations arc_prune and arc_evict would consume 100% of 2 cores and make ZFS really slow. If you are not running into that issue then nocache etc. shouldn't be needed. I don't run into this issue, but I have a huge perf difference when using nocache in the nightly periodic runs. 4h instead of 12-24h (22 jails on this system). On my laptop I set ARC to 1G since I don't use swap and in the past ARC would consume to much memory and things would die. When the nullfs holds a bunch of vnodes then ZFS couldn't release them. FYI, on my laptop with nocache and limited vnodes I haven't run into this problem. I haven't tried the patch to let ZFS free it's and nullfs vnodes on my laptop. I have only tried it via I have this patch and your mount patch installed now, without nocache and reduced arc reclaim settings (100, 1). I will check the runtime for the next 2 days. 9-10h runtime with the above settings (compared to 4h with nocache and 12-24h without any patch and without nocache). I changed the sysctls back to the defaults and will see in the next run (in 7h) what the result is with just the patches. And again 9-10h runtime (I've seen a lot of the find processes in the periodic daily run of those 22 jails in the state "*vnode"). Seems nocache gives the best perf for me in this case. Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.orgnetch...@freebsd.org : PGP 0x8F31830F9F2772BF pgpbbgb6PWKOs.pgp Description: Digitale PGP-Signatur
