Re: Does CDS archive generation work for crossbuilds?
On 11/5/21 9:38 AM, Magnus Ihse Bursie wrote: On 2021-10-29 21:35, Anton Kozlov wrote: QEMU user space emulation [1] works pretty fast. On my linux-x86 laptop native java -Xshare:dump completes in 0.4 sec, and with qemu-aarch64 it's about 2.3 sec. GHA now provides a runnable instruction on how to create the sysroot [2]. But that time excludes booting up the virtual machine, I presume. I think user space emulation doesn't spin up a new virtual machine. It runs on the same kernel as the build host. QEMU does the translation between the the user program and the OS (system calls, signals, etc). So the start-up overhead is pretty minimal. Nevertheless, if someone is interested in going down this road to get CDS generation for cross-compilation, I would not object to putting it in the build system. I filed https://bugs.openjdk.java.net/browse/JDK-8276791 in case there are any volunteers :-) Thanks - Ioi /Magnus Regarding problems, different page size for linux-aarch64 first comes to mind, but it should be fixed by JDK-8268396: "CDS archive with 4K alignment unusable on machines with 64k pages" Thanks, Anton [1] https://www.qemu.org/docs/master/user/main.html [2] https://github.com/openjdk/jdk/blob/master/.github/workflows/submit.yml#L514 On 10/29/21 14:35, Magnus Ihse Bursie wrote: On 2021-10-28 22:56, Ioi Lam wrote: How reliable would it be to use qemu to run the cross-compiled binaries? Has anyone tried that recently? I experimented with qemu and the JDK build some years ago (so not really recently). As I remember it, the takeaway was that it probably was reliable, but it was slow as h*ll, up to the point of being practically unusable. Add to this the fact that you need to prepare an entire OS image, complete with all tools needed to build the JDK... I set up a few such images for my own use (I think it was emulating linux-aarch64 on x64) to run testing. I had an idea of describing how I did to share the knowledge, but in the end it was just too complicated and slow to even consider recommending. /Magnus On 10/23/21 5:48 AM, Thomas Stüfe wrote: Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas -Alan
Re: Does CDS archive generation work for crossbuilds?
On 2021-10-29 21:35, Anton Kozlov wrote: QEMU user space emulation [1] works pretty fast. On my linux-x86 laptop native java -Xshare:dump completes in 0.4 sec, and with qemu-aarch64 it's about 2.3 sec. GHA now provides a runnable instruction on how to create the sysroot [2]. But that time excludes booting up the virtual machine, I presume. Nevertheless, if someone is interested in going down this road to get CDS generation for cross-compilation, I would not object to putting it in the build system. /Magnus Regarding problems, different page size for linux-aarch64 first comes to mind, but it should be fixed by JDK-8268396: "CDS archive with 4K alignment unusable on machines with 64k pages" Thanks, Anton [1] https://www.qemu.org/docs/master/user/main.html [2] https://github.com/openjdk/jdk/blob/master/.github/workflows/submit.yml#L514 On 10/29/21 14:35, Magnus Ihse Bursie wrote: On 2021-10-28 22:56, Ioi Lam wrote: How reliable would it be to use qemu to run the cross-compiled binaries? Has anyone tried that recently? I experimented with qemu and the JDK build some years ago (so not really recently). As I remember it, the takeaway was that it probably was reliable, but it was slow as h*ll, up to the point of being practically unusable. Add to this the fact that you need to prepare an entire OS image, complete with all tools needed to build the JDK... I set up a few such images for my own use (I think it was emulating linux-aarch64 on x64) to run testing. I had an idea of describing how I did to share the knowledge, but in the end it was just too complicated and slow to even consider recommending. /Magnus On 10/23/21 5:48 AM, Thomas Stüfe wrote: Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas -Alan
Re: Does CDS archive generation work for crossbuilds?
QEMU user space emulation [1] works pretty fast. On my linux-x86 laptop native java -Xshare:dump completes in 0.4 sec, and with qemu-aarch64 it's about 2.3 sec. GHA now provides a runnable instruction on how to create the sysroot [2]. Regarding problems, different page size for linux-aarch64 first comes to mind, but it should be fixed by JDK-8268396: "CDS archive with 4K alignment unusable on machines with 64k pages" Thanks, Anton [1] https://www.qemu.org/docs/master/user/main.html [2] https://github.com/openjdk/jdk/blob/master/.github/workflows/submit.yml#L514 On 10/29/21 14:35, Magnus Ihse Bursie wrote: On 2021-10-28 22:56, Ioi Lam wrote: How reliable would it be to use qemu to run the cross-compiled binaries? Has anyone tried that recently? I experimented with qemu and the JDK build some years ago (so not really recently). As I remember it, the takeaway was that it probably was reliable, but it was slow as h*ll, up to the point of being practically unusable. Add to this the fact that you need to prepare an entire OS image, complete with all tools needed to build the JDK... I set up a few such images for my own use (I think it was emulating linux-aarch64 on x64) to run testing. I had an idea of describing how I did to share the knowledge, but in the end it was just too complicated and slow to even consider recommending. /Magnus On 10/23/21 5:48 AM, Thomas Stüfe wrote: Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas -Alan
Re: Does CDS archive generation work for crossbuilds?
On 2021-10-28 22:56, Ioi Lam wrote: How reliable would it be to use qemu to run the cross-compiled binaries? Has anyone tried that recently? I experimented with qemu and the JDK build some years ago (so not really recently). As I remember it, the takeaway was that it probably was reliable, but it was slow as h*ll, up to the point of being practically unusable. Add to this the fact that you need to prepare an entire OS image, complete with all tools needed to build the JDK... I set up a few such images for my own use (I think it was emulating linux-aarch64 on x64) to run testing. I had an idea of describing how I did to share the knowledge, but in the end it was just too complicated and slow to even consider recommending. /Magnus On 10/23/21 5:48 AM, Thomas Stüfe wrote: Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas -Alan
Re: Does CDS archive generation work for crossbuilds?
How reliable would it be to use qemu to run the cross-compiled binaries? Has anyone tried that recently? On 10/23/21 5:48 AM, Thomas Stüfe wrote: Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas -Alan
Re: Does CDS archive generation work for crossbuilds?
Hi Alan, On Sat, Oct 23, 2021 at 9:58 AM Alan Bateman wrote: > > > On 23/10/2021 07:57, Thomas Stüfe wrote: > > Hi, > > > > when I crossbuild (for linux aarch64, using a devkit, building on linux > > x64), for some reason I don't > > get the classes.jsa generated inside the images directory. > > > > My configure options: > > > > > --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu > > --openjdk-target=aarch64-linux-gnu > > --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 > > > --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk > > --with-gtest=/shared/projects/openjdk/gtest/googletest > > --with-debug-level=fastdebug > > > > The build jdk is a freshly build x64 release VM from the same source > tree. > > > > Am I missing something obvious? Is CDS archive generation even supported > > for crossbuilds? > It needs the generate run-time to execute "java -Xshare:dump" so I don't > expect so. hotspot-runtime-dev is probably the place to discuss the > details. BTW: this came up recently in the context of the jlink plugin > that generates the CDS archive. The plugin needed a check to ensure that > the target platform matched the current platform as it could launch the > target VM to create the dump. > > Thinking for a second, probably it cannot work since we copy binary structures verbatim to the archive; I guess the chance that they are binary compatible between platforms is very small. But it should be easily rectified by calling Xshare:dump on the target platform. Thank you! ..Thomas > -Alan >
Re: Does CDS archive generation work for crossbuilds?
On 23/10/2021 07:57, Thomas Stüfe wrote: Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? It needs the generate run-time to execute "java -Xshare:dump" so I don't expect so. hotspot-runtime-dev is probably the place to discuss the details. BTW: this came up recently in the context of the jlink plugin that generates the CDS archive. The plugin needed a check to ensure that the target platform matched the current platform as it could launch the target VM to create the dump. -Alan
Does CDS archive generation work for crossbuilds?
Hi, when I crossbuild (for linux aarch64, using a devkit, building on linux x64), for some reason I don't get the classes.jsa generated inside the images directory. My configure options: --with-devkit=/shared/projects/openjdk/devkits/x86_64-linux-gnu-to-aarch64-linux-gnu --openjdk-target=aarch64-linux-gnu --with-boot-jdk=/shared/projects/openjdk/jdks/sapmachine17 --with-build-jdk=/shared/projects/openjdk/jdk-jdk/output-release/images/jdk --with-gtest=/shared/projects/openjdk/gtest/googletest --with-debug-level=fastdebug The build jdk is a freshly build x64 release VM from the same source tree. Am I missing something obvious? Is CDS archive generation even supported for crossbuilds? Thanks, Thomas
