Re: Preliminary review for new WINENV support
fixpath.sh`. Logs are here: https://gist.github.com/lewurm/2fbf2971a55a3a4a4f5933b105925976 = ## Epilog = I hope this has met your bar of exotic setup. Unfortunately I can't provide much feedback on the automagic. I would love to see your work landing eventually! Thank you, -Bernhard ____ From: build-dev on behalf of Magnus Ihse Bursie Sent: Friday, August 28, 2020 11:09 To: Ludovic Henry; Yasumasa Suenaga; Monica Beckwith; build-dev Subject: Re: Preliminary review for new WINENV support On 2020-08-24 19:21, Ludovic Henry wrote: Hi Magnus, I want to follow up on this work of yours, as we've particular interest in it for the Windows-AArch64 port. Let me know how I could assist you in this effort. The big blockers for this fix is basically these two issues: I have also temporarily disabled the javac server, and building without absolute paths. I believe a better way forward is to address these with improved functionality in separate patches, instead of trying to work around the issues introduced by them in this patch. Until that is fixed, any integration of this patch is a no-go. I have been working on those, but a lot of other stuff has kept me from gaining much progress; everything from long vacations to preparations to the switch to github. Unfortunately, I don't think there's an easy way for you to help on those two blockers. However, there are two more things that I'd like to look into, if possible, before integrating my patch, and there you might be able to help. One is performance. I got a somewhat different performance profile with my patch than without -- some things were faster, and some were slower. I'd like to understand why some things got slower, and if it is possible to do anything about. (Why things got faster I don't care about, I'm just happy :)) So maybe you can help me with some performance analysis? With the limited availability of tooling for makefiles, this is not really trivial. What I usually do is I run with JOBS=1 (to be able to track each task individually without effects of concurrency), and LOG=info,profile. This will give you time stamps for each individual execution in build.log. And see if there is any interesting differences with and without the patch. The other thing is that WSL2 works awfully bad. :( Maybe there's nothing to be done about it, if the thing being shipped is just too buggy, but maybe we do things incorrectly, or that unnecessarily triggers bugs. Or maybe there are workarounds we could use. But I think it could be worth some more effort to try to get it working. Maybe you can help with this? /Magnus
Re: Preliminary review for new WINENV support
build arm64 == Same problem with permissions. I must be doing something wrong. ``` $ sudo bash configure --with-devkit="/mnt/c/lewurm/VS2019-16.6.1-devkit" --with-boot-jdk="/mnt/c/lewurm/jdk-16-ea+19-windows-aarch64" --disable-precompiled-headers [...] Configuration summary: * Debug level:release * HS debug level: product * JVM variants: server * JVM features: server: 'cds compiler1 compiler2 epsilongc g1gc jfr jni-check jvmti management nmt parallelgc serialgc services shenandoahgc vm-structs zgc' * OpenJDK target: OS: windows, CPU architecture: aarch64, address length: 64 * Version string: 16-internal+0-adhoc.root.openjdk-jdk (16-internal) Tools summary: * Environment:wsl1 version 4.4.0-19607-Microsoft, #1000-Microsoft Wed Apr 09 15:02:00 PST 2020 (Ubuntu 20.04 LTS); windows version 10.0.19607.1000; prefix "/mnt"; root "[unavailable]" * Boot JDK: openjdk version "16-ea" 2020-10-09 OpenJDK Runtime Environment Microsoft (build 16-ea+19-windows-aarch64) OpenJDK 64-Bit Server VM Microsoft (build 16-ea+19-windows-aarch64, mixed mode) (at /mnt/c/lewurm/jdk-16~1) * Toolchain: microsoft (Microsoft Visual Studio 2019 16.6.1 (devkit)) * C Compiler: Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe) * C++ Compiler: Version 19.27.29112 (at /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64/cl.exe) Build performance summary: * Cores to use: 64 * Memory limit: 524204 MB ``` So a couple of things that I hacked up: * I manually copied x86 MSVC binaries into /mnt/c/lewurm/vs2019-16.6.1-devkit/vc/bin/arm64 * I manually copied arm64 binaries from the Windows SDK into /mnt/c/lewurm/vs2019-16.6.1-devkit/10/bin/arm64 Eventually I need to update the devkit script to create a proper archive and/or update the build script to be aware that x86 MSVC binaries must be used on arm64 (the hope is that this eventually won't be true anymore). * add detection for WSL on aarch64: https://github.com/lewurm/openjdk/commit/4da4b6df16d05c97b7f48bef166e6d2080d37511 * The boot-jdk is from https://github.com/microsoft/openjdk-aarch64/releases * `--disable-precompiled-headers` is needed because otherwise a bug in the binary translator (xtajit) is triggered when running x86 cl.exe with it. Note to myself: File a bug report. Let's build a jdk image: ``` $ sudo make jdk-image JOBS=64 LOG=info [...] /mnt/c/lewurm/openjdk-jdk/make/scripts/fixpath.sh: line 405: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/bin/java: No such file or directory make[3]: *** [ExplodedImageOptimize.gmk:41: /mnt/c/lewurm/openjdk-jdk/build/windows-aarch64-server-release/jdk/_optimize_image_exec.marker] Error 127 make[2]: *** [make/Main.gmk:437: exploded-image-optimize] Error 2 $ ./build/windows-aarch64-server-release/jdk/bin/java.exe -version openjdk version "16-internal" 2021-03-16 OpenJDK Runtime Environment (build 16-internal+0-adhoc.root.openjdk-jdk) OpenJDK 64-Bit Server VM (build 16-internal+0-adhoc.root.openjdk-jdk, mixed mode) ``` So same issue as in scenario 3 (WSL1 on x86_64 building x86_64), which is _great_ news :-) == ## Performance == I did a comparision run on WSL1 on x86_64 to build x86_64 (scenario 3), once with the WINENV patches applied and once without it. Overall it seems there is some performance regression, but not too bad. The profile logs aren't very meaningful to me, but if I would need to guess the difference is due to having a native `fixpath.exe` vs. `fixpath.sh`. Logs are here: https://gist.github.com/lewurm/2fbf2971a55a3a4a4f5933b105925976 = ## Epilog = I hope this has met your bar of exotic setup. Unfortunately I can't provide much feedback on the automagic. I would love to see your work landing eventually! Thank you, -Bernhard ____ From: build-dev on behalf of Magnus Ihse Bursie Sent: Friday, August 28, 2020 11:09 To: Ludovic Henry; Yasumasa Suenaga; Monica Beckwith; build-dev Subject: Re: Preliminary review for new WINENV support On 2020-08-24 19:21, Ludovic Henry wrote: > Hi Magnus, > > I want to follow up on this work of yours, as we've particular interest in it > for the Windows-AArch64 port. > > Let me know how I could assist you in this effort. The big blockers for this fix is basically these two issues: >> I have also temporarily disabled the javac server, and building without >> absolute paths. I believe a better way forward is to address these with >> improved functionality in separate patches, instead of trying to work around >> the issues introduced by them in this patch. Until that is fixed, any integration of this patch is a no-go. I have been working on those, but a lot of other stuff has kept me from gaining much progress; everything from long vacations to preparations to the switch to git
Re: Preliminary review for new WINENV support
On 2020-08-24 19:21, Ludovic Henry wrote: Hi Magnus, I want to follow up on this work of yours, as we've particular interest in it for the Windows-AArch64 port. Let me know how I could assist you in this effort. The big blockers for this fix is basically these two issues: I have also temporarily disabled the javac server, and building without absolute paths. I believe a better way forward is to address these with improved functionality in separate patches, instead of trying to work around the issues introduced by them in this patch. Until that is fixed, any integration of this patch is a no-go. I have been working on those, but a lot of other stuff has kept me from gaining much progress; everything from long vacations to preparations to the switch to github. Unfortunately, I don't think there's an easy way for you to help on those two blockers. However, there are two more things that I'd like to look into, if possible, before integrating my patch, and there you might be able to help. One is performance. I got a somewhat different performance profile with my patch than without -- some things were faster, and some were slower. I'd like to understand why some things got slower, and if it is possible to do anything about. (Why things got faster I don't care about, I'm just happy :)) So maybe you can help me with some performance analysis? With the limited availability of tooling for makefiles, this is not really trivial. What I usually do is I run with JOBS=1 (to be able to track each task individually without effects of concurrency), and LOG=info,profile. This will give you time stamps for each individual execution in build.log. And see if there is any interesting differences with and without the patch. The other thing is that WSL2 works awfully bad. :( Maybe there's nothing to be done about it, if the thing being shipped is just too buggy, but maybe we do things incorrectly, or that unnecessarily triggers bugs. Or maybe there are workarounds we could use. But I think it could be worth some more effort to try to get it working. Maybe you can help with this? /Magnus
RE: Preliminary review for new WINENV support
Hi Magnus, I want to follow up on this work of yours, as we've particular interest in it for the Windows-AArch64 port. Let me know how I could assist you in this effort. Thank you, -- Ludovic -Original Message- From: build-dev On Behalf Of Yasumasa Suenaga Sent: Wednesday, July 8, 2020 9:47 PM To: Derek Keeler ; Monica Beckwith ; Magnus Ihse Bursie ; build-dev Subject: Re: Preliminary review for new WINENV support On 2020/07/09 12:12, Derek Keeler wrote: > After applying Suenaga's patch, I see this error: > > checking wsl distribution... > /home/dekeeler/dev/openjdk.net/sandbox/build/.configure-support/generated-configure.sh: > line 32952: -d: command not found > > I'm running Debian 10 in my WSL2 instance and it does not include > `lsb_release` by default. Installed it with `sudo apt-get install > lsb-release`. It's better if we add AC_CHECK_PROG for lsb_release for Windows (for just WSL?) > The next trouble I ran into is that I only have Visual Studio 2019 installed. > I tried to make it work but lack the necessary experience to wrangle this > code. Will start again in the morning after installing VS2017. FYI: I can run configure script with VS2019 community edition on Ubuntu 20.04 on both WSL 1 and 2, however I could not complete building OpenJDK image on WSL 2 due to I/O performance issue. Thanks, Yasumasa > -Derek > > > From: build-dev on behalf of Derek Keeler > > Sent: July 8, 2020 5:53 PM > To: Monica Beckwith ; Magnus Ihse Bursie > ; build-dev > Subject: Re: Preliminary review for new WINENV support > > This is fantastic! > > I'm synching to this branch tonight and will start building it locally on my > WSL2 environment. I'll send questions/comments to this thread unless there is > a better place? > > (Note: our internal email system may obfuscate the links below in future > replies, my apologies). > > -Derek > > > From: build-dev on behalf of Monica > Beckwith > Sent: July 5, 2020 11:34 AM > To: Magnus Ihse Bursie ; build-dev > > Subject: Re: Preliminary review for new WINENV support > > Wow! Impressive work, @Magnus Ihse > Bursie<mailto:magnus.ihse.bur...@oracle.com> . When working with > cross-compilation mods for aarch64-win port, I realized we needed to clean > up/better support the quirks around fixpath and also I was hoping to expand > the support to wsl2, etc. From what I am reading, you seem to have touched > all of these and more! I am mostly offline for the next two days, but will > start testing your changes when I am back online. Thank you for doing the > needful. > > Monica > > Get Outlook for Android<https://aka.ms/ghei36> > > ____________ > From: build-dev on behalf of Magnus Ihse > Bursie > Sent: Saturday, July 4, 2020 7:08:20 PM > To: build-dev > Subject: Preliminary review for new WINENV support > > I've been working for some time on a complete rewrite of how we handle the > pecularities of the Windows build environment. This new solution supports > Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed > different "winenvs". > > One of the main design goals has been to minimize the difference for the > configure script and make files for the different winenvs, and leverage as > much shared code between them as possible. Another has been to try to collect > all the "trickiness" needed in as few places as possible, ideally just one, > instead of having it spread out all over the configure script. A third design > goal has been to prepare for cross-compilation for Windows from Linux, using > Wine. > > It pretty soon turned out that I needed to get a better grip on how we detect > tools in configure, so a complete overhaul of this is included in the change. > Now we have more or less fully parted with the original autoconf functions, > which has long been too limited for us, and now finally has reached their end > of usefulness. > > At this point, I have a prototype / preview that basically works, but has > some limitations. > > I'd like to ask anyone interested in building OpenJDK on Windows to take the > patch for a spin. Especially if you have an esoteric or exotic setup! > > Webrev: > https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2Fdata=02%7C01%7Cluhenry%40microsoft.com%7C55d5bddc82f142c2312c08d823c33111%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298668546029073sdata=k7tQ6%2BfwdrD8AGryNZdqAD%2Fk3Lnc%2B%2B75opEaWASISo0%3Dreserved=0 > > (If you prefer, you can check out the branch "ihse-winenv-bran
Re: Preliminary review for new WINENV support
After applying Suenaga's patch, I see this error: checking wsl distribution... /home/dekeeler/dev/openjdk.net/sandbox/build/.configure-support/generated-configure.sh: line 32952: -d: command not found I'm running Debian 10 in my WSL2 instance and it does not include `lsb_release` by default. Installed it with `sudo apt-get install lsb-release`. The next trouble I ran into is that I only have Visual Studio 2019 installed. I tried to make it work but lack the necessary experience to wrangle this code. Will start again in the morning after installing VS2017. -Derek From: build-dev on behalf of Derek Keeler Sent: July 8, 2020 5:53 PM To: Monica Beckwith ; Magnus Ihse Bursie ; build-dev Subject: Re: Preliminary review for new WINENV support This is fantastic! I'm synching to this branch tonight and will start building it locally on my WSL2 environment. I'll send questions/comments to this thread unless there is a better place? (Note: our internal email system may obfuscate the links below in future replies, my apologies). -Derek From: build-dev on behalf of Monica Beckwith Sent: July 5, 2020 11:34 AM To: Magnus Ihse Bursie ; build-dev Subject: Re: Preliminary review for new WINENV support Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com> . When working with cross-compilation mods for aarch64-win port, I realized we needed to clean up/better support the quirks around fixpath and also I was hoping to expand the support to wsl2, etc. From what I am reading, you seem to have touched all of these and more! I am mostly offline for the next two days, but will start testing your changes when I am back online. Thank you for doing the needful. Monica Get Outlook for Android<https://aka.ms/ghei36> From: build-dev on behalf of Magnus Ihse Bursie Sent: Saturday, July 4, 2020 7:08:20 PM To: build-dev Subject: Preliminary review for new WINENV support I've been working for some time on a complete rewrite of how we handle the pecularities of the Windows build environment. This new solution supports Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed different "winenvs". One of the main design goals has been to minimize the difference for the configure script and make files for the different winenvs, and leverage as much shared code between them as possible. Another has been to try to collect all the "trickiness" needed in as few places as possible, ideally just one, instead of having it spread out all over the configure script. A third design goal has been to prepare for cross-compilation for Windows from Linux, using Wine. It pretty soon turned out that I needed to get a better grip on how we detect tools in configure, so a complete overhaul of this is included in the change. Now we have more or less fully parted with the original autoconf functions, which has long been too limited for us, and now finally has reached their end of usefulness. At this point, I have a prototype / preview that basically works, but has some limitations. I'd like to ask anyone interested in building OpenJDK on Windows to take the patch for a spin. Especially if you have an esoteric or exotic setup! Webrev: https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2Fdata=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355sdata=LkTMXBZRcaOe%2FRCvcckFViwDOysi7nS9H1Edve%2B4Alk%3Dreserved=0 (If you prefer, you can check out the branch "ihse-winenv-branch" on https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandboxdata=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355sdata=g8UElT7Zx15fMItd8yGGjS%2BuPcyKd3i1EEVHJLumfUo%3Dreserved=0 instead of downloading the patch from the webrev.) I am leaving on vacation next week, so I won't be doing any more work on this for a while, but I promise to read all emails when I get back and try to rectify all issues that are reported. This means you have some time to try it out, too. Here are some technical notes of what is changing, compared to the current Windows build. The native "fixpath.exe" tool is gone. This means that we do not need to compile it during configure, which was always tricky to get right (mostly since we did not have fixpath in place to help us...). Instead, there is a new fixpath.sh shell script, that does the same job, and more. The script is written in highly optimized shell code (yeah, I see the oxymoron) so only bash internal functionality is used, to avoid calling external tools, which is expensive on Windows. This makes the performance pr
Re: Preliminary review for new WINENV support
This is fantastic!
I'm synching to this branch tonight and will start building it locally on my
WSL2 environment. I'll send questions/comments to this thread unless there is a
better place?
(Note: our internal email system may obfuscate the links below in future
replies, my apologies).
-Derek
From: build-dev on behalf of Monica Beckwith
Sent: July 5, 2020 11:34 AM
To: Magnus Ihse Bursie ; build-dev
Subject: Re: Preliminary review for new WINENV support
Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
. When working with cross-compilation mods for aarch64-win port, I realized we
needed to clean up/better support the quirks around fixpath and also I was
hoping to expand the support to wsl2, etc. From what I am reading, you seem to
have touched all of these and more! I am mostly offline for the next two days,
but will start testing your changes when I am back online. Thank you for doing
the needful.
Monica
Get Outlook for Android<https://aka.ms/ghei36>
From: build-dev on behalf of Magnus Ihse
Bursie
Sent: Saturday, July 4, 2020 7:08:20 PM
To: build-dev
Subject: Preliminary review for new WINENV support
I've been working for some time on a complete rewrite of how we handle the
pecularities of the Windows build environment. This new solution supports
Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed
different "winenvs".
One of the main design goals has been to minimize the difference for the
configure script and make files for the different winenvs, and leverage as much
shared code between them as possible. Another has been to try to collect all
the "trickiness" needed in as few places as possible, ideally just one, instead
of having it spread out all over the configure script. A third design goal has
been to prepare for cross-compilation for Windows from Linux, using Wine.
It pretty soon turned out that I needed to get a better grip on how we detect
tools in configure, so a complete overhaul of this is included in the change.
Now we have more or less fully parted with the original autoconf functions,
which has long been too limited for us, and now finally has reached their end
of usefulness.
At this point, I have a prototype / preview that basically works, but has some
limitations.
I'd like to ask anyone interested in building OpenJDK on Windows to take the
patch for a spin. Especially if you have an esoteric or exotic setup!
Webrev: http://cr.openjdk.java.net/~ihse/winenv-preview-1/webrev.01/
(If you prefer, you can check out the branch "ihse-winenv-branch" on
http://hg.openjdk.java.net/jdk/sandbox instead of downloading the patch from
the webrev.)
I am leaving on vacation next week, so I won't be doing any more work on this
for a while, but I promise to read all emails when I get back and try to
rectify all issues that are reported. This means you have some time to try it
out, too.
Here are some technical notes of what is changing, compared to the current
Windows build.
The native "fixpath.exe" tool is gone. This means that we do not need to
compile it during configure, which was always tricky to get right (mostly since
we did not have fixpath in place to help us...).
Instead, there is a new fixpath.sh shell script, that does the same job, and
more. The script is written in highly optimized shell code (yeah, I see the
oxymoron) so only bash internal functionality is used, to avoid calling
external tools, which is expensive on Windows. This makes the performance
practically roughly at par with the native fixpath.exe.
Fixpath also has a "print" and "import" mode, apart from the traditional"exec"
mode. This makes it possible to use the same tool for converting individual
paths at runtime to Windows style ("print"), and it takes the logic needed to
"import" paths given by the user to configure, into a format usable internally
by the build system, and moves it into a centralized location in the fixpath
script.
A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, ENVROOT
and WINTEMP. For instance, for "cygwin", these are:
PATHTOOL=cygpath
DRIVEPREFIX=/cygdrive (typically)
ENVROOT=C:\Cygwin64 (typically)
WINTEMP=/tmp
These are needed for fixpath to do it's magic. Fixpath can auto-detect those,
but to save on execution time they are normally detected by configure and sent
as arguments to fixpath.
Detection of the Visual Studio environment has been massively simplified. Using
fixpath, conversion between Windows and unix paths is not so complex anymore.
The bridge Windows batch file that is needed to extract the environment
variables is no longer created on the fly, but is instead stored in
make/scripts/extract-vs-env.cmd. This is called with fixpath, so all arg
Re: Preliminary review for new WINENV support
On 2020/07/09 12:12, Derek Keeler wrote: After applying Suenaga's patch, I see this error: checking wsl distribution... /home/dekeeler/dev/openjdk.net/sandbox/build/.configure-support/generated-configure.sh: line 32952: -d: command not found I'm running Debian 10 in my WSL2 instance and it does not include `lsb_release` by default. Installed it with `sudo apt-get install lsb-release`. It's better if we add AC_CHECK_PROG for lsb_release for Windows (for just WSL?) The next trouble I ran into is that I only have Visual Studio 2019 installed. I tried to make it work but lack the necessary experience to wrangle this code. Will start again in the morning after installing VS2017. FYI: I can run configure script with VS2019 community edition on Ubuntu 20.04 on both WSL 1 and 2, however I could not complete building OpenJDK image on WSL 2 due to I/O performance issue. Thanks, Yasumasa -Derek From: build-dev on behalf of Derek Keeler Sent: July 8, 2020 5:53 PM To: Monica Beckwith ; Magnus Ihse Bursie ; build-dev Subject: Re: Preliminary review for new WINENV support This is fantastic! I'm synching to this branch tonight and will start building it locally on my WSL2 environment. I'll send questions/comments to this thread unless there is a better place? (Note: our internal email system may obfuscate the links below in future replies, my apologies). -Derek From: build-dev on behalf of Monica Beckwith Sent: July 5, 2020 11:34 AM To: Magnus Ihse Bursie ; build-dev Subject: Re: Preliminary review for new WINENV support Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com> . When working with cross-compilation mods for aarch64-win port, I realized we needed to clean up/better support the quirks around fixpath and also I was hoping to expand the support to wsl2, etc. From what I am reading, you seem to have touched all of these and more! I am mostly offline for the next two days, but will start testing your changes when I am back online. Thank you for doing the needful. Monica Get Outlook for Android<https://aka.ms/ghei36> From: build-dev on behalf of Magnus Ihse Bursie Sent: Saturday, July 4, 2020 7:08:20 PM To: build-dev Subject: Preliminary review for new WINENV support I've been working for some time on a complete rewrite of how we handle the pecularities of the Windows build environment. This new solution supports Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed different "winenvs". One of the main design goals has been to minimize the difference for the configure script and make files for the different winenvs, and leverage as much shared code between them as possible. Another has been to try to collect all the "trickiness" needed in as few places as possible, ideally just one, instead of having it spread out all over the configure script. A third design goal has been to prepare for cross-compilation for Windows from Linux, using Wine. It pretty soon turned out that I needed to get a better grip on how we detect tools in configure, so a complete overhaul of this is included in the change. Now we have more or less fully parted with the original autoconf functions, which has long been too limited for us, and now finally has reached their end of usefulness. At this point, I have a prototype / preview that basically works, but has some limitations. I'd like to ask anyone interested in building OpenJDK on Windows to take the patch for a spin. Especially if you have an esoteric or exotic setup! Webrev: https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2Fdata=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355sdata=LkTMXBZRcaOe%2FRCvcckFViwDOysi7nS9H1Edve%2B4Alk%3Dreserved=0 (If you prefer, you can check out the branch "ihse-winenv-branch" on https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandboxdata=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355sdata=g8UElT7Zx15fMItd8yGGjS%2BuPcyKd3i1EEVHJLumfUo%3Dreserved=0 instead of downloading the patch from the webrev.) I am leaving on vacation next week, so I won't be doing any more work on this for a while, but I promise to read all emails when I get back and try to rectify all issues that are reported. This means you have some time to try it out, too. Here are some technical notes of what is changing, compared to the current Windows build. The native "fixpath.exe" tool is gone. This means that we do not need to compile it during configure, which was always tricky to get right (mostly since we did
Re: Preliminary review for new WINENV support
MSVC_DLL=`$FIND "$WIN_VS_TOOLS_DIR" -name $DLL_NAME \
- | $GREP -i /x86/ | $HEAD --lines 1`
-fi
+POSSIBLE_MSVC_DLL=`$FIND "$WIN_VS_TOOLS_DIR" -name $DLL_NAME \
+ | $GREP -i /$vs_target_cpu/ | $HEAD --lines 1`
TOOLCHAIN_CHECK_POSSIBLE_MSVC_DLL([$DLL_NAME], [$POSSIBLE_MSVC_DLL],
[search of VS100COMNTOOLS])
fi
@@ -554,17 +532,17 @@ AC_DEFUN([TOOLCHAIN_SETUP_MSVC_DLL],
# Probe: Search wildly in the VCINSTALLDIR. We've probably lost by now.
# (This was the original behaviour; kept since it might turn something up)
if test "x$VCINSTALLDIR" != x; then
- if test "x$OPENJDK_TARGET_CPU_BITS" = x64; then
-POSSIBLE_MSVC_DLL=`$FIND "$VCINSTALLDIR" -name $DLL_NAME \
-| $GREP x64 | $HEAD --lines 1`
- else
+ if test "x$vs_target_cpu" = xx86; then
POSSIBLE_MSVC_DLL=`$FIND "$VCINSTALLDIR" -name $DLL_NAME \
-| $GREP x86 | $GREP -v ia64 | $GREP -v x64 | $HEAD --lines 1`
+ | $GREP x86 | $GREP -v ia64 | $GREP -v x64 | $GREP -v arm64 | $HEAD
--lines 1`
if test "x$POSSIBLE_MSVC_DLL" = x; then
# We're grasping at straws now...
POSSIBLE_MSVC_DLL=`$FIND "$VCINSTALLDIR" -name $DLL_NAME \
- | $HEAD --lines 1`
+| $HEAD --lines 1`
fi
+ else
+POSSIBLE_MSVC_DLL=`$FIND "$VCINSTALLDIR" -name $DLL_NAME \
+ | $GREP $vs_target_cpu | $HEAD --lines 1`
fi
TOOLCHAIN_CHECK_POSSIBLE_MSVC_DLL([$DLL_NAME], [$POSSIBLE_MSVC_DLL],
@@ -628,9 +606,9 @@ AC_DEFUN([TOOLCHAIN_SETUP_VS_RUNTIME_DLLS],
fi
AC_ARG_WITH(vcruntime-1-dll, [AS_HELP_STRING([--with-vcruntime-1-dll],
- [path to microsoft C++ runtime dll (vcruntime*_1.dll) (Windows x64 only)
@<:@probed@:>@])])
+ [path to microsoft C++ runtime dll (vcruntime*_1.dll) (Windows 64-bits
only) @<:@probed@:>@])])
- if test "x$VCRUNTIME_1_NAME" != "x" && test "x$OPENJDK_TARGET_BITS" = x64;
then
+ if test "x$VCRUNTIME_1_NAME" != "x" && test "x$OPENJDK_TARGET_CPU_BITS" =
x64; then
if test "x$with_vcruntime_1_dll" != x; then
# If given explicitly by user, do not probe. If not present, fail
directly.
TOOLCHAIN_CHECK_POSSIBLE_MSVC_DLL($VCRUNTIME_1_NAME,
[$with_vcruntime_1_dll],
@@ -695,3 +673,25 @@ AC_DEFUN([TOOLCHAIN_SETUP_VS_RUNTIME_DLLS],
fi
AC_SUBST(UCRT_DLL_DIR)
])
+
+# Setup the sysroot flags and add them to global CFLAGS and LDFLAGS so
+# that configure can use them while detecting compilers.
+# TOOLCHAIN_TYPE is available here.
+# Param 1 - Optional prefix to all variables. (e.g BUILD_)
+AC_DEFUN([TOOLCHAIN_SETUP_VISUAL_STUDIO_SYSROOT_FLAGS],
+[
+ OLDIFS="$IFS"
+ IFS=":"
+
+ # Convert $1VS_INCLUDE into $1SYSROOT_CFLAGS
+ for ipath in [$]$1VS_INCLUDE; do
+$1SYSROOT_CFLAGS="[$]$1SYSROOT_CFLAGS -I$ipath"
+ done
+
+ # Convert $1VS_LIB into $1SYSROOT_LDFLAGS
+ for libpath in [$]$1VS_LIB; do
+$1SYSROOT_LDFLAGS="[$]$1SYSROOT_LDFLAGS -libpath:$libpath"
+ done
+
+ IFS="$OLDIFS"
+])
diff --git a/make/hotspot/gensrc/GensrcAdlc.gmk
b/make/hotspot/gensrc/GensrcAdlc.gmk
index 0947133c16e..bcdea62590d 100644
--- a/make/hotspot/gensrc/GensrcAdlc.gmk
+++ b/make/hotspot/gensrc/GensrcAdlc.gmk
@@ -88,6 +88,10 @@ ifeq ($(call check-jvm-feature, compiler2), true)
ADLCFLAGS += -DAIX=1
else ifeq ($(call isTargetOs, macosx), true)
ADLCFLAGS += -D_ALLBSD_SOURCE=1 -D_GNU_SOURCE=1
+ else ifeq ($(call isTargetOs, windows), true)
+ifeq ($(call isTargetCpuBits, 64), true)
+ ADLCFLAGS += -D_WIN64=1
+endif
endif
ifeq ($(call isTargetOs, windows), false)
diff --git a/make/modules/java.base/gensrc/GensrcMisc.gmk
b/make/modules/java.base/gensrc/GensrcMisc.gmk
index cd12f2ab318..5390fcabe85 100644
--- a/make/modules/java.base/gensrc/GensrcMisc.gmk
+++ b/make/modules/java.base/gensrc/GensrcMisc.gmk
@@ -60,6 +60,10 @@ ifneq ($(filter $(TOOLCHAIN_TYPE), gcc clang), )
CPP_FLAGS += -x c
else ifeq ($(TOOLCHAIN_TYPE), microsoft)
CPP_FLAGS += -nologo
+
+ # cl.exe does only recognize few file extensions as valid (ex: .c, .h,
.cpp), so
+ # make sure *.java.template files are recognized as valid input files
+ CPP_FILEPREFIX = -Tc
endif
# Generate a java source file from a template through the C preprocessor for
the
@@ -73,7 +77,7 @@ define generate-preproc-src
$(call MakeDir, $(@D))
$(call ExecuteWithLog, $(SUPPORT_OUTPUTDIR)/gensrc/java.base/_$(@F), \
( $(NAWK) '/@@END_COPYRIGHT@@/{exit}1' $< && \
- $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB) $< \
+ $(CPP) $(CPP_FLAGS) $(SYSROOT_CFLAGS) $(CFLAGS_JDKLIB)
$(CPP_FILEPREFIX) $< \
2> >($(GREP) -v '^$(&2) \
Re: Preliminary review for new WINENV support
Hi Magnus,
It's awesome work! I tested your patch, but I saw some errors (configure
--enable-debug --with-boot-jdk=/path/to/jdk14):
1) script error
checking for gdiff...
/mnt/d/test/jdk-master/build/.configure-support/generated-configure.sh: line
9989: test: /mnt/c/Program: binary operator expected
/mnt/d/test/jdk-master/build/.configure-support/generated-configure.sh: line
9993: test: /mnt/c/Program: binary operator expected
/mnt/d/test/jdk-master/build/.configure-support/generated-configure.sh: line
9989: test: too many arguments
2) warning in awk
configure: Found potential Boot JDK using configure arguments
gawk: cmd. line:1: warning: regexp escape sequence `\"' is not a known regexp
operator
checking for Boot JDK... /mnt/c/java/jdk-14.0.1
3) command not found at fixpath.sh (it happens on `make images`)
/mnt/d/test/jdk-master/make/scripts/fixpath.sh: line 402:
/mnt/d/test/jdk-master/build/windows-x86_64-server-fastdebug/jdk/bin/java: No
such file or directory
I fixed them with following patch, and it works fine on my WSL 1.
```
diff --git a/make/autoconf/boot-jdk.m4 b/make/autoconf/boot-jdk.m4
index 7059558b2..db73eba15 100644
--- a/make/autoconf/boot-jdk.m4
+++ b/make/autoconf/boot-jdk.m4
@@ -77,7 +77,7 @@ AC_DEFUN([BOOTJDK_DO_CHECK],
# Additional [] needed to keep m4 from mangling shell constructs.
java_to_test="$BOOT_JDK/bin/java"
UTIL_FIXUP_EXECUTABLE(java_to_test)
- [ BOOT_JDK_VERSION=`$java_to_test $USER_BOOT_JDK_OPTIONS -version 2>&1 | $AWK
'/version \"[0-9a-zA-Z\._\-]+\"/{print $ 0; exit;}'` ]
+ [ BOOT_JDK_VERSION=`$java_to_test $USER_BOOT_JDK_OPTIONS -version 2>&1 | $AWK
'/version "[0-9a-zA-Z\._\-]+"/{print $ 0; exit;}'` ]
if [ [[ "$BOOT_JDK_VERSION" =~ "Picked up" ]] ]; then
AC_MSG_NOTICE([You have _JAVA_OPTIONS or JAVA_TOOL_OPTIONS set.
This can mess up the build. Please use --with-boot-jdk-jvmargs instead.])
AC_MSG_NOTICE([Java reports: "$BOOT_JDK_VERSION".])
diff --git a/make/autoconf/util_paths.m4 b/make/autoconf/util_paths.m4
index 8dec82fdc..3d20d1700 100644
--- a/make/autoconf/util_paths.m4
+++ b/make/autoconf/util_paths.m4
@@ -377,11 +377,11 @@ AC_DEFUN([UTIL_LOOKUP_PROGS],
continue
fi
full_path="$elem/$name"
- if test ! -e $full_path && test "x$OPENJDK_BUILD_OS" = "xwindows";
then
+ if test ! -e "$full_path" && test "x$OPENJDK_BUILD_OS" = "xwindows";
then
# Try again with .exe
full_path="$elem/$name.exe"
fi
- if test -e $full_path; then
+ if test -e "$full_path"; then
$1="$full_path"
UTIL_FIXUP_EXECUTABLE($1, $3)
result="[$]$1"
diff --git a/make/scripts/fixpath.sh b/make/scripts/fixpath.sh
index 14eacbec6..f8293a798 100644
--- a/make/scripts/fixpath.sh
+++ b/make/scripts/fixpath.sh
@@ -393,6 +393,10 @@ function exec_command_line() {
fi
done
+ if [ ! -e "$command" ]; then
+command="$command".exe
+ fi
+
# Now execute it
if [[ -v DEBUG_FIXPATH ]]; then
echo fixpath: debug: "$command" "${collected_args[@]}" >&2
```
I tried to build on WSL 2, but I couldn't because the process seemed to hangup.
configure script could work normally, but I saw following message on my console
in the end.
I guess it is not an issue in your patch because I haven't seen it on WSL 1.
```
cat: standard output: No such file or directory
```
Also I saw LNK1158 error as following, but it might be caused by my environment
- my Windows box has been installed both Visual Studio 2019 and Windows SDK.
(My PC is set locale to Japanese, sorry :)
* For target hotspot_variant-server_libjvm_objs_BUILD_LIBJVM_link:
ライブラリ
d:\test\jdk-master\build\windows-x86_64-server-fastdebug\hotspot\variant-server\libjvm\objs\jvm.lib
とオブジェクト
d:\test\jdk-master\build\windows-x86_64-server-fastdebug\hotspot\variant-server\libjvm\objs\jvm.exp
を作成中
LINK : fatal error LNK1158: 'rc.exe' を実行できません。
I could solve the problem in the way the following URL indicate.
https://stackoverflow.com/questions/35215971/lnk1158-cannot-run-rc-exe-x64-visual-studio
Thanks,
Yasumasa
On 2020/07/05 9:08, Magnus Ihse Bursie wrote:
I've been working for some time on a complete rewrite of how we handle the pecularities
of the Windows build environment. This new solution supports Cygwin, Msys2, WSL1 and WSL2
(after a fashion, see below), what I have termed different "winenvs".
One of the main design goals has been to minimize the difference for the configure script
and make files for the different winenvs, and leverage as much shared code between them
as possible. Another has been to try to collect all the "trickiness" needed in
as few places as possible, ideally just one, instead of having it spread out all over the
configure script. A third design goal has been to prepare for cross-compilation for
Windows from Linux, using Wine.
It pretty soon turned out that I
Re: Preliminary review for new WINENV support
Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
. When working with cross-compilation mods for aarch64-win port, I realized we
needed to clean up/better support the quirks around fixpath and also I was
hoping to expand the support to wsl2, etc. From what I am reading, you seem to
have touched all of these and more! I am mostly offline for the next two days,
but will start testing your changes when I am back online. Thank you for doing
the needful.
Monica
Get Outlook for Android<https://aka.ms/ghei36>
From: build-dev on behalf of Magnus Ihse
Bursie
Sent: Saturday, July 4, 2020 7:08:20 PM
To: build-dev
Subject: Preliminary review for new WINENV support
I've been working for some time on a complete rewrite of how we handle the
pecularities of the Windows build environment. This new solution supports
Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed
different "winenvs".
One of the main design goals has been to minimize the difference for the
configure script and make files for the different winenvs, and leverage as much
shared code between them as possible. Another has been to try to collect all
the "trickiness" needed in as few places as possible, ideally just one, instead
of having it spread out all over the configure script. A third design goal has
been to prepare for cross-compilation for Windows from Linux, using Wine.
It pretty soon turned out that I needed to get a better grip on how we detect
tools in configure, so a complete overhaul of this is included in the change.
Now we have more or less fully parted with the original autoconf functions,
which has long been too limited for us, and now finally has reached their end
of usefulness.
At this point, I have a prototype / preview that basically works, but has some
limitations.
I'd like to ask anyone interested in building OpenJDK on Windows to take the
patch for a spin. Especially if you have an esoteric or exotic setup!
Webrev:
https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2Fdata=02%7C01%7Cmonica.beckwith%40microsoft.com%7C78ed6cf8f63042ee176908d82077a6cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637295045569644419sdata=EnmrU3hXkAGeXuc4h0tXeQFljPwbmiCtaz%2BU2%2BrJMbE%3Dreserved=0
(If you prefer, you can check out the branch "ihse-winenv-branch" on
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandbox%2Fdata=02%7C01%7Cmonica.beckwith%40microsoft.com%7C78ed6cf8f63042ee176908d82077a6cf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637295045569654420sdata=7IDqLnBCiL%2FSoSyddEabAuKsehUaRo%2BnK9pesr%2F1x50%3Dreserved=0
instead of downloading the patch from the webrev.)
I am leaving on vacation next week, so I won't be doing any more work on this
for a while, but I promise to read all emails when I get back and try to
rectify all issues that are reported. This means you have some time to try it
out, too.
Here are some technical notes of what is changing, compared to the current
Windows build.
The native "fixpath.exe" tool is gone. This means that we do not need to
compile it during configure, which was always tricky to get right (mostly since
we did not have fixpath in place to help us...).
Instead, there is a new fixpath.sh shell script, that does the same job, and
more. The script is written in highly optimized shell code (yeah, I see the
oxymoron) so only bash internal functionality is used, to avoid calling
external tools, which is expensive on Windows. This makes the performance
practically roughly at par with the native fixpath.exe.
Fixpath also has a "print" and "import" mode, apart from the traditional"exec"
mode. This makes it possible to use the same tool for converting individual
paths at runtime to Windows style ("print"), and it takes the logic needed to
"import" paths given by the user to configure, into a format usable internally
by the build system, and moves it into a centralized location in the fixpath
script.
A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, ENVROOT
and WINTEMP. For instance, for "cygwin", these are:
PATHTOOL=cygpath
DRIVEPREFIX=/cygdrive (typically)
ENVROOT=C:\Cygwin64 (typically)
WINTEMP=/tmp
These are needed for fixpath to do it's magic. Fixpath can auto-detect those,
but to save on execution time they are normally detected by configure and sent
as arguments to fixpath.
Detection of the Visual Studio environment has been massively simplified. Using
fixpath, conversion between Windows and unix paths is not so complex anymore.
The bridge Windows batch file that is needed to extract the environment
variables is no longer created on the fly, but is instead stored in
make/scripts/extract-vs-env.cm
Preliminary review for new WINENV support
I've been working for some time on a complete rewrite of how we handle the
pecularities of the Windows build environment. This new solution supports
Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed
different "winenvs".
One of the main design goals has been to minimize the difference for the
configure script and make files for the different winenvs, and leverage as much
shared code between them as possible. Another has been to try to collect all
the "trickiness" needed in as few places as possible, ideally just one, instead
of having it spread out all over the configure script. A third design goal has
been to prepare for cross-compilation for Windows from Linux, using Wine.
It pretty soon turned out that I needed to get a better grip on how we detect
tools in configure, so a complete overhaul of this is included in the change.
Now we have more or less fully parted with the original autoconf functions,
which has long been too limited for us, and now finally has reached their end
of usefulness.
At this point, I have a prototype / preview that basically works, but has some
limitations.
I'd like to ask anyone interested in building OpenJDK on Windows to take the
patch for a spin. Especially if you have an esoteric or exotic setup!
Webrev: http://cr.openjdk.java.net/~ihse/winenv-preview-1/webrev.01/
(If you prefer, you can check out the branch "ihse-winenv-branch" on
http://hg.openjdk.java.net/jdk/sandbox/ instead of downloading the patch from
the webrev.)
I am leaving on vacation next week, so I won't be doing any more work on this
for a while, but I promise to read all emails when I get back and try to
rectify all issues that are reported. This means you have some time to try it
out, too.
Here are some technical notes of what is changing, compared to the current
Windows build.
The native "fixpath.exe" tool is gone. This means that we do not need to
compile it during configure, which was always tricky to get right (mostly since
we did not have fixpath in place to help us...).
Instead, there is a new fixpath.sh shell script, that does the same job, and
more. The script is written in highly optimized shell code (yeah, I see the
oxymoron) so only bash internal functionality is used, to avoid calling
external tools, which is expensive on Windows. This makes the performance
practically roughly at par with the native fixpath.exe.
Fixpath also has a "print" and "import" mode, apart from the traditional"exec"
mode. This makes it possible to use the same tool for converting individual
paths at runtime to Windows style ("print"), and it takes the logic needed to
"import" paths given by the user to configure, into a format usable internally
by the build system, and moves it into a centralized location in the fixpath
script.
A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, ENVROOT
and WINTEMP. For instance, for "cygwin", these are:
PATHTOOL=cygpath
DRIVEPREFIX=/cygdrive (typically)
ENVROOT=C:\Cygwin64 (typically)
WINTEMP=/tmp
These are needed for fixpath to do it's magic. Fixpath can auto-detect those,
but to save on execution time they are normally detected by configure and sent
as arguments to fixpath.
Detection of the Visual Studio environment has been massively simplified. Using
fixpath, conversion between Windows and unix paths is not so complex anymore.
The bridge Windows batch file that is needed to extract the environment
variables is no longer created on the fly, but is instead stored in
make/scripts/extract-vs-env.cmd. This is called with fixpath, so all arguments
to it can be unix paths.
Furthermore, TOOLCHAIN_SETUP_VISUAL_STUDIO_ENV is now clearly designed to have
a single responsibility, to set SYSROOT flags (and TOOLCHAIN_PATH) for the
microsoft toolchain. As a result of this, it is now called from
FLAGS_SETUP_SYSROOT_FLAGS. Also, the file toolchain_windows.m4 is now more
correctly named toolchain_microsoft.m4. A price we had to pay for this was that
the old idea that you should be able to start a "Visual Studio console" and
then run configure from it to extract the variables do not work anymore. (It
had not been tested for ages, and might have been broken anyway.)
Fixpath also knows about the difference between unix path lists
(/foo/bar:/my/dir) and windows path lists (c:\dir;d:\data) and can convert
freely and automatically between them. This furthermore means that PATH_SEP is
removed, and we only use unix style (colon separated) path lists internally.
The logic automatically detects if and when .exe is needed, so EXE_SUFFIX is
removed too. There are some limitations; when code needs to explicitly test for
the presence of a file, the suffix needs to be correct. Also when make files
check for e.g. the generated bin/java.exe, the suffix needs to be present. For
this, I have introduced an EXECUTABLE_SUFFIX, that has the same value as
EXE_SUFFIX -- but not the same semantics! The old
