-Derek
________________________________
From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Derek Keeler
<dekee...@microsoft.com>
Sent: July 8, 2020 5:53 PM
To: Monica Beckwith <monica.beckw...@microsoft.com>; Magnus Ihse Bursie
<magnus.ihse.bur...@oracle.com>; build-dev <build-dev@openjdk.java.net>
Subject: Re: Preliminary review for new WINENV support
This is fantastic!
I'm synching to this branch tonight and will start building it locally on my
WSL2 environment. I'll send questions/comments to this thread unless there is a
better place?
(Note: our internal email system may obfuscate the links below in future
replies, my apologies).
-Derek
________________________________
From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Monica Beckwith
<monica.beckw...@microsoft.com>
Sent: July 5, 2020 11:34 AM
To: Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>; build-dev
<build-dev@openjdk.java.net>
Subject: Re: Preliminary review for new WINENV support
Wow! Impressive work, @Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
. When working with cross-compilation mods for aarch64-win port, I realized we needed
to clean up/better support the quirks around fixpath and also I was hoping to expand
the support to wsl2, etc. From what I am reading, you seem to have touched all of
these and more! I am mostly offline for the next two days, but will start testing
your changes when I am back online. Thank you for doing the needful.
Monica
Get Outlook for Android<https://aka.ms/ghei36>
________________________________
From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Magnus Ihse Bursie
<magnus.ihse.bur...@oracle.com>
Sent: Saturday, July 4, 2020 7:08:20 PM
To: build-dev <build-dev@openjdk.java.net>
Subject: Preliminary review for new WINENV support
I've been working for some time on a complete rewrite of how we handle the pecularities
of the Windows build environment. This new solution supports Cygwin, Msys2, WSL1 and WSL2
(after a fashion, see below), what I have termed different "winenvs".
One of the main design goals has been to minimize the difference for the configure script
and make files for the different winenvs, and leverage as much shared code between them
as possible. Another has been to try to collect all the "trickiness" needed in
as few places as possible, ideally just one, instead of having it spread out all over the
configure script. A third design goal has been to prepare for cross-compilation for
Windows from Linux, using Wine.
It pretty soon turned out that I needed to get a better grip on how we detect
tools in configure, so a complete overhaul of this is included in the change.
Now we have more or less fully parted with the original autoconf functions,
which has long been too limited for us, and now finally has reached their end
of usefulness.
At this point, I have a prototype / preview that basically works, but has some
limitations.
I'd like to ask anyone interested in building OpenJDK on Windows to take the
patch for a spin. Especially if you have an esoteric or exotic setup!
Webrev:
https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2F&data=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355&sdata=LkTMXBZRcaOe%2FRCvcckFViwDOysi7nS9H1Edve%2B4Alk%3D&reserved=0
(If you prefer, you can check out the branch "ihse-winenv-branch" on
https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandbox&data=02%7C01%7Cdekeeler%40microsoft.com%7C28e792b26f604f86be7108d823a29bf5%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298528593747355&sdata=g8UElT7Zx15fMItd8yGGjS%2BuPcyKd3i1EEVHJLumfUo%3D&reserved=0
instead of downloading the patch from the webrev.)
I am leaving on vacation next week, so I won't be doing any more work on this
for a while, but I promise to read all emails when I get back and try to
rectify all issues that are reported. This means you have some time to try it
out, too.
Here are some technical notes of what is changing, compared to the current
Windows build.
The native "fixpath.exe" tool is gone. This means that we do not need to
compile it during configure, which was always tricky to get right (mostly since we did
not have fixpath in place to help us...).
Instead, there is a new fixpath.sh shell script, that does the same job, and
more. The script is written in highly optimized shell code (yeah, I see the
oxymoron) so only bash internal functionality is used, to avoid calling
external tools, which is expensive on Windows. This makes the performance
practically roughly at par with the native fixpath.exe.
Fixpath also has a "print" and "import" mode, apart from the traditional"exec" mode. This makes it
possible to use the same tool for converting individual paths at runtime to Windows style ("print"), and it takes the
logic needed to "import" paths given by the user to configure, into a format usable internally by the build system, and
moves it into a centralized location in the fixpath script.
A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, ENVROOT and
WINTEMP. For instance, for "cygwin", these are:
PATHTOOL=cygpath
DRIVEPREFIX=/cygdrive (typically)
ENVROOT=C:\Cygwin64 (typically)
WINTEMP=/tmp
These are needed for fixpath to do it's magic. Fixpath can auto-detect those,
but to save on execution time they are normally detected by configure and sent
as arguments to fixpath.
Detection of the Visual Studio environment has been massively simplified. Using
fixpath, conversion between Windows and unix paths is not so complex anymore.
The bridge Windows batch file that is needed to extract the environment
variables is no longer created on the fly, but is instead stored in
make/scripts/extract-vs-env.cmd. This is called with fixpath, so all arguments
to it can be unix paths.
Furthermore, TOOLCHAIN_SETUP_VISUAL_STUDIO_ENV is now clearly designed to have a single
responsibility, to set SYSROOT flags (and TOOLCHAIN_PATH) for the microsoft toolchain. As
a result of this, it is now called from FLAGS_SETUP_SYSROOT_FLAGS. Also, the file
toolchain_windows.m4 is now more correctly named toolchain_microsoft.m4. A price we had
to pay for this was that the old idea that you should be able to start a "Visual
Studio console" and then run configure from it to extract the variables do not work
anymore. (It had not been tested for ages, and might have been broken anyway.)
Fixpath also knows about the difference between unix path lists
(/foo/bar:/my/dir) and windows path lists (c:\dir;d:\data) and can convert
freely and automatically between them. This furthermore means that PATH_SEP is
removed, and we only use unix style (colon separated) path lists internally.
The logic automatically detects if and when .exe is needed, so EXE_SUFFIX is removed too.
There are some limitations; when code needs to explicitly test for the presence of a
file, the suffix needs to be correct. Also when make files check for e.g. the generated
bin/java.exe, the suffix needs to be present. For this, I have introduced an
EXECUTABLE_SUFFIX, that has the same value as EXE_SUFFIX -- but not the same semantics!
The old code added EXE_SUFFIX here and there, not caring about if we meant
"microsoft" as a toolchain or if we were running on Windows as a build
platform. Clearly not well adapted for future cross-compilation.
The old ways of locating programs in configure were messy, complicated and not always correct. We
used a mixture of the original autoconf macros, and our own UTIL_PATH_PROGS and UTIL_CHECK_TOOLS.
These did not have very well defined semantics, and were frequently mixed up. Also,
UTIL_CHECK_TOOLS required UTIL_FIXUP_EXECUTABLE tobe called afterwards, to "rewrite" the
result in a second step. This was not needed after UTIL_PATH_PROGS but was frequently done anyway
to "be on the safe side".
I have now replaced:
AC_(PATH|CHECK)_(PROG|PROGS) with UTIL_LOOKUP_PROGS
AC_(PATH|CHECK)_(TOOL|TOOLS) with UTIL_LOOKUP_TOOLCHAIN_PROGS
This is actually almost the same semantic, but unless you're an autoconf aficionado, you ar not likely to
understand the difference between "PROG" and "TOOL". The only difference is that
UTIL_LOOKUP_TOOLCHAIN_PROGS will try to look for "host-prefixed" tools first, when cross-compiling,
and should therefore be used for all toolchain lookups.
There is also a fail-fast version of both, UTIL_REQUIRE_PROGS and
UTIL_REQUIRE_TOOLCHAIN_PROGS.
UTIL_LOOKUP_PROGS is the core function, with the rest being thin wrappers
around it. This function is created from scratch, to do exactly what we want,
on Unix platforms and Windows. So there is no need anymore to call
UTIL_FIXUP_EXECUTABLE, unless you have input from elsewhere (e.g. user flag)
that you need to verify. I have also collected all this logic in a single file,
util_paths.m4, moving parts from util.m4, and just removing util_windows.m4.
UTIL_LOOKUP_PROGS will use the new and nifty function
UTIL_CHECK_WINENV_EXEC_TYPE, which will automatically determine if an
executable needs to be prefixed by fixpath! That means that you can match and
mix Windows-style and Unix-style programs however you like, with very few
limitations. For instance, you can have a Linux version of the BootJDK on WSL.
For this to work, the $FIXPATH prefix is now stored in the variables themselves
(e.g. in $CC), rather than added as @FIXPATH@ in spec.gmk.in. This has
generally worked out OK, but caused some headaches when the code thought that
$CC (etc) was not a way to launch a program, but was a file that could be
tested for existence, etc.
I reintroduced support for msys2. This was mostly free, since msys2 is so close
to cygwin (which msys never where). To underline the difference, I renamed the
winenv windows.msys2 (instead of windows.msys). Msys (version 1) will never be
supported in modern OpenJDK due to critical packages being far too old, and not
updated. I also clearly separate between WSL1 (which is using a kernel
emulation layer, somewhat like Wine) and WSL2 (which is using a Hyper-V VM to
run an actual Linux kernel), as windows.wsl1 and windows.wsl2.
I have also done a ton of small fixes, to make things more convenient and
smooth when working in these winenvs. I have adjusted the output from configure
to be less verbose and more streamlined. Overall, a lot of odd code in
configure has been removed. A few changes that are strictly unneeded for this
patch has also been included; mostly removal of dead code I came across, and a
few fixes for additional debuggability that I needed when developing this
patch, like ExecuteithLog.
I have also temporarily disabled the javac server, and building without
absolute paths. I believe a better way forward is to address these with
improved functionality in separate patches, instead of trying to work around
the issues introduced by them in this patch.
I have done substantial testing of the core functionality (building jdk-images)
on Cygwin, Msys2 and WSL2, both on my personal machine and on Oracle's CI
system. The performance on Cygwin (which we use for our CI builds) is roughly
on par with the old Cygwin performance, but the WSL1 performance is roughly 20%
faster, so I think we should investigate if it is possible for us to switch to
WSL1. Everything seems stable so far, but more testing is definitely needed. I
have also not even started testing autxillary functionality, such as the
compare script, IDE project file generation etc.
However, the big disappointment in all of this is WSL2. That was the main driver that got me
started on this rewrite. But it turned out that WSL2 is still very immature. :-( There are lot of
issues, like stdout and stderr getting messed up, time synchronization issues causing make to
complain about "Clock skew detected", extreme slowness when accessing disks cross the OS
boundary. But worst of all has been that WSL2 is *extremly* unstable. After a few calls of Window
executables, I get the impression that a conversion daemon dies. All further calls to Window
binaries just stalls, and the WSL2 instance is unusable until I terminate it using "wsl -t
Ubuntu-2004" from a cmd shell. :-(
So the WSL2 functionality is not very well tested, since I have not been able to build it
completely a single time. I do believe that everything is correct, in "theory".
So in case this is something broken with my WSL2 installation, I'd encourage anyone with
a WSL2 installation to try it out.
/Magnus
