Hi Magnus, I want to follow up on this work of yours, as we've particular interest in it for the Windows-AArch64 port.
Let me know how I could assist you in this effort. Thank you, -- Ludovic -----Original Message----- From: build-dev <build-dev-r...@openjdk.java.net> On Behalf Of Yasumasa Suenaga Sent: Wednesday, July 8, 2020 9:47 PM To: Derek Keeler <dekee...@microsoft.com>; Monica Beckwith <monica.beckw...@microsoft.com>; Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>; build-dev <build-dev@openjdk.java.net> Subject: Re: Preliminary review for new WINENV support On 2020/07/09 12:12, Derek Keeler wrote: > After applying Suenaga's patch, I see this error: > > checking wsl distribution... > /home/dekeeler/dev/openjdk.net/sandbox/build/.configure-support/generated-configure.sh: > line 32952: -d: command not found > > I'm running Debian 10 in my WSL2 instance and it does not include > `lsb_release` by default. Installed it with `sudo apt-get install > lsb-release`. It's better if we add AC_CHECK_PROG for lsb_release for Windows (for just WSL?) > The next trouble I ran into is that I only have Visual Studio 2019 installed. > I tried to make it work but lack the necessary experience to wrangle this > code. Will start again in the morning after installing VS2017. FYI: I can run configure script with VS2019 community edition on Ubuntu 20.04 on both WSL 1 and 2, however I could not complete building OpenJDK image on WSL 2 due to I/O performance issue. Thanks, Yasumasa > -Derek > > ________________________________ > From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Derek Keeler > <dekee...@microsoft.com> > Sent: July 8, 2020 5:53 PM > To: Monica Beckwith <monica.beckw...@microsoft.com>; Magnus Ihse Bursie > <magnus.ihse.bur...@oracle.com>; build-dev <build-dev@openjdk.java.net> > Subject: Re: Preliminary review for new WINENV support > > This is fantastic! > > I'm synching to this branch tonight and will start building it locally on my > WSL2 environment. I'll send questions/comments to this thread unless there is > a better place? > > (Note: our internal email system may obfuscate the links below in future > replies, my apologies). > > -Derek > > ________________________________ > From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Monica > Beckwith <monica.beckw...@microsoft.com> > Sent: July 5, 2020 11:34 AM > To: Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>; build-dev > <build-dev@openjdk.java.net> > Subject: Re: Preliminary review for new WINENV support > > Wow! Impressive work, @Magnus Ihse > Bursie<mailto:magnus.ihse.bur...@oracle.com> . When working with > cross-compilation mods for aarch64-win port, I realized we needed to clean > up/better support the quirks around fixpath and also I was hoping to expand > the support to wsl2, etc. From what I am reading, you seem to have touched > all of these and more! I am mostly offline for the next two days, but will > start testing your changes when I am back online. Thank you for doing the > needful. > > Monica > > Get Outlook for Android<https://aka.ms/ghei36> > > ________________________________ > From: build-dev <build-dev-r...@openjdk.java.net> on behalf of Magnus Ihse > Bursie <magnus.ihse.bur...@oracle.com> > Sent: Saturday, July 4, 2020 7:08:20 PM > To: build-dev <build-dev@openjdk.java.net> > Subject: Preliminary review for new WINENV support > > I've been working for some time on a complete rewrite of how we handle the > pecularities of the Windows build environment. This new solution supports > Cygwin, Msys2, WSL1 and WSL2 (after a fashion, see below), what I have termed > different "winenvs". > > One of the main design goals has been to minimize the difference for the > configure script and make files for the different winenvs, and leverage as > much shared code between them as possible. Another has been to try to collect > all the "trickiness" needed in as few places as possible, ideally just one, > instead of having it spread out all over the configure script. A third design > goal has been to prepare for cross-compilation for Windows from Linux, using > Wine. > > It pretty soon turned out that I needed to get a better grip on how we detect > tools in configure, so a complete overhaul of this is included in the change. > Now we have more or less fully parted with the original autoconf functions, > which has long been too limited for us, and now finally has reached their end > of usefulness. > > At this point, I have a prototype / preview that basically works, but has > some limitations. > > I'd like to ask anyone interested in building OpenJDK on Windows to take the > patch for a spin. Especially if you have an esoteric or exotic setup! > > Webrev: > https://nam06.safelinks.protection.outlook.com/?url=http:%2F%2Fcr.openjdk.java.net%2F~ihse%2Fwinenv-preview-1%2Fwebrev.01%2F&data=02%7C01%7Cluhenry%40microsoft.com%7C55d5bddc82f142c2312c08d823c33111%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298668546029073&sdata=k7tQ6%2BfwdrD8AGryNZdqAD%2Fk3Lnc%2B%2B75opEaWASISo0%3D&reserved=0 > > (If you prefer, you can check out the branch "ihse-winenv-branch" on > https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhg.openjdk.java.net%2Fjdk%2Fsandbox&data=02%7C01%7Cluhenry%40microsoft.com%7C55d5bddc82f142c2312c08d823c33111%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637298668546029073&sdata=yoknkwhOhji8D43VzTyj4ryiNtNx2CSnCliVTd420nI%3D&reserved=0 > instead of downloading the patch from the webrev.) > > > I am leaving on vacation next week, so I won't be doing any more work on this > for a while, but I promise to read all emails when I get back and try to > rectify all issues that are reported. This means you have some time to try it > out, too. > > Here are some technical notes of what is changing, compared to the current > Windows build. > > The native "fixpath.exe" tool is gone. This means that we do not need to > compile it during configure, which was always tricky to get right (mostly > since we did not have fixpath in place to help us...). > > Instead, there is a new fixpath.sh shell script, that does the same job, and > more. The script is written in highly optimized shell code (yeah, I see the > oxymoron) so only bash internal functionality is used, to avoid calling > external tools, which is expensive on Windows. This makes the performance > practically roughly at par with the native fixpath.exe. > > Fixpath also has a "print" and "import" mode, apart from the > traditional"exec" mode. This makes it possible to use the same tool for > converting individual paths at runtime to Windows style ("print"), and it > takes the logic needed to "import" paths given by the user to configure, into > a format usable internally by the build system, and moves it into a > centralized location in the fixpath script. > > A "winenv" is defined by a quartet of variables: PATHTOOL, DRIVEPREFIX, > ENVROOT and WINTEMP. For instance, for "cygwin", these are: > PATHTOOL=cygpath > DRIVEPREFIX=/cygdrive (typically) > ENVROOT=C:\Cygwin64 (typically) > WINTEMP=/tmp > > These are needed for fixpath to do it's magic. Fixpath can auto-detect those, > but to save on execution time they are normally detected by configure and > sent as arguments to fixpath. > > Detection of the Visual Studio environment has been massively simplified. > Using fixpath, conversion between Windows and unix paths is not so complex > anymore. The bridge Windows batch file that is needed to extract the > environment variables is no longer created on the fly, but is instead stored > in make/scripts/extract-vs-env.cmd. This is called with fixpath, so all > arguments to it can be unix paths. > > Furthermore, TOOLCHAIN_SETUP_VISUAL_STUDIO_ENV is now clearly designed to > have a single responsibility, to set SYSROOT flags (and TOOLCHAIN_PATH) for > the microsoft toolchain. As a result of this, it is now called from > FLAGS_SETUP_SYSROOT_FLAGS. Also, the file toolchain_windows.m4 is now more > correctly named toolchain_microsoft.m4. A price we had to pay for this was > that the old idea that you should be able to start a "Visual Studio console" > and then run configure from it to extract the variables do not work anymore. > (It had not been tested for ages, and might have been broken anyway.) > > Fixpath also knows about the difference between unix path lists > (/foo/bar:/my/dir) and windows path lists (c:\dir;d:\data) and can convert > freely and automatically between them. This furthermore means that PATH_SEP > is removed, and we only use unix style (colon separated) path lists > internally. > > The logic automatically detects if and when .exe is needed, so EXE_SUFFIX is > removed too. There are some limitations; when code needs to explicitly test > for the presence of a file, the suffix needs to be correct. Also when make > files check for e.g. the generated bin/java.exe, the suffix needs to be > present. For this, I have introduced an EXECUTABLE_SUFFIX, that has the same > value as EXE_SUFFIX -- but not the same semantics! The old code added > EXE_SUFFIX here and there, not caring about if we meant "microsoft" as a > toolchain or if we were running on Windows as a build platform. Clearly not > well adapted for future cross-compilation. > > The old ways of locating programs in configure were messy, complicated and > not always correct. We used a mixture of the original autoconf macros, and > our own UTIL_PATH_PROGS and UTIL_CHECK_TOOLS. These did not have very well > defined semantics, and were frequently mixed up. Also, UTIL_CHECK_TOOLS > required UTIL_FIXUP_EXECUTABLE tobe called afterwards, to "rewrite" the > result in a second step. This was not needed after UTIL_PATH_PROGS but was > frequently done anyway to "be on the safe side". > > I have now replaced: > AC_(PATH|CHECK)_(PROG|PROGS) with UTIL_LOOKUP_PROGS > AC_(PATH|CHECK)_(TOOL|TOOLS) with UTIL_LOOKUP_TOOLCHAIN_PROGS > > This is actually almost the same semantic, but unless you're an autoconf > aficionado, you ar not likely to understand the difference between "PROG" > and "TOOL". The only difference is that UTIL_LOOKUP_TOOLCHAIN_PROGS will try > to look for "host-prefixed" tools first, when cross-compiling, and should > therefore be used for all toolchain lookups. > > There is also a fail-fast version of both, UTIL_REQUIRE_PROGS and > UTIL_REQUIRE_TOOLCHAIN_PROGS. > > UTIL_LOOKUP_PROGS is the core function, with the rest being thin wrappers > around it. This function is created from scratch, to do exactly what we want, > on Unix platforms and Windows. So there is no need anymore to call > UTIL_FIXUP_EXECUTABLE, unless you have input from elsewhere (e.g. user flag) > that you need to verify. I have also collected all this logic in a single > file, util_paths.m4, moving parts from util.m4, and just removing > util_windows.m4. > > UTIL_LOOKUP_PROGS will use the new and nifty function > UTIL_CHECK_WINENV_EXEC_TYPE, which will automatically determine if an > executable needs to be prefixed by fixpath! That means that you can match and > mix Windows-style and Unix-style programs however you like, with very few > limitations. For instance, you can have a Linux version of the BootJDK on > WSL. For this to work, the $FIXPATH prefix is now stored in the variables > themselves (e.g. in $CC), rather than added as @FIXPATH@ in spec.gmk.in. This > has generally worked out OK, but caused some headaches when the code thought > that $CC (etc) was not a way to launch a program, but was a file that could > be tested for existence, etc. > > I reintroduced support for msys2. This was mostly free, since msys2 is so > close to cygwin (which msys never where). To underline the difference, I > renamed the winenv windows.msys2 (instead of windows.msys). Msys (version 1) > will never be supported in modern OpenJDK due to critical packages being far > too old, and not updated. I also clearly separate between WSL1 (which is > using a kernel emulation layer, somewhat like Wine) and WSL2 (which is using > a Hyper-V VM to run an actual Linux kernel), as windows.wsl1 and windows.wsl2. > > I have also done a ton of small fixes, to make things more convenient and > smooth when working in these winenvs. I have adjusted the output from > configure to be less verbose and more streamlined. Overall, a lot of odd code > in configure has been removed. A few changes that are strictly unneeded for > this patch has also been included; mostly removal of dead code I came across, > and a few fixes for additional debuggability that I needed when developing > this patch, like ExecuteithLog. > > I have also temporarily disabled the javac server, and building without > absolute paths. I believe a better way forward is to address these with > improved functionality in separate patches, instead of trying to work around > the issues introduced by them in this patch. > > I have done substantial testing of the core functionality (building > jdk-images) on Cygwin, Msys2 and WSL2, both on my personal machine and on > Oracle's CI system. The performance on Cygwin (which we use for our CI > builds) is roughly on par with the old Cygwin performance, but the WSL1 > performance is roughly 20% faster, so I think we should investigate if it is > possible for us to switch to WSL1. Everything seems stable so far, but more > testing is definitely needed. I have also not even started testing autxillary > functionality, such as the compare script, IDE project file generation > etc. > > However, the big disappointment in all of this is WSL2. That was the main > driver that got me started on this rewrite. But it turned out that WSL2 is > still very immature. :-( There are lot of issues, like stdout and stderr > getting messed up, time synchronization issues causing make to complain about > "Clock skew detected", extreme slowness when accessing disks cross the OS > boundary. But worst of all has been that WSL2 is *extremly* unstable. After a > few calls of Window executables, I get the impression that a conversion > daemon dies. All further calls to Window binaries just stalls, and the WSL2 > instance is unusable until I terminate it using "wsl -t Ubuntu-2004" from a > cmd shell. :-( > > So the WSL2 functionality is not very well tested, since I have not been able > to build it completely a single time. I do believe that everything is > correct, in "theory". So in case this is something broken with my WSL2 > installation, I'd encourage anyone with a WSL2 installation to try it out. > > /Magnus >
