On 2018-12-13 08:48, Andrew Luo wrote:
Hi,
I attached the latest patch, which now can use Windows paths.
Great!
I looked at the code, and it looks really good. Just one request. I
understand why you want to unify the similar code between msys and
wsl, but unless you have actually verified on an msys system that this
does not break anything -- please do not do it. This entire system of
getting Windows environments to behave is very brittle, and even
things that looks like they "should" work, often turn out not to do so
in practice. So even though code duplication is a bad thing in
general, in this case I'd rather see that you just added the support
for WSL, instead of changing the old code. Unless, of course, you have
verified that it does not break msys.
I can also add that the code here is really horrendous. I'm sure
there's a more efficient way of achieving what we want -- sane,
space-free, universally usable paths that can be easily turned into
windows paths by fixpath, but there's been too many corner-cases, too
much of "oh no, now it breaks on cygwin if the code is in the users
home dir!" nasty surprises. Solving this properly would probably
involve creating some automated test that can be run on all two (soon
three) Windows unix environment. And I've never felt it worth it. So
it's been a lot of "well, just add this rewrite here too, just in
case, see, now it works!". I'm not proud of it, but it does it's thing.
(I also note that you didn't care about tr:ing the 8.3 path to lower
case. It's of course a matter of taste, but since the goal is to
transform the path to a unix-style path, I tend to think that a path
like /mnt/c/progra~1/micros~1/vc/cl.exe is more easier to the eye than
/mnt/c/PROGRA~1/MICROS~1/VC/cl.exe or whatever.)
The new instructions to build are (assuming 8.3 paths are enabled
on your system...):
Right, it's possible to disable 8.3 paths nowadays, yes? We should
probably add that to the build documentation.
1. wsl must be started from a Windows Developer command prompt. To
ensure the correct environment variables are propagated from Windows
to WSL, you can run the following commands:
set WSLENV=INCLUDE/l:LIBPATH/l
2. Start wsl (bash):
wsl
3. Run configure:
./configure
--with-boot-jdk=/mnt/c/Users/Andrew/Downloads/openjdk-11.0.1_windows-
x64_bin/jdk-11.0.1 --with-tools-dir="C:\Program Files (x86)\Microsoft
Visual Studio\2017\Enterprise\VC\Auxiliary"
--with-ucrt-dll-dir="C:\Program Files (x86)\Windows
Kits\10\Redist\ucrt\DLLs\x64"
4. Run make
I’ve tested make with the default target as well as “make images”
Great, those are much simplified build instructions! If you are happy
with them, I can end here. However, I can't refrain from at least
mentioning that we do have code in place that should make even a lot
of these steps unnecessary. It's up to you if you want to make the
additional effort to adapt them to the WSL environment. In order:
1) You should not have to start the Developer command prompt, nor
export the INCLUDE and LIBPATH environment variables. We locate the
vcenv.bat file (or whatever it's called), run this, and then extract
the relevant environment variables. But, maybe that is not possible
with that kind of env sharing between bat files and the unix
environment in WSL?
2) We should be able to locate the relevant Visual Studio installation
and associated helper dll:s automatically, if if is installed in a
standard location. If the path rewriting is now working properly, I
see no reason why this would not work under WSL as well.
3) The official stance is that only unix-style paths is allowed as
argument to configure flags. Otherwise we can't do things like read
the value of the flag and check if the file exists. For certain
arguments, this might work anyway, out of pure chance. But it's not
recommended. So if it ends up that you really need to point to the
visual studio installation, the example in the build confiuration
should be something like "--with-tools-dir="/mnt/c/Program Files
(x86)/Microsoft Visual Studio/2017/Enterprise/VC/Auxiliary".
The issues regarding the console input redirection I'm still
investigating, but I doubt it's a bug in the build scripts, meaning
it is likely a bug in the (boot) JDK or WSL. Even if we fix the JDK,
we probably still have to support older boot JDKs (even if the patch
is backported), and waiting for Microsoft to fix a bug in WSL could
take a while (and might only be fixed in a later version of Windows).
Thus, I think what we have is a good start, unless you think it's
necessary to root cause the redirection issue first. That said, I
will keep this thread updated with my progress once I figure out the
root cause of the issue.
No, it's not necessary to find out the root cause. It would be nice to
know, but if the issue is only involving these two tools, and it
happens deterministically if it happens, I'm fine. Especially since
the workaround was actually an improvement. :-)
/Magnus
Thanks,
-Andrew
-----Original Message-----
From: Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>
Sent: Wednesday, December 12, 2018 10:54 AM
To: Erik Joelsson <erik.joels...@oracle.com>; Andrew Luo
<andrewluotechnolog...@outlook.com>; build-dev@openjdk.java.net
Subject: Re: [PATCH] Support for building using WSL (Windows
Subsystem for Linux) on Windows
On 2018-12-12 18:30, Erik Joelsson wrote:
Hello,
I had the same trouble you describe trying to call cmd to create a
short path from WSL with an inline script. I managed to it working
by creating a script file like this:
shortName.cmd:
---
@ECHO OFF
if '%1' NEQ '' echo %~s1
---
$ /mnt/c/Windows/System32/cmd.exe /c shortName.cmd "C:\Program Files"
C:\PROGRA~1
We could put such a script in make/scripts.
That's a clever workaround. Andrew, can you see if you can use that
technique to get the proper space safe path resolution to work? I
think you can copy the msys code straight as it is, and just replace
the call to cmd echo %~sA with cmd /c
$TOPDIR/make/script/shortName.cmd, or whatever you end up calling it.
/Erik
On 2018-12-11 22:44, Andrew Luo wrote:
For the stdin/stdout redirection: basically, the issue was only
occurring with those two executables. Oddly enough, it would occur
every time I tried (for SPP the output would be cutoff, presumably
because the input was cutoff, and for the other executable,
available0 would throw an exception consistently for System.in). I
have a hunch this is due to using WINAPI console functions for
reading from a WSL shell, but I'm not 100% certain. I plan to try
to build a smaller sample that can reproduce the issue, and if it
seems to be a Windows bug I will file a bug with Microsoft.
So what you are saying is that the issue was not intermittent, but
always happened for those tools? If so, I can reluctantly agree to
this fix. But I'd appreciate if you could drill down a bit and see
what the problem really is.
/Magnus
As for the short paths, calling cmd.exe from WSL bash seems to be a
bit buggy. cmd.exe, when called from a standard Windows
environment, works properly and prints the path, however, when
called from WSL, I
get:
"(C:\Program Files (x86))" was unexpected at this time.
I verified that the correct path was being passed to cmd.exe (not a
bash escaping issue) by creating my own test executable (C++) that
just prints argv[0] ... argv[argc - 1] and when running my text
executable from both Windows and WSL I get the same result, however
when running cmd.exe with the exact same arguments, it works in
Windows but not WSL. I will file a bug with Microsoft for this issue.
Thanks,
-Andrew
-----Original Message-----
From: Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>
Sent: Tuesday, December 11, 2018 6:18 AM
To: Andrew Luo <andrewluotechnolog...@outlook.com>; Erik Joelsson
<erik.joels...@oracle.com>; build-dev@openjdk.java.net
Subject: Re: [PATCH] Support for building using WSL (Windows
Subsystem for Linux) on Windows
On 2018-12-11 14:37, Magnus Ihse Bursie wrote:
On 2018-12-11 06:25, Andrew Luo wrote:
Hi,
Yes, I've signed an OCA (I've also contributed changes to other
groups before, but not build).
Okay, I have fixed the autconf-config* files.
Unfortunately, as Erik mentioned, there is no
(supported/reliable) way to access the WSL root / from
/cygdrive/c, or even from Windows (there is a way in reality,
however, it's not documented/supported by Microsoft and the
location changes depending on the distribution/store app id/etc.
so best to avoid using it.) I can see if we can print information about
versions however.
Right, WSL requires the .exe extension when accessing an
executable, as this is Linux behavior (Linux doesn't have
extensions for executables generally, but that's besides the
point)...
I fixed BASIC_REMOVE_SYMBOLIC_LINKS - a leftover from another
approach I tried.
For the redirect, redirect doesn't seem to be working when you
have a bash shell input piped into a Win32 executable reading
from stdin using WINAPI. I'm not sure this is supported by the
OpenJDK, more likely it might be a Microsoft issue. For some
reason, the stdin would be cut off (or I would see an exception
thrown from
available0 in FileInputStream). I personally didn't see any harm
in changing piping into input/output files (since all the
inputs/outputs are files anyways!).
Ok, let me be sure I get this right. It is only the redirect of
*input* that fails? (But you fixed both because of consistency). I
agree that the change itself is fine, even better than it is right
now
-- I was mostly worried about the consequences of redirects is not
working; there might be other places that fail. But if redirecting
output works, I think we're mostly fine. That's something we do
all the time, for each executed command, so if that did not work
reliably it would be really bad.
But still... I tried greping for "<" and there's a lot of places,
20+, that redirects input.
Or did this problem only happen when running *java* as the
recipient of the redirected input?
This worries me, and while I do think your change makes the tools
have a better UI, I don't like this as a workaround that will not
solve all potential problems. :(
I fixed TOOLCHAIN_FIND_VISUAL_STUDIO_BAT_FILE - this was also
from a few things I had tried earlier.
I disabled the $BASH code because to call bash from Win32 the
correct way is either "wsl /bin/bash" or just "bash". $BASH
correctly evaluates to /bin/bash, however
BASIC_WINDOWS_REWRITE_AS_WINDOWS_MIXED_PATH is implemented in
terms of wslpath, which can only convert a path under /mnt/c back
to a Windows path. Other files under /, for example /bin and
/bin/bash, cannot be converted to a Windwos path.
The escaping changes I made because it wasn't working. This does
work with spaces in the path on WSL. I don't have a Cygwin
environment to check, perhaps someone else here could help out?
Otherwise I can refactor that code to use that echo statement for
WSL and use the old echo statement for Cygwin.
I can check it out the next time I'm on a Windows machine.
I have fixed the extraneous debug print statement.
As for Windows vs Linux output - you can still force it to build
a Linux output binary. You just need to run configure as follows:
./configure --with-boot-jdk=/home/andrew/jdk-11.0.1
----build=x86_64-unknown-linux-gnu
--host=x86_64-unknown-linux-gnu
However, there is a behavior change: now, on WSL, by default,
Windows binaries are targeted. Previously, Linux binaries were
the default target. (Also, you can run configure twice and two
sets of configurations will be generated, you can actually build
both images by setting CONF=linux-x86_64-server-release or
CONF=windows-x86_64-server-release)
If you run on WLS, it's reasonable that the default is Windows.
The --build --host combo is good enough for me as a way to force a
linux build; we don't need an extra flag for this somewhat odd
build configuration.
As for BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN, wslpath does not
support
8.3 names. But perhaps the symlink workaround is acceptable for
now and we can handle the 8.3 naming on WSL in a separate change,
what do you guys think - personally I think what we have
(assuming Cygwin still works) is at least a MVP for WSL devs.
Anyways, at least some people may have to use the symlink
workaround if they've disabled 8.3 on NTFS.
That's too bad, since it really helped with getting around the
issue with spaces in path that's mandatory on Windows using
default installation of Visual Studio. :(
Again, sorry if I don't know enough about WSL to know if this is
possible, but on msys we do the following:
new_path=`cmd /c "for %A in (\"$input_path\") do @echo
%~sA"|$TR \\\\\\\\ / | $TR 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
'abcdefghijklmnopqrstuvwxyz'`
That is, we call the Windows cmd.exe using the "%~sA" variable
syntax to print the 8.3 version of the path (input_path is a
"normal" Windows path). Is there any way it's possible to do this
on WSL? It seems reasonable that you should be able to call
cmd.exe and redirect the output.
I think it will be worth trying to jump through some loops or
doing some dirty tricks to get this to work, because everything
will be
*soooo* much simpler if you can reliably turn paths into
space-safe paths; our normal Windows build depends on it.
I also realized that if you make a WSL version of
BASIC_FIXUP_EXECUTABLE, then you might be able to add .exe in it.
You will still need the EXE_SUFFIX when e.g. looking for the
java.exe binary in locating the Boot JDK, but perhaps it will
simplify some of the places.
I see now that the call to java in Images.gmk really should have
been prefixed with $(FIXPATH) instead. Then you would not have
needed the EXE_SUFFIX fix there.
Also, I suggest you look more closely on how we do things on msys
than on cygwin; I have the feeling wsl is closer to msys (in terms
of functionality from our perspective) than cygwin.
/Magnus
Attaching my latest patch (generated using powershell, so to
properly import you may need to convert UTF16 -> UTF8 and change
CRLF to just LF, hg tends to be picky)...
Looks much better now!
Before accepting it, I'd like to understand more about the redirect
issue, and see if there really is no way to rewrite the paths in a
space-safe manner. Then I think this is good to go.
/Magnus
Thanks,
-Andrew
-----Original Message-----
From: Erik Joelsson <erik.joels...@oracle.com>
Sent: Monday, December 10, 2018 9:19 AM
To: Magnus Ihse Bursie <magnus.ihse.bur...@oracle.com>; Andrew
Luo <andrewluotechnolog...@outlook.com>;
build-dev@openjdk.java.net
Subject: Re: [PATCH] Support for building using WSL (Windows
Subsystem for Linux) on Windows
Hello,
On 2018-12-10 02:06, Magnus Ihse Bursie wrote:
On 2018-12-09 20:11, Andrew Luo wrote:
One important thing to note is that the WSL build targets Windows.
It is also possible to use WSL to target itself (a WSL Linux
binary) or even other distributions of Linux. I have not
implemented that yet, but I think I could do that as a next
step if you guys think it would be useful (at least I think it
would be useful, then you can test your changes for both
Windows and Linux on one system...).
I think if you just run configure ordinarily, it will behave
like a Linux system and build the Linux image right out-of-the-box..?
But then again, maybe that behavior is negated by your changes
to config.guess and platform.m4. So maybe we need a flag to
configure to control this...
It is indeed possible to build a pure Linux binary in WSL today
so I think it would be bad to lose that functionality. We
certainly need a configure flag to control if a Windows or Linux
build should be produced in this case. This is something I have
been thinking about when I started tackling WSL builds some time
ago but didn't really come up with a good solution. I didn't have
the time to spend to really see it through though, so it's nice
to see that someone else is trying.
We could simply use the --with-openjdk-target, that would perhaps
be the cleanest, but it's also a bit cumbersome. We may need some
simplification similar to how we have --with-target-bits=32/64 as
a simple switch (e.g. --with-wsl-target=linux/windows?).
Steps in case you want to try this out:
1. Due to autotools not handling spaces well, you have to
create symlinks in Windows that will allow you to access
Windows Kits and the VC++ compiler without spaces in the path:
mklink /D C:\VS "C:\Program Files (x86)\Microsoft Visual Studio"
mklink /D C:\WindowsKits "C:\Program Files (x86)\Windows Kits"
That's a bit odd. We encounter spaces in paths on Windows
normally on cygwin and msys, and that works fine. I suspect
there is something missing with the rewriting functions. What we
do, is that we rewrite paths with spaces to paths without
spaces, by using the old 8+3 compatibility names, so we get
something like "/cygdrive/c/progra~1/microso~2" from "C:\Program
Files (x86)\Microsoft Visual Studio". Have a look at
BASIC_MAKE_WINDOWS_SPACE_SAFE_CYGWIN. I think you need a WSL
version of that, as well as of BASIC_FIXUP_PATH_CYGWIN. (And you
need to call the BASIC_FIXUP_PATH_WSL from BASIC_FIXUP_PATH.)
If you get these parts right, I don't think you will need any of
the special instructions below to build. In fact, as long as
C:\... is properly remapped, the normal VS autodetect code
should work just fine. And perhaps you can even revert some of
the scarier changes in toolchain_windows.m4.
I definitely agree with Magnus that to make WSL truly supported,
the path handling macros need to be replicated. I'm not sure how
to solve it properly. The root path Magnus is asking for is not
defined in WSL. In fact, from windows you cannot reach any path
in the WSL filesystem. Only Windows drives are mounted in WSL,
not the other way around. To convert to old style paths in Cygwin
we rely on the cygpath utility. There is a wslpath utility but
does it support old style path conversions? If not, maybe it's
possible to write such a tool in CMD/PowerShell?
/Erik
2. wsl must be started from a Windows Developer command
prompt. To ensure the correct environment variables are
propagated from Windows to WSL, you can run the following
commands:
set WSLENV=INCLUDE/l:LIBPATH/l
3. Start wsl (bash):
wsl
4. After starting bash you must set your compiler
variables to explicitly point to the correct tools:
export
AR=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Host
x6
4/
x64/lib.exe
export
CC=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Host
x6
4/
x64/cl.exe
export
CXX=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Hos
tx
64
/x64/cl.exe
export
LD=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin/Host
x6
4/
x64/link.exe
export RC=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/rc.exe
export MT=/mnt/c/WindowsKits/10/bin/10.0.17763.0/x64/mt.exe
export
DUMPBIN=/mnt/c/VS/2017/Enterprise/VC/Tools/MSVC/14.16.27023/bin
/H
os
tx64/x64/dumpbin.exe
5. Run configure:
./configure
--with-boot-jdk=/mnt/c/Users/Andrew/Downloads/openjdk-11.0.1_wi
nd
ow
s-x64_bin/jdk-11.0.1
--with-tools-dir="C:\VS\2017\Enterprise\VC\Auxiliary"
--with-ucrt-dll-dir="/mnt/c/WindowsKits/10/Redist/ucrt/DLLs/x64"
6. Run make
I've tested make with the default target as well as "make images"
Let me know if you have any feedback/comments.
Thanks,
-Andrew
