Just a quick reply (I haven't read the details yet, will do so and
re-reply if I find anything more to add).
It's important to differentiate two different aspects here.
1) Using msys2 instead of cygwin
2) Compiling using gcc on windows.
1 should be fairly trivial. We once supported msys, and it just recently
bitrotted due to lack of testing. I don't see any good reasons not to
revive this support.
2 has never been done before, and can probably be quite tricky. I'm not
sure what to gain from it, but if someone is prepared to go through the
effort of fixing this, I think it can be a quality boost to the code
base -- right now we don't properly differentiate between "windows" and
"microsoft toolchain", and this effort would help us clean up that mess.
But this is likely a huge undertaking, the prelimininary results showed
multiple tier1 failures, which indicates (I believe) subtle issues in
the compiled hotspot code.
/Magnus
On 2017-11-27 14:36, Thomas Stüfe wrote:
Hi Peter,
sorry if I came over too harsh. I gave this some thoughts and now I think
getting the openjdk to build with gcc on Windows may be beneficial. Please
find more comments inline.
On Mon, Nov 27, 2017 at 12:12 PM, Peter Budai <peterbu...@hotmail.com>
wrote:
MSYS2 or Cygwin should be similar from the technical point of view IMHO.
I’m familiar with MSYS2, but not with Cygwin.
I would prefer cygwin, as this is the officially adopted tool chain, and
the work is already done and we need to maintain it anyway. I can see the
value in having a different compiler, but not in having a different tool
chain. Again, my personal opinions, others may think differently.
A quick grep through our sources yields a number of hits on cygwin specific
code. Some are really cygwin specific (e.g. calls to cygpath.exe to
calculate the cygwin specific path) whereas others seem to just mean "any
posix like windows environment as opposed to native windows cmd.exe". If we
add msys2 to the list, would we not have to revise all these places?
The big change as you called out is to make sure that gcc toolchain can
build successfully also for a Windows platform and produce a properly
working binaries.
BTW I don’t belie there are big performance differences whether you are
building with GCC or MSC
Maybe you are right on the performance front, but I personally would still
be careful using gcc as a compiler for production builds. On one hand I
have an established tool chain with a company behind it I can pay for
support if something goes wrong. On the other hand a quiet exotic solution
for this platform.
Once again, please advise on what is the best way to move forward, or stop
this effort.
It is not my place to advice you or to decide anything. People from the
build group would have to decide whether we want to add support for
msys2+gcc and whether we want to support it.
..Thomas
Peter
------------------------------
*From:* Thomas Stüfe <thomas.stu...@gmail.com>
*Sent:* Thursday, November 23, 2017 10:36:07 PM
*To:* Lussier, Denis
*Cc:* Jonathan Gibbons; build-dev@openjdk.java.net; Peter Budai
*Subject:* Re: Building OpenJDK9 on MSYS2
On Thu, Nov 23, 2017 at 12:40 AM, Lussier, Denis <den...@bigsql.com>
wrote:
This is good stuff. Back in 2010 I got OpenJDK compiled and running on
the
Microsoft Toolchain. I never ran the jtreg tests, but, the binaries
worked for running basic Tomcat but definitely not for trying to run
Eclipse.
These days I am a pretty big fan of MSYS2. In my personal experiences
with
building a PostgreSQL Windoze distribution, the MSYS2 binaries run a
little
faster than using the old MS Toolchain.
https://www.openscg.com/2016/02/building-postgresql-on-windo
ws-for-performance/
Well, the linked article does claim the msys2 toolchain to be faster than
the Visual Studio, but there are no information I can see which compiler
and linker options were used. Without this information, the article is
difficult to take seriously.
While I really respect these efforts, I regret seeing the fragmentation in
windows build environments. I rather have one well kept build environment
than of multiple buggy ones - and we have not that many folks interested in
the windows port in the first place (it always feels like second place to
the Posix platforms to me). The official build environment of the openjdk
windows port is Cygwin + Visual Studio. Even if we think the gcc toolchain
would be an attractive alternative to VS - why not just use the cygwin gcc
instead? That way we would at least share the same posix layer.
Just my 5c.
Kind Regards, Thomas
--Luss
On Wed, Nov 22, 2017 at 4:28 PM, Jonathan Gibbons <
jonathan.gibb...@oracle.com> wrote:
Peter,
With over 120 :tier1 test failures, it would be worth understanding the
test failures as part of moving forward. There should normally be no
:tier1
test failures.
-- Jon
On 11/22/2017 01:10 PM, Peter Budai wrote:
Let me give an update on the current status:
* I have managed to build both the 64 and the 32-bit version.
* The 64-bit version seems reasonable:
==============================
Test summary - w64-bit
==============================
TEST TOTAL PASS FAIL
ERROR
jtreg:jdk/test:tier1 1645 1610 34
1
<<
jtreg:langtools/test:tier1 3907 3819 79
9 <<
jtreg:nashorn/test:tier1 0 0
0
0
jtreg:jaxp/test:tier1 0 0 0
0
==============================
* However the 32-bit version of hotspot contains a bunch of inline
assembly pieces which uses an MSC specific syntax, and my ASM
knowledge is
really limited. I have managed to get a many inline assembly codes
from the
Linux version, but there are few pieces where I was not able to find a
good
solution yet, so although the 32-bit version compiles, unfortunately
it is
not working
* I have installed VS Express 2013 and made sure that the
“traditional” Microsoft toolchain is still working with the changes
* My OCA has been processed
I have a changeset which cleanly applies to JDK9 v181, but I recall you
have mentioned earlier that you would like to consider this to JDK10
So what is the best way to open a PR for review? This page:
http://openjdk.java.net/contribute/ says I should send the changeset
as
attachment, but earlier somebody mentioned here that attachments are
removed from the developer mailing lists.
Best regards,
Peter
From: Erik Joelsson<mailto:erik.joels...@oracle.com>
Sent: Thursday, November 2, 2017 1:07 AM
To: Peter Budai<mailto:peterbu...@hotmail.com>; Magnus Ihse
Bursie<mailto:magnus.ihse.bur...@oracle.com>
Cc: build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
I would say a few hours is way too long for tier1, but it of course
depends on the hardware used. We typically run it on windows in less
than 30 minutes with concurrency set to 6. Not sure what concurrency
you
used by default.
/Erik
On 2017-10-27 04:33, Peter Budai wrote:
Hi Magnus, after a little poking I managed to install and use jtreg,
thanks for the guidance.
Make run-test-tier1 resulted a pretty OK result for a first try, at
least for run-test-jdk:
Test results: passed: 1,610; failed: 34; error: 1
It took a few hors to run – is that normal?
I’ll review the patchset, and then share that with you.
P.
From: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
Sent: Thursday, October 26, 2017 11:00 AM
To: Peter Budai<mailto:peterbu...@hotmail.com>
Cc: build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
On 2017-10-26 00:01, Peter Budai wrote:
OK, I have found what was missing, it was actually my fault with a
missing exception handler.
So finally OpenJDK build has finished on Windows using gcc toolchain
running in MSYS2/MINGW64 shell. I ran hotspot unit tests, and it looks
promising:
./build/windows-x86_64-normal-server-fastdebug/hotspot/varia
nt-server/libjvm/gtest/gtestLauncher.exe
--jdk=/home/peterbud/jdk9/buil
d/windows-x86_64-normal-server-fastdebug/jdk
….
….
….
[----------] Global test environment tear-down
[==========] 346 tests from 54 test cases ran. (3859 ms total)
[ PASSED ] 346 tests.
I'm impressed! :-)
Would you care to share your current patchset, just to still my
curiosity? :-)
What is the best way to test the current build more thoroughly?
"make run-test-tier1". As Erik says, you'll need jtreg, and call
"configure --with-jtreg=...". You can get it from the Adopt OpenJDK
group
here: https://adopt-openjdk.ci.cloudbees.com/job/jtreg/lastSuccess
fulBuild/artifact/
/Magnus
P.
From: Bob Vandette<mailto:bob.vande...@oracle.com>
Sent: Tuesday, October 24, 2017 8:10 PM
To: Peter Budai<mailto:peterbu...@hotmail.com>
Cc: David Holmes<mailto:david.hol...@oracle.com>; Erik
Joelsson<mailto:
erik.joels...@oracle.com>; Magnus Ihse Bursie<mailto:magnus.ihse.burs
i...@oracle.com>; build-dev@openjdk.java.net<mailto:
build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
Can you provide some details about your toolchain, processor and os
plus
a dump of the native instructions around the SEGV. This might give us
enough info to be able to figure out what’s going on.
Bob.
On Oct 24, 2017, at 1:21 PM, Peter Budai <peterbu...@hotmail.com
<mailto:
peterbu...@hotmail.com>> wrote:
I get that error running in the debugger but also running
without/outside of the debugger as well.
I saw the comment in the code about generating SEGV in function
generate_get_cpu_info(), however the debugger can execute that, and
the
SEGV I experience is coming later in the
VM_Version::get_processor_features()
function
Peter
From: Bob Vandette<mailto:bob.vande...@oracle.com>
Sent: Tuesday, October 24, 2017 6:28 PM
To: Peter Budai<mailto:peterbu...@hotmail.com>
Cc: David Holmes<mailto:david.hol...@oracle.com>; Erik
Joelsson<mailto:
erik.joels...@oracle.com>; Magnus Ihse Bursie<mailto:magnus.ihse.burs
i...@oracle.com>; build-dev@openjdk.java.net<mailto:
build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
Was this a SEGV while you were running the debugger?
There is an intentional SEGV generated. This is used to determine if
we
can use some of newer
CPU features. Try to allow the SEGV to continue to see if you run
normally.
Bob.
On Oct 24, 2017, at 11:37 AM, Peter Budai <peterbu...@hotmail.com
<mailto
:peterbu...@hotmail.com>> wrote:
It seems that the compile is progressing well, I have 49
executables/tools and 38 compiled shared libraries already in the JDK
folder.
I have tried to run the product with the simplest ‘java -version’
command, however I get a SIGSEGV at get_cpu_info_stub() in
vm_version_x86.cpp, VM_Version::get_processor_features()
get_cpu_info_stub(&_cpuid_info);
Thread 5 received signal SIGSEGV, Segmentation fault.
0x000000002d9a072f in ?? ()
I can debug using gdb, but unfortunately this is pure ASM, and my
knowledge on this is close to 0.
Any idea help or pointer how could I tackle this?
Peter
From: David Holmes<mailto:david.hol...@oracle.com>
Sent: Sunday, October 15, 2017 10:37 PM
To: Peter Budai<mailto:peterbu...@hotmail.com>; Erik
Joelsson<mailto:
erik.joels...@oracle.com>; Magnus Ihse Bursie<mailto:
magnus.ihse.burs
i...@oracle.com>
Cc: build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net
<mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
On 16/10/2017 12:41 AM, Peter Budai wrote:
A quick status update:
* Hotspot successfully compiled without warnings
* I’d like to run the unit tests, but as I see ‘make check’
does
not work, and gtestlauncher expects a command line parameter jdk.
Tried to
look up some documentation on this, but have not found. So the
question is:
how can I run unit tests for hotspot? Do I need also JDK compiled
for that?
Or the bootJDK is good enough? Any help would be appreciated.
Hotspot has to be tested as part of a full JDK - you can't load the
JVM
without having the "J" part :)
You should be able to drop your built dll into an existing JDK 9
windows
JDK and test it that way.
David
-----
* Also I’m making progress on compiling jdk, but there are some
very interesting solutions on windows linking which makes a bit more
difficult to compile with gcc: LIBS_windows contains sometimes
simple
library names (which I believe is correct) and other times library
names
with full path (which I believe is not the best solution). I’m
trying to
rework those places and use simple library names and passing search
path
for libraries -L<path> (for gcc toolchain) and /LIBPATH:<path> (for
Microsoft toolchain). Also I was surprised by a few manual function
name
exports…
* jdk code base contains apparently more MSVC specific part,
many places casts/lack of casts are generating errors, static
attributes
were missing etc. a bit tedious work.
P.
From: Erik Joelsson<mailto:erik.joels...@oracle.com>
Sent: Wednesday, October 11, 2017 4:16 PM
To: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>; Peter
Budai<mailto:peterbu...@hotmail.com>
Cc: build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net
<mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
Hello,
On 2017-10-11 15:48, Magnus Ihse Bursie wrote:
For gcc, we let the compiler generate the .d file. For the Microsoft
tool chain, we use a clever sed script to extract and create it
ourself.
I think that logic is checking for "Windows", not "Microsoft". That
might be your cause of trouble.
Look in NativeCompilation.gmk.
That was my initial thought as well, but we do correctly check for
microsoft. Also it's not the .d files that are the problem. As
Peter just
wrote, they look fine. It's the .d.target files which we create
using the
same technique on all platforms. What we don't account for is the
compiler
putting Windows mixed paths in the .d files.
/Magnus
11 okt. 2017 kl. 14:43 skrev Peter Budai <peterbu...@hotmail.com
<mailto:peterbu...@hotmail.com><mailto:peterbu...@hotmail.com>>:
Hi Erik,
The .d file looks like this:
C:/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/hotspot/variant-server/tools/adlc/objs/adlparse.obj: \
C:/msys64/home/peterbud/jdk9/hotspot/src/share/vm/adlc/adlparse.cpp
\
I have checked .d.targets file, and looks like it has the first line
has not been deleted, and the file names below are also wrong:
/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-serve
r-release/hotspot/variant-server/tools/adlc/objs/adlparse.obj : :
/msys64/home/peterbud/jdk9/hotspot/src/share/vm/adlc/adlparse.cpp :
/msys64/home/peterbud/jdk9/hotspot/src/share/vm/adlc/adlc.hpp :
I guess this part in the DEPENDENCY_TARGET_SED_PATTERN is fooled by
the “C:/”
-e 's/^[^:]*: *//'
Yes, that does indeed look like the problem. I suppose the regexp is
unnecessarily strict. It should be ok to rewrite it as something
like this:
-e 's/^.*: *//'
Basically just make sure it ends with : and any number of spaces.
/Erik
Peter
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
From: Erik Joelsson<mailto:erik.joels...@oracle.com>
Sent: Wednesday, October 11, 2017 12:16 PM
To: Peter Budai<mailto:peterbu...@hotmail.com>; Magnus Ihse
Bursie<mailto:magnus.ihse.bur...@oracle.com>;
build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net
<mailto:
build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
Hello Peter,
On 2017-10-11 00:18, Peter Budai wrote:
Thanks Magnus & Erik
First thanks for your support and kind words!
Magnus, I have checked .bash_profile, .bashrc but they seem to be
empty (everything is commented out). You can check with a default
MSYS2
install, I have not changed these files at all. If you find thee
something
specific I can give a try here as well.
Let me give also a quick status update, where am I with building
hotspot:
· I guess its still the beginning, but I have managed to
compile
jvm.dll with almost 700 object file: with debug info the dll is
around 700
MB 😊
· I made only surgical, minimal changes to the source, and so
far it looks reasonable. I have encountered 3 scenarios where
changes were
necessary:
o When in makefiles conditionals were using assuming that if
target_os is windows then it is visual studio compiler/linker.
Obviously
these conditionals had to be reviewed in a few places and if
necessary were
changes to check the toolchain=Microsoft
These are not surprising and should be pretty straight forward to
fix
and it seems you know what to do.
·
o I got a few warnings as gcc 7.2 uncovered some code problems in
windows specific codes, where before that MSVC I guess did not say
a word…
To get around this you can configure with
--disable-warnings-as-errors
until you get things working properly. This is commonly needed when
using
compiler versions that we normally don't use.
·
o And I had like 6-7 places where the code was using MSVC specific
__try … __except structures which gcc does not know. Do you have a
suggestion how to approach them? I can do ugly #ifdefs (I would
avoid that)
but I have also seen some solutions to replace them with a code
which gcc
can compile (http://www.programmingunlimit
ed.net/siteexec/content.cgi?
page=mingw-seh ) – but before doing that though I would ask first
you on the purpose of those
This kind of question is probably best to bring to the hotspot
mailing
list.
· What bothers me is that I was not able to do incremental
builds: when an error occurs, and build stops, then after making
change in
the CPP source the build cannot continue, I always got an error
message:
/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/hotspot/variant-server/tools/adlc/objs/adlparse.
d.targets:1:
*** missing target pattern. Stop.
make[2]: *** [make/Main.gmk:256: hotspot-server-gensrc] Error 2
If I do a ‘make clean’ and restart the build then it nicely
compiles.
Question 1: Is there a way to resume such builds without ‘make
clean’?
Well, incremental builds is supposed to work well. We have several
extra tricks in there to handle cases where normal make builds
would fail.
The *.d.targets files is one such trick and it seems to backfire
for you.
The contents of that file should be something like:
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/adlparse.cpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/adlc.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/opto/opcodes.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/opto/classes.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/arena.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/opto/adlcVMDeps.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/filebuff.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/dict2.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/forms.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/formsopt.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/formssel.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/archDesc.hpp :
/localhome/hg/jdk9-dev/hotspot/src/share/vm/adlc/adlparse.hpp :
Basically an empty rule for each dependency for the corresponding
object file. Declaring these rules makes it possible to delete
source files
without having to build clean. It seems your file is not generated
correctly so please have a look inside it. The file is in
make/common/NativeCompilation.gmk, look for
DEPENDENCY_TARGET_SED_PATTERN.
Question 2: What would be the best way to submit/share the patches
for
your thorough review?
Well, first of all, have you signed the OCA?
As for publishing patches and reviews, there is a bit of chicken and
egg problem. Once you become an "author" in any of the OpenJDK
projects,
you get a user name and should be able to publish reviews on
cr.openjdk.java.net<http://cr.openjdk.java.net/><http://cr.o
penjdk.java.net<http://cr.openjdk.java.net/<http://cr.openjd
k.java.net/%3e%3chttp:/cr.openjdk.java.net%3chttp:/cr.
openjdk.java.net/>>>. Before that, if the patch is small, it can be
posted inline in an email to the list. If it's large, you will need
a
current OpenJDK user to host it for you. At least that's how I
understand
it. Hopefully someone who knows the process better can chime in
here.
I should also let you know that getting this into JDK 9 is most
likely
not going to happen. AFAIK we are only doing security updates for
9. It
would have to go into the currently active release. I should also
warn you
that new ports generally need a certain amount of backing to be
accepted.
It may be that this would have to live in a porting side project.
Hopefully
someone who knows this better can chime in here as well.
/Erik
P.
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
From: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
Sent: Tuesday, October 10, 2017 10:04 AM
To: Peter Budai<mailto:peterbu...@hotmail.com>; Erik
Joelsson<mailto:
erik.joels...@oracle.com>; build-dev@openjdk.java.net<mailto:
build-dev@openjdk.java.net><mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
On 2017-10-07 10:14, Peter Budai wrote:
The configure of OpenJDK overwrites the SHELL. Actually it is using
bash, but for the arguments it was using “-e -o pipefail”. I have
figured
that for MSYS2 bash what is needed as bash arguments is “-e -l -c -o
pipefail”
That looks like solving this problem, and now the real issues are
surfacing.
FWIW, "-l" makes bash behave like a login shell. Most likely you are
changing bash's behavior in one of your login scripts, and that
change is
what's really needed.
/Magnus
Peter
From: Peter Budai<mailto:peterbu...@hotmail.com>
Sent: Friday, October 6, 2017 6:43 PM
To: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>; Erik
Joelsson<mailto:erik.joels...@oracle.com>;
build-dev@openjdk.java.net
<mailto:build-dev@openjdk.java.net><mailto:build-dev@openjdk
.java.net>
Subject: RE: Building OpenJDK9 on MSYS2
Magnus,
I have followed your suggestion and removed the fixpath prefixes
from
gcc related compile tools, and left only the fixpath prefix _only_
for the
Boot JDK related tools in place.
1) As I follow the process, all java and javac related compile
steps are running properly
2) When the process reaches gcc related steps I got the error
message at the same place as before (no fixpath). If I execute that
command
from the bash prompt, it creates the output:
$ ( /usr/bin/gawk '/@@END_COPYRIGHT@@/{exit}1'
/C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes
/sun/nio/ch/SocketOptionRegistry.java.template &&
/C/msys64/mingw64/bin/gcc -E -x c /C/msys64/home/peterbud/jdk9/j
dk/src/java.base/share/classes/sun/nio/ch/SocketOptionRegist
ry.java.template
2> >(/usr/bin/grep -v '^SocketOptionRegistry.java.template$' >&2) |
/usr/bin/gawk '/@@START_HERE@@/,0' | /usr/bin/sed -e
's/@@START_HERE@@/\/\/
AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' -e 's/PREFIX_//' -e
's/^#.*//'
) > /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/support/gensrc/java.base/sun/nio/ch/SocketOption
Registry.java
As I have mentioned the parameters are replaced by the bash
automatically
3) Then build continues, then little later stops at a super
simple command:
mv /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/support/gensrc/java.base/java/nio/ByteBuffer.java.tmp
/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/support/gensrc/java.base/java/nio/ByteBuffer.java
Needless to say, the ByteBuffer.java.tmp file DOES
exist.
And running the above command from the bash works, and build
continues.
4) A few similar cases (stops) with DirectByteBuffer and
DirectByteBufferR
Currently I try to explore how that might relate to the MSYS2 bash
and
make, somehow it behaves differently
If you have any other suggestion, let me know.
Best regards,
Peter
From: Peter Budai<mailto:peterbu...@hotmail.com>
Sent: Thursday, October 5, 2017 3:52 PM
To: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>; Erik
Joelsson<mailto:erik.joels...@oracle.com>;
build-dev@openjdk.java.net
<mailto:build-dev@openjdk.java.net><mailto:build-dev@openjdk
.java.net>
Subject: RE: Building OpenJDK9 on MSYS2
Hi Magnus,
So first of all, here is the current patch, which I was not able to
attach: https://pastebin.com/pwT4Ynxc
That's surprising, since gcc is prefixed with fixpath, which it
should
not.
Actually you DO need fixpath IMHO.
This is a mingw64 version of the gcc (/C/msys64/mingw64/bin/gcc),
which is a fully functional Windows executable, which expects
Windows
formatted path arguments.
As the updated build process uses EXPORT MSYS2_ARG_CONV_EXCL=* (see
that patch), none of the command line arguments are converted from
the
unix path to Windows, but fixpath does that conversion. There is a
wiki
describing more details on this:
https://github.com/msys2/msys2/wiki/Porting#user-content-
filesystem-namespaces
I have a hard time believing this is a race condition. On the other
hand, this stuff is weird, we're misusing the C preprocessor to
process
defines in java code, so I'm not surprised it breaks down.
I don't know why it succeeded when run on the command line,
though.
When I execute that command from the bash command line there is no
EXPORT MSYS2_ARG_CONV_EXCL, but the bash itself does the automatic
conversion of the arguments. Maybe it has something to do how
fixpath does
CreateProcess?
Does that help?
Best regards,
Peter
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
From: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
Sent: Thursday, October 5, 2017 12:13 PM
To: Peter Budai<mailto:peterbu...@hotmail.com>; Erik
Joelsson<mailto:
erik.joels...@oracle.com>; build-dev@openjdk.java.net<mailto:
build-dev@openjdk.java.net><mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
On 2017-10-05 11:59, Peter Budai wrote:
Hi Magnus and Erik,
I really appreciate your quick feedback. I assumed that it won’t be
easy, but I just don’t feel I should give up now - maybe later
when I see
the real scale of work. So bear with me for a time being.
Attached is a patch which already includes Magnus’ changes, plus a
few
which I have added:
· basically enabling gcc for windows,
· and modifying a logic for compiling fixpath (before that it
was using hard-coded MS VSC compile flags)
Actually, you must make sure fixpath is *not* used for the
toolchain,
since gcc uses unix style paths.
(However, other tools such as java will still need it.)
So here is what I have as the result of configure:
====================================================
The existing configuration has been successfully updated in
/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release
using configure arguments '--disable-freetype-bundling
--disable-javac-server'.
Configuration summary:
* Debug level: release
* HS debug level: product
* JDK variant: normal
* JVM variants: server
* OpenJDK target: OS: windows, CPU architecture: x86, address
length:
64
* Version string: 9-internal+0-adhoc.peterbud.jdk9 (9-internal)
Tools summary:
* Environment: msys version 2.9.0(0.318/5/3) (root at /C/msys64)
* Boot JDK: java version "1.8.0_144" Java(TM) SE Runtime
Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server
VM (build
25.144-b01, mixed mode) (at /c/progra~1/java/jdk18~1.0_1)
* Toolchain: gcc (GNU Compiler Collection)
* C Compiler: Version 7.2.0 (at /C/msys64/mingw64/bin/gcc)
* C++ Compiler: Version 7.2.0 (at /c/msys64/mingw64/bin/g++)
Build performance summary:
* Cores to use: 4
* Memory limit: 16216 MB
Its clear says that the toolchain is gcc 7.2 (BTW there is no Visual
Studio on this machine)
Now for the details of the config log, you can see here:
https://pastebin.com/MN2ZYcHH
And about the build process and the error I get:
$ make JOBS=1
Building target 'default (exploded-image)' in configuration
'windows-x86_64-normal-server-release'
Compiling 8 files for BUILD_TOOLS_LANGTOOLS
Compiling 17 properties into resource bundles for jdk.compiler
Parsing 1 properties into enum-like class for jdk.compiler
Compiling 19 properties into resource bundles for jdk.javadoc
Compiling 12 properties into resource bundles for jdk.jdeps
Compiling 7 properties into resource bundles for jdk.jshell
Compiling 117 files for BUILD_INTERIM_java.compiler
Compiling 396 files for BUILD_INTERIM_jdk.compiler
Compiling 61 files for BUILD_INTERIM_jdk.jdeps
Compiling 457 files for BUILD_INTERIM_jdk.javadoc
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Compiling 159 files for BUILD_TOOLS_JDK
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/b
uild/windows-x86_64-normal-server-release/support/gensrc/jav
a.base/sun/nio/ch/SocketOptionRegistry.java] Error 1
make[3]: *** Deleting file '/C/msys64/home/peterbud/jdk9/
build/windows-x86_64-normal-server-release/support/gensrc/ja
va.base/sun/nio/ch/SocketOptionRegistry.java'
make[2]: *** [make/Main.gmk:115: java.base-gensrc-jdk] Error 2
ERROR: Build failed for target 'default (exploded-image)' in
configuration 'windows-x86_64-normal-server-release' (exit code 2)
No indication of failed target found.
Hint: Try searching the build log for '] Error'.
Hint: See common/doc/building.html#troubleshooting for assistance.
make[1]: *** [/home/peterbud/jdk9/make/Init.gmk:296: main] Error 2
make: *** [/home/peterbud/jdk9/make/Init.gmk:185: default] Error 2
If I run here
make JOBS=1 LOG=debug
The failing line seems to be this:
( /usr/bin/gawk '/@@END_COPYRIGHT@@/{exit}1'
/C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes
/sun/nio/ch/SocketOptionRegistry.java.template &&
/C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/configure-support/bin/fixpath.exe
-m/C/msys64/@/c/msys64/@/c/progra~ /C/msys64/mingw64/bin/gcc -E -x
c
/C/msys64/home/peterbud/jdk9/jdk/src/java.base/share/classes
/sun/nio/ch/SocketOptionRegistry.java.template 2> >(/usr/bin/grep
-v
'^SocketOptionRegistry.java.template$' >&2) | /usr/bin/gawk
'/@@START_HERE@@/,0' | /usr/bin/sed -e 's/@@START_HERE@@/\/\/
AUTOMATICALLY GENERATED FILE - DO NOT EDIT/' -e 's/PREFIX_//' -e
's/^#.*//'
) > /C/msys64/home/peterbud/jdk9/build/windows-x86_64-normal-ser
ver-release/support/gensrc/java.base/sun/nio/ch/SocketOption
Registry.java
make[3]: *** [GensrcMisc.gmk:78: /C/msys64/home/peterbud/jdk9/b
uild/windows-x86_64-normal-server-release/support/gensrc/jav
a.base/sun/nio/ch/SocketOptionRegistry.java] Error 1
Now the interesting is: if I copy this line above to the bash
prompt,
it runs without problem, and the file support/gensrc/java.base/sun/n
io/ch/SocketOptionRegistry.java
That's surprising, since gcc is prefixed with fixpath, which it
should
not.
I have a hard time believing this is a race condition. On the other
hand, this stuff is weird, we're misusing the C preprocessor to
process
defines in java code, so I'm not suprised it breaks down. I don't
know why
it succeeded when run on the command line, though. My suggestion is
to just
do some quick and dirty hack around this: take the file you manage
to
generate and just copy it in during the build instead. If you can
get round
this, you might start seeing some *real* problems. :-)
Also, my suggestion is that you try running "make hotspot" to cut to
the chase. Compiling hotspot will likely be the hardest thing. Or
even
"make -k hotspot" to get an assessment of the amount of work ahead
of you.
/Magnus
Is produced.
Then I can again issue
make JOBS=1 LOG=debug
And the compile process is being continued, until a similar error
pops
up with a different generated file. I have an assumption that this
happens
because make is still running parallel jobs, despite JOBS=1 but I’m
not
sure.
How could I best tackle this?
Thank you and best regards,
Peter
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986> for
Windows 10
From: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
Sent: Thursday, October 5, 2017 11:33 AM
To: Erik Joelsson<mailto:erik.joels...@oracle.com>; Peter
Budai<mailto:peterbu...@hotmail.com>; build-dev@openjdk.java.net
<mai
lto:build-dev@openjdk.java.net><mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
On 2017-10-05 10:10, Erik Joelsson wrote:
Hello Peter,
On 2017-10-04 21:15, Peter Budai wrote:
Hi Magnus,
Thanks for the quick reply I’ll check these patches with msys2.
Let me specify with more details what I’d like to achieve: I’d
like
to build OpenJDK9 with MSYS2 MINGW64 environment using gcc
toolchain.
(I’m not sure how familiar are you with MSYS2, but there are 3
different environments: MSYS2, MINGW32 and MINGW64). In theory
MINGW64 with gcc is the closes you can get on Windows platform as
a
gcc unix like build environment, which produces still a native
64-bit
executable on Windows.
I’m not very familiar with OpenJDK yet, so therefore I’d like to
hear
your opinion: how realistic is that?
Sorry to disappoint, but I would say that requires major work.
There
is a strong historic assumption that windows builds are done using
Visual Studio. We have abstracted away some of it in configure (see
TOOLCHAIN_TYPE), but it's very far from enough to change compiler
environment for a Windows build. The native sources are also bound
to
make a lot of such assumptions. I would expect the changes needed
to
be in the thousands of lines of code.
I agree that it requires hard work (even if "thousands" might be an
overestimation I think, but "hundreds" is not enough, so it's the
right
magnitude). On the other hand, it would be really good if we did
sort
things out, so that we had proper conditions based on OS vs
compiler/toolchain.
If you really want to start, the first thing is to patch
toolchain.m4
to
VALID_TOOLCHAINS_windows="microsoft gcc"
and then call configure using "bash configure
--with-toolchain-type=gcc".
As Erik, I doubt you will come very far before things starts
tumbling
down.
When we say supporting the build in msys2 instead of cygwin, we just
mean using msys2 as the unix emulating layer for our tools like
make/bash/grep/sed etc.
One think I have done successfully is running the build in WSL
(Windows Subsystem for Linux), but that isn't all that helpful as
WSL
for practical purposes is more or less like running Linux in a VM,
so
the build sees a Linux system and builds a Linux binary.
As a side note: with MINGW64 I have managed to run configure phase
successfully for OpenJDK. The compile process has also started and
went for a while, but interestingly I run into some kind of race
conditions as make stopped with an error. Using LOG=debug I have
fond
the failing line and then copying the failed command and pasting
it
to the bash prompt it successfully generated the output target,
and
then the build process run further when a similar situation
happened.
Also pasting the failed command run in the bash without any
problem,
and build continued… until the next.
Without seeing the errors I can't say much. I very much doubt that
you
are running with gcc as the compiler though. Configure isn't easily
fooled into using a different compiler to what it prefers, and I
would
expect things to crash and burn pretty early if you actually did.
/Erik
I have tried to run make JOBS=1, but did not help, strangely I
have
still seen in the log make[3] and make[4] logs which suggested
that
there are more than one make jobs were running. Also tried
.configure
--with-output-sync=recurse without success (same symptoms)
Let me know your thoughts.
Best regards,
Peter
Sent from Mail<https://go.microsoft.com/fwlink/?LinkId=550986>
for
Windows 10
From: Magnus Ihse Bursie<mailto:magnus.ihse.bur...@oracle.com>
Sent: Wednesday, October 4, 2017 1:04 AM
To: Peter Budai<mailto:peterbu...@hotmail.com>;
build-dev@openjdk.java.net<mailto:build-dev@openjdk.java.net
<mailto:build-dev@openjdk.java.net><mailto:build-dev@openjd
k.java.net><mailto:build-dev@openjdk.java.net>
Subject: Re: Building OpenJDK9 on MSYS2
Actually, it wasn't so much remaining trouble. :-) I fired up
msys2
and
checked out where I left off. It turned out that the remaining
snag
was
that msys2 tries to convert command lines automatically, from
"unix"
style paths to "windows" style paths. Unfortunately, it does not
do
this
very well and it breaks all sorts of things. We already have a
FIXPATH
solution in place which deals with this, so basically all I had
to do
was disable this (by setting MSYS2_ARG_CONV_EXCL to "*"). However,
this
broke our cygpath replacement hack (!) so I had to disable it
there.
Sigh. Anyway, with those fixes it ran and worked well. (I also
discovered and fixed a bug related to how we set up the FIXPATH
variable
on msys, but it only triggers in certain circumstances).
With this patch I now jdk9 seems to build fine on msys2. It should
apply
cleanly on jdk9/jdk9. Since it turned out to be so trivial, I'll
try
to
get it in in jdk10.
Here's the patch if you want to apply it yourself:
diff -r a08cbfc0e4ec common/autoconf/basics_windows.m4
--- a/common/autoconf/basics_windows.m4 Thu Aug 03 18:56:56
2017
+0000
+++ b/common/autoconf/basics_windows.m4 Wed Oct 04 00:53:58
2017
+0200
@@ -42,7 +42,7 @@
windows_path=`$CYGPATH -m "$unix_path"`
$1="$windows_path"
elif test "x$OPENJDK_BUILD_OS_ENV" = "xwindows.msys"; then
- windows_path=`cmd //c echo $unix_path`
+ windows_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $unix_path`
$1="$windows_path"
fi
])
@@ -136,6 +136,16 @@
fi
])
+AC_DEFUN([BASIC_MSYS_UPDATE_FIXPATH],
+[
+ # Take all collected prefixes and turn them into a
-m/c/foo@/c/bar@... command line
+ # @ was chosen as separator to minimize risk of other tools
messing
around with it
+ all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:>@}" \
+ | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
+ fixpath_argument_list=`echo $all_unique_prefixes | tr ' ' '@'`
+ FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
+])
+
AC_DEFUN([BASIC_FIXUP_PATH_MSYS],
[
path="[$]$1"
@@ -143,7 +153,7 @@
new_path="$path"
if test "x$has_colon" = x; then
# Not in mixed or Windows style, start by that.
- new_path=`cmd //c echo $path`
+ new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $path`
fi
BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
@@ -155,6 +165,8 @@
# Save the first 10 bytes of this path to the storage, so
fixpath
can work.
all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
"${new_path:0:10}")
+ # We might need to re-evaluate FIXPATH.
+ BASIC_MSYS_UPDATE_FIXPATH
])
AC_DEFUN([BASIC_FIXUP_EXECUTABLE_CYGWIN],
@@ -293,7 +305,7 @@
# Do not save /bin paths to all_fixpath_prefixes!
else
# Not in mixed or Windows style, start by that.
- new_path=`cmd //c echo $new_path`
+ new_path=`MSYS2_ARG_CONV_EXCL= cmd //c echo $new_path`
BASIC_MAKE_WINDOWS_SPACE_SAFE_MSYS([$new_path])
# Output is in $new_path
BASIC_WINDOWS_REWRITE_AS_UNIX_PATH(new_path)
@@ -302,6 +314,8 @@
# Save the first 10 bytes of this path to the storage, so
fixpath
can work.
all_fixpath_prefixes=("${all_fixpath_prefixes@<:@@@:>@}"
"${new_path:0:10}")
+ # We might need to re-evaluate FIXPATH.
+ BASIC_MSYS_UPDATE_FIXPATH
fi
])
@@ -347,6 +361,10 @@
WINDOWS_ENV_VENDOR='msys'
WINDOWS_ENV_VERSION="$MSYS_VERSION"
+ # Prohibit msys2 path conversion from trying to be
"intelligent",
and rely
+ # on fixpath instead.
+ export MSYS2_ARG_CONV_EXCL="*"
+
AC_MSG_CHECKING([msys root directory as unix-style path])
# The cmd output ends with Windows line endings (CR/LF),
the grep
command will strip that away
MSYS_ROOT_PATH=`cd / ; cmd /c cd | $GREP ".*"`
@@ -391,10 +409,7 @@
elif test "x$OPENJDK_BUILD_OS_ENV" = xwindows.msys; then
# Take all collected prefixes and turn them into a
-m/c/foo@/c/bar@... command line
# @ was chosen as separator to minimize risk of other
tools
messing around with it
- all_unique_prefixes=`echo "${all_fixpath_prefixes@<:@@@:
@}"
\
- | tr ' ' '\n' | $GREP '^/./' | $SORT | $UNIQ`
- fixpath_argument_list=`echo $all_unique_prefixes | tr ' '
'@'`
- FIXPATH="$FIXPATH_BIN -m$fixpath_argument_list"
+ BASIC_MSYS_UPDATE_FIXPATH
fi
FIXPATH_SRC_W="$FIXPATH_SRC"
FIXPATH_BIN_W="$FIXPATH_BIN"
diff -r a08cbfc0e4ec common/autoconf/build-aux/config.sub
--- a/common/autoconf/build-aux/config.sub Thu Aug 03 18:56:56
2017 +0000
+++ b/common/autoconf/build-aux/config.sub Wed Oct 04 00:53:58
2017 +0200
@@ -30,7 +30,7 @@
DIR=`dirname $0`
# First, filter out everything that doesn't begin with
"aarch64-"
-if ! echo $* | grep '^aarch64-' >/dev/null ; then
+if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
. $DIR/autoconf-config.sub "$@"
# autoconf-config.sub exits, so we never reach here, but
just in
# case we do:
@@ -38,13 +38,17 @@
fi
while test $# -gt 0 ; do
- case $1 in
+ case $1 in
-- ) # Stop option processing
shift; break ;;
aarch64-* )
config=`echo $1 | sed 's/^aarch64-/arm-/'`
sub_args="$sub_args $config"
shift; ;;
+ *-msys )
+ config=`echo $1 | sed 's/msys/mingw32/'`
+ sub_args="$sub_args $config"
+ shift; ;;
- ) # Use stdin as input.
sub_args="$sub_args $1"
shift; break ;;
diff -r a08cbfc0e4ec common/autoconf/spec.gmk.in
--- a/common/autoconf/spec.gmk.in Thu Aug 03 18:56:56 2017
+0000
+++ b/common/autoconf/spec.gmk.in Wed Oct 04 00:53:58 2017
+0200
@@ -120,6 +120,13 @@
# On Windows, the Visual Studio toolchain needs the PATH
to be
adjusted
# to include Visual Studio tools (this needs to be in
cygwin/msys
style).
export PATH:=@VS_PATH@
+
+endif
+
+ifeq ($(OPENJDK_TARGET_OS_ENV), windows.msys)
+ # On msys2, prohibit msys path conversion from trying to be
+ # "intelligent", and rely on fixpath instead.
+ export MSYS2_ARG_CONV_EXCL:=*
endif
SYSROOT_CFLAGS := @SYSROOT_CFLAGS@
/Magnus
On 2017-10-03 22:34, Magnus Ihse Bursie wrote:
I gave msys2 a shot some time ago, but it ended up too much
trouble.
I'll share some of my notes from that attempt, for what it's
worth.
To install package X/Y, run "pacman -S X/Y". Missing tools and
packages where to find them:
cmp: msys/diffutils
tar: msys/tar
make: msys/make
unzip: msys/unzip
zip: msys/zip
config.sub reports msys as "x86_64-pc-mingw32" but msys2 as
"x86_64-pc-msys". This patch adds postprocessing in "our"
config.sub
to report msys2 similar to msys. (Opinions, including my own :-)
may
vary if this really is the best way..)
diff -r b88023f46daa common/autoconf/build-aux/config.sub
--- a/common/autoconf/build-aux/config.sub Fri Jan 27
10:15:41
2017 +0100
+++ b/common/autoconf/build-aux/config.sub Fri Feb 03
05:00:25
2017 -0700
@@ -30,7 +30,7 @@
DIR=`dirname $0`
# First, filter out everything that doesn't begin with
"aarch64-"
-if ! echo $* | grep '^aarch64-' >/dev/null ; then
+if ! echo $* | grep -e '^aarch64-' -e 'msys' >/dev/null ; then
. $DIR/autoconf-config.sub "$@"
# autoconf-config.sub exits, so we never reach here, but
just in
# case we do:
@@ -45,6 +45,10 @@
config=`echo $1 | sed 's/^aarch64-/arm-/'`
sub_args="$sub_args $config"
shift; ;;
+ *-msys )
+ config=`echo $1 | sed 's/msys/mingw32/'`
+ sub_args="$sub_args $config"
+ shift; ;;
- ) # Use stdin as input.
sub_args="$sub_args $1"
shift; break ;;
If I remember correctly, this got me past the configure stage at
the
time.
I don't think it's very hard to get it to work on msys2, I just
ran
into one snag too many and didn't think msys2 would be used by
anyone.
/Magnus
On 2017-10-03 17:20, Peter Budai wrote:
Hello,
According to
http://hg.openjdk.java.net/jdk9/jdk9/file/a08cbfc0e4ec/commo
n/doc/building.html
“msys2 and the new Windows Subsystem for Linux (WSL) would
likely
be
possible to support in a future version but that would require a
community effort to implement”
I’d like to help making the OpenJDK 9 build working on msys2.
What
is
the best way to move forward? Is there a similar effort in
progress?
Thank you and best regards,
Peter
