Re: [11] RFR(S) 8204113: Upgrade linker used in AOT tests to be same version as build toolchain
Hi Vladimir, The fix looks good to me. — Igor > On Jun 11, 2018, at 6:15 PM, Vladimir Kozlov > wrote: > > http://cr.openjdk.java.net/~kvn/8204113/webrev.01/ > https://bugs.openjdk.java.net/browse/JDK-8204113 > > AOT tests need a local linker to generate AOT libraries. They load devkits > from our infrastructure when there is no local linker on testing machine. The > names of devkits are hardcoded in AotCompiler.java (java wrapper which > launches 'jaotc' tool). > > We recently updated compilers for JDK 11. As result devkit names in > AotCompiler.java should be updated too. > > Tested by running AOT tests on all our supported platforms. > > -- > Thanks, > Vladimir
[11] RFR(S) 8204113: Upgrade linker used in AOT tests to be same version as build toolchain
http://cr.openjdk.java.net/~kvn/8204113/webrev.01/ https://bugs.openjdk.java.net/browse/JDK-8204113 AOT tests need a local linker to generate AOT libraries. They load devkits from our infrastructure when there is no local linker on testing machine. The names of devkits are hardcoded in AotCompiler.java (java wrapper which launches 'jaotc' tool). We recently updated compilers for JDK 11. As result devkit names in AotCompiler.java should be updated too. Tested by running AOT tests on all our supported platforms. -- Thanks, Vladimir
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
Looks good to me. /Jesper > On 11 Jun 2018, at 22:42, Erik Joelsson wrote: > > Hello, > > Based on the discussion here, I have reverted back to something more similar > to webrev.02, but with a few changes. Mainly fixing a bug that caused > JVM_FEATURES_hardened to not actually be the same as for server (if you have > custom additions in configure). I also added a check so that configure fails > if you try to enable either variant hardened or feature no-speculative-cti > and the flags aren't available. > > Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.05/index.html > > /Erik > > On 2018-06-11 00:10, Magnus Ihse Bursie wrote: >> On 2018-06-08 23:50, Erik Joelsson wrote: >>> On 2018-06-07 17:30, David Holmes wrote: On 8/06/2018 6:11 AM, Erik Joelsson wrote: > I just don't think the extra work is warranted or should be prioritized > at this point. I also cannot think of a combination of options required > for what you are suggesting that wouldn't be confusing to the user. If > someone truly feels like these flags are forced on them and can't live > with them, we or preferably that person can fix it then. I don't think > that's dictatorship. OpenJDK is still open source and anyone can > contribute. I don't see why --enable-hardened-jdk and --enable-hardened-hotspot to add to the right flags would be either complicated or confusing. >>> For me the confusion surrounds the difference between >>> --enable-hardened-hotspot and --with-jvm-variants=server, hardened and >>> making the user understand it. But sure, it is doable. Here is a new webrev >>> with those two options as I interpret them. Here is the help text: >>> >>> --enable-hardened-jdk enable hardenening compiler flags for all jdk >>> libraries (except the JVM), typically disabling >>> speculative cti. [disabled] >>> --enable-hardened-hotspot >>> enable hardenening compiler flags for hotspot (all >>> jvm variants), typically disabling speculative >>> cti. >>> To make hardening of hotspot a runtime choice, >>> consider the "hardened" jvm variant instead of >>> this >>> option. [disabled] >>> >>> Note that this changes the default for jdk libraries to not enable >>> hardening unless the user requests it. >>> >>> Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/ >> >> Hold it, hold it! I'm not sure how we ended up here, but I don't like it at >> all. :-( >> >> I think Eriks initial patch is much better than this. Some arguments in >> random order to defend this position: >> >> 1) Why should we have a configure option to disable security relevant flags >> for the JDK, if there has been no measured negative effect? We don't do this >> for any other compiler flags, especially not security relevant ones! >> >> I've re-read the entire thread to see if I could understand what could >> possibly motivate this, but the only thing I can find is David Holmes vague >> fear that these flags would not be well-tested enough. Let me counter with >> my own vague guesses: I believe the spectre mitigation methods to have been >> fully and properly tested, since they are rolled-out massively on all >> products. And let me complement with my own fear: the PR catastrophe if >> OpenJDK were *not* built with spectre mitigations, and someone were to >> exploit that! >> >> In fact, I could even argue that "server" should be hardened *by default*, >> and that we should instead introduce a non-hardened JVM named something akin >> to "quick-but-dangerous-server" instead. But I realize that a 25% >> performance hit is hard to swallow, so I won't push this agenda. >> >> 2) It is by no means clear that "--enable-hardened-jdk" does not harden all >> aspects of the JDK! If we should keep the option (which I definitely do not >> think we should!) it should be renamed to "--enable-hardened-libraries", or >> something like that. And it should be on by default, so it should be a >> "--disabled-hardened-jdk-libraries". >> >> Also, the general-sounding name "hardened" sounds like it might encompass >> more things than it does. What if I disabled a hardened jdk build, should I >> still get stack banging protection? If so, you need to move a lot more >> security-related flags to this option. (And, just to be absolutely clear: I >> don't think you should do that.) >> >> 3) Having two completely different ways of turning on Spectre protection for >> hotspot is just utterly confusing! This was a perfect example of how to use >> the JVM features, just as in the original patch. >> >> If you want to have spectre mitigation enabled for both server and client, >> by default, you would just need to run "configure >> --with-jvm-variants=server,client
Re: bash configure - LINK : fatal error LNK1104: cannot open file ...fixpath.exe
Hello,
I have tried the MSYS2_ARG_CONV_EXCL environment variable, thanks for
the suggestion. It's supposed to be a semi-colon separated list of
argument prefixes (they have some examples like
`/switch;/sdcard;--root=`), so I tried setting it to `/out:`, `-Fe`
(from the .m4 file) and `/out`, and also just the entire path that is
being mangled. But none of them worked :( (still the same error).
I'm trying to build the amber repo, which I think is parallel with
jdk/jdk tip? Any ways, there is no autogen.sh in /make/autoconf and
anytime I make changes to the .m4 file it mentions something about
having to regenerate the configure file. I can also see the changes I
make in `/build/.configure-support/generated-configure.sh`, which
currently looks like this (the part I mentioned):
```
#$RM -rf $FIXPATH_BIN $FIXPATH_DIR
#$MKDIR -p $FIXPATH_DIR $CONFIGURESUPPORT_OUTPUTDIR/bin
cd $CURDIR
echo "#" here
#if test ! -x $FIXPATH_BIN; then
# cd $FIXPATH_DIR
# $CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log
2>&1
# cd $CURDIR
#fi
echo "#" there
if test ! -x $FIXPATH_BIN; then
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5
$as_echo "no" >&6; }
cat $FIXPATH_DIR/fixpath1.log
as_fn_error $? "Could not create $FIXPATH_BIN" "$LINENO" 5
fi
```
Which gives me this output (the last few lines):
```
checking if fixpath can be created...
# here
# there
no
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
configure: error: Could not create
/J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/cofixpath.cupport/bin/fixpath.exe
J:/Projects/openjdk/amber/make/src/native/fixpath.c(49): warning C4477:
'fprintf' : format string '%s' requires an argument of type 'char *',
but variadic argument 3 has type 'LPVOID'
Microsoft (R) Incremental Linker Version 14.00.24215.1
Copyright (C) Microsoft Corporation. All rights reserved.
/out:J:J:/msys64/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe
fixpath.obj
LINK : fatal error LNK1104: cannot open file
'J:J:/msys64/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'
configure exiting with result code 1
```
So the changes I'm making seem to be going through... well... at least
as far as the echo statements go. I also tried using -e on the check
that is not comment out, but with no different results (I'm also using
autoconf 2.69 btw). This is kind of a head scratcher éh. I'm calling it
a night, I think I'll also try taking this up with the msys guys, some
other time.
Thanks for the help so far,
Jorn Vernee
Erik Joelsson schreef op 2018-06-11 22:19:
Hello,
On 2018-06-11 13:00, jbvernee wrote:
Hello Erik,
Thank you for the suggestion. Unfortunately it didn't help. TBH, I've
been banging my head against trying to build the JDK on my machine on
and off for a few months. So at this point I really appreciate any
help that gets me even an inch further, thanks.
After your suggestion, I have tracked down the call site of the
compile command and checked the paths that are being used in
basics_windows.m4 (line 406) to compile fixpath.exe:
```
cd $FIXPATH_DIR
$CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log
2>&1
cd $CURDIR
```
They are:
$CC = /j/progra~2/micros~2.0/vc/bin/amd64/cl
$FIXPATH_BIN_W =
J:/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe
$FIXPATH_DIR =
/J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/fixpath
Note that the second one is a windows style path. I can change that to
use the unix style path, and that does affect the error message, I now
see:
`'/J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'`
as the path in the linker error. But of course the Visual Studios
linker can't do anything with a unix style path.
What's weird is that either path is working for the C compiler (cl),
but it is being mangled before being passed to the linker, and I can't
find where the linker command is actually being fired off. It seems to
be done by that same line. I was hoping you could tell me more about
that?
AFAIK, we compile fixpath from a single source file directly into an
executable, so it's cl that calls link. Somewhere along the way, msys
decides to mangle your path argument incorrectly. You could try using
MSYS2_ARG_CONV_EXCL to disable the mangling. I don't remember exactly
how it works but I know you can affect the mangling using this env
variable.
One other idea I had, but haven't been able to implement, is to check
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
Hello, Based on the discussion here, I have reverted back to something more similar to webrev.02, but with a few changes. Mainly fixing a bug that caused JVM_FEATURES_hardened to not actually be the same as for server (if you have custom additions in configure). I also added a check so that configure fails if you try to enable either variant hardened or feature no-speculative-cti and the flags aren't available. Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.05/index.html /Erik On 2018-06-11 00:10, Magnus Ihse Bursie wrote: On 2018-06-08 23:50, Erik Joelsson wrote: On 2018-06-07 17:30, David Holmes wrote: On 8/06/2018 6:11 AM, Erik Joelsson wrote: I just don't think the extra work is warranted or should be prioritized at this point. I also cannot think of a combination of options required for what you are suggesting that wouldn't be confusing to the user. If someone truly feels like these flags are forced on them and can't live with them, we or preferably that person can fix it then. I don't think that's dictatorship. OpenJDK is still open source and anyone can contribute. I don't see why --enable-hardened-jdk and --enable-hardened-hotspot to add to the right flags would be either complicated or confusing. For me the confusion surrounds the difference between --enable-hardened-hotspot and --with-jvm-variants=server, hardened and making the user understand it. But sure, it is doable. Here is a new webrev with those two options as I interpret them. Here is the help text: --enable-hardened-jdk enable hardenening compiler flags for all jdk libraries (except the JVM), typically disabling speculative cti. [disabled] --enable-hardened-hotspot enable hardenening compiler flags for hotspot (all jvm variants), typically disabling speculative cti. To make hardening of hotspot a runtime choice, consider the "hardened" jvm variant instead of this option. [disabled] Note that this changes the default for jdk libraries to not enable hardening unless the user requests it. Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/ Hold it, hold it! I'm not sure how we ended up here, but I don't like it at all. :-( I think Eriks initial patch is much better than this. Some arguments in random order to defend this position: 1) Why should we have a configure option to disable security relevant flags for the JDK, if there has been no measured negative effect? We don't do this for any other compiler flags, especially not security relevant ones! I've re-read the entire thread to see if I could understand what could possibly motivate this, but the only thing I can find is David Holmes vague fear that these flags would not be well-tested enough. Let me counter with my own vague guesses: I believe the spectre mitigation methods to have been fully and properly tested, since they are rolled-out massively on all products. And let me complement with my own fear: the PR catastrophe if OpenJDK were *not* built with spectre mitigations, and someone were to exploit that! In fact, I could even argue that "server" should be hardened *by default*, and that we should instead introduce a non-hardened JVM named something akin to "quick-but-dangerous-server" instead. But I realize that a 25% performance hit is hard to swallow, so I won't push this agenda. 2) It is by no means clear that "--enable-hardened-jdk" does not harden all aspects of the JDK! If we should keep the option (which I definitely do not think we should!) it should be renamed to "--enable-hardened-libraries", or something like that. And it should be on by default, so it should be a "--disabled-hardened-jdk-libraries". Also, the general-sounding name "hardened" sounds like it might encompass more things than it does. What if I disabled a hardened jdk build, should I still get stack banging protection? If so, you need to move a lot more security-related flags to this option. (And, just to be absolutely clear: I don't think you should do that.) 3) Having two completely different ways of turning on Spectre protection for hotspot is just utterly confusing! This was a perfect example of how to use the JVM features, just as in the original patch. If you want to have spectre mitigation enabled for both server and client, by default, you would just need to run "configure --with-jvm-variants=server,client --with-jvm-features=no-speculative-cti", which will enable that feature for all variants. That's not really hard *at all* for anyone building OpenJDK. And it's way clearer what will happen, than a --enable-hardened-hotspot. 4) If you are a downstream provider building OpenJDK and you are dead set on not including Spectre mitigations in the JDK libraries, despite being shown to have no negative
Re: bash configure - LINK : fatal error LNK1104: cannot open file ...fixpath.exe
Hello, On 2018-06-11 13:00, jbvernee wrote: Hello Erik, Thank you for the suggestion. Unfortunately it didn't help. TBH, I've been banging my head against trying to build the JDK on my machine on and off for a few months. So at this point I really appreciate any help that gets me even an inch further, thanks. After your suggestion, I have tracked down the call site of the compile command and checked the paths that are being used in basics_windows.m4 (line 406) to compile fixpath.exe: ``` cd $FIXPATH_DIR $CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log 2>&1 cd $CURDIR ``` They are: $CC = /j/progra~2/micros~2.0/vc/bin/amd64/cl $FIXPATH_BIN_W = J:/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe $FIXPATH_DIR = /J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/fixpath Note that the second one is a windows style path. I can change that to use the unix style path, and that does affect the error message, I now see: `'/J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'` as the path in the linker error. But of course the Visual Studios linker can't do anything with a unix style path. What's weird is that either path is working for the C compiler (cl), but it is being mangled before being passed to the linker, and I can't find where the linker command is actually being fired off. It seems to be done by that same line. I was hoping you could tell me more about that? AFAIK, we compile fixpath from a single source file directly into an executable, so it's cl that calls link. Somewhere along the way, msys decides to mangle your path argument incorrectly. You could try using MSYS2_ARG_CONV_EXCL to disable the mangling. I don't remember exactly how it works but I know you can affect the mangling using this env variable. One other idea I had, but haven't been able to implement, is to check whether the fixpath tool exists before trying to compile it. That way I could just compile it manually. I have tried this snippet in basics_windows.m4 at line 404: ``` #$RM -rf $FIXPATH_BIN $FIXPATH_DIR #$MKDIR -p $FIXPATH_DIR $CONFIGURESUPPORT_OUTPUTDIR/bin cd $CURDIR if test ! -x $FIXPATH_BIN; then cd $FIXPATH_DIR $CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log 2>&1 cd $CURDIR fi ``` i.e. check if fixpath.exe exists, otherwise compile it (I don't know the source language though, so I just copied that from somewhere else). That didn't work, the check seems to be failing and it's still trying to compile (and I don't know why, I hope it's not a tabs vs. spaces issue?). I also tried just commenting that part out completely, but it's STILL trying to compile. I have no idea why that is happening a this point. It's also immediately spitting out 'no' after printing 'checking if fixpath can be created...', even before all the output from the compiler, so it almost seems like the command is being fired off from somewhere else? Or maybe it's just a race condition, idk. What version of OpenJDK are you trying to build? As in which repository did you clone. Depending on which, you may need to explicitly regenerate the configure script after making changes to .m4 files. There is a script, autogen.sh, in the same directory as the .m4 files to do it correctly. This requires autoconf 2.69 to be available. The language in .m4 is autoconf, which (in our case) is bash shell with m4 macros on top. Most of the source, including your snippet above is just bash. So your change above looks correct, you just need to get it to be used. You could try changing the -x to -e in case execute permissions aren't translated properly between msys and windows. /Erik If you have any more suggestions I really appreciate it, but if it's too much trouble for an unsupported build system I understand. Best regards, Jorn Vernee Erik Joelsson schreef op 2018-06-11 19:51: Hello Jorn, I don't have access to an msys2 environment to try this myself and as we don't regularly build there, it's unfortunately not expected to work. Msys has a habit of trying to outsmart the user when treating paths, by automatically converting path formats from unix style to windows style by trying to detect what kind of process is receiving it. In my experience, this automatic behavior trips you up more often than it helps you and it looks like that is what's happening here. The tool fixpath.exe is our internal solution to the path troubles. We build it in configure and then use it to convert paths to windows style for any process that we know need it. Unfortunately for you, your setup is failing before managing to create the tool itself. I would try to supply the value for the variable in Unix style (/j/msys/...) and see if msys converts it correctly then. /Erik On
Re: bash configure - LINK : fatal error LNK1104: cannot open file ...fixpath.exe
Hello Erik, Thank you for the suggestion. Unfortunately it didn't help. TBH, I've been banging my head against trying to build the JDK on my machine on and off for a few months. So at this point I really appreciate any help that gets me even an inch further, thanks. After your suggestion, I have tracked down the call site of the compile command and checked the paths that are being used in basics_windows.m4 (line 406) to compile fixpath.exe: ``` cd $FIXPATH_DIR $CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log 2>&1 cd $CURDIR ``` They are: $CC = /j/progra~2/micros~2.0/vc/bin/amd64/cl $FIXPATH_BIN_W = J:/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe $FIXPATH_DIR = /J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/fixpath Note that the second one is a windows style path. I can change that to use the unix style path, and that does affect the error message, I now see: `'/J/Projects/openjdk/amber/make/autoconf/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'` as the path in the linker error. But of course the Visual Studios linker can't do anything with a unix style path. What's weird is that either path is working for the C compiler (cl), but it is being mangled before being passed to the linker, and I can't find where the linker command is actually being fired off. It seems to be done by that same line. I was hoping you could tell me more about that? One other idea I had, but haven't been able to implement, is to check whether the fixpath tool exists before trying to compile it. That way I could just compile it manually. I have tried this snippet in basics_windows.m4 at line 404: ``` #$RM -rf $FIXPATH_BIN $FIXPATH_DIR #$MKDIR -p $FIXPATH_DIR $CONFIGURESUPPORT_OUTPUTDIR/bin cd $CURDIR if test ! -x $FIXPATH_BIN; then cd $FIXPATH_DIR $CC $FIXPATH_SRC_W -Fe$FIXPATH_BIN_W > $FIXPATH_DIR/fixpath1.log 2>&1 cd $CURDIR fi ``` i.e. check if fixpath.exe exists, otherwise compile it (I don't know the source language though, so I just copied that from somewhere else). That didn't work, the check seems to be failing and it's still trying to compile (and I don't know why, I hope it's not a tabs vs. spaces issue?). I also tried just commenting that part out completely, but it's STILL trying to compile. I have no idea why that is happening a this point. It's also immediately spitting out 'no' after printing 'checking if fixpath can be created...', even before all the output from the compiler, so it almost seems like the command is being fired off from somewhere else? Or maybe it's just a race condition, idk. If you have any more suggestions I really appreciate it, but if it's too much trouble for an unsupported build system I understand. Best regards, Jorn Vernee Erik Joelsson schreef op 2018-06-11 19:51: Hello Jorn, I don't have access to an msys2 environment to try this myself and as we don't regularly build there, it's unfortunately not expected to work. Msys has a habit of trying to outsmart the user when treating paths, by automatically converting path formats from unix style to windows style by trying to detect what kind of process is receiving it. In my experience, this automatic behavior trips you up more often than it helps you and it looks like that is what's happening here. The tool fixpath.exe is our internal solution to the path troubles. We build it in configure and then use it to convert paths to windows style for any process that we know need it. Unfortunately for you, your setup is failing before managing to create the tool itself. I would try to supply the value for the variable in Unix style (/j/msys/...) and see if msys converts it correctly then. /Erik On 2018-06-11 07:38, jbvernee wrote: Hello, I've been trying to build the JDK using an msys2 toolchain, which seems to be possible according to this thread: http://mail.openjdk.java.net/pipermail/build-dev/2017-October/019862.html (although I'm not trying to use gcc, I'm using VS 15). I know it's advised to use cygwin, but I can't get that to work at all on my machine (some error about base heap offset after a fresh install. I might try troubleshooting that next...) I'm running into the error mentioned in the subject line while running `bash configure`, namely: LINK : fatal error LNK1104: cannot open file 'J:J:/msys64/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe' I have also created a gist of the full configure log: https://gist.github.com/JornVernee/6b579e6d13d1fce306d0d370a381d1b3 Looking at this, it seems like the path is getting mangled. The correct path is 'J:/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe' But as you can see there is an extra `J:/msys64`
Re: RFR: JDK-8204664 PrepareFailureLogs should be done after sequential make targets
On 2018-06-11 11:09, Magnus Ihse Bursie wrote: On 2018-06-11 17:42, Erik Joelsson wrote: Hello, Looks ok. Perhaps this warrants a comment somewhere about the failure logs only working in parallel targets? Do you mean a comment in the code, or in the build documentation? I'm thinking in the code, to remind us in the future when we edit in these areas. /Erik /Magnus /Erik On 2018-06-11 01:50, Magnus Ihse Bursie wrote: When running a compound make line such as "make reconfigure clean jdk-image test-image", make will first single out the "sequential" targets reconfigure and clean, and execute them single-threaded, in sequence, and then it will build the remaining targets in parallel. However, the macro PrepareFailureLogs was called before this sequential calling, meaning that the directories created by it will be destroyed moments after by the clean target. The result is that if there is a compile error, the build will exit with something along these lines: /bin/cp: cannot create regular file `/export/users/dh198349/jdk-dev2/build/linux-x64-debug/make-support/failure-logs/hotspot_variant-server_libjvm_objs_thread.o.log': No such file or directory lib/CompileJvm.gmk:149: recipe for target '/export/users/dh198349/jdk-dev2/build/linux-x64-debug/hotspot/variant-server/libjvm/objs/thread.o' failed Bug: https://bugs.openjdk.java.net/browse/JDK-8204664 Patch inline: diff --git a/make/Init.gmk b/make/Init.gmk --- a/make/Init.gmk +++ b/make/Init.gmk @@ -298,7 +298,6 @@ main: $(INIT_TARGETS) ifneq ($(SEQUENTIAL_TARGETS)$(PARALLEL_TARGETS), ) $(call RotateLogFiles) - $(call PrepareFailureLogs) $(PRINTF) "Building $(TARGET_DESCRIPTION)\n" $(BUILD_LOG_PIPE) ifneq ($(SEQUENTIAL_TARGETS), ) # Don't touch build output dir since we might be cleaning. That @@ -308,6 +307,7 @@ $(SEQUENTIAL_TARGETS) ) endif ifneq ($(PARALLEL_TARGETS), ) + $(call PrepareFailureLogs) $(call StartGlobalTimer) $(call PrepareSmartJavac) # JOBS will only be empty for a bootcycle-images recursive call /Magnus
Re: RFR: JDK-8204664 PrepareFailureLogs should be done after sequential make targets
On 2018-06-11 17:42, Erik Joelsson wrote: Hello, Looks ok. Perhaps this warrants a comment somewhere about the failure logs only working in parallel targets? Do you mean a comment in the code, or in the build documentation? /Magnus /Erik On 2018-06-11 01:50, Magnus Ihse Bursie wrote: When running a compound make line such as "make reconfigure clean jdk-image test-image", make will first single out the "sequential" targets reconfigure and clean, and execute them single-threaded, in sequence, and then it will build the remaining targets in parallel. However, the macro PrepareFailureLogs was called before this sequential calling, meaning that the directories created by it will be destroyed moments after by the clean target. The result is that if there is a compile error, the build will exit with something along these lines: /bin/cp: cannot create regular file `/export/users/dh198349/jdk-dev2/build/linux-x64-debug/make-support/failure-logs/hotspot_variant-server_libjvm_objs_thread.o.log': No such file or directory lib/CompileJvm.gmk:149: recipe for target '/export/users/dh198349/jdk-dev2/build/linux-x64-debug/hotspot/variant-server/libjvm/objs/thread.o' failed Bug: https://bugs.openjdk.java.net/browse/JDK-8204664 Patch inline: diff --git a/make/Init.gmk b/make/Init.gmk --- a/make/Init.gmk +++ b/make/Init.gmk @@ -298,7 +298,6 @@ main: $(INIT_TARGETS) ifneq ($(SEQUENTIAL_TARGETS)$(PARALLEL_TARGETS), ) $(call RotateLogFiles) - $(call PrepareFailureLogs) $(PRINTF) "Building $(TARGET_DESCRIPTION)\n" $(BUILD_LOG_PIPE) ifneq ($(SEQUENTIAL_TARGETS), ) # Don't touch build output dir since we might be cleaning. That @@ -308,6 +307,7 @@ $(SEQUENTIAL_TARGETS) ) endif ifneq ($(PARALLEL_TARGETS), ) + $(call PrepareFailureLogs) $(call StartGlobalTimer) $(call PrepareSmartJavac) # JOBS will only be empty for a bootcycle-images recursive call /Magnus
Re: bash configure - LINK : fatal error LNK1104: cannot open file ...fixpath.exe
Hello Jorn,
I don't have access to an msys2 environment to try this myself and as we
don't regularly build there, it's unfortunately not expected to work.
Msys has a habit of trying to outsmart the user when treating paths, by
automatically converting path formats from unix style to windows style
by trying to detect what kind of process is receiving it. In my
experience, this automatic behavior trips you up more often than it
helps you and it looks like that is what's happening here.
The tool fixpath.exe is our internal solution to the path troubles. We
build it in configure and then use it to convert paths to windows style
for any process that we know need it. Unfortunately for you, your setup
is failing before managing to create the tool itself.
I would try to supply the value for the variable in Unix style
(/j/msys/...) and see if msys converts it correctly then.
/Erik
On 2018-06-11 07:38, jbvernee wrote:
Hello,
I've been trying to build the JDK using an msys2 toolchain, which
seems to be possible according to this thread:
http://mail.openjdk.java.net/pipermail/build-dev/2017-October/019862.html
(although I'm not trying to use gcc, I'm using VS 15). I know it's
advised to use cygwin, but I can't get that to work at all on my
machine (some error about base heap offset after a fresh install. I
might try troubleshooting that next...)
I'm running into the error mentioned in the subject line while running
`bash configure`, namely:
LINK : fatal error LNK1104: cannot open file
'J:J:/msys64/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'
I have also created a gist of the full configure log:
https://gist.github.com/JornVernee/6b579e6d13d1fce306d0d370a381d1b3
Looking at this, it seems like the path is getting mangled. The
correct path is
'J:/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'
But as you can see there is an extra `J:/msys64` in there, which
happens to be the root of my msys2 installation (though I'm running
configure in PowerShell, and not in the included msys2 environment.
Switching between them doesn't fix the problem either though).
I thought I had traced the value of this path to the `OUTPUTDIR`
variable in `\make\autoconf\basics.m4`, where it is created inside
this if-else block:
```
if test "x$CUSTOM_ROOT" != x; then
OUTPUTDIR="${CUSTOM_ROOT}/build/${CONF_NAME}"
else
OUTPUTDIR="${TOPDIR}/build/${CONF_NAME}"
fi
```
Where it is then used to create the `CONFIGURESUPPORT_OUTPUTDIR` path
like: `CONFIGURESUPPORT_OUTPUTDIR="$OUTPUTDIR/configure-support"`, and
finally the output path for fixpath.exe is constructed from that in
`\make\autoconf\basics_windows.m4`:
```
FIXPATH_BIN="$CONFIGURESUPPORT_OUTPUTDIR/bin/fixpath.exe"
FIXPATH_DIR="$CONFIGURESUPPORT_OUTPUTDIR/fixpath"
```
I have tried setting the `CUSTOM_ROOT` variable, but the error
remained the same. As a sanity check I rewrote the if-statement to:
`OUTPUTDIR="J:/Projects/openjdk/amber/build/${CONF_NAME}", but the
error message still remains the same.
I have also tried manually compiling fixpath.c into fixpath.exe and
placing that file into
`/build/windows-x86_64-normal-server-release/configure-support/bin/`,
but the file is just ignored by the configuration script.
At this point I don't know what else I could try. So I'm wondering if
anyone here has any suggestions?
Best regards,
Jorn Vernee
bash configure - LINK : fatal error LNK1104: cannot open file ...fixpath.exe
Hello,
I've been trying to build the JDK using an msys2 toolchain, which seems
to be possible according to this thread:
http://mail.openjdk.java.net/pipermail/build-dev/2017-October/019862.html
(although I'm not trying to use gcc, I'm using VS 15). I know it's
advised to use cygwin, but I can't get that to work at all on my machine
(some error about base heap offset after a fresh install. I might try
troubleshooting that next...)
I'm running into the error mentioned in the subject line while running
`bash configure`, namely:
LINK : fatal error LNK1104: cannot open file
'J:J:/msys64/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'
I have also created a gist of the full configure log:
https://gist.github.com/JornVernee/6b579e6d13d1fce306d0d370a381d1b3
Looking at this, it seems like the path is getting mangled. The correct
path is
'J:/Projects/openjdk/amber/build/windows-x86_64-normal-server-release/configure-support/bin/fixpath.exe'
But as you can see there is an extra `J:/msys64` in there, which happens
to be the root of my msys2 installation (though I'm running configure in
PowerShell, and not in the included msys2 environment. Switching between
them doesn't fix the problem either though).
I thought I had traced the value of this path to the `OUTPUTDIR`
variable in `\make\autoconf\basics.m4`, where it is created inside this
if-else block:
```
if test "x$CUSTOM_ROOT" != x; then
OUTPUTDIR="${CUSTOM_ROOT}/build/${CONF_NAME}"
else
OUTPUTDIR="${TOPDIR}/build/${CONF_NAME}"
fi
```
Where it is then used to create the `CONFIGURESUPPORT_OUTPUTDIR` path
like: `CONFIGURESUPPORT_OUTPUTDIR="$OUTPUTDIR/configure-support"`, and
finally the output path for fixpath.exe is constructed from that in
`\make\autoconf\basics_windows.m4`:
```
FIXPATH_BIN="$CONFIGURESUPPORT_OUTPUTDIR/bin/fixpath.exe"
FIXPATH_DIR="$CONFIGURESUPPORT_OUTPUTDIR/fixpath"
```
I have tried setting the `CUSTOM_ROOT` variable, but the error remained
the same. As a sanity check I rewrote the if-statement to:
`OUTPUTDIR="J:/Projects/openjdk/amber/build/${CONF_NAME}", but the error
message still remains the same.
I have also tried manually compiling fixpath.c into fixpath.exe and
placing that file into
`/build/windows-x86_64-normal-server-release/configure-support/bin/`,
but the file is just ignored by the configuration script.
At this point I don't know what else I could try. So I'm wondering if
anyone here has any suggestions?
Best regards,
Jorn Vernee
Re: RFR(XS): 8204684: [AIX] Build of libjli_static broken after change 8204572 (SetupJdkLibrary)
Hi Erik, Thomas, thanks for the quick review! Regards, Volker On Mon, Jun 11, 2018 at 5:59 PM, Thomas Stüfe wrote: > This looks good and I can confirm fixes our AIX build, > > Thanks for fixing! > > > ..Thomas > > > On Mon, Jun 11, 2018 at 4:48 PM, Volker Simonis > wrote: >> Hi, >> >> can I please have a review for the following trivial build change >> which fixes a build problem on AIX when building libjli_static: >> >> http://cr.openjdk.java.net/~simonis/webrevs/2018/8204684/ >> https://bugs.openjdk.java.net/browse/JDK-8204684 >> >> The reason is that change 8204572 forgot to add LIBJLI_SRC_DIRS as >> include path for the libjli_static build. >> >> Thank you and best regards, >> Volker
Re: RFR(XS): 8204684: [AIX] Build of libjli_static broken after change 8204572 (SetupJdkLibrary)
This looks good and I can confirm fixes our AIX build, Thanks for fixing! ..Thomas On Mon, Jun 11, 2018 at 4:48 PM, Volker Simonis wrote: > Hi, > > can I please have a review for the following trivial build change > which fixes a build problem on AIX when building libjli_static: > > http://cr.openjdk.java.net/~simonis/webrevs/2018/8204684/ > https://bugs.openjdk.java.net/browse/JDK-8204684 > > The reason is that change 8204572 forgot to add LIBJLI_SRC_DIRS as > include path for the libjli_static build. > > Thank you and best regards, > Volker
Re: RFR(XS): 8204684: [AIX] Build of libjli_static broken after change 8204572 (SetupJdkLibrary)
Looks ok. /Erik On 2018-06-11 07:48, Volker Simonis wrote: Hi, can I please have a review for the following trivial build change which fixes a build problem on AIX when building libjli_static: http://cr.openjdk.java.net/~simonis/webrevs/2018/8204684/ https://bugs.openjdk.java.net/browse/JDK-8204684 The reason is that change 8204572 forgot to add LIBJLI_SRC_DIRS as include path for the libjli_static build. Thank you and best regards, Volker
Re: RFR: JDK-8204682 Parsing for LOG=report=none is broken when combined with other keywords
Looks good. Sneaky problem! /Erik On 2018-06-11 06:29, Magnus Ihse Bursie wrote: Parsing for LOG=report=none is broken when combined with other keywords, e.g. "LOG=info,report=none". Bug: https://bugs.openjdk.java.net/browse/JDK-8204682 WebRev: http://cr.openjdk.java.net/~ihse/JDK-8204682-LOG-report-parsing-broken/webrev.01 /Magnus
Re: RFR: JDK-8204127: Change bundle format on Windows to zip
I agree, will change. /Erik On 2018-06-11 01:55, Magnus Ihse Bursie wrote: On 2018-06-09 02:05, Erik Joelsson wrote: The compressed archive bundles we create are all tar.gz format. On Windows that's not a very common format, requiring third party applications to handle, while the zip format can be natively extracted. This patch changes the bundle format for the JDK on Windows to zip. This only applies to the actual product distributions. I think we prefer tar.gz for all the internal bundles, like symbols and tests. Bug: https://bugs.openjdk.java.net/browse/JDK-8204127 Webrev: http://cr.openjdk.java.net/~erikj/8204127/webrev.01 It's okay, but I think it would be nicer to do it like this: ifeq ($(OPENJDK_TARGET_OS), windows) JDK_BUNDLE_EXTENSION := zip else JDK_BUNDLE_EXTENSION := tar.gz endif JDK_BUNDLE_NAME := jdk-$(BASE_NAME)_bin$(DEBUG_PART).$(JDK_BUNDLE_EXTENSION) If you accept, you don't need to redo the review. If you disagree, push your version. :-) /Magnus /Erik
Re: RFR: JDK-8204664 PrepareFailureLogs should be done after sequential make targets
Hello, Looks ok. Perhaps this warrants a comment somewhere about the failure logs only working in parallel targets? /Erik On 2018-06-11 01:50, Magnus Ihse Bursie wrote: When running a compound make line such as "make reconfigure clean jdk-image test-image", make will first single out the "sequential" targets reconfigure and clean, and execute them single-threaded, in sequence, and then it will build the remaining targets in parallel. However, the macro PrepareFailureLogs was called before this sequential calling, meaning that the directories created by it will be destroyed moments after by the clean target. The result is that if there is a compile error, the build will exit with something along these lines: /bin/cp: cannot create regular file `/export/users/dh198349/jdk-dev2/build/linux-x64-debug/make-support/failure-logs/hotspot_variant-server_libjvm_objs_thread.o.log': No such file or directory lib/CompileJvm.gmk:149: recipe for target '/export/users/dh198349/jdk-dev2/build/linux-x64-debug/hotspot/variant-server/libjvm/objs/thread.o' failed Bug: https://bugs.openjdk.java.net/browse/JDK-8204664 Patch inline: diff --git a/make/Init.gmk b/make/Init.gmk --- a/make/Init.gmk +++ b/make/Init.gmk @@ -298,7 +298,6 @@ main: $(INIT_TARGETS) ifneq ($(SEQUENTIAL_TARGETS)$(PARALLEL_TARGETS), ) $(call RotateLogFiles) - $(call PrepareFailureLogs) $(PRINTF) "Building $(TARGET_DESCRIPTION)\n" $(BUILD_LOG_PIPE) ifneq ($(SEQUENTIAL_TARGETS), ) # Don't touch build output dir since we might be cleaning. That @@ -308,6 +307,7 @@ $(SEQUENTIAL_TARGETS) ) endif ifneq ($(PARALLEL_TARGETS), ) + $(call PrepareFailureLogs) $(call StartGlobalTimer) $(call PrepareSmartJavac) # JOBS will only be empty for a bootcycle-images recursive call /Magnus
RFR(XS): 8204684: [AIX] Build of libjli_static broken after change 8204572 (SetupJdkLibrary)
Hi, can I please have a review for the following trivial build change which fixes a build problem on AIX when building libjli_static: http://cr.openjdk.java.net/~simonis/webrevs/2018/8204684/ https://bugs.openjdk.java.net/browse/JDK-8204684 The reason is that change 8204572 forgot to add LIBJLI_SRC_DIRS as include path for the libjli_static build. Thank you and best regards, Volker
RFR: JDK-8204682 Parsing for LOG=report=none is broken when combined with other keywords
Parsing for LOG=report=none is broken when combined with other keywords, e.g. "LOG=info,report=none". Bug: https://bugs.openjdk.java.net/browse/JDK-8204682 WebRev: http://cr.openjdk.java.net/~ihse/JDK-8204682-LOG-report-parsing-broken/webrev.01 /Magnus
Re: RFR: JDK-8204127: Change bundle format on Windows to zip
On 2018-06-09 02:05, Erik Joelsson wrote: The compressed archive bundles we create are all tar.gz format. On Windows that's not a very common format, requiring third party applications to handle, while the zip format can be natively extracted. This patch changes the bundle format for the JDK on Windows to zip. This only applies to the actual product distributions. I think we prefer tar.gz for all the internal bundles, like symbols and tests. Bug: https://bugs.openjdk.java.net/browse/JDK-8204127 Webrev: http://cr.openjdk.java.net/~erikj/8204127/webrev.01 It's okay, but I think it would be nicer to do it like this: ifeq ($(OPENJDK_TARGET_OS), windows) JDK_BUNDLE_EXTENSION := zip else JDK_BUNDLE_EXTENSION := tar.gz endif JDK_BUNDLE_NAME := jdk-$(BASE_NAME)_bin$(DEBUG_PART).$(JDK_BUNDLE_EXTENSION) If you accept, you don't need to redo the review. If you disagree, push your version. :-) /Magnus /Erik
RFR: JDK-8204664 PrepareFailureLogs should be done after sequential make targets
When running a compound make line such as "make reconfigure clean jdk-image test-image", make will first single out the "sequential" targets reconfigure and clean, and execute them single-threaded, in sequence, and then it will build the remaining targets in parallel. However, the macro PrepareFailureLogs was called before this sequential calling, meaning that the directories created by it will be destroyed moments after by the clean target. The result is that if there is a compile error, the build will exit with something along these lines: /bin/cp: cannot create regular file `/export/users/dh198349/jdk-dev2/build/linux-x64-debug/make-support/failure-logs/hotspot_variant-server_libjvm_objs_thread.o.log': No such file or directory lib/CompileJvm.gmk:149: recipe for target '/export/users/dh198349/jdk-dev2/build/linux-x64-debug/hotspot/variant-server/libjvm/objs/thread.o' failed Bug: https://bugs.openjdk.java.net/browse/JDK-8204664 Patch inline: diff --git a/make/Init.gmk b/make/Init.gmk --- a/make/Init.gmk +++ b/make/Init.gmk @@ -298,7 +298,6 @@ main: $(INIT_TARGETS) ifneq ($(SEQUENTIAL_TARGETS)$(PARALLEL_TARGETS), ) $(call RotateLogFiles) - $(call PrepareFailureLogs) $(PRINTF) "Building $(TARGET_DESCRIPTION)\n" $(BUILD_LOG_PIPE) ifneq ($(SEQUENTIAL_TARGETS), ) # Don't touch build output dir since we might be cleaning. That @@ -308,6 +307,7 @@ $(SEQUENTIAL_TARGETS) ) endif ifneq ($(PARALLEL_TARGETS), ) + $(call PrepareFailureLogs) $(call StartGlobalTimer) $(call PrepareSmartJavac) # JOBS will only be empty for a bootcycle-images recursive call /Magnus
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
Sorry this is making my head spin.
Doh! jvm-features only apply to the JVM.
So I retract my last email - sorry.
And with that I'm going to just bow out. You and Erik can figure it out.
Thanks,
David
On 11/06/2018 6:19 PM, Magnus Ihse Bursie wrote:
On 2018-06-11 09:38, David Holmes wrote:
Hi Magnus,
On 11/06/2018 5:10 PM, Magnus Ihse Bursie wrote:
On 2018-06-08 23:50, Erik Joelsson wrote:
On 2018-06-07 17:30, David Holmes wrote:
On 8/06/2018 6:11 AM, Erik Joelsson wrote:
I just don't think the extra work is warranted or should be
prioritized at this point. I also cannot think of a combination of
options required for what you are suggesting that wouldn't be
confusing to the user. If someone truly feels like these flags are
forced on them and can't live with them, we or preferably that
person can fix it then. I don't think that's dictatorship. OpenJDK
is still open source and anyone can contribute.
I don't see why --enable-hardened-jdk and --enable-hardened-hotspot
to add to the right flags would be either complicated or confusing.
For me the confusion surrounds the difference between
--enable-hardened-hotspot and --with-jvm-variants=server, hardened
and making the user understand it. But sure, it is doable. Here is a
new webrev with those two options as I interpret them. Here is the
help text:
--enable-hardened-jdk enable hardenening compiler flags for all jdk
libraries (except the JVM), typically
disabling
speculative cti. [disabled]
--enable-hardened-hotspot
enable hardenening compiler flags for
hotspot (all
jvm variants), typically disabling
speculative cti.
To make hardening of hotspot a runtime
choice,
consider the "hardened" jvm variant
instead of this
option. [disabled]
Note that this changes the default for jdk libraries to not enable
hardening unless the user requests it.
Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/
Hold it, hold it! I'm not sure how we ended up here, but I don't like
it at all. :-(
I think Eriks initial patch is much better than this. Some arguments
in random order to defend this position:
1) Why should we have a configure option to disable security relevant
flags for the JDK, if there has been no measured negative effect? We
don't do this for any other compiler flags, especially not security
relevant ones!
I've re-read the entire thread to see if I could understand what
could possibly motivate this, but the only thing I can find is David
Holmes vague fear that these flags would not be well-tested enough.
Let me counter with my own vague guesses: I believe the spectre
mitigation methods to have been fully and properly tested, since they
are rolled-out massively on all products. And let me complement with
my own fear: the PR catastrophe if OpenJDK were *not* built with
spectre mitigations, and someone were to exploit that!
All I'm looking for is the ability to select whether you can build
with or without this "hardening". The default OpenJDK build can of
course churn out a "hardened" implementation. Anyone who opts out of
that is on their own.
With Erik's original proposal (webrev.02), you will, by default, get a
hotspot "server" JVM variant that is identical to what you got without
the patch. This should definitely cover your case.
You will also get all the non-hotspot libraries built as hardened. You
*can* get the JDK libraries built non-hardened, by removing the
${$2NO_SPECULATIVE_CTI_CFLAGS} from the line
$1_CFLAGS_JDK="${$1_DEFINES_CPU_JDK} ${$1_CFLAGS_CPU}
${$1_CFLAGS_CPU_JDK} ${$1_TOOLCHAIN_CFLAGS}
${$2NO_SPECULATIVE_CTI_CFLAGS}".
As I said, I believe this is enough to support that case.
I don't share your faith or confidence in the quality of any software
rushed out in a fairly short space of time. Prudence, if nothing else,
says you should be able to not build this way IMHO.
AFAIU, these compiler flags has received extensive testing inside
Oracle. It is also part of a global, high-visibility project, where key
players have had time to prepare for handling the issues ahead of the
public awareness of the exploits. *And* it's been almost half a year
since the Spectre exploit was made public.
I have much more faith in enabling these flags than I'd have for e.g.
upgrading to a newer version of Solaris Studio. :-)
In fact, I could even argue that "server" should be hardened *by
default*, and that we should instead introduce a non-hardened JVM
named something akin to "quick-but-dangerous-server" instead. But I
realize that a 25% performance hit is hard to swallow, so I won't
push this agenda.
2) It is by no means clear that "--enable-hardened-jdk" does not
harden all aspects of the JDK! If we should keep the option (which I
definitely do not think we should!) it should be renamed
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
On 2018-06-11 09:38, David Holmes wrote:
Hi Magnus,
On 11/06/2018 5:10 PM, Magnus Ihse Bursie wrote:
On 2018-06-08 23:50, Erik Joelsson wrote:
On 2018-06-07 17:30, David Holmes wrote:
On 8/06/2018 6:11 AM, Erik Joelsson wrote:
I just don't think the extra work is warranted or should be
prioritized at this point. I also cannot think of a combination of
options required for what you are suggesting that wouldn't be
confusing to the user. If someone truly feels like these flags are
forced on them and can't live with them, we or preferably that
person can fix it then. I don't think that's dictatorship. OpenJDK
is still open source and anyone can contribute.
I don't see why --enable-hardened-jdk and --enable-hardened-hotspot
to add to the right flags would be either complicated or confusing.
For me the confusion surrounds the difference between
--enable-hardened-hotspot and --with-jvm-variants=server, hardened
and making the user understand it. But sure, it is doable. Here is a
new webrev with those two options as I interpret them. Here is the
help text:
--enable-hardened-jdk enable hardenening compiler flags for all jdk
libraries (except the JVM), typically
disabling
speculative cti. [disabled]
--enable-hardened-hotspot
enable hardenening compiler flags for
hotspot (all
jvm variants), typically disabling
speculative cti.
To make hardening of hotspot a runtime
choice,
consider the "hardened" jvm variant
instead of this
option. [disabled]
Note that this changes the default for jdk libraries to not enable
hardening unless the user requests it.
Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/
Hold it, hold it! I'm not sure how we ended up here, but I don't like
it at all. :-(
I think Eriks initial patch is much better than this. Some arguments
in random order to defend this position:
1) Why should we have a configure option to disable security relevant
flags for the JDK, if there has been no measured negative effect? We
don't do this for any other compiler flags, especially not security
relevant ones!
I've re-read the entire thread to see if I could understand what
could possibly motivate this, but the only thing I can find is David
Holmes vague fear that these flags would not be well-tested enough.
Let me counter with my own vague guesses: I believe the spectre
mitigation methods to have been fully and properly tested, since they
are rolled-out massively on all products. And let me complement with
my own fear: the PR catastrophe if OpenJDK were *not* built with
spectre mitigations, and someone were to exploit that!
All I'm looking for is the ability to select whether you can build
with or without this "hardening". The default OpenJDK build can of
course churn out a "hardened" implementation. Anyone who opts out of
that is on their own.
With Erik's original proposal (webrev.02), you will, by default, get a
hotspot "server" JVM variant that is identical to what you got without
the patch. This should definitely cover your case.
You will also get all the non-hotspot libraries built as hardened. You
*can* get the JDK libraries built non-hardened, by removing the
${$2NO_SPECULATIVE_CTI_CFLAGS} from the line
$1_CFLAGS_JDK="${$1_DEFINES_CPU_JDK} ${$1_CFLAGS_CPU}
${$1_CFLAGS_CPU_JDK} ${$1_TOOLCHAIN_CFLAGS} ${$2NO_SPECULATIVE_CTI_CFLAGS}".
As I said, I believe this is enough to support that case.
I don't share your faith or confidence in the quality of any software
rushed out in a fairly short space of time. Prudence, if nothing else,
says you should be able to not build this way IMHO.
AFAIU, these compiler flags has received extensive testing inside
Oracle. It is also part of a global, high-visibility project, where key
players have had time to prepare for handling the issues ahead of the
public awareness of the exploits. *And* it's been almost half a year
since the Spectre exploit was made public.
I have much more faith in enabling these flags than I'd have for e.g.
upgrading to a newer version of Solaris Studio. :-)
In fact, I could even argue that "server" should be hardened *by
default*, and that we should instead introduce a non-hardened JVM
named something akin to "quick-but-dangerous-server" instead. But I
realize that a 25% performance hit is hard to swallow, so I won't
push this agenda.
2) It is by no means clear that "--enable-hardened-jdk" does not
harden all aspects of the JDK! If we should keep the option (which I
definitely do not think we should!) it should be renamed to
"--enable-hardened-libraries", or something like that. And it should
be on by default, so it should be a "--disabled-hardened-jdk-libraries".
Also, the general-sounding name "hardened" sounds like it might
encompass more things than it does.
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
Hi Magnus, On 11/06/2018 5:10 PM, Magnus Ihse Bursie wrote: On 2018-06-08 23:50, Erik Joelsson wrote: On 2018-06-07 17:30, David Holmes wrote: On 8/06/2018 6:11 AM, Erik Joelsson wrote: I just don't think the extra work is warranted or should be prioritized at this point. I also cannot think of a combination of options required for what you are suggesting that wouldn't be confusing to the user. If someone truly feels like these flags are forced on them and can't live with them, we or preferably that person can fix it then. I don't think that's dictatorship. OpenJDK is still open source and anyone can contribute. I don't see why --enable-hardened-jdk and --enable-hardened-hotspot to add to the right flags would be either complicated or confusing. For me the confusion surrounds the difference between --enable-hardened-hotspot and --with-jvm-variants=server, hardened and making the user understand it. But sure, it is doable. Here is a new webrev with those two options as I interpret them. Here is the help text: --enable-hardened-jdk enable hardenening compiler flags for all jdk libraries (except the JVM), typically disabling speculative cti. [disabled] --enable-hardened-hotspot enable hardenening compiler flags for hotspot (all jvm variants), typically disabling speculative cti. To make hardening of hotspot a runtime choice, consider the "hardened" jvm variant instead of this option. [disabled] Note that this changes the default for jdk libraries to not enable hardening unless the user requests it. Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/ Hold it, hold it! I'm not sure how we ended up here, but I don't like it at all. :-( I think Eriks initial patch is much better than this. Some arguments in random order to defend this position: 1) Why should we have a configure option to disable security relevant flags for the JDK, if there has been no measured negative effect? We don't do this for any other compiler flags, especially not security relevant ones! I've re-read the entire thread to see if I could understand what could possibly motivate this, but the only thing I can find is David Holmes vague fear that these flags would not be well-tested enough. Let me counter with my own vague guesses: I believe the spectre mitigation methods to have been fully and properly tested, since they are rolled-out massively on all products. And let me complement with my own fear: the PR catastrophe if OpenJDK were *not* built with spectre mitigations, and someone were to exploit that! All I'm looking for is the ability to select whether you can build with or without this "hardening". The default OpenJDK build can of course churn out a "hardened" implementation. Anyone who opts out of that is on their own. I don't share your faith or confidence in the quality of any software rushed out in a fairly short space of time. Prudence, if nothing else, says you should be able to not build this way IMHO. In fact, I could even argue that "server" should be hardened *by default*, and that we should instead introduce a non-hardened JVM named something akin to "quick-but-dangerous-server" instead. But I realize that a 25% performance hit is hard to swallow, so I won't push this agenda. 2) It is by no means clear that "--enable-hardened-jdk" does not harden all aspects of the JDK! If we should keep the option (which I definitely do not think we should!) it should be renamed to "--enable-hardened-libraries", or something like that. And it should be on by default, so it should be a "--disabled-hardened-jdk-libraries". Also, the general-sounding name "hardened" sounds like it might encompass more things than it does. What if I disabled a hardened jdk build, should I still get stack banging protection? If so, you need to move a lot more security-related flags to this option. (And, just to be absolutely clear: I don't think you should do that.) 3) Having two completely different ways of turning on Spectre protection for hotspot is just utterly confusing! This was a perfect example of how to use the JVM features, just as in the original patch. Okay. I have had some confusion over "features" versus "variants" based on Eriks earlier comments. Erik's email from June 6 first states: "I agree, and you sort of can. By adding the jvm feature "no-speculative-cti" to any jvm variant, you get the flags." but then later said: "We don't see the point in giving the choice on the JDK libraries ..." by which I now think he meant not giving the choice at the VM variant level, but I mistook it for meaning at the "feature" level. Hence I came back with the two flags suggestion. If we can already select features arbitrarily at configure time then this is all addressed already.
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
On 2018-06-10 15:28, David Holmes wrote: On 9/06/2018 7:50 AM, Erik Joelsson wrote: On 2018-06-07 17:30, David Holmes wrote: On 8/06/2018 6:11 AM, Erik Joelsson wrote: I just don't think the extra work is warranted or should be prioritized at this point. I also cannot think of a combination of options required for what you are suggesting that wouldn't be confusing to the user. If someone truly feels like these flags are forced on them and can't live with them, we or preferably that person can fix it then. I don't think that's dictatorship. OpenJDK is still open source and anyone can contribute. I don't see why --enable-hardened-jdk and --enable-hardened-hotspot to add to the right flags would be either complicated or confusing. For me the confusion surrounds the difference between --enable-hardened-hotspot and --with-jvm-variants=server, hardened and That's the problem: "hardened" is not a jvm-variant as we have always defined that! "hardened" is a variation in the same way as product vs fastdebug versus slow-debug versus (the old) optimized. It is _not_ at all the same kind of thing as server versus client versus zero etc. The desire to ship "hardened" in the same image as non-hardened is what is causing the semantic conflict here. It is like shipping a product and debug VM together. Sure you can do it, but that's not how we've categorised things in the past. I disagree. The "no-speculative-cti" is a perfectly fine JVM feature, which can be applied to any JVM variant. It is not a JVM feature as a separate software component (like cmsgc or compiler1) that could be left in or kept out and that affects the functionality of hotspot. Instead, it is a JVM feature very much like the existing link-time-opt, in that it affects all aspects of the JVM; not the functionality, but the performance (and security). The way we bundle a certain set of JVM features as a named JVM variant has always been a bit, well, semantically odd, but it has served us okay in the past and serve us just as well for this fix. I understand the need to make things work this way, so in that sense selecting jvm-variant=hardened should be seen as specifying "--enable-hardened-hotspot --enable-hardened-jdk". But jvm-variant=hardened is really jvm-variant=hardened-server. Yes, jvm-variant=hardened is actually hardned-server. Despite the longer name, it might be more clear to use that name. It ties in into a bit into Erik's original "altserver" proposal. I think the reason just "hardened" sounds like a reasonable alternative to the more proper but longer "hardened-server" is due to how "server" has become the mainstream variant, even for clients, and "client" feels like it's being put on death row. Nobody really believes that it will survive in the long term, and nowadays Oracle don't even build it anymore (we stopped doing that when we stopped doing 32-bit builds). So "server" is increasingly incorrectly named, and should really just be considered a legacy name for what should perhaps be "default" or so. /Magnus making the user understand it. But sure, it is doable. Here is a new webrev with those two options as I interpret them. Here is the help text: --enable-hardened-jdk enable hardenening compiler flags for all jdk libraries (except the JVM), typically disabling speculative cti. [disabled] --enable-hardened-hotspot enable hardenening compiler flags for hotspot (all jvm variants), typically disabling speculative cti. To make hardening of hotspot a runtime choice, consider the "hardened" jvm variant instead of this option. [disabled] Note that this changes the default for jdk libraries to not enable hardening unless the user requests it. That's your call. I don't care what the default is as long as the developer has control over it. Thanks, David Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/ /Erik
Re: RFR: JDK-8202384: Introduce altserver jvm variant with speculative execution disabled
On 2018-06-08 23:50, Erik Joelsson wrote: On 2018-06-07 17:30, David Holmes wrote: On 8/06/2018 6:11 AM, Erik Joelsson wrote: I just don't think the extra work is warranted or should be prioritized at this point. I also cannot think of a combination of options required for what you are suggesting that wouldn't be confusing to the user. If someone truly feels like these flags are forced on them and can't live with them, we or preferably that person can fix it then. I don't think that's dictatorship. OpenJDK is still open source and anyone can contribute. I don't see why --enable-hardened-jdk and --enable-hardened-hotspot to add to the right flags would be either complicated or confusing. For me the confusion surrounds the difference between --enable-hardened-hotspot and --with-jvm-variants=server, hardened and making the user understand it. But sure, it is doable. Here is a new webrev with those two options as I interpret them. Here is the help text: --enable-hardened-jdk enable hardenening compiler flags for all jdk libraries (except the JVM), typically disabling speculative cti. [disabled] --enable-hardened-hotspot enable hardenening compiler flags for hotspot (all jvm variants), typically disabling speculative cti. To make hardening of hotspot a runtime choice, consider the "hardened" jvm variant instead of this option. [disabled] Note that this changes the default for jdk libraries to not enable hardening unless the user requests it. Webrev: http://cr.openjdk.java.net/~erikj/8202384/webrev.04/ Hold it, hold it! I'm not sure how we ended up here, but I don't like it at all. :-( I think Eriks initial patch is much better than this. Some arguments in random order to defend this position: 1) Why should we have a configure option to disable security relevant flags for the JDK, if there has been no measured negative effect? We don't do this for any other compiler flags, especially not security relevant ones! I've re-read the entire thread to see if I could understand what could possibly motivate this, but the only thing I can find is David Holmes vague fear that these flags would not be well-tested enough. Let me counter with my own vague guesses: I believe the spectre mitigation methods to have been fully and properly tested, since they are rolled-out massively on all products. And let me complement with my own fear: the PR catastrophe if OpenJDK were *not* built with spectre mitigations, and someone were to exploit that! In fact, I could even argue that "server" should be hardened *by default*, and that we should instead introduce a non-hardened JVM named something akin to "quick-but-dangerous-server" instead. But I realize that a 25% performance hit is hard to swallow, so I won't push this agenda. 2) It is by no means clear that "--enable-hardened-jdk" does not harden all aspects of the JDK! If we should keep the option (which I definitely do not think we should!) it should be renamed to "--enable-hardened-libraries", or something like that. And it should be on by default, so it should be a "--disabled-hardened-jdk-libraries". Also, the general-sounding name "hardened" sounds like it might encompass more things than it does. What if I disabled a hardened jdk build, should I still get stack banging protection? If so, you need to move a lot more security-related flags to this option. (And, just to be absolutely clear: I don't think you should do that.) 3) Having two completely different ways of turning on Spectre protection for hotspot is just utterly confusing! This was a perfect example of how to use the JVM features, just as in the original patch. If you want to have spectre mitigation enabled for both server and client, by default, you would just need to run "configure --with-jvm-variants=server,client --with-jvm-features=no-speculative-cti", which will enable that feature for all variants. That's not really hard *at all* for anyone building OpenJDK. And it's way clearer what will happen, than a --enable-hardened-hotspot. 4) If you are a downstream provider building OpenJDK and you are dead set on not including Spectre mitigations in the JDK libraries, despite being shown to have no negative effects, then you can do just as any other downstream user with highly specialized requirements, and patch the source. I have no sympathies for this; I can't stop it but I don't think there's any reason for us to complicate the code to support this unlikely case. So, to recap, I think the webrev as published in http://cr.openjdk.java.net/~erikj/8202384/webrev.02/ (with "altserver" renamed to "hardened") is the way to go. /Magnus /Erik
Re: RFR: JDK-8204602 Add devkit for linux-arm32
On 2018-06-10 15:35, David Holmes wrote: On 8/06/2018 6:38 PM, Magnus Ihse Bursie wrote: With some simple changes, our devkit scripts can be made to generate devkits for cross-compiling on linux-x64 to linux-arm (32-bit). Also add a jib profile to use this new linux-arm devkit, as opposed to the legacy devkits used by the traditional linux-arm profiles. I'm not at all clear what this is providing or for whom. The legacy linux-arm-vfp-hflt requires pointers to the X11R6 installation to build. The linux-arm32 jib-profile has none, but instead lists freetype as being bundled. ?? The target audience here is mainly Erik and me (and the Hotspot engineers who have requested a simple way to build the arm port). This provides us with an up-to-date, standard configured, devkit for building arm-32. In contrast to the legacy jib profile, we now have a devkit that is build according to our standard, so configure need a minimal amount of flags (hence no X11 flags). The bundled freetype is to simplify things; we now have a version of freetype included in the OpenJDK source, and it's by far the easiest to use it, rather than to include it in the devkit, but this is not enabled by default on linux-arm. I hope that answers your questions! /Magnus David Note that this change does not imply any change in Oracle's support of the linux-arm platform. Bug: https://bugs.openjdk.java.net/browse/JDK-8204602 WebRev: http://cr.openjdk.java.net/~ihse/JDK-8204602-add-linux-arm-32-devkit/webrev.01 /Magnus
