Re: Help with build changes for: 8214796: Create a jlink plugin for stripping debug info symbols from native libraries

[Mandy Chung]

Hi Severin,

Another alternative would be to support per-jlink-plugin resource
bundle to avoid merging .properties files at build time.  The
plugin-specific messages should only be used by the plugin itself
and it would be cleaner for each plugin to manage its resource bundle.

Mandy

On 1/25/19 7:27 AM, Severin Gehwolf wrote:

Hi,

I'm working on an enhancement for jlink. In particular a platform
specific plugin. I.e. It would only get built on unix/linux platforms.
My trouble is getting some resouce properties set up properly. In my
example there is two versions of plugins.properties: one in
shared/classes one in unix/classes. These, need to get merged into one
via the MergeProperties build tool class. So far so good. But for the
cases where there is only one resource property it should just get
copied to support/gensrc and that source root be used for compiling
those properties into actual ListResourceBundle Java classes.

The copying from the src tree to the gensrc tree doesn't seem to work.
I've tried using $(CP) and SetupCopyFiles to no avail. Would anybody
willing to help?

WIP webrev is:
http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8214796-wip/01/webrev/

I thought I'd do the copying in an else branch in SetupPropertiesMerge,
where I've put in a FIXME comment. Am I on the wrong track?

Thanks,
Severin

RFR JDK-8217761: Enhance run-test-prebuilt jib profile to support running tests with JCov

[Alexandre (Shura) Iline]

Hi,

Please take a look on a change to allow JCov test execution through jib.

Please notice that this fix also changes behavior of RunTest.gmk when 
conflicting combination of options is used. JCOV_IMAGE_DIR  is now expected to 
be set when TEST_OPTS_JCOV is true.

Shura

Bug: https://bugs.openjdk.java.net/browse/JDK-8217761
Webrev: http://cr.openjdk.java.net/~shurailine/8217761/webrev.02/

Re: Failure at rung of get_source on windows with cygwin

[Magnus Ihse Bursie]

On 2019-01-25 22:12, Simonovsky, Pavel wrote:

Hi.
I am attempting to build Open jdk 8u from sources.
At Linux (RH6 it worked smoothly.
At windows I am trying to build using Cygwin version 2.8...

I am failing at windows build on step of running get_source.sh:

   nashorn:   abort: stream ended unexpectedly (got 1297 bytes, 
expected 192443)
 jaxws:   abort: stream ended unexpectedly (got 1957 bytes, 
expected 21072)
  jaxp:   rollback completed
   hotspot:   abort: stream ended unexpectedly (got 99623 bytes, 
expected 483101)
  jaxp:   abort: stream ended unexpectedly (got 28110 bytes, 
expected 37505)
 langtools:   transaction abort!
 langtools:   rollback completed
 langtools:   abort: stream ended unexpectedly (got 810 bytes, 
expected 2288)
WARNING: /tmp/forest.34500/hotspot.pid.rc exited abnormally.
WARNING: /tmp/forest.34500/jaxp.pid.rc exited abnormally.
WARNING: /tmp/forest.34500/jaxws.pid.rc exited abnormally.
WARNING: /tmp/forest.34500/langtools.pid.rc exited abnormally.
WARNING: /tmp/forest.34500/nashorn.pid.rc exited abnormally.

I have enough disk space:

$ df -k .
Filesystem 1K-blocks  Used Available Use% Mounted on
C: 248548552 191203664  57344888  77% /cygdrive/c

And fast network connection.
What have I to do to troubleshoot this problem?
It sounds like the unfortunately well-known problems of underperforming 
mercurial server at hg.openjdk.java.net, combined with some issues with 
mercurial running on cygwin. :-(


The only advice I can give is to keep re-trying. Eventually you will get 
a clone, and subsequent operations will go smoother. If you have a 
forest on a local linux machine, try starting by making a clone manually 
from that one. (You'll need to clone each repo individually.)


/Magnus

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Martin Buchholz]

On Fri, Jan 25, 2019 at 10:07 AM Andrew Haley  wrote:
>
> On 1/25/19 5:01 PM, Martin Buchholz wrote:
> > I re-ran my linker performance experiment using  configure
> > --with-native-debug-symbols="internal"
> > lld is a big winner here:
>
> It looks to me like lld and multi-threaded gold would be a near tie. I
> think that lld uses multi-threading; I wonder why gold doesn't. But
> either is so fast linking libjvm.so that I no longer care.

repeating my last experiment with explicit --threads:

--- ld=gold ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--threads -Wl,--hash-style=both
-Wl,-z,defs  9.18s user 4.98s system 329% cpu 4.292 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--threads -Wl,--hash-style=both
-Wl,-z,defs  9.12s user 5.12s system 333% cpu 4.266 total
--- ld=lld ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--threads -Wl,--hash-style=both
-Wl,-z,defs  2.70s user 1.09s system 324% cpu 1.169 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--threads -Wl,--hash-style=both
-Wl,-z,defs  2.74s user 1.00s system 328% cpu 1.141 total

and with explicit --no-threads:

--- ld=gold ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--no-threads -Wl,--hash-style=both
-Wl,-z,defs   4.61s user 0.61s system 99% cpu 5.213 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--no-threads -Wl,--hash-style=both
-Wl,-z,defs   4.60s user 0.61s system 99% cpu 5.212 total
--- ld=lld ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--no-threads -Wl,--hash-style=both
-Wl,-z,defs   1.67s user 0.60s system 99% cpu 2.289 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--no-threads -Wl,--hash-style=both
-Wl,-z,defs   1.66s user 0.60s system 99% cpu 2.283 total

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Andrew Haley]

On 1/25/19 5:01 PM, Martin Buchholz wrote:
> I re-ran my linker performance experiment using  configure
> --with-native-debug-symbols="internal"
> lld is a big winner here:

It looks to me like lld and multi-threaded gold would be a near tie. I
think that lld uses multi-threading; I wonder why gold doesn't. But
either is so fast linking libjvm.so that I no longer care.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Philip Race]

Category 3 has to deal with plenty of native code too ...

-phil

On 1/25/19, 9:15 AM, Martin Buchholz wrote:

In our own wrappers around configure, we've introduced the concept of
a "developer mode".  But this thread suggests there are 3 populations
of users invoking configure:

1. release engineers
2. hotspot developers
3. java library developers

Category 1 doesn't care about edit-compile-debug cycle - they just
want to build a reliable high-performance jdk without pitfalls.  This
is the VAST MAJORITY of users, for whom we should design defaults in
configure.
Category 2 wants debug info and maybe incremental linking.  They might
cheat and rebuild only libjvm.so and drop that one file into a
previously built jdk as part of their development cycle.
Category 3 doesn't care about native debug symbols at all

Re: Help with build changes for: 8214796: Create a jlink plugin for stripping debug info symbols from native libraries

[Erik Joelsson]

Hello Severin,

To get the conditional running of either MergeProperties or copy, I 
would do something like this:


$$($1_TARGET): $$($1_DEPS)
$$(call MakeTargetDir)
ifneq ($$($1_NUM_INPUT_FILES),1) $$(call LogInfo, Merging $$(words 
$$($1_INPUT_FILES)) properties files into a single properties file for 
jdk.jlink)

$(TOOL_MERGE_PROPERTIES) \
$(GENERATED_JLINK_RESOURCES_DIR)/$$($1_PROP).properties \
$$($1_INPUT_FILES)
else
  $$(call LogInfo, Copying $$(patsubst $(OUTPUTDIR)/%,%,$$@)))
  $$(call install-file)
endif

  $1 := $$($1_TARGET)

Note that the make conditional lines need to be indented with space 
while the recipe lines (including the $$(call )) needs to be tabbed. We 
still like them to align though, assuming 8 spaces tabs.


Down in the call loop, the first argument to SetupPropertiesMerge must 
be unique. This can be achieved with MERGE_JLINK_PROPS_$(prop). Then add 
a line like this last in the loop:


$(eval TARGETS += $(MERGE_JLINK_PROPS_$(prop)))

To find all the src dirs for the jlink module, there are helper macros 
in make/common/Modules.gmk.


JLINK_RESOURCE_DIRS := $(addsuffix /jdk/tools/jlink/resources, $(call 
FindModuleSrcDirs, jdk.jlink))


In the loop I would recommend reusing the already built list of all 
properties files instead of calling find again. Also, if all properties 
files in the resource dir are in the same directory with no sub 
directories, then please consider using $(wildcard ) to find them as 
that's much less taxing on Windows.


Hope this helps!

/Erik

On 2019-01-25 07:27, Severin Gehwolf wrote:

Hi,

I'm working on an enhancement for jlink. In particular a platform
specific plugin. I.e. It would only get built on unix/linux platforms.
My trouble is getting some resouce properties set up properly. In my
example there is two versions of plugins.properties: one in
shared/classes one in unix/classes. These, need to get merged into one
via the MergeProperties build tool class. So far so good. But for the
cases where there is only one resource property it should just get
copied to support/gensrc and that source root be used for compiling
those properties into actual ListResourceBundle Java classes.

The copying from the src tree to the gensrc tree doesn't seem to work.
I've tried using $(CP) and SetupCopyFiles to no avail. Would anybody
willing to help?

WIP webrev is:
http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8214796-wip/01/webrev/

I thought I'd do the copying in an else branch in SetupPropertiesMerge,
where I've put in a FIXME comment. Am I on the wrong track?

Thanks,
Severin

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Martin Buchholz]

In our own wrappers around configure, we've introduced the concept of
a "developer mode".  But this thread suggests there are 3 populations
of users invoking configure:

1. release engineers
2. hotspot developers
3. java library developers

Category 1 doesn't care about edit-compile-debug cycle - they just
want to build a reliable high-performance jdk without pitfalls.  This
is the VAST MAJORITY of users, for whom we should design defaults in
configure.
Category 2 wants debug info and maybe incremental linking.  They might
cheat and rebuild only libjvm.so and drop that one file into a
previously built jdk as part of their development cycle.
Category 3 doesn't care about native debug symbols at all

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Andrew Haley]

On 1/25/19 4:45 PM, Magnus Ihse Bursie wrote:
> On 2019-01-25 16:42, Andrew Haley wrote:
>> On 1/24/19 2:04 PM, Magnus Ihse Bursie wrote:
>>> But I'm leaning more towards just enabling gold on x86_64 -- for other
>>> platforms, we might as well keep the good ol' bfd linker. Does that
>>> sound like a good solution to you?
>> I'd love it to be an option for AArch64.
>>
> I hear you.
> 
> My current thinking is that I'm going to introduce a configure argument, 
> something like --with-toolchain-linker=gold, and if it is enabled, I'll 
> verify that gold is indeed present, and if it supports --threads. It 
> will be available on all gcc platforms.
> 
> This will take some more time to write than my initial one-liner, so 
> I'll take this back to the drawing board for awhile.

Thank you.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Martin Buchholz]

I re-ran my linker performance experiment using  configure
--with-native-debug-symbols="internal"
lld is a big winner here:

--- ld=bfd ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  7.30s user 1.26s system 99% cpu 8.559 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  6.73s user 1.18s system 99% cpu 7.908 total
--- ld=gold ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  4.57s user 0.62s system 99% cpu 5.191 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  4.67s user 0.53s system 99% cpu 5.209 total
--- ld=lld ---
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  2.61s user 1.10s system 330% cpu 1.124 total
/usr/bin/g++ -fuse-ld=$ld -Wl,--hash-style=both -Wl,-z,defs
-Wl,-z,noexecstac  2.72s user 1.03s system 326% cpu 1.146 total


On Thu, Jan 24, 2019 at 10:49 AM Martin Buchholz  wrote:
>
> Here's an experiment using the 3 competing open source linkers to link
> hotspot.  This confirms that lld is faster than gold is faster than
> bfd, but is the one second saving worth the engineering effort?
>
>  $ (BUILDDIR=$HOME/ws/jdk/build/linux-x86_64-server-release; for
> linker in bfd gold lld; do echo --- $linker ---; time /usr/bin/g++
> -fuse-ld=$linker -Wl,--hash-style=both -Wl,-z,defs -Wl,-z,noexecstack
> -Wl,-O1 -Wl,-z,relro -m64 -static-libstdc++ -static-libgcc -shared
> -m64 -Wl,-version-script=$BUILDDIR/hotspot/variant-server/libjvm/mapfile
> -Wl,-soname=libjvm.so -o
> $BUILDDIR/support/modules_libs/java.base/server/libjvm.so
> @$BUILDDIR/hotspot/variant-server/libjvm/objs/_BUILD_LIBJVM_objectfilenames.txt
> -lm -ldl -lpthread; done)
> --- bfd ---
> /usr/bin/g++ -fuse-ld=$linker -Wl,--hash-style=both -Wl,-z,defs
> -Wl,-O1  -m6  1.31s user 0.36s system 99% cpu 1.669 total
> --- gold ---
> /usr/bin/g++ -fuse-ld=$linker -Wl,--hash-style=both -Wl,-z,defs
> -Wl,-O1  -m6  0.42s user 0.11s system 99% cpu 0.537 total
> --- lld ---
> /usr/bin/g++ -fuse-ld=$linker -Wl,--hash-style=both -Wl,-z,defs
> -Wl,-O1  -m6  0.25s user 0.20s system 145% cpu 0.310 total

Re: Help with build changes for: 8214796: Create a jlink plugin for stripping debug info symbols from native libraries

[Magnus Ihse Bursie]

On 2019-01-25 16:27, Severin Gehwolf wrote:

Hi,

I'm working on an enhancement for jlink. In particular a platform
specific plugin. I.e. It would only get built on unix/linux platforms.
My trouble is getting some resouce properties set up properly. In my
example there is two versions of plugins.properties: one in
shared/classes one in unix/classes. These, need to get merged into one
via the MergeProperties build tool class. So far so good. But for the
cases where there is only one resource property it should just get
copied to support/gensrc and that source root be used for compiling
those properties into actual ListResourceBundle Java classes.

The copying from the src tree to the gensrc tree doesn't seem to work.
I've tried using $(CP) and SetupCopyFiles to no avail. Would anybody
willing to help?
SetupCopyFile should work, but in this case I think it's simplest to 
just use install-file. Like this perhaps:


ifneq ($$($1_NUM_INPUT_FILES), 1)
  $$($1_TARGET): $$($1_INPUT_FILES)
  $$(call LogInfo, Merging $$($1_NUM_INPUT_FILES) properties files 
into a single properties file for jdk.jlink)

  $(TOOL_MERGE_PROPERTIES) \
$(GENERATED_JLINK_RESOURCES_DIR)/$$($1_PROP).properties \
  $$($1_INPUT_FILES)
else
  $$($1_TARGET): $$($1_INPUT_FILES)
    $$(install-file)
endif

$1 += $$($1_TARGET)

/Magnus

$$($1_TARGET): $$($1_DEPS)





WIP webrev is:
http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8214796-wip/01/webrev/

I thought I'd do the copying in an else branch in SetupPropertiesMerge,
where I've put in a FIXME comment. Am I on the wrong track?

Thanks,
Severin

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Magnus Ihse Bursie]

On 2019-01-25 16:42, Andrew Haley wrote:

On 1/24/19 2:04 PM, Magnus Ihse Bursie wrote:

But I'm leaning more towards just enabling gold on x86_64 -- for other
platforms, we might as well keep the good ol' bfd linker. Does that
sound like a good solution to you?

I'd love it to be an option for AArch64.


I hear you.

My current thinking is that I'm going to introduce a configure argument, 
something like --with-toolchain-linker=gold, and if it is enabled, I'll 
verify that gold is indeed present, and if it supports --threads. It 
will be available on all gcc platforms.


This will take some more time to write than my initial one-liner, so 
I'll take this back to the drawing board for awhile.


/Magnus

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Andrew Haley]

On 1/24/19 2:04 PM, Magnus Ihse Bursie wrote:
> But I'm leaning more towards just enabling gold on x86_64 -- for other 
> platforms, we might as well keep the good ol' bfd linker. Does that 
> sound like a good solution to you?

I'd love it to be an option for AArch64.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Andrew Haley]

On 1/25/19 2:39 PM, Andrew Haley wrote:
> Massive win for gold here, on my (very old) AArch64 system:

And on another, faster AArch64 system:

BFD linker:

real0m14.532s
user0m13.070s
sys 0m1.419s

gold, single-threaded:

real0m6.313s
user0m6.123s
sys 0m0.190s

gold, multi-threaded:

real0m4.129s
user0m8.724s
sys 0m1.741s

Verdict: yes, please!

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Help with build changes for: 8214796: Create a jlink plugin for stripping debug info symbols from native libraries

[Severin Gehwolf]

Hi,

I'm working on an enhancement for jlink. In particular a platform
specific plugin. I.e. It would only get built on unix/linux platforms.
My trouble is getting some resouce properties set up properly. In my
example there is two versions of plugins.properties: one in
shared/classes one in unix/classes. These, need to get merged into one
via the MergeProperties build tool class. So far so good. But for the
cases where there is only one resource property it should just get
copied to support/gensrc and that source root be used for compiling
those properties into actual ListResourceBundle Java classes.

The copying from the src tree to the gensrc tree doesn't seem to work.
I've tried using $(CP) and SetupCopyFiles to no avail. Would anybody
willing to help?

WIP webrev is:
http://cr.openjdk.java.net/~sgehwolf/webrevs/JDK-8214796-wip/01/webrev/

I thought I'd do the copying in an else branch in SetupPropertiesMerge,
where I've put in a FIXME comment. Am I on the wrong track?

Thanks,
Severin

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[Ao Qi]

Hi John,

On Fri, Jan 25, 2019 at 11:10 PM Ao Qi  wrote:
>
> On Fri, Jan 25, 2019 at 10:51 PM John Paul Adrian Glaubitz
>  wrote:
> >
> > On 1/25/19 3:44 PM, Ao Qi wrote:> FYI, if one wants to debug this issue, 
> > below is some sctripts I used
> > > to cross compile a mips zero jdk, which can trigger the bug.
> >
> > Native builds on Debian unstable on mips* work?
> >
>

Correction. We use Debian stable and testing to do night build (only
mips64el). We did not use Debian unstable, Is the difference big?

> Worked. We nightly native (using Loongson CPUs) build zero mips of
> jdk8-12 and jdk. But zero mips is not stable as boot jdk, some random
> crash occurs. I am not sure it is an issue of zero mips or an issue of
> MIPS for example maybe some memory consistency issue.
>
>
> > > do one patch [1] then make images
> >
> > What about the alignment patch? Is that still required?
> >
>
> We do not use this patch.
>
> > > https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff
> >
> > Adrian
> >
> > --
> >  .''`.  John Paul Adrian Glaubitz
> > : :' :  Debian Developer - glaub...@debian.org
> > `. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
> >   `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[Ao Qi]

On Fri, Jan 25, 2019 at 10:51 PM John Paul Adrian Glaubitz
 wrote:
>
> On 1/25/19 3:44 PM, Ao Qi wrote:> FYI, if one wants to debug this issue, 
> below is some sctripts I used
> > to cross compile a mips zero jdk, which can trigger the bug.
>
> Native builds on Debian unstable on mips* work?
>

Worked. We nightly native (using Loongson CPUs) build zero mips of
jdk8-12 and jdk. But zero mips is not stable as boot jdk, some random
crash occurs. I am not sure it is an issue of zero mips or an issue of
MIPS for example maybe some memory consistency issue.


> > do one patch [1] then make images
>
> What about the alignment patch? Is that still required?
>

We do not use this patch.

> > https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff
>
> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaub...@debian.org
> `. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
>   `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[Ao Qi]

Hi,

On Fri, Jan 25, 2019 at 10:31 PM John Paul Adrian Glaubitz
 wrote:
>
> Hi!
>
> On 1/24/19 3:28 AM, Leslie Zhai wrote:
> > Please give me some advice about how to fix the root cause, thanks a lot!
>
> Just as a heads-up: In Debian we have two patches required for OpenJDK on the 
> mips
> targets:
>
> > https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff
> > https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/mips-sigset.diff
>
> I think the stuff with sigset was being worked on upstream at some point, but 
> it
> wasn't finished if I remember correctly.
>

Yes, I wish to solve this problem, I tried, and it was basically the
way Debian and AIX used [1]. But David Holmes wishes a POSIX version,
that patch failed to be pushed. I think I will go on to work on it,
however I do not have Solaris and macOS for test, I am working on
preparing these OSs.

What is 
https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff
for? We build mips zero without this patch.

Cheers,
Ao Qi

[1] 
https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2018-December/031839.html

> Adrian
>
> --
>  .''`.  John Paul Adrian Glaubitz
> : :' :  Debian Developer - glaub...@debian.org
> `. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
>   `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[John Paul Adrian Glaubitz]

On 1/25/19 3:44 PM, Ao Qi wrote:> FYI, if one wants to debug this issue, below 
is some sctripts I used
> to cross compile a mips zero jdk, which can trigger the bug.

Native builds on Debian unstable on mips* work?

> do one patch [1] then make images

What about the alignment patch? Is that still required?

> https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[Ao Qi]

Hi,

FYI, if one wants to debug this issue, below is some sctripts I used
to cross compile a mips zero jdk, which can trigger the bug.

build machine: x86, debian testing (in a VirtualBox; an ubuntu may
fail to configure)

known worked targets: ppc64le/server and s390x/server (aarch64/server
may fail during make images)

repository: http://hg.openjdk.java.net/jdk/jdk

geting a sysroot:

#export arch=arm64
#export arch=ppc64el
#export arch=s390x
export arch=mips64el
export ver=stretch
apt-get install qemu-user-static sudo debootstrap binfmt-support qemu
(not sure if all of them are required)
sudo qemu-debootstrap --arch=${arch} --verbose \
   
--include=fakeroot,build-essential,libx11-dev,libxext-dev,libxrender-dev,libxrandr-dev,libxtst-dev,libxt-dev,libcups2-dev,libfontconfig1-dev,libasound2-dev,libfreetype6-dev,libpng-dev,libffi-dev
\
  --resolve-deps ${ver} /chroots/${arch}_${ver}
http://httpredir.debian.org/debian/

configure of jdk:

#arch1=aarch64 ; arch2=arm64; gnuname=gnu
#arch1=powerpc64le ; arch2=ppc64el; gnuname=gnu
#arch1=s390x ; arch2=s390x; gnuname=gnu
arch1=mips64el ; arch2=mips64el; gnuname=gnuabi64
apt install g++-${arch1}-linux-${gnuname} gcc-${arch1}-linux-${gnuname}
ver=stretch
CC=$arch1-linux-${gnuname}-gcc CXX=$arch1-linux-${gnuname}-g++ \
  sh ./configure --openjdk-target=$arch1-linux-${gnuname}
--with-sysroot=/chroots/${arch2}_${ver}/
--with-toolchain-path=/chroots/${arch2}_${ver}/ \
  --with-boot-jdk=/home/loongson/aoqi/jdk-13/ --disable-warnings-as-errors \
  --with-jvm-variants=zero \
  --x-libraries=/chroots/${arch2}_${ver}/usr/lib/$arch1-linux-${gnuname}
--x-include=/chroots/${arch2}_${ver}/usr/include/X11/ \
  --with-freetype-lib=/chroots/${arch2}_${ver}/usr/lib/$arch1-linux-${gnuname}/
--with-freetype-include=/chroots/${arch2}_${ver}/usr/include/freetype2

make:

do one patch [1] then make images

I am trying Magnus's approach. When there are results, I will reply.

[1] 
https://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2018-December/031839.html

On Fri, Jan 25, 2019 at 1:19 AM Erik Joelsson  wrote:
>
> Hello,
>
> I agree with the conclusion here, there is certainly a bug in the build
> logic. When creating the "Build JDK" during a cross compilation build,
> we are, as Magnus said, cutting a few corners for efficiency. One of
> those corners is assuming all the java classes are the same and can
> simply be reused. It has been shown here that that assumption is not
> always correct.
>
> I think the most reasonable solution is to assume all of java.base needs
> to be built specifically for the "Build JDK". I think this can be
> reasonably easy to implement (compared to the existing Build JDK logic,
> which is arguably rather hairy).
>
> Magnus' suggested workaround should work fine for now however.
>
> Filed https://bugs.openjdk.java.net/browse/JDK-8217739
>
> /Erik
>
> On 2019-01-24 05:30, Magnus Ihse Bursie wrote:
> > On 2019-01-24 13:22, Leslie Zhai wrote:
> >> Hi Magnus,
> >>
> >> Thanks for your kind response!
> >>
> >> 在 2019/1/24 下午8:05, Magnus Ihse Bursie 写道:
> >>>
> >>>
> >>> On 2019-01-24 05:45, David Holmes wrote:
>  On 24/01/2019 1:51 pm, Ao Qi wrote:
> > Hi David,
> >
> > On Thu, Jan 24, 2019 at 10:47 AM David Holmes
> >  wrote:
> >>
> >> Hi Leslie,
> >>
> >> I'm not Erik :) however 
> >>
> >> On 24/01/2019 12:28 pm, Leslie Zhai wrote:
> >>> Hi Erik,
> >>>
> >>> Because the flags' macro of MIPS is different from other
> >>> targets[1], it
> >>> will effect the `UnixConstants` of
> >>> src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template,
> >>> for
> >>> example, PREFIX_O_APPEND, so the `Flags` in `FileChannel` is
> >>> different
> >>> from other targets.  Then failed[2] to cross compiling jdk12 for
> >>> mips64el-linux-gnu target by following the document[3].
> >>
> >> It's been quite a while since we built for a platform where the
> >> flags
> >> had different values, but it is supposed to work. I suspect that
> >> the way
> >> we build now with the module system may not be quite correct in this
> >> regard - but that is just supposition on my part. Because
> >> compile-time
> >> constants are stored directly into class files that use them, we
> >> have to
> >> generate UnixConstants.java before compiling any class that needs
> >> to use
> >> it, then we must compile all classes that do use it. Those
> >> classes must
> >> only be executed in conjunction with a tool executing on the target
> >> platform (javac, jmod etc). My suspicion is that we may be
> >> executing a
> >> tool on the build platform using the newly compiled classes for the
> >> target platform - and that would be wrong.
> >>
> >
> > Yes. Some variables in UnixConstants.java is defined according to the
> > target (fcntl.h of mips in this case), but would be executed on host
> > (x86 in this 

Re: RFR: JDK-8217723 Switch ld from bfd to gold on gcc toolchain

[Andrew Haley]

Massive win for gold here, on my (very old) AArch64 system:

BFD linker:

 $ time /home/aph/gcc-x/aarch64-trunk-install/bin/g++ -Wl,-z,defs -fuse-ld=bfd 
-Wl,-z,noexecstack ...

real0m26.090s
user0m24.570s
sys 0m1.480s

gold, single-threaded:

 $ time /home/aph/gcc-x/aarch64-trunk-install/bin/g++ -Wl,-z,defs -fuse-ld=gold 
-Wl,-z,noexecstack ...

real0m12.325s
user0m12.030s
sys 0m0.280s

gold, multi-threaded:

 $ time /home/aph/gcc-x/aarch64-trunk-install/bin/g++ -Wl,-z,defs -fuse-ld=gold 
-Wl,--threads -Wl,-z,noexecstack ...

real0m8.297s
user0m14.680s
sys 0m1.650s

I would very much like to see this change go in, with a probe to make
sure that -fuse-ld=gold works. There is unfortunately one gotcha: on
some systems gold is built without threading support (why,
fercrineoutloud?) so it's necessary to probe for -Wl,--threads
separately.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[John Paul Adrian Glaubitz]

Hi!

On 1/24/19 3:28 AM, Leslie Zhai wrote:
> Please give me some advice about how to fix the root cause, thanks a lot!

Just as a heads-up: In Debian we have two patches required for OpenJDK on the 
mips
targets:

> https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/hotspot-mips-align.diff
> https://git.launchpad.net/~openjdk/ubuntu/+source/openjdk/+git/openjdk/tree/debian/patches/mips-sigset.diff

I think the stuff with sigset was being worked on upstream at some point, but it
wasn't finished if I remember correctly.

Adrian

-- 
 .''`.  John Paul Adrian Glaubitz
: :' :  Debian Developer - glaub...@debian.org
`. `'   Freie Universitaet Berlin - glaub...@physik.fu-berlin.de
  `-GPG: 62FF 8A75 84E0 2956 9546  0006 7426 3B37 F5B5 F913

Re: Fail to cross compiling jdk12 for mips64el-linux-gnu target

[Magnus Ihse Bursie]

On 2019-01-25 02:47, Leslie Zhai wrote:

Hi Magnus,

Failed to make images:

$ make images CONF=mips
Building target 'images' in configuration 
'linux-mips64el-normal-server-fastdebug'
GenerateLinkOptData.gmk:61: recipe for target 
'/home/loongson/zhaixiang/jdk12-mips/build/linux-mips64el-normal-server-fastdebug/support/link_opt/classlist' 
failed
make[3]: *** 
[/home/loongson/zhaixiang/jdk12-mips/build/linux-mips64el-normal-server-fastdebug/support/link_opt/classlist] 
Error 1
make[3]: *** Deleting file 
'/home/loongson/zhaixiang/jdk12-mips/build/linux-mips64el-normal-server-fastdebug/support/link_opt/classlist'

make/Main.gmk:441: recipe for target 'generate-link-opt-data' failed
make[2]: *** [generate-link-opt-data] Error 2


This should have worked, but sounds like something that can typically 
fail when cross-compiling. As a workaround, disable class list 
generation: "configure --disable-generate-classlist", or even "configure 
--disable-cds".


/Magnus


ERROR: Build failed for target 'images' in configuration 
'linux-mips64el-normal-server-fastdebug' (exit code 2)


=== Make failed targets repeated here ===
GenerateLinkOptData.gmk:61: recipe for target 
'/home/loongson/zhaixiang/jdk12-mips/build/linux-mips64el-normal-server-fastdebug/support/link_opt/classlist' 
failed

make/Main.gmk:441: recipe for target 'generate-link-opt-data' failed
=== End of repeated output ===

Hint: Try searching the build log for the name of the first failed 
target.

Hint: See doc/building.html#troubleshooting for assistance.

/home/loongson/zhaixiang/jdk12-mips/make/Init.gmk:300: recipe for 
target 'main' failed

make[1]: *** [main] Error 2
/home/loongson/zhaixiang/jdk12-mips/make/Init.gmk:186: recipe for 
target 'images' failed

make: *** [images] Error 2

- 8<  8<  8<  8<  8<  8< ---

My cross-compile configure option:

CC=mips64el-linux-gnuabi64-gcc CXX=mips64el-linux-gnuabi64-g++ sh 
./configure \

  --openjdk-target=mips64el-linux-gnuabi64 \
  --with-sysroot=/chroots/mips64el_stretch \
  --with-toolchain-path=/chroots/mips64el_stretch \
  --with-boot-jdk=/home/loongson/aoqi/jdk-11.0.1 \
  --disable-warnings-as-errors \
--with-freetype-lib=/chroots/mips64el_stretch/usr/lib/mips64el-linux-gnuabi64 
\

--with-freetype-include=/chroots/mips64el_stretch/usr/include/freetype2 \
  --with-debug-level=fastdebug \
  --with-build-jdk=build/buildjdk/jdk

- 8<  8<  8<  8<  8<  8< ---

buildjdk is OK:

$ sh configure --with-conf-name=buildjdk 
--with-boot-jdk=/home/loongson/aoqi/jdk-11.0.1 
--disable-warnings-as-errors


$ make jdk CONF=buildjdk

Building target 'jdk' in configuration 'buildjdk'
Finished building target 'jdk' in configuration 'buildjdk'

Regards,

Leslie Zhai


在 2019/1/24 下午9:30, Magnus Ihse Bursie 写道:

On 2019-01-24 13:22, Leslie Zhai wrote:

Hi Magnus,

Thanks for your kind response!

在 2019/1/24 下午8:05, Magnus Ihse Bursie 写道:



On 2019-01-24 05:45, David Holmes wrote:

On 24/01/2019 1:51 pm, Ao Qi wrote:

Hi David,

On Thu, Jan 24, 2019 at 10:47 AM David Holmes 
 wrote:


Hi Leslie,

I'm not Erik :) however 

On 24/01/2019 12:28 pm, Leslie Zhai wrote:

Hi Erik,

Because the flags' macro of MIPS is different from other 
targets[1], it

will effect the `UnixConstants` of
src/java.base/unix/classes/sun/nio/fs/UnixConstants.java.template, 
for
example, PREFIX_O_APPEND, so the `Flags` in `FileChannel` is 
different

from other targets.  Then failed[2] to cross compiling jdk12 for
mips64el-linux-gnu target by following the document[3].


It's been quite a while since we built for a platform where the 
flags
had different values, but it is supposed to work. I suspect that 
the way
we build now with the module system may not be quite correct in 
this
regard - but that is just supposition on my part. Because 
compile-time
constants are stored directly into class files that use them, we 
have to
generate UnixConstants.java before compiling any class that 
needs to use
it, then we must compile all classes that do use it. Those 
classes must

only be executed in conjunction with a tool executing on the target
platform (javac, jmod etc). My suspicion is that we may be 
executing a

tool on the build platform using the newly compiled classes for the
target platform - and that would be wrong.



Yes. Some variables in UnixConstants.java is defined according to 
the

target (fcntl.h of mips in this case), but would be executed on host
(x86 in this case) during "make images". These variables of ppc64le,
aarch64 and s390x are the same as x86, but the ones of mips are not.
However, mips is not an official supported target and I did not find
other targets have similar issue, I am not sure if this is a bug 
or if

this is the right mailing list :)


This is the right mailing list to discuss. :)

So I went back to find when this happened in the past and I 
mis-remembered how we solved it. We previously had a problem with 
the Oracle 

RE: RFR (S) [11u backport]: 8207849: Allow the addition of more number to the Java version string

[Langer, Christoph]

Thanks, Magnus for the review.

I shall try wiggle 

> -Original Message-
> From: Magnus Ihse Bursie 
> Sent: Freitag, 25. Januar 2019 11:21
> To: Langer, Christoph ; 'build-
> d...@openjdk.java.net' 
> Cc: Zeller, Arno 
> Subject: Re: RFR (S) [11u backport]: 8207849: Allow the addition of more
> number to the Java version string
> 
> 
> 
> On 2019-01-25 10:31, Langer, Christoph wrote:
> > Hi,
> >
> > may I please get reviews for the backport of this issue to jdk11u.
> >
> > Bug: https://bugs.openjdk.java.net/browse/JDK-8207849
> > Original Commit: http://hg.openjdk.java.net/jdk/jdk/rev/1edc62f9ba3a
> > Original review thread: https://mail.openjdk.java.net/pipermail/build-
> dev/2018-July/022719.html
> > Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8207849.11u/
> >
> > The patch did not apply cleanly, but only because in
> "make/autoconf/version-numbers" some version numbers have changed in
> the meanwhile and the apply tool did not find the right spot to insert the
> additional variables.
> >
> > Thanks
> > Christoph
> 
> Looks good to me.
> 
> When a patch fail to apply cleanly, I regularly try with the "wiggle"
> tool. [1] It is much more resilient to harmless changes than "patch".
> The backport rule of "applies cleanly" still requires you to send a RFR,
> but using wiggle saves your effort, and including a note about it in the
> RFR will increase reviewers confidence.
> 
> /Magnus
> 
> [1] https://github.com/neilbrown/wiggle

Re: RFR (S) [11u backport]: 8207849: Allow the addition of more number to the Java version string

[Magnus Ihse Bursie]

On 2019-01-25 10:31, Langer, Christoph wrote:

Hi,

may I please get reviews for the backport of this issue to jdk11u.

Bug: https://bugs.openjdk.java.net/browse/JDK-8207849
Original Commit: http://hg.openjdk.java.net/jdk/jdk/rev/1edc62f9ba3a
Original review thread: 
https://mail.openjdk.java.net/pipermail/build-dev/2018-July/022719.html
Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8207849.11u/

The patch did not apply cleanly, but only because in 
"make/autoconf/version-numbers" some version numbers have changed in the 
meanwhile and the apply tool did not find the right spot to insert the additional 
variables.

Thanks
Christoph


Looks good to me.

When a patch fail to apply cleanly, I regularly try with the "wiggle" 
tool. [1] It is much more resilient to harmless changes than "patch". 
The backport rule of "applies cleanly" still requires you to send a RFR, 
but using wiggle saves your effort, and including a note about it in the 
RFR will increase reviewers confidence.


/Magnus

[1] https://github.com/neilbrown/wiggle

RFR (S) [11u backport]: 8207849: Allow the addition of more number to the Java version string

[Langer, Christoph]

Hi,

may I please get reviews for the backport of this issue to jdk11u.

Bug: https://bugs.openjdk.java.net/browse/JDK-8207849
Original Commit: http://hg.openjdk.java.net/jdk/jdk/rev/1edc62f9ba3a
Original review thread: 
https://mail.openjdk.java.net/pipermail/build-dev/2018-July/022719.html
Webrev: http://cr.openjdk.java.net/~clanger/webrevs/8207849.11u/

The patch did not apply cleanly, but only because in 
"make/autoconf/version-numbers" some version numbers have changed in the 
meanwhile and the apply tool did not find the right spot to insert the 
additional variables.

Thanks
Christoph