On 2015-09-15 02:53, Maurizio Cimadamore wrote:
Hi Erik,
thanks for the explanation.
Regarding build times, the current heuristics scores ok on my
high-end machine (I get more or less same time as with JDK 8 build) -
but with a lower spec machine (i.e. laptop with dual core intel i5) it
gets much much worse - i.e. I used to be able to build in 7 minutes on
my laptop (using ccache) - now build time is at least double that figure.
I believe the major reason for your degradation of build performance in
JDK 9 vs JDK 8 on a low end machine is caused by us splitting the java
compilation into a per module model. In JDK 8, all java code was
compiled in one chunk. On a high end machine, splitting doesn't cause as
much degradation as many modules are compiled in parallel, making up for
the lost time of restarting the JVM each time. I think most of this loss
will be made up when we introduce server javac, where the jvm will be
warmed up and reused for all java compilations.
Have you tested reducing the number of jobs (make JOBS=2) on your i5? If
that does indeed improve build times on JDK 9, I would be surprised, but
then we definitely need to change the heuristics. In my experience
lowering the JOBS number does not have a positive impact on build times
any system.
/Erik
I know it's an hard problem to decide how many cores to use but there
seem to be a pattern emerging:
* low-end machines get completely swamped by the build load
* CPU bound tests run into troubles when reusing same concurrency
settings, even on high-end hardware. Without playing with timeouts
it's impossible to get a clean test sheet.
* on relatively high-end HW, current build concurrency settings seem
to be doing ok.
Realistically, I believe anything that uses more than n/2 virtual
processors is going to face troubles sooner or later; the build might
be ok since there's so much IO going on (reading/writing files) - but
the more the build will become CPU intensive (and sjavac might help
with that) the more current settings could become a bottleneck.
Maurizio
On 14/09/15 17:05, Erik Joelsson wrote:
Hello,
When I implemented the heuristic to choose a suitable default
concurrency, I only ever worried about the build. I think having
tests use the same concurrency setting must be a new feature? In any
case, it seems like there is a case for reducing concurrency when
running tests.
Another note. It at least used to be quite tricky to get correct
information about cores vs hyperthreading from the OS. I know today
we aren't even consistent with this across platforms. Perhaps we
should revisit this heuristic and take hyperthreading into
consideration too.
The current implemenation uses 100% of number of virtual cpus when 1
to 4 of them, then 90% at 5 to 16. After that it caps out at 16. (I
might remember some detail wrong here)
/Erik
On 2015-09-14 04:10, Maurizio Cimadamore wrote:
The information I posted was slightly incorrect, sorry - my machine
has 8 cores (and 16 virtual processors) - so you see why choosing
concurrency factor of 14 is particularly bad in this setup.
Maurizio
On 14/09/15 12:03, Maurizio Cimadamore wrote:
Hi,
I realized that the concurrency factor inferred by the JDK build
might be too high; on a 16 core machine, concurrency is set to 14 -
which then leads to absurd load averages (50-ish) when
building/running tests. High load when building is not a big issue,
but when running test this almost always turns into spurious
failures due to timeouts. I know I can override the concurrency
factor with --with-jobs - but I was curious as to why the default
parameter is set to such aggressive value?
Thanks
Maurizio
