[
https://issues.apache.org/jira/browse/HBASE-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack reopened HBASE-23779:
-----------------------------------
Reopening. Was resolved because thought there no elbow room here but have been
making progress on this master project off in side issues. Let me bring the
work together back here under this issue.
High-level, trying to run w/ more parallelism -- forkcount and the mvn -T -- we
ran into limits. Experiments trying to up our forkcount to 0.5C (8 cores on
apache jenkins) ran into container memory and host/user ulimit upper-bounds.
Returning fork count down to 0.25C re-stabilized builds (along w/ test fixes).
Minimally would at least like to add -T0.25C to maven before resolving this
issue again. Maximally, would like to get build running on jenkins at 0.5C (In
local tests build takes twice as long if forkcount+ T arg are halved... say
from 0.5C to 0.25C).
In local tests, I can run with 1.0C. On a 40CPU linux machine, it is a bit
wobbly. It was wobbly on big GCE instances but I should revisit. On a 16CPU
mac, it seems fine as on a 4CPU VM. For Jenkins, 0.5C might be doable. Lets try
again now we know more.
> Up the default fork count to make builds complete faster; make count relative
> to CPU count
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-23779
> URL: https://issues.apache.org/jira/browse/HBASE-23779
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Michael Stack
> Assignee: Michael Stack
> Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments: addendum2.patch, test_yetus_934.0.patch
>
>
> Tests take a long time. Our fork count running all tests are conservative --
> 1 (small) for first part and 5 for second part (medium and large). Rather
> than hardcoding we should set the fork count to be relative to machine size.
> Suggestion here is 0.75C where C is CPU count. This ups the CPU use on my box.
> Looking up at jenkins, it seems like the boxes are 24 cores... at least going
> by my random survey. The load reported on a few seems low though this not
> representative (looking at machine/uptime).
> More parallelism willl probably mean more test failure. Let me take a look
> see.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
