Hi Martin,
On 9/16/2015 7:36 PM, Martin Buchholz wrote:
OK, thanks, I managed to run my first tier test.
What I (sometimes) really want is to apply a test filter to my test set:
- those tests that are non-flaky and don't take "too many" resources
to run
You could harvest both of those qualities from your many quality
engineering test runs.
FWIW, we see reasonable run times (20 minutes or less) on many platforms
for running each of tier 1 and tier 2 tests if run on a well-sized
server when using jtreg's agentvm mode and concurrency options.
-Joe
On Wed, Sep 16, 2015 at 6:32 PM, Joseph D. Darcy <joe.da...@oracle.com
<mailto:joe.da...@oracle.com>> wrote:
Hi Martin,
On 9/16/2015 6:17 PM, Martin Buchholz wrote:
Thanks Joe.
I see those tier definitions in the various TEST.groups files but
it's not obvious to me how to run them all with one simple
invocation (neither make nor jtreg).
Make support is coming, but not there yet:
JDK-8075571: Support tiered testing make targets
The command line you gave doesn't seem to tell jtreg which
TEST.ROOT directories to operate on.
It looks like
jtreg ... langtools/test:tier1 jdk/test:tier1
works, but I was looking for a simpler way to "run all the tier1
tests" without enumerating all the roots (there are currently 7!)
That is the simple(st) way currently.
The jdk, langtools, nashorn, and jaxp repos currently use the
tiered testing approach as defined in jtreg groups so you would
"only" need four roots.
The tier 2 tests are pretty stable too, but we certainly have
instances were new repeatable failures are introduced, but we're
getting better at addressing them quickly :-)
-Joe
On Wed, Sep 16, 2015 at 5:25 PM, Joseph D. Darcy
<joe.da...@oracle.com <mailto:joe.da...@oracle.com>> wrote:
Dropping the jtreg alias.
On 9/16/2015 5:18 PM, Martin Buchholz wrote:
Just now for the first time I ran all the jtreg tests in
the "jdk_stable" test set using
make test TEST=jdk_stable
and I got around 50 failures out of 5000.
Is that expected?
Instead, I expect that with all the work on the test
suite y'all have been doing, that the expected failure
rate of something with a name like "jdk_stable" would be 0.
For JDK 9 dev, the tier 1 tests across repos should be
stable. Currently, only the langtools and jdk repos have
non-empty tier 1 tests sets defined.
If you run
jtreg ... -exclude:ProblemList.txt -k:intermittent :tier1
there are some tier 1 test that are known to fail
intermittently, but at this point with very low frequency (at
least with the testing I'm exposed to).
HTH,
-Joe
