Hi Kevin,
On 05/11/16 22:33, Kevin Burton wrote:
We use TeamCity internally (which is great btw) for Maven testing.
We have about 2000 tests which we continually integrate on every commit.
The problem is that testing takes about 15 minutes from start to end.
Is this really the time for the tests only or is this the whole build
time including the test time? If it is only the time for running the
tests it means those tests are slow...
2000 Test / 15 Min = 2.2 Tests / Second...
A usual value is about by 10-100+ Tests / Second...
(My current project is running 5000 unit tests within 350 seconds (which
is slow as well)...but has 14.X tests / second.
Having a log file of the maven run available you can summarize the run
time of the tests only by using something like this (on a linux like
system):
cat logFile | egrep -a "^Tests run: [0-9]+, Failures: [0-9]+, Errors:
[0-9]+, Skipped: [0-9]+, Time elapsed: " | cut -d":" -f 6 | cut -d" "
-f2 | paste -sd+ | bc -l
This will summarize the time for the tests only...
We use -T 16 on our tests and our boxes have 8 cores so this allows some
tests to block on IO while others execute.
-T 16 tries to parallellize the module build and not parallize the
tests...furthermore I have my doubts that using -T 16 is faster than -T
4 etc..
I don't know how many modules you have in your build. If you have really
many modules like 100+ it might be a good idea to consider using Smart
Builder[1].
If you like to parallize the unit tests themself you have to configure
maven-surefire-plugin accordingly...I recommend reading the doc about
that subject[2].
Kind regards
Karl Heinz Marbaise
[1]: http://takari.io/book/30-team-maven.html
[2]:
http://maven.apache.org/components/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html
However, I'm not happy with this. I want our tests to finish in 2 minutes.
The only way this could happen is for us to hand parallelize everything (no
fun), or do it automatically, or buy REALLY expensive hardware (no fun).
I think one could do this with JUnit categories and maven profiles.
I think what one could do is hash the name of the test and then based on
the hash prefix, create N buckets.
Right now we have 4 CI boxes so we would run 4 profiles, one per box. Then
TeamCity could integrate the results.
I think this is technically possible now but I think a lot of plugins would
need to cooperate here.
It would be better if maven didn't need to explicitly have to split these
out.
Maybe update surefire to have a way to only run 1/Nth of the tests and make
the set determinstic?
This way you can use two build boxes and each have two sets of tests with
no intersection.
Thoughts on this approach? I think it would be pretty awesome.
I'm sure someone here knows the gang at TeamCity and could recommend this
to the proper decision makers.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
