So I tried using unit-threaded to run Phobos unit tests again and
had problems (which I'll look into later) with its compile-time
reflection. Then I realised I was an idiot since I don't need to
reflect on anything: all Phobos tests are in unittest blocks so
all I need to do is include them in the build and unit-threaded
will run them for me.
I tried a basic sanity check by running them in one thread only
with the -s option and got a segfault, and a failing test before
that. None of this should happen, and I'll be taking a look at
that as well.
But I carried on by removing the troublesome modules from the
build. These turned out to be:
std.datetime (fails)
std.process (fails and causes the segfault)
std.stdio (fails)
All the others pass in single threaded mode. After this I tried
using threads and std.parallelism failed, so I took that away
from the build as well.
Another thing to mention is that although the tests are running
in threads, since when I wrote the library the getUnitTests
__traits wasn't available (and since then I wasn't interested in
using it), each module's unit tests run as one test. So they only
interleave with other modules, not with each other.
Running in one thread took 39 +/- 1 seconds.
Running in 8 threads took... ~41 seconds.
Oops. I noticed some tests take a lot longer so I tried removing
those. They were:
std.file
std.conv
std.regex
std.random
std.container
std.xml
std.utf
std.numeric
std.uuid
std.exception
I also removed any modules that were likely to be problematic
like std.concurrency and std.socket. With the reduced sample size
the results were:
1 thread: ~1.9s
8 threads: 4.1s +/- 0.2
So the whole threading thing isn't looking so great. Or at least
not how I implemented it. This got me thinking about my own
projects. The tests run so fast I never really paid attention to
how fast they were running. I compared running the unit tests in
Cerealed in one or more threads and got the same result: running
in one thread was faster.
I have to look to be sure but maybe the bottleneck is output. As
in actually printing the results to the screen. I had to jump
through a few hoops to make sure the output wasn't interleaved,
and in the end decided to have one thread be responsible for
that, with the tests sending it output messages.
For reference, I copied all of the std/*.d modules into a local
std directory, compiled all of them with dmd -unittest -c, then
used this as the build command:
dmd -unittest -I~/coding/d/unit-threaded/source ut.d
std/algorithm.o std/array.o std/ascii.o std/base64.o std/bigint.o
std/bitmanip.o std/compiler.o std/complex.o std/container.o
std/cstream.o std/csv.o std/demangle.o std/encoding.o
std/format.o std/functional.o std/getopt.o std/json.o std/math.o
std/mathspecial.o std/metastrings.o std/mmfile.o std/numeric.o
std/outbuffer.o std/range.o std/signals.o std/stdint.o
std/stdiobase.o std/stream.o std/string.o std/syserror.o
std/system.o std/traits.o std/typecons.o std/typelist.o
std/typetuple.o std/uri.o std/variant.o std/zip.o std/zlib.o
libunit-threaded.a -ofphobos_ut
I got libunit-threaded.a by running "dub build" in the root
directory of unit-threaded.
I might just implement a random order option now. Hmm.
Atila
