On Wednesday, 4 December 2013 at 12:49:13 UTC, Atila Neves wrote:
It all started here:
http://forum.dlang.org/thread/[email protected]
Not wanting to be outdone, my buddy Jeff (the Go guy) went and
wrote a new benchmark. My implementation didn't do as well on
that one so I went back to the code and optimised it. Also, our
other buddy Patrick from work decided to get in on this and use
it as an excuse to learn Erlang. So now 4 different
implementations in 4 different languages are duking it out.
There are two benchmarks, both written in Go. The first is
loadtest and it measures throughput. I varied the number of
client connections to the server and measured how many messages
were sent per second.
The second benchmark is pingtest and it measures latency. The
quantity I varied here was the number of wildcard subscribers
(it'd take a while to explain what they are to anyone who isn't
familiar with MQTT). It has a bunch of client connections talk
to each other, and instead of trying to send as many messages
as computerly possible they wait for the acknowledgement from
the partner connection. The results are as follows (I tried
making charts but LibreOffice wasn't helping and then I got
bored), with the numbers being thousands of messages per second:
loadtest
Connections: 100 500 750 1k
D + vibe.d: 121.7 +/- 1.5 166.9 +/- 1.5 171.1 +/- 3.3
167.9 +/- 1.3
C (mosquitto): 106.1 +/- 0.8 122.4 +/- 0.4 95.2 +/- 1.3
74.7 +/- 0.4
Erlang: 104.1 +/- 2.2 124.2 +/- 5.9 117.6 +/- 4.6
117.7 +/- 3.2
Go: 90.9 +/- 11 100.1 +/- 0.1 99.3 +/- 0.2
98.8 +/- 0.3
pingtest
wsubs: 20 200 400
D + vibe.d: 50.9 +/- 0.3 38.3 +/- 0.2 20.1 +/- 0.1
C (mosquitto): 65.4 +/- 4.4 45.2 +/- 0.2 20.0 +/- 0.0
Erlang: 49.1 +/- 0.8 30.9 +/- 0.3 15.6 +/- 0.1
Go: 45.2 +/- 0.2 27.5 +/- 0.1 16.0 +/- 0.1
So, D was faster than the other contenders by far in
throughput, 2nd place losing to the C implementation on
latency. I'm still not sure why that is. Profiling in this case
is tricky. I'm pretty sure the profiler is still ticking away
when a fiber yields - the top function is the one that reads
from the network, which I can't do much about.
GC was pretty much a non-issue, which I'm both surprised at and
happy about. I allocated as much memory as I wanted at first,
and by the time I tried to optmise memory usage by allocating
less all I managed to eke out in performance terms was a paltry
extra 8% or so.
All in all though, so much for his "nothing is going to be
faster than Go because this is what it's good at". If he'd
measured it against mosquitto first and realised it was already
losing to C I probably wouldn't have written an MQTT broker in
D, so I guess that was a good thing. :)
Atila
P.S. vibe.d is awesome, although I wish it'd compile with ldc
or gdc
Nice benchmark! :)
So, a DMD-compiled D program vs GCC-compiled (?) C program
fighting for latency? That might be a reason. Interesting we
still win with throughput.
The variance/standard deviation of pingtest-C-20 with 4.4 stands
out. What happens there?
