On Tuesday, September 5, 2017 at 12:41:04 PM UTC+3, Jay McCarthy wrote:
> Is the benchmarking client core the same core as the server core?
> Could that help explain why single threaded performance is best?

The not-quite-yes-or-no answer is that they were limited to separate virtual 
cores inside a VirtualBox VM. When a VirtualBox VM has N virtual cores on a 
physical CPU with M cores, it can use roughly up to N/M of the CPU's resources. 
In that benchmark the applications, which had access to two virtual cores, had 
to themselves 50% of the four-core physical CPU, and the load generator had 25% 
with one virtual core.

I've run three variants of the benchmark to see if running in a VM had a 
noticeable effect on single-threaded vs. multi-threaded performance and to 
address your question about whether "many" underperformed because of a virtual 

First, I got rid of the VM. Both the applications and the load generator ran in 
containers on the same machine, but not in a VM. This meant they were limited 
to different physical cores. The results were similar to those in a VM. The 
numbers were lower overall due to the slightly weaker hardware.

results/caddy.txt:Requests per second:    2758.06 [#/sec] (mean)
results/compojure.txt:Requests per second:    2670.11 [#/sec] (mean)
results/custom-many-places.txt:Requests per second:    4326.27 [#/sec] (mean)
results/custom-many.txt:Requests per second:    4655.04 [#/sec] (mean)
results/custom-places.txt:Requests per second:    4584.75 [#/sec] (mean)
results/custom-single.txt:Requests per second:    5191.93 [#/sec] (mean)
results/flask.txt:Requests per second:    1111.25 [#/sec] (mean)
results/guile.txt:Requests per second:    1933.10 [#/sec] (mean)
results/plug.txt:Requests per second:    3346.99 [#/sec] (mean)
results/scgi.txt:Requests per second:    2092.03 [#/sec] (mean)
results/sinatra.txt:Requests per second:    293.60 [#/sec] (mean)
results/stateful.txt:Requests per second:    532.61 [#/sec] (mean)
results/stateless.txt:Requests per second:    625.02 [#/sec] (mean)

Second, I ran the benchmark over a gigabit local network. Yesterday I pushed a 
script for this (`remote-benchmark.exp`) to the repository. The applications 
ran on one machine (in a Docker container with access to two virtual cores). 
The load generator ran on another (my laptop).

remote-results/caddy.txt-Requests per second:    3119.23 [#/sec] (mean)
remote-results/compojure.txt-Requests per second:    4009.71 [#/sec] (mean)
remote-results/custom-many-places.txt-Requests per second:    4409.48 [#/sec] 
remote-results/custom-many.txt-Requests per second:    5499.20 [#/sec] (mean)
remote-results/custom-places.txt-Requests per second:    5072.63 [#/sec] (mean)
remote-results/custom-single.txt-Requests per second:    6246.09 [#/sec] (mean)
remote-results/flask.txt-Requests per second:    1106.43 [#/sec] (mean)
remote-results/guile.txt-Requests per second:    2062.53 [#/sec] (mean)
remote-results/plug.txt-Requests per second:    4034.74 [#/sec] (mean)
remote-results/scgi.txt-Requests per second:    2046.91 [#/sec] (mean)
remote-results/sinatra.txt-Requests per second:    288.52 [#/sec] (mean)
remote-results/stateful.txt-Requests per second:    542.27 [#/sec] (mean)
remote-results/stateless.txt-Requests per second:    614.18 [#/sec] (mean)

In both cases the ordering is still "single" > "many" > "places" > 
"many-places". Though "many" and "places" are pretty close in the first case, 
"many" consistently comes out ahead if you retest.

Last, I ran the benchmark over the Internet with two machines about 1.89×10^-10 
light years apart. The applications ran on a very humble VPS. Due to its 
humbleness I had to reduce the number of concurrent connections to 25. 
"places", "many-places", and racket-scgi ran out of memory with as few as 10 
concurrent connections (racket-scgi seemingly due to nginx), so I decided to 
exclude them rather than reduce the number of connections further.

> env CONNECTIONS=25 ./remote-benchmark.exp vps
remote-results/caddy.txt:Requests per second:    231.37 [#/sec] (mean)
remote-results/compojure.txt:Requests per second:    242.41 [#/sec] (mean)
remote-results/custom-many.txt:Requests per second:    250.35 [#/sec] (mean)
remote-results/custom-single.txt:Requests per second:    255.21 [#/sec] (mean)
remote-results/flask.txt:Requests per second:    235.26 [#/sec] (mean)
remote-results/guile.txt:Requests per second:    242.38 [#/sec] (mean)
remote-results/plug.txt:Requests per second:    244.98 [#/sec] (mean)
remote-results/sinatra.txt:Requests per second:    239.78 [#/sec] (mean)
remote-results/stateful.txt:Requests per second:    239.60 [#/sec] (mean)
remote-results/stateless.txt:Requests per second:    238.71 [#/sec] (mean)

You received this message because you are subscribed to the Google Groups 
"Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to racket-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to