This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "snap-website".
The branch, master has been updated
via 8e0b5fd325dcb98f3ac8d916c7d0edf47d1b2cef (commit)
from e1327700b300eb75e93121117abc5aded9c07706 (commit)
Summary of changes:
.../content/2010/05/23/snap-0.1-benchmarks.md | 6 +
blogdata/content/2010/11/17/snap-0.3-benchmarks.md | 151 ++++++++++++++++++++
static/media/img/file-bench-20101117.png | Bin 0 -> 24629 bytes
static/media/img/pong-bench-20101117.png | Bin 0 -> 25808 bytes
static/media/img/snap-0.1-0.3-bench.png | Bin 0 -> 18293 bytes
5 files changed, 157 insertions(+), 0 deletions(-)
copy templates/benchmarks.md =>
blogdata/content/2010/05/23/snap-0.1-benchmarks.md (93%)
create mode 100644 blogdata/content/2010/11/17/snap-0.3-benchmarks.md
create mode 100644 static/media/img/file-bench-20101117.png
create mode 100644 static/media/img/pong-bench-20101117.png
create mode 100644 static/media/img/snap-0.1-0.3-bench.png
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 8e0b5fd325dcb98f3ac8d916c7d0edf47d1b2cef
Author: Mighty Byte <[email protected]>
Date: Wed Nov 17 18:47:11 2010 -0500
Blog post with new benchmarks.
diff --git a/blogdata/content/2010/05/23/snap-0.1-benchmarks.md
b/blogdata/content/2010/05/23/snap-0.1-benchmarks.md
new file mode 100644
index 0000000..ab5c9e2
--- /dev/null
+++ b/blogdata/content/2010/05/23/snap-0.1-benchmarks.md
@@ -0,0 +1,112 @@
+| title: Snap 0.1 Benchmarks
+| author: Doung Beardsley <[email protected]>
+| published: 2010-05-23T10:15:00-0400
+| updated: 2010-05-23T10:15:00-0400
+| summary: Snap benchmarks and comparison with other web frameworks.
+
+Web Framework Benchmark Description
+===================================
+
+To compare the performance of Snap to that of other web servers/frameworks, we
+ran some simple benchmarks using httperf: pong and file. In the pong
+benchmark, servers sent a 4-byte response of "PONG". The file benchmark sent a
+50KB PNG image file as its response. Our results and nodes are described
+below.
+
+Most of the benchmarks were run with the following httperf command line
+options:
+
+~~~~~~~~~~~~~~~~~~
+httperf --hog --num-conns 1000 --num-calls 1000 --burst-length 20 --port 3000
--rate 1000 --uri=/pong
+~~~~~~~~~~~~~~~~~~
+
+All benchmarks were run on a dual quad-core Xeon E5345 2.33 GHz workstation
+with 8 gigs of RAM. In all these tests, we compiled Snap with "-threaded" and
+ran it with "+RTS -A4M -N4 -qg0 -qb -g1". Using more than four threads caused
+the performance to decrease. This is certainly a good area for future
+improvement in Snap.
+
+Disclaimer: We are not experts at optimizing these web frameworks. It
+is almost certain that the results for other frameworks are unfairly
+low. If you're interested in the details, the code we used is in a
+[github repository](http://github.com/snapframework/snap-benchmarks).
+We're always willing to update our benchmarks with optimizations that
+don't take an inordinate amount of effort to set up.
+
+### Pong Benchmark
+
+The pong benchmark is quite simple, but there was a large amount of variation
+in performance. Here is a chart of responses per second (higher numbers
+are better):
+
+
+
+#### The Numbers
+
+<table>
+ <tr>
+ <td>RoR</td>
+ <td>Grails</td>
+ <td>Happstack</td>
+ <td>Apache+PHP</td>
+ <td>Snap</td>
+ </tr>
+ <tr>
+ <td>258</td>
+ <td>796</td>
+ <td>8578</td>
+ <td>8843</td>
+ <td>23997</td>
+ </tr>
+</table>
+<br />
+
+It should be noted that the Happstack benchmark is not multithreaded. We
+compiled Happstack with -threaded and when run with "+RTS -N4", the test never
+finished. When we aborted httperf, it reported a rate of 12.6
+requests/second. We also tried running it with with the same "+RTS -A4M -N4
+-qg0 -qb -g1" runtime options that we used with Snap. This caused a stack
+overflow.
+
+### File Benchmark
+
+The file benchmark tests the ability to serve static content from disk. The
+results here were a little closer together than the pong results, but we still
+see the same general pattern.
+
+
+
+#### The Numbers
+
+<table>
+ <tr>
+ <td>RoR</td>
+ <td>Grails</td>
+ <td>Happstack</td>
+ <td>Apache</td>
+ <td>Snap</td>
+ </tr>
+ <tr>
+ <td>360</td>
+ <td>1769</td>
+ <td>2455</td>
+ <td>4126</td>
+ <td>7206</td>
+ </tr>
+</table>
+<br />
+
+In this test, we had more problems because Happstack does not support
+pipelining. So we switched to the following httperf line for Happstack:
+
+~~~~~~~~~~~~~~~~~~
+httperf --hog --num-conns 100000 --num-calls 1 --burst-length 1 --port 8080
--rate 10000 --uri=/FiringGeometry.png
+~~~~~~~~~~~~~~~~~~
+
+This means that Happstack's results in this test are not really an
+apples-to-apples comparison with the other frameworks, but they're still
+somewhat meaningful since they reflect a deficiency in the server.
+
+See the [raw test output](/bench-raw-results.txt) for more detailed
+httperf output.
+
diff --git a/blogdata/content/2010/11/17/snap-0.3-benchmarks.md
b/blogdata/content/2010/11/17/snap-0.3-benchmarks.md
new file mode 100644
index 0000000..382f305
--- /dev/null
+++ b/blogdata/content/2010/11/17/snap-0.3-benchmarks.md
@@ -0,0 +1,151 @@
+| title: Snap 0.3 Benchmarks with GHC 7.0.1
+| author: Doug Beardsley <[email protected]>
+| published: 2010-11-17T21:48:00+0100
+| updated: 2010-11-17T21:48:00+0100
+| summary: New benchmarking results for Snap 0.3 show a ~50% improvement since
Snap 0.1 and a 6% improvement due to GHC 7.0.1.
+
+The recent release of GHC 7.0.1 features some significant performance
+improvements, and because the Snap benchmarks have not been updated since we
+released six months ago we thought it would be a good time to do so. We kept
+as many things constant as possible, including benchmarks (pong and file),
+hardware, OS (although the kernel version is probably different), and httperf
+options.
+
+### Pong Benchmark
+
+First let's look at the results for the pong benchmark. The units are
+replies/second, higher numbers are better. We should note that we are
+benchmarking Snap 0.3 which is still in development and scheduled for release
+in January.
+
+
+
+<table>
+ <tr>
+ <td>RoR</td>
+ <td>Apache+PHP</td>
+ <td>Happstack</td>
+ <td>Snap (loging)</td>
+ <td>Node.js</td>
+ <td>Snap (no logging)</td>
+ </tr>
+ <tr>
+ <td>258</td>
+ <td>8843</td>
+ <td>16299</td>
+ <td>17800</td>
+ <td>22733</td>
+ <td>35316</td>
+ </tr>
+</table>
+<br />
+
+If you're comparing this with our [previous
+results](/blog/2010/05/23/snap-0.1-benchmarks) you will notice that we left
+out Grails. We discovered that our previous results for Grails may have been
+too low because the JVM had not been given time to warm up. The problem is
+that after the JVM warms up for some reason httperf isn't able to get any
+samples from which to generate a replies/sec measurement, so it outputs 0.0
+replies/sec. There are also 1000 connreset errors, so we decided the Grails
+numbers were not reliable enough to use.
+
+Happstack has improved since the last benchmark, but it has a problem where
+performance decreases under heavy loads. We discovered that using an httperf
+rate of 34 almost doubles happstack's performance on the pong benchmark
+compared to the rate of 1000 used to test all the other frameworks. This
+brings it to within about 10% of Snap's numbers. But on these benchmarks
+we're more interested in a server's performance under heavy load than it's
+peak throughput in optimal conditions, so our graphs show the numbers for a
+rate of 1000.
+
+One source of variation among applications is whether they do access logging
+or not. Ruby on Rails, Node.js, and Happstack do not do any access logging.
+Our previous Snap benchmarks also did not have access logging turned on. But
+Apache does write access logs, so we decided to add a second entry for Snap
+with access logging to facilitate a more accurate comparison.
+
+### File Benchmark
+
+Here are the results for the file benchmark. Again, higher numbers are
+better.
+
+
+
+<table>
+ <tr>
+ <td>RoR</td>
+ <td>Apache+PHP</td>
+ <td>Node.js</td>
+ <td>Snap (loging)</td>
+ <td>Happstack</td>
+ <td>Snap (no logging)</td>
+ </tr>
+ <tr>
+ <td>360</td>
+ <td>4126</td>
+ <td>4540</td>
+ <td>6152</td>
+ <td>7554</td>
+ <td>10832</td>
+ </tr>
+</table>
+<br />
+
+The relative performance of the frameworks stays pretty much the same, except
+that Happstack is good at serving files, while Node.js is slower. We didn't
+try to figure out what rate would optimize Happstack's throughput on the file
+benchmark, but it seems reasonable to assume that it won't make as big of a
+difference here as it did with pong.
+
+After we published the last results comments
+[here](http://news.ycombinator.com/item?id=1380405) and
+[here](http://news.ycombinator.com/item?id=1369852) suggested that it is
+unfair to benchmark Rails with the Webrick server and that we should use
+Thin/Mongrel/nginx+passenger instead. Benchmarking best of breed against best
+of breed is certainly a useful thing to do, but it is also useful to benchmark
+a similar level of effort. Webrick is the out-of-the-box server for Rails, so
+it seems fair to compare this with Snap's out-of-the-box solution. I was able
+to benchmark Thin and it performed worse than webrick. Another person got
+similar results and also found that Webrick beat Mongrel as well. We did not
+benchmark nginx+passenger because that requires significant configuration and
+doesn't qualify as an out-of-the-box solution. We also tried to benchmark the
+Play web framework but it generated an InternalError exception and stopped
+responding to requests.
+
+### Snap Comparison
+
+So what has happened to Snap since our last benchmarks in May? Here is a
+chart comparing the performance of Snap 0.1 and 0.3.
+
+
+
+On the pong benchmark we went from 23,997 to 35,316 (replies/sec), a 47%
+improvement. The file benchmark improved from 7,206 to 10,832, a 50%
+improvement. We were curious to see how much of this came from improvements
+to Snap and how much was from better GHC code generation. When compiled with
+GHC 6.12.3 the pong benchmark gets 33196 replies/sec. So upgrading to GHC
+7.0.1 gave us a drop-in 6% performance improvement.
+
+This time all of our Snap runs used the runtime options "+RTS -A4M -N4". The
+new version of GHC eliminated the need for the "-qg0 -qb -g1" that we used
+before. Also, we were really looking forward to the new IO manager that is in
+GHC 7.0.1, and hoping that it would eliminate the need for our libev backend.
+Unfortunately our libev backend still performs significantly better, so we
+will continue to ship it.
+
+For those who are interested, here are the versions of the software we used:
+
+* Ruby on Rails 2.3.5
+* Grails 1.2.2
+* Apache 2.2.16
+* PHP 5.2.14
+* Node.js 0.2.4
+* Snap 0.3
+* Play 1.1
+
+Full results are
+[here](https://github.com/snapframework/snap-benchmarks/blob/51b1c5b24c09ced61e937d7d5b1d88f8e513663b/results.txt).
+Or visit the [snap-benchmarks github
+repository](https://github.com/snapframework/snap-benchmarks) if you want the
+code.
+
diff --git a/static/media/img/file-bench-20101117.png
b/static/media/img/file-bench-20101117.png
new file mode 100644
index 0000000..ba34510
Binary files /dev/null and b/static/media/img/file-bench-20101117.png differ
diff --git a/static/media/img/pong-bench-20101117.png
b/static/media/img/pong-bench-20101117.png
new file mode 100644
index 0000000..20fde7a
Binary files /dev/null and b/static/media/img/pong-bench-20101117.png differ
diff --git a/static/media/img/snap-0.1-0.3-bench.png
b/static/media/img/snap-0.1-0.3-bench.png
new file mode 100644
index 0000000..8cd880b
Binary files /dev/null and b/static/media/img/snap-0.1-0.3-bench.png differ
-----------------------------------------------------------------------
hooks/post-receive
--
snap-website
_______________________________________________
Snap mailing list
[email protected]
http://mailman-mail5.webfaction.com/listinfo/snap
