Hi Ludovic, I’m usually highly sceptical about any network benchmarks over localhost. I’d suggest you to rerun your benchmarks using two different machines connected over a gigabit network. Please make sure that your server process loads the CPU 100%, and ideally you should use several processes for clients.
Yury > On Feb 14, 2016, at 1:31 PM, Ludovic Gasc <gml...@gmail.com> wrote: > > Hi, > > This is some values I've found during the update attempt of the AsyncIO test > suite for FrameworkBenchmarks: > https://www.techempower.com/benchmarks/#section=intro > > Long story short: After Python 3.5 update, dependencies update and > async/await syntax usage, on my test setup, I loose around 15% of efficiency. > You can see the difference between old test setup and now: > https://github.com/Eyepea/FrameworkBenchmarks/commit/a40c82ed720a53e04ebdafd7db90302ca67b5226 > > If somebody has any suggestion to find my mistake or how to find the new > bottleneck, be my guest. > > However, even if the absolute values aren't exact, the relative values > between each test should give us an idea of the evolution. > > For each test, I've launched 5 times, and I've taken the best one. > > The values on this setup before to change something in the test setup (python > 3.4.2, aiohttp 0.16.3, aiopg 0.7.0): > > $ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20 > Running 1m test @ http://127.0.0.1:8080/queries?queries=20 > 8 threads and 256 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 216.97ms 105.14ms 1.14s 75.31% > Req/Sec 148.88 17.97 206.00 69.38% > 71647 requests in 1.00m, 54.98MB read > Requests/sec: 1194.19 > Transfer/sec: 0.92MB > > Now, I upgrade only Python 3.4 to Python 3.5 (python 3.5.1, aiohttp 0.16.3, > aiopg 0.7.0): > > $ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20 > Running 1m test @ http://127.0.0.1:8080/queries?queries=20 > 8 threads and 256 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 237.25ms 118.33ms 1.17s 74.74% > Req/Sec 134.77 13.24 171.00 66.98% > 65051 requests in 1.00m, 49.92MB read > Requests/sec: 1084.09 > Transfer/sec: 851.93KB > > And now, I update aiohttp and aiopg (python 3.5.1, aiohttp 0.21.1, aiopg > 0.9.2) > > $ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20 > Running 1m test @ http://127.0.0.1:8080/queries?queries=20 > 8 threads and 256 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 254.25ms 181.80ms 1.56s 75.51% > Req/Sec 129.43 24.09 204.00 68.37% > 62122 requests in 1.00m, 46.25MB read > Requests/sec: 1035.44 > Transfer/sec: 789.32KB > > And now, I use async/await syntax: > > $ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20 > Running 1m test @ http://127.0.0.1:8080/queries?queries=20 > 8 threads and 256 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 259.27ms 121.86ms 842.74ms 70.02% > Req/Sec 126.29 17.70 207.00 75.82% > 60740 requests in 1.00m, 45.22MB read > Requests/sec: 1014.01 > Transfer/sec: 772.99KB > > To be sure it isn't a problem with my hardware (CPU too hot after a while...) > at the end, I've relaunched the first test without any updates: > > $ wrk -t8 -c256 -d1m http://127.0.0.1:8080/queries?queries=20 > Running 1m test @ http://127.0.0.1:8080/queries?queries=20 > 8 threads and 256 connections > Thread Stats Avg Stdev Max +/- Stdev > Latency 220.24ms 98.73ms 798.70ms 72.78% > Req/Sec 147.85 19.10 215.00 70.30% > 70967 requests in 1.00m, 54.46MB read > Requests/sec: 1183.54 > Transfer/sec: 0.91MB > > Thanks for your remarks. > > On Monday, December 28, 2015 at 2:54:06 PM UTC+1, Ludovic Gasc wrote: > Hi everybody, > > For now, I can't contribute like I want because I'm handling a lot of > personal and professional changes. > > However, I continue to keep an eye. > FrameworkBenchmarks Round 11 results are available: > https://www.techempower.com/benchmarks/#section=data-r11&hw=peak&test=fortune&l=1kw > > If you don't know what is FrameworkBenchmarks: > https://www.techempower.com/benchmarks/#section=intro > > No big surprises, AsyncIO+aiohttp continues to have good results, except for > plain text, need to dig where is the bottleneck. > > For the next round, I've a small todo list: > 1. Add tests with MySQL, because MySQL is more optimized by techempower than > PostgreSQL setup and all other Python frameworks with better results use > MySQL. > 2. Upgrade aiohttp, because the latest version has several performance > improvements: https://github.com/KeepSafe/aiohttp/releases/tag/v0.20.0 > 3. Use Python 3.5 with async/await. Apparently, I could hope some performance > improvements. > 4. If I have the time, take a try with MicroPython. > > If somebody wants to help me or suggests other improvements, be my guest. > > Happy holidays.
