Re: Echo server in Lua

[Willy Tarreau]

On Thu, Nov 19, 2015 at 07:00:32AM +, Thrawn wrote:
> Hmm...I haven't tuned anything (this is just my workstation, not a server).
> /proc/sys/net/core/somaxconn 128
> /proc/sys/net/ipv4/tcp_max_backlog doesn't exist
> 
>  /proc/sys/net/ipv4/tcp_max_syn_backlog512

Yep sorry it's this one.

> I guess those are pretty low for trying to thrash an echo server...

Yes they're low even for 16k/s.

> any recommendations?

Start by multiplying both of them by 4. If nothing improves it doesn't
come from there.

Willy

Re: Echo server in Lua

[Thrawn]

Nope :|. I've increased somaxconn to 512 and tcp_max_syn_backlog to 2048, but 
with one process, 1M requests, concurrency 500, I still get a bunch of 
timeouts, and the longest transaction still takes most of the test (eg 32 
seconds out of a test that takes 35).

Any other obvious possibilities?

On Friday, 20 November 2015, 4:12, Willy Tarreau  wrote:



On Thu, Nov 19, 2015 at 07:00:32AM +, Thrawn wrote:
> Hmm...I haven't tuned anything (this is just my workstation, not a server).
> /proc/sys/net/core/somaxconn 128
> /proc/sys/net/ipv4/tcp_max_backlog doesn't exist
> 
>  /proc/sys/net/ipv4/tcp_max_syn_backlog512

Yep sorry it's this one.

> I guess those are pretty low for trying to thrash an echo server...

Yes they're low even for 16k/s.

> any recommendations?

Start by multiplying both of them by 4. If nothing improves it doesn't
come from there.


Willy

Re: Echo server in Lua

[Willy Tarreau]

hi,

just chiming in regarding this specific point :

On Tue, Nov 10, 2015 at 10:46:32PM +, Thrawn wrote:
> I ran ab with concurrency 1000 and a total of 3 requests, against each
> server, 5 times, plus one run each with 15 requests (sum of the previous
> 5 tests).For Apache+PHP, this typically resulted in 5-15ms response time for
> 99% of requests, with the remaining few either taking tens of seconds or
> eventually disconnecting with an error.For HAProxy+Lua, 99% response times
> were 1ms, or sometimes 2ms, with the last few taking about 200ms. So, HAProxy
> worked much better (of course).
> However, on the larger run (150k), HAProxy too had a small percentage of
> disconnections (apr_socket_recv: Connection reset by peer). I've been able to
> reproduce this with moderate consistency whenever I push it beyond about
> 35000 total requests. It's still a better error rate than PHP, but I'd like
> to understand why the errors are occurring. For all I know, it's a problem
> with ab.

I've been used to get errors with ab around 1000 concurrent conns because it
doesn't perform a setrlimit(RLIMIT_FD_CUR) so it's limited to the default
1024. I seem to remember another limitation to 1024 somewhere in the code
though I don't remember which one. There are some moments where the previous
FDs are possibly not completely released before trying to allocate a new one
and it experiences failures slightly below 1024. Also, past a few hundred
conns, it can sometimes show uneven time distribution for various conns, I
seem to remember that since it doesn't use an event cache, connections with
the lowest file descriptor number are always served more often, or something
similar.

Thus I tend to use it for anything below 500 and start to look at it with
a suspicious eye above, or switch to something else. But I too like to use
it because it's simple and convenient. You just need to take its limits into
consideration.

You may want to retry with httperf, httpress, wrk or I-don't-know-what since
there are a myriad. httpress is easy to build and to use and looks very
similar to ab. It can sometimes bug when you start multiple threads where
performance can go down and the load appear uneven between the threads but
I didn't observe this often enough to care to try to debug it. And since
it's already fast enough with a single thread, I really don't care :-)

Cheers,
Willy

Re: Echo server in Lua

[Thrawn]

OK, I've decided to go a bit easier on the stress tools ;), and limited 
concurrency to 500, using siege. That's resulted in some useful data 
(attached). 
HAProxy HTTP endpoint, with 1 process, handled up to 500K total requests 
without breaking a sweat, in just under 30 seconds, achieving an actual 
concurrency of 160-170 (I'm guessing the requests finished too fast to go any 
higher). For some reason, the longest request took almost all that time; maybe 
that's again a limitation of the stress tool? Higher request volumes introduced 
a small percentage of timeouts.
Increasing to two HAProxy processes made a marked difference. At 500K requests, 
total time halved to approximately 10-15 seconds. At higher volumes, timeouts 
were much rarer; I was able to run 5 tests at 1.5M requests each with no 
timeouts at all. Just for fun, I increased to 750 concurrency (2.25M requests), 
with no trouble (the same test with 1 proc resulted in 57 timeouts and about 
30% longer run time).
PHP surprised me, actually; its performance characteristics were somewhat 
better than HAProxy with 1 process (though I don't know how the memory usage 
compared). However, compared to HAProxy with two processes, it had a somewhat 
higher error rate and noticeably slower response time.
Looks like the conclusion is: use HAProxy, but tune nbproc wisely :)

On Thursday, 19 November 2015, 9:49, Willy Tarreau  wrote:
 

 On Wed, Nov 18, 2015 at 10:53:18PM +, Thrawn wrote:
> Thanks, Willy.
> I've likewise noticed that ab didn't seem capable of going beyond about 1024 
> concurrency. I've also tried siege, since it's available in the Ubuntu 
> repositories and has a decent reputation, but it keeps hitting errors, too:
> [error] descriptor table full sock.c:119: Too many open files
> or after setting ulimit -n 1,
> *** buffer overflow detected ***: siege terminated

"great" ...

> That's with --reps=30 and --concurrent=1000.
> So it may be that HAProxy is actually more capable than the stress tool(s)...
> I'll see whether I can draw some meaningful comparisons between Lua and PHP 
> at lower concurrency.

Indeed sometimes that's true. Just yesterday I had to set up a second
machine to host the server during a test because I wasn't able to have
both the client and the server on the same machine to hammer haproxy.

Willy



 Date & Time,  Trans,  Elap Time,  Data Trans,  Resp Time,  Trans Rate,  Throughput,  Concurrent,OKAY,   Failed
--concurrent=500
2015-11-19 12:22:54, 50,  28.98,   4,   0.01,17253.28,0.14,  164.61,  50,   0
2015-11-19 12:23:27, 50,  27.32,   4,   0.01,18301.61,0.15,  170.11,  50,   0
2015-11-19 12:29:44, 50,  27.39,   4,   0.01,18254.84,0.15,  159.64,  50,   0
2015-11-19 12:46:38, 499970,  32.20,   4,   0.01,15527.02,0.12,  129.63,  499970,  30
2015-11-19 12:47:28, 50,  14.40,   4,   0.01,34722.22,0.28,  329.47,  50,   0
2015-11-19 12:24:52, 749985,  31.74,   6,   0.01,23629.02,0.19,  209.15,  749985,  15
2015-11-19 12:25:39, 749955,  32.81,   6,   0.01,22857.51,0.18,  174.32,  749955,  45
2015-11-19 12:30:27, 75,  29.08,   6,   0.01,25790.92,0.21,  253.69,  75,   0
2015-11-19 12:44:32, 749989,  31.56,   6,   0.01,23763.91,0.19,  194.78,  749989,  11
2015-11-19 12:45:11, 749965,  32.46,   6,   0.01,23104.28,0.18,  191.89,  749965,  35
2015-11-19 12:20:41, 75,  32.20,   8,   0.01,31055.12,0.25,  276.73,  75,  25
2015-11-19 12:21:26, 94,  33.50,   8,   0.01,29850.57,0.24,  285.38,  94,   6
2015-11-19 12:22:18, 69,  33.38,   8,   0.01,29957.13,0.24,  260.23,  69,  31
2015-11-19 12:41:46, 47,  33.63,   8,   0.01,29733.78,0.24,  242.28,  47,  53
2015-11-19 12:42:31, 99,  31.27,   8,   0.01,31979.50,0.26,  317.10,  99,   1
2015-11-19 13:31:04,1499983,  38.05,  12,   0.01,39421.37,0.32,  390.14, 1499983,  17
2015-11-19 13:31:56,1499989,  36.71,  12,   0.01,40860.50,0.33,  407.91, 1499989,  11
2015-11-19 13:32:52,146,  42.09,  12,   0.01,35637.82,0.29,  373.48, 146,   4
2015-11-19 13:34:17,1499940,  48.67,  12,   0.01,30818.58,0.25,  270.53, 1499940,  60
2015-11-19 13:35:53,1499928,  39.12,  12,   0.01,38341.72,0.31,  312.84, 1499928,  72

--concurrent=750
2015-11-19 12:56:51,2249943,  65.67, 

Re: Echo server in Lua

[Willy Tarreau]

Hi,

On Thu, Nov 19, 2015 at 03:37:33AM +, Thrawn wrote:
> OK, I've decided to go a bit easier on the stress tools ;), and limited
> concurrency to 500, using siege. That's resulted in some useful data
> (attached). 
> HAProxy HTTP endpoint, with 1 process, handled up to 500K total requests
> without breaking a sweat, in just under 30 seconds, achieving an actual
> concurrency of 160-170 (I'm guessing the requests finished too fast to go any
> higher). For some reason, the longest request took almost all that time;
> maybe that's again a limitation of the stress tool? Higher request volumes
> introduced a small percentage of timeouts.

Maybe you're hitting the accept backlog here.

> Increasing to two HAProxy processes made a marked difference. At 500K
> requests, total time halved to approximately 10-15 seconds. At higher
> volumes, timeouts were much rarer; I was able to run 5 tests at 1.5M requests
> each with no timeouts at all. Just for fun, I increased to 750 concurrency
> (2.25M requests), with no trouble (the same test with 1 proc resulted in 57
> timeouts and about 30% longer run time).

That also tends to confirm a backlog issue. What's your sys.net.core.somaxconn
and sys.net.ipv4.tcp_max_backlog ?

> PHP surprised me, actually; its performance characteristics were somewhat
> better than HAProxy with 1 process (though I don't know how the memory usage
> compared). However, compared to HAProxy with two processes, it had a somewhat
> higher error rate and noticeably slower response time.
> Looks like the conclusion is: use HAProxy, but tune nbproc wisely :)

Normally at 500k in 30s (16k/s) you clearly don't need to use nbproc, so
I'm pretty sure that you're hitting some network settings limits and your
system needs to be tuned now.

Willy

Re: Echo server in Lua

[Thrawn]

Hmm...I haven't tuned anything (this is just my workstation, not a server).
/proc/sys/net/core/somaxconn 128
/proc/sys/net/ipv4/tcp_max_backlog doesn't exist

 /proc/sys/net/ipv4/tcp_max_syn_backlog512

I guess those are pretty low for trying to thrash an echo server...any 
recommendations?

On Thursday, 19 November 2015, 16:40, Willy Tarreau  wrote:
 

 Hi,

On Thu, Nov 19, 2015 at 03:37:33AM +, Thrawn wrote:
> OK, I've decided to go a bit easier on the stress tools ;), and limited
> concurrency to 500, using siege. That's resulted in some useful data
> (attached). 
> HAProxy HTTP endpoint, with 1 process, handled up to 500K total requests
> without breaking a sweat, in just under 30 seconds, achieving an actual
> concurrency of 160-170 (I'm guessing the requests finished too fast to go any
> higher). For some reason, the longest request took almost all that time;
> maybe that's again a limitation of the stress tool? Higher request volumes
> introduced a small percentage of timeouts.

Maybe you're hitting the accept backlog here.

> Increasing to two HAProxy processes made a marked difference. At 500K
> requests, total time halved to approximately 10-15 seconds. At higher
> volumes, timeouts were much rarer; I was able to run 5 tests at 1.5M requests
> each with no timeouts at all. Just for fun, I increased to 750 concurrency
> (2.25M requests), with no trouble (the same test with 1 proc resulted in 57
> timeouts and about 30% longer run time).

That also tends to confirm a backlog issue. What's your sys.net.core.somaxconn
and sys.net.ipv4.tcp_max_backlog ?

> PHP surprised me, actually; its performance characteristics were somewhat
> better than HAProxy with 1 process (though I don't know how the memory usage
> compared). However, compared to HAProxy with two processes, it had a somewhat
> higher error rate and noticeably slower response time.
> Looks like the conclusion is: use HAProxy, but tune nbproc wisely :)

Normally at 500k in 30s (16k/s) you clearly don't need to use nbproc, so
I'm pretty sure that you're hitting some network settings limits and your
system needs to be tuned now.

Willy



   

Re: Echo server in Lua

[Thrawn]

Hi, again.
Actually, when I think more about it, this service needs to support SSL, so a 
Layer 7 implementation is needed.Also, when I put the TCP applet under any 
degree of stress (as little as 100 request with concurrency 10), it hangs? The 
other two implementations (http-req vs http applet) seem to behave very 
similarly to each other, though. 
Increasing the number of processes (nbproc 2) makes a difference. At 
concurrency 1000 and 3 requests, the 90% response time rises to 2-3ms, but 
there is typically no 200ms spike on the last few requests; they take only 
10-15ms.

On Thursday, 12 November 2015, 5:50, PiBa-NL <piba.nl@gmail.com> wrote:
 

  Hi Thrawn,
 
 I tried these configs, and there doesn't seem to be much if any difference. 
The tcp one might even be the slowest in my limited virtualized tests, but only 
my a few milliseconds..
 
 frontend lua-replyip
     bind            192.168.0.120:9010   
     mode            http
     http-request use-service lua.lua-replyip 
 frontend lua-replyip-copy
     bind            192.168.0.120:9011  
     mode            tcp
     tcp-request content use-service lua.lua-replyip-tcp 
 frontend lua-replyip-httpreq
     bind            192.168.0.120:9012   
     mode            http
     http-request lua.lua-replyip-http-req 
 
 core.register_service("lua-replyip", "http", function(applet) 
    local response = applet.f:src()
    applet:set_status(200)
    applet:add_header("Server", "haproxy-lua/echo") 
    applet:add_header("content-length", string.len(response)) 
    applet:add_header("content-type", "text/plain") 
    applet:start_response() 
    applet:send(response) 
 end)
 
 core.register_service("lua-replyip-tcp", "tcp", function(applet)
    local buffer = applet.f:src()
    applet:send("HTTP/1.0 200 OK\r\nServer: haproxy-lua/echo\r\nContent-Type: 
text/html\r\nContent-Length: " .. buffer:len() .. "\r\nConnection: 
close\r\n\r\n" .. buffer)
 end)
 
 core.register_action("lua-replyip-http-req", { "http-req" }, function (txn)
     local buffer = txn.f:src()
     txn.res:send("HTTP/1.0 200 OK\r\nServer: haproxy-lua/echo\r\nContent-Type: 
text/html\r\nContent-Length: " .. buffer:len() .. "\r\nConnection: 
close\r\n\r\n" .. buffer)
     txn:done()
 end)
  

Re: Echo server in Lua

[PiBa-NL]

Hi Thrawn,

I tried these configs, and there doesn't seem to be much if any 
difference. The tcp one might even be the slowest in my limited 
virtualized tests, but only my a few milliseconds..


frontend lua-replyip
bind192.168.0.120:9010
modehttp
http-request use-service lua.lua-replyip
frontend lua-replyip-copy
bind192.168.0.120:9011
modetcp
tcp-request content use-service lua.lua-replyip-tcp
frontend lua-replyip-httpreq
bind192.168.0.120:9012
modehttp
http-request lua.lua-replyip-http-req

core.register_service("lua-replyip", "http", function(applet)
   local response = applet.f:src()
   applet:set_status(200)
   applet:add_header("Server", "haproxy-lua/echo")
   applet:add_header("content-length", string.len(response))
   applet:add_header("content-type", "text/plain")
   applet:start_response()
   applet:send(response)
end)

core.register_service("lua-replyip-tcp", "tcp", function(applet)
   local buffer = applet.f:src()
   applet:send("HTTP/1.0 200 OK\r\nServer: 
haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " .. 
buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)

end)

core.register_action("lua-replyip-http-req", { "http-req" }, function (txn)
local buffer = txn.f:src()
txn.res:send("HTTP/1.0 200 OK\r\nServer: 
haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " .. 
buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)

txn:done()
end)


Op 11-11-2015 om 3:07 schreef Thrawn:
Hmm...I seem to be able to set up something in TCP mode, and it 
returns the expected response via curl, but its performance is awful. 
I must be doing something wrong?


Lua:

core.register_action("tcp-echo", {"tcp-req"}, function (txn)
local buffer = txn.f:src()
txn.res:send("HTTP/1.0 200 OK\r\nServer: 
haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " .. 
buffer:len() .. "\r\nConnection: close\r\n\r\n" ..

missing the appending of 'buffer' in the end on the line above?

txn:done()
end)

I couldn't find a way for a TCP applet to retrieve the client IP 
address; suggestions are welcome.


HAProxy config:

frontend tcp-echo
bind 127.0.2.1:1610
timeout client 1
mode tcp
tcp-request content lua.tcp-echo

Testing this with ab frequently hangs and times out even at tiny loads 
(10 requests with concurrency 3).




On Wednesday, 11 November 2015, 10:19, PiBa-NL <piba.nl@gmail.com> 
wrote:



b.t.w. if sole purpose of the frontend is to echo the ip back to the 
client.
You should probably also check the 'use-service' applet syntax, i dont 
know if that could be faster for your purpose.
Then another thing to check would be if you want to use the tcp or 
http service mode. A TCP service could be almost 1 line of lua code.. 
And i kinda expect to be a bit faster.


http://www.arpalert.org/src/haproxy-lua-api/1.6/index.html#haproxy-lua-hello-world
Instead of sending 'hello world' you could send the client-ip..

Op 10-11-2015 om 23:46 schreef Thrawn:

OK, some explanation seems in order :).

I ran ab with concurrency 1000 and a total of 3 requests, against 
each server, 5 times, plus one run each with 15 requests (sum of 
the previous 5 tests).
For Apache+PHP, this typically resulted in 5-15ms response time for 
99% of requests, with the remaining few either taking tens of seconds 
or eventually disconnecting with an error.
For HAProxy+Lua, 99% response times were 1ms, or sometimes 2ms, with 
the last few taking about 200ms. So, HAProxy worked much better (of 
course).


However, on the larger run (150k), HAProxy too had a small percentage 
of disconnections (apr_socket_recv: Connection reset by peer). I've 
been able to reproduce this with moderate consistency whenever I push 
it beyond about 35000 total requests. It's still a better error rate 
than PHP, but I'd like to understand why the errors are occurring. 
For all I know, it's a problem with ab.


I've also tried a couple of runs with 15 requests but concurrency 
only 100, and neither server had trouble serving that, although 
interestingly, PHP is slightly more consistent: 99% within 4-5ms, 
then about 200ms for the last few, whereas HAProxy returns 99% within 
1-2ms and 1800ms for the last few.


The box is just my workstation, 8 cores and 16GB RAM, running Ubuntu 
15.10, with no special tuning.


Any ideas on why the HAProxy tests showed disconnections or 
occasional slow response times at high loads?




On Wednesday, 11 November 2015, 8:29, Baptiste <bed...@gmail.com> 
<mailto:bed...@gmail.com> wrote:



On Tue, Nov 10, 2015 at 10:46 PM, Thrawn
<shell_layer-git...@yahoo.com.au 
<mailto:shell_layer-git...@yahoo.com.au>> wrote:

> OK,

Re: Echo server in Lua

[Thrawn]

quests)
Transfer rate:  350.99 [Kbytes/sec] received

Connection Times (ms)
  min  mean[+/-sd] median   max
Connect:    0    9  92.7  0    1001
Processing: 1   50 703.8  5   13838
Waiting:    1   50 703.8  5   13837
Total:  3   59 711.9  5   13847

Percentage of the requests served within a certain time (ms)
  50%  5
  66%  5
  75%  5
  80%  5
  90%  5
  95%  6
  98% 14
  99%   1199
 100%  13847 (longest request)
~  ᐅ ab -n 3 -c 1000 -H 'Host: echo-local' http://127.0.0.1/echo.php
This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 29602 requests completed
~  ᐅ ab -n 3 -c 1000 -H 'Host: echo-local' http://127.0.0.1/echo.php
This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 29937 requests completed
~  ᐅ ~  ᐅ ab -n 3 -c 1000 http://127.0.1.1:1610/   
This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.1.1 (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
Completed 3 requests
Finished 3 requests


Server Software:    haproxy-lua/echo
Server Hostname:    127.0.1.1
Server Port:    1610

Document Path:  /
Document Length:    9 bytes

Concurrency Level:  1000
Time taken for tests:   0.711 seconds
Complete requests:  3
Failed requests:    0
Total transferred:  351 bytes
HTML transferred:   27 bytes
Requests per second:    42182.05 [#/sec] (mean)
Time per request:   23.707 [ms] (mean)
Time per request:   0.024 [ms] (mean, across all concurrent requests)
Transfer rate:  4819.63 [Kbytes/sec] received

Connection Times (ms)
  min  mean[+/-sd] median   max
Connect:    0    0   0.3  0   9
Processing: 0    1   1.7  1 208
Waiting:    0    0   1.7  0 208
Total:  0    1   1.7  1 217

Percentage of the requests served within a certain time (ms)
  50%  1
  66%  1
  75%  1
  80%  1
  90%  1
  95%  1
  98%  1
  99%  1
 100%    217 (longest request)
~  ᐅ ab -n 3 -c 1000 http://127.0.1.1:1610/
This is ApacheBench, Version 2.3 <$Revision: 1638069 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.1.1 (be patient)
Completed 3000 requests
Completed 6000 requests
Completed 9000 requests
Completed 12000 requests
Completed 15000 requests
Completed 18000 requests
Completed 21000 requests
Completed 24000 requests
Completed 27000 requests
Completed 3 requests
Finished 3 requests


Server Software:    haproxy-lua/echo
Server Hostname:    127.0.1.1
Server Port:    1610

Document Path:  /
Document Length:    9 bytes

Concurrency Level:  1000
Time taken for tests:   0.752 seconds
Complete requests:  3
Failed requests:    0
Total transferred:  351 bytes
HTML transferred:   27 bytes
Requests per second:    39900.14 [#/sec] (mean)
Time per request:   25.063 [ms] (mean)
Time per request:   0.025 [ms] (mean, across all concurrent requests)
Transfer rate:  4558.90 [Kbytes/sec] received

Connection Times (ms)
  min  mean[+/-sd] median   max
Connect:    0    0   0.6  0  16
Processing: 0    1   1.6  1 199
Waiting:    0    0   1.6  0 199
Total:  1    1   1.8  1 213

Percentage of the requests served within a certain time (ms)
  50%  1
  66%  1
  75%  1
  80%  1
  90%  1
  95%  1
  98%  1
  99%  1
 100%    213 (longest request)
~  ᐅ ab -n 3 -c 1000 http://127.0.1.1:1610/
This is ApacheBench, Version 2.

Re: Echo server in Lua

[Baptiste]

On Tue, Nov 10, 2015 at 10:46 PM, Thrawn
 wrote:
> OK, I've set this up locally, and tested it against PHP using ab.
>
> HAProxy was consistently faster (99% within 1ms, vs 5-15ms for PHP), but at
> request volumes over about 35000, with concurrency 1000, it consistently had
> a small percentage of socket disconnections. PHP had timeouts - or very long
> response times - and disconnections at pretty much any request volume with
> that concurrency, but I'm wondering where the errors stem from, or even if
> it's a limitation of ab.
>
> HAProxy config:
>
> global
> maxconn 4096
> daemon
> nbproc 1
> stats socket localhost:9461 level admin
> chroot /etc/haproxy/jail
> user haproxy
> group haproxy
> lua-load /etc/haproxy/jail/echo.lua
>
> defaults
> log 127.0.0.1 local0
> mode http
> timeout client 6
> timeout server 6
> timeout connect 6
> option forwardfor
> balance roundrobin
> option abortonclose
> maxconn 20
>
> frontend echo
> bind 127.0.1.1:1610
> timeout client 1
> mode http
> http-request lua.echo
>
> Lua:
> core.register_action("echo", { "http-req" }, function (txn)
> local buffer = txn.f:src()
> txn.res:send("HTTP/1.0 200 OK\r\nServer:
> haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " ..
> buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)
> txn:done()
> end)
>

Hi Thrawn

I'm sorry, but I don't understand anything to all your benchmarcks!
If you could at least give an explanation  before running each ab,
this may help.

Furthermore, you don't share anything about your hardware environment
neither the tuning you did on each box.
So it's impossible to help you.

At least, I can say that Lua seems to perform very well :)

Baptiste

Re: Echo server in Lua

[Thrawn]

OK, some explanation seems in order :).
I ran ab with concurrency 1000 and a total of 3 requests, against each 
server, 5 times, plus one run each with 15 requests (sum of the previous 5 
tests).For Apache+PHP, this typically resulted in 5-15ms response time for 99% 
of requests, with the remaining few either taking tens of seconds or eventually 
disconnecting with an error.For HAProxy+Lua, 99% response times were 1ms, or 
sometimes 2ms, with the last few taking about 200ms. So, HAProxy worked much 
better (of course).
However, on the larger run (150k), HAProxy too had a small percentage of 
disconnections (apr_socket_recv: Connection reset by peer). I've been able to 
reproduce this with moderate consistency whenever I push it beyond about 35000 
total requests. It's still a better error rate than PHP, but I'd like to 
understand why the errors are occurring. For all I know, it's a problem with ab.

I've also tried a couple of runs with 15 requests but concurrency only 100, 
and neither server had trouble serving that, although interestingly, PHP is 
slightly more consistent: 99% within 4-5ms, then about 200ms for the last few, 
whereas HAProxy returns 99% within 1-2ms and 1800ms for the last few.

 The box is just my workstation, 8 cores and 16GB RAM, running Ubuntu 15.10, 
with no special tuning.
Any ideas on why the HAProxy tests showed disconnections or occasional slow 
response times at high loads?
 


 On Wednesday, 11 November 2015, 8:29, Baptiste  wrote:
   

 On Tue, Nov 10, 2015 at 10:46 PM, Thrawn
 wrote:
> OK, I've set this up locally, and tested it against PHP using ab.
>
> HAProxy was consistently faster (99% within 1ms, vs 5-15ms for PHP), but at
> request volumes over about 35000, with concurrency 1000, it consistently had
> a small percentage of socket disconnections. PHP had timeouts - or very long
> response times - and disconnections at pretty much any request volume with
> that concurrency, but I'm wondering where the errors stem from, or even if
> it's a limitation of ab.
>
> HAProxy config:
>
> global
>    maxconn 4096
>    daemon
>    nbproc 1
>    stats socket localhost:9461 level admin
>    chroot /etc/haproxy/jail
>    user haproxy
>    group haproxy
>    lua-load /etc/haproxy/jail/echo.lua
>
> defaults
>    log 127.0.0.1 local0
>    mode http
>    timeout client 6
>    timeout server 6
>    timeout connect 6
>    option forwardfor
>    balance roundrobin
>    option abortonclose
>    maxconn 20
>
> frontend echo
>    bind 127.0.1.1:1610
>    timeout client 1
>    mode http
>    http-request lua.echo
>
> Lua:
> core.register_action("echo", { "http-req" }, function (txn)
>    local buffer = txn.f:src()
>    txn.res:send("HTTP/1.0 200 OK\r\nServer:
> haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " ..
> buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)
>    txn:done()
> end)
>

Hi Thrawn

I'm sorry, but I don't understand anything to all your benchmarcks!
If you could at least give an explanation  before running each ab,
this may help.

Furthermore, you don't share anything about your hardware environment
neither the tuning you did on each box.
So it's impossible to help you.

At least, I can say that Lua seems to perform very well :)

Baptiste


  

Re: Echo server in Lua

[PiBa-NL]

b.t.w. if sole purpose of the frontend is to echo the ip back to the client.
You should probably also check the 'use-service' applet syntax, i dont 
know if that could be faster for your purpose.
Then another thing to check would be if you want to use the tcp or http 
service mode. A TCP service could be almost 1 line of lua code.. And i 
kinda expect to be a bit faster.


http://www.arpalert.org/src/haproxy-lua-api/1.6/index.html#haproxy-lua-hello-world
Instead of sending 'hello world' you could send the client-ip..

Op 10-11-2015 om 23:46 schreef Thrawn:

OK, some explanation seems in order :).

I ran ab with concurrency 1000 and a total of 3 requests, against 
each server, 5 times, plus one run each with 15 requests (sum of 
the previous 5 tests).
For Apache+PHP, this typically resulted in 5-15ms response time for 
99% of requests, with the remaining few either taking tens of seconds 
or eventually disconnecting with an error.
For HAProxy+Lua, 99% response times were 1ms, or sometimes 2ms, with 
the last few taking about 200ms. So, HAProxy worked much better (of 
course).


However, on the larger run (150k), HAProxy too had a small percentage 
of disconnections (apr_socket_recv: Connection reset by peer). I've 
been able to reproduce this with moderate consistency whenever I push 
it beyond about 35000 total requests. It's still a better error rate 
than PHP, but I'd like to understand why the errors are occurring. For 
all I know, it's a problem with ab.


I've also tried a couple of runs with 15 requests but concurrency 
only 100, and neither server had trouble serving that, although 
interestingly, PHP is slightly more consistent: 99% within 4-5ms, then 
about 200ms for the last few, whereas HAProxy returns 99% within 1-2ms 
and 1800ms for the last few.


The box is just my workstation, 8 cores and 16GB RAM, running Ubuntu 
15.10, with no special tuning.


Any ideas on why the HAProxy tests showed disconnections or occasional 
slow response times at high loads?




On Wednesday, 11 November 2015, 8:29, Baptiste  wrote:


On Tue, Nov 10, 2015 at 10:46 PM, Thrawn
> wrote:

> OK, I've set this up locally, and tested it against PHP using ab.
>
> HAProxy was consistently faster (99% within 1ms, vs 5-15ms for PHP), 
but at
> request volumes over about 35000, with concurrency 1000, it 
consistently had
> a small percentage of socket disconnections. PHP had timeouts - or 
very long
> response times - and disconnections at pretty much any request 
volume with
> that concurrency, but I'm wondering where the errors stem from, or 
even if

> it's a limitation of ab.
>
> HAProxy config:
>
> global
>maxconn 4096
>daemon
>nbproc 1
>stats socket localhost:9461 level admin
>chroot /etc/haproxy/jail
>user haproxy
>group haproxy
>lua-load /etc/haproxy/jail/echo.lua
>
> defaults
>log 127.0.0.1 local0
>mode http
>timeout client 6
>timeout server 6
>timeout connect 6
>option forwardfor
>balance roundrobin
>option abortonclose
>maxconn 20
>
> frontend echo
>bind 127.0.1.1:1610
>timeout client 1
>mode http
>http-request lua.echo
>
> Lua:
> core.register_action("echo", { "http-req" }, function (txn)
>local buffer = txn.f:src()
>txn.res:send("HTTP/1.0 200 OK\r\nServer:
> haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " ..
> buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)
>txn:done()
> end)
>

Hi Thrawn

I'm sorry, but I don't understand anything to all your benchmarcks!
If you could at least give an explanation  before running each ab,
this may help.

Furthermore, you don't share anything about your hardware environment
neither the tuning you did on each box.
So it's impossible to help you.

At least, I can say that Lua seems to perform very well :)

Baptiste

Re: Echo server in Lua

[Thrawn]

Hmm...I seem to be able to set up something in TCP mode, and it returns the 
expected response via curl, but its performance is awful. I must be doing 
something wrong?
Lua:
core.register_action("tcp-echo", {"tcp-req"}, function (txn)
    local buffer = txn.f:src()
    txn.res:send("HTTP/1.0 200 OK\r\nServer: haproxy-lua/echo\r\nContent-Type: 
text/html\r\nContent-Length: " .. buffer:len() .. "\r\nConnection: 
close\r\n\r\n" ..
    txn:done()
end)

I couldn't find a way for a TCP applet to retrieve the client IP address; 
suggestions are welcome.

HAProxy config:
frontend tcp-echo
    bind 127.0.2.1:1610
    timeout client 1
    mode tcp
    tcp-request content lua.tcp-echo

Testing this with ab frequently hangs and times out even at tiny loads (10 
requests with concurrency 3). 


 On Wednesday, 11 November 2015, 10:19, PiBa-NL  
wrote:
   

  b.t.w. if sole purpose of the frontend is to echo the ip back to the client.
 You should probably also check the 'use-service' applet syntax, i dont know if 
that could be faster for your purpose.
 Then another thing to check would be if you want to use the tcp or http 
service mode. A TCP service could be almost 1 line of lua code.. And i kinda 
expect to be a bit faster.
 
http://www.arpalert.org/src/haproxy-lua-api/1.6/index.html#haproxy-lua-hello-world
 Instead of sending 'hello world' you could send the client-ip..
 
 Op 10-11-2015 om 23:46 schreef Thrawn:
  
  OK, some explanation seems in order :). 
  I ran ab with concurrency 1000 and a total of 3 requests, against each 
server, 5 times, plus one run each with 15 requests (sum of the previous 5 
tests). For Apache+PHP, this typically resulted in 5-15ms response time for 99% 
of requests, with the remaining few either taking tens of seconds or eventually 
disconnecting with an error. For HAProxy+Lua, 99% response times were 1ms, or 
sometimes 2ms, with the last few taking about 200ms. So, HAProxy worked much 
better (of course). 
  However, on the larger run (150k), HAProxy too had a small percentage of 
disconnections (apr_socket_recv: Connection reset by peer). I've been able to 
reproduce this with moderate consistency whenever I push it beyond about 35000 
total requests. It's still a better error rate than PHP, but I'd like to 
understand why the errors are occurring. For all I know, it's a problem with ab.
  
  I've also tried a couple of runs with 15 requests but concurrency only 
100, and neither server had trouble serving that, although interestingly, PHP 
is slightly more consistent: 99% within 4-5ms, then about 200ms for the last 
few, whereas HAProxy returns 99% within 1-2ms and 1800ms for the last few.
  
   The box is just my workstation, 8 cores and 16GB RAM, running Ubuntu 15.10, 
with no special tuning. 
  Any ideas on why the HAProxy tests showed disconnections or occasional slow 
response times at high loads?
   
 
 
   On Wednesday, 11 November 2015, 8:29, Baptiste  wrote:
   
 
 On Tue, Nov 10, 2015 at 10:46 PM, Thrawn
  wrote:
 > OK, I've set this up locally, and tested it against PHP using ab.
 >
 > HAProxy was consistently faster (99% within 1ms, vs 5-15ms for PHP), but at
 > request volumes over about 35000, with concurrency 1000, it consistently had
 > a small percentage of socket disconnections. PHP had timeouts - or very long
 > response times - and disconnections at pretty much any request volume with
 > that concurrency, but I'm wondering where the errors stem from, or even if
 > it's a limitation of ab.
 >
 > HAProxy config:
 >
 > global
 >    maxconn 4096
 >    daemon
 >    nbproc 1
 >    stats socket localhost:9461 level admin
 >    chroot /etc/haproxy/jail
 >    user haproxy
 >    group haproxy
 >    lua-load /etc/haproxy/jail/echo.lua
 >
 > defaults
 >    log 127.0.0.1 local0
 >    mode http
 >    timeout client 6
 >    timeout server 6
 >    timeout connect 6
 >    option forwardfor
 >    balance roundrobin
 >    option abortonclose
 >    maxconn 20
 >
 > frontend echo
 >    bind 127.0.1.1:1610
 >    timeout client 1
 >    mode http
 >    http-request lua.echo
 >
 > Lua:
 > core.register_action("echo", { "http-req" }, function (txn)
 >    local buffer = txn.f:src()
 >    txn.res:send("HTTP/1.0 200 OK\r\nServer:
 > haproxy-lua/echo\r\nContent-Type: text/html\r\nContent-Length: " ..
 > buffer:len() .. "\r\nConnection: close\r\n\r\n" .. buffer)
 >    txn:done()
 > end)
 >
 
 Hi Thrawn
 
 I'm sorry, but I don't understand anything to all your benchmarcks!
 If you could at least give an explanation  before running each ab,
 this may help.
 
 Furthermore, you don't share anything about your hardware environment
 neither the tuning you did on each box.
 So it's impossible to help you.
 
 At least, I can say that Lua seems to perform very well :)
 
 Baptiste
 
 
  
 
 

  

Re: Echo server in Lua

[Thrawn]

Thanks, Baptiste.
I've had a go at setting that up, but found an issue in the process: I can't 
build HAProxy with Lua support unless I install the OpenSSL headers. Which I 
don't have any particular interest in doing, since I don't need SSL support.I 
notice that there was a patch in March to be able to build without actually 
enabling OpenSSL, but can there also be patching to allow it to be entirely 
absent from the system?
By the way, I'm not sure how much JIT optimisation is done in Lua, but it's 
probably fastest to inline the code?
core.register_action("echo", { "http-req" }, function (txn)
    txn.res:send("HTTP/1.0 200 OK\r\nServer: haproxy-lua/echo\r\nContent-Type: 
text/html\r\nContent-Length: " .. buffer:len() .. "\r\nConnection: 
close\r\n\r\n" .. txn.f:src())    txn.done()end)
Regards
Thrawn

  bed...@gmail.com  


 On Wednesday, 4 November 2015, 17:56, Baptiste <bed...@gmail.com> wrote:
   

 On Tue, Nov 3, 2015 at 5:53 AM, Thrawn <shell_layer-git...@yahoo.com.au> wrote:
> Now that HAProxy has Lua support, I'm looking at the possibility of setting 
> up an echo server, which simply responds with the observed remote address of 
> the client (currently implemented in PHP as  $_SERVER['REMOTE_ADDRESS']; ?>).
>
>
> Does anyone have a suggestion of the most efficient possible implementation 
> of this? If possible, it should handle millions of clients polling it 
> regularly, so speed is essential.
>
>
> Thanks
>

Hi,

content of echo.lua file:
-- a simple echo server
-- it generates a response whose body contains the client IP address
core.register_action("echo", { "http-req" }, function (txn)
        local buffer = ""
        local response = ""

        buffer = txn.f:src()

        response = response .. "HTTP/1.0 200 OK\r\n"
        response = response .. "Server: haproxy-lua/echo\r\n"
        response = response .. "Content-Type: text/html\r\n"
        response = response .. "Content-Length: " .. buffer:len() .. "\r\n"
        response = response .. "Connection: close\r\n"
        response = response .. "\r\n"
        response = response .. buffer

        txn.res:send(response)
        txn:done()
end)

content of haproxy's configuration:

global
  log 127.0.0.1 local0
  lua-load echo.lua

frontend echo
  bind *:10004
  mode http
  http-request lua.echo


Don't forget to setup timeouts, etc...

Baptiste


  

Re: Echo server in Lua

[Baptiste]

On Tue, Nov 3, 2015 at 5:53 AM, Thrawn <shell_layer-git...@yahoo.com.au> wrote:
> Now that HAProxy has Lua support, I'm looking at the possibility of setting 
> up an echo server, which simply responds with the observed remote address of 
> the client (currently implemented in PHP as  $_SERVER['REMOTE_ADDRESS']; ?>).
>
>
> Does anyone have a suggestion of the most efficient possible implementation 
> of this? If possible, it should handle millions of clients polling it 
> regularly, so speed is essential.
>
>
> Thanks
>

Hi,

content of echo.lua file:
-- a simple echo server
-- it generates a response whose body contains the client IP address
core.register_action("echo", { "http-req" }, function (txn)
local buffer = ""
local response = ""

buffer = txn.f:src()

    response = response .. "HTTP/1.0 200 OK\r\n"
response = response .. "Server: haproxy-lua/echo\r\n"
response = response .. "Content-Type: text/html\r\n"
response = response .. "Content-Length: " .. buffer:len() .. "\r\n"
response = response .. "Connection: close\r\n"
response = response .. "\r\n"
response = response .. buffer

txn.res:send(response)
txn:done()
end)

content of haproxy's configuration:

global
  log 127.0.0.1 local0
  lua-load echo.lua

frontend echo
  bind *:10004
  mode http
  http-request lua.echo


Don't forget to setup timeouts, etc...

Baptiste

Echo server in Lua

[Thrawn]

Now that HAProxy has Lua support, I'm looking at the possibility of setting up 
an echo server, which simply responds with the observed remote address of the 
client (currently implemented in PHP as ).


Does anyone have a suggestion of the most efficient possible implementation of 
this? If possible, it should handle millions of clients polling it regularly, 
so speed is essential.


Thanks