Oh, forgot to answer the question about siege; I've configured siege to
run 15 threads and to generate a non-stop random set of URL's so all the
subsystems get tested by it.
On 09/28/2012 06:45 AM, Baptiste wrote:
Hi,
First question: how does siege generate the requests: how much and
what frequency?
I don't know how does your application works, but a single HTTP
request may generate a MySQL, a PH
P-FPM and a memcache request.
So a single HTTP request may require at least 4 connections (maybe more).
Since you're using Xen, which type of virtual interface are you using?
The Virtual interface choice can divide by 10 the performance in Xen
and KVM, so hopefully you did the right choice :)
How many virtual CPU have you dedicated to your virtual LB?
Could you provide us (or to me in private) a screenshot of the stats
page right after the test?
And yes, which version of HAProxy are you running?
Something weird is that your test lasts only 5s... That amount of
errors in a so short time is not normal.
A few improvements you could add:
- enable option httplog on every HTTP frontend (to get very verbose
logs) which could give us information
- enable option htp-server-close on every HTTP frontend and backend
(but I think the impact of such feature will be low cause your problem
is not related to it)
- configure some maxconn on servers depending on their capacity and/or
he limit you setup them in term of connections
cheers
On Fri, Sep 28, 2012 at 6:24 AM, Fred Leeflang<[email protected]> wrote:
BTW, I've also just run siege as root, thinking I might have been hitting my
own ulimit somehow (my ulimit -n was set to 1024 which seemed all too
coincidental :). This didn't make any difference, alas.
-Fred Leeflang
On 09/28/2012 06:01 AM, Fred Leeflang wrote:
Hello,
I have set up haproxy on two virtual (Xen) machines, listening to two
virtual IP addresses (heartbeat).
It's loadbalancing nginx, varnish, memcache, php-fpm and mysql and
functionally it works. I seem to be running into some sort of resource
limitation however; When I run siege on the website it always ends up
saying:
siege aborted due to excessive socket failure; you
can change the failure threshold in $HOME/.siegerc
Transactions: 1038 hits
Availability: 50.34 %
Elapsed time: 5.26 secs
Data transferred: 8.24 MB
Response time: 0.07 secs
Transaction rate: 197.34 trans/sec
Throughput: 1.57 MB/sec
Concurrency: 14.05
Successful transactions: 1038
Failed transactions: 1024
Longest transaction: 1.04
Shortest transaction: 0.01
The number of failed transactions and the number of transactions is always
the same.
I've tweaked the kernel with some things I've found on the net:
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_fin_timeout = 3
net.ipv4.tcp_tw_recycle = 0
net.core.netdev_max_backlog = 30000
net.ipv4.tcp_no_metrics_save=1
net.core.somaxconn = 262144
net.ipv4.tcp_syncookies = 0
net.ipv4.tcp_max_orphans = 262144
net.ipv4.tcp_max_syn_backlog = 262144
net.ipv4.tcp_synack_retries = 2
net.ipv4.ip_nonlocal_bind=1
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_tw_buckets = 400000
I've included my config as well.
Any ideas as to what I have forgotten about that makes the system run out
of resources here? I can't find anything in the syslog that indicates any
problems. It might be the Xen host logs anything but have not checked that
yet, or it may also be the BSD firewall (in front of these two haproxy
servers) that runs out of resources but haven't checked those yet as I'd
first like to get my haproxy setup checked.
Thanks,
Fred Leeflang
