Hi Willy, Thank you for your detailed explanation; you are entirely right and the limit that was screwing me was the limit of fds/process which I have now fixed.
Can you explain which parameter should be set to 2 to prevent malloc() fails when there is no more memory left? Many thanks, Alex On Tue, Oct 4, 2011 at 6:49 AM, Willy Tarreau <[email protected]> wrote: > Hi Alex, > > On Tue, Oct 04, 2011 at 01:38:40AM +0100, Alex Davies wrote: > > Hi, > > > > I am running haproxy 1.4.18 on RHEL6 (64 bit). I am trying to run a > > large number of connections (I have long pooling backends), ideally > > several hundred k. My servers have 24G RAM and are doing nothing else. > > I have two problems. > > > > Firstly, if I increase teh "nofile" hard limit in > > /etc/security/limits.conf to anything over 1048576 (1 Megabyte, > > although I think this is defined as a # of files) I can't SSH to the > > box or start haproxy - new SSH connections are immediately closed > > without entries in the logs. The kernel limit (/proc/sys/fs/file-max) > > is set to >200k without problem. Haproxy balks as follows: > > > > Starting haproxy: [WARNING] 276/012759 (2924) : > > [/usr/sbin/haproxy.main()] Cannot raise FD limit to 2000046. > > [WARNING] 276/012759 (2924) : [/usr/sbin/haproxy.main()] FD limit > > (1048576) too low for maxconn=1000000/maxsock=2000046. Please raise > > 'ulimit-n' to 2000046 or more to avoid any trouble. > > There is a default limit of 1048576 fds per process, which you can > change with nr_open. But be careful when increasing all those values! > Some programs might not like it at all and some scripts may become > slow as hell when they close every possible fd before a fork(). So > you don't want to have everything running unlimited. Ideally, keep > the default ulimit at 1024 and let haproxy change it without impacting > other processes. > > Note: the highest test that was done on haproxy was with 1M fds, nothing > above (about 480k conns). > > > Secondly, I notice that as my number of connections rises I get "Out > > of socket memory" errors in the kernel log. Google led me to believe > > that this would be a problem caused by excessive orphan sockets, but > > this seems not to be the case: > > I don't believe in orphans too for this case, because normally you see > a message "too many of orphans" long before running out of memory, since > orphans are often limited to a small number. > > > [root@frontend2 log]# cat /proc/net/sockstat | grep orphan > > TCP: inuse 134376 orphan 0 tw 87 alloc 150249 mem 100217 > > (output taken during a time of problem, where haproxy was mostly > > inaccessible and most connections were refused) > > > > I also seem to have plenty of spare sockets: > > > > [root@frontend2 log]# cat /proc/sys/net/ipv4/tcp_mem > > 2303712 3071616 4607424 > > [root@frontend2 log]# cat /proc/net/sockstat > > sockets: used 150458 > > TCP: inuse 134376 orphan 1 tw 69 alloc 150250 mem 100217 > > UDP: inuse 12 mem 3 > > UDPLITE: inuse 0 > > RAW: inuse 0 > > FRAG: inuse 0 memory 0 > > > > (100217 < 4607424 by some margin). > > The usual thing is when some socket buffers are too large. For instance, > the default 87k read and 64k write buffers imply that you consume 150k > by default for each socket, until the kernel starts to reclaim socket > memory. All the people I know who use very large amounts of sockets use > small buffers, both in haproxy and in the kernel, because most often > it's for chat/websocket or such things which transport few data. You > could very well set your default kernel TCP buffers to 8k and let them > grow larger if needed : > > tcp_rmem 4096 8192 65536 > tcp_wmem 4096 8192 65536 > > You should never forget that a full connection through haproxy uses : > - 2 TCP sockets, so 2 tcp_rmem and 2 tcp_wmem > - 2 haproxy buffers (one per direction), each of global.bufsize. > > So if you set haproxy to 8kB and your kernel to 8kB, you will need > 16kB for haproxy and 32kB for the system = 48kB per connection. That's > already too much for 1M connections at 24 GB. The kernel will never > go below 4kB, and at this point you'll have 32kB total = 32 GB of RAM > per million connection. > > > Ideally i'd like to set the number of open files and sockets to as > > close to infinite as possible - and allow them to use all the system > > RAM. > > You will use all the system's RAM if socket buffers are properly tuned. > I'm used to say that you need 1/3 of the RAM for TCP buffers, another 1/3 > for haproxy and the last 1/3 for the rest of the system (caches, ...). > With 24 GB, you can even assume close to 12+12 and almost nothing for > the system. Your TCP settings already allow you to use up to 18 GB RAM > for sockets. Haproxy will allocate what it needs for its maxconn, so in > practice you can max out memory. It is even a bit dangerous, you should > ensure that the system will never overcommit memory (set the param to 2) > so that it fails on malloc() when the limit is reached instead of > pretending the memory exists and killing the process when it tries to > use it. But that's for a second point. > > Regards, > Willy > > -- Alex Davies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender immediately by e-mail and delete this e-mail permanently.
