On 06/05/2015 12:03 μμ, Baptiste wrote: > On Wed, May 6, 2015 at 7:15 AM, Krishna Kumar (Engineering) > <[email protected]> wrote: >> Hi Baptiste, >> >> On Wed, May 6, 2015 at 1:24 AM, Baptiste <[email protected]> wrote: >>> >>>> Also, during the test, the status of various backend's change often >>>> between >>>> OK to DOWN, >>>> and then gets back to OK almost immediately: >>>> >>>> >>>> www-backend,nginx-3,0,0,0,10,30000,184,23843,96517588,,0,,27,0,0,180,DOWN >>>> >>>> 1/2,1,1,0,7,3,6,39,,7,3,1,,220,,2,0,,37,L4CON,,0,0,184,0,0,0,0,0,,,,0,0,,,,,6,Out >>>> of local source ports on the system,,0,2,3,92, >>> >>> this error is curious with the type of traffic your generating! >>> Maybe you should let HAProxy manage the source ports on behalf of the >>> server. >>> Try adding the "source 0.0.0.0:1024-65535" parameter in your backend >>> description. >> >> >> Yes, this has fixed the issue - I no longer get state change after an hour >> testing. >> The performance didn't improve though. I will check the sysctl parameters >> that >> were different between haproxy/nginx nodes. >> >> Thanks, >> - Krishna Kumar > > > You have to investigate why this issue happened. > I mean, it is not normal. As Pavlos mentionned, you connection rate is > very low, since you do keep alive and you opened only 500 ports. > > Wait, I know, could you share the keep-alive connection from your nginx > servers? > By default, they close connections every 100 requests... This might be > the root of the issue. >
That reminds that in my setup I configured nginx backend servers to have keepalive_requests 100000. You need to increase the keepalive limit because stress tool use keepalive as well. So, let's say you have 500 TCP concurrent connections open and stress test does 5M requests in total, you need to allow 10K keepalived http requests on nginx . I have a suggestion, rerun the test with haproxy and on your nginx server, and nginx on your haproxy server. Cheers, Pavlos
signature.asc
Description: OpenPGP digital signature
