I found the source of the problem. One of the backends was being shared with another person who was testing iptables rules/tunnel setups, and that might have caused some connection drops. I have now removed that backend from my setup and use dedicated systems, after which the original configuration without specifying source port is working, no connection flaps now.
Thanks, - Krishna Kumar On Wed, May 6, 2015 at 4:53 PM, Willy Tarreau <[email protected]> wrote: > On Wed, May 06, 2015 at 12:03:12PM +0200, Baptiste wrote: > > On Wed, May 6, 2015 at 7:15 AM, Krishna Kumar (Engineering) > > <[email protected]> wrote: > > > Hi Baptiste, > > > > > > On Wed, May 6, 2015 at 1:24 AM, Baptiste <[email protected]> wrote: > > >> > > >> > Also, during the test, the status of various backend's change often > > >> > between > > >> > OK to DOWN, > > >> > and then gets back to OK almost immediately: > > >> > > > >> > > > >> > > www-backend,nginx-3,0,0,0,10,30000,184,23843,96517588,,0,,27,0,0,180,DOWN > > >> > > > >> > > 1/2,1,1,0,7,3,6,39,,7,3,1,,220,,2,0,,37,L4CON,,0,0,184,0,0,0,0,0,,,,0,0,,,,,6,Out > > >> > of local source ports on the system,,0,2,3,92, > > >> > > >> this error is curious with the type of traffic your generating! > > >> Maybe you should let HAProxy manage the source ports on behalf of the > > >> server. > > >> Try adding the "source 0.0.0.0:1024-65535" parameter in your backend > > >> description. > > > > > > > > > Yes, this has fixed the issue - I no longer get state change after an > hour > > > testing. > > > The performance didn't improve though. I will check the sysctl > parameters > > > that > > > were different between haproxy/nginx nodes. > > > > > > Thanks, > > > - Krishna Kumar > > > > > > You have to investigate why this issue happened. > > I mean, it is not normal. As Pavlos mentionned, you connection rate is > > very low, since you do keep alive and you opened only 500 ports. > > > > Wait, I know, could you share the keep-alive connection from your nginx > servers? > > By default, they close connections every 100 requests... This might be > > the root of the issue. > > But even then there is no reason why the local ports would remain in use. > There definitely is a big problem. It also explains why servers are going > up and down all the time and errors are reported. > > Willy > >
