Re: HAProxy bottleneck/tuning
Thanks for the answer. Le lun. 30 sept. 2019 à 05:51, Willy Tarreau a écrit : > Hi, > > On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote: > > Hi, > > > > We use HAProxy as a LB for many usage, including LB for Squid and user > > acces on Internet. > > Users frequently grumble about "Internet speed" (classical...) and we > want > > to be sure that it is not a LB issue (msconfig, or system bottleneck ?). > > > > How could we find is haproxy is "undersized" in config or if the hosting > > system (CentOS) is not correctly configured (file desc ? memory ?) > > Well, this sounds more like a sysadmin training than an haproxy-specific > question, but a few things you should have a look at : > - is your system swapping ? (it must never) > - do you see in your system logs "conntrack table full" ? If so you're > hitting some misconfigured conntrack limits > - do you see in your system logs "TCP: sending SYN cookies" ? If so > you're likely running haproxy with too low a maxconn setting, resulting > in connections not being accepted > - do you see in your haproxy logs flags "sC" or "SC" while you know > your servers are working fine ? If so that could indicate a failure to > allocate a source port or a file descriptor for an outgoing connection > - do you see your CPU steadily running above, say, 50% ? If so you cannot > exclude frequent 100% peaks possibly causing some sub-second delays not > reported by the system tools. > - and if you're running in a VM, you can redo all this above inside the > hypervisor, and in the guest you should also look at the "st" field in > vmstat to make sure your CPU is not stolen by other VMs (or goodbye low > latency), and you can also run "ping" to your VM from an external host > on the same LAN and make sure the latency never jumps above 1 > millisecond > on a mostly idle network or 2-3 ms on a loaded network or it can > indicate that you're have performance issues caused by noisy neighbors > on your machine. > - connection setup timers exhibiting multiple of 3s in haproxy logs > usually indicate packet drops between haproxy and the servers > - client timeout errors during request like "cR" often indicate drops or > MTU issues between the client and haproxy (sometimes caused by bogus > virtualized network drivers). > > Hoping this helps, > Willy >
Re: HAProxy bottleneck/tuning
Hi, On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote: > Hi, > > We use HAProxy as a LB for many usage, including LB for Squid and user > acces on Internet. > Users frequently grumble about "Internet speed" (classical...) and we want > to be sure that it is not a LB issue (msconfig, or system bottleneck ?). > > How could we find is haproxy is "undersized" in config or if the hosting > system (CentOS) is not correctly configured (file desc ? memory ?) Well, this sounds more like a sysadmin training than an haproxy-specific question, but a few things you should have a look at : - is your system swapping ? (it must never) - do you see in your system logs "conntrack table full" ? If so you're hitting some misconfigured conntrack limits - do you see in your system logs "TCP: sending SYN cookies" ? If so you're likely running haproxy with too low a maxconn setting, resulting in connections not being accepted - do you see in your haproxy logs flags "sC" or "SC" while you know your servers are working fine ? If so that could indicate a failure to allocate a source port or a file descriptor for an outgoing connection - do you see your CPU steadily running above, say, 50% ? If so you cannot exclude frequent 100% peaks possibly causing some sub-second delays not reported by the system tools. - and if you're running in a VM, you can redo all this above inside the hypervisor, and in the guest you should also look at the "st" field in vmstat to make sure your CPU is not stolen by other VMs (or goodbye low latency), and you can also run "ping" to your VM from an external host on the same LAN and make sure the latency never jumps above 1 millisecond on a mostly idle network or 2-3 ms on a loaded network or it can indicate that you're have performance issues caused by noisy neighbors on your machine. - connection setup timers exhibiting multiple of 3s in haproxy logs usually indicate packet drops between haproxy and the servers - client timeout errors during request like "cR" often indicate drops or MTU issues between the client and haproxy (sometimes caused by bogus virtualized network drivers). Hoping this helps, Willy
HAProxy bottleneck/tuning
Hi, We use HAProxy as a LB for many usage, including LB for Squid and user acces on Internet. Users frequently grumble about "Internet speed" (classical...) and we want to be sure that it is not a LB issue (msconfig, or system bottleneck ?). How could we find is haproxy is "undersized" in config or if the hosting system (CentOS) is not correctly configured (file desc ? memory ?) Regards,