Re: HAProxy bottleneck/tuning

2019-09-30 Thread Emmanuel BILLOT
Thanks for the answer.

Le lun. 30 sept. 2019 à 05:51, Willy Tarreau  a écrit :

> Hi,
>
> On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote:
> > Hi,
> >
> > We use HAProxy as a LB for many usage, including LB for Squid and user
> > acces on Internet.
> > Users frequently grumble about "Internet speed" (classical...) and we
> want
> > to be sure that it is not a LB issue (msconfig, or system bottleneck ?).
> >
> > How could we find is haproxy is "undersized" in config or if the hosting
> > system (CentOS) is not correctly configured (file desc ? memory ?)
>
> Well, this sounds more like a sysadmin training than an haproxy-specific
> question, but a few things you should have a look at :
>   - is your system swapping ? (it must never)
>   - do you see in your system logs "conntrack table full" ? If so you're
> hitting some misconfigured conntrack limits
>   - do you see in your system logs "TCP: sending SYN cookies" ? If so
> you're likely running haproxy with too low a maxconn setting, resulting
> in connections not being accepted
>   - do you see in your haproxy logs flags "sC" or "SC" while you know
> your servers are working fine ? If so that could indicate a failure to
> allocate a source port or a file descriptor for an outgoing connection
>   - do you see your CPU steadily running above, say, 50% ? If so you cannot
> exclude frequent 100% peaks possibly causing some sub-second delays not
> reported by the system tools.
>   - and if you're running in a VM, you can redo all this above inside the
> hypervisor, and in the guest you should also look at the "st" field in
> vmstat to make sure your CPU is not stolen by other VMs (or goodbye low
> latency), and you can also run "ping" to your VM from an external host
> on the same LAN and make sure the latency never jumps above 1
> millisecond
> on a mostly idle network or 2-3 ms on a loaded network or it can
> indicate that you're have performance issues caused by noisy neighbors
> on your machine.
>   - connection setup timers exhibiting multiple of 3s in haproxy logs
> usually indicate packet drops between haproxy and the servers
>   - client timeout errors during request like "cR" often indicate drops or
> MTU issues between the client and haproxy (sometimes caused by bogus
> virtualized network drivers).
>
> Hoping this helps,
> Willy
>


Re: HAProxy bottleneck/tuning

2019-09-29 Thread Willy Tarreau
Hi,

On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote:
> Hi,
> 
> We use HAProxy as a LB for many usage, including LB for Squid and user
> acces on Internet.
> Users frequently grumble about "Internet speed" (classical...) and we want
> to be sure that it is not a LB issue (msconfig, or system bottleneck ?).
> 
> How could we find is haproxy is "undersized" in config or if the hosting
> system (CentOS) is not correctly configured (file desc ? memory ?)

Well, this sounds more like a sysadmin training than an haproxy-specific
question, but a few things you should have a look at :
  - is your system swapping ? (it must never)
  - do you see in your system logs "conntrack table full" ? If so you're
hitting some misconfigured conntrack limits
  - do you see in your system logs "TCP: sending SYN cookies" ? If so
you're likely running haproxy with too low a maxconn setting, resulting
in connections not being accepted
  - do you see in your haproxy logs flags "sC" or "SC" while you know
your servers are working fine ? If so that could indicate a failure to
allocate a source port or a file descriptor for an outgoing connection
  - do you see your CPU steadily running above, say, 50% ? If so you cannot
exclude frequent 100% peaks possibly causing some sub-second delays not
reported by the system tools.
  - and if you're running in a VM, you can redo all this above inside the
hypervisor, and in the guest you should also look at the "st" field in
vmstat to make sure your CPU is not stolen by other VMs (or goodbye low
latency), and you can also run "ping" to your VM from an external host
on the same LAN and make sure the latency never jumps above 1 millisecond
on a mostly idle network or 2-3 ms on a loaded network or it can
indicate that you're have performance issues caused by noisy neighbors
on your machine.
  - connection setup timers exhibiting multiple of 3s in haproxy logs
usually indicate packet drops between haproxy and the servers
  - client timeout errors during request like "cR" often indicate drops or
MTU issues between the client and haproxy (sometimes caused by bogus
virtualized network drivers).

Hoping this helps,
Willy