Re: Help tracking "connection refused" under pressure on v2.9

2024-03-29 Thread Ricardo Nabinger Sanchez
Hi Willy, On Fri, 29 Mar 2024 07:17:56 +0100 Willy Tarreau wrote: > > These "connection refused" is from our watchdog; but the effects are as > > perceptible from the outside. When our watchdog hits this situation, > > it will forcefully restart HAProxy (we have 2 instances) because there > >

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-29 Thread Willy Tarreau
Hi Ricardo, On Thu, Mar 28, 2024 at 06:21:16PM -0300, Ricardo Nabinger Sanchez wrote: > Hi Willy, > > On Thu, 28 Mar 2024 04:37:11 +0100 > Willy Tarreau wrote: > > > Thanks guys! So there seems to be an annoying bug. However I'm not sure > > how this is related to your "connection refused",

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-28 Thread Ricardo Nabinger Sanchez
Hi Willy, On Thu, 28 Mar 2024 04:37:11 +0100 Willy Tarreau wrote: > Thanks guys! So there seems to be an annoying bug. However I'm not sure > how this is related to your "connection refused", except if you try to > connect at the moment the process crashes and restarts, of course. I'm > seeing

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-27 Thread Willy Tarreau
On Wed, Mar 27, 2024 at 02:26:47PM -0300, Ricardo Nabinger Sanchez wrote: > On Wed, 27 Mar 2024 11:06:39 -0300 > Felipe Wilhelms Damasio wrote: > > > kernel: traps: haproxy[2057993] trap invalid opcode ip:5b3e26 > > sp:7fd7c002f100 error:0 in haproxy[42c000+1f7000] > > We managed to get a core

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-27 Thread Ricardo Nabinger Sanchez
On Wed, 27 Mar 2024 11:06:39 -0300 Felipe Wilhelms Damasio wrote: > kernel: traps: haproxy[2057993] trap invalid opcode ip:5b3e26 > sp:7fd7c002f100 error:0 in haproxy[42c000+1f7000] We managed to get a core file, and so created ticket #2508 (https://github.com/haproxy/haproxy/issues/2508) with

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-27 Thread Ricardo Nabinger Sanchez
On Wed, 27 Mar 2024 11:06:39 -0300 Felipe Wilhelms Damasio wrote: > kernel: traps: haproxy[2057993] trap invalid opcode ip:5b3e26 sp:7fd7c002f100 > error:0 in haproxy[42c000+1f7000] In our build, this would be where instruction pointer was: (gdb) list *0x5b10e6 0x5b10e6 is in __task_queue

Re: Help tracking "connection refused" under pressure on v2.9

2024-03-27 Thread Felipe Wilhelms Damasio
Hi, We've confirmed a few findings after we poured ~75-80Gbps of traffic on purpose on a single machine: - haproxy does indeed crashes; - hence, we have no stats socket to collect a few things; It seems that under pressure (not sure which conditions yet) the kernel seems to be killing it. dmesg

Help tracking "connection refused" under pressure on v2.9

2024-03-26 Thread Felipe Wilhelms Damasio
Hi, Since we don't really know how to track this one, we thought it might be better to reach out here to get feedback. We're using haproxy to deliver streaming files under pressure (80-90Gbps per machine). When using h1/http, splice-response is a great help to keep load under control. We use