Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-13 Thread Maciej Zdeb
Ok, I'll wait for update from you! :) pon., 13 maj 2019 o 08:03 Willy Tarreau napisał(a): > Hi Maciej, > > On Mon, May 13, 2019 at 07:21:59AM +0200, Maciej Zdeb wrote: > > Hi, > > > > I'm not observing any issues, so I think it's fixed. :) > > > > Willy, Olivier thank you very much! > > Great,

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-13 Thread Willy Tarreau
Hi Maciej, On Mon, May 13, 2019 at 07:21:59AM +0200, Maciej Zdeb wrote: > Hi, > > I'm not observing any issues, so I think it's fixed. :) > > Willy, Olivier thank you very much! Great, thanks. I'm going to issue 1.9.8 with the patch I sent you then. However after discussing about it with

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-12 Thread Maciej Zdeb
Hi, I'm not observing any issues, so I think it's fixed. :) Willy, Olivier thank you very much! Kind regards, niedz., 12 maj 2019 o 12:11 Willy Tarreau napisał(a): > On Sun, May 12, 2019 at 12:00:44PM +0200, Maciej Zdeb wrote: > > As for now it is stable, no looping. I'll leave it till

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-12 Thread Willy Tarreau
On Sun, May 12, 2019 at 12:00:44PM +0200, Maciej Zdeb wrote: > As for now it is stable, no looping. I'll leave it till monday and return > with feedback! :) Many thanks Maciej for this positive feedback. I'll merge it and issue 1.9.8 then. There are definitely enough pending fixes for a release!

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-12 Thread Maciej Zdeb
As for now it is stable, no looping. I'll leave it till monday and return with feedback! :) Thanks, sob., 11.05.2019, 15:44 użytkownik Maciej Zdeb napisał: > Patch applied, finger crossed, testing! :-) > > Thanks! > > sob., 11 maj 2019 o 14:58 Willy Tarreau napisał(a): > >> On Sat, May 11,

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Maciej Zdeb
Patch applied, finger crossed, testing! :-) Thanks! sob., 11 maj 2019 o 14:58 Willy Tarreau napisał(a): > On Sat, May 11, 2019 at 11:01:42AM +0200, Willy Tarreau wrote: > > On Sat, May 11, 2019 at 10:52:35AM +0200, Willy Tarreau wrote: > > > I certainly made a few reasoning mistakes above but

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Willy Tarreau
On Sat, May 11, 2019 at 11:01:42AM +0200, Willy Tarreau wrote: > On Sat, May 11, 2019 at 10:52:35AM +0200, Willy Tarreau wrote: > > I certainly made a few reasoning mistakes above but I don't see anything > > in the code preventing this case from happening. > > > > Thus I'd like you to try the

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Maciej Zdeb
> What I find very strange is why you're possibly the only one seeing this > (and maybe also @serimin on github issue #94). If we could figure what > makes your case specific it could help narrow the issue down. I'm seeing > that you have a very simple Lua service to respond to health checks, so >

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Willy Tarreau
On Sat, May 11, 2019 at 10:52:35AM +0200, Willy Tarreau wrote: > I certainly made a few reasoning mistakes above but I don't see anything > in the code preventing this case from happening. > > Thus I'd like you to try the attached patch which is supposed to prevent > this scenario from happening.

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Willy Tarreau
On Sat, May 11, 2019 at 09:56:18AM +0200, Willy Tarreau wrote: > I'm back to auditing the code to figure how we can free an h2s without > first detaching it from the lists. I hope to have yet another patch to > propose to you. So I'm seeing something which bothers me in the code. Since I'm not at

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-11 Thread Willy Tarreau
Hi Maciej, On Fri, May 10, 2019 at 06:45:21PM +0200, Maciej Zdeb wrote: > Olivier, it's still looping, but differently: > > 2609list_for_each_entry_safe(h2s, h2s_back, >send_list, list) { > (gdb) n > 2610if (h2c->st0 >= H2_CS_ERROR || h2c->flags & > H2_CF_MUX_BLOCK_ANY) >

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-10 Thread Maciej Zdeb
Olivier, it's still looping, but differently: 2609list_for_each_entry_safe(h2s, h2s_back, >send_list, list) { (gdb) n 2610if (h2c->st0 >= H2_CS_ERROR || h2c->flags & H2_CF_MUX_BLOCK_ANY) (gdb) 2609list_for_each_entry_safe(h2s, h2s_back, >send_list, list) { (gdb) 2613

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-10 Thread Maciej Zdeb
I've just sent some additional data to Willy. :) Sure, I'll test your patch! pt., 10 maj 2019 o 15:11 Olivier Houchard napisał(a): > Hi Maciej, > > On Thu, May 09, 2019 at 07:25:54PM +0200, Maciej Zdeb wrote: > > Hi again, > > > > I have bad news, HAProxy 1.9.7-35b44da still looping :/ > > > >

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-10 Thread Olivier Houchard
Hi Maciej, On Thu, May 09, 2019 at 07:25:54PM +0200, Maciej Zdeb wrote: > Hi again, > > I have bad news, HAProxy 1.9.7-35b44da still looping :/ > > gdb session: > h2_process_mux (h2c=0x1432420) at src/mux_h2.c:2609 > 2609list_for_each_entry_safe(h2s, h2s_back, >send_list, list) { >

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-10 Thread Maciej Zdeb
I'm gettingh old... I failed to remember to dump core :( And already killed the process. Sorry, but the issue must reoccur and I can't say how long it may take. As soon as I get core dump I'll return. pt., 10.05.2019, 10:35 użytkownik Willy Tarreau napisał: > Hi Maciej, > > On Thu, May 09,

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-10 Thread Willy Tarreau
Hi Maciej, On Thu, May 09, 2019 at 07:25:54PM +0200, Maciej Zdeb wrote: > Hi again, > > I have bad news, HAProxy 1.9.7-35b44da still looping :/ Well, it's getting really annoying. Something's definitely wrong in this list and I can't figure what. > 2609list_for_each_entry_safe(h2s,

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Maciej Zdeb
Hi again, I have bad news, HAProxy 1.9.7-35b44da still looping :/ gdb session: h2_process_mux (h2c=0x1432420) at src/mux_h2.c:2609 2609list_for_each_entry_safe(h2s, h2s_back, >send_list, list) { (gdb) n 2610if (h2c->st0 >= H2_CS_ERROR || h2c->flags & H2_CF_MUX_BLOCK_ANY)

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Maciej Zdeb
I'm happy to help! :) Checking Olivier patch. Thanks! czw., 9 maj 2019 o 14:34 Willy Tarreau napisał(a): > On Thu, May 09, 2019 at 02:31:58PM +0200, Maciej Zdeb wrote: > > What a bad luck :D I must have compiled it just before you pushed that > > change (segfault above is from haproxy

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Willy Tarreau
On Thu, May 09, 2019 at 02:31:58PM +0200, Maciej Zdeb wrote: > What a bad luck :D I must have compiled it just before you pushed that > change (segfault above is from haproxy 1.9.7-9b8ac0f). Great, so there's still some hope. I really appreciate your help and feedback here, such issues are

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Willy Tarreau
On Thu, May 09, 2019 at 02:19:26PM +0200, Maciej Zdeb wrote: > Hi Willy, > > I've built 1.9 from head, unfortunately something is wrong, right now I've > got segfault: > > Core was generated by `/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p > /var/run/haproxy.pid -D -sf 75'. > Program

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Maciej Zdeb
Hi Willy, I've built 1.9 from head, unfortunately something is wrong, right now I've got segfault: Core was generated by `/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 75'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00484ab4 in

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-09 Thread Willy Tarreau
Hi Maciej, I've just pushed a number of fixes into 1.9-master, including the one I was talking about, if you want to try again. Cheers, Willy

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Maciej Zdeb
Ok, thanks! śr., 8.05.2019, 16:50 użytkownik Willy Tarreau napisał: > On Wed, May 08, 2019 at 04:48:55PM +0200, Maciej Zdeb wrote: > > I'll gladly test Olivier patch after backporting. :) > > Thank you Maciej. I still have other stuff to deal with before going back > to the pending 1.9

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Maciej Zdeb
I'll gladly test Olivier patch after backporting. :) śr., 8.05.2019, 15:29 użytkownik Willy Tarreau napisał: > On Wed, May 08, 2019 at 03:03:23PM +0200, Olivier Houchard wrote: > > > > I can't seem to remember :) > > > > > > Given the number of bugs we've dealt with in the last few weeks,

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Willy Tarreau
On Wed, May 08, 2019 at 03:03:23PM +0200, Olivier Houchard wrote: > > > I can't seem to remember :) > > > > Given the number of bugs we've dealt with in the last few weeks, you're > > forgiven :-) > > > > I'm afraid I'm getting old :/ Ah! you see how it feels! > > > I think that patch is safe

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Olivier Houchard
On Wed, May 08, 2019 at 02:30:07PM +0200, Willy Tarreau wrote: > On Wed, May 08, 2019 at 01:56:05PM +0200, Olivier Houchard wrote: > > > > One of processes stuck in infinite loop, admin socket is not responsive > > > > so > > > > I've got information only from gdb: > > > > > > > >

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Willy Tarreau
On Wed, May 08, 2019 at 01:56:05PM +0200, Olivier Houchard wrote: > > > One of processes stuck in infinite loop, admin socket is not responsive so > > > I've got information only from gdb: > > > > > > 0x00484ab8 in h2_process_mux (h2c=0x2e8ff30) at src/mux_h2.c:2589 > > > 2589

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-08 Thread Olivier Houchard
Hi, On Wed, May 08, 2019 at 07:11:27AM +0200, Willy Tarreau wrote: > Hi Maciej, > > On Tue, May 07, 2019 at 07:08:47PM +0200, Maciej Zdeb wrote: > > Hi, > > > > I've got another bug with 100% CPU on HAProxy process, it is built from > > HEAD of 1.9 branch. > > > > One of processes stuck in

Re: [1.9 HEAD] HAProxy using 100% CPU

2019-05-07 Thread Willy Tarreau
Hi Maciej, On Tue, May 07, 2019 at 07:08:47PM +0200, Maciej Zdeb wrote: > Hi, > > I've got another bug with 100% CPU on HAProxy process, it is built from > HEAD of 1.9 branch. > > One of processes stuck in infinite loop, admin socket is not responsive so > I've got information only from gdb: >

[1.9 HEAD] HAProxy using 100% CPU

2019-05-07 Thread Maciej Zdeb
Hi, I've got another bug with 100% CPU on HAProxy process, it is built from HEAD of 1.9 branch. One of processes stuck in infinite loop, admin socket is not responsive so I've got information only from gdb: 0x00484ab8 in h2_process_mux (h2c=0x2e8ff30) at src/mux_h2.c:2589 2589