Re: 1.8.1 Segfault + slowdown

2017-12-21 Thread Christopher Faulet 

Le 21/12/2017 à 08:42, Peter Lindegaard Hansen a écrit :

update:

we've disabled h2 on 1.8, and everything is running as expected again. 
haproxy does not degrade performance anymore nor does it segfault.

so it issues seem to be related to the h2



Med venlig hilsen


*Peter Lindegaard Hansen*
/Softwareudvikler / Partner
/

Telefon: +45 96 500 300 | Direkte: 69 14 97 04 | Email: 
p...@tigermedia.dk 
Tiger Media A/S | Gl. Gugvej 17C | 9000 Aalborg | Web: www.tigermedia.dk 



For supportspørgsmål kontakt os da på supp...@tigermedia.dk 
 eller på tlf. 96 500 300

og din henvendelse vil blive besvaret af første ledige medarbejder.


2017-12-19 11:36 GMT+01:00 Peter Lindegaard Hansen >:


Hi list,

We upgraded from 1.5 to 1.8 recently - then to 1.8.1

Now we're seeing segfaults and slowdowns with haproxy

Repeating:
Dec 19 11:14:26 haproxy02 kernel: [122635.295196] haproxy[29582]:
segfault at 55d5152279b2 ip 7f9c2dcc5a28 sp 7fff07caf4b8
error 6 in libc-2.23.so [7f9c2dc26000+1c]
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:26 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:26 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:26 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 kernel: [122636.578738] haproxy[31479]:
segfault at 56409a8c1de2 ip 7fa5fa349a28 sp 7ffe66f4f688
error 6 in libc-2.23.so [7fa5fa2aa000+1c]
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:27 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:28 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 kernel: [122637.569863] haproxy[31487]:
segfault at 55cb4bd59857 ip 7f71e678aa28 sp 7fffb94427b8
error 6 in libc-2.23.so [7f71e66eb000+1c]
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:28 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:29 haproxy02 systemd[1]: Started HAProxy Load Balancer.


At same time in haproxy.log

(lots of ssl handshake failures...) then
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579)
: Current worker 29582 left with exit code 139
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:26 haproxy02 haproxy[29579]: [WARNING] 352/090058
(29579) : All workers are left. Leaving... (139)
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476)
: Current worker 31479 left with exit code 139
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:27 haproxy02 haproxy[31476]: [WARNING] 352/111426
(31476) : All workers are left. Leaving... (139)
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485)
: Current worker 31487 left with exit code 139
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:28 haproxy02 haproxy[31485]: [WARNING] 352/111428
(31485) : All workers are left. Leaving... (139)
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493)
: Current worker 31496 left with exit code 139
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:29 haproxy02

Re: 1.8.1 Segfault + slowdown

2017-12-20 Thread Peter Lindegaard Hansen 
update:

we've disabled h2 on 1.8, and everything is running as expected again.
haproxy does not degrade performance anymore nor does it segfault.
so it issues seem to be related to the h2



Med venlig hilsen


*Peter Lindegaard Hansen*

*Softwareudvikler / Partner*

Telefon: +45 96 500 300 | Direkte: 69 14 97 04 | Email: p...@tigermedia.dk
Tiger Media A/S | Gl. Gugvej 17C | 9000 Aalborg | Web: www.tigermedia.dk

For supportspørgsmål kontakt os da på supp...@tigermedia.dk eller på tlf.
96 500 300
og din henvendelse vil blive besvaret af første ledige medarbejder.

2017-12-19 11:36 GMT+01:00 Peter Lindegaard Hansen :

> Hi list,
>
> We upgraded from 1.5 to 1.8 recently - then to 1.8.1
>
> Now we're seeing segfaults and slowdowns with haproxy
>
> Repeating:
> Dec 19 11:14:26 haproxy02 kernel: [122635.295196] haproxy[29582]: segfault
> at 55d5152279b2 ip 7f9c2dcc5a28 sp 7fff07caf4b8 error 6 in
> libc-2.23.so[7f9c2dc26000+1c]
> Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Main process
> exited, code=exited, status=139/n/a
> Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Unit entered failed
> state.
> Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Failed with result
> 'exit-code'.
> Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Service hold-off
> time over, scheduling restart.
> Dec 19 11:14:26 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
> Dec 19 11:14:26 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
> Dec 19 11:14:26 haproxy02 systemd[1]: Started HAProxy Load Balancer.
> Dec 19 11:14:27 haproxy02 kernel: [122636.578738] haproxy[31479]: segfault
> at 56409a8c1de2 ip 7fa5fa349a28 sp 7ffe66f4f688 error 6 in
> libc-2.23.so[7fa5fa2aa000+1c]
> Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Main process
> exited, code=exited, status=139/n/a
> Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Unit entered failed
> state.
> Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Failed with result
> 'exit-code'.
> Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Service hold-off
> time over, scheduling restart.
> Dec 19 11:14:27 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
> Dec 19 11:14:27 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
> Dec 19 11:14:28 haproxy02 systemd[1]: Started HAProxy Load Balancer.
> Dec 19 11:14:28 haproxy02 kernel: [122637.569863] haproxy[31487]: segfault
> at 55cb4bd59857 ip 7f71e678aa28 sp 7fffb94427b8 error 6 in
> libc-2.23.so[7f71e66eb000+1c]
> Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Main process
> exited, code=exited, status=139/n/a
> Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Unit entered failed
> state.
> Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Failed with result
> 'exit-code'.
> Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Service hold-off
> time over, scheduling restart.
> Dec 19 11:14:28 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
> Dec 19 11:14:28 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
> Dec 19 11:14:29 haproxy02 systemd[1]: Started HAProxy Load Balancer.
>
>
> At same time in haproxy.log
>
> (lots of ssl handshake failures...) then
> Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579) :
> Current worker 29582 left with exit code 139
> Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579) :
> exit-on-failure: killing every workers with SIGTERM
> Dec 19 11:14:26 haproxy02 haproxy[29579]: [WARNING] 352/090058 (29579) :
> All workers are left. Leaving... (139)
> Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476) :
> Current worker 31479 left with exit code 139
> Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476) :
> exit-on-failure: killing every workers with SIGTERM
> Dec 19 11:14:27 haproxy02 haproxy[31476]: [WARNING] 352/111426 (31476) :
> All workers are left. Leaving... (139)
> Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485) :
> Current worker 31487 left with exit code 139
> Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485) :
> exit-on-failure: killing every workers with SIGTERM
> Dec 19 11:14:28 haproxy02 haproxy[31485]: [WARNING] 352/111428 (31485) :
> All workers are left. Leaving... (139)
> Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493) :
> Current worker 31496 left with exit code 139
> Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493) :
> exit-on-failure: killing every workers with SIGTERM
> Dec 19 11:14:29 haproxy02 haproxy[31493]: [WARNING] 352/111429 (31493) :
> All workers are left. Leaving... (139)
> Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503) :
> Current worker 31505 left with exit code 139
> Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503) :
> exit-on-failure: killing every workers with SIGTERM
> Dec 19 11:14:30 haproxy02 haproxy[31503]: [WARNING]

1.8.1 Segfault + slowdown

2017-12-19 Thread Peter Lindegaard Hansen 
Hi list,

We upgraded from 1.5 to 1.8 recently - then to 1.8.1

Now we're seeing segfaults and slowdowns with haproxy

Repeating:
Dec 19 11:14:26 haproxy02 kernel: [122635.295196] haproxy[29582]: segfault
at 55d5152279b2 ip 7f9c2dcc5a28 sp 7fff07caf4b8 error 6 in
libc-2.23.so[7f9c2dc26000+1c]
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Main process exited,
code=exited, status=139/n/a
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Unit entered failed
state.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Failed with result
'exit-code'.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Service hold-off
time over, scheduling restart.
Dec 19 11:14:26 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:26 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:26 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 kernel: [122636.578738] haproxy[31479]: segfault
at 56409a8c1de2 ip 7fa5fa349a28 sp 7ffe66f4f688 error 6 in
libc-2.23.so[7fa5fa2aa000+1c]
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Main process exited,
code=exited, status=139/n/a
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Unit entered failed
state.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Failed with result
'exit-code'.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Service hold-off
time over, scheduling restart.
Dec 19 11:14:27 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:28 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 kernel: [122637.569863] haproxy[31487]: segfault
at 55cb4bd59857 ip 7f71e678aa28 sp 7fffb94427b8 error 6 in
libc-2.23.so[7f71e66eb000+1c]
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Main process exited,
code=exited, status=139/n/a
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Unit entered failed
state.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Failed with result
'exit-code'.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Service hold-off
time over, scheduling restart.
Dec 19 11:14:28 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:29 haproxy02 systemd[1]: Started HAProxy Load Balancer.


At same time in haproxy.log

(lots of ssl handshake failures...) then
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579) :
Current worker 29582 left with exit code 139
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:26 haproxy02 haproxy[29579]: [WARNING] 352/090058 (29579) :
All workers are left. Leaving... (139)
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476) :
Current worker 31479 left with exit code 139
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:27 haproxy02 haproxy[31476]: [WARNING] 352/111426 (31476) :
All workers are left. Leaving... (139)
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485) :
Current worker 31487 left with exit code 139
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:28 haproxy02 haproxy[31485]: [WARNING] 352/111428 (31485) :
All workers are left. Leaving... (139)
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493) :
Current worker 31496 left with exit code 139
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:29 haproxy02 haproxy[31493]: [WARNING] 352/111429 (31493) :
All workers are left. Leaving... (139)
Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503) :
Current worker 31505 left with exit code 139
Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:30 haproxy02 haproxy[31503]: [WARNING] 352/111429 (31503) :
All workers are left. Leaving... (139)
Dec 19 11:14:30 haproxy02 haproxy[31511]: [ALERT] 352/111430 (31511) :
Current worker 31515 left with exit code 139
Dec 19 11:14:30 haproxy02 haproxy[31511]: [ALERT] 352/111430 (31511) :
exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:30 haproxy02 haproxy[31511]: [WARNING] 352/111430 (31511) :
All workers are left. Leaving... (139)



Until haproxy does not respond to requests :-(


I dont know if it is related, but we do see huge slowdows running 1.8 (with
h2 enabled)

After say 10-20 hours haproxy will degrade to serving requests at
4-500kb/s.. when we do a restart of the haproxy service, it will go back to
regular full-line speeds.

We're running ubuntu servers and using Vincent Bernat's PPA.

I can post conf if needed.

Any