Le 21/12/2017 à 08:42, Peter Lindegaard Hansen a écrit :
update:
we've disabled h2 on 1.8, and everything is running as expected again.
haproxy does not degrade performance anymore nor does it segfault.
so it issues seem to be related to the h2
Med venlig hilsen
*Peter Lindegaard Hansen*
/Softwareudvikler / Partner
/
Telefon: +45 96 500 300 | Direkte: 69 14 97 04 | Email:
[email protected] <mailto:[email protected]>
Tiger Media A/S | Gl. Gugvej 17C | 9000 Aalborg | Web: www.tigermedia.dk
<http://www.tigermedia.dk>
For supportspørgsmål kontakt os da på [email protected]
<mailto:[email protected]> eller på tlf. 96 500 300
og din henvendelse vil blive besvaret af første ledige medarbejder.
2017-12-19 11:36 GMT+01:00 Peter Lindegaard Hansen <[email protected]
<mailto:[email protected]>>:
Hi list,
We upgraded from 1.5 to 1.8 recently - then to 1.8.1
Now we're seeing segfaults and slowdowns with haproxy
Repeating:
Dec 19 11:14:26 haproxy02 kernel: [122635.295196] haproxy[29582]:
segfault at 55d5152279b2 ip 00007f9c2dcc5a28 sp 00007fff07caf4b8
error 6 in libc-2.23.so <http://libc-2.23.so>[7f9c2dc26000+1c0000]
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:26 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:26 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:26 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:26 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 kernel: [122636.578738] haproxy[31479]:
segfault at 56409a8c1de2 ip 00007fa5fa349a28 sp 00007ffe66f4f688
error 6 in libc-2.23.so <http://libc-2.23.so>[7fa5fa2aa000+1c0000]
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:27 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:27 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:27 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:28 haproxy02 systemd[1]: Started HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 kernel: [122637.569863] haproxy[31487]:
segfault at 55cb4bd59857 ip 00007f71e678aa28 sp 00007fffb94427b8
error 6 in libc-2.23.so <http://libc-2.23.so>[7f71e66eb000+1c0000]
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Main process
exited, code=exited, status=139/n/a
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Unit entered
failed state.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Failed with
result 'exit-code'.
Dec 19 11:14:28 haproxy02 systemd[1]: haproxy.service: Service
hold-off time over, scheduling restart.
Dec 19 11:14:28 haproxy02 systemd[1]: Stopped HAProxy Load Balancer.
Dec 19 11:14:28 haproxy02 systemd[1]: Starting HAProxy Load Balancer...
Dec 19 11:14:29 haproxy02 systemd[1]: Started HAProxy Load Balancer.
At same time in haproxy.log
(lots of ssl handshake failures...) then
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579)
: Current worker 29582 left with exit code 139
Dec 19 11:14:26 haproxy02 haproxy[29579]: [ALERT] 352/090058 (29579)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:26 haproxy02 haproxy[29579]: [WARNING] 352/090058
(29579) : All workers are left. Leaving... (139)
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476)
: Current worker 31479 left with exit code 139
Dec 19 11:14:27 haproxy02 haproxy[31476]: [ALERT] 352/111426 (31476)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:27 haproxy02 haproxy[31476]: [WARNING] 352/111426
(31476) : All workers are left. Leaving... (139)
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485)
: Current worker 31487 left with exit code 139
Dec 19 11:14:28 haproxy02 haproxy[31485]: [ALERT] 352/111428 (31485)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:28 haproxy02 haproxy[31485]: [WARNING] 352/111428
(31485) : All workers are left. Leaving... (139)
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493)
: Current worker 31496 left with exit code 139
Dec 19 11:14:29 haproxy02 haproxy[31493]: [ALERT] 352/111429 (31493)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:29 haproxy02 haproxy[31493]: [WARNING] 352/111429
(31493) : All workers are left. Leaving... (139)
Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503)
: Current worker 31505 left with exit code 139
Dec 19 11:14:30 haproxy02 haproxy[31503]: [ALERT] 352/111429 (31503)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:30 haproxy02 haproxy[31503]: [WARNING] 352/111429
(31503) : All workers are left. Leaving... (139)
Dec 19 11:14:30 haproxy02 haproxy[31511]: [ALERT] 352/111430 (31511)
: Current worker 31515 left with exit code 139
Dec 19 11:14:30 haproxy02 haproxy[31511]: [ALERT] 352/111430 (31511)
: exit-on-failure: killing every workers with SIGTERM
Dec 19 11:14:30 haproxy02 haproxy[31511]: [WARNING] 352/111430
(31511) : All workers are left. Leaving... (139)
Until haproxy does not respond to requests :-(
I dont know if it is related, but we do see huge slowdows running
1.8 (with h2 enabled)
After say 10-20 hours haproxy will degrade to serving requests at
4-500kb/s.. when we do a restart of the haproxy service, it will go
back to regular full-line speeds.
We're running ubuntu servers and using Vincent Bernat's PPA.
I can post conf if needed.
Any help appreciated.
Is it related to h2 or 1.8?
We've been running 1.5.9 (ubuntu repos default) for years on the
same requests with no problem.
Thanks
Med venlig hilsen
*Peter Lindegaard Hansen*
/Softwareudvikler / Partner
/
Telefon: +45 96 500 300 | Direkte: 69 14 97 04 | Email:
[email protected] <mailto:[email protected]>
Tiger Media A/S | Gl. Gugvej 17C | 9000 Aalborg | Web:
www.tigermedia.dk <http://www.tigermedia.dk>
For supportspørgsmål kontakt os da på [email protected]
<mailto:[email protected]> eller på tlf. 96 500 300
og din henvendelse vil blive besvaret af første ledige medarbejder.
Hi,
There have been several bugfixes on the H2 part since the 1.8.1. Could
you try the latest 1.8 source snapshot[1] please ?
Then, the segfault part may be unrelated. This is really strange. How
and when does it happens ? after a reload or when the traffic slow down
? And what is your CPU/RAM usage when it happens ?
If possible, share your configuration, it could help. And finally, if
you compile HAProxy by hand, compile it with debug info and enable
coredumps. Then, send us the full backtrace of the segfault. It will be
really useful to know what happens.
[1]
http://www.haproxy.org/download/1.8/src/snapshot/haproxy-ss-LATEST.tar.gz
--
Christopher Faulet
