Re: Health check logging differences between 1.9 and 2.0

2020-10-29 Thread Veiko Kukk
On 2020-10-24 10:32, Willy Tarreau wrote: That sounds strange, I don't like this. This sounds like an uninitialized variable. Did you observe that the facility used is stable inside a backend for example, or does it seem to be affected by other activity ? After investigation, it appears that

Re: Health check logging differences between 1.9 and 2.0

2020-10-28 Thread Veiko Kukk
On 2020-10-28 13:11, Veiko Kukk wrote: Another difference between 1.9 and 2.0 here is that 2.0 is compiled with systemd support and executed using -Ws and Type=notify instead of 1.9 -W and Type=forking. With the exactlty same HAproxy 2.0 (compiled with systemd support), I've changed

Re: Health check logging differences between 1.9 and 2.0

2020-10-28 Thread Veiko Kukk
On 2020-10-26 13:57, Christopher Faulet wrote: health-check log messages are emitted in the same way in 1.9 and 2.0. And at first glance, the code responsible to set the syslog priority is the same too. So there is probably something we missed. Could you confirm you still have the same issue

Re: Health check logging differences between 1.9 and 2.0

2020-10-23 Thread Veiko Kukk
On 2020-10-22 10:38, Veiko Kukk wrote: Indeed, in HAproxy 2.0, 'option log-health-checks' messages are emitted usig syslog facility 'daemon' and not the facility configured with global configuration keyword 'log'. In 1.9, health check logs were emitted as defined by 'log' facility value. I

Re: Health check logging differences between 1.9 and 2.0

2020-10-22 Thread Veiko Kukk
On 2020-10-20 11:56, Veiko Kukk wrote: I've upgraded some servers from 1.9.15 to 2.0.18. Log config is very simple. ... Without any changes to rsyslog configuration/filters, health checks are now filtered to /var/log/messages and not into specified haproxy log files as was before. Answering

Health check logging differences between 1.9 and 2.0

2020-10-20 Thread Veiko Kukk
Hi I've upgraded some servers from 1.9.15 to 2.0.18. Log config is very simple. global log /dev/log local0 defaults log global option httplog option log-health-checks Without any changes to rsyslog configuration/filters, health checks are now filtered to /var/log/messages and not into

Re: 2.0.14 PCRE2 JIT compilation failed

2020-04-24 Thread Veiko Kukk
On 2020-04-24 12:47, Veiko Kukk wrote: HAproxy 2.0.14 on CentOS 7.7.1908 with PCRE2 JIT enabled (USE_PCRE2=1 USE_PCRE2_JIT=1). When starting it with configuration that has following ACL regex line, it fails: acl path_is_foo path_reg ^\/video\/[a-zA-Z0-9_-]{43}\/[a-z0-9]{8}\/videos\/ Error

2.0.14 PCRE2 JIT compilation failed

2020-04-24 Thread Veiko Kukk
Hi Since 1.9 support ends soon, I'm trying to start using 2.0 series. HAproxy 2.0.14 on CentOS 7.7.1908 with PCRE2 JIT enabled (USE_PCRE2=1 USE_PCRE2_JIT=1). When starting it with configuration that has following ACL regex line, it fails: acl path_is_foo path_reg

Understanding resolvers usage

2020-03-20 Thread Veiko Kukk
Hi I'd like to have better understanding how server-template and resolvers work together. HAproxy 1.9.14. Relevant sections from config: resolvers dns accepted_payload_size 1232 parse-resolv-conf hold valid 90s resolve_retries 3 timeout resolve 1s timeout retry 1s

Re: 1.9 external health checks fail suddenly

2019-09-23 Thread Veiko Kukk
On 2019-08-28 11:13, Veiko Kukk wrote: Applied it to 1.9.10, after ~ 12h it ran into spinlock using 400% cpu (4 threads configured). Not sure if this is related to patch or is some new bug in 1.9.10. I've now replaced running instance with 1.9.10 without external check patch to see

Re: 1.9 external health checks fail suddenly

2019-08-28 Thread Veiko Kukk
On 2019-07-11 08:35, Willy Tarreau wrote: against your version. Normally it should work for 1.9 to 2.1. Applied it to 1.9.10, after ~ 12h it ran into spinlock using 400% cpu (4 threads configured). Not sure if this is related to patch or is some new bug in 1.9.10. I've now replaced running

Re: 1.9 external health checks fail suddenly

2019-07-10 Thread Veiko Kukk
On 2019-07-09 13:59, Lukas Tribus wrote: How are you currently working around this issue? Did you disable external checks? I'd assume failing checks have negative impact on production systems also. Since this has happened so far only 3 times during 2 months, we've just reloaded HAproxy when

Re: 1.9 external health checks fail suddenly

2019-07-10 Thread Veiko Kukk
On 2019-07-09 14:29, Willy Tarreau wrote: I didn't have a patch but just did it. It was only compile-tested, please verify that it works as expected on a non-sensitive machine first! Hi, Willy Against what version should I run this patch? Veiko

Re: 1.9 external health checks fail suddenly

2019-07-09 Thread Veiko Kukk
On 2019-07-08 16:06, Lukas Tribus wrote: The bug you may be affected by is: https://github.com/haproxy/haproxy/issues/141 Can you check what happens with: nbthread 1 I'm afraid I can't because those are production systems that won't be able to service with single thread, they have relatively

Re: 1.9 external health checks fail suddenly

2019-07-01 Thread Veiko Kukk
On 2019-07-01 10:11, Veiko Kukk wrote: Hi Sometimes (infrequently) all external checks hang and time out: * Has happened with versions 1.9.4 and 1.9.8 on multiple servers with nbproc 1 and nbthread set to (4-12) depending on server. * Happens infrequently, last one happened after 10 days

1.9 external health checks fail suddenly

2019-07-01 Thread Veiko Kukk
Hi Sometimes (infrequently) all external checks hang and time out: * Has happened with versions 1.9.4 and 1.9.8 on multiple servers with nbproc 1 and nbthread set to (4-12) depending on server. * Happens infrequently, last one happened after 10 days of uptime. * External checks are written in

Re: [PATCH v2 1/2] MINOR: systemd: Use the variables from /etc/default/haproxy

2019-05-08 Thread Veiko Kukk
On 2019-05-06 11:00, Tim Duesterhus wrote: From: Apollon Oikonomopoulos This will allow seamless upgrades from the sysvinit system while respecting any changes the users may have made. It will also make local configuration easier than overriding the systemd unit file. Note by Tim: This

Re: Early connection close, incomplete transfers

2019-02-20 Thread Veiko Kukk
On 2019-02-19 06:47, Willy Tarreau wrote: This is interesting. As you observed in the trace you sent me, the lighttpd server closes just after sending the response headers. This indeed matches the "SD" log that aproxy emits. If it doesn't happen in TCP mode nor with Nginx, it means that

Re: Early connection close, incomplete transfers

2019-02-14 Thread Veiko Kukk
On 2019-02-14 18:29, Aleksandar Lazic wrote: Replaced HAproxy with Nginx for testing and with Nginx, not a single connection was interrupted, did millions of requests. In 1.9.4 are a lot of fixed added. please can you try your tests with 1.9.4, thanks. Already did before writing my previous

Re: Early connection close, incomplete transfers

2019-02-14 Thread Veiko Kukk
On 2019-02-01 13:30, Veiko Kukk wrote: On 2019-02-01 12:34, Aleksandar Lazic wrote: Do you have any errors in lighthttpds log? Yes, it has error messages about not being enable to write to socket. Unrecoverable error writing to socket! errno 32, retries 12, ppoll return 1, send return

Re: Early connection close, incomplete transfers

2019-02-04 Thread Veiko Kukk
On 2019-02-01 17:02, Willy Tarreau wrote: Hi Veiko, Are you certain that 1.9 and 1.7 have the same issue ? I mean, you could be observing two different cases looking similar. If you're sure it's the same issue, it could rule out a number of parts that differ quite a lot between the two (idle

Re: Early connection close, incomplete transfers

2019-02-01 Thread Veiko Kukk
On 2019-02-01 12:34, Aleksandar Lazic wrote: Do you have any errors in lighthttpds log? Yes, it has error messages about not being enable to write to socket. Unrecoverable error writing to socket! errno 32, retries 12, ppoll return 1, send return -1 ERROR: Couldn't write header data to

Re: Early connection close, incomplete transfers

2019-02-01 Thread Veiko Kukk
On 2019-01-31 12:57, Aleksandar Lazic wrote: Willy have found some issues which are added in the code of 2.0 tree. Do you have a chance to test this branch or do you want to wait for the next 1.9 release? I tested stable 1.9.3 and 1.9 preview version Willy gave link here

Early connection close, incomplete transfers

2019-01-31 Thread Veiko Kukk
HAproxy 1.9.3, but happens also with 1.7.10, 1.7.11. Connections are getting closed during data transfer phase at random sizes on backend. Sometimes just as little as 420 bytes get transferred, but usually more is transferred before sudden end of connection. HAproxy logs have connection

Re: HA Proxy Load Balancer

2018-12-21 Thread Veiko Kukk
On 2018-12-20 20:41, Lance Melancon wrote: Thanks for the info. Unfortunately I am not a programmer by a long shot and syntax is a big problem for me. I tried a few things but no luck and I can't find any examples of a redirect. So do I need both the backend and acl statements? I'm simply trying

Re: 1.7.11 with gzip compression serves incomplete files

2018-12-06 Thread Veiko Kukk
Hi, Willy On 2018-12-06 04:43, Willy Tarreau wrote: In the mean time it would be useful to see if adding "option http-pretend-keepalive" helps. This way we'll know if it's the server closing first or haproxy closing first which triggers this. And if it turns out that it fixes the issue for you,

Re: 1.7.11 with gzip compression serves incomplete files

2018-12-05 Thread Veiko Kukk
On 2018-11-30 09:40, Christopher Faulet wrote: Now, I'm still puzzled with this issue. Because I can't reproduce it for now. And it is even more strange because when the compression is enabled and used on a response, it cannot be switched in TUNNEL mode. So I don't really understand how the

1.7.11 with gzip compression serves incomplete files

2018-11-26 Thread Veiko Kukk
Hi! There is not much to add, just that it's been broken before in 1.7.9 and is again broken in 1.7.11. Works with 1.7.10. When applying patch provided here https://www.mail-archive.com/haproxy@formilux.org/msg27155.html 1.7.11 also works. Testing is really simple, just configure haproxy

Understanding certain balance configuration

2018-07-30 Thread Veiko Kukk
Hi, I'm trying to understand how balance url_param hash-type consistent should work. Haproxy 1.7.11. Lets say, we have a config of two haproxy instances that balance content between local and remote (sibling). server0 (10.0.0.1) would have config section like this: backend load_balancer

Re: force-persist and use_server combined

2018-07-30 Thread Veiko Kukk
On 07/25/2018 03:05 PM, Veiko Kukk wrote: The idea here is that HAproxy statistics page, some other backend statistics and also some remote health checks running against path under /dl/ would always reach only local_http_frontend, never go anywhere else even when local really is down, not just

force-persist and use_server combined

2018-07-25 Thread Veiko Kukk
Hi, I'd like to understand if I've made a mistake in configuration or there might be a bug in HAproxy 1.7.11. defaults section has "option redispatch". backend load_balancer mode http option httplog option httpchk HEAD /load_balance_health HTTP/1.1\r\nHost:\ foo.bar balance url_param

Re: Truly seamless reloads

2018-06-01 Thread Veiko Kukk
On 31/05/18 23:15, William Lallemand wrote: Sorry but unfortunately we are not backporting features in stable branches, those are only meant for maintenance. People who want to use the seamless reload should migrate to HAProxy 1.8, the stable team won't support this feature in previous

Re: Truly seamless reloads

2018-04-30 Thread Veiko Kukk
On 26/04/18 17:11, Veiko Kukk wrote: Hi, According to https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ : "The patchset has already been merged into the HAProxy 1.8 development branch and will soon be backported to HAProxy Enterprise Edition

Truly seamless reloads

2018-04-26 Thread Veiko Kukk
Hi, According to https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ : "The patchset has already been merged into the HAProxy 1.8 development branch and will soon be backported to HAProxy Enterprise Edition 1.7r1 and possibly 1.6r2." Has it been backported to

Re: 1.7.10 and 1.6.14 always compress response

2018-04-10 Thread Veiko Kukk
On 04/10/2018 03:51 PM, William Lallemand wrote: On Tue, Apr 10, 2018 at 03:43:12PM +0300, Veiko Kukk wrote: Hi, Hi, This happens even when either compression algo nor compression type are specified in haproxy configuration file. If you didn't specify any compression keyword

1.7.10 and 1.6.14 always compress response

2018-04-10 Thread Veiko Kukk
Hi, Lets run simple query against host (real hostnames replaced). curl https://testhost01.tld -o /dev/null -vvv Request headers: > GET / HTTP/1.1 > Host: testhost01.tld > User-Agent: curl/7.58.0 > Accept: */* Response headers: < HTTP/1.1 200 OK < Date: Tue, 10 Apr 2018 12:23:44 GMT <

Re: Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk
Hi Lukas, On 11/03/2017 02:53 PM, Lukas Tribus wrote: # service haproxy reload [ALERT] 306/110738 (29225) : sendmsg logger #1 failed: Resource temporarily unavailable (errno=11) Well the destination logging socket is unavailable. I don't think there is a lot to do here on the haproxy side,

Re: Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk
On 11/03/2017 01:21 PM, Veiko Kukk wrote: Hi, I noticed, while trying to reproduce conditions for another bug about processes never closing after restart, that sometimes reload causes logging errors displayed. Should read here "never closing after *reload*". Veiko

Logging errors during reload of haproxy

2017-11-03 Thread Veiko Kukk
Hi, I noticed, while trying to reproduce conditions for another bug about processes never closing after restart, that sometimes reload causes logging errors displayed. Following config section might be relevant: global log /dev/log local0 nbproc 3 defaults log /dev/log local0

Re: Possible regression in 1.6.12

2017-06-16 Thread Veiko Kukk
Hi, Willy On 16/06/17 12:15, Willy Tarreau wrote: So I have more info on this now. Veiko, first, I'm assuming that your config was using "resolvers dns_resolvers" on the "server" line, otherwise resolvers are not used. My real world configs use resolvers, but timeouts happen even when

Re: Possible regression in 1.6.12

2017-06-15 Thread Veiko Kukk
On 14/06/17 17:37, Willy Tarreau wrote: Could you try to revert the attached patch which was backported to 1.6 to fix an issue where nbproc and resolvers were incompatible ? To do that, please use "patch -Rp1 < foo.patch". I have applied the patch. Now HAproxy working as in 1.6.11 version, no

Possible regression in 1.6.12

2017-06-14 Thread Veiko Kukk
Possible regression in 1.6.12 I might have discovered a haproxy bug. It occurs when all of the following configuration conditions are satisfied: * haproxy version 1.6.12 * multiple processes * resolvers section with more than one server configured (not even used anywhere) * haproxy is either

Re: Multiple url parameter based session limiting

2016-10-05 Thread Veiko Kukk
On 04/10/16 18:21, Veiko Kukk wrote: Lets say, we have URL http://domain.tld?foo=abc=def. I'd like to have current session limiting with sticky tables when both foo and bar values match, but I'm not sure how to achieve this (in most optimal way). I found similar post from 2013. Not exactly

Multiple url parameter based session limiting

2016-10-04 Thread Veiko Kukk
Hi, Lets say, we have URL http://domain.tld?foo=abc=def. I'd like to have current session limiting with sticky tables when both foo and bar values match, but I'm not sure how to achieve this (in most optimal way). Sticky tables are somewhat hard to understand for me. stick-table type string

Re: Haproxy 1.6.9 failed to compile regex

2016-09-07 Thread Veiko Kukk
On 07/09/16 14:37, Veiko Kukk wrote: I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed by haproxy 1.6.9. Any ideas, why? Another strange issue is that 1.6.9 shows: Running on OpenSSL version : OpenSSL 1.0.0-fips 29 Mar 2010 System does have openssl 1.0.1e-48.el6_8.1

Haproxy 1.6.9 failed to compile regex

2016-09-07 Thread Veiko Kukk
Hi, I tried to upgrade from 1.6.8 to 1.6.9, but found strange errors printed by haproxy 1.6.9. Any ideas, why? [ALERT] 250/112901 (12026) : parsing [/etc/haproxy/haproxy.cfg:57] : 'reqirep' : regular expression '^([^ :]*) /(.*)' : failed to compile regex '^([^ :]*) /(.*)' (error=unknown or

Re: 100% cpu , epoll_wait()

2016-05-23 Thread Veiko Kukk
On 18/05/16 15:42, Willy Tarreau wrote: Hi Sebastian, On Thu, May 12, 2016 at 09:58:22AM +0200, Sebastian Heid wrote: Hi Lukas, starting from around 200mbit/s in, haproxy processes (nbproc 6) are hitting 100% cpu regularly (noticed up to 3 processes at the same time with 100%), but recover

Re: 100% cpu , epoll_wait()

2016-04-20 Thread Veiko Kukk
On 20/04/16 11:43, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 09:53:36PM +0300, Veiko Kukk wrote: On 19/04/16 18:52, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 04:15:08PM +0200, Willy Tarreau wrote: OK in fact it's different. Above we have a busy polling loop, which may very be caused

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Veiko Kukk
On 19/04/16 18:52, Willy Tarreau wrote: On Tue, Apr 19, 2016 at 04:15:08PM +0200, Willy Tarreau wrote: OK in fact it's different. Above we have a busy polling loop, which may very be caused by the buffer space miscalculation bug and which results in a process not completing its job until a

Re: 100% cpu , epoll_wait()

2016-04-19 Thread Veiko Kukk
On 16/04/16 01:53, Jim Freeman wrote: I'm suspecting that a connection to the stats port goes wonky with a '-sf' reload, but I'll have to wait for it to re-appear to poke further. I'll look first for a stats port connection handled by the pegged process, then use 'tcpkill' to kill just that

Choosing backend based on constant

2015-04-30 Thread Veiko Kukk
Hi everybody I'd like to simplify my haproxy configuration management by using almost identical configurations for different groups of haproxy installations that use different backends based on string comparision. The only difference in haproxy configuration files of different groups would be

Re: Choosing backend based on constant

2015-04-30 Thread Veiko Kukk
30, 2015 at 11:49 AM, Veiko Kukk vk...@xvidservices.com wrote: Hi everybody I'd like to simplify my haproxy configuration management by using almost identical configurations for different groups of haproxy installations that use different backends based on string comparision. The only difference