Re: [RFC PATCH] BUG/MINOR: systemd: Remove configuration check for reloads

[William Lallemand]

On Tue, Oct 01, 2019 at 01:13:27AM +0200, Tim Düsterhus wrote:
> William,
> 
> Am 01.10.19 um 01:09 schrieb William Lallemand:
> > Found this in the Debian BTS:
> > 
> > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=936032
> > 
> 
> Apparently my Google Foo was insufficient to find that bug. As it's a
> systemd bug and as it's recognized as such by systemd upstream we
> shouldn't change anything within the HAProxy provided unit. I'll just go
> and apply my patch to the unit files on my machines for my own sanity.
> 

Well, it will just break the reload status in systemd, so that's not that
bad :-)

-- 
William Lallemand

Re: [RFC PATCH] BUG/MINOR: systemd: Remove configuration check for reloads

[Tim Düsterhus]

William,

Am 01.10.19 um 01:09 schrieb William Lallemand:
> Found this in the Debian BTS:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=936032
> 

Apparently my Google Foo was insufficient to find that bug. As it's a
systemd bug and as it's recognized as such by systemd upstream we
shouldn't change anything within the HAProxy provided unit. I'll just go
and apply my patch to the unit files on my machines for my own sanity.

Best regards
Tim Düsterhus

Re: [RFC PATCH] BUG/MINOR: systemd: Remove configuration check for reloads

[William Lallemand]

On Tue, Oct 01, 2019 at 12:41:33AM +0200, Tim Duesterhus wrote:
> Willy, William, Vincent,
> 

Hello Tim,

> apparently systemd's behaviour changed regarding reloads. When one of the
> reload commands fails it apparently starts stopping the service.
> 
> At first I thought it was some bad systemd configuration on my end, but
> after seeing HAProxy die on me for the bazillionth time because of a bad
> configuration I started investigating.
> 
> I cannot reproduce the issue on my Ubuntu 16.04 with systemd 229. I can
> reproduce it on a Debian Buster machine with systemd 232.
> 

Looks like a bug to me, they can't break the compatibility like that.

> I'm not fully convinced that this patch:
> 
> - is correct, because I am not sure whether "waitpid" mode is doing
>   something bad. That's why I Cc William.

It's not doing something bad, but it's not convenient, and there is a warning 
on the CLI.

> - is the best solution. It certainly isn't because the `reload` succeeds,
>   while actually it does not. Maybe there's some systemd knob that
>   improves on that behaviour. I Cc Vincent as the Debian maintainer of
>   HAProxy, because that's the thing I run in production.

We must not do that. I don't want to change these reload lines until we have a
synchronous reload command which returns a failure. It's going to be worst if we
remove the -c line, I know it takes time to check the conf, but it's the only
way to return a failure. Only people who knows what they are doing should remove
it, we shouldn't do it by default.

> So basically this email is to bring attention to the issue and to confirm
> it's not just me who is holding systemd wrong (I'm seeing that same issue
> with Debian's nginx package, though). This patch is absolutely RFC only
> and probably should not be merged as is without some discussion. It's
> certainly the longest commit message I ever wrote ...
> 
> Best regards
> Tim Düsterhus
> 

Found this in the Debian BTS:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=936032

Looks like this exact bug.

Regards,

-- 
William Lallemand

[RFC PATCH] BUG/MINOR: systemd: Remove configuration check for reloads

[Tim Duesterhus]

Willy, William, Vincent,

apparently systemd's behaviour changed regarding reloads. When one of the
reload commands fails it apparently starts stopping the service.

At first I thought it was some bad systemd configuration on my end, but
after seeing HAProxy die on me for the bazillionth time because of a bad
configuration I started investigating.

I cannot reproduce the issue on my Ubuntu 16.04 with systemd 229. I can
reproduce it on a Debian Buster machine with systemd 232.

I'm not fully convinced that this patch:

- is correct, because I am not sure whether "waitpid" mode is doing
  something bad. That's why I Cc William.
- is the best solution. It certainly isn't because the `reload` succeeds,
  while actually it does not. Maybe there's some systemd knob that
  improves on that behaviour. I Cc Vincent as the Debian maintainer of
  HAProxy, because that's the thing I run in production.

So basically this email is to bring attention to the issue and to confirm
it's not just me who is holding systemd wrong (I'm seeing that same issue
with Debian's nginx package, though). This patch is absolutely RFC only
and probably should not be merged as is without some discussion. It's
certainly the longest commit message I ever wrote ...

Best regards
Tim Düsterhus

Apply with `git am --scissors` to automatically cut the commit message.
-- >8 --
Subject: [RFC PATCH] BUG/MINOR: systemd: Remove configuration check for reloads

Apparently newer systemd versions kill the service when one of the
`ExecReload=` commands fails instead of just failing the reload.

See the following example with `haproxy.service` being as provided by
the Debian project.

Note the "Active: failed" after the first bad reload. After adjusting
the unit file HAProxy continues running in "waitpid" mode when the config
is bad, systemd does not return an error though. Thus the admin might not
notice that there was an issue.

Good config
===

root@example:~# systemctl start haproxy
root@example:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: 
enabled)
   Active: active (running) since Mon 2019-09-30 22:17:18 UTC; 2s ago
 Docs: man:haproxy(1)
   file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 16227 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS 
(code=exited, status=0/SUCCESS)
 Main PID: 16228 (haproxy)
Tasks: 2 (limit: 2301)
   Memory: 6.8M
   CGroup: /system.slice/haproxy.service
   ├─16228 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p 
/run/haproxy.pid -S /run/haproxy-master.sock
   └─16229 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p 
/run/haproxy.pid -S /run/haproxy-master.sock

Sep 30 22:17:18 example.com systemd[1]: Starting HAProxy Load Balancer...
Sep 30 22:17:18 example.com haproxy[16228]: Proxy http started.
Sep 30 22:17:18 example.com haproxy[16228]: Proxy https started.
Sep 30 22:17:18 example.com haproxy[16228]: [NOTICE] 272/221718 (16228) : New 
worker #1 (16229) forked
Sep 30 22:17:18 example.com systemd[1]: Started HAProxy Load Balancer.

Reload with bad config
==

root@example:~# echo 'foo' >> /etc/haproxy/haproxy.cfg
root@example:~# systemctl reload haproxy
Job for haproxy.service failed because the control process exited with error 
code.
See "systemctl status haproxy.service" and "journalctl -xe" for details.
root@example:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: 
enabled)
   Active: failed (Result: exit-code) since Mon 2019-09-30 22:17:45 UTC; 315ms 
ago
 Docs: man:haproxy(1)
   file:/usr/share/doc/haproxy/configuration.txt.gz
  Process: 16255 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS 
(code=exited, status=1/FAILURE)

Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Control process 
exited, code=exited, status=1/FAILURE
Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Failed with result 
'exit-code'.
Sep 30 22:17:45 example.com systemd[1]: Failed to start HAProxy Load Balancer.
Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Service 
RestartSec=100ms expired, scheduling restart.
Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Scheduled restart job, 
restart counter is at 6.
Sep 30 22:17:45 example.com systemd[1]: Stopped HAProxy Load Balancer.
Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Start request repeated 
too quickly.
Sep 30 22:17:45 example.com systemd[1]: haproxy.service: Failed with result 
'exit-code'.
Sep 30 22:17:45 example.com systemd[1]: Failed to start HAProxy Load Balancer.

Fix config
==

root@example:~# sed -i '/foo/d' /etc/haproxy/haproxy.cfg
root@example:~# systemctl restart haproxy
root@example:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/lib/systemd/system/haproxy.service; enabled

Re: HAProxy bottleneck/tuning

[Emmanuel BILLOT]

Thanks for the answer.

Le lun. 30 sept. 2019 à 05:51, Willy Tarreau  a écrit :

> Hi,
>
> On Sat, Sep 28, 2019 at 07:07:02PM +0200, Emmanuel BILLOT wrote:
> > Hi,
> >
> > We use HAProxy as a LB for many usage, including LB for Squid and user
> > acces on Internet.
> > Users frequently grumble about "Internet speed" (classical...) and we
> want
> > to be sure that it is not a LB issue (msconfig, or system bottleneck ?).
> >
> > How could we find is haproxy is "undersized" in config or if the hosting
> > system (CentOS) is not correctly configured (file desc ? memory ?)
>
> Well, this sounds more like a sysadmin training than an haproxy-specific
> question, but a few things you should have a look at :
>   - is your system swapping ? (it must never)
>   - do you see in your system logs "conntrack table full" ? If so you're
> hitting some misconfigured conntrack limits
>   - do you see in your system logs "TCP: sending SYN cookies" ? If so
> you're likely running haproxy with too low a maxconn setting, resulting
> in connections not being accepted
>   - do you see in your haproxy logs flags "sC" or "SC" while you know
> your servers are working fine ? If so that could indicate a failure to
> allocate a source port or a file descriptor for an outgoing connection
>   - do you see your CPU steadily running above, say, 50% ? If so you cannot
> exclude frequent 100% peaks possibly causing some sub-second delays not
> reported by the system tools.
>   - and if you're running in a VM, you can redo all this above inside the
> hypervisor, and in the guest you should also look at the "st" field in
> vmstat to make sure your CPU is not stolen by other VMs (or goodbye low
> latency), and you can also run "ping" to your VM from an external host
> on the same LAN and make sure the latency never jumps above 1
> millisecond
> on a mostly idle network or 2-3 ms on a loaded network or it can
> indicate that you're have performance issues caused by noisy neighbors
> on your machine.
>   - connection setup timers exhibiting multiple of 3s in haproxy logs
> usually indicate packet drops between haproxy and the servers
>   - client timeout errors during request like "cR" often indicate drops or
> MTU issues between the client and haproxy (sometimes caused by bogus
> virtualized network drivers).
>
> Hoping this helps,
> Willy
>

Do you need redesign of your site at the address https://www.haproxy.com or another original software?

[Igor]

Hello.

I can make a new high-quality fast website in the adaptive layout at the price 
from $300 for your nice project at the address https://www.haproxy.com (HAProxy 
Technologies | The World’s Fastest and Most Widely Used Software Load Balancer).

As well I can do complicated web applications, accounting apps or any other 
software, made and configured specifically for you, and much more, about what 
you can find out on my site https://www.programs.gq/en/ 

With best regards, Igor,
flashscript1...@gmail.com

RE: WebSocket connection upgrade not using tunnel timeout

[Rabie Van Der Merwe]

I seem to have found the cause.
If I specify cache for that backend then the 'timeout tunnel' no longer
works, if I comment out caching then the timeout applies.

Is this expected behaviour?

I'm using the following in my backend:
--
http-request cache-use cache01
http-response cache-store cache01
--

And this is my Cache config
--
cache cache01
total-max-size 256
max-age 60
--

Regards
Rabie

-Original Message-
From: Rabie Van Der Merwe 
Sent: Monday, 30 September 2019 15:53
To: haproxy@formilux.org
Subject: RE: WebSocket connection upgrade not using tunnel timeout

Small correction I'm still on 1.8.20.
I have since managed to do a tcpdump on the HAProxy box for the backend
connection to the web/socket server.
I can see that HAProxy forwards the connection upgrade request, the
connection is upgraded and I can see websocket traffic (viewing dump in
WireShark).

The connection still times out based on the server timeout and not the
tunnel timeout setting.

How can I debug this on the HAProxy side?

R

-Original Message-
From: Rabie Van Der Merwe 
Sent: Friday, 27 September 2019 12:14
To: 'haproxy@formilux.org' 
Subject: WebSocket connection upgrade not using tunnel timeout

Hi,

I'm on 1.8.21 and have a webapp that uses websockets.
The application works, but the app complains very quickly that the
connection to the server has been lost.
On the second line we get "sD--" Which is:
sD  The server did not send nor acknowledge any data for as long as
the "timeout server" setting during the data phase. This is often caused by
too short timeouts on L4 equipments before the server (firewalls,
load-balancers, .), as well as keep-alive sessions maintained between the
client and the server expiring first on haproxy.

I have added a tunnel timeout statement, however I'm unsure if haproxy is
aware of this and if it is why is it still timing out on server timeout? I
have noticed that I can increase the server timeout which would delay the
disconnect, however this is not a solution.

Any suggestions? How can I check to see if haproxy 'sees' the connection
upgrade?

--snip--
Sep 27 11:43:52 localhost haproxy[22504]: 1.2.3.4:50137
[27/Sep/2019:11:43:52.201] mainhttps~ smartcm01/apxblvcm01 0/0/0/7/7 200
1217 - -  9/9/1/2/0 0/0 {|} "GET /nui/images/settings.gif HTTP/1.1"
Sep 27 11:44:51 localhost haproxy[22504]: 1.2.3.4:48390
[27/Sep/2019:11:43:50.389] mainhttps~ smartcm01/apxblvcm01 0/0/1/7/61486
101 606 - - sD-- 3/3/0/0/0 0/0 {|} "GET /rest/events HTTP/1.1"
Sep 27 11:51:25 localhost haproxy[22504]: 1.2.3.4:9
[27/Sep/2019:11:51:25.423] mainhttps~ smartcm01/apxblvcm01 0/0/2/7/9 304
100 - -  8/8/0/1/0 0/0 {|} "GET / HTTP/1.1"
--snip--

Regards
Rabie van der Merwe

RE: WebSocket connection upgrade not using tunnel timeout

[Rabie Van Der Merwe]

Small correction I'm still on 1.8.20.
I have since managed to do a tcpdump on the HAProxy box for the backend
connection to the web/socket server.
I can see that HAProxy forwards the connection upgrade request, the
connection is upgraded and I can see websocket traffic (viewing dump in
WireShark).

The connection still times out based on the server timeout and not the
tunnel timeout setting.

How can I debug this on the HAProxy side?

R

-Original Message-
From: Rabie Van Der Merwe 
Sent: Friday, 27 September 2019 12:14
To: 'haproxy@formilux.org' 
Subject: WebSocket connection upgrade not using tunnel timeout

Hi,

I'm on 1.8.21 and have a webapp that uses websockets.
The application works, but the app complains very quickly that the
connection to the server has been lost.
On the second line we get "sD--" Which is:
sD  The server did not send nor acknowledge any data for as long as
the "timeout server" setting during the data phase. This is often caused
by too short timeouts on L4 equipments before the server (firewalls,
load-balancers, .), as well as keep-alive sessions maintained between the
client and the server expiring first on haproxy.

I have added a tunnel timeout statement, however I'm unsure if haproxy is
aware of this and if it is why is it still timing out on server timeout? I
have noticed that I can increase the server timeout which would delay the
disconnect, however this is not a solution.

Any suggestions? How can I check to see if haproxy 'sees' the connection
upgrade?

--snip--
Sep 27 11:43:52 localhost haproxy[22504]: 1.2.3.4:50137
[27/Sep/2019:11:43:52.201] mainhttps~ smartcm01/apxblvcm01 0/0/0/7/7 200
1217 - -  9/9/1/2/0 0/0 {|} "GET /nui/images/settings.gif HTTP/1.1"
Sep 27 11:44:51 localhost haproxy[22504]: 1.2.3.4:48390
[27/Sep/2019:11:43:50.389] mainhttps~ smartcm01/apxblvcm01 0/0/1/7/61486
101 606 - - sD-- 3/3/0/0/0 0/0 {|} "GET /rest/events HTTP/1.1"
Sep 27 11:51:25 localhost haproxy[22504]: 1.2.3.4:9
[27/Sep/2019:11:51:25.423] mainhttps~ smartcm01/apxblvcm01 0/0/2/7/9 304
100 - -  8/8/0/1/0 0/0 {|} "GET / HTTP/1.1"
--snip--

Regards
Rabie van der Merwe

Re: using hashicorp vault for storing SSL certs

[Pavlos Parissis]

On Κυριακή, 29 Σεπτεμβρίου 2019 7:52:12 Μ.Μ. CEST Илья Шипицин wrote:
> hello,
> 
> is anybody using https://www.vaultproject.io/docs/secrets/pki/index.html
> for storing certs ? (I want to avoid reinventing the wheel here)
> 
> thanks,
> Ilya Shipitcin
> 

I have, but with a different system, but it doesn't matter a lot from HAProxy 
point of view.
Try to avoid the typical mistake where haproxy can't reload anymore as the 
backend for the
Certs is down and as a result the certs aren't available anymore. I know some 
people care so much
about security that they forget about availability.

My 2 cents,
Pavlos

HAProxyConf 2019 - Early Bird Tickets Deadline Extended

[Senad Caus]

Hello,
Due to popular demand we've decided to extend the sale of Early Bird 
tickets for the inaugural HAProxyConf for one more week.


The new deadline is October 7th. After that, you can still purchase 
tickets at the regular price.


The number of attendees is limited, so please make sure to purchase your 
ticket on time. More info about the ticket prices and the agenda can be 
found on the website. https://www.haproxyconf.com/


Also, don’t forget about the preferential rates we have prepared in 
cooperation with three hotels near our venue. They are available until 
October 28th.


Let me know if you have any questions about the agenda, location or the 
conference in general.


Cheers,
Senad Caus 

Re: [PATCH 2/2] BUG/MINOR: lua: Make the arrays in the list of headers 1-indexed

[Thierry Fournier]

> On 30 Sep 2019, at 05:40, Willy Tarreau  wrote:
> 
> Hi guys,
> 
> On Thu, Sep 26, 2019 at 04:35:31PM +0200, Tim Düsterhus wrote:
>> Adis,
>> 
>> Am 26.09.19 um 16:01 schrieb Adis Nezirovic:
>>> While I agree that using zero based array indexing is a mistake (wearing
>>> my Lua hat), I don't think your proposal will make situation any better.
>>> Actually ot might hurt existing Lua code in unforeseen ways.
>> 
>> Yes, I understand that it breaks backwards compatibility. That's why I
>> added the cover letter with the rationale of duplicating the first value
>> instead of making the 0-index a nil.
> 
> To be honest I'm totally ignorant of this code area and it's even not
> very clear to me (even after reading the commit messages) what is the
> extent of this change nor what this headers array corresponds to.
> 
> However what I can say is that when implementing a transition towards
> a final solution by having an intermediate solution (like you're
> proposing by keeping index 0), it's important to have a plan to get rid
> of the intermediate solution at some point. Here, by duplicating entries
> I'm not seeing any way to properly cover this final transition, nor to
> safely avoid breakage of existing applications. Thus it will turn an
> existing problem into two other ones.
> 
> I could instead propose two solutions :
>  - either we knowingly break existing code for 2.1+ by changing the
>array numbering to match Lua standards ; those migrating from 2.0
>to 2.1 or 2.2 will have to modify their code. We can explain them
>how to make portable code by checking the haproxy version from
>within Lua if required;
> 
>  - or we decide to rename this "headers" field on the long term (if
>it's the only one, which is really the part I'm ignoring) and create
>a new one ("hdrs" ?) numbered from 1. We can thus keep the two in
>parallel till 2.2 included so that at least one LTS version covers
>both the old and new API, and in 2.3 we remove headers. If we can
>emit an API warning when "headers" is first met, it's even better.
>I just don't know how we can make one structure field alias another
>one, I can understand that it's not just a matter of setting +1 on
>a pointer, but I guess that we have hooks behind the curtains which
>intercept this "headers" thing.
> 
> There may also be other solutions, but definitely an API change needs
> to pass through something visible so that affected code at some point
> has to be fixed if we want to get rid of a past mistake.


Hi, hard question because everyone has right :-) I vote for a new function
which return an array of header starting at the index 1.

Thierry