stable-bot: Bugfixes waiting for a release 2.2 (8), 2.0 (4)

[stable-bot]

Hi,

This is a friendly bot that watches fixes pending for the next haproxy-stable 
release!  One such e-mail is sent periodically once patches are waiting in the 
last maintenance branch, and an ideal release date is computed based on the 
severity of these fixes and their merge date.  Responses to this mail must be 
sent to the mailing list.


Last release 2.2.11 was issued on 2021-03-18.  There are currently 8 
patches in the queue cut down this way:
- 1 BUG, first one merged on 2021-03-23
- 6 MEDIUM, first one merged on 2021-03-23
- 1 MINOR, first one merged on 2021-03-24

Thus the computed ideal release date for 2.2.12 would be 2021-04-20, which is 
in three weeks or less.

Last release 2.0.21 was issued on 2021-03-18.  There are currently 4 
patches in the queue cut down this way:
- 1 BUG, first one merged on 2021-03-24
- 3 MEDIUM, first one merged on 2021-03-24

Thus the computed ideal release date for 2.0.22 would be 2021-05-19, which is 
in seven weeks or less.

The current list of patches in the queue is:
 - 2.0, 2.2  - BUG : mworker/cli: do not use the unix_bind 
prefix for the master CLI socket
 - 2.2   - MEDIUM  : fd: do not wait on FD removal in 
fd_delete()
 - 2.0, 2.2  - MEDIUM  : lua: Always init the lua stack before 
referencing the context
 - 2.0, 2.2  - MEDIUM  : freq_ctr/threads: use the 
global_now_ms variable
 - 2.2   - MEDIUM  : debug/lua: Don't dump the lua stack if 
not dumpable
 - 2.2   - MEDIUM  : fd: Take the fd_mig_lock when closing 
if no DWCAS is available.
 - 2.0, 2.2  - MEDIUM  : debug/lua: Use internal hlua function 
to dump the lua traceback
 - 2.2   - MINOR   : ssl: Prevent disk access when using 
"add ssl crt-list"

-- 
The haproxy stable-bot is freely provided by HAProxy Technologies to help 
improve the quality of each HAProxy release.  If you have any issue with these 
emails or if you want to suggest some improvements, please post them on the 
list so that the solutions suiting the most users can be found.

Re: Table sticky counters decrementation problem

[Vincent Bernat]

 ❦ 30 mars 2021 11:21 +02, Thomas SIMON:

> And I confirm you than when rolling back with source compilation and
> 2.3.7 version (can't do this with repository as only last version is 
> available) , counters decrements well.

The old debs are still here, so you can still download them manually if
you need to. I need to switch to aptly at some point since reprepro is
unlikely to ever support several versions for the same source package. I
would also have to host and build packages for Ubuntu as well as it is
common request.
-- 
Choose variable names that won't be confused.
- The Elements of Programming Style (Kernighan & Plauger)

Re: [ANNOUNCE] haproxy-2.3.9

[Willy Tarreau]

On Tue, Mar 30, 2021 at 07:08:25PM +0200, Tim Düsterhus wrote:
> Willy,
> 
> On 3/30/21 6:58 PM, Willy Tarreau wrote:
> > Note: I've just discovered that building with -DDEBUG_THREAD fails :-(
> >   I've now fixed it, and since this normally only affects haproxy
> >   developers who are supposed to use git versions, I don't expect
> >   this to be an issue for anyone. However if it's an issue for you,
> >   just let me know and I'll emit 2.3.10. I just don't want to spend
> >   my time creating releases for no reason.
> > 
> 
> may I request one for 1.7 if you have some spare cycles:
> https://github.com/haproxy/haproxy/issues/760#issuecomment-805280316?
> 
> While the issue for Docker is fixed by manually applying the patch [1]
> the current official release fails the build for musl.

Ah yes, already forgot about this one, thank you :-/  We'll check
tomorrow, or my head will explode and that would be dirty here.

thanks,
Willy

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 07:31:34PM +0200, Sander Klein wrote:
> Yes! It works. Sometimes you just need to go home, eat something and look
> again.

Oh I know how it feels, it works the same when emitting releases sometimes...

> It did need a full restart to get it going again though.

You mean "as opposed to a simple reload" ? This ought to be decorellated
since the master always performs an exec() on reload to make sure the new
binary is properly taken into account. Or maybe it failed to restart for
whatever reason and remained on the old one.

Anyway, thanks for your tests, that makes me more confident in 2.2.12.
It's probably not for this evening though (too high risk of mistakes
caused by high context-switch rate, I need to cool down first) but likely
tomorrow morning.

Cheers,
Willy

Re: Table sticky counters decrementation problem

[Sander Klein]

On 2021-03-30 19:15, Willy Tarreau wrote:

On Tue, Mar 30, 2021 at 07:07:41PM +0200, Sander Klein wrote:

On 2021-03-30 18:14, Willy Tarreau wrote:

> No, my chance is already gone :-)
>
> OK, I'm pushing this one into 2.3, re-running the tests a last time,
> and issuing 2.3.9. We'll be able to issue 2.2.12 soon finally, as users
> of 2.2 are still into trouble between 2.2.9 and 2.2.11 depending on the
> bug they try to avoid :-/

Somehow either my patching skillz have gone down the drain or this fix
doesn't work for me on 2.2.11. I still see the same behavior.


No worries, I'll backport whatever is needed so that you can test the
latest maintenance version, it will make you more confident in your
tests.


Yes! It works. Sometimes you just need to go home, eat something and 
look again. It did need a full restart to get it going again though.


Sander

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 07:07:41PM +0200, Sander Klein wrote:
> On 2021-03-30 18:14, Willy Tarreau wrote:
> 
> > No, my chance is already gone :-)
> > 
> > OK, I'm pushing this one into 2.3, re-running the tests a last time,
> > and issuing 2.3.9. We'll be able to issue 2.2.12 soon finally, as users
> > of 2.2 are still into trouble between 2.2.9 and 2.2.11 depending on the
> > bug they try to avoid :-/
> 
> Somehow either my patching skillz have gone down the drain or this fix
> doesn't work for me on 2.2.11. I still see the same behavior.

No worries, I'll backport whatever is needed so that you can test the
latest maintenance version, it will make you more confident in your
tests.

Thanks!
Willy

Re: [ANNOUNCE] haproxy-2.3.9

[Tim Düsterhus]

Willy,

On 3/30/21 6:58 PM, Willy Tarreau wrote:
> Note: I've just discovered that building with -DDEBUG_THREAD fails :-(
>   I've now fixed it, and since this normally only affects haproxy
>   developers who are supposed to use git versions, I don't expect
>   this to be an issue for anyone. However if it's an issue for you,
>   just let me know and I'll emit 2.3.10. I just don't want to spend
>   my time creating releases for no reason.
> 

may I request one for 1.7 if you have some spare cycles:
https://github.com/haproxy/haproxy/issues/760#issuecomment-805280316?

While the issue for Docker is fixed by manually applying the patch [1]
the current official release fails the build for musl.

Best regards
Tim Düsterhus

[1]
https://github.com/docker-library/haproxy/commit/42989bf9ab08e05e7333261ce62fff00d0dcb08a

Re: Table sticky counters decrementation problem

[Sander Klein]

On 2021-03-30 18:14, Willy Tarreau wrote:


No, my chance is already gone :-)

OK, I'm pushing this one into 2.3, re-running the tests a last time,
and issuing 2.3.9. We'll be able to issue 2.2.12 soon finally, as users
of 2.2 are still into trouble between 2.2.9 and 2.2.11 depending on the
bug they try to avoid :-/


Somehow either my patching skillz have gone down the drain or this fix 
doesn't work for me on 2.2.11. I still see the same behavior.


Sander

[ANNOUNCE] haproxy-2.3.9

[Willy Tarreau]

Hi,

HAProxy 2.3.9 was released on 2021/03/30. It added 5 new commits
after version 2.3.8.

This essentially fixes the rate counters issue that popped up in 2.3.8
after the previous fix for the rate counters already.

What happened is that the internal time in millisecond wraps every 49.7
days and that the new global counter used to make sure rate counters are
now stable across threads starts at zero and is initialized when older
than the current thread's current date. It just happens that the wrapping
happened a few hours ago at "Mon Mar 29 23:59:46 CEST 2021" exactly and
that any process started since this date and for the next 24 days doesn't
validate this condition anymore, hence doesn't rotate its rate counters
anymore.

Another issue possibly not affecting the same users that I met on a 8-core
16-thread xeon is that there was still some important contention on the
idle conns in case the limit of the file descriptors was about to be
reached and threads fight to choose a queue to put it into, to the point
of regularly triggering the watchdog. We already fixed a similar one
recently but it existed at two places.

And the fix for the rare crashes in mux-h1 on double shutdown reported in
issue #1197 was backported.

The rest is quite minor (payload sample fetch not properly waiting, and
restarting servers not using the correct color on the stats page).

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Wiki : https://github.com/haproxy/wiki/wiki
   Sources  : http://www.haproxy.org/download/2.3/src/
   Git repository   : http://git.haproxy.org/git/haproxy-2.3.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git
   Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Note: I've just discovered that building with -DDEBUG_THREAD fails :-(
  I've now fixed it, and since this normally only affects haproxy
  developers who are supposed to use git versions, I don't expect
  this to be an issue for anyone. However if it's an issue for you,
  just let me know and I'll emit 2.3.10. I just don't want to spend
  my time creating releases for no reason.

Willy
---
Complete changelog :
Christopher Faulet (1):
  BUG/MINOR: payload: Wait for more data if buffer is empty in 
payload/payload_lv

Florian Apolloner (1):
  BUG/MINOR: stats: Apply proper styles in HTML status page.

Willy Tarreau (3):
  BUG/MEDIUM: mux-h1: make h1_shutw_conn() idempotent
  MEDIUM: backend: use a trylock to grab a connection on high FD counts as 
well
  BUG/MEDIUM: time: make sure to always initialize the global tick

---

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 06:33:12PM +0200, Thomas SIMON wrote:
> Hi willy,
> 
> just to confirm that sticky counter decrement is okay with your patch on
> 2.3.8 version, so no objection for 2.3.9 patching neither :)

Great, thanks for the test. I've just committed the fix and am preparing
2.3.9 now.

Willy

Re: Table sticky counters decrementation problem

[Thomas SIMON]

Hi willy,

just to confirm that sticky counter decrement is okay with your patch on 
2.3.8 version, so no objection for 2.3.9 patching neither :)


thomas

Le 30/03/2021 à 15:47, Willy Tarreau a écrit :

On Tue, Mar 30, 2021 at 03:17:34PM +0200, Sander Klein wrote:

On 2021-03-30 15:13, Willy Tarreau wrote:


diff --git a/src/time.c b/src/time.c
index 0cfc9bf3c..fafe3720e 100644
--- a/src/time.c
+++ b/src/time.c
@@ -268,7 +268,7 @@ void tv_update_date(int max_wait, int interrupted)
 old_now_ms = global_now_ms;
 do {
 new_now_ms = old_now_ms;
-   if (tick_is_lt(new_now_ms, now_ms))
+   if (tick_is_lt(new_now_ms, now_ms) || !new_now_ms)
 new_now_ms = now_ms;
 }  while (!_HA_ATOMIC_CAS(&global_now_ms, &old_now_ms,
new_now_ms));

Do I need to apply this on top of the other fixes? Or should this be done on
the vanilla 2.2.11?

It's indeed on top of other fixes like those present in 2.3.8 or queued
in 2.2-maint.

Just let me know if you need some help with the patch or if you need another
one. I've mostly focused on 2.3 for now since 2.3.8 was expected to be
definitely fixed and I wanted to do 2.2.12 today based on it.

Thanks!
Willy


--
Thomas SIMON
Responsable Infrastructures
Neteven

Clients occasionally see truncated responses

[Nathan Konopinski]

Sometimes clients (clients are only http 1.1 and use connection: close) are
reporting a body length of ~4000 is less than the content length of ~14000.
The issue does not appear when using nginx as an LB and I've verified
complete responses are being sent from the backends for the requests
clients report errors on.

It's not clear why a portion of the clients aren't receiving the entire
response. I'm unable to replicate the issue with curl. I have a vanilla
config using https, prometheus metrics, and a h1-case-adjust-bogus-client
option to adjust a couple headers.

Has anyone come across similar issues? I see an option for request
buffering but nothing for response buffering. Are there options I can
adjust that could be related to this type of issue?

```
$ haproxy -vv
HA-Proxy version 2.2.11 2021/03/18 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2
2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.11.html
Running on: Linux 4.9.0-13-amd64 #1 SMP Debian 4.9.228-1 (2020-07-05) x86_64
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1
  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 -PCRE2_JIT
+POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H
+GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB -SLZ
+CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD
-OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=48).
Built with OpenSSL version : OpenSSL 1.1.0l  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.0l  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 6.3.0 20170516
Built with the Prometheus exporter as a service

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
fcgi : mode=HTTP   side=BEmux=FCGI
: mode=HTTP   side=FE|BE mux=H1
  h2 : mode=HTTP   side=FE|BE mux=H2
: mode=TCPside=FE|BE mux=PASS

Available services : prometheus-exporter
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
[CACHE] cache
[FCGI] fcgi-app


$ cat haproxy.cfg
global
log /dev/log local0 notice
chroot /var/lib/haproxy
stats timeout 30s
user haproxy
daemon
maxconn 1024000
nbproc 1
nbthread 30
ssl-default-bind-options ssl-min-ver TLSv1.0 no-tls-tickets
h1-case-adjust content-length Content-Length
h1-case-adjust date Date

defaults
log global
modehttp
option  httplog
option  dontlognull
timeout connect 5000
timeout client  5
timeout server  5
option h1-case-adjust-bogus-client

frontend https
bind *:443 ssl crt /cert.pem
acl PATH path /path
http-request deny deny_status 404 unless PATH
default_backend servers

backend servers
balance roundrobin
option forwardfor if-none
option httpchk GET /metrics
server one 192.168.1.10:8080 check
server two 192.168.1.11:8080 check

frontend stats
bind 192.168.1.1:9001
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /
stats realm Haproxy\ Statistics
```

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 06:15:28PM +0200, William Dauchy wrote:
> On Tue, Mar 30, 2021 at 5:57 PM Willy Tarreau  wrote:
> > out of curiosity I wanted to check when the overflow happened:
> >
> > $ date --date=@$$(date +%s) * 1000) & -0x800) / 1000))
> > Mon Mar 29 23:59:46 CEST 2021
> >
> > So it only affects processes started since today. I'm quite tempted not
> > to wait further and to emit 2.3.9 urgently to fix this before other
> > people get trapped after reloading their process. Any objection ?
> 
> I do confirm the timestamp on our side but do not have the necessary
> tooling to test the fix.

Many thanks William!

willy

Re: Table sticky counters decrementation problem

[William Dauchy]

On Tue, Mar 30, 2021 at 5:57 PM Willy Tarreau  wrote:
> out of curiosity I wanted to check when the overflow happened:
>
> $ date --date=@$$(date +%s) * 1000) & -0x800) / 1000))
> Mon Mar 29 23:59:46 CEST 2021
>
> So it only affects processes started since today. I'm quite tempted not
> to wait further and to emit 2.3.9 urgently to fix this before other
> people get trapped after reloading their process. Any objection ?

I do confirm the timestamp on our side but do not have the necessary
tooling to test the fix.

Thanks,
-- 
William

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 06:09:09PM +0200, Lukas Tribus wrote:
> Hi Willy,
> 
> On Tue, 30 Mar 2021 at 17:56, Willy Tarreau  wrote:
> >
> > Guys,
> >
> > out of curiosity I wanted to check when the overflow happened:
> >
> > $ date --date=@$$(date +%s) * 1000) & -0x800) / 1000))
> > Mon Mar 29 23:59:46 CEST 2021
> >
> > So it only affects processes started since today. I'm quite tempted not
> > to wait further and to emit 2.3.9 urgently to fix this before other
> > people get trapped after reloading their process. Any objection ?
> 
> No objection, but also: what a coincidence. I suggest you get a
> lottery ticket today.

No, my chance is already gone :-)

OK, I'm pushing this one into 2.3, re-running the tests a last time,
and issuing 2.3.9. We'll be able to issue 2.2.12 soon finally, as users
of 2.2 are still into trouble between 2.2.9 and 2.2.11 depending on the
bug they try to avoid :-/

Willy

Re: Table sticky counters decrementation problem

[Lukas Tribus]

Hi Willy,

On Tue, 30 Mar 2021 at 17:56, Willy Tarreau  wrote:
>
> Guys,
>
> out of curiosity I wanted to check when the overflow happened:
>
> $ date --date=@$$(date +%s) * 1000) & -0x800) / 1000))
> Mon Mar 29 23:59:46 CEST 2021
>
> So it only affects processes started since today. I'm quite tempted not
> to wait further and to emit 2.3.9 urgently to fix this before other
> people get trapped after reloading their process. Any objection ?

No objection, but also: what a coincidence. I suggest you get a
lottery ticket today.


cheers,
lukas

Re: Table sticky counters decrementation problem

[Willy Tarreau]

Guys,

out of curiosity I wanted to check when the overflow happened:

$ date --date=@$$(date +%s) * 1000) & -0x800) / 1000))
Mon Mar 29 23:59:46 CEST 2021

So it only affects processes started since today. I'm quite tempted not
to wait further and to emit 2.3.9 urgently to fix this before other
people get trapped after reloading their process. Any objection ?

Willy

Re: [PATCH] Apply proper styles in HTML status page.

[Florian Apolloner]

Hi Willy,

On Tue, Mar 30, 2021, at 17:04, Willy Tarreau wrote:
> Well, for a first one it was pretty good :-)

Thank you for the kind words and the quick merge.

> And I don't recall having seen that orange for a long time now.

Ok, so I was not going mad either. I remembered haproxy from like 10 years ago 
to have more colours on that page :D

Thanks again,
Florian

Re: [PATCH] Apply proper styles in HTML status page.

[Willy Tarreau]

Hi Florian,

On Tue, Mar 30, 2021 at 01:42:55PM +0200, Florian Apolloner wrote:
> Hi there,
> 
> I am a first time contributor, so please be gentle -- I most certainly did
> something wrong. I tried to follow as much as possible of the CONTRIBUTING
> file but I most likely CC'ed the wrong person or something (if that is the
> case, I apologize in advance).

Well, for a first one it was pretty good :-)  I've readjusted the subject
as we use some tags to qualify the nature of the change (feature or bug)
and the approximate area ("stats" here) to help triage them, but that's
all.

> While playing with HAProxy I noticed that the status page currently does not
> ever seem to show a backend as orange ("active DOWN, going up"). If I read
> the code correctly this is due to a broken usage of strcmp which tries to
> compare "DOWN 3/5" with "DOWN " (note the trailing blank). I have adjusted
> it to strncmp and now backends that are down and going up are displayed
> orange again. I have also done the same for the "NOLB " case.

This is interesting because on my first reading I was surprised because I
was used to see them in yellow and thought that it was what you called
orange. But actually you're totally right, yellow is supposed to be for
up going down, not down going up. And I don't recall having seen that
orange for a long time now.

I introduced this issue myself 5 years ago when we've split the stats
output from its processing path, with commit 0c378efe8 ("MEDIUM: stats:
compute the color code only in the HTML form"). This may explain why I
don't recall having seen orange for a while!

Now applied, thank you!
Willy

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 03:17:34PM +0200, Sander Klein wrote:
> On 2021-03-30 15:13, Willy Tarreau wrote:
> 
> > diff --git a/src/time.c b/src/time.c
> > index 0cfc9bf3c..fafe3720e 100644
> > --- a/src/time.c
> > +++ b/src/time.c
> > @@ -268,7 +268,7 @@ void tv_update_date(int max_wait, int interrupted)
> > old_now_ms = global_now_ms;
> > do {
> > new_now_ms = old_now_ms;
> > -   if (tick_is_lt(new_now_ms, now_ms))
> > +   if (tick_is_lt(new_now_ms, now_ms) || !new_now_ms)
> > new_now_ms = now_ms;
> > }  while (!_HA_ATOMIC_CAS(&global_now_ms, &old_now_ms,
> > new_now_ms));
> 
> Do I need to apply this on top of the other fixes? Or should this be done on
> the vanilla 2.2.11?

It's indeed on top of other fixes like those present in 2.3.8 or queued
in 2.2-maint.

Just let me know if you need some help with the patch or if you need another
one. I've mostly focused on 2.3 for now since 2.3.8 was expected to be
definitely fixed and I wanted to do 2.2.12 today based on it.

Thanks!
Willy

Re: zlib vs slz (perfoarmance)

[Willy Tarreau]

On Tue, Mar 30, 2021 at 05:57:55PM +0500,  ??? wrote:
> also, I read that slz stops compressing when CPU level reaches some
> threshold. is it related to all gzip, including zlib ?

It's neither, it's haproxy which does this, based on "maxcompcpuusage"
(but it defaults to 100, i.e. unlimited). And similarly you can limit
zlib's memory usage using maxzlibmem, which will make it refrain from
compressing once the limit is reached. For me the memory is the real
problem here. 30k concurrent streams will consume 9 GB.

> if so, we can safely stick with any compression lib (but I agree that
> having benchmarks would help people)

There are a few benchmarks on the site and the readme, and some can be
found in dedicated tools. We could indeed produce some more detailed
tests on cheap dedicated servers with limited bandwidth/unlimited data
to see which one is better for which case, but I recall a few figures
such as "use zlib below 50 Mbps output link size on Atoms if you've
got lots of RAM otherwise use slz".

Willy

Re: Table sticky counters decrementation problem

[Sander Klein]

On 2021-03-30 15:13, Willy Tarreau wrote:


diff --git a/src/time.c b/src/time.c
index 0cfc9bf3c..fafe3720e 100644
--- a/src/time.c
+++ b/src/time.c
@@ -268,7 +268,7 @@ void tv_update_date(int max_wait, int interrupted)
old_now_ms = global_now_ms;
do {
new_now_ms = old_now_ms;
-   if (tick_is_lt(new_now_ms, now_ms))
+   if (tick_is_lt(new_now_ms, now_ms) || !new_now_ms)
new_now_ms = now_ms;
}  while (!_HA_ATOMIC_CAS(&global_now_ms, &old_now_ms, 
new_now_ms));


Do I need to apply this on top of the other fixes? Or should this be 
done on the vanilla 2.2.11?


Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 02:54:55PM +0200, Willy Tarreau wrote:
> And I've tested using the same method (http_req_rate(2s) and 500ms this
> time to cover both >1s and <1s). So I don't know what to say. I'm now
> extremely tempted to revert all these fixes because in the end the
> original problem was much less visible for most users :-(
> 
> I'm now trying with this *exact* config in case I missed something else.

So I was not crazy. The reason is that... the date changed since my
tests one week ago :-(

The new date is updated if it's in the past compared to the newest one,
except that it starts at zero. Last week during my tests, the now_ms
date was ~1.6 billion. 0-1.6 billion is negative so the new_now_ms date
was updated to reflect it.

But today the date it ~2.2 billion, which is higher than 2^31, thus
0-2.2 billion is positive and the new date is not after the old one so
the old one is not updated.

Thus we need to special case the update to reflect this. What saddens
me is that I hesitated to completely rewrite this part to simplify it
and concluded that I'd rather stay on the safe side. Someone getting
rid of legacy would be better, really!

This patch fixes it.

Sander, Thomas, please check again with it, it MUST work this time!

Thanks,
Willy

diff --git a/src/time.c b/src/time.c
index 0cfc9bf3c..fafe3720e 100644
--- a/src/time.c
+++ b/src/time.c
@@ -268,7 +268,7 @@ void tv_update_date(int max_wait, int interrupted)
old_now_ms = global_now_ms;
do {
new_now_ms = old_now_ms;
-   if (tick_is_lt(new_now_ms, now_ms))
+   if (tick_is_lt(new_now_ms, now_ms) || !new_now_ms)
new_now_ms = now_ms;
}  while (!_HA_ATOMIC_CAS(&global_now_ms, &old_now_ms, new_now_ms));
 

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

вт, 30 мар. 2021 г. в 17:44, Willy Tarreau :

> On Tue, Mar 30, 2021 at 02:25:14PM +0500,  ??? wrote:
> > I would wait for feedback from some other high load projects. From my
> > observation it is significant benefit.
>
> Originally slz was written for use in haproxy to solve the huge memory
> consumption that comes with zlib (~288kB of context per stream IIRC).
> SLZ needs much less and uses ~40 bytes or so per stream, and by keeping
> less context, has less opportunities for high compression ratios. This
> simplified its lookup algorithms, and as a byproduct made it significantly
> faster.
>


for example, silesia xml, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz

lzbench suite

[root@localhost lzbench]# ./lzbench -ezlib,1/slz_zlib,1  silezia/xml
lzbench 1.8 (64-bit Linux)   Assembled by P.Skibinski
Compressor name Compress. Decompress. Compr. size  Ratio Filename
memcpy  10948 MB/s 15766 MB/s 5345280 100.00 silezia/xml
zlib 1.2.11 -1125 MB/s   429 MB/s  965248  18.06 silezia/xml
slz_zlib 1.2.0 -1 329 MB/s   331 MB/s 1294302  24.21 silezia/xml
done... (cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)
[root@localhost lzbench]#


I observe that slz is more than 2 times faster compared to zlib, while
compressed file size is 20% bigger.
but I did not yet checked zlib-ng.

also, I read that slz stops compressing when CPU level reaches some
threshold. is it related to all gzip, including zlib ?
if so, we can safely stick with any compression lib (but I agree that
having benchmarks would help people)


>
> The only sane way I've seen to use zlib is to limit its CPU usage. When it
> reaches a target ratio (typically 80%), compression automatically disables
> itself, so overall the savings-to-cpu usage makes it less interesting for
> zlib in our use cases. The only case where zlib is more interesting is for
> those having a small line and who prefer to spend more CPU cycles to get
> more savings.
>
> One reasonable approach we could think of would consist in importing SLZ
> directly into haproxy now. I didn't want to do it initially because the
> code was young and I wanted it to live its own life and get its own fixes.
> Now the burn-in test is long finished, and the only updates over the last
> 4 years were for performance improvements or fixes for use cases outside
> of haproxy. That's only 1500 lines of code to integrate and it would
> certainly simplify a lot of things. We could even imagine to always build
> with compression enabled, SLZ when not specified otherwise zlib when
> asked for it.
>
> Just a few ideas.
> Willy
>

Re: Table sticky counters decrementation problem

[Willy Tarreau]

On Tue, Mar 30, 2021 at 10:17:01AM +0200, Lukas Tribus wrote:
> Hello Thomas,
> 
> 
> this is a known issue in any release train other than 2.3 ...
> 
> https://github.com/haproxy/haproxy/issues/1196
> 
> However neither 2.3.7 (does not contain the offending commits), nor
> 2.3.8 (contains all the fixes) should be affected by this.
> 
> 
> Are you absolutely positive that you are running 2.3.8 and not
> something like 2.2 or 2.0 ? Can you provide the full output of haproxy
> -vv?

I must say that I'm completely puzzled because that's exactly the issue
that affected 2.3.7, and which was fixed in 2.3.8. Here it seems to do
exactly the opposite!

And I've tested using the same method (http_req_rate(2s) and 500ms this
time to cover both >1s and <1s). So I don't know what to say. I'm now
extremely tempted to revert all these fixes because in the end the
original problem was much less visible for most users :-(

I'm now trying with this *exact* config in case I missed something else.

Thanks,
Willy

Re: zlib vs slz (perfoarmance)

[Willy Tarreau]

On Tue, Mar 30, 2021 at 02:25:14PM +0500,  ??? wrote:
> I would wait for feedback from some other high load projects. From my
> observation it is significant benefit.

Originally slz was written for use in haproxy to solve the huge memory
consumption that comes with zlib (~288kB of context per stream IIRC).
SLZ needs much less and uses ~40 bytes or so per stream, and by keeping
less context, has less opportunities for high compression ratios. This
simplified its lookup algorithms, and as a byproduct made it significantly
faster.

The only sane way I've seen to use zlib is to limit its CPU usage. When it
reaches a target ratio (typically 80%), compression automatically disables
itself, so overall the savings-to-cpu usage makes it less interesting for
zlib in our use cases. The only case where zlib is more interesting is for
those having a small line and who prefer to spend more CPU cycles to get
more savings.

One reasonable approach we could think of would consist in importing SLZ
directly into haproxy now. I didn't want to do it initially because the
code was young and I wanted it to live its own life and get its own fixes.
Now the burn-in test is long finished, and the only updates over the last
4 years were for performance improvements or fixes for use cases outside
of haproxy. That's only 1500 lines of code to integrate and it would
certainly simplify a lot of things. We could even imagine to always build
with compression enabled, SLZ when not specified otherwise zlib when
asked for it.

Just a few ideas.
Willy

[PATCH] Apply proper styles in HTML status page.

[Florian Apolloner]

Hi there,

I am a first time contributor, so please be gentle -- I most certainly 
did something wrong. I tried to follow as much as possible of the 
CONTRIBUTING file but I most likely CC'ed the wrong person or something 
(if that is the case, I apologize in advance).


While playing with HAProxy I noticed that the status page currently does 
not ever seem to show a backend as orange ("active DOWN, going up"). If 
I read the code correctly this is due to a broken usage of strcmp which 
tries to compare "DOWN 3/5" with "DOWN " (note the trailing blank). I 
have adjusted it to strncmp and now backends that are down and going up 
are displayed orange again. I have also done the same for the "NOLB " case.


I'll be happy to answer any questions you have or work further on the patch.

I am not subscribed to the list and would appreciate if you kept me in CC.

Cheers,
Florian
>From 07fc8dbcd996203fec6d78c550584306f6225ded Mon Sep 17 00:00:00 2001
From: Florian Apolloner 
Date: Tue, 30 Mar 2021 13:28:35 +0200
Subject: [PATCH] Apply proper styles in HTML status page.

When a backend is in status DOWN and going UP it is currently displayed
as yellow ("active UP, going down") instead of orange ("active UP, going
down"). This patches restyles the table rows to actually match the
legend.
---
 src/stats.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/stats.c b/src/stats.c
index e54d13adf..bfd05bbe2 100644
--- a/src/stats.c
+++ b/src/stats.c
@@ -1079,13 +1079,13 @@ static int stats_dump_fields_html(struct buffer *out,
 		strcmp(field_str(stats, ST_F_STATUS), "DOWN (agent)") == 0) {
 			style = "down";
 		}
-		else if (strcmp(field_str(stats, ST_F_STATUS), "DOWN ") == 0) {
+		else if (strncmp(field_str(stats, ST_F_STATUS), "DOWN ", strlen("DOWN ")) == 0) {
 			style = "going_up";
 		}
 		else if (strcmp(field_str(stats, ST_F_STATUS), "DRAIN") == 0) {
 			style = "draining";
 		}
-		else if (strcmp(field_str(stats, ST_F_STATUS), "NOLB ") == 0) {
+		else if (strncmp(field_str(stats, ST_F_STATUS), "NOLB ", strlen("NOLB ")) == 0) {
 			style = "going_down";
 		}
 		else if (strcmp(field_str(stats, ST_F_STATUS), "NOLB") == 0) {
-- 
2.30.2

Re: Stick table counter not working after upgrade to 2.2.11

[Lukas Tribus]

Hi Willy,


On Tue, 23 Mar 2021 at 09:32, Willy Tarreau  wrote:
>
> Guys,
>
> These two patches address it for me, and I could verify that they apply
> on top of 2.2.11 and work there as well. This time I tested with two
> counters at different periods 500 and 2000ms.

Both Sander and Thomas now report that the issue is at least partially
still there in 2.3.8 (which has all fixes, or 2.2.11 with patches),
and that downgrading to 2.3.7 (which has none of the commits) works
around the issue:

https://www.mail-archive.com/haproxy@formilux.org/msg40093.html
https://www.mail-archive.com/haproxy@formilux.org/msg40092.html


Let's not yet publish stable bugfix releases, until this is properly diagnosed.



Lukas

Re: zlib vs slz (perfoarmance)

[Илья Шипицин]

I would wait for feedback from some other high load projects. From my
observation it is significant benefit.

On Tue, Mar 30, 2021, 12:18 PM Dinko Korunic 
wrote:

> On 30.03.2021., at 08:17, Илья Шипицин  wrote:
> >
> > I would really like to know whether zlib was chosen for purpose or by
> chance.
> >
> > And yes, some marketing campaign makes sense
> >
>
> Hi Илья,
>
> People tend to spawn Docker images in dozens and/or even hundreds and we
> always have to consider that adding a single library on top of what’s
> already present in the minimal Docker base image (libz is pretty much
> always present in base images) will result in additional size which is in
> general frown upon by Docker users.
>
> But then again, if the community (aka you) thinks that pros (performance)
> outweigh the cons (increased target size), I’ll take care of it for
> haproxytech images and these changes would be also propagated to Ingress
> Controller image as well.
>
>
> Kind regards,
> D.
>
> --
> Dinko Korunic   ** Standard disclaimer applies **
> Sent from OSF1 osf1v4b V4.0 564 alpha
>
>
>

Re: Table sticky counters decrementation problem

[Thomas SIMON]

Hi Lukas,

I'm on 2.3.8 yes

root@web12:~# haproxy -vv
HA-Proxy version 2.3.8-1~bpo10+1 2021/03/25 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2022.
Known bugs: http://www.haproxy.org/bugs/bugs-2.3.8.html
Running on: Linux 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 
14:26:17 +0100) x86_64

Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-2.3.8=. 
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time 
-D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered 
-Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond 
-Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 
USE_ZLIB=1 USE_SYSTEMD=1

  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 
+PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE 
-STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT 
+CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB 
-SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL 
+SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with the Prometheus exporter as a service
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTTP   side=FE|BE mux=H2
    fcgi : mode=HTTP   side=BE    mux=FCGI
    : mode=HTTP   side=FE|BE mux=H1
    : mode=TCP    side=FE|BE mux=PASS

Available services : prometheus-exporter
Available filters :
    [SPOE] spoe
    [CACHE] cache
    [FCGI] fcgi-app
    [COMP] compression
    [TRACE] trace

I'm using buster-backports repository, and I've updated package 
yesterday morning


[11:15:44]root@web12:~# cat /etc/apt/sources.list.d/haproxy.list
deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] 
http://haproxy.debian.net buster-backports-2.3 main


root@web12:~# aptcp haproxy
haproxy:
  Installed: 2.3.8-1~bpo10+1
  Candidate: 2.3.8-1~bpo10+1
  Version table:
 *** 2.3.8-1~bpo10+1 100
    100 http://haproxy.debian.net buster-backports-2.3/main amd64 
Packages

    100 /var/lib/dpkg/status

root@web12:~# grep haproxy /var/log/dpkg.log
2021-03-29 09:31:56 upgrade haproxy:amd64 2.3.7-1~bpo10+1 2.3.8-1~bpo10+1
2021-03-29 09:31:56 status half-configured haproxy:amd64 2.3.7-1~bpo10+1
2021-03-29 09:31:56 status unpacked haproxy:amd64 2.3.7-1~bpo10+1
2021-03-29 09:31:56 status half-installed haproxy:amd64 2.3.7-1~bpo10+1
2021-03-29 09:31:56 status unpacked haproxy:amd64 2.3.8-1~bpo10+1
2021-03-29 09:31:57 configure haproxy:amd64 2.3.8-1~bpo10+1 
2021-03-29 09:31:57 status unpacked haproxy:amd64 2.3.8-1~bpo10+1
2021-03-29 09:32:06 conffile /etc/logrotate.d/haproxy keep
2021-03-29 09:32:06 status half-configured haproxy:amd64 2.3.8-1~bpo10+1
2021-03-29 09:32:08 status installed haproxy:amd64 2.3.8-1~bpo10+1

And I confirm you than when rolling back with source compilation and 
2.3.7 version (can't do this with repository as only last version is 
available) , counters decrements well.


thanks

thomas

Le 30/03/2021 à 10:17, Lukas Tribus a écrit :

Hello Thomas,


this is a known issue in any release train other than 2.3 ...

https://github.com/haproxy/haproxy/issues/1196

However neither 2.3.7 (does not contain the offending commits), nor
2.3.8 (contains all the fixes) should be affected by this.


Are you absolutely positive that you are running 2.3.8 and not
something like 2.2 or 2.0 ? Can you provide the full output of haproxy
-vv?



Thanks,

Lukas


--
Thomas SIMON
Responsable Infrastructures
Neteven

Re: Table sticky counters decrementation problem

[Sander Klein]

On 2021-03-30 10:17, Lukas Tribus wrote:

Hello Thomas,


this is a known issue in any release train other than 2.3 ...

https://github.com/haproxy/haproxy/issues/1196

However neither 2.3.7 (does not contain the offending commits), nor
2.3.8 (contains all the fixes) should be affected by this.


Are you absolutely positive that you are running 2.3.8 and not
something like 2.2 or 2.0 ? Can you provide the full output of haproxy
-vv?



I can confirm I'm seeing this on 2.3.8 as well. But moreover, I also see 
this happening on 2.2.11 with Willy's patches in it as well.


I am very confused because I am pretty sure this problem was gone last 
week when I tested the patches and took that version in production.


Sander



0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Re: Table sticky counters decrementation problem

[Lukas Tribus]

Hello Thomas,


this is a known issue in any release train other than 2.3 ...

https://github.com/haproxy/haproxy/issues/1196

However neither 2.3.7 (does not contain the offending commits), nor
2.3.8 (contains all the fixes) should be affected by this.


Are you absolutely positive that you are running 2.3.8 and not
something like 2.2 or 2.0 ? Can you provide the full output of haproxy
-vv?



Thanks,

Lukas

Table sticky counters decrementation problem

[Thomas SIMON]

Hi all,

Since version 2.3.8, I've noticed problem with come sticky counters, 
which only increments, and never decrements. The behavior was OK in 2.3.7



frontend web
    bind *:443 ssl crt /etc/ssl/certs/...
    http-request track-sc0 src table global_limits

backend global_limits
 stick-table type ip size 1m expire 1h store 
conn_cur,http_req_rate(20s),http_err_rate(1h)



Stick table

echo "show table global_limits" | socat stdio 
unix-connect:/run/haproxy/admin.sock
0x7ff0f4027d40: key=195.219.xxx.xxx use=2 exp=3599384 conn_cur=2 
http_req_rate(2)=607 http_err_rate(360)=0


One minute after :

0x7ff0f4027d40: key=195.219.250.105 use=2 exp=3599923 conn_cur=2 
http_req_rate(2)=689 http_err_rate(360)=0


Conn_cur increments and decrements well, but http_req_rate and 
http_err_rate doesn't.



regards
thomas

Re: zlib vs slz (perfoarmance)

[Dinko Korunic]

On 30.03.2021., at 08:17, Илья Шипицин  wrote:
> 
> I would really like to know whether zlib was chosen for purpose or by chance.
> 
> And yes, some marketing campaign makes sense
> 

Hi Илья,

People tend to spawn Docker images in dozens and/or even hundreds and we always 
have to consider that adding a single library on top of what’s already present 
in the minimal Docker base image (libz is pretty much always present in base 
images) will result in additional size which is in general frown upon by Docker 
users.

But then again, if the community (aka you) thinks that pros (performance) 
outweigh the cons (increased target size), I’ll take care of it for haproxytech 
images and these changes would be also propagated to Ingress Controller image 
as well.


Kind regards,
D.

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha

Re: zlib vs slz (perfoarmance)

[Aleksandar Lazic]

+1

On 30.03.21 08:17, Илья Шипицин wrote:

I would really like to know whether zlib was chosen for purpose or by chance.

And yes, some marketing campaign makes sense

On Tue, Mar 30, 2021, 10:35 AM Dinko Korunic mailto:dinko.koru...@gmail.com>> wrote:


 > On 29.03.2021., at 23:06, Lukas Tribus mailto:lu...@ltri.eu>> wrote:
 >

[…]

 > Like I said last year, this needs a marketing campaign:
 > https://www.mail-archive.com/haproxy@formilux.org/msg38044.html 

 >
 >
 > What about the docker images from haproxytech? Are those zlib or slz
 > based? Perhaps that would be a better starting point?
 >
 > https://hub.docker.com/r/haproxytech/haproxy-alpine 




Hi Lukas,

I am maintaining haproxytech Docker images and I can easily make that (slz 
being used) happen, if that’s what community would like to see.


Kind regards,
D.

-- 
Dinko Korunic                   ** Standard disclaimer applies **

Sent from OSF1 osf1v4b V4.0 564 alpha