Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Dauchy
On Thu, Jan 7, 2021 at 4:02 PM Willy Tarreau  wrote:
> No, we'd rather indeed make the types exclude broken by default.
> Let's just preset "REGTESTS_TYPES=default,bug,devel,slow" in the makefile
> and that's done without changing anything for anyone who already forces
> them.
>
> Patches welcome, as usual :-)

thanks, will do.

-- 
William



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Willy Tarreau
On Thu, Jan 07, 2021 at 03:12:20PM +0100, William Dauchy wrote:
> On Thu, Jan 7, 2021 at 1:54 PM  ???  wrote:
> > We can make regtest type mandatory
> 
> honestly not a fan :) makes it more complicated for people trying to
> contribute simple things

No, we'd rather indeed make the types exclude broken by default.
Let's just preset "REGTESTS_TYPES=default,bug,devel,slow" in the makefile
and that's done without changing anything for anyone who already forces
them.

Patches welcome, as usual :-)
Willy



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Dauchy
On Thu, Jan 7, 2021 at 1:54 PM Илья Шипицин  wrote:
> We can make regtest type mandatory

honestly not a fan :) makes it more complicated for people trying to
contribute simple things

-- 
William



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Илья Шипицин
On Thu, Jan 7, 2021, 5:19 PM William Dauchy  wrote:

> On Thu, Jan 7, 2021 at 12:53 PM Илья Шипицин  wrote:
> > They are excluded in ci builds
> > - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests
> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
> cat $folder/INFO $folder/LOG; done && exit
>
> yeah I understood, but I'm thinking about small contributors trying to
> understand what is wrong before sending their patches; a default
> working `make reg-tests` would be nice in my own opinion.
>
> --
> William
>

We can make regtest type mandatory

>


Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Dauchy
On Thu, Jan 7, 2021 at 12:53 PM Илья Шипицин  wrote:
> They are excluded in ci builds
> - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests 
> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do 
> cat $folder/INFO $folder/LOG; done && exit

yeah I understood, but I'm thinking about small contributors trying to
understand what is wrong before sending their patches; a default
working `make reg-tests` would be nice in my own opinion.

-- 
William



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Илья Шипицин
On Thu, Jan 7, 2021, 4:45 PM William Dauchy  wrote:

> On Thu, Jan 7, 2021 at 11:54 AM William Lallemand
>  wrote:
> > These reg-tests are of types "slow" and "broken" not launched by the CI.
>
> thanks folks for your inputs. my vtest was indeed out of date.
> now I'm left with:
>
> ## Starting vtest ##
> Testing with haproxy version: 2.4-dev5-761d64-4
> #top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.194)
> exit=2
> 1 tests failed, 0 tests skipped, 107 tests passed
> ## Gathering results ##
> ## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
> ## test results in:
> "/tmp/haregtests-2021-01-07_12-38-36.OFayg8/vtc.132412.2a7ac563"
>  h1Bad exit status: 0x exit 0x0 signal 0 core 0
>
> but indeed:
> reg-tests/seamless-reload/abns_socket.vtc:#REGTEST_TYPE=broken
>
> maybe we should exclude them from a `make reg-tests` to avoid confusion?
> --
> William
>

They are excluded in ci builds
- env VTEST_PROGRAM=../vtest/vtest gmake reg-tests
REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
cat $folder/INFO $folder/LOG; done && exit

>


Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Dauchy
On Thu, Jan 7, 2021 at 11:54 AM William Lallemand
 wrote:
> These reg-tests are of types "slow" and "broken" not launched by the CI.

thanks folks for your inputs. my vtest was indeed out of date.
now I'm left with:

## Starting vtest ##
Testing with haproxy version: 2.4-dev5-761d64-4
#top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.194) exit=2
1 tests failed, 0 tests skipped, 107 tests passed
## Gathering results ##
## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
## test results in:
"/tmp/haregtests-2021-01-07_12-38-36.OFayg8/vtc.132412.2a7ac563"
 h1Bad exit status: 0x exit 0x0 signal 0 core 0

but indeed:
reg-tests/seamless-reload/abns_socket.vtc:#REGTEST_TYPE=broken

maybe we should exclude them from a `make reg-tests` to avoid confusion?
-- 
William



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Frederic Lecaille

On 1/7/21 11:03 AM, William Dauchy wrote:

unclear whether this is on my side only because I did not investigate
but I have two tests failing for a while now:

## Starting vtest ##
Testing with haproxy version: 2.4-dev5-761d64-4
#top  TEST reg-tests/filters/random-forwarding.vtc FAILED (0.192) exit=2
#top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.215) exit=2
2 tests failed, 0 tests skipped, 106 tests passed
## Gathering results ##
## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
## test results in:
"/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.67e5fada"
 h1   Bad exit status: 0x exit 0x0 signal 0 core 0
## Test case: reg-tests/filters/random-forwarding.vtc ##
## test results in:
"/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.72f8fd5f"
 c1   Assert error in vtc_gunzip(), src/vtc_gzip.c line 114:


Your vtest program does not seem to be up to date. Here is the vtest 
issue which is similar to yours:


https://github.com/vtest/VTest/issues/20



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Lallemand
On Thu, Jan 07, 2021 at 11:21:07AM +0100, Willy Tarreau wrote:
> On Thu, Jan 07, 2021 at 11:03:14AM +0100, William Dauchy wrote:
> > On Wed, Jan 6, 2021 at 5:45 PM Willy Tarreau  wrote:
> > > HAProxy 2.4-dev5 was released on 2021/01/06. It added 91 new commits
> > > after version 2.4-dev4.
> > 
> > unclear whether this is on my side only because I did not investigate
> > but I have two tests failing for a while now:
> > 
> > ## Starting vtest ##
> > Testing with haproxy version: 2.4-dev5-761d64-4
> > #top  TEST reg-tests/filters/random-forwarding.vtc FAILED (0.192) exit=2
> > #top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.215) 
> > exit=2
> 
> I didn't observe them, they all succeed on my side, and just checking
> the CI, it seems they succeed. But that doesn't mean they're reliable
> or anything, very often regtests start to fail sporadically in a single
> environment before we figure the problem.
> 

These reg-tests are of types "slow" and "broken" not launched by the CI.
 

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Willy Tarreau
On Thu, Jan 07, 2021 at 11:03:14AM +0100, William Dauchy wrote:
> On Wed, Jan 6, 2021 at 5:45 PM Willy Tarreau  wrote:
> > HAProxy 2.4-dev5 was released on 2021/01/06. It added 91 new commits
> > after version 2.4-dev4.
> 
> unclear whether this is on my side only because I did not investigate
> but I have two tests failing for a while now:
> 
> ## Starting vtest ##
> Testing with haproxy version: 2.4-dev5-761d64-4
> #top  TEST reg-tests/filters/random-forwarding.vtc FAILED (0.192) exit=2
> #top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.215) exit=2

I didn't observe them, they all succeed on my side, and just checking
the CI, it seems they succeed. But that doesn't mean they're reliable
or anything, very often regtests start to fail sporadically in a single
environment before we figure the problem.

Is it 100% reproducible for you ?

> 2 tests failed, 0 tests skipped, 106 tests passed
> ## Gathering results ##
> ## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
> ## test results in:
> "/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.67e5fada"
>  h1   Bad exit status: 0x exit 0x0 signal 0 core 0
> ## Test case: reg-tests/filters/random-forwarding.vtc ##
> ## test results in:
> "/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.72f8fd5f"
>  c1   Assert error in vtc_gunzip(), src/vtc_gzip.c line 114:
> Condition(vz.total_out < l) not true.  Errno=0 Success

I remember seeing this one with an old version of vtest, and it
went away after an upgrade.

> I build with:
> 
> make -j5 CPU="generic" TARGET="linux-glibc" USE_OPENSSL=1 USE_PCRE=1
> USE_PCRE_JIT=1 USE_ZLIB=1 USE_GETADDRINFO=1 USE_LINUX_TPROXY=1
> ADDLIB="-ljemalloc" DEFINE=-DTCP_USER_TIMEOUT=18 USE_SYSTEMD=1
> USE_TFO=1 USE_NS=1
> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

Using the exact same options (except systemd) worked fine for me
with 106 successful tests. So at least it's not cause exclusively
by the set of options.

Willy



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread Илья Шипицин
чт, 7 янв. 2021 г. в 15:06, William Dauchy :

> On Wed, Jan 6, 2021 at 5:45 PM Willy Tarreau  wrote:
> > HAProxy 2.4-dev5 was released on 2021/01/06. It added 91 new commits
> > after version 2.4-dev4.
>
> unclear whether this is on my side only because I did not investigate
> but I have two tests failing for a while now:
>
> ## Starting vtest ##
> Testing with haproxy version: 2.4-dev5-761d64-4
> #top  TEST reg-tests/filters/random-forwarding.vtc FAILED (0.192)
> exit=2
> #top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.215)
> exit=2
> 2 tests failed, 0 tests skipped, 106 tests passed
> ## Gathering results ##
> ## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
> ## test results in:
> "/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.67e5fada"
>  h1   Bad exit status: 0x exit 0x0 signal 0 core 0
> ## Test case: reg-tests/filters/random-forwarding.vtc ##
> ## test results in:
> "/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.72f8fd5f"
>  c1   Assert error in vtc_gunzip(), src/vtc_gzip.c line 114:
>

did you clone vtest recently ? some gunzip asserts were fixed like 1 or 2
months ago.
if it is fresh master, error most probably should be reported to vtest


> Condition(vz.total_out < l) not true.  Errno=0 Success
>  s1   HTTP header is incomplete
> make: *** [Makefile:1053: reg-tests] Error 1
>
> I build with:
>
> make -j5 CPU="generic" TARGET="linux-glibc" USE_OPENSSL=1 USE_PCRE=1
> USE_PCRE_JIT=1 USE_ZLIB=1 USE_GETADDRINFO=1 USE_LINUX_TPROXY=1
>

USE_LINUX_TPROXY is automatic. no need to specify


> ADDLIB="-ljemalloc" DEFINE=-DTCP_USER_TIMEOUT=18 USE_SYSTEMD=1
> USE_TFO=1 USE_NS=1
> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"
>
> I will spend more time later if nobody does.
> --
> William
>
>


Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-07 Thread William Dauchy
On Wed, Jan 6, 2021 at 5:45 PM Willy Tarreau  wrote:
> HAProxy 2.4-dev5 was released on 2021/01/06. It added 91 new commits
> after version 2.4-dev4.

unclear whether this is on my side only because I did not investigate
but I have two tests failing for a while now:

## Starting vtest ##
Testing with haproxy version: 2.4-dev5-761d64-4
#top  TEST reg-tests/filters/random-forwarding.vtc FAILED (0.192) exit=2
#top  TEST reg-tests/seamless-reload/abns_socket.vtc FAILED (2.215) exit=2
2 tests failed, 0 tests skipped, 106 tests passed
## Gathering results ##
## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
## test results in:
"/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.67e5fada"
 h1   Bad exit status: 0x exit 0x0 signal 0 core 0
## Test case: reg-tests/filters/random-forwarding.vtc ##
## test results in:
"/tmp/haregtests-2021-01-07_10-59-10.xOIJpc/vtc.115543.72f8fd5f"
 c1   Assert error in vtc_gunzip(), src/vtc_gzip.c line 114:
Condition(vz.total_out < l) not true.  Errno=0 Success
 s1   HTTP header is incomplete
make: *** [Makefile:1053: reg-tests] Error 1

I build with:

make -j5 CPU="generic" TARGET="linux-glibc" USE_OPENSSL=1 USE_PCRE=1
USE_PCRE_JIT=1 USE_ZLIB=1 USE_GETADDRINFO=1 USE_LINUX_TPROXY=1
ADDLIB="-ljemalloc" DEFINE=-DTCP_USER_TIMEOUT=18 USE_SYSTEMD=1
USE_TFO=1 USE_NS=1
EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o"

I will spend more time later if nobody does.
-- 
William



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-06 Thread Willy Tarreau
On Wed, Jan 06, 2021 at 06:32:14PM +0100, Tim Düsterhus wrote:
> Willy,
> 
> Am 06.01.21 um 17:43 schrieb Willy Tarreau:
> > important factor. Finally responses using unknown encodings are not
> > cached anymore (the list of supported ones is already wide and easy
> > to extend).
> > 
> 
> I don't think that is the case (yet). To the best of my knowledge
> unknown encodings still use the crc32. The hash is not used for known
> encodings, though.

Ah you might be right. Between the discussions, the reviews and the merges,
I end up not remembering the exact current state.

Willy



Re: [ANNOUNCE] haproxy-2.4-dev5

2021-01-06 Thread Tim Düsterhus
Willy,

Am 06.01.21 um 17:43 schrieb Willy Tarreau:
> important factor. Finally responses using unknown encodings are not
> cached anymore (the list of supported ones is already wide and easy
> to extend).
> 

I don't think that is the case (yet). To the best of my knowledge
unknown encodings still use the crc32. The hash is not used for known
encodings, though.

Best regards
Tim Düsterhus



[ANNOUNCE] haproxy-2.4-dev5

2021-01-06 Thread Willy Tarreau
Hi,

HAProxy 2.4-dev5 was released on 2021/01/06. It added 91 new commits
after version 2.4-dev4.

This version was mostly focused on new features, but a few bugs were
also addressed:

  - Fred's experimental QUIC code made its entrance! OK OK OK please calm
down, it's just a part of the code that's needed to get merged to
continue the required infrastructure changes and there is absolutely
nothing functional at this step. I think at best it will handle a
handshake. But these elements are terribly important to continue the
parallel work on connections and muxes so they're better here than
out-of-tree.

  - Rémi fixed the early issues reported by Tim on the handling of
accept-encoding and Vary and improved the performance of the header
processing. In addition, the header normalization should mechanically
result in an increased cache hit ratio for those for whom this is an
important factor. Finally responses using unknown encodings are not
cached anymore (the list of supported ones is already wide and easy
to extend).

  - Thayne McCombs implemented the ability to configure stickiness on the
servers' addresses instead of just their ID or name, which will allow
to persist connections over clusters even when DNS is involved and
dynamic cookies are not usable for whatever reason.

  - Christopher finally found the cause of the corrupted stats output that
a few already noticed and reported (there was a non-thread-safe variable
in use in the middle of the chain, which indicates that those suffering
from this issue are dumping stats from multiple points at the same time,
possibly from various bots).

  - a build error triggered by gcc 11 was worked around by slightly changing
the code (this way there's no pressure and the issue can be discussed
calmly with the gcc team)

  - a significant amount of tree-wide code and doc cleanups was contributed
by Tim and Ilya

  - Dragan upgraded XXHash to v0.8.0 to use the faster and even better XXH3.
All exposed occurrences continue to use XXH2 however (e.g. converters,
dynamic cookies etc).

  - Tim improved the makefile's help message to try to give more hints to
the user about suggested build options. We've indeed seen a few times
some users forgetting to enable SSL and admittedly it's not trivial to
guess when you don't know where to start from.

  - Olivier fixed an interesting issue on the MacOS assembler which uses
the semi-colon as a comment starter (like the old DOS-based assemblers)
while other forms tend to use it as an instruction delimiter (which I
used to ignore). This caused some recent issues on the new Macs with
the M1 CPU where the double-word CAS was causing an endless loop.

  - Amaury allowed http-checks to set the Connection header so that it
becomes possible to send WebSocket health checks now, and fixed two
recently issues (crash with pool-max-conn 0 and disabled backends).

  - Another small change for those often debugging using strace, a very
very long time ago, before the dinosaurs' fate, we used to force the
poller to wake up every second to check the proxies state.  This is
long gone but the wake up every second remained. When running haproxy
under "strace -f" with 20 threads, it was quite annoying to see plenty
of lines scroll all over the screen. And probably that in some VMs it
would cause a small but measurable CPU usage for a totally idle
system. This could have been completely removed but this frequent
wakeup is also used to better detect and correct time drift in VMs.
So the maximum sleep delay was increased to 60 seconds. This will
still allow to correct serious time drifts and drastically reduce the
unneeded wakeups on idle systems.

  - and a long tail of janitor stuff

It's fun to see that during this end-of-year period, while the usual
suspects were almost absent from the changelog, the usually more discrete
ones were very active, with Fred being far ahead with 36 patches! My
obvious conclusion is that we should take vacation more often :-)

Jokes aside, a few of us are currently busy eliminating recently reported
problems and backporting the missing fixes to issue a new set of stable
releases. I got a private report of at least one isolated issue still
affecting 2.2 which doesn't look like a recently fixed one but overall
it's rather clean.

As soon as I find enough time I'll do another set of 1.7 and 1.6 versions
with pending important fixes and close 1.6 as planned since 2020-Q4 is
behind us (and is encouraged to stay far away). I've heard plenty of times
from various people that 1.6 used to be "the best one". Let's allow it to
end its life in peace with all the fixes it deserves. As usual I don't
count on any of the rare users to upgrade that late, but sometimes it can
help a few to smoothen an upgrade.

Ah, last minute report, Christopher noticed that some