Re: [ANNOUNCE] haproxy-2.0-dev7
Am 11.06.2019 um 22:01 schrieb Willy Tarreau:
> On Tue, Jun 11, 2019 at 09:42:13PM +0200, Aleksandar Lazic wrote:
>> Sorry to say that but there is no dev7 and in
>> http://www.haproxy.org/download/2.0/src/devel/
>> also not.
>
> Ah silly me! And who forgot to run "pubish-release" in your opinion ? :-)
The cat which hasn't passed the keyboard ;-)
> It's OK now. Thanks for reporting this!
Perfect.
Build log: https://gitlab.com/aleks001/haproxy20-centos/-/jobs/229375977
Image with tls 1.3: https://hub.docker.com/r/me2digital/haproxy20-centos
```
docker run --rm --entrypoint /usr/local/sbin/haproxy [MASKED]/haproxy20-centos
-vv
HA-Proxy version 2.0-dev7 2019/06/11 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_THREAD=1 USE_PTHREAD_PSHARED=1
USE_REGPARM=1 USE_LINUX_SPLICE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1
USE_SLZ=1 USE_TFO=1 USE_NS=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE +PCRE_JIT
-PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD +PTHREAD_PSHARED +REGPARM
-STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT
+CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 -ZLIB
+SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD
-OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.1.1c 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using 'proto' keyword)
h2 : mode=HTXside=FE|BE mux=H2
h2 : mode=HTTP side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services :
prometheus-exporter
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
```
> Willy
>
Re: [ANNOUNCE] haproxy-2.0-dev7
On Tue, Jun 11, 2019 at 10:02:48PM +0200, Tim Düsterhus wrote: > Should I file an issue (referring to this thread) to keep track of the > whole issue and possible solutions? Oh yes please! > In Travis we have a clearly defined environment, so that kind of test > should work there as well. Good point! Tomorrow I'll talk about this one with Fred and we could decide to disable it for the release, then later think how we can add categorize and use such delicate tests. Thanks, Willy
Re: [ANNOUNCE] haproxy-2.0-dev7
Willy,
Am 11.06.19 um 21:58 schrieb Willy Tarreau:
>> So clearly there is no *seamless* reload happening here which is the
>> very thing the test is testing. The question is: Is the issue with the
>> 'abns' socket or is the issue with the 'fd@${testme}' socket? Can
>> fd-sockets be seamlessly passed?
>
> That's an excellent question. I tend to think we cannot pass them by
> definition since the fd is supposed to be passed by the caller in
> this case! So very likely the issue stems from the fact that we're
> relying on passing an fd in a situation where it has no chance of
> being passed due to the way it's configured.
>
> I don't know if we can run a vtest client against an abns address, in
> which case a solution could consist in using exclusively abns sockets
> for the tests. Another solution could consist in not using the client
> at all and relying on health checks instead to send the request. But
> in all cases that's racy.
Should I file an issue (referring to this thread) to keep track of the
whole issue and possible solutions?
>> In any case this is a bad test, because it is inherently racy.
>
> Yes I agree. Typically the type of thing I tend to be cautious about
> in regression testing, I know that when we try to go too far in this
> direction we tend to spend more time and energy fixing problems
> invented from scratch in the test scenarios instead of addressing
> real issues. There's a mid-point to find, which is always difficult.
>
> Note, we could also imagine having a few "unit" tests that require a
> very specific environment and which would be compatible with testing
> by hand (e.g. binding to a fixed port, etc). I see how this could be
> useful.
In Travis we have a clearly defined environment, so that kind of test
should work there as well.
Best regards
Tim Düsterhus
Re: [ANNOUNCE] haproxy-2.0-dev7
On Tue, Jun 11, 2019 at 09:42:13PM +0200, Aleksandar Lazic wrote: > Sorry to say that but there is no dev7 and in > http://www.haproxy.org/download/2.0/src/devel/ > also not. Ah silly me! And who forgot to run "pubish-release" in your opinion ? :-) It's OK now. Thanks for reporting this! Willy
Re: [ANNOUNCE] haproxy-2.0-dev7
On Tue, Jun 11, 2019 at 09:27:37PM +0200, Tim Düsterhus wrote:
> Reading the log I believe this might actually be a real problem within
> HAProxy. Note the log lines starting at 370 in the attached file:
>
> line 370: The old worker starts stopping its proxies (due to having
> received the SIGUSR1 from the re-executed master)
> line 401: The client connects
> line 406: The client starts waiting for a response
> line 408: The old worker stopped all proxies
> line 409: The new worker starts
> line 410: The client receives a connection reset
> line 411: The old worker exits
>
> So clearly there is no *seamless* reload happening here which is the
> very thing the test is testing. The question is: Is the issue with the
> 'abns' socket or is the issue with the 'fd@${testme}' socket? Can
> fd-sockets be seamlessly passed?
That's an excellent question. I tend to think we cannot pass them by
definition since the fd is supposed to be passed by the caller in
this case! So very likely the issue stems from the fact that we're
relying on passing an fd in a situation where it has no chance of
being passed due to the way it's configured.
I don't know if we can run a vtest client against an abns address, in
which case a solution could consist in using exclusively abns sockets
for the tests. Another solution could consist in not using the client
at all and relying on health checks instead to send the request. But
in all cases that's racy.
> In any case this is a bad test, because it is inherently racy.
Yes I agree. Typically the type of thing I tend to be cautious about
in regression testing, I know that when we try to go too far in this
direction we tend to spend more time and energy fixing problems
invented from scratch in the test scenarios instead of addressing
real issues. There's a mid-point to find, which is always difficult.
Note, we could also imagine having a few "unit" tests that require a
very specific environment and which would be compatible with testing
by hand (e.g. binding to a fixed port, etc). I see how this could be
useful.
Willy
Re: [ANNOUNCE] haproxy-2.0-dev7
Am 11.06.2019 um 20:02 schrieb Willy Tarreau: > Hi, > > HAProxy 2.0-dev7 was released on 2019/06/11. It added 34 new commits > after version 2.0-dev6. [snipp] > Please find the usual URLs below : >Site index : http://www.haproxy.org/ >Discourse: http://discourse.haproxy.org/ >Slack channel: https://slack.haproxy.org/ >Issue tracker: https://github.com/haproxy/haproxy/issues >Sources : http://www.haproxy.org/download/2.0/src/ Sorry to say that but there is no dev7 and in http://www.haproxy.org/download/2.0/src/devel/ also not. >Git repository : http://git.haproxy.org/git/haproxy.git/ >Git Web browsing : http://git.haproxy.org/?p=haproxy.git >Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG >Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ > > Willy Regards Aleks
Re: [ANNOUNCE] haproxy-2.0-dev7
Willy,
Am 11.06.19 um 20:53 schrieb Willy Tarreau:
> Hi Tim,
>
> On Tue, Jun 11, 2019 at 08:14:41PM +0200, Tim Düsterhus wrote:
>> Willy,
>>
>> Am 11.06.19 um 20:02 schrieb Willy Tarreau:
>>> I noticed that no regtest failed on Travis since commit 7b3a79f6 which
>>
>> The abns_socket.vtc one is incredibly flaky though. I've noticed two
>> "failed" tests for Travis in in the list of commits today where a single
>> configuration failed to succeed because of this test. I've pushed the
>> button to re-run the failing configurations and they immediately
>> succeeded. We really must find a solution for that one.
>
> Ah that's interesting indeed. I remember seeing it fail very often in
> the past and was almost happy not to see it anymore. Now I understand,
> you're behind the curtains with your crank to restart it :-)
Only today! There's one failed job due to that test from 7 days before,
though: https://travis-ci.com/haproxy/haproxy/jobs/205334815#L529. I've
attached the relevant part of the log for posterity.
I guess if one pushes the "restart" button it will succeed without issue :/
> Yes we definitely need to address this. Either the test is bogus (which
> is always possible) or it reveals a real problem that either we need to
> address or to document if it's a technical limitation, all of which are
> unknown to me at this point.
>
Reading the log I believe this might actually be a real problem within
HAProxy. Note the log lines starting at 370 in the attached file:
line 370: The old worker starts stopping its proxies (due to having
received the SIGUSR1 from the re-executed master)
line 401: The client connects
line 406: The client starts waiting for a response
line 408: The old worker stopped all proxies
line 409: The new worker starts
line 410: The client receives a connection reset
line 411: The old worker exits
So clearly there is no *seamless* reload happening here which is the
very thing the test is testing. The question is: Is the issue with the
'abns' socket or is the issue with the 'fd@${testme}' socket? Can
fd-sockets be seamlessly passed?
In any case this is a bad test, because it is inherently racy.
Best regards
Tim Düsterhus
Test case: reg-tests/seamless-reload/abns_socket.vtc
*top 0.0 TEST reg-tests/seamless-reload/abns_socket.vtc starting
top 0.0 extmacro def pwd=/home/travis/build/haproxy/haproxy
top 0.0 extmacro def no-htx=
top 0.0 extmacro def localhost=127.0.0.1
top 0.0 extmacro def bad_backend=127.0.0.1 46119
top 0.0 extmacro def bad_ip=192.0.2.255
top 0.0 macro def testdir=/home/travis/build/haproxy/haproxy/reg-tests/seamless-reload
top 0.0 macro def tmpdir=/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2
** top 0.0 === varnishtest "Seamless reload issue with abns sockets"
*top 0.0 VTEST Seamless reload issue with abns sockets
** top 0.0 === feature ignore_unknown_macro
** top 0.0 === haproxy h1 -W -conf {
h10.0 macro def h1_cli_sock=127.0.0.1 35461
h10.0 macro def h1_cli_addr=127.0.0.1
h10.0 macro def h1_cli_port=35461
h10.0 setenv(cli, 4)
h10.0 macro def h1_testme_sock=127.0.0.1 43423
h10.0 macro def h1_testme_addr=127.0.0.1
h10.0 macro def h1_testme_port=43423
h10.0 setenv(testme, 5)
** h10.0 haproxy_start
h10.0 opt_worker 1 opt_daemon 0 opt_check_mode 0
h10.0 argv|exec "/home/travis/build/haproxy/haproxy/haproxy" -d -W -f "/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2/h1/cfg" -p "/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2/h1/pid"
h10.0 conf|global
h10.0 conf|\tstats socket "/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2/h1/stats.sock" level admin mode 600
h10.0 conf|stats socket "fd@${cli}" level admin
h10.0 conf|
h10.0 conf| global
h10.0 conf|stats socket "/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2/h1/stats" level admin expose-fd listeners
h10.0 conf|
h10.0 conf| defaults
h10.0 conf|mode http
h10.0 conf| option http-use-htx
h10.0 conf|log global
h10.0 conf|option httplog
h10.0 conf|timeout connect 15ms
h10.0 conf|timeout client 20ms
h10.0 conf|timeout server 20ms
h10.0 conf|
h10.0 conf| listen testme
h10.0 conf|bind "fd@${testme}"
h10.0 conf|server test_abns_server abns@wpproc1 send-proxy-v2
h10.0 conf|
h10.0 conf| frontend test_abns
h10.0 conf|bind abns@wpproc1 accept-proxy
h10.0 conf|http-request deny deny_status 200
h10.0 XXX 7 @637
*** h10.0 PID: 6050
h10.0 macro def h1_pid=6050
h10.0 macro def h1_name=/tmp/haregtests-2019-06-04_14-58-10.evaaZM/vtc.5888.1e49b2c2/h1
*** h10.0 wait-pid-file
*** h1
Re: [ANNOUNCE] haproxy-2.0-dev7
Hi Tim, On Tue, Jun 11, 2019 at 08:14:41PM +0200, Tim Düsterhus wrote: > Willy, > > Am 11.06.19 um 20:02 schrieb Willy Tarreau: > > I noticed that no regtest failed on Travis since commit 7b3a79f6 which > > The abns_socket.vtc one is incredibly flaky though. I've noticed two > "failed" tests for Travis in in the list of commits today where a single > configuration failed to succeed because of this test. I've pushed the > button to re-run the failing configurations and they immediately > succeeded. We really must find a solution for that one. Ah that's interesting indeed. I remember seeing it fail very often in the past and was almost happy not to see it anymore. Now I understand, you're behind the curtains with your crank to restart it :-) Yes we definitely need to address this. Either the test is bogus (which is always possible) or it reveals a real problem that either we need to address or to document if it's a technical limitation, all of which are unknown to me at this point. Willy
Re: [ANNOUNCE] haproxy-2.0-dev7
Willy, Am 11.06.19 um 20:02 schrieb Willy Tarreau: > I noticed that no regtest failed on Travis since commit 7b3a79f6 which The abns_socket.vtc one is incredibly flaky though. I've noticed two "failed" tests for Travis in in the list of commits today where a single configuration failed to succeed because of this test. I've pushed the button to re-run the failing configurations and they immediately succeeded. We really must find a solution for that one. Best regards Tim Düsterhus
[ANNOUNCE] haproxy-2.0-dev7
Hi, HAProxy 2.0-dev7 was released on 2019/06/11. It added 34 new commits after version 2.0-dev6. We're getting pretty good! I really was hesitating about marking it final already or not, but given that we've addressed a few last minute non-trivial issues and that there are still enough code cleanups and doc updates left to keep anyone busy, I thought I'd rather slightly postpone the final release to focus on maximal stability and cleanliness than perfect schedule. In short, there was an issue with the H1 to H2 upgrade making the performance irregular, another issue affecting the end of compression in HTX, a case where H1 could erroneously report being connected on the backend side before the end of the handshake (possibly the one causing me some irregular performance in H2 to H1 tests), the unsafe thread initialization issues, and a more delicate fix for HTX to address situations where buffer defragmentation was sometimes abused resulting in low performance. >From now on I estimate that the code is the code we'll release, more or less the fixes for any bug which will be reported in between. Let's focus solely on tidying the remaining FIXMEs, the doc, and possibly exporting a few more info to the CLI/stats to help with debugging (e.g. a few H1/H2 counters are terribly missing). I noticed that no regtest failed on Travis since commit 7b3a79f6 which concluded the changes for the stacking of the connection layers that begun after the start of the periodic failures which appeared between dev3 and dev4 mainly. Thus I would not completely rule out the possibility that some deeply burried bug in the old connection code was responsible for a part of them. If this is the case, good riddance! Let's set on anything between Friday and Tuesday for the final release, the former if no more bugs are reported, the latter if some are fixed late. Please, really, no more dev for 2.0, if you have anything to submit which doesn't belong to the exceptions above, this will go into -next (which I just rebased now) but I would appreciate it if you could delay your submissions so that we stay focused on 2.0. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy.git Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Christopher Faulet (5): BUG/MINOR: cache/htx: Fix the counting of data already sent by the cache applet BUG/MEDIUM: compression/htx: Fix the adding of the last data block MINOR: flt_trace: Don't scrash the original offset during the random forwarding MAJOR: htx: Rework how free rooms are tracked in an HTX message MINOR: htx: Add the function htx_move_blk_before() Daniel Corbett (4): MINOR: contrib/spoa_server: Upgrade SPOP to 2.0 BUG/MEDIUM: contrib/spoa_server: Set FIN flag on agent frames MINOR: contrib/spoa_server: Add random IP score DOC/MINOR: contrib/spoa_server: Fix typo in README Frédéric Lécaille (6): MINOR peers: data structure simplifications for server names dictionary cache. DOC: peers: Update for dictionary cache entries for peers protocol. MINOR: dict: Store the length of the dictionary entries. MINOR: peers: A bit of optimization when encoding cached server names. MINOR: peers: Optimization for dictionary cache lookup. BUG/MINOR: dict: race condition fix when inserting dictionary entries. Olivier Houchard (6): MINOR: chunks: Make sure trash_size is only set once. BUG/MEDIUM: H1: When upgrading, make sure we don't free the buffer too early. BUG/MEDIUM: stream_interface: Make sure we call si_cs_process() if CS_FL_EOI. Revert "BUG/MEDIUM: H1: When upgrading, make sure we don't free the buffer too early." BUG/MEDIUM: h1: Don't try to subscribe if we had a connection error. BUG/MEDIUM: h1: Don't consider we're connected if the handshake isn't done. Willy Tarreau (13): BUG/MEDIUM: mux-h2: make sure the connection timeout is always set MINOR: tools: add new bitmap manipulation functions MINOR: logs: use the new bitmap functions instead of fd_sets for encoding maps Revert "MINOR: chunks: Make sure trash_size is only set once." MINOR: threads: serialize threads initialization MEDIUM: tools: improve time format error detection MINOR: threads: avoid clearing harmless twice in thread_release() MEDIUM: threads: add thread_sync_release() to synchronize steps BUG/MEDIUM: init/threads: prevent initialized threads from starting before others OP
