Re: [ANNOUNCE] haproxy-1.9-dev3
> On 29 Sep 2018, at 21:41, Willy Tarreau wrote: > > Ah, a small change is that we now build with -Wextra after having addressed > all warnings reported up to gcc 7.3 and filtered a few useless ones. Hello, here are some warnings from clang version 6.0.0: cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/cfgparse.o src/cfgparse.c src/cfgparse.c:5131:34: warning: implicit conversion from 'int' to 'char' changes value from 130 to -126 [-Wconstant-conversion] curproxy->check_req[5] = 130; ~ ^~~ src/cfgparse.c:5157:33: warning: implicit conversion from 'int' to 'char' changes value from 128 to -128 [-Wconstant-conversion] curproxy->check_req[5] = 128; ~ ^~~ cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/stick_table.o src/stick_table.c src/stick_table.c:2018:14: warning: equality comparison with extraneous parentheses [-Wparentheses-equality] if ((stkctr == )) ~~~^ src/stick_table.c:2018:14: note: remove extraneous parentheses around the comparison to silence this warning if ((stkctr == )) ~ ^~ src/stick_table.c:2018:14: note: use '=' to turn this equality comparison into an assignment if ((stkctr == )) ^~ cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/mux_h2.o src/mux_h2.c src/mux_h2.c:3532:195: warning: implicit conversion from enumeration type 'enum h1m_state' to different enumeration type 'enum h1_state' [-Wenum-conversion] ...= %d bytes out (%u in, st=%s, ep=%u, es=%s, h2cws=%d h2sws=%d) data=%u", h2c->st0, h2s->id, size+9, (unsigned int)total, h1_msg_state_str(h1m->state), h1m->err_pos, h1_ms... ~^ cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS-DTPROXY -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/peers.o src/peers.c src/peers.c:253:16: warning: implicit conversion
[ANNOUNCE] haproxy-1.9-dev3
Subject: [ANNOUNCE] haproxy-1.9-dev3
To: haproxy@formilux.org
Hi,
Now that Kernel Recipes is over (it was another awesome edition), I'm back
to my haproxy activities. Well, I was pleased to see that my coworkers
reserved me a nice surprise by fixing the pending bugs that were plaguing
dev2. I should go to conferences more often, maybe it's a message from
them to make me understand I'm disturbing them when I'm at the office ;-)
So I thought that it was a good opportunity to issue dev3 now and make it
what dev2 should have been, and forget that miserable one, eventhough I
was told that I'll soon get another batch of patches to merge, but then
we'll simply emit dev4 so there's no need to further delay pending fixes.
HAProxy 1.9-dev3 was released on 2018/09/29. It added 35 new commits
after version 1.9-dev2.
There's nothing fancy here. The connection issues are supposedly addressed
(please expect a bit more in this area soon). The HTTP/1 generic parser is
getting smarter since we're reimplementing the features that were in the
old HTTP code (content-length and transfer-encoding now handled). Lua now
can access stick-tables. I haven't checked precisely how but I saw that
Adis updated the doc so all info should be there.
Ah, a small change is that we now build with -Wextra after having addressed
all warnings reported up to gcc 7.3 and filtered a few useless ones. If you
get some build warnings, please report them along with your gcc version and
your build options. I personally build with -Werror in addition to this one,
and would like to keep this principle to catch certain bugs or new compiler
jokes earlier in the future.
As usual, this is an early development version. It's fine if you want to
test the changes, but avoid putting this into production if it can cost
you your job!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse: http://discourse.haproxy.org/
Sources : http://www.haproxy.org/download/1.9/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog: http://www.haproxy.org/download/1.9/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Adis Nezirovic (1):
MEDIUM: lua: Add stick table support for Lua.
Bertrand Jacquin (1):
DOC: Fix typos in lua documentation
Christopher Faulet (3):
MINOR: h1: Add H1_MF_XFER_LEN flag
BUG/MEDIUM: h1: Really skip all updates when incomplete messages are
parsed
BUG/MEDIUM: http: Don't parse chunked body if there is no input data
Dragan Dosen (1):
BUG/MEDIUM: patterns: fix possible double free when reloading a pattern
list
Moemen MHEDHBI (1):
DOC: Update configuration doc about the maximum number of stick counters.
Olivier Houchard (4):
BUG/MEDIUM: process_stream: Don't use si_cs_io_cb() in process_stream().
MINOR: h2/stream_interface: Reintroduce te wake() method.
BUG/MEDIUM: h2: Wake the task instead of calling h2_recv()/h2_process().
BUG/MEDIUM: process_stream(): Don't wake the task if no new data was
received.
Willy Tarreau (24):
BUG/MINOR: h1: don't consider the status for each header
MINOR: h1: report in the h1m struct if the HTTP version is 1.1 or above
MINOR: h1: parse the Connection header field
MINOR: http: add http_hdr_del() to remove a header from a list
MINOR: h1: add headers to the list after controls, not before
MEDIUM: h1: better handle transfer-encoding vs content-length
MEDIUM: h1: deduplicate the content-length header
CLEANUP/CONTRIB: hpack: remove some h1 build warnings
BUG/MINOR: tools: fix set_net_port() / set_host_port() on IPv4
BUG/MINOR: cli: make sure the "getsock" command is only called on
connections
MINOR: stktable: provide an unchecked version of stktable_data_ptr()
MINOR: stream-int: make si_appctx() never fail
BUILD: ssl_sock: remove build warnings on potential null-derefs
BUILD: stats: remove build warnings on potential null-derefs
BUILD: stream: address null-deref build warnings at -Wextra
BUILD: http: address a couple of null-deref warnings at -Wextra
BUILD: log: silent build warnings due to unchecked __objt_{server,applet}
BUILD: dns: fix null-deref build warning at -Wextra
BUILD: checks: silence a null-deref build warning at -Wextra
BUILD: connection: silence a couple of null-deref build warnings at
-Wextra
BUILD: backend: fix 3 build warnings related to null-deref at -Wextra
BUILD: sockpair: silence a build warning at -Wextra
BUILD: build with -Wextra and sort out certain warnings
BUG/CRITICAL: hpack: fix improper sign check on the header index value
---
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi Willy,
I thought lets give those reg-test another try :) as its easy to run and
dev3 just came out.
All tests pass on my FreeBSD system, except this one, new reg-test attached.
Pretty much the same test as previously send, but now with only 4 x 10
connections. Which should be fine for conntrack and sysctls (i hope..).
It seems those stats numbers are 'off', or is my expected value not as
fixed as i thought it would be?
Tested with:
HA-Proxy version 1.9-dev3-27010f0 2018/09/29
FreeBSD freebsd11 11.1-RELEASE
Results:
h1 0.0 CLI recv|CumConns: 33
h1 0.0 CLI recv|CumReq: 65
h1 0.0 CLI expect failed ~ "CumConns: 41"
If my 'expect' is correct, would the patch be suitable for inclusion
with the other reg-tests this way?
If you want to rename loadtest, to heavytest, or any other tweaks please
feel free to do so.
Regards,
PiBa-NL (Pieter)
Op 20-9-2018 om 22:25 schreef PiBa-NL:
Hi Willy,
Op 20-9-2018 om 13:56 schreef Willy Tarreau:
For me the test produces like 345 lines of output as attached. which seems
not to bad (if the test succeeds).
It's already far too much for a user.
Well those 345 lines are if it succeeds while in 'verbose' mode, in
'normal' mode it only produces 1 line of output when successful.
Pretty much all tests produce 100+ lines of 'logging' if they fail for
some reason. From what ive seen varnishtest either produces a bulk of
logging on a failure, or it only logs the failures. There isn't much
in between.
As for all the rest of the email, thanks for your elaborate response :).
Regards,
PiBa-NL (Pieter)
From 28377ffe246ed1db0e0d898fa6263eccdc68c490 Mon Sep 17 00:00:00 2001
From: PiBa-NL
Date: Sat, 15 Sep 2018 01:51:54 +0200
Subject: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
after running a 4 x 10 looping requests
---
reg-tests/loadtest/b0-loadtest.vtc | 99 ++
1 file changed, 99 insertions(+)
create mode 100644 reg-tests/loadtest/b0-loadtest.vtc
diff --git a/reg-tests/loadtest/b0-loadtest.vtc
b/reg-tests/loadtest/b0-loadtest.vtc
new file mode 100644
index ..590924e1
--- /dev/null
+++ b/reg-tests/loadtest/b0-loadtest.vtc
@@ -0,0 +1,99 @@
+# Checks that request and connection counters are properly kept
+
+varnishtest "Connection counters check"
+feature ignore_unknown_macro
+
+server s1 {
+rxreq
+expect req.http.TESTsize == 10
+txresp
+} -repeat 4 -start
+
+syslog Slg_1 -level notice {
+recv
+} -repeat 15 -start
+
+haproxy h1 -W -conf {
+ global
+nbthread 3
+log ${Slg_1_addr}:${Slg_1_port} local0
+#nokqueue
+
+ defaults
+mode http
+option dontlog-normal
+log global
+option httplog
+timeout connect 3s
+timeout client 4s
+timeout server 15s
+
+ frontend fe1
+bind "fd@${fe_1}"
+acl donelooping hdr(TEST) -m len 10
+http-request set-header TEST "%[hdr(TEST)]x"
+use_backend b2 if donelooping
+default_backend b1
+
+ backend b1
+server srv1 ${h1_fe_1_addr}:${h1_fe_1_port}
+
+ frontend fe2
+bind "fd@${fe_2}"
+default_backend b2
+
+ backend b2
+# haproxy 1.8 does not have the ,length converter.
+#acl OK hdr(TEST) -m len 10
+#http-request deny deny_status 200 if OK
+#http-request deny deny_status 400
+
+# haproxy 1.9 does have a ,length converter.
+http-request set-header TESTsize "%[hdr(TEST),length]"
+http-request del-header TEST
+server srv2 ${s1_addr}:${s1_port}
+
+} -start
+
+barrier b1 cond 4
+
+client c1 -connect ${h1_fe_1_sock} {
+ timeout 17
+ barrier b1 sync
+txreq -url "/"
+rxresp
+expect resp.status == 200
+} -start
+client c2 -connect ${h1_fe_1_sock} {
+ timeout 17
+ barrier b1 sync
+txreq -url "/"
+rxresp
+expect resp.status == 200
+} -start
+client c3 -connect ${h1_fe_1_sock} {
+ timeout 17
+ barrier b1 sync
+txreq -url "/"
+rxresp
+expect resp.status == 200
+} -start
+client c4 -connect ${h1_fe_1_sock} {
+ timeout 17
+ barrier b1 sync
+txreq -url "/"
+rxresp
+expect resp.status == 200
+} -start
+
+client c1 -wait
+client c2 -wait
+client c3 -wait
+client c4 -wait
+
+haproxy h1 -cli {
+send "show info"
+expect ~ "CumConns: 41"
+send "show info"
+expect ~ "CumReq: 42"
+}
--
2.18.0.windows.1
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi Pieter, On Sun, Sep 30, 2018 at 12:05:14AM +0200, PiBa-NL wrote: > Hi Willy, > > I thought lets give those reg-test another try :) as its easy to run and > dev3 just came out. > All tests pass on my FreeBSD system, except this one, new reg-test attached. > > Pretty much the same test as previously send, but now with only 4 x 10 > connections. Which should be fine for conntrack and sysctls (i hope..). It > seems those stats numbers are 'off', or is my expected value not as fixed as > i thought it would be? Well, at least it works fine on 1.8 and not on 1.9-dev3 so I think you spotted a regression that we have to analyse. However, I'd like to merge the fix before merging the regtest otherwise it will kill the reg-test feature until we manage to get the issue fixed! I'm also seeing that you rely on threads, I think I noticed another test involving threads. Probably that we should have a specific directory for these ones that we can disable completely when threads are not enabled, otherwise this will also destroy tests (and make them extremely slow due to varnishtest waiting for the timeout if haproxy refuses to parse the config). I think that we should think a bit forward based on these tests. We must not let varnishtest stop on the first error but rather just log it. Then at the end we could produce a report of successes and failures that would be easy to diff from the previous (or expected) one. That will be particularly useful when running the tests on older releases. As an example, I had to run your test manually on 1.8 because for I-don't-know- what-reason, the one about the proxy protocol now fails while it used to work fine last week for the 1.8.14 release. That's a shame that we can't complete tests just because one randomly fails. Thanks, Willy
Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
On Sun, Sep 30, 2018 at 07:46:24AM +0200, Willy Tarreau wrote: > Well, at least it works fine on 1.8 and not on 1.9-dev3 so I think you > spotted a regression that we have to analyse. However, I'd like to merge > the fix before merging the regtest otherwise it will kill the reg-test > feature until we manage to get the issue fixed! By the way, could you please explain in simple words the issue you've noticed ? I tried to reverse the vtc file but I don't understand the details nor what it tries to achieve. When I'm running a simple test on a simple config, the CummConns always matches the CumReq, and when running this test I'm seeing random values there in the output, but I also see that they are retrieved before all connections are closed, so I'm not even sure the test is correct :-/ Thanks, Willy
Re: H2O - an optimized HTTP server
Hi. Am 29.09.2018 um 07:33 schrieb Dave Cottlehuber: > On Sat, 29 Sep 2018, at 00:31, Aleksandar Lazic wrote: >> Hi. >> >> Have anyone used this server in production setup behind haproxy? >> >> https://h2o.examp1e.net/ > > Yes for the last 2 years at least. but from a pure speed and http2 perspective > you’re best off running them beside each other. It’s solid web server and the > embedded mruby is very useful but its proxy support is primitive still. I use > its OCSP script to handle things for haproxy though. thanks for answer. Just to get you right the setup looks like this. client -> www.company.com (h2o) -> files -> anything else (haproxy) -> backends > A+ > Dave Regards Aleks
Re: [PATCH] MEDIUM: lua: Add stick table support for Lua
Hi Adis, On Thu, Sep 27, 2018 at 05:32:22PM +0200, Adis Nezirovic wrote: > On Thu, Sep 27, 2018 at 04:52:29PM +0200, Thierry Fournier wrote: > > I Adis, > > > > Sorry for the delay, I processed a quick review, and all seems to be ok for > > me! > > > > BR, > > Thierry > > Great, happy to hear that, I hope guys will merge it soon. OK, just merged now. Thanks to you both! Willy
Re: [ANNOUNCE] haproxy-1.9-dev3
Hi Dmitry, On Sat, Sep 29, 2018 at 11:05:19PM +0300, Dmitry Sivachenko wrote: > > > On 29 Sep 2018, at 21:41, Willy Tarreau wrote: > > > > Ah, a small change is that we now build with -Wextra after having addressed > > all warnings reported up to gcc 7.3 and filtered a few useless ones. > > Hello, > > here are some warnings from clang version 6.0.0: > > cc -Iinclude -Iebtree -Wall -Wextra -O2 -pipe -fstack-protector > -fno-strict-aliasing -fno-strict-aliasing -Wdeclaration-after-statement > -fwrapv -fno-strict-overflow -Wno-address-of-packed-member -Wno-unused-label > -Wno-sign-compare -Wno-unused-parameter -Wno-ignored-qualifiers > -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wtype-limits > -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS-DTPROXY > -DCONFIG_HAP_CRYPT -DUSE_GETADDRINFO -DUSE_ZLIB -DENABLE_POLL > -DENABLE_KQUEUE -DUSE_CPU_AFFINITY -DUSE_ACCEPT4 -DCONFIG_REGPARM=3 > -DUSE_THREAD -DUSE_OPENSSL -I/usr/include -DUSE_PCRE -I/usr/local/include > -DUSE_PCRE_JIT -DCONFIG_HAPROXY_VERSION=\"1.9-dev3\" > -DCONFIG_HAPROXY_DATE=\"2018/09/29\" -c -o src/cfgparse.o src/cfgparse.c > src/cfgparse.c:5131:34: warning: implicit conversion from 'int' to 'char' > changes value from 130 to -126 [-Wconstant-conversion] > > curproxy->check_req[5] = 130; (...) Oh yes I remember seeing these ones reported once, as well as in peers.c. > src/stick_table.c:2018:14: warning: equality comparison with extraneous > parentheses [-Wparentheses-equality] > if ((stkctr == )) > ~~~^ > src/stick_table.c:2018:14: note: remove extraneous parentheses around the > comparison to silence this warning > if ((stkctr == )) > ~ ^~ > src/stick_table.c:2018:14: note: use '=' to turn this equality comparison > into an assignment > if ((stkctr == )) > ^~ Interesting ones! I admit that without context around they clearly look suspicious, though we tend to know this code works. These ones are easy to fix. Thank you! Willy
Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?
Hello all! I've played with `tune.rcvbuf.server`, `tune.sndbuf.server`, `tune.rcvbuf.client`, and `tune.sndbuf.client` and explicitly set them to various values ranging from 4k to 256k. Unfortunately in all cases it seems that this generates too large TCP packets (larger than the advertised and agreed MSS in both direction), which in turn leads to TCP fragmentation and reassembly. (Both client and server are Linux >4.10. The protocol used was HTTP 1.1 over TLS 1.2.) The resulting outcome was a bandwidth of about 100 KB (for a client-server latency of 160ms). The only setting that din't have this effect was not to set them. The resulting bandwidth was around 10 MB. (I have tested the backend server without HAProxy, in fact two different webservers, both with and without HAProxy, and I would exclude them as the issue.) Thus I was wondering if anyone encountered similar issues and how they've fixed it. (My guess is that it's more due to the Linux TCP implementation stack than from HAProxy.) As a sidenote is the following interpretation correct: * `tune.*buf.server` refers to the TCP sockets that the frontend binds to and listens for actual clients; * `tune.*buf.client` refers to the TCP sockets that the backend creates and connects to the actual servers; Thanks, Ciprian.
