Re: Understanding resolvers usage
On Fri, Mar 20, 2020 at 5:02 PM Veiko Kukk wrote: > Hi > > I'd like to have better understanding how server-template and resolvers > work together. HAproxy 1.9.14. > > Relevant sections from config: > > resolvers dns >accepted_payload_size 1232 >parse-resolv-conf >hold valid 90s >resolve_retries 3 >timeout resolve 1s >timeout retry 1s > > server-template srv 4 _foo._tcp.server.name.tld ssl check resolvers dns > resolve-prefer ipv4 resolve-opts prevent-dup-ip > > After some time, when I check statistics from socket: > > echo "show resolvers" |/usr/bin/socat /var/run/haproxy.sock.stats1 stdio > > Resolvers section dns > nameserver 127.0.0.1: >sent:33508 >snd_error: 0 >valid: 33502 >update: 2 >cname: 0 >cname_error: 0 >any_err: 0 >nx: 0 >timeout: 0 >refused: 0 >other: 0 >invalid: 0 >too_big: 0 >truncated: 0 >outdated:6 > nameserver 8.8.8.8: >sent:33508 >snd_error: 0 >valid: 0 >update: 0 >cname: 0 >cname_error: 0 >any_err: 0 >nx: 0 >timeout: 0 >refused: 0 >other: 0 >invalid: 0 >too_big: 0 >truncated: 0 >outdated:33508 > nameserver 8.8.4.4: >sent:33508 >snd_error: 0 >valid: 0 >update: 0 >cname: 0 >cname_error: 0 >any_err: 0 >nx: 0 >timeout: 0 >refused: 0 >other: 0 >invalid: 0 >too_big: 0 >truncated: 0 >outdated:33508 > nameserver 64.6.64.6: >sent:33508 >snd_error: 0 >valid: 6 >update: 0 >cname: 0 >cname_error: 0 >any_err: 0 >nx: 0 >timeout: 0 >refused: 0 >other: 0 >invalid: 0 >too_big: 0 >truncated: 0 >outdated:33502 > > What I wonder about here is why are all nameservers used instead of only > the first one when there are no issues/errors with local caching server > 127.0.0.1:53. From the statistics, the 'sent:' value leaves me > impression that all DNS servers get all requests. I that true? > > /etc/resolv.conf itself: > > nameserver 127.0.0.1 > > nameserver 8.8.8.8 > nameserver 8.8.4.4 > nameserver 64.6.64.6 > > options timeout:1 attempts:2 > > I'd like to achieve situation where other nameservers would be used only > when local caching server fails. Don't want to manually configure only > local one in resolvers section (no failover) and would very much prefer > not to duplicate name server config in resolv.conf and HAproxy config. > > -- > Veiko > > > Hi Veiko You are correct, all servers are queried at the same time and we pick up the fastest non-error response. Other responses will be simply ignored. So if your local cache answers faster than google DNS servers, then you're already covered. Baptiste
[PATCH] fix memory leak, issue 559
Hello, I attached patch that fixes memory leak, described in #559 Cheers, Ilya Shipitcin From a52fad9b32959afc78b4e3086b4d3e17458d19b1 Mon Sep 17 00:00:00 2001 From: Ilya Shipitsin Date: Sat, 21 Mar 2020 19:21:10 +0500 Subject: [PATCH] BUG/MINOR: src/ssl_sock.c: free "find_chain" upon exit non-recurring memory leak was detected in issue #559 --- src/ssl_sock.c | 4 1 file changed, 4 insertions(+) diff --git a/src/ssl_sock.c b/src/ssl_sock.c index 9b44e9d0b..efd464255 100644 --- a/src/ssl_sock.c +++ b/src/ssl_sock.c @@ -3702,6 +3702,10 @@ static int ssl_sock_put_ckch_into_ctx(const char *path, const struct cert_key_an #endif end: + +if (find_chain) +sk_X509_pop_free(find_chain, X509_free); + return errcode; } -- 2.25.1
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 03:13:07PM +0500, ??? wrote: > > Really you *need* to have some response data, not just an empty 200. > > Probably that you can do it with something like this to build 10kB of > > garbage: > > > > am I correct, dealing with large blocks is something HPACK related ? No it's unrelated. HPACK only deals with headers encoding. Here it's more about making sure there are still bytes on the wire to be sent when a window is reopening after a WINDOW_UPDATE etc. > so, for example, if we decide, we can split into several steps, like http2, > HPACK, ... Quite frankly given that a generic config works fine there's no point in having distinct setups for all the tests, it would be a burden to maintain and would cause extra confusion. > h2spec can report in junit format. but no CI can import it (well, gitlab-ci > can, but I did not try). > I'd actually prefer to see history test by test (and for reg-tests as well) By default in verbose more you have all the output and a summary of failed tests at the end, which is very easy to read. > the most we can do (and it is relatively cheap) is to define several steps > in github actions. > like, I already done for "build" and "download h2spec", we can define > several steps for h2spec itself. I think it's asking for more difficulties and pain than really needed. The normal case is that h2spec should work fine so we don't need to have details about what fails, if it fails we can check the output and see what test failed. You'll never know why anyway, it will always require manual attempts to reproduce. Just my two cents, Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:59, Willy Tarreau : > On Sat, Mar 21, 2020 at 02:51:16PM +0500, ??? wrote: > > ??, 21 ???. 2020 ?. ? 14:29, Willy Tarreau : > > > > > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > > > how do you call h2spec ? > > > > > > Like this (for example): > > > > > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > > > > > > > indeed. 146/146 > > > > https://github.com/chipitsine/haproxy/runs/523821049 > > > > ok, I'll try to run it multiple times to figure out how stable it is > > Really you *need* to have some response data, not just an empty 200. > Probably that you can do it with something like this to build 10kB of > garbage: > am I correct, dealing with large blocks is something HPACK related ? so, for example, if we decide, we can split into several steps, like http2, HPACK, ... h2spec can report in junit format. but no CI can import it (well, gitlab-ci can, but I did not try). I'd actually prefer to see history test by test (and for reg-tests as well) the most we can do (and it is relatively cheap) is to define several steps in github actions. like, I already done for "build" and "download h2spec", we can define several steps for h2spec itself. ok, I'll try it. > ># 40B >http-request set-var(req.v) > str(0123456789.123456789.123456789.123456789) ># 160B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 640B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 2560B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) ># 10240B >http-request set-var(req.v) > var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) >http-request return status 200 content-type text/plain lf-string > %[var(req.v)] > > Note that I didn't test it, and might have written a bit of crap, but you > get the idea. > > Willy >
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 02:51:16PM +0500, ??? wrote: > ??, 21 ???. 2020 ?. ? 14:29, Willy Tarreau : > > > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > > how do you call h2spec ? > > > > Like this (for example): > > > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > > > > indeed. 146/146 > > https://github.com/chipitsine/haproxy/runs/523821049 > > ok, I'll try to run it multiple times to figure out how stable it is Really you *need* to have some response data, not just an empty 200. Probably that you can do it with something like this to build 10kB of garbage: # 40B http-request set-var(req.v) str(0123456789.123456789.123456789.123456789) # 160B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 640B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 2560B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) # 10240B http-request set-var(req.v) var(req.v),concat(,req.v),concat(,req.v),concat(,req.v) http-request return status 200 content-type text/plain lf-string %[var(req.v)] Note that I didn't test it, and might have written a bit of crap, but you get the idea. Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:29, Willy Tarreau : > On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > > how do you call h2spec ? > > Like this (for example): > > $ h2spec -Svtk -h 127.0.0.1 -p 4443 > indeed. 146/146 https://github.com/chipitsine/haproxy/runs/523821049 ok, I'll try to run it multiple times to figure out how stable it is > > > in my case, I'm getting 95/95 tests. > > OK you only have the H2 tests, it doesn't test HPACK nor the generic > part, you just have not to pass "http2" and it will run everything. I > don't exactly remember the difference between http2 and generic though, > I think only one of them checks the framing. > > > of course we do not have to run all tests. we can run just specific > > We could indeed drop unreliable ones but I'd rather make sure that we're > in good condition to trust them. > > Willy >
Re: Segfault on 2.1.3
On Sat, Mar 21, 2020 at 10:08:15AM +0100, Willy Tarreau wrote: > On Fri, Mar 20, 2020 at 08:10:25AM -0600, Sean Reifschneider wrote: > > I grabbed the source from the PPA and rebuilt it, installed the dbg > > package, and here's one of the "bt full"s: > > Thanks! > > > (gdb) bt full > > #0 pattern_exec_match (head=head@entry=0x55e4dd275478, > > smp=smp@entry=0x7fbf9ef650c0, > > fill=fill@entry=0) at src/pattern.c:2541 > > __pl_l = > > __pl_r = > > list = 0x0 > > pat = > > This is very strange. The "list" field is null for the expression. That > doesn't make much sense in a linked list. This makes me suspect that the > previous element was added then freed without being unlinked and was then > reused and zeroed. > > I wanted to issue dev5 right now but I'll first try to figure if this is > reproducible and if so, how. I obviously can't reproduce it and the only line in your config making use of L4 rules is perfectly fine and straightforward. Thus I'm having two questions: - is it the new or the old process that occasionally crashes on reload ? If it's the new one, the service is down. If it's the old one, the service continues and you only know about it from your logs. - do you have anything that tries to update the "rate_whitelist" ACL over the stats socket ? We could for example imagine that you're maintaining a whitelist in a separate file that you're uploading upon reloads. Thanks, Willy
Re: running h2spec in CI ?
On Sat, Mar 21, 2020 at 02:04:04PM +0500, ??? wrote: > how do you call h2spec ? Like this (for example): $ h2spec -Svtk -h 127.0.0.1 -p 4443 > in my case, I'm getting 95/95 tests. OK you only have the H2 tests, it doesn't test HPACK nor the generic part, you just have not to pass "http2" and it will run everything. I don't exactly remember the difference between http2 and generic though, I think only one of them checks the framing. > of course we do not have to run all tests. we can run just specific We could indeed drop unreliable ones but I'd rather make sure that we're in good condition to trust them. Willy
Re: Segfault on 2.1.3
On Fri, Mar 20, 2020 at 08:10:25AM -0600, Sean Reifschneider wrote: > I grabbed the source from the PPA and rebuilt it, installed the dbg > package, and here's one of the "bt full"s: Thanks! > (gdb) bt full > #0 pattern_exec_match (head=head@entry=0x55e4dd275478, > smp=smp@entry=0x7fbf9ef650c0, > fill=fill@entry=0) at src/pattern.c:2541 > __pl_l = > __pl_r = > list = 0x0 > pat = This is very strange. The "list" field is null for the expression. That doesn't make much sense in a linked list. This makes me suspect that the previous element was added then freed without being unlinked and was then reused and zeroed. I wanted to issue dev5 right now but I'll first try to figure if this is reproducible and if so, how. Thanks, Willy
Re: running h2spec in CI ?
сб, 21 мар. 2020 г. в 14:00, Willy Tarreau : > Hi Ilya, > > On Sat, Mar 21, 2020 at 01:44:40AM +0500, ??? wrote: > > Hello, > > > > I played with "special purpose" job, which runs h2spec > > > > here's code: > > > > > https://github.com/chipitsine/haproxy/commit/8c90ea82fd32c0ca9bd3df0ae7d9361525eda590 > > > > output: > > > > https://github.com/chipitsine/haproxy/runs/522959386 > > > > > > I think such jobs might be run on schedule, for example weekly ? > > I'm hesitating. While h2spec is a fantastic tool to detect some breakage, > it also relies on extremely precise behaviors and timing. Typically I > never managed to get it to work reproducibly by sending less than 8kB or > so of response data. This is related to the fact that it will for example > send an RST_STREAM after a request and will check if some data flow back, > which will essentially depend on the bytes in flight (hence bandwidth times > latency) between h2spec, haproxy and the server. That's typically what > makes > the success rate vary from 141 to 146 tests for me. > how do you call h2spec ? in my case, I'm getting 95/95 tests. of course we do not have to run all tests. we can run just specific > > Now that we have the return directive we could imagine creating a second > layer and returning a large response there. But as you see I'm slightly > worried of having to deal with even more false positives while we haven't > still completely addressed the abns+reload randomness :-/ > > What's others' opinion on this ? > > Thanks, > Willy >
Re: running h2spec in CI ?
Hi Ilya, On Sat, Mar 21, 2020 at 01:44:40AM +0500, ??? wrote: > Hello, > > I played with "special purpose" job, which runs h2spec > > here's code: > > https://github.com/chipitsine/haproxy/commit/8c90ea82fd32c0ca9bd3df0ae7d9361525eda590 > > output: > > https://github.com/chipitsine/haproxy/runs/522959386 > > > I think such jobs might be run on schedule, for example weekly ? I'm hesitating. While h2spec is a fantastic tool to detect some breakage, it also relies on extremely precise behaviors and timing. Typically I never managed to get it to work reproducibly by sending less than 8kB or so of response data. This is related to the fact that it will for example send an RST_STREAM after a request and will check if some data flow back, which will essentially depend on the bytes in flight (hence bandwidth times latency) between h2spec, haproxy and the server. That's typically what makes the success rate vary from 141 to 146 tests for me. Now that we have the return directive we could imagine creating a second layer and returning a large response there. But as you see I'm slightly worried of having to deal with even more false positives while we haven't still completely addressed the abns+reload randomness :-/ What's others' opinion on this ? Thanks, Willy
