Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters

[PiBa-NL]

Hi Willy,

Op 19-9-2018 om 7:36 schreef Willy Tarreau:

Hi Pieter,

I took some time this morning to give it a test. For now it fails here,
after dumping 2200 lines of not really usable output that I didn't
investigate. From what I'm seeing it seems to moderately stress the
local machine so it has many reasons for failing (lack of source
ports, improperly tuned conntrack, ulimit, etc), and it takes far too
long a time to be usable as a default test, or this one alone will be
enough to discourage anyone from regularly running "make reg-tests".
Test takes like 5 seconds to run here, and while that is a bit long if 
you get a hundred more similar tests and want to continue tweaking 
developments while running tests in between. It wouldn't hurt to run 
such a (series) of longer tests before creating a patch and submitting 
it for inclusion on the official git repository in my opinion, or before 
a release.?. My attempt was to test a bit differently than just looking 
for regressions of known fixed bugs, and putting a little load on 
haproxy so that threads and simultaneous actions 'might' get into 
conflicts/locks/stuff which might by chance, show up, which is why i 
choose to go a little higher on the number of round-trips with ever 
slightly increasing payload..


For me the test produces like 345 lines of output as attached. which 
seems not to bad (if the test succeeds).. Besides the 2 instances of cli 
output for stats, its seems not that much different from other tests..
And with 1.8.13 on FreeBSD (without qkueue) it succeeds:  #    top TEST 
./test/b0-loadtest.vtc passed (4.800


Taking into account conntrack and ulimit, would that mean we can never 
'reg-test' if haproxy can really handle like 1 connections without 
issue? Or should the environment be configured by the test?? ,that seems 
very tricky at least and probably would be undesirable.. (i just today 
figured i could run reg-tests also on my production box to compare if a 
new build showed issues there that my test-box might not.. i wouldn't 
want system settings to changed by a reg-test run..)



I think we should create a distinct category for such tests
Agreed, which is why i used the currently non-existing '/loadtest/' 
folder. If '/heavy/' is better thats of course alright for me to.

, because
I see some value in it when it's made to reproduce a very specific
class of issues which is very unlikely to trigger unless someone is
working on it. In this case it is not a problem that it dumps a lot
of output, as it will be useful for the person knowing what to look
for there. Probably that such tests should be run by hand and dump
their log into a related file. Imagine for example that we would
have this :

  $ make reg-tests/heavy/conn-counter-3000-req.log
I'm not exactly sure..("make: don't know how to make reg-tests. Stop"). 
i would still like to have a way to run all 'applicable' tests with 1 
command, even if it takes a hour or so to verify haproxy is working 
'perfectly'. But like abns@ tests cant work on FreeBSD, but they should 
not 'fail', perhaps get skipped automatically though.?. Anyhow thats a 
question for my other mail-topic ( 
https://www.mail-archive.com/haproxy@formilux.org/msg31195.html )

It would run the test on reg-tests/heavy/conn-counter-3000-req.vtc and
would produce the log into reg-tests/heavy/conn-counter-3000-req.log.
We could use a similar thing to test for select/poll/epoll/kqueue, to
test for timeouts, race conditions (eg show sess in threads). This is
very likely something to brainstorm about. You might have other ideas
related to certain issues you faced in the past. Fred is unavailable
this week but I'd be very interested in his opinion on such things.

Thus for now I'm not applying your patch, but I'm interested in seeing
what can be done with it.
Okay no problem :) , ill keep running this particular test myself for 
the moment, it 'should' be able to pass normally..  (On my environment 
anyhow..)

Thanks,
Willy


Thanks for your comments, and thoughts.

I'm interested in Fred's and anyone elses opinion ;) , and well maybe 
this particular test-case could be replaced by something simpler/faster/ 
with more or less the same likelihood of catching yet unknown issues..? 
Looking forward to reactions :) .


Thanks and regards,

PiBa-NL (Pieter)

 top   0.0 extmacro def pwd=/usr/ports/net/haproxy-devel
 top   0.0 extmacro def localhost=127.0.0.1
 top   0.0 extmacro def bad_backend=127.0.0.1 58530
 top   0.0 extmacro def bad_ip=192.0.2.255
 top   0.0 macro def testdir=/usr/ports/net/haproxy-devel/./test
 top   0.0 macro def tmpdir=/tmp/vtc.35996.290f74a9
*top   0.0 TEST ./test/b0-loadtest.vtc starting
**   top   0.0 === varnishtest "Seamless reload issue with abns sockets"
*top   0.0 TEST Seamless reload issue with abns sockets
**   top   0.0 === feature ignore_unknown_macro
**   top   0.0 === server s1 {
**   s10.0 Starting server
 s10.0 macro def 

Re: invalid port numbers in 'show servers state' with SRV records

[francis Lavalliere]

Hello,

In server.c on line 2912


I believe the server state file is unsigned long and when loaded back its
being compared with USHRT_MAX

In my own development environment I had commented the following if
statement, as server state is fully controlled by the haproxy and
everything was working fine. But since this isn't the "Real" fix Ii assume
there will be needed of some kind of conversion to have a valid comparison.


```
if (port > USHRT_MAX) {
chunk_appendf(msg, ", invalid srv_port value '%s'", port_str);
port_str = NULL;
```

I

On Wed, Sep 19, 2018 at 12:32 PM Peter Erickson  wrote:

> Hello - I'm attempting to use HAProxy's service discovery (SRV records)
> along with `load-server-state-from-file` but the port numbers from `show
> servers state` are incorrect.
>
> Here is my example (HA-Proxy version 1.9-dev2 2018/09/12)
>
> DNS configuration:
>
> $ dig SRV _nginx._tcp.alpha.int
> ;; ANSWER SECTION:
> _nginx._tcp.alpha.int.  18  IN  SRV 1 1 32772
> 56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int.
> _nginx._tcp.alpha.int.  18  IN  SRV 1 1 32773
> c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int.
>
> haroxy.cfg:
>
> global
> pidfile /var/run/haproxy.pid
> stats socket /haproxy-run/haproxy.sock
> server-state-file /haproxy-run/haproxy.state
>
> defaults
> modehttp
> timeout connect 1
> timeout client 10
> timeout server 10
> load-server-state-from-file global
>
> resolvers resolver0
> nameserver dns0 172.20.0.2:53
>
> frontend http
> bind *:80
> default_backend b1
>
> backend b1
> server-template www 2 _nginx._tcp.alpha.int check resolvers resolver0
>
> server state:
>
> $ echo "show servers state b1" | sudo socat haproxy-run/haproxy.sock stdio
> 1
> # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state
> srv_uweight srv_iweight srv_time_since_last_change srv_check_status
> srv_check_result srv_check_health srv_check_state srv_agent_state
> bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
> 3 b1 1 www1 172.20.2.109 2 0 1 1 13 6 3 4 6 0 0 0
> 56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int 4294934532
> _nginx._tcp.alpha.int
> 3 b1 2 www2 172.20.0.192 2 0 1 1 12 6 3 4 6 0 0 0
> c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int 4294934533
> _nginx._tcp.alpha.int
>
>
> As you can see, the port number in the SRV records are:
> 32772
> 32773
>
> However, the srv_port value from `show servers state` is:
> 4294934532
> 4294934533
>
> This causes the following error when haproxy attempts to read the state
> file:
> [WARNING] 261/162616 (1) : server-state application failed for server
> 'b1/www1', invalid srv_port value '4294934532'
> [WARNING] 261/162616 (1) : server-state application failed for server
> 'b1/www2', invalid srv_port value '4294934533'
>
> Can anyone help me out?
>
> Thanks,
> Pete
>

invalid port numbers in 'show servers state' with SRV records

[Peter Erickson]

Hello - I'm attempting to use HAProxy's service discovery (SRV records)
along with `load-server-state-from-file` but the port numbers from `show
servers state` are incorrect.

Here is my example (HA-Proxy version 1.9-dev2 2018/09/12)

DNS configuration:

$ dig SRV _nginx._tcp.alpha.int
;; ANSWER SECTION:
_nginx._tcp.alpha.int.  18  IN  SRV 1 1 32772
56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int.
_nginx._tcp.alpha.int.  18  IN  SRV 1 1 32773
c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int.

haroxy.cfg:

global
pidfile /var/run/haproxy.pid
stats socket /haproxy-run/haproxy.sock
server-state-file /haproxy-run/haproxy.state

defaults
modehttp
timeout connect 1
timeout client 10
timeout server 10
load-server-state-from-file global

resolvers resolver0
nameserver dns0 172.20.0.2:53

frontend http
bind *:80
default_backend b1

backend b1
server-template www 2 _nginx._tcp.alpha.int check resolvers resolver0

server state:

$ echo "show servers state b1" | sudo socat haproxy-run/haproxy.sock stdio
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state
srv_uweight srv_iweight srv_time_since_last_change srv_check_status
srv_check_result srv_check_health srv_check_state srv_agent_state
bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord
3 b1 1 www1 172.20.2.109 2 0 1 1 13 6 3 4 6 0 0 0
56b9c088-9bc5-4466-99af-ebebd3550791._nginx._tcp.alpha.int 4294934532
_nginx._tcp.alpha.int
3 b1 2 www2 172.20.0.192 2 0 1 1 12 6 3 4 6 0 0 0
c6032796-8c7a-4d78-8934-d98900c18383._nginx._tcp.alpha.int 4294934533
_nginx._tcp.alpha.int


As you can see, the port number in the SRV records are:
32772
32773

However, the srv_port value from `show servers state` is:
4294934532
4294934533

This causes the following error when haproxy attempts to read the state
file:
[WARNING] 261/162616 (1) : server-state application failed for server
'b1/www1', invalid srv_port value '4294934532'
[WARNING] 261/162616 (1) : server-state application failed for server
'b1/www2', invalid srv_port value '4294934533'

Can anyone help me out?

Thanks,
Pete

Client Timeout - undeterministic behaviour with tcp frontends

[Sven Buesing]

Hi All,

I think we stumbled over a bug in haproxy 1.8.3 regarding "timeout client" and 
tcp frontends.

We are using the following version of haproxy:
HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau 

We have quite a complex setup, as we use a tcp listener for ssl termination 
which forwards the requests to another internal listener via socket where the 
decision for ecc or rsa certificates is made.
After that the request is getting forwarded to an http mode frontend, which 
then decides which backend to use.

Our default "timeout client" is set to 18s.
Default "timeout server" is 5m.

Now comes our problem.
A client connects to our tcp listener; ssl termination etc. went well and the 
request is submitted to our backend server.
After that the backend is computing which takes more than 18s. Sometimes it's 
about 20s, sometimes it's below 18s. However if we consistently create a 
request which response (ttfb) needs more than 18s we can observe a quite 
undeterministic behaviour. 3 out of 10 of those requests get terminated and the 
browser shows ERR:EMPTY_RESPONSE.
If we observe the network traffic we can see that the tcp session is closed by 
the loadbalancer (haproxy). We assume it's haproxy, because setting the 
"timeout client" on the tcp listener to higher values correctly prevents this 
from happening.

The unusual part is, that this behaviour is only reproducible in about 30% of 
requests and the documentation states, that "timeout client" should only apply 
to inactivity when the client is expected to acknowledge or send data, which is 
not the case here as the client already send it's full request and is waiting 
fort he response.

Has anyone already stumbled across this problem, or can explain what the 
problem could be here?

Regards,
Sven

Re: Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections

[Lukas Tribus]

Hello,

On Wednesday, 19 September 2018, Shishir Kumar Yadav <
shis...@purestorage.com> wrote:

> I am able to get logs and I see these errors -
>
> 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for
> backend ir-http-server-backend: no free ports.
>

Make sure you enable net.ipv4.tcp_tw_reuse:

https://vincent.bernat.ch/en/blog/2014-tcp-time-wait-state-linux


Lukas

Re: Where can I find explanations for all the columns of the web interface?

[Aleksandar Lazic]

Hi.

Am 19.09.2018 um 12:01 schrieb Бобровский Дмитрий:
> Hello!
> 
> Where can I find explanations for all the columns of the web interface?

https://cbonte.github.io/haproxy-dconv/1.8/management.html#9

> http://172.29.84.64/haproxy?stats
> 
> Best regards,
> 
> Bobrovsky Dmitry

Regards
Aleks

> *Tel**: *
> 
> +7-(910)-416-85-00
> 
> *EMail: *
> 
> _ministrbob...@gmail.com _
> 
> *Telegram**: *
> 
> _@BobrD _
> 
> *Skype:*
> 
> _MinistrBob _

Where can I find explanations for all the columns of the web interface?

[Бобровский Дмитрий]

Hello !

Where can I find explanations for all the columns of the web interface?

http://172.29.84.64/haproxy?stats

[cid:image001.png@01D45018.EB070510]

Best regards,
Bobrovsky Dmitry


Tel:

+7-(910)-416-85-00

EMail:

ministrbob...@gmail.com

Telegram:

@BobrD

Skype:

MinistrBob