Re: [PATCH] REGTEST/MINOR: loadtest: add a test for connection counters
Hi Pieter, On Sat, Sep 15, 2018 at 02:03:45AM +0200, PiBa-NL wrote: > Hi List, Willy, > > I've created a regtest that checks that when concurrent connections are > being handled that the connection counters are kept properly. > > I think it could be committed as attached. It takes a few seconds to run. It > currently fails on 1.9-dev2 (also fails on 1.8.13 with kqueue on FreeBSD, > adding a 'nokqueue' on 1.8.13 makes it succeed though..). > > I think it might be a good and reproducible test to run.? > > Or does it need more tweaking? Thoughts appreciated :). I took some time this morning to give it a test. For now it fails here, after dumping 2200 lines of not really usable output that I didn't investigate. From what I'm seeing it seems to moderately stress the local machine so it has many reasons for failing (lack of source ports, improperly tuned conntrack, ulimit, etc), and it takes far too long a time to be usable as a default test, or this one alone will be enough to discourage anyone from regularly running "make reg-tests". I think we should create a distinct category for such tests, because I see some value in it when it's made to reproduce a very specific class of issues which is very unlikely to trigger unless someone is working on it. In this case it is not a problem that it dumps a lot of output, as it will be useful for the person knowing what to look for there. Probably that such tests should be run by hand and dump their log into a related file. Imagine for example that we would have this : $ make reg-tests/heavy/conn-counter-3000-req.log It would run the test on reg-tests/heavy/conn-counter-3000-req.vtc and would produce the log into reg-tests/heavy/conn-counter-3000-req.log. We could use a similar thing to test for select/poll/epoll/kqueue, to test for timeouts, race conditions (eg show sess in threads). This is very likely something to brainstorm about. You might have other ideas related to certain issues you faced in the past. Fred is unavailable this week but I'd be very interested in his opinion on such things. Thus for now I'm not applying your patch, but I'm interested in seeing what can be done with it. Thanks, Willy
Re: Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections
https://www.google.com.au/search?q=Connect()+failed+for+backend+no+free+ports On Wed, 19 Sep 2018 at 12:32, Shishir Kumar Yadav wrote: > I am able to get logs and I see these errors - > > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: ::: > 10.21.214.31:39846 [18/Sep/2018:23:39:22.405] haproxy-frontend > ir-http-server-backend/server1 0/0/-1/-1/0 503 212 - - SC-- 16/15/13/13/3 > 0/0 "GET > /splunk-data/puretest/db/f2/63/1~DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/guidSplunk-DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/1536613068-1536610572-12314536523543509297.tsidx > HTTP/1.1" > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for > backend ir-http-server-backend: no free ports. > 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: ::: > 10.21.214.31:39840 [18/Sep/2018:23:39:22.405] haproxy-frontend > ir-http-server-backend/server1 0/0/-1/-1/0 503 212 - - SC-- 36/36/33/33/3 > 0/0 "GET > /splunk-data/puretest/db/f2/63/1~DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/guidSplunk-DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/splunk-autogen-params.dat > HTTP/1.1" > > > My sysctl port range is - > > net.ipv4.ip_local_port_range = 32768 61000 > > > On Tue, Sep 18, 2018 at 10:20 AM, Shishir Kumar Yadav < > shis...@purestorage.com> wrote: > >> Thanks Lucas. >> >> I am trying to enable the logging. Will provide the logs when it is >> available. >> >> The default value 2000 is much higher to cause problem as I am seeing >> problem at 250 connections. >> >> -Shishir >> >> On Tue, Sep 18, 2018 at 2:40 AM, Lukas Tribus wrote: >> >>> Hello, >>> >>> >>> On Tue, 18 Sep 2018 at 02:36, Shishir Kumar Yadav >>> wrote: >>> > >>> > Hi All, >>> > >>> > I am using haproxy 1.8.3 >>> >>> Which has 169 unfixed bugs: >>> http://www.haproxy.org/bugs/bugs-1.8.3.html >>> >>> I'd strongly suggest you use latest stable, although that doesn't mean >>> it has something to do with your specific issue. >>> >>> >>> >>> > I am testing a setup where a bunch of clients are generating number of >>> GET >>> > requests and when number of connection reaches somewhere around 250, >>> > the clients intermittently get 503 Service Unavailable error. >>> >>> Looking at the haproxy logs is the logical next step. They will most >>> likely contain the reason for this 503 error. >>> >>> >>> Also note that when you *not* set maxconn, the default value of 2000 >>> applies (per process and per frontend, not per server). >>> >>> >>> >>> Lukas >>> >> >> >
Re: Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections
I am able to get logs and I see these errors - 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[569]: :::10.21.214.31:39846 [18/Sep/2018:23:39:22.405] haproxy-frontend ir-http-server-backend/server1 0/0/-1/-1/0 503 212 - - SC-- 16/15/13/13/3 0/0 "GET /splunk-data/puretest/db/f2/63/1~DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/guidSplunk-DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/1536613068-1536610572-12314536523543509297.tsidx HTTP/1.1" 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: Connect() failed for backend ir-http-server-backend: no free ports. 2018-09-18 23:39:22+00:00 127.0.0.1 haproxy[572]: :::10.21.214.31:39840 [18/Sep/2018:23:39:22.405] haproxy-frontend ir-http-server-backend/server1 0/0/-1/-1/0 503 212 - - SC-- 36/36/33/33/3 0/0 "GET /splunk-data/puretest/db/f2/63/1~DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/guidSplunk-DFFEABF3-6CE0-4107-88E9-DAC84082BE3A/splunk-autogen-params.dat HTTP/1.1" My sysctl port range is - net.ipv4.ip_local_port_range = 32768 61000 On Tue, Sep 18, 2018 at 10:20 AM, Shishir Kumar Yadav < shis...@purestorage.com> wrote: > Thanks Lucas. > > I am trying to enable the logging. Will provide the logs when it is > available. > > The default value 2000 is much higher to cause problem as I am seeing > problem at 250 connections. > > -Shishir > > On Tue, Sep 18, 2018 at 2:40 AM, Lukas Tribus wrote: > >> Hello, >> >> >> On Tue, 18 Sep 2018 at 02:36, Shishir Kumar Yadav >> wrote: >> > >> > Hi All, >> > >> > I am using haproxy 1.8.3 >> >> Which has 169 unfixed bugs: >> http://www.haproxy.org/bugs/bugs-1.8.3.html >> >> I'd strongly suggest you use latest stable, although that doesn't mean >> it has something to do with your specific issue. >> >> >> >> > I am testing a setup where a bunch of clients are generating number of >> GET >> > requests and when number of connection reaches somewhere around 250, >> > the clients intermittently get 503 Service Unavailable error. >> >> Looking at the haproxy logs is the logical next step. They will most >> likely contain the reason for this 503 error. >> >> >> Also note that when you *not* set maxconn, the default value of 2000 >> applies (per process and per frontend, not per server). >> >> >> >> Lukas >> > >
Re: Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections
Thanks Lucas. I am trying to enable the logging. Will provide the logs when it is available. The default value 2000 is much higher to cause problem as I am seeing problem at 250 connections. -Shishir On Tue, Sep 18, 2018 at 2:40 AM, Lukas Tribus wrote: > Hello, > > > On Tue, 18 Sep 2018 at 02:36, Shishir Kumar Yadav > wrote: > > > > Hi All, > > > > I am using haproxy 1.8.3 > > Which has 169 unfixed bugs: > http://www.haproxy.org/bugs/bugs-1.8.3.html > > I'd strongly suggest you use latest stable, although that doesn't mean > it has something to do with your specific issue. > > > > > I am testing a setup where a bunch of clients are generating number of > GET > > requests and when number of connection reaches somewhere around 250, > > the clients intermittently get 503 Service Unavailable error. > > Looking at the haproxy logs is the logical next step. They will most > likely contain the reason for this 503 error. > > > Also note that when you *not* set maxconn, the default value of 2000 > applies (per process and per frontend, not per server). > > > > Lukas >
Re: [ANNOUNCE] haproxy-1.9-dev2
> Le 18 sept. 2018 à 11:54, Lukas Tribus a écrit : > > Hi Manu, > > > On Fri, 14 Sep 2018 at 15:45, Emmanuel Hocdet wrote: >> >> Hi, >> >> Quick test with 1.9-dev2, and i see latency (in seconds) to connect to >> haproxy with SSL (tcp mode). >> It’s ok in master with 9f9b0c6a. >> No time to investigate more for the moment. > > I cannot reproduce it in a simple SSL termination + tcp mode > configuration. There is probably something more to it. > > perhaps with: tcp-request inspect-delay 5s ++ Manu
Re: [ANNOUNCE] haproxy-1.9-dev2
Hi guys, On Tue, Sep 18, 2018 at 11:54:59AM +0200, Lukas Tribus wrote: > Hi Manu, > > > On Fri, 14 Sep 2018 at 15:45, Emmanuel Hocdet wrote: > > > > Hi, > > > > Quick test with 1.9-dev2, and i see latency (in seconds) to connect to > > haproxy with SSL (tcp mode). > > It's ok in master with 9f9b0c6a. > > No time to investigate more for the moment. > > I cannot reproduce it in a simple SSL termination + tcp mode > configuration. There is probably something more to it. We definitely have some issues related to connection setup and tear down that we're investigating. They're all related to the changes consisting in orienting the recv/send calls from up to down and not relying on waking up everything bottom-to-top. There's a very closely related issue that Pieter reported with TCP connections without data causing issues, another one that Christopher just faced this morning where connection errors wake up process_stream() in loops, and some cases where client errors on H2 can crash the process. I'm sorry for all these issues, but having the code merged as a first step was the only option to make forward progress on this part. Having everyone constantly rebase his own code on hypothetic changes was not workable anymore. Among the possible solutions currently being studied, it seems that there are too many places where the "process" part of the mux is called instead of being scheduled (typically the cases where we update the polling). But this is still under investigation. Thus if you're still finishing your devs for 1.9, 1.9-dev2 gives a preview of the forthcoming changes that will apply to the connection layers, at the price of accepting to work around the current limitations. If you want something to play with on your own servers, it definitely isn't something to play with. I'm currently trying to stabilize this so that we can at least continue the development with less disturbance, but it's really not easy, as this merge has at least uncovered some limitations of the current model among those inherited from the prehistoric code :-/ Thanks, Willy
Re: [ANNOUNCE] haproxy-1.9-dev2
Hi Manu, On Fri, 14 Sep 2018 at 15:45, Emmanuel Hocdet wrote: > > Hi, > > Quick test with 1.9-dev2, and i see latency (in seconds) to connect to > haproxy with SSL (tcp mode). > It’s ok in master with 9f9b0c6a. > No time to investigate more for the moment. I cannot reproduce it in a simple SSL termination + tcp mode configuration. There is probably something more to it. Regards, Lukas
Re: Intermittent HTTP 503 Error (Service Unavailable) with about 250 Connections
Hello, On Tue, 18 Sep 2018 at 02:36, Shishir Kumar Yadav wrote: > > Hi All, > > I am using haproxy 1.8.3 Which has 169 unfixed bugs: http://www.haproxy.org/bugs/bugs-1.8.3.html I'd strongly suggest you use latest stable, although that doesn't mean it has something to do with your specific issue. > I am testing a setup where a bunch of clients are generating number of GET > requests and when number of connection reaches somewhere around 250, > the clients intermittently get 503 Service Unavailable error. Looking at the haproxy logs is the logical next step. They will most likely contain the reason for this 503 error. Also note that when you *not* set maxconn, the default value of 2000 applies (per process and per frontend, not per server). Lukas
