Re: Request rate limiting on the backend section

[Krishna Kumar (Engineering)]

On Tue, Nov 7, 2017 at 11:57 PM, Lukas Tribus  wrote:

Hi Lukas,

> Yes, in 1.7 you can change server maxconn values in real time using
> the admin socket:
> https://cbonte.github.io/haproxy-dconv/1.7/management.
html#9.3-set%20maxconn%20server

Thanks, will take a look at if we can use this. The only issue is that we
want to
be able to change rps very often, and some backend sections contain upto
500 servers (and much more during sales), and doing that on the fly at high
frequency may not scale.

> You are reluctant to elaborate on the bigger picture, so I guess
> generic advice is not what you are looking for. I just hope you are
> not trying to build some kind of distributed rate-limiting
> functionality with this.

Sorry, not reluctance, I thought giving too much detail would put off people
from taking a look :) So you are right, we are trying to build a
distributed rate
limiting feature, and the control plane is mostly ready (thanks to HAProxy
developers for making such a performant/configurable system). The service
monitors current http_request_rate and current RPS setting via uds every
second, and updates these values to a central repository (zookeeper), and
on demand, tries to increase capacity by requesting capacity from other
servers so as to keep capacity constant at the configured value (e.g. 1000
RPS). Is this something you would not recommend?

> I don't have enough experience with stick-tables to comment on this
> generally, but I would suggest you upgrade to a current 1.7 release
> first of all and retry your tests. There are currently 223 bugs fixed
> in releases AFTER 1.6.3:
> http://www.haproxy.org/bugs/bugs-1.6.3.html

Thanks, we are considering moving to this version.

> Maybe someone more stick-table savvy can comment on your specific
question.

If anyone else has done something similar, would really like to hear from
you on
how to control RPS in the backend.

Regards,
- Krishna

lua socket api settimeout in seconds vs. milliseconds

[Nick Galbreath]

Hello,

Thanks for a great Lua API!

I have a question regarding the socket:settimeout function.

It appears to only accept an integer value for seconds, but internally
converts to milliseconds (see below).

Is there a reason to enforce whole seconds, or could this be relaxed to
accept a lua number (float/double).?

Happy to provide a patch, but just checking first since I'm new to HAProxy.

Thanks for your time,

regards

nickg

around line 2442 in hluc.c:

__LJMP static int hlua_socket_settimeout(struct lua_State *L)
{
struct hlua_socket *socket;
int tmout;
struct xref *peer;
struct appctx *appctx;
struct stream_interface *si;
struct stream *s;

MAY_LJMP(check_args(L, 2, "settimeout"));

socket = MAY_LJMP(hlua_checksocket(L, 1));
tmout = MAY_LJMP(luaL_checkinteger(L, 2)) * 1000;

Re: Request rate limiting on the backend section

[Krishna Kumar (Engineering)]

Hi Lukas,

On Tue, Nov 7, 2017 at 6:46 PM, Lukas Tribus  wrote:

> I'd suggest to use maxconn. This limits the amount of connections opened
> to a single server, and is therefor equivalent to in-flight requests.
That's is
> a more appropriate limit than RPS because it doesn't matter if the
responses
> take a long time to compute or not.

Thanks for your suggestion. Unfortunately per server maxconn may be
unsuitable
for our particular case due to 2 reasons:

1. We want to modify the limit dynamically at high frequency, e.g. every
second.
maxconn setting on server does not seem to be modifiable at present
without
a reload. We are using haproxy-1.6.3, is this feature present in a
newer release?

2. We have haproxy running on *many* servers, even for a single service
(many
configuration files also use nbproc). It is easier to give a RPS=1000
for the entire
service instead of breaking up per process and per server, which may
not be
possible at the rate we plan.

Is there any way that this can be done at the backend using stick-tables? I
was
wondering if there was an elementary mistake in the configuration options
being
used.

Regards,
- Krishna

Re: Problem: Connect() failed for backend: no free ports.

[Lukas Tribus]

Hello,


>> If you don't require specific source IP's per server, than just remove
>> the "source ip:port-range" keyword altogether, the kernel will take
>> care of everything. Just make sure that your sysctls permit a similar
>> source port range.
>
> thanks. That helps.
>
>
>> If you need specific source IPs (for reasons unrelated to source port
>> exhaustion), then drop the port range and specify only the IP. However
>> for the kernel to be able to use the full 5-table, you will need
>> IP_BIND_ADDRESS_NO_PORT [1], which requires haproxy 1.7, linux 4.2 and
>> libc 2.23.
>
> We will see if we can install a 4.2 or later kernel.

This is only necessary if you need the to use specific source IPs. If
you can remove the source keyword completely, then you don't need to
do this at all.


Grüße,
Lukas

Re: Problem: Connect() failed for backend: no free ports.

[Michael Schwartzkopff]

Am 06.11.2017 um 23:43 schrieb Lukas Tribus:
> Hallo Michael,
>
>
>
> 2017-11-06 22:47 GMT+01:00 Michael Schwartzkopff :
>> Am 06.11.2017 um 22:39 schrieb Baptiste:
>>> On Mon, Nov 6, 2017 at 10:14 PM, Michael Schwartzkopff  wrote:
>>>
 Hi,

 I have a problem setting up a haproxy 1.6.13 that starts several
 processes. In the config I have nbroc 3. In the logs I find lots of
 entries like:

 haproxy[: Connect() failed for backend XXX: no free ports
>> global
>>   maxconn 200
>>   nbproc 3
>>   cpu-map 1 0
>>   cpu-map 2 1
>>   cpu-map 3 2
>> [...]
>> backend IMAP-be
>>   option tcp-check
>>   tcp-check connect port 143
>>   tcp-check expect string * OK
>>   default-server on-marked-down shutdown-sessions
>>   fullconn 40
>>   server proxy01 192.168.0.101 source 192.168.0.201:1-6 check
>>   server proxy02 192.168.0.102 source 192.168.0.202:1-6 check
>>   server proxy03 192.168.0.103 source 192.168.0.203:1-6 check
>>   server proxy04 192.168.0.104 source 192.168.0.204:1-6 check
>
> You are using multiprocess mode together with static source port
> ranges. That's a bad idea, because the processes will compete for the
> same exact source ports and the syscalls will continue to fail as
> different processes are trying to use the same ports.
>
> There are a few possibilities here, but we will have to know:
>
> - why are you using different source IP's for each backend server?
> - why are you using static port ranges?
>
> What I would suggest is to make sure that the kernel does the source
> port selection, but the kernel needs to be able to use the full
> 5-tuple at this point, otherwise I imagine you'd run into source port
> exhaustion soon.
>
> If you don't require specific source IP's per server, than just remove
> the "source ip:port-range" keyword altogether, the kernel will take
> care of everything. Just make sure that your sysctls permit a similar
> source port range.

thanks. That helps.


> If you need specific source IPs (for reasons unrelated to source port
> exhaustion), then drop the port range and specify only the IP. However
> for the kernel to be able to use the full 5-table, you will need
> IP_BIND_ADDRESS_NO_PORT [1], which requires haproxy 1.7, linux 4.2 and
> libc 2.23.

We will see if we can install a 4.2 or later kernel.


Mit freundlichen Grüßen,

-- 

[*] sys4 AG
 
https://sys4.de, +49 (89) 30 90 46 64
Schleißheimer Straße 26/MG,80333 München
 
Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263
Vorstand: Patrick Ben Koetter, Marc Schiffbauer, Wolfgang Stief
Aufsichtsratsvorsitzender: Florian Kirstein




signature.asc
Description: OpenPGP digital signature

Re: Request rate limiting on the backend section

[Lukas Tribus]

Hello,



2017-11-07 10:46 GMT+01:00 Krishna Kumar (Engineering)
:
> Hi all,
>
> I am trying to implement request rate limiting to protect our servers from
> too
> many requests. While we were able to get this working correctly in the
> frontend section, it is required to implement the same in the backend
> section
> due to the configuration we use in our data center for different services.

I'd suggest to use maxconn. This limits the amount of connections opened
to a single server, and is therefor equivalent to in-flight requests. That's is
a more appropriate limit than RPS because it doesn't matter if the responses
take a long time to compute or not.

You simply specify maxconn on each server:

backend bk_default
 server s1 10.0.0.1:80 maxconn 5000
 server s2 10.0.0.2:80 maxconn 3000
 server s3 10.0.0.3:80 maxconn 2500


http://cbonte.github.io/haproxy-dconv/1.7/configuration.html#5.2-maxconn



Regards,
Lukas

Re: Rename SPIN/RWLOCK macros using HA_

[Willy TARREAU]

On Tue, Nov 07, 2017 at 10:51:16AM +0100, Christopher Faulet wrote:
> Hi Willy,
> 
> Here is the patch to rename SPIN/RWLOCK macros and another one to 
> implicitly enable USE_THREAD on Solaris.

now applied, thank you!

Willy

Request rate limiting on the backend section

[Krishna Kumar (Engineering)]

Hi all,

I am trying to implement request rate limiting to protect our servers from
too
many requests. While we were able to get this working correctly in the
frontend section, it is required to implement the same in the backend
section
due to the configuration we use in our data center for different services.

This is the current configuration that I tried to get 1000 RPS for the
entire
backend section:

backend HTTP-be
http-request track-sc2 fe_id
stick-table type integer size 1m expire 60s store
 http_req_rate(100ms),gpc0,gpc0_rate(100ms)
acl mark_seen sc2_inc_gpc0 ge 0
acl outside_limit sc2_gpc0_rate() gt 100
http-request deny deny_status 429 if mark_seen outside_limit
server my-server :80

But running "wrk -t 1 -c 1 " gives:
 RPS: 19.60 (Total requests: 18669 Good: 100 Errors: 18569)
with following haproxy metrics:
 0x2765d1c: key=3 use=1 exp=6 gpc0=7270 gpc0_rate(100)=364
http_req_rate(100)=364

and "wrk -t 20 -c 1000 " gives:
RPS: 6.62 (Total requests: 1100022 Good: 100 Errors: 1099922)
with following haproxy metrics:
0x2765d1c: key=3 use=94 exp=5 gpc0=228218 gpc0_rate(100)=7229
http_req_rate(100)=7229

As seen above, only a total of 100 requests succeeded, not 100 * 10 * 20 =
20K
(for the 20 second duration). gpc0_rate does not *seem* to get reset to
zero at
configured interval (100ms), may be due to a mistake in the configuration
settings
above.

Another setting that we tried was:

backend HTTP-be
acl too_fast be_sess_rate gt 1000
tcp-request inspect-delay 1s
tcp-request content accept if ! too_fast
tcp-request content accept if WAIT_END
server my-server :80

While this behaved much better than earlier, I still get a lot of
discrepancy based
on the traffic volume:
wrk -t 40 -c 200 -d 15s 
RPS: 841.14 (Total requests: 12634 Good: 12634 Errors: 0 Time: 15.02)

wrk -t 40 -c 400 -d 15s 
RPS: 1063.78 (Total requests: 15978 Good: 15978 Errors: 0 Time: 15.02)

wrk -t 40 -c 800 -d 15s 
RPS: 1123.03 (Total requests: 16868 Good: 16868 Errors: 0 Time: 15.02)

wrk -t 40 -c 1600 -d 15s 
RPS: 1382.98 (Total requests: 20883 Good: 20883 Errors: 0 Time: 15.10)

The last one is off by 38%

Could someone help on what I am doing wrong, or how to achieve this? I
prefer using the first approach with stick table if possible, as it
provides finer
time granularity.

Thanks for any help.

Regards,
- Krishna

Editorial Ideas for your consideration

[Jason Grills]

Hello, 
 
My name is Jason; I am a Writer & Technical Support Executive at ProProfs.com ( 
https://www.proprofs.com/ ) , a website for building and testing knowledge. 
 
I have recently contributed articles to sites like Mycustomer ( 
https://www.mycustomer.com/service/channels/do-emojis-have-a-role-in-live-chat-support
 ) , Saasgenious ( 
http://www.saasgenius.com/blog/5-signs-your-support-really-needs-live-chat-software
 ) & Hyken ( 
https://hyken.com/customer-care/guest-blog-hire-live-chat-agents-delight-customers/
 ) and I was wondering whether you would be interested in having me contribute 
some articles to your website too. For example, here are a couple of story 
ideas that I have for you: 
 5 Easy Peasy Ways to Make Your Brand in Demand Why Is Your Marketing Agency 
Not Getting New Clients How Deploying a Live Chat Can Improve Your Business 
Easiest Way to Make Your Support Team More ProductivePlease let me know if you 
are interested, and I will be happy to send over articles for review 
 
Regards 
Jason

Re: Diagnose a PD-- status

[Christopher Faulet]

Le 06/11/2017 à 16:47, Mildis a écrit :

Hi Christopher,

Configuration is attached.
The domain2backend map sends data mostly to bck-traefik.



Hi Mildis,

At first glance, I don't see anything strange here. The compression 
filter is not supposed to fail this way. So there is definitely 
something strange I don't understand for now.


Could you reproduce the bug on a single request on an HAProxy instance 
without load, maybe using a dedicated frontend ? If yes, it could be 
useful to enable the trace filter adding following lines in your 
configuration, in the used frontend section:


  filter trace name BEFORE
  filter compression
  filter trace name AFTER

Then, you can retry, starting HAProxy in foreground. The output is 
verbose, but it give more information about the message filtering.


Thanks,
--
Christopher Faulet