Re: haproxy backend server template service discovery questions

[Pavlos Parissis]

On Sat, 6 Apr 2024 at 20:17, Илья Шипицин  wrote:
>
> Consul template is something done by consul itself, after that haproxy.conf 
> is rendered
>
> Do you mean "how haproxy deals with rendered template"?
>

He doesn't use that method of discovery, he uses DNS resolvers so
haproxy gets the SRV records with the list of
servers of backends.

Cheers,
Pavlos

Re: haproxy backend server template service discovery questions

[Pavlos Parissis]

On Fri, 5 Apr 2024 at 15:00, Andrii Ustymenko
 wrote:
>
> Dear list!
>
> My name is Andrii. I work for Adyen. We are using haproxy as our main
> software loadbalancer at quite large scale.
> Af of now our main use-case for backends routing based on
> server-template and dynamic dns from consul as service discovery. Below
> is the example of simple backend configuration:
>
> ```
> backend example
>balance roundrobin
>server-template application 10 _application._tcp.consul resolvers
> someresolvers init-addr last,libc,none resolve-opts allow-dup-ip
> resolve-prefer ipv4 check ssl verify none
> ```
>
> and in global configuration
>
> ```
> resolvers someresolvers
>nameserver ns1 10.10.10.10:53
>nameserver ns2 10.10.10.11:53
> ```
>
> As we see haproxy will create internal table for backends with some
> be_id and be_name=application and allocate 10 records for each server
> with se_id from 1 to 10. Then those records get populated and updated
> with the data from resolvers.
> I would like to understand couple of things with regards to this
> structure and how it works, which I could not figure out myself from the
> source code:
>
> 1) In tcpdump for dns queries we see that haproxy asynchronously polls
> all the nameservers simultaneously. For instance:
>
> ```
> 11:06:17.587798 eth2  Out ifindex 4 aa:aa:aa:aa:aa:aa ethertype IPv4
> (0x0800), length 108: 10.10.10.50.24050 > 10.10.10.10.53: 34307+ [1au]
> SRV? _application._tcp.consul. (60)
> 11:06:17.587802 eth2  Out ifindex 4 aa:aa:aa:aa:aa:aa ethertype IPv4
> (0x0800), length 108: 10.10.10.50.63155 > 10.10.10.11.53: 34307+ [1au]
> SRV? _application._tcp.consul. (60)
> 11:06:17.588097 eth2  In  ifindex 4 ff:ff:ff:ff:ff:ff ethertype IPv4
> (0x0800), length 205: 10.10.10.10.53 > 10.10.10.50.24050: 2194 2/0/1 SRV
> 0a5099e5.addr.consul.:25340 1 1, SRV 0a509934.addr.consul.:26010 1 1 (157)
> 11:06:17.588097 eth2  In  ifindex 4 ff:ff:ff:ff:ff:ff ethertype IPv4
> (0x0800), length 205: 10.10.10.11.53 > 10.10.10.50.63155: 2194 2/0/1 SRV
> 0a5099e5.addr.consul.:25340 1 1, SRV 0a509934.addr.consul.:26010 1 1 (157)
> ```
>
> Both nameservers reply with the same response. But what if they are out
> of sync? Let's say one says: server1, server2 and the second one says
> server2, server3? So far testing this locally - I see sometimes the
> reply overrides the table, but sometimes it seems to just gets merged
> with the rest.
>

Most service discovery systems employ "eventually consistency".
Therefore, some instances of
the DNS servers in front of that system may have "stale" data. The
instance usually becomes
consistent with milliseconds or seconds; it depends on the topology.

I don't think haproxy performs any kind of merging, I believe it uses
the 1st valid response.


> 2) Each entry from SRV reply will be placed into the table under
> specific se_id. Most of the times that placement won't change. So, for
> the example above the most likely 0a5099e5.addr.consul. and
> 0a509934.addr.consul. will have se_id 1 and 2 respectively. However
> sometimes we have the following scenario:
>
> 1. We admistratively disable the server (drain traffic) with the next
> command:
>
> ```
> echo "set server example/application1 state maint" | nc -U
> /var/lib/haproxy/stats
> ```
>
> the MAINT flag will be added to the record with se_id 1
>
> 2. Instance of application goes down and gets de-registered from consul,
> so also evicted from srv replies and out of discovery of haproxy.
>
> 3. Instance of application goes up and gets registered by consul and
> discovered by haproxy, but haproxy allocates different se_id for it.
> Haproxy healthchecks will control the traffic to it in this case.
>
> 4. We will still have se_id 1 with MAINT flag and application instance
> dns record placed into different se_id.
>
> The problem comes that any new discovered record which get placed into
> se_id 1 will never be active until either command:
>
> ```
> echo "set server example/application1 state ready" | nc -U
> /var/lib/haproxy/stats
> ```
>
> executed or haproxy gets reloaded without state file. With this pattern
> we basically have persistent "records pollution" due to operations made
> directly with control socket.
>

This situation could lead to minor incidents where most newly
registered servers are
assigned to se_ids that were previously in maintenance mode.
So, you end up with a backend that has, let's say, 75% of servers in
maintenance mode.

> I am not sure is there anything to do about this. Maybe, if haproxy
> could cache the state not only of se_id but also associated record with
> that and then if that gets changed - re-schedule healtchecks. Or instead
> of integer ids use some hashed ids based on dns/ip-addresses of
> discovered records, in this case binding will happen exactly in the same
> slot.
>

Another approach could be haproxy resetting the `srv_admin_state `
field for IDs for which an
entry was de-registered.

My 2 cents,
Pavlos

Incorrect href for 2.9 version on Latest versions table

[Pavlos Parissis]

Hoi all,

I noticed on the main page of http://www.haproxy.org/ that the `Latest
versions` table has an incorrect URL for the git repo of the 2.9
version.
In the below table the `href` for 2.9 should point to
http://git.haproxy.org/?p=haproxy-2.9.git and not to
http://git.haproxy.org/?p=haproxy.git

```
[...snip...]

 http://git.haproxy.org/?p=haproxy.git;>2.9
 2023-12-05
 2025-Q1 (stable)
 2.9.4
 2024/01/31
 http://git.haproxy.org/git/haproxy.git/;>git / http://git.haproxy.org/?p=haproxy.git;>web / dir / https://www.mail-archive.com/search?l=haproxy%40formilux.org=announce+subject%3A%22[ANNOUNCE]+haproxy-2.9.4%22+-subject%3A%22re:%22;>announce
/ http://www.haproxy.org/bugs/bugs-2.9.html;>bugs

[...snip...]
```

Please stop for a second from improving haproxy and making it faster
to fix the HTML code. :-)

Thanks,
Pavlos

Re: Some notes about what happens with HTTP/1.0 requests

[Pavlos Parissis]

On Wednesday, July 5, 2023 8:25:35 PM CEST Shawn Heisey wrote:
> I have a backend in haproxy for my Solr server.  Solr lives unencrypted 
> on port 8983, haproxy provides TLS for it, on a name like 
> `solr.example.com`.
> 
> Everything works fully as expected with HTTP 1.1, 2, or 3.
> 
> If I send a request with curl using any HTTP version to 
> https://solr.example.com/, it results in a 302 response.
> 
> If the request is HTTP/1.0, Solr is revealing the internal IP address -- 
> the location header is https://172.31.8.104:8983/solr/ which will not 
> work -- the port isn't exposed to the Internet, isn't using TLS, and the 
> private IP address is only valid within the AWS VPC.  An interesting 
> detail:  If I send the HTTP/1.0 request directly to Solr, it does NOT 
> reveal the internal address.  That only happens for requests relayed by 
> haproxy.
> 
> The backend connection is HTTP/2, as I have "proto h2" on the server line.
> 
> The curl command gets a response that's HTTP/1.1 even though it sent 1.0.
> 
> What I would like to do is deny HTTP/1.0 requests, but I have not been 
> able to figure out a way to do that. 

There is a list of pre-defined ACLs, see 
http://docs.haproxy.org/2.8/configuration.html#7.4, and in that list
you have HTTP_1.0 acl to match traffic for that version of HTTP protocol.

So, you can add below snippet to block traffic for HTTP 1.0 version

http-request deny if HTTP_1.0.

I hope it helps,
Pavlos

Re: Opinions desired on HTTP/2 config simplification

[Pavlos Parissis]

On Saturday, April 15, 2023 11:32:49 AM CEST Willy Tarreau wrote:
> Hi everyone,

[...snip...]
> Even if I wouldn't share my feelings, some would consider that I'm
> trying to influence their opinion, so I'll share them anyway :-)  I
> think that with the status change from "experimental-but-supported" to
> "production" for QUIC in 2.8, having to manually and explicitly deal
> with 3 HTTP versions in modern configs while the default (h1) only
> corresponds to 7% of what clients prefer is probably an indicator that
> it's the right moment to simplify these a little bit. But I'm open to
> any argument in any direction.
> 
> It would be nice to be able to decide (and implement a change if needed)
> before next week's dev8, so that it leaves some time to collect feedback
> before end of May, so please voice in!
> 

I agree with enabling H2 by default and I also agree that 2.8 version is very 
good candidate for 
introducing that change.

My 2 cents,
Pavlos

Re: Is adaptive circuit breaking in the roadmap for 2.3/2.4?

[Pavlos Parissis]

On Τρίτη, 15 Σεπτεμβρίου 2020 9:24:32 Π.Μ. CEST Willy Tarreau wrote:
> Hi Pavlos!
> 
> On Sat, Sep 12, 2020 at 11:45:12AM +0200, Pavlos Parissis wrote:
> > Hi old friends!,
> > 
> > Is in the roadmap the addition of a circuit breaking which adapts its 
> > settings using real-time data?
> > I believe we discussed this in the last HAProxyConf with a group of people, 
> > but I don't remember if there were
> > , back then, concrete plans to work on it.
> > 
> > I know that something similar can  be accomplished by using agent check, 
> > but IMHO it less ideal.
> > - watch 
> > https://www.youtube.com/watch?v=CQvmSXlnyeQ=PLj6h78yzYM2O1wlsM-Ma-RYhfT5LKq0XC=21
> > 
> > - read 
> > https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/http/http_filters/adaptive_concurrency_filter
> 
> Yep as you say we can indeed already do it using agent checks. I know
> it's not ideal, but what matters to me is that it indicates we already
> have the required core functionalities to do something like this.
> 

This is excellent!

> The proper way to implement such a mechanism is by using feedback
> reaction, similar to how MPPT solar panel controllers work: you never
> know if you're sending enough load to keep the servers busy, so you
> constantly need to try to send a bit more to see if any service
> degradation occurs or not. The degradation happens by response time
> suddenly becoming affine to the number of concurrent requests. Then
> the decision of where to stick needs to be made based on the choice
> of lower latency (bottom of curve) or higher bandwidth (top of curve).
> A simpler approach would consist in having a setting indicating how
> much extra response time is acceptable compared to the measued optimal
> one.
> 
> I also think that it remains important to let the user configure lower
> and upper bounds that guarantee safe operations. And that's typically
> what could be done using the minconn and maxconn values. Instead of
> using the backend's fullconn setting we would rely on a response time
> measurement.
> 
> Or maybe that could be the right opportunity for splitting connection
> concurrency from request concurrency, keeping connections for the hard
> limits and using request concurrency for softer limits.
> 

What do we gain by doing this? IMHO, most application will suffer from
concurrent requests rather from connections. Having said that, we still have
applications that handling many idle or not SSL connection comes with
performance issues.

> There's no such ongoing work that I'm aware of but that has always
> been a subject of interest to me (I even wrote down the algorithm to
> compute weights by measured response times using a low-pass filter a
> decade ago but I lost my notes and never felt like doing work again).
> So if anyone is interested in this subject, we can continue this
> conversation till we reach something that looks like a possible design
> roadmap.
> 

I can volunteering in testing at my spare time, I don't have a valid use-case 
at my
current work, but I am still very much interested in helping haproxy community
to support this feature.

Thanks Willy for getting back to me,
Pavlos

> Cheers,
> Willy
> 

Is adaptive circuit breaking in the roadmap for 2.3/2.4?

[Pavlos Parissis]

Hi old friends!,

Is in the roadmap the addition of a circuit breaking which adapts its settings 
using real-time data?
I believe we discussed this in the last HAProxyConf with a group of people, but 
I don't remember if there were
, back then, concrete plans to work on it.

I know that something similar can  be accomplished by using agent check, but 
IMHO it less ideal.



- watch 
https://www.youtube.com/watch?v=CQvmSXlnyeQ=PLj6h78yzYM2O1wlsM-Ma-RYhfT5LKq0XC=21

- read 
https://www.envoyproxy.io/docs/envoy/v1.15.0/configuration/http/http_filters/adaptive_concurrency_filter

Cheers,
Pavlos

Re: TLS tickets prone to MITM attacks (was: [PR] Docs tls tickets)

[Pavlos Parissis]

On Τετάρτη, 11 Μαρτίου 2020 9:10:56 Π.Μ. CET Lukas Tribus wrote:
> Hello,
> 
> On Wed, 11 Mar 2020 at 08:32, Илья Шипицин  wrote:
> >> On 09.03.20 20:37, Lukas Tribus wrote:
> >> >> I think the wording from the patch is still quite relaxed :). One of 
> >> >> the best
> >> >> summaries describing the session ticket flaws, which I recommend is 
> >> >> this:
> >> >> https://blog.filippo.io/we-need-to-talk-about-session-tickets/
> >> > Nothing about this is a MITM attack. The point in the article is that
> >> > when the ticket-key is compromised, captured traffic can be passively
> >> > decrypted (which is what broken PFS means, as explained by the Apache
> >> > docs).
> >>
> >> take also this article, which clearly states that session tickets are
> >> vulnerable to replay attacks (which are a kind of MITM):
> >> https://eprint.iacr.org/2019/228.pdf
> >
> >
> >
> > major players of a big picture are 0RTT and session tickets.
> > indeed, if you wish to fight replay attack, you cannot use 0RTT (also, you 
> > are supposed to maintain keys rotation).
> >
> > as for keys rotation, for unfamiliar people it is not clear why haproxy 
> > itself does not provide such rotation.
> > at least, it should be better documented.
> 
> Sure. But we are not gonna use the documentation to spread wrong
> information and FUD, based on partial, incorrect and out of context
> quotes.
> 
> We already explain the forward secrecy issue with TLS ticket
> resumption, just as we explain replay-attacks for TLSv1.3 0RTT. If
> anyone thinks this still needs improvement, feel free to send RFC
> patches based on FACTUAL information.
> 
> But let's stop making up things and then go on a fishing expeditions
> to justify it.
> 
> 
> As for automatic key rotation features, I'm not aware of anyone doing
> this by default, expect some niche projects (Caddy I believe does
> this). Not nginx, not Apache. These are features that someone has to
> actually develop.
> 
> 
> 
> -lukas
> 

+1 from me who had deployed TLS tickets across multiple HAProxy servers.
The people who experiment with these advantaged features know very well the 
pros and cons.

My 2 cents,
Pavlos

Re: kernel panics after updating to 2.0

[Pavlos Parissis]

On Παρασκευή, 6 Δεκεμβρίου 2019 10:36:18 Π.Μ. CET Sander Hoentjen wrote:
> 
> On 12/6/19 10:20 AM, Pavlos Parissis wrote:
> > On Παρασκευή, 6 Δεκεμβρίου 2019 9:23:24 Π.Μ. CET Sander Hoentjen wrote:
> >> Hi list,
> >>
> >> After updating from 1.8.13 to 2.0.5 (also with 2.0.10) we are seeing
> >> kernel panics on our production servers. I haven't been able to trigger
> >> them on a test server, and we rollbacked haproxy to 1.8 for now.
> >>
> >> I am attaching a panic log, hope something useful is in there.
> >>
> >> Anybody an idea what might be going on here?
> >>
> > Have you noticed any high CPU utilization prior the panic?
> >
> Nothing out of the ordinary, but I have only minute data, so I don't 
> know for sure things about seconds before crash.
> 

Then I suggest to configure sar tool to pull/store metrics every 1 second for 
some period in order to
see if the panic is the result of CPU(s) spinning at 100%, either at user or 
system level. That will provide some hints
to haproxy developers.

Another idea is to try haproxy version 2.1.x.

Cheers,
Pavlos

Re: kernel panics after updating to 2.0

[Pavlos Parissis]

On Παρασκευή, 6 Δεκεμβρίου 2019 9:23:24 Π.Μ. CET Sander Hoentjen wrote:
> Hi list,
> 
> After updating from 1.8.13 to 2.0.5 (also with 2.0.10) we are seeing 
> kernel panics on our production servers. I haven't been able to trigger 
> them on a test server, and we rollbacked haproxy to 1.8 for now.
> 
> I am attaching a panic log, hope something useful is in there.
> 
> Anybody an idea what might be going on here?
> 

Have you noticed any high CPU utilization prior the panic?

Cheers,
Pavlos

Re: using hashicorp vault for storing SSL certs

[Pavlos Parissis]

On Κυριακή, 29 Σεπτεμβρίου 2019 7:52:12 Μ.Μ. CEST Илья Шипицин wrote:
> hello,
> 
> is anybody using https://www.vaultproject.io/docs/secrets/pki/index.html
> for storing certs ? (I want to avoid reinventing the wheel here)
> 
> thanks,
> Ilya Shipitcin
> 

I have, but with a different system, but it doesn't matter a lot from HAProxy 
point of view.
Try to avoid the typical mistake where haproxy can't reload anymore as the 
backend for the
Certs is down and as a result the certs aren't available anymore. I know some 
people care so much
about security that they forget about availability.

My 2 cents,
Pavlos

Re: fullconn not working

[Pavlos Parissis]

On Παρασκευή, 28 Ιουνίου 2019 5:50:48 Μ.Μ. CEST Patrick Hemmer wrote:
> I'm trying to get fullconn working, and can't seem to do so. I dunno if 
> it's a bug, or if it's my understanding that's wrong.
> Basically my goal is to prevent the cumulative total of all connections 
> to all servers in a pool from exceeding a certain value.
> For example I might have 10 servers, each with a maxconn of 10. But I 
> want to configure haproxy with a pool-wide limit of 50, so that even if 
> the connections are well distributed and no one server is maxed out, 
> after 50 connections to all servers, haproxy will still start to queue 
> instead.
> 
> fullconn seems like the right way to accomplish this, however I cannot 
> get it to work. I've tried a simple setup of 2 servers, each with 
> `maxconn 3`, and then a backend `fullconn 2`, which should result in 
> queuing after 2 simultaneous connections, however it doesn't. If I send 
> 4 connections, all 4 are simultaneously sent to the backend servers.
> 
> Here's my test config:
> defaults
>  log 127.0.0.1:1234 daemon
>  mode http
>  option httplog
>  timeout queue 5s
> frontend f1
>  bind :8001
>  default_backend b1
> backend b1
>  fullconn 2
>  server s1 127.0.0.1:8081 minconn 1 maxconn 3
>  server s2 127.0.0.1:8081 minconn 1 maxconn 3
> 
> Here's how I test:
> for i in {1..4}; do curl "http://localhost:8001/?sleep=2=$i; & done
> 
> And here's the logs:
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55119 
> [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
> 4/4/3/2/0 0/0 "GET /?sleep=2=3 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55117 
> [28/Jun/2019:11:37:45.658] f1 b1/s2 0/0/0/2003/2003 200 75 - -  
> 4/4/2/1/0 0/0 "GET /?sleep=2=4 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55118 
> [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
> 4/4/1/2/0 0/0 "GET /?sleep=2=1 HTTP/1.1"
> <30>Jun 28 11:37:47 haproxy[75322]: 127.0.0.1:55120 
> [28/Jun/2019:11:37:45.658] f1 b1/s1 0/0/0/2003/2003 200 75 - -  
> 4/4/0/1/0 0/0 "GET /?sleep=2=2 HTTP/1.1"
> 
> 

Your e-mail client mangled above log lines and as a result they are bit 
unreadable.
The 4th field from `4/4/3/2/0`  is srv_conn  *2* which is below the maxconn of 
*3*, so haproxy did
the right thing as it didn't allow more than *full_conn* connections to be 
concurrently opened
against the server.

Cheers,
Pavlos

Re: How to get overall stats in haproxy?

[Pavlos Parissis]

On Tue, 11 Jun 2019 at 10:03,  wrote:
>
> Hi
>
> I'm using multiple processes haproxy, how can I get overall stats in haproxy?
>
> I have to access every process's stats page, and get the stats of that 
> process and count.
>
> Is there any convenient way to get overall stats?
>
> Thanks
>
>

>From the command line, you can use
https://github.com/unixsurfer/haproxytool

and for pushing stats to Graphite you can use
https://github.com/unixsurfer/haproxystats

Alternatively, you can switch to threads where stats are aggregated by haproxy

Cheers,
Pavlos

Re: two maps with the same id

[Pavlos Parissis]

On 11/4/19 2:56 μ.μ., Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Thu, Apr 11, 2019 at 09:47:41AM +0200, Pavlos Parissis wrote:
>> Hi,
>>
>> I am having a rather simple config[1] for testing few changes in 
>> haproxyadmin library and I noticed
>> that two different map files have the same ID:
>>
>> echo 'show map' | socat /run/haproxy/admin1.sock  -
>> # id (file) description
>> -1 (/etc/haproxy/test_map.map) pattern loaded from file 
>> '/etc/haproxy/test_map.map' used by map at
>> file '/etc/haproxy/haproxy.cfg-haproxystats' line 75
>> -1 (/etc/haproxy/test2_map.map) pattern loaded from file 
>> '/etc/haproxy/test2_map.map' used by map at
>> file '/etc/haproxy/haproxy.cfg-haproxystats' line 79
> (...)
> 
> Thanks for your detailed report and the reproducer, I thought I was
> becoming crazy until I figured it only happens for maps created with
> use_backend, log-format, or unique-id-format rules. -1 is the
> uninitialized value. These maps are simply not fully initialized
> because the initialization was done before they were loaded very
> late in the boot process, so you cannot correctly update them on
> the CLI and they will not use the pattern cache.
> 
> That's fixed with the attached patch that I just merged, all versions
> since 1.5 have the same bug!

Thanks a lot for this, as always, very fast response and solution to the issue!

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

two maps with the same id

[Pavlos Parissis]

Hi,

I am having a rather simple config[1] for testing few changes in haproxyadmin 
library and I noticed
that two different map files have the same ID:

echo 'show map' | socat /run/haproxy/admin1.sock  -
# id (file) description
-1 (/etc/haproxy/test_map.map) pattern loaded from file 
'/etc/haproxy/test_map.map' used by map at
file '/etc/haproxy/haproxy.cfg-haproxystats' line 75
-1 (/etc/haproxy/test2_map.map) pattern loaded from file 
'/etc/haproxy/test2_map.map' used by map at
file '/etc/haproxy/haproxy.cfg-haproxystats' line 79

% echo 'show map #-1' | socat /run/haproxy/admin1.sock  -


0x557ffcbe8970 foor bar


% echo 'show map /etc/haproxy/test_map.map' | socat /run/haproxy/admin1.sock  -


0x557ffcbe8970 foor bar


% echo 'show map /etc/haproxy/test2_map.map' | socat /run/haproxy/admin1.sock  -


0x557ffcc63be0 for bar
0x557ffcc65d20 one two
0x557ffcc65da0 2 three


Is this expected behavior ?

I am using HAProxy version 1.8.19-a3cfe8-8.

Cheers,
Pavlos

[1]
global
log 127.0.0.1 len 2048 local2
chroot /var/lib/haproxy
stats socket /run/haproxy/admin1.sock mode 666 level admin process 1
stats socket /run/haproxy/admin2.sock mode 666 level admin process 2
stats socket /run/haproxy/admin3.sock mode 666 level admin process 3
stats socket /run/haproxy/admin4.sock mode 666 level admin process 4
#stats socket  127.0.0.1: level admin process 1
#stats socket  127.0.0.1:5556 level admin process 2
#stats socket  127.0.0.1:5557 level admin process 3
#stats socket  127.0.0.1:5558 level admin process 3
#stats socket  ::1:5558   level admin process 3
stats timeout 30s
stats maxconn 5
user haproxy
group haproxy
daemon
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 1
cpu-map 4 0
maxconnrate 333
defaults
log global
backlog 65535
modehttp
option  httplog
option  dontlognull
timeout connect 5000
timeout client  5
timeout server  5

frontend frontend_all
bind :83 process 1
bind :83 process 2
bind :83 process 3
bind :83 process 4
acl wl_dev src -f /etc/haproxy/wl_stats
default_backend backend_all

backend backend_all
bind-process 1-4
default-server inter 1000s
option httpchk GET / HTTP/1.1\r\nHost:\ app.foo.com\r\nUser-Agent:\ HAProxy
server srv1_all 127.0.0.1:8001 check fall 2 inter 5s rise 3
server srv2_all 127.0.0.1:8002 check fall 2 inter 5s rise 3
server srv3_all 127.0.0.1:8003 check fall 2 inter 5s rise 3
server srv4_all 127.0.0.1:8004 check fall 2 inter 5s rise 3

frontend frontend_proc34
bind :84 process 3
bind :84 process 4
default_backend backend_proc34

backend backend_proc34
bind-process 3-4
default-server inter 1000s
option httpchk GET / HTTP/1.1\r\nHost:\ app.foo.com\r\nUser-Agent:\ HAProxy
server srv1_proc34 127.0.0.1:83 check fall 2 inter 25s rise 3

frontend frontend_proc4
bind :85 process 4
default_backend backend_proc4

backend backend_proc4
bind-process 4
default-server inter 1000s
option httpchk GET / HTTP/1.1\r\nHost:\ app.foo.com\r\nUser-Agent:\ HAProxy
server srv1_proc4 127.0.0.1: check fall 2 inter 25s rise 3

frontend frontend2_proc34
   bind :85 process 3
   bind :85 process 4
   use_backend %[req.hdr(host),lower,map(/etc/haproxy/test_map.map)]

frontend frontend2_proc3
   bind :86 process 3
   use_backend %[req.hdr(host),lower,map(/etc/haproxy/test2_map.map)]

backend backend_dynamic
server-template srv 1-5 127.0.0.5:8080 check disabled




signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] haproxy-2.0-dev2

[Pavlos Parissis]

Hoi Willy!

What a detailed and fruitful response, as always

On 27/3/19 10:56 μ.μ., Willy Tarreau wrote:
> Hi Pavlos!
> 
> On Wed, Mar 27, 2019 at 09:57:32PM +0100, Pavlos Parissis wrote:
>> Have you considered enabling SO_INCOMING_CPU socket option in
>> order to increase data locality and CPU cache hits?
> 
> No, really for our use case I'm not convinced at all by it, I'm only
> seeing risks of making things much worse for the vast majority of use
> cases. First, my experience of haproxy+network stack has always shown
> that sharing the same CPU for both is the second worst solution (the
> worst one being to use different sockets on a NUMA system). Indeed,
> haproxy requires quite some CPU and the network stack as well. When
> running in keep-alive it's common to see 50/50 user/system. So when
> both are on the same CPU, each cycle used by the kernel is not
> available to do user stuff an conversely. With processes, I found
> that the best model was to have haproxy on half of the cores (plus
> their respective siblings if HT was in use) and half of the cores
> for the network interrupts. This means that data would flow via the
> L3 cache, which usually is pretty fast (typically 10ns access time)
> and moves only once per direction. Also, haproxy manipulates a lot
> of data and causes lots of L1 cache thrashing, something you
> absolutely don't want to experience in the kernel where you need to
> have a very fast access to your socket hash tables and various other
> structures. Keep in mind that simply memcpy() 16 kB from one socket
> to a buffer and, sending them again to the SSL stack then back to the
> kernel destroys 48kB of cache. My skylake has 32kB of L1... 
> 

OK, I have to admit that I didn't think about thrashing L1 cache. My bad.

> Placing the network on the first thread of all cores and haproxy on
> the second thread of all cores was sometimes better for SSL since
> you can have more real cores for crypto code, but then you have the
> fact that you still trash the L1d a lot, and quite a part of the L1i
> as well, both of which are shared between the two siblings, with
> apparently varying limits that manage to protect both from the other
> one to some extents. In general when doing this you'd observer a lower
> max connection rate but a higher SSL rate.
> 
> Nowadays with threads I'm seeing that running haproxy and the system
> on siblings of the same core doesn't have as negative an impact as it
> used to. One reason certainly is that threads share the same code and
> the same data, and that by having all this code and data readily
> available in L2, L1 can quickly be refilled (3 ns).
> 
> The other thing is that if you are certain to control your network
> card's affinity (i.e. no single-flow attack will keep a single CPU
> busy), then it's reasonable to let haproxy run on all cores/threads
> and the same for the network because the level of performance you
> can reach is so much indecent that the comfort provided by such a
> simple and adaptable configuration completely overweights the losses
> caused by bad cache interactions since you'll hardly ever need to
> see the two compete.

This is the setup we ended up, but with HT disabled and having the first
two CPUs handling resources from side-cars(rsyslog and etc), so haproxy
uses 12 out of 14 CPUs while IRQ queues are handled by CPUs. During
peak traffic CPU utilization for SoftIrq level never goes above 6% and
only during stress testing we noticed usage close to 40%. As you rightly
wrote simplicity matters and we opted for simplicity.

> 
> Now to get back to the socket vs CPU affinity, the only case where
> it would not have a negative effect is when each thread is bound to
> a single CPU and your NIC is perfectly bound with one queue per CPU
> as well. 

But, on different CPUs, otherwise we go back to the situation of thrashing CPU
cache.

> First, you get back to the very painful configuration that
> can regularly break (we've seen NICs lose their bindings on link
> reset for example), and in this case when everything is bound by
> hand you're back to the risk of a single-flow attack that fills one
> core from a trivial ACK flood on a single TCP stream.
> 

Above is a general problem and irqbalancer can solve it, but irqbalaner
has proven almost always to take the wrong decision. I believe the majority of
people have it disabled and use either affinity scripts from NIC provider
or custom code in a configuration system(puppet and etc).

> In all other cases, cpu-map missing, "process" missing on the bind
> lines or more than one thread referenced, RPS enabled on the system
> to be more resistant to attacks, or simply some intense traffic on
> the network side preventing haproxy from making good progress, well,
> you can hav

Re: [ANNOUNCE] haproxy-2.0-dev2

[Pavlos Parissis]

On 26/3/19 6:24 π.μ., Willy Tarreau wrote:
> Hi,
> 
> HAProxy 2.0-dev2 was released on 2019/03/26. It added 176 new commits
> after version 2.0-dev1.
> 
> This version starts more important changes. One of the most visible
> ones is that haproxy will now automatically start with threads enabled
> if neither "nbthread" nor "nbproc" is configured. It will check the
> number of CPUs it's running on and will start as many threads. This
> means that it will not be necessary anymore to adjust the configuration
> to adjust the number of threads and the CPUs bound, by just setting the
> affinity in the service's configuration, haproxy will automatically
> adapt and use the same number of threads. 

Nice work. Have you considered enabling SO_INCOMING_CPU socket option in
order to increase data locality and CPU cache hits?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] CONTRIB: contrib/prometheus-exporter: Add a Prometheus exporter for HAProxy

[Pavlos Parissis]

On 8/2/19 11:11 π.μ., Willy Tarreau wrote:
> Hi Christopher,
> 
> 
> On Thu, Feb 07, 2019 at 10:09:52PM +0100, Christopher Faulet wrote:
>> Hi,
>>
>> This patch adds a new component in contrib. It is a Prometheus exporter for
>> HAProxy.
> (...)
> 
> Thanks for doing this. After reading the whole patch, I measure how
> uninteresting an experience this must have been! After all, using C
> to produce yet-another-format is akin to using python to write
> yet-another-http-proxy :-)
> 
> I totally agree with your approach of placing it under contrib/. After
> all we've already done the same with other extensions explicitly
> targetting products (mod_security, mod_defender, systemd, ...). 
> We support standards, not products. And products which respect standards
> are naturally supported via the standards, so there is indeed no reason
> for opening a new can of worms by inviting $PRODUCT_OF_THE_MONTH into
> src/ especially when these products change faster than our maintenance
> cycle.
> 
> In my opinion the right place for a stats exporter is : outside. However
> I'm well aware that our export formats are not necessarily friendly to
> such exporters. For example, the fact that prometheus uses this funny
> ordering forces a gateway to keep all metrics in memory before being
> able to dump them. It's not cool either. We could have a long-term
> approach consisting in trying to implement multiple tree walk methods
> combined with a few formats so that implementing external exporters in
> various languages becomes trivial. In this case such tools could provide
> high quality agents to collect our metrics by default without having to
> work around some limitations or constraints.
> 
> This is probably a design discussion to have for the long term here on
> the list : what are the main stats export mechanisms desired in field.
> I can imagine that most agents will want to poll haproxy and dump the
> whole stats once in a while, some will rely on it to send a full dump
> once in a while (this may already become an issue during reloads), some
> might possibly want to subscribe to change notification of certain
> metrics, or receive a diff from the previous dump once in a while. And
> for all these variations I guess we may have to implement 2 or 3 dump
> styles :
>   - Front -> Back -> Server -> metric
>   - { Front, Back, Server } -> metric
>   - metric: { front, back, server }
> 
> I don't know if I'm completely off or not, but I do think that those who
> have experience with such tools should definitely join the discussion to
> share their observations and deployment difficulties met in field.
> 


There are mainly two ways to get metrics out of software:
1. Push, where foobar software uploads stats to a remote/local entry point. 
Graphite is one of the
most used system for this.
2. Pull, the metrics pipeline/infra scrapes foobar software to fetch metrics.

The above is the easy part of getting metrics, the most challenging is data 
types(counters, gauges,
summaries) and format.

Graphite has a very simply format and data type, you send strings over TCP 
connections:
metricname value timestamp(epoch)

where metricname looks like a FS tree:
loadbalancers.edge.lb-01.haproxy.frontend.www_haproxy_org.lbtol 124 1549661217

Prometheus uses the pull method and it is a bit more complicated.

You also have "proxy" systems like telegraf/fluentd that can work on any method 
and build a bridge
between foobar software and any metric pipeline. All those "proxy" systems 
allow you to write any
transformer you want, so the options are countless.

I have to agree with you that supporting all possible combinations is quite 
hard.
Some software support both, some only one of them. More and more you see new 
software have
"instrumentation" out of the box using the pull method and default to 
Prometheus model.

I personally find the cvs data we get out of STATS socket easy to use. I can 
easily write a software
to support both models. I have written one to support the Push method and use 
Graphite as a metric
pipeline and it would be trivial to write an exporter for Prometheus and for 
another X system in two
years.

I prefer the foorbar software to give me raw data and let me decide how I will 
use them. I don't
want to have any kind of aggregation at the source of the metrics or any other 
funny things, which
can use issues when I do aggregation at the store or during visualization.

My 2cents,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: Balance based on network/cpu load

[Pavlos Parissis]

On 13/11/18 9:37 π.μ., Bruno Henc wrote:
> Hello,
> 
> 
> Not sure if there is a direct way to do this, but you can always create a 
> monitoring process that
> will use the haproxy runtime API to MAINT or DRAIN a server until the CPU / 
> network load drops. So
> you have a simple watchdog process which reads the output from your 
> monitoring tools to decide if a
> server needs to be disabled or re-enabled.
> 

Another approach is to use agent-check from haproxy to query a sidecar process 
on the backend
servers. That sidecar process determines, based on various criteria, the load 
of the server and
instructs haproxy to either drain or reduce the percentage of traffic that it 
gets.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] DOC: fix various typos in the CHANGELOG

[Pavlos Parissis]

On 8/11/18 5:48 π.μ., Willy Tarreau wrote:
> Hi Joseph,
> 
> On Wed, Nov 07, 2018 at 06:03:14PM -0800, Joseph Herlant wrote:
>> Hi,
>> 
>> Please find attached a patch to cleanup typos in the changelog.
>> I'm not sure where you stand on cleaning up typos in changelogs,
>> some people totally refuse to allow that, hence a totally separate
>> patch for this. Anyway, here's a cleanup proposal for it.
> 

[..snip..]
> Just thinking about something, how did you figure all these typos ?
> By reading everything, or aided by some automated tool or maybe a mix
> of all of this ? I'm asking in case we could try to improve our tools
> to spot mistakes before and after patch submissions.

I'm not a developer but an engineer who writes code when it is needed,
thus I rely on understandable commit description. Once I was told to
write commits like that the first line could be append to the sentence
"If you apply commit it will" and it will make sense. The long commit
description will follow the typical flow of What is the problem->Why it
is a problem->how we solve it.

To avoid typos I simply force my self to "git show" after 5secs I
finished git commit -av. I use spell checker on my editor and if
the commit message is 100lines I may even print it. Do I avoid typos?
Not always, I am getting better.

About understandable commit description, I love Willy's commit
description his commits taught me how I should write them as I have
learned so many things about haproxy by reading his commits.

My 2cents,
Pavlos







signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] haproxy-1.9-dev4

[Pavlos Parissis]

On 10/22/18 11:15 AM, Willy Tarreau wrote:
> Hi Pavlos!
> 
> On Mon, Oct 22, 2018 at 11:01:37AM +0200, Pavlos Parissis wrote:
>> On 10/21/18 9:05 PM, Willy Tarreau wrote:
>>> Hi,
>>>
>>> HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
>>> after version 1.9-dev3.
>>>
>>> There's not much user-visible here, it's mostly another merge of some
>>> pending infrastructure changes. The most sensitive changes consist in
>>> the finalization of the connection reorientation from top to bottom,
>>> so that we don't need the conn_want_* tricks from the upper layers nor
>>> the update_poll() calls anymore. Everything is attempted directly and
>>> a subscription to the lower layer is made in case of failure. The perf
>>> is slightly better than with dev3, but more importantly the code becomes
>>> much cleaner and straightforward. An optimization was made in the
>>> scheduler regarding the wait queues, most of which are lockfree now.
>>> Another one concerns the FD lock which is taken less often as well.
>>> All in all the overall multi-thread performance has increased quite
>>> a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.
>>>
>>
>> Nice, very nice. HAProxy version 1.9 would be very exciting release.
> 
> Yes, we all hope so, but we also expect that it may still have rough edges,
> being a technical version.
> 
>> Any ideas if we get see gRPC support on both ends(client/server side) in 1.9
>> version ?
> 
> While we won't be doing anything specific for gRPC, my previous reading of
> the spec taught me that if we simply support H2 end to end it *should* work.

So, haproxy wont be able to understand services and calls of those services and 
take
load balancing decisions in the same way we do with HTTP1 where we can take 
decisions
base on verbs(HTTP Methods) and nouns(URL, cookie and etc).

I have to mention that I know very little about gRPC/brotobuf and I have looked 
what
other products offer when they claim that they support gRPC and I didn't find 
any
other software loadbalancer that is able to route traffic based on the 
verbs/nouns.
It could be that my expectation is unrealistic and lb software doesn't need to
take any decisions based on those verbs/nouns.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] haproxy-1.9-dev4

[Pavlos Parissis]

On 10/21/18 9:05 PM, Willy Tarreau wrote:
> Hi,
> 
> HAProxy 1.9-dev4 was released on 2018/10/21. It added 97 new commits
> after version 1.9-dev3.
> 
> There's not much user-visible here, it's mostly another merge of some
> pending infrastructure changes. The most sensitive changes consist in
> the finalization of the connection reorientation from top to bottom,
> so that we don't need the conn_want_* tricks from the upper layers nor
> the update_poll() calls anymore. Everything is attempted directly and
> a subscription to the lower layer is made in case of failure. The perf
> is slightly better than with dev3, but more importantly the code becomes
> much cleaner and straightforward. An optimization was made in the
> scheduler regarding the wait queues, most of which are lockfree now.
> Another one concerns the FD lock which is taken less often as well.
> All in all the overall multi-thread performance has increased quite
> a bit. I measured a gain of 60% over 1.8 using only H2 on 4 threads.
> 

Nice, very nice. HAProxy version 1.9 would be very exciting release.

Any ideas if we get see gRPC support on both ends(client/server side) in 1.9 
version ?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: gRPC protocol

[Pavlos Parissis]

On 5/24/18 5:54 PM, Daniel Corbett wrote:
> Hello Aleks,
> 
> 
> On 05/24/2018 10:54 AM, Aleksandar Lazic wrote:
>>
>> I remembert that Willy mentioned this in any of his mail.
>> Do you have any rough timeline, this year, next year something like this
>> ;-)
>>
> 
> We're aiming to have the native internal HTTP representation completed for 
> 1.9 which is slated for
> an -rc1 around the end of September with a potential release between 
> mid-October and the end of
> November.  While I cannot make any promises we're hoping to have gRPC added 
> within this release as
> well.
> 

Can you share with us any update on this? Will HAProxy version 1.9 support gRPC?

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: Considering adding support for TCP Zero Copy

[Pavlos Parissis]

On 03/05/2018 07:24 μμ, Willy Tarreau wrote:
> On Thu, May 03, 2018 at 02:51:12PM +0200, Pavlos Parissis wrote:
>> On 03/05/2018 02:45 uu, Olivier Houchard wrote:
>>> Hi Pavlos,
>>>
>>> On Thu, May 03, 2018 at 12:45:42PM +0200, Pavlos Parissis wrote:
>>>> Hi,
>>>>
>>>> Linux kernel version 4.14 adds support for zero-copy from user memory to 
>>>> TCP sockets by setting
>>>> MSG_ZEROCOPY flag. This is for the sending side of the socket, for the 
>>>> receiving side of the socket
>>>> we need to wait for kernel version 4.18.
>>>>
>>>> Will you consider enabling this on HAProxy?
>>>>
>>>> More info can be found here, 
>>>> https://www.kernel.org/doc/html/latest/networking/msg_zerocopy.html
>>>
>>> After some discussion with Willy, we're not sure it is worth it.
>>> It would force us to release buffer much later than we do actually, it can't
>>> be used with SSL, and we already achieve zero-copy by using splicing.
>>>
>>> Is there any specific case where you think it'd be a huge win ?
>>>
>>
>> The only use case that I can think of is HTTP streaming. But, without 
>> testing it we can't say a lot.
> 
> In fact, for HTTP streaming, splicing already does it all and even
> better since it only manipulates a few pointers in the kernel between
> the source and destination socket buffers. Userspace is not even
> involved.
> 
> Also it's important to remember that while copies are best avoided
> whenever possible, they aren't that dramatic at the common traffic
> rates. I've already reached 60 Gbps of forwarded traffic with and
> without splicing on a 4-core machine.
> 
> One aspect to keep in mind is the following. A typical Xeon system will
> achieve around 20 GB/s of in-L3 memcpy() bandwidth. For a typical 16kB
> buffer, that's only 760 ns to copy the whole buffer, which is roughly the
> cost of the extra syscall needed to check that the transfer completed.
> At 10 Gbps, this represents only 6.25% of the total processing time.
> And there's something much more important : with the copy operation,
> the buffer is released after these 760 ns and immediately recycled for
> other connections. This ensures that the memory usage remains low and
> that most transfer operations are made in L3 instead of RAM. If you
> use zero-copy here, instead your memory will be pinned for the time
> it takes to cycle on many other connections and get back to processing
> this FD. It can very easily become 10-100 microseconds, or 15-150 times
> more, resulting in much more RAM usage for temporary buffers, and thus
> a much higher cache footprint.
> 
> In my opinion MSG_ZEROCOPY was designed for servers, those which stream
> video and so on, and which produce their own data, and which don't need
> to recycle their buffers. We're definitely not in this case at all here,
> we're just forwarding ephemeral data so we can recycle buffers very quickly
> and through splicing we can even avoid to see these data at all.
> 
> Hoping this helps,
> Willy
> 

Thanks for this very detailed response, once again I learned a lot.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Considering adding support for TCP Zero Copy

[Pavlos Parissis]

On 03/05/2018 02:45 μμ, Olivier Houchard wrote:
> Hi Pavlos,
> 
> On Thu, May 03, 2018 at 12:45:42PM +0200, Pavlos Parissis wrote:
>> Hi,
>>
>> Linux kernel version 4.14 adds support for zero-copy from user memory to TCP 
>> sockets by setting
>> MSG_ZEROCOPY flag. This is for the sending side of the socket, for the 
>> receiving side of the socket
>> we need to wait for kernel version 4.18.
>>
>> Will you consider enabling this on HAProxy?
>>
>> More info can be found here, 
>> https://www.kernel.org/doc/html/latest/networking/msg_zerocopy.html
> 
> After some discussion with Willy, we're not sure it is worth it.
> It would force us to release buffer much later than we do actually, it can't
> be used with SSL, and we already achieve zero-copy by using splicing.
> 
> Is there any specific case where you think it'd be a huge win ?
> 

The only use case that I can think of is HTTP streaming. But, without testing 
it we can't say a lot.

Thanks,
Pavlos



signature.asc
Description: OpenPGP digital signature

Considering adding support for TCP Zero Copy

[Pavlos Parissis]

Hi,

Linux kernel version 4.14 adds support for zero-copy from user memory to TCP 
sockets by setting
MSG_ZEROCOPY flag. This is for the sending side of the socket, for the 
receiving side of the socket
we need to wait for kernel version 4.18.

Will you consider enabling this on HAProxy?

More info can be found here, 
https://www.kernel.org/doc/html/latest/networking/msg_zerocopy.html

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Updating the unofficial GitHub mirror

[Pavlos Parissis]

On 12/03/2018 08:07 πμ, Willy Tarreau wrote:
> Hi Dan,
> 
> On Sun, Mar 11, 2018 at 12:55:59PM -0400, Dan Kohn wrote:
>> I'm the author of the CNCF Cloud Native Interactive Landscape, which tracks 
>> projects and
>> products used in open source. We include HAProxy:
>> 
>> https://landscape.cncf.io/grouping=no=ha-proxy=latest-commit
>> 
>> However, our info on latest commit is out of date because 
>> https://github.com/haproxy/haproxy is
>> no longer mirroring http://www.haproxy.org/. Would it be possible to update 
>> the mirror,
>> please?
> 
> The Github link above is not a mirror, it's an unofficial fork with a few 
> local patches which has
> already caused some confusion in the past as it was receiving issues and pull 
> requests that were
> ignored.

Have you asked the maintainer of the github fork to change the project to 
'unofficial-haproxy' or
something else?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH v2 0/3] Add SystemD's sandboxing options

[Pavlos Parissis]

On 27/02/2018 08:19 μμ, Tim Duesterhus wrote:
> Willy,
> 
> okay. I added an additional comment about the nature of those options in
> the first commit and then added the various settings in commented out
> versions. For reference, these are the settings I add on top of Debian's
> default unit file (haproxy 1.8.4 om Debian Stretch) for one of my production
> instances of haproxy:
> 
> # /lib/systemd/system/haproxy.service.d/config.conf
> [Service]
> Environment=CONFIG=/usr/share/haproxy/
> # /lib/systemd/system/haproxy.service.d/no-pidfile.conf
> [Service]
> ExecStart=
> ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG
> # /lib/systemd/system/haproxy.service.d/security.conf
> [Service]
> ProtectSystem=strict
> ProtectHome=true
> ProtectKernelTunables=true
> ProtectKernelModules=true
> ProtectControlGroups=true
> SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
> NoNewPrivileges=true
> # /lib/systemd/system/haproxy.service.d/state.conf
> [Service]
> RuntimeDirectory=haproxy
> ExecReload=
> ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
> ExecReload=/bin/sh -c "echo show servers state |nc -U 
> /var/run/haproxy/admin.sock > /run/haproxy/global-state"
> ExecReload=/bin/kill -USR2 $MAINPID
> 
> I'm open for further feedback from the other participants in this thread
> as well!
> 
> Best regards
> 
> Tim Duesterhus (3):
>   MINOR: systemd: Add section for SystemD sandboxing to unit file
>   MINOR: systemd: Add SystemD's Protect*= options to the unit file
>   MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file
> 

I am fine with adding the comments and thanks for accepting the feedback.
BTW: The commit message is a bit misleading because If I don't read the code I 
will
think that those options are enabled, which isn't true. So, you may want to 
mention they aren't
enabled by default.

Thanks,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH 0/2] Add SystemD's sandboxing options

[Pavlos Parissis]

On 27/02/2018 04:00 μμ, Willy Tarreau wrote:
> Hi Tim,
> 
> On Thu, Feb 22, 2018 at 03:03:58PM +0100, Tim Duesterhus wrote:
>> I'm running this exact settings on my Debian Stretch machine using haproxy
>> 1.8.x, without issues so far.
>>
>> The first patch could cause issues for users that store their configuration
>> in /home or /root, but I consider this unlikely.
>>

How do you know that?

>> Tim Duesterhus (2):
>>   MINOR: systemd: Add SystemD's Protect*= options to the unit file
>>   MINOR: systemd: Add SystemD's SystemCallFilter option to the unit file
> 
> I took a look, but my systemd incompetence limited my ability to understand
> what this really does. How does systemd act to do this exactly ? I'm very
> worried that the only way it could proceed would be by running the process
> under ptrace causing a tremendous slowdown, and additionally making the
> process unobservable/undebuggable. Do you know how it proceeds internally ?
> 
> Thanks,
> Willy
> 

I am pretty much against this. systemd allows users to extend the systemd 
configuration
of a service (haproxy in this case), by dropping a file under
etc/systemd/system/haproxy.service.d directory. If user X or Distribution X 
wants to harden
the default systemd configuration of HAProxy then they can do it. But, I don't 
think it is
the task of haproxy devs to ship a configuration with zero Return Of Investment 
and potential
breakage.

My 2 cents,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: What is a nice way to bypass the maintenance mode for certain IP's?

[Pavlos Parissis]

On 20/02/2018 09:11 πμ, Pieter Vogelaar wrote:
> Hi Willy,
> 
> Thanks I will look into that!
> 
> On the statistics report page it's possible to set all servers of a backend 
> in maintence mode. Is it also possible to set the servers of all backends in 
> maintenance mode?
>  
> Best regards,
> Pieter Vogelaar
> 

You may want to look at https://github.com/unixsurfer/haproxytool, as it does 
exactly what you want
  % haproxytool server -d bck_all_srv1
  Are you sure we want to disable 3 servers y/n?: y
  bck_all_srv1 disabled in backend1_proc34 backend
  bck_all_srv1 disabled in backend_proc1 backend
  bck_all_srv1 disabled in backend2_proc34 backend

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

[PATCH] DOC: Mention -Ws in the list of available options

[Pavlos Parissis]

Hi,

Please consider applying the attached patch.

It is a patch for the management.txt and adds the '-Ws' in the list of 
available options. haproxy
--help reports that option, so I thought we need to have in the document as 
well.

The patch mentions that haproxy must be build with `USE_SYSTEMD` build option 
enabled, otherwise
this option is not available.

Cheers,
Pavlos
From 426a4b37f7f1347a3e017db8f577b81f34f51f13 Mon Sep 17 00:00:00 2001
From: Pavlos Parissis <pavlos.paris...@booking.com>
Date: Wed, 7 Feb 2018 21:42:16 +0100
Subject: [PATCH] DOC: Mention -Ws in the list of available options

---
 doc/management.txt | 4 
 1 file changed, 4 insertions(+)

diff --git a/doc/management.txt b/doc/management.txt
index af216521a..1488862e7 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -176,6 +176,10 @@ list of options is :
 is compatible either with the foreground or daemon mode.  It is
 recommended to use this mode with multiprocess and systemd.
 
+  -Ws : master-worker mode with support of `notify` type of systemd service.
+This option is only available when HAProxy was built with `USE_SYSTEMD`
+build option enabled.
+
   -c : only performs a check of the configuration files and exits before trying
 to bind. The exit status is zero if everything is OK, or non-zero if an
 error is encountered.
-- 
2.11.0



signature.asc
Description: OpenPGP digital signature

Re: -Ws argument isn't document?

[Pavlos Parissis]

On 03/02/2018 03:53 μμ, Lucas Rolff wrote:
> haproxy --help:
> -W master-worker mode.
> -Ws master-worker mode with systemd notify support.

Stupid me. But, I think it should be mentioned in the document as well. I 
prepared a patched for it.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

-Ws argument isn't document?

[Pavlos Parissis]

Hi,

In contrib/systemd/haproxy.service.in we see -Ws used in ExecStart as it is the 
recommended way to
start haproxy under systemd:

 ExecStart=@SBINDIR@/haproxy -Ws -f $CONFIG -p $PIDFILE

But, it isn't documented in doc/management.txt, only -W is mentioned while I 
failed to find any
references for '-s'.
I assume that '-s' adds certain functionality as I find it in other argument 
combinations (-st and -sf).

Is my understanding wrong?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Warning: upgrading to openssl master+ enable_tls1_3 (coming v1.1.1) could break handshakes for all protocol versions .

[Pavlos Parissis]

On 13/01/2018 04:22 μμ, Lukas Tribus wrote:
> Hello,
> 
> 
> On 13 January 2018 at 15:17, Pavlos Parissis <pavlos.paris...@gmail.com> 
> wrote:
>>> Not exactly, the moment you force a cipher list that does not include a
>>> TLSv1.3 cipher in the server side (which has TLSv1.3 enabled) the TLS
>>> handshake will break regardless of what is in the Client hello.
>>>
>>
>> But, can we have TLSv3 enabled on server side and still accept TLSv2 
>> sessions?
> 
> Only if your cipher-list contains TLSv1.3 ciphers, otherwise nothing
> will work (regardless of the TLS version).
> 

and all those ciphers are supported by all recent browsers, right ?

> OpenSSL really goes the extra mile to make everyone's life miserable.
> 
> 

Is this the result of the implementation or of the TLSv1.3 design ?


Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Warning: upgrading to openssl master+ enable_tls1_3 (coming v1.1.1) could break handshakes for all protocol versions .

[Pavlos Parissis]

On 13/01/2018 01:22 μμ, Moemen MHEDHBI wrote:
> HI Pavlos,
> 
> 
> On 12/01/2018 22:53, Pavlos Parissis wrote:
>> On 12/01/2018 03:57 μμ, Emeric Brun wrote:
>>> Hi All,
>>>
>>> FYI: upgrading to next openssl-1.1.1 could break your prod if you're using 
>>> a forced cipher list because
>>> handshake will fail regardless the tls protocol version if you don't 
>>> specify a cipher valid for TLSv1.3
>>> in your cipher list.
>>>
>>> https://github.com/openssl/openssl/issues/5057
>>>
>>> https://github.com/openssl/openssl/issues/5065
>>>
>>> Openssl's team doesn't seem to consider this as an issue and I'm just bored 
>>> to discuss with them.
>>>
>>> R,
>>> Emeric
>>>
>>
>> So, If we enable TLSv1.3, together with TLSv1.2, on the server side, then 
>> client must support
>> TLSv1.3 otherwise it will get a nice SSL error. Am I right? If I am right, I 
>> hope I'm not, then we
>> have to wait for all clients to support TLSv1.3 before we enabled it on the 
>> server side, this
>> doesn't sound right and I am pretty sure I am completely wrong here.
>>
>> Cheers,
>> Pavlos
>>
>>
> 
> Not exactly, the moment you force a cipher list that does not include a
> TLSv1.3 cipher in the server side (which has TLSv1.3 enabled) the TLS
> handshake will break regardless of what is in the Client hello.
> 

But, can we have TLSv3 enabled on server side and still accept TLSv2 sessions?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Warning: upgrading to openssl master+ enable_tls1_3 (coming v1.1.1) could break handshakes for all protocol versions .

[Pavlos Parissis]

On 12/01/2018 03:57 μμ, Emeric Brun wrote:
> Hi All,
> 
> FYI: upgrading to next openssl-1.1.1 could break your prod if you're using a 
> forced cipher list because
> handshake will fail regardless the tls protocol version if you don't specify 
> a cipher valid for TLSv1.3
> in your cipher list.
> 
> https://github.com/openssl/openssl/issues/5057
> 
> https://github.com/openssl/openssl/issues/5065
> 
> Openssl's team doesn't seem to consider this as an issue and I'm just bored 
> to discuss with them.
> 
> R,
> Emeric
> 


So, If we enable TLSv1.3, together with TLSv1.2, on the server side, then 
client must support
TLSv1.3 otherwise it will get a nice SSL error. Am I right? If I am right, I 
hope I'm not, then we
have to wait for all clients to support TLSv1.3 before we enabled it on the 
server side, this
doesn't sound right and I am pretty sure I am completely wrong here.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: AW: Poll: haproxy 1.4 support ?

[Pavlos Parissis]

On 03/01/2018 08:50 πμ, Maximilian Böhm wrote:
> Debian (Jessie) distributes Haproxy 1.5.8.3
> 

Well, Debian users can also use https://haproxy.debian.net/ to get any version 
they want.
For more details, please read
https://packages.debian.org/search?suite=jessie-backports=names=haproxy

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Poll: haproxy 1.4 support ?

[Pavlos Parissis]

On 02/01/2018 04:23 μμ, Jonathan Matthews wrote:
> On 2 January 2018 at 15:12, Willy Tarreau  wrote:
>> So please simply voice in. Just a few "please keep it alive" will be
>> enough to convince me, otherwise I'll mark it unmaintained.
> 
> I don't use 1.4, but I do have a small reason to say please *do* mark
> it as unmaintained.
> 
> The sustainability of haproxy is linked to the amount of work you (and
> a /relatively/ small set of people) both have to do and want to do.
> I would very much like it to continue happily, so I would vote to
> reduce your mental load and to mark 1.4 as unmaintained.
> 

+1

BTW: I don't use/look at haproxy 1.5 version as well...

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Incredible performance bump between 1.5.12 and 1.8.1 ?

[Pavlos Parissis]

On 09/12/2017 05:01 πμ, Christopher Lane wrote:
> It is plausible/expected that my version upgrade performance goes like (about 
> 4K connections,
> long lived and short lived mix, TCP only, no HTTP:
> 
> 1.5.12 (nproc 1, old connections causing about 100 old -sf processes to 
> linger) uses 100% CPU
> almost all the time, frequently with >1 process.  (100%, 75%, 48%, ... ).  
> Highest CPU user has
> 2-3K connections.
> 
> 1.8.1 (nbproc 4, with hard-stop-after 600s) using like 1%, 3%, 3%, 8% CPU  
> Also 2-3K
> connections.
> 

You compare multi-process setup with a single process setup, which isn't a fair 
comparison.
Do you see the same the performance increase with 1.8.1, when you configure it 
with one process?

BTW: I am not surprised when I see better performance with newer versions,  as 
the development
community is very much performance orientated and always try to add 
functionality without dropping
performance.

Having said that, 1.8 version came with two major functionalities, HTTP2 and 
threading. Both
aforementioned functionalities required internal changes, therefore it would be 
very interesting to
see any possible negative impact on the performance. For that, you need to run 
a detailed benchmark
between 1.7 and 1.8 versions. I am planning to do that (hopefully) next month 
as I have the hardware
to produce 1M HTTPs/sec.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Pavlos Parissis]

On 08/12/2017 10:38 πμ, Pavlos Parissis wrote:
> On 08/12/2017 10:21 πμ, Christopher Faulet wrote:
>> Le 08/12/2017 à 05:52, Willy Tarreau a écrit :
>>>
>>>> pparissis at poseidonas in ~/repo/haproxy-1.8 on (master u=)
>>>> sudo gdb ./haproxy
>>>> GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
>>>> Copyright (C) 2016 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later 
>>>> <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-linux-gnu".
>>>> Type "show configuration" for configuration details.
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>> Find the GDB manual and other documentation resources online at:
>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>> For help, type "help".
>>>> Type "apropos word" to search for commands related to "word"...
>>>> Reading symbols from ./haproxy...done.
>>>> (gdb) run -f /etc/haproxy/haproxy-ams4-dc.cfg
>>>> Starting program: /home/pparissis/repo/haproxy-1.8/haproxy -f
>>>> /etc/haproxy/haproxy-ams4-dc.cfg
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
>>>> [WARNING] 340/221611 (13628) : parsing 
>>>> [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
>>>> 'http-request' rule placed after a 'use_backend' rule will still be 
>>>> processed before.
>>>> ^C
>>>> Program received signal SIGINT, Interrupt.
>>>> 0x5562e302 in register_name (name=0x558ee294 
>>>> "selected_dc_backup",
>>>> len=18, scope=scope@entry=0x7fffd83c,
>>>>  alloc=alloc@entry=1, err=0x7fffdb28) at src/vars.c:215
>>>> 215    HA_RWLOCK_WRLOCK(VARS_LOCK, _names_rwlock);
>>>
>>> Ah, this looks like a double-lock :-)
>>> CCing Christopher who knows better than me how to track them.
>>>
>> Hi,
>>
>> The problem is not a double-lock but an unlock on a free lock, leaving it in 
>> a bad state. This is
>> not obvious on this trace, but compiling HAProxy with the debug on threads 
>> it is easier to observe it.
>>
>> This patch is joined. It should fix the bug. Thanks Pavlos!
> 
> Applied to 1.8 repo and solved the issue as I can start haproxy.
> 

I meant to my local cloned repo:-)

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Pavlos Parissis]

On 07/12/2017 10:18 μμ, Pavlos Parissis wrote:
> On 07/12/2017 07:41 μμ, Willy Tarreau wrote:
>> Hi Pavlos!
>>
>> On Thu, Dec 07, 2017 at 07:16:54PM +0100, Pavlos Parissis wrote:
>>> Hi,
>>>
>>> OK, I haven't read the ML for ~2 weeks and a quick scan didn't reveal 
>>> anything.
>>> So, here I am asking something that may have been addressed already.
>>>
>>> Today, I decided to switch my dev env to haproxy-1.8 using current master 
>>> and I
>>> started haproxy in the same way as I have been doing with older releases:
>>>
>>> sudo ./haproxy -f /etc/haproxy/haproxy-ams4-dc.cfg
>>> [WARNING] 340/173007 (3104) : parsing 
>>> [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
>>> 'http-request' rule placed after a 'use_backend' rule will still be 
>>> processed before.
>>>
>>> above it didn't return and wasn't printing, expect the warning. I curled 
>>> against
>>> the IPs and got back connection error, see attached file for process 
>>> output, lsof
>>> info, build verion and haproxy.cfg.
>>>
>>> I also started in the way it is mentioned in section 3 of management 
>>> document:
>>> sudo ./haproxy  -f /etc/haproxy/haproxy-ams4-dc.cfg -D -p 
>>> /run/haproxy-ams4.pid
>>> -sf $(cat /run/haproxy-ams4.pid)
>>> cat: /run/haproxy-ams4.pid: No such file or directory
>>> [WARNING] 340/173007 (3104) : parsing 
>>> [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
>>> 'http-request' rule placed after a 'use_backend' rule will still be 
>>> processed before.
>>>
>>> but same result, haproxy didn't return and I had to CTRL-C it.
>>>
>>> I am pretty sure I am doing something stupid but I can't find it.
>>>
>>> Any ideas?
>>
>> It looks like it doesn't finish to startup in fact. Are you seeing it spin
>> on the CPU maybe ?
> 
> Yes, it does. I am sorry but I didn't notice it
> 


Found the issue. If I remove all set-var and var(req.) statements from the
config then haproxy loads.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: 1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Pavlos Parissis]

On 07/12/2017 07:41 μμ, Willy Tarreau wrote:
> Hi Pavlos!
> 
> On Thu, Dec 07, 2017 at 07:16:54PM +0100, Pavlos Parissis wrote:
>> Hi,
>>
>> OK, I haven't read the ML for ~2 weeks and a quick scan didn't reveal 
>> anything.
>> So, here I am asking something that may have been addressed already.
>>
>> Today, I decided to switch my dev env to haproxy-1.8 using current master 
>> and I
>> started haproxy in the same way as I have been doing with older releases:
>>
>> sudo ./haproxy -f /etc/haproxy/haproxy-ams4-dc.cfg
>> [WARNING] 340/173007 (3104) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] 
>> : a
>> 'http-request' rule placed after a 'use_backend' rule will still be 
>> processed before.
>>
>> above it didn't return and wasn't printing, expect the warning. I curled 
>> against
>> the IPs and got back connection error, see attached file for process output, 
>> lsof
>> info, build verion and haproxy.cfg.
>>
>> I also started in the way it is mentioned in section 3 of management 
>> document:
>> sudo ./haproxy  -f /etc/haproxy/haproxy-ams4-dc.cfg -D -p 
>> /run/haproxy-ams4.pid
>> -sf $(cat /run/haproxy-ams4.pid)
>> cat: /run/haproxy-ams4.pid: No such file or directory
>> [WARNING] 340/173007 (3104) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] 
>> : a
>> 'http-request' rule placed after a 'use_backend' rule will still be 
>> processed before.
>>
>> but same result, haproxy didn't return and I had to CTRL-C it.
>>
>> I am pretty sure I am doing something stupid but I can't find it.
>>
>> Any ideas?
> 
> It looks like it doesn't finish to startup in fact. Are you seeing it spin
> on the CPU maybe ?

Yes, it does. I am sorry but I didn't notice it

> Otherwise probably that starting it by hand in gdb and
> stopping it to see what it's doing will help.
>


pparissis at poseidonas in ~/repo/haproxy-1.8 on (master u=)
sudo gdb ./haproxy
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./haproxy...done.
(gdb) run -f /etc/haproxy/haproxy-ams4-dc.cfg
Starting program: /home/pparissis/repo/haproxy-1.8/haproxy -f
/etc/haproxy/haproxy-ams4-dc.cfg
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[WARNING] 340/221611 (13628) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] : 
a
'http-request' rule placed after a 'use_backend' rule will still be processed 
before.
^C
Program received signal SIGINT, Interrupt.
0x5562e302 in register_name (name=0x558ee294 "selected_dc_backup",
len=18, scope=scope@entry=0x7fffd83c,
alloc=alloc@entry=1, err=0x7fffdb28) at src/vars.c:215
215 HA_RWLOCK_WRLOCK(VARS_LOCK, _names_rwlock);
(gdb) bt full
#0  0x5562e302 in register_name (name=0x558ee294 
"selected_dc_backup",
len=18, scope=scope@entry=0x7fffd83c,
alloc=alloc@entry=1, err=0x7fffdb28) at src/vars.c:215
i = 
var_names2 = 
tmp = 
res = 0x0
#1  0x5562f489 in vars_check_arg (arg=0x558ee600, err=)
at src/vars.c:502
name = 
scope = SCOPE_REQ
err = 
arg = 0x558ee600
#2  0x55609b94 in sample_parse_expr (str=str@entry=0x558ee598,
idx=idx@entry=0x7fffd964,
file=file@entry=0x558df420 "/etc/haproxy/haproxy-ams4-dc.cfg",
line=line@entry=106, err_msg=err_msg@entry=0x7fffdb28,
al=al@entry=0x558e85d8) at src/sample.c:901
begw = 
endw = 0x558df4c2 "req.selected_dc_backup)"
endt = 0x558df4d8 ")"
expr = 0x558ee5c0
fetch = 0x558ade30 <sample_fetch_keywords+16>
conv = 
prev_type = 6
fkw = 0x558ee2f0 "var"
ckw = 0x0
err_arg = 1
#3  0x55630144 in parse_acl_expr (args=args@entry=0x558ee598,
err=err@entry=0x7fffdb28, al=al@entry=0x558e85d8,
file=file@entry=0x558df420 &quo

1.8.1-fe66fd doesn't finish startup and doesn't listen to sockets

[Pavlos Parissis]

Hi,

OK, I haven't read the ML for ~2 weeks and a quick scan didn't reveal anything.
So, here I am asking something that may have been addressed already.

Today, I decided to switch my dev env to haproxy-1.8 using current master and I
started haproxy in the same way as I have been doing with older releases:

sudo ./haproxy -f /etc/haproxy/haproxy-ams4-dc.cfg
[WARNING] 340/173007 (3104) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
'http-request' rule placed after a 'use_backend' rule will still be processed 
before.

above it didn't return and wasn't printing, expect the warning. I curled against
the IPs and got back connection error, see attached file for process output, 
lsof
info, build verion and haproxy.cfg.

I also started in the way it is mentioned in section 3 of management document:
sudo ./haproxy  -f /etc/haproxy/haproxy-ams4-dc.cfg -D -p /run/haproxy-ams4.pid
-sf $(cat /run/haproxy-ams4.pid)
cat: /run/haproxy-ams4.pid: No such file or directory
[WARNING] 340/173007 (3104) : parsing [/etc/haproxy/haproxy-ams4-dc.cfg:103] : a
'http-request' rule placed after a 'use_backend' rule will still be processed 
before.

but same result, haproxy didn't return and I had to CTRL-C it.

I am pretty sure I am doing something stupid but I can't find it.

Any ideas?

Cheers,
Pavlos

foo at me in ~
curl -v http://10.52.12.2   
  
* Rebuilt URL to: http://10.52.12.2/
*   Trying 10.52.12.2...
* TCP_NODELAY set
* connect to 10.52.12.2 port 80 failed: Connection refused
* Failed to connect to 10.52.12.2 port 80: Connection refused
* Closing connection 0
foo at me in ~ *7
ps|grep haproxy 
  
root  8244  0.0  0.0  51352  3800 pts/25   S+   17:14   0:00  |   |   \_ 
sudo ./haproxy -f /etc/haproxy/haproxy-ams4-dc.cfg
root  8245 99.5  0.0  32468  5704 pts/25   R+   17:14   0:50  |   |   
\_ ./haproxy -f /etc/haproxy/haproxy-ams4-dc.cfg
ppariss+ 10834  0.0  0.0  12788   968 pts/26   S+   17:15   0:00  |   \_ 
grep --colour=auto haproxy

foo at me in ~
sudo lsof -n|grep haproxy   
  
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs
  Output information may be incomplete.
rsyslogd542  root4u unix 0x9b4879369c000t0  
17715 /var/lib/haproxy/dev/log type=DGRAM
in:imuxso   542   570root4u unix 0x9b4879369c000t0  
17715 /var/lib/haproxy/dev/log type=DGRAM
in:imklog   542   571root4u unix 0x9b4879369c000t0  
17715 /var/lib/haproxy/dev/log type=DGRAM
in:imudp542   572root4u unix 0x9b4879369c000t0  
17715 /var/lib/haproxy/dev/log type=DGRAM
rs:main 542   573root4u unix 0x9b4879369c000t0  
17715 /var/lib/haproxy/dev/log type=DGRAM
sudo   8244  root  cwd   DIR  254,4   4096  
  6307210 /home/foo/repo/haproxy-1.8
haproxy8245  root  cwd   DIR  254,4   4096  
  6307210 /home/foo/repo/haproxy-1.8
haproxy8245  root  rtd   DIR  254,1   4096  
2 /
haproxy8245  root  txt   REG  254,47643016  
  6307218 /home/foo/repo/haproxy-1.8/haproxy
haproxy8245  root  mem   REG  254,1  47632  
   915005 /lib/x86_64-linux-gnu/libnss_files-2.24.so
haproxy8245  root  mem   REG  254,1  47688  
   915010 /lib/x86_64-linux-gnu/libnss_nis-2.24.so
haproxy8245  root  mem   REG  254,1  89064  
   915001 /lib/x86_64-linux-gnu/libnsl-2.24.so
haproxy8245  root  mem   REG  254,1  31616  
   915003 /lib/x86_64-linux-gnu/libnss_compat-2.24.so
haproxy8245  root  mem   REG  254,11689360  
   914991 /lib/x86_64-linux-gnu/libc-2.24.so
haproxy8245  root  mem   REG  254,1 468920  
   914073 /lib/x86_64-linux-gnu/libpcre.so.3.13.3
haproxy8245  root  mem   REG  254,1  10128  
   392554 /usr/lib/x86_64-linux-gnu/libpcreposix.so.3.13.3
haproxy8245  root  mem   REG  254,12686672  
   396263 /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
haproxy8245  root  mem   REG  254,1 442920  
   396761 /usr/lib/x86_64-linux-gnu/libssl.so.1.1
haproxy8245  root  mem   REG  254,1 135440  
   915013 /lib/x86_64-linux-gnu/libpthread-2.24.so
haproxy8245  root  mem   REG  254,1  14640  
   914996 

Re: [ANNOUNCE] haproxy-1.8.0

[Pavlos Parissis]

On 26/11/2017 07:57 μμ, Willy Tarreau wrote:
> Hi all,
> 
> After one year of intense development and almost one month of debugging,
> polishing, and cross-review work trying to prevent our respective coworkers
> from winning the first bug award, I'm pleased to announce that haproxy 1.8.0
> is now officially released!
> 

Congratulations to everyone involved in releasing HAProxy 1.8 version.

Well done and keep up the hard and good work.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] MINOR: mworker: do not store child pid anymore in the pidfile

[Pavlos Parissis]

On 06/11/2017 03:19 μμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Mon, Nov 06, 2017 at 03:09:10PM +0100, Pavlos Parissis wrote:
>> That will be very much appreciated as it will allow us to have a smooth
>> migration to the new master process model.
> 
> In fact the current behaviour is to continue to dump the pids if you're
> not working in master-worker. If you need to perform some adaptations
> for master-worker, I think it would be easier to do them only once to
> support the single pid instead of having a temporary period where you
> have to deal with N+1 pids and be careful not to touch the workers by
> accident.
> 

Valid point. So, no objections from me about this commit:-)

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH] MINOR: mworker: do not store child pid anymore in the pidfile

[Pavlos Parissis]

On 06/11/2017 01:35 μμ, William Lallemand wrote:
> On Mon, Nov 06, 2017 at 12:11:13PM +0100, Pavlos Parissis wrote:
>> On 06/11/2017 11:16 πμ, William Lallemand wrote:
>>> The parent process supervises itself the children, we don't need to
>>> store the children pids anymore in the pidfile in master-worker mode.
>>
>> I have a small objection against this. Having PIDs in a file allows external 
>> tools to monitor the
>> workers. If those PIDs aren't written in a the pidfile then those tools have 
>> to use pgrep to find
>> them, unless the master process can provide them in some way(stats socket?).
>>
>> My 2cents,
>> Pavlos
>>
> 
> Hi Pavlos,
> 
> This patch was made to prevent scripts to send signals to the children 
> directly
> instead of sending them to the master which will forward them. That could end
> in reporting a wrong exit code in the master for example.
> 
> One of the problem of the pidfile, is that only the latest children were
> written, so you wouldn't have the remaining PID in this list on a reload.
> With pgrep -P you will have the full list of processes attached to the master.
> 
> Unfortunately there is no stats socket on the master (yet?), so it's not
> possible to do what you suggest.
> 
> However, we can maybe add an option to write all new PIDs in the pidfile if
> it's easier for supervision.
> 

That will be very much appreciated as it will allow us to have a smooth 
migration to the new master
process model.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: [PATCH] MINOR: mworker: do not store child pid anymore in the pidfile

[Pavlos Parissis]

On 06/11/2017 11:16 πμ, William Lallemand wrote:
> The parent process supervises itself the children, we don't need to
> store the children pids anymore in the pidfile in master-worker mode.

I have a small objection against this. Having PIDs in a file allows external 
tools to monitor the
workers. If those PIDs aren't written in a the pidfile then those tools have to 
use pgrep to find
them, unless the master process can provide them in some way(stats socket?).

My 2cents,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: 1.8-RC1 100% cpu usage

[Pavlos Parissis]

On 02/11/2017 02:24 μμ, Mihail Samoylov wrote:
> Hi.
> 
> I've tried 1.8-RC1 and in my case it ate 100% CPU and didn't work. I found 
> out that this is caused
> by option httpchk. When I commented this line everything became fine. Some 
> details:
> 

Willy  mentioned in the announcement that checks ares broken, so the behavior 
you observed is expected.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Happroxy and TCP SYN flood attacks

[Pavlos Parissis]

On 18/10/2017 01:06 μμ, Pooja Patel wrote:
> Respected Sir,
> 
> I am Pooja from University of Hyerabad. Currently I am working on networking 
> project for which I am
> using HAProxy as a load balancer. I have one doubt and that is:
> 
> *Does HAProxy by default protect itself from DOS or TCP SYN flood attack? If 
> not then how can
> protect it from these attacks?
> *
> 
> I have done simulation on my server using hping3 and hynae tool by flooding 
> HAProxy server with TCP
> SYN packets but I am not able to see any changes in my statistics.
> 
> Kindly go through my questions and Waiting for your reply.
> 

Before a TCP connection is handled by HAProxy, the Linux kernel processes it. 
So, you need to look
at defense mechanisms there. Newer kernels (see 4.9 and higher) provide very 
good ways to handle TCP
Syn flooding, one of them is about lockless listener, see commits
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4d54d86546f62c7c4a0fe3b36a64c5e3b98ce1a9

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=e6934f3ec00b04234acb24a1a2c28af59763d3b5

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c3fc7ac9a0b978ee8538058743d21feef25f7b33


With above patches CPU utilization stays the same when a server is under TCP 
Syn flood, while older
kernels suffer from CPU saturation.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Experimental / broken HTTP/2 support

[Pavlos Parissis]

On 16/10/2017 11:27 πμ, Willy Tarreau wrote:
> Hi Pavlos!
> 
> On Mon, Oct 16, 2017 at 10:46:44AM +0200, Pavlos Parissis wrote:
>>> Now after several failed attempts and with a lot of design sessions
>>> with my coworkers, I've made a good progress on a totally different
>>> approach which will later allow us to implement HTTP/2 on both sides,
>>> as well as implement support for QUIC. I have not merged anything yet
>>> because as I'm picking code from the first implementation, 
>>
>> HTTP/2 on client and servers side, and at the same time QUIC support!!
>> 1.8 version will be very hot release, thanks a lot for your hard work.
> 
> Wait wait wait... I never said that! I'm speaking about *design*. Look,
> "will later allow us". For 1.8, client-side only is expected. I don't know
> how long it will take to support server-side, as there are some painful
> shortcomings related to maintaining the server connections alive and being
> able to share them (and my beloved coworkers working on threads will hate
> me when they'll see that they'll have to deal with connections floating
> between multiple threads due to the full mesh you can have between client
> streams and servers streams).

I guess following step-by-step approach, 1st client side, it makes sense as it 
reduces
the size of breakage:-)

On a serious note, having support for QUIC is a huge advantage over other LB 
softwares.
Having said, we don't really know if QUIC will be used as it requires changes 
on several places,
Firewalls, bot/robot detection... It is interesting that Google decided to 
switch protocol, from TCP
to UDP, as they decided to not use SCTP for HTTP2 as it required changes on 
Firewall and etc.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Experimental / broken HTTP/2 support

[Pavlos Parissis]

On 15/10/2017 07:02 μμ, Willy Tarreau wrote:
> Hi Sander,
> 
> On Sun, Oct 15, 2017 at 04:27:15PM +0200, Sander Klein wrote:
>> Hi,
>>
>> I haven't been paying much attention to the list lately, but I am wondering
>> what the current status of http/2 support is in 1.8-(dev|snapshot).
>>
>> Is it in a usable-but-needs testing state? Or more like
>> stay-away-because-it-kills-kittens state?
> 
> The code I posted was not merged because it was experimental and I was
> not satisfied with what the architecture would look like in the long
> term. So I kept it handy "just in case" but didn't want to merge it.
> 
> Now after several failed attempts and with a lot of design sessions
> with my coworkers, I've made a good progress on a totally different
> approach which will later allow us to implement HTTP/2 on both sides,
> as well as implement support for QUIC. I have not merged anything yet
> because as I'm picking code from the first implementation, 

HTTP/2 on client and servers side, and at the same time QUIC support!!
1.8 version will be very hot release, thanks a lot for your hard work.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Reload takes about 3 minutes

[Pavlos Parissis]

On 13/10/2017 10:32 πμ, Joel W Kall wrote:
> We're using HAProxy 1.6.3 to load balance and route HTTP traffic to hundreds 
> of backend servers. We
> reload the config often (several times a day), both automatically when a 
> server fails and manually
> for administrative reasons.
> 
> The problem is that running the reload command takes 3 minutes on one of our 
> HAProxy servers (Ubuntu
> server 16.04). It does not seem to matter whether the server has traffic or 
> not. On our other
> servers with the same version of OS and HAProxy, the reload takes 1-5 
> seconds, regardless of load.
> We have a bunch of long-running requests but as I said it does not seem to 
> matter if the server has
> traffic or not.
> 
> We can see that a new process is spawned, but then it takes a few minutes 
> before it starts accepting
> traffic (or at least until the CPU usage of the new process climbs above 0% 
> and the it emits log
> messages about starting backends).
> 

Without seeing the conf, it is quite difficult to get a proper advice.
Loading SSL certificates can take some time, how many do you have?

> Questions:
> 
> - What can cause HAProxy to take so long to reload?
> 

Does the issue occur only on restart? If not then start haproxy with strace 
-ttt -T and see where it
spends most of the time

> - How can we find out what it's doing? For instance, what level of logging do 
> we need to enable and
> what would we look for in the log?
> 

Use strace as I mentioned.

> - Do you have a suggestion for a fix?
> 

We can only provide a fix even we know the root cause of the problem and right 
now we don't know
what is causing the problem.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Consider mentioning haproxystats in the site

[Pavlos Parissis]

On 03/10/2017 01:55 μμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Wed, Sep 27, 2017 at 11:40:07AM +0200, Pavlos Parissis wrote:
>> On 27/09/2017 10:36 ?u, Pavlos Parissis wrote:
>>> Hello all,
>>>
>>> haproxystats has been running for more than one year in production and has 
>>> been proven stable and
>>> very useful.
>>>
>>
>> Forgot to mention that link for the project:
>> https://github.com/unixsurfer/haproxystats
> 
> Sure! Where do you want it to appear exactly ?
> 
> Willy
> 

Under the quick links section in the main page, below the link for 'Herald: 
load feedback agent'.

Thanks,
Pavlos





signature.asc
Description: OpenPGP digital signature

Re: Consider mentioning haproxystats in the site

[Pavlos Parissis]

On 27/09/2017 10:36 πμ, Pavlos Parissis wrote:
> Hello all,
> 
> haproxystats has been running for more than one year in production and has 
> been proven stable and
> very useful.
> 

Forgot to mention that link for the project:
https://github.com/unixsurfer/haproxystats

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Consider mentioning haproxystats in the site

[Pavlos Parissis]

Hello all,

haproxystats has been running for more than one year in production and has been 
proven stable and
very useful.

Thus, I would like to request to mention it in www.haproxy.org site as a 
satellite tool, under the
list with HAtop and etc.

Thanks in advance,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Issue while using Proxy protocol in TCP mode

[Pavlos Parissis]

On 06/14/2017 11:17 AM, Vijay Bais wrote:
> Upgraded haproxy to 1.5.18.
> 

Why not 1.5.19? 1.5.18 was released a year ago.
Do you have a reason for not switching to 1.7.5?

Having said this, your problem could be a configuration issue or a new bug.

Cheers.
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Issue while using Proxy protocol in TCP mode

[Pavlos Parissis]

On 13/06/2017 02:23 μμ, Christopher Faulet wrote:
> Le 13/06/2017 à 10:07, Vijay Bais a écrit :
>> Hello,
>>
>> I am using HAProxy version 1.5-dev25-a339395.
>>
> 
> Hi,
> 
> This is a *very* old version and not a stable one. Please, first of all, 
> upgrade
> your HAProxy to a stable version. For HAProxy 1.5, the last stable version is 
> 1.5.19.
> 

Or 1.7.5 version, which is the latest and greatest version:-)

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Regarding migration on version haproxy-1.7.3

[Pavlos Parissis]

On 06/08/2017 07:31 AM, Devendra Joshi wrote:
> Hi,
> currently we are using haproxy-1.5.14 & now we want to migrate on 
> haproxy-1.7.3
> 
> 
> is there any config changes or need to do some setting.
> 

You should always test a software before you push it to production.
In most of the cases I have seen the upgrade is very smooth, but it depends on 
the configuration.
Some people never change the config they created with 1.4 version and get in 
troubles when they
upgrade to 1.6 or 1.7 version.

Just build a test machine, load the config, check for errors and then test it 
with some traffic.
In case of troubles in production after the upgrade, you can easily roll back 
by downgrading the
package.

1.7.6 version is coming soon, so you may want to wait few days for that one to 
be released before
you upgrade production.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Seeing server termination_state SD after updating from 1.6.11 to 1.7.5

[Pavlos Parissis]

On 06/01/2017 04:53 PM, Bernard McCormack wrote:
>  
> 
> After updating haproxy from 1.6.11 we saw a number of errors with the 
> termination state “SD “  that
> is flooding our logs with errors. It seems to only occur on a random subset 
> of requests.  Attached
> is a pcap of failed request. I was wondering the if anyone else was seeing 
> this?  I’ve tried turning
> off gzip on the backend server but it seems to make no difference.
> 
>  
> 
> The request all appear to be functioning correctly but it is generating a 
> large number of log
> messages. The flow is browser -> nginx -> haproxy -> nginx.
> 
>  
> 
> Jun 01 09:11:01 haproxy haproxy[10519]: haproxy  haproxy[10519]: 
> 127.0.0.1:24849
> [01/Jun/2017:09:11:01.190] MAINSITE_HTTP_MASTER phpmaster_80/webphpmaster01 
> 0/0/0/790/790 200 19
> 
> 22 - - SD-- 3/1/1/0/0 0/0 {CuQGKFkwEmU5eRCEAwZoAg==} "GET /favicon.ico 
> HTTP/1.0"
> 


In the network capture, I didn't see the HTTP response header Content-Length, 
which could confuse
haproxy and mark the connection as terminated from server side during data 
transfer as haproxy
didn't know the actual size of response. I am assuming here that, haproxy reads 
the value of
Content-Length and it will mark a request with SD if the server closes the 
connection, either by
sending a FIN/ACK or RST, before haproxy has received the amount of data 
announced in Content-Length.

My assumption could be wrong, so let's wait for some more experience people to 
answer your question.

Last but not least, when you supply a network capture it is very useful to 
mention the system behind
the IPs, in your case you have nginx-haproxy-nginx and I assumed that 1st HTTP 
response (frame 16)
was from nginx to haproxy. The size of reassembled TCP payload also matches the 
response size
mentioned in the log.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Download site horrendously slow

[Pavlos Parissis]

On 05/31/2017 06:28 PM, William Lallemand wrote:
> On Wed, May 31, 2017 at 04:15:18PM +, Skarbek, John wrote:
>> Hey guys,
>>
>> Just an FYI, I'm not sure how you host the downloads for haproxy, but for 
>> the past couple of days, they've been horridly slow.
>>
> 
> You are right, I'd just test it and the download of the tarball fluctuate
> between 1 and 2 kB/s.
> 
> 

I experience the same poor performance.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: New feature request

[Pavlos Parissis]

On 05/30/2017 11:56 AM, Willy Tarreau wrote:
> On Tue, May 30, 2017 at 11:04:35AM +0200, Pavlos Parissis wrote:
>> On 05/29/2017 02:58 PM, John Dison wrote:
>>> Hello,
>>>
>>> in ROADMAP I see:
>>> - spare servers : servers which are used in LB only when a minimum farm
>>> weight threshold is not satisfied anymore. Useful for inter-site LB with
>>> local pref by default.
>>>
>>>
>>> Is it possible to push this item priority to get it done for 1.8 please?  
>>> It looks like it should not require major code refactoring, just another LB 
>>> scheme.
>>>
>>> What I want to achieve is an ability to route request to "local" pool until 
>>> is get some
>>> pre-defined maximum load, and route extra request to "remote" pool of 
>>> servers.
>>>
>>> Thanks in advance.
>>>
>>
>>
>> +1 as I also find it very useful. But I am afraid it is too late for 1.8.
> 
> I'd love to have it as well for the same reasons. I think by now it
> shouldn't be too complicated to implement anymore, but all the usual
> suspects are busy on more important devs. I'm willing to take a look
> at it before 1.8 is released if we're in time with everything planned,
> but not more. However if someone wants to give it a try and doesn't
> need too much code review (which is very time consuming), I think this
> could get merged if the impact on existing code remains low (otherwise
> postponed to 1.9-dev).
> 
> In the mean time it's quite possible to achieve something more or less
> similar using two backends, one with the local servers, one with all
> servers, and to only use the second backend when the first one is full.
> It's not exactly the same, but can sometimes provide comparable results.
> 
> Willy
> 

True. I use the following to achieve it, it also avoids flipping users between 
data centers:

# Data center availability logic.
# Based on the destination IP we select the pool.
# NOTE: Destination IP is the public IP of a site and for each data center
# we use different IP address. So, in case we see IP address of dc1
# arriving in dc2 we know that dc is broken
http-request set-header X-Pool
%[str(www.foo.bar)]%[dst,map_ip(/etc/haproxy/dst_ip_dc.map,env(DATACENTER)]
use_backend %[hdr(X-Pool)] if { hdr(X-Pool),nbsrv ge 1 }

# Check for the availability of app in a data canter.
# NOTE: Two acl's with the same name produces a logical or.
acl www.foo.bardc1_down nbsrv(www.foo.bardc1) lt 1
acl www.foo.bardc1_down queue(www.foo.bardc1) ge 1
acl www.foo.bardc2_down nbsrv(www.foo.bardc2) lt 1
acl www.foo.bardc2_down queue(www.foo.bardc2) ge 1
acl www.foo.bardc3_down nbsrv(www.foo.bardc3) lt 1
acl www.foo.bardc3_down queue(www.foo.bardc3) ge 1

# We end up here if the selected pool of a data center is down.
# We don't want to use the all_dc pool as it would flip users between data
# centers, thus we are going to balance traffic across the two remaining
# data centers using a hash against the client IP. Unfortunately, we will
# check again for the availability of the data center, for which we know
# already is down. I should try to figure out a way to somehow dynamically
# know the remaining two data centers, so if dc1 is down then I should
# only check dc2 and dc3.

http-request set-var(req.selected_dc_backup) src,djb2,mod(2)

#Balance if www.foo.bardc1 is down
use_backend www.foo.bardc2 if www.foo.bardc1_down !www.foo.bardc2_down { 
var(req.selected_dc_backup)
eq 0 }
use_backend www.foo.bardc3 if www.foo.bardc1_down !www.foo.bardc3_down { 
var(req.selected_dc_backup)
eq 1 }

#Balance if www.foo.bardc2 is down
use_backend www.foo.bardc1 if www.foo.bardc2_down !www.foo.bardc1_down { 
var(req.selected_dc_backup)
eq 0 }
use_backend www.foo.bardc3 if www.foo.bardc2_down !www.foo.bardc3_down { 
var(req.selected_dc_backup)
eq 1 }

#Balance if www.foo.bardc3 is down
use_backend www.foo.bardc1 if www.foo.bardc3_down !www.foo.bardc1_down { 
var(req.selected_dc_backup)
eq 0 }
use_backend www.foo.bardc2 if www.foo.bardc3_down !www.foo.bardc2_down { 
var(req.selected_dc_backup)
eq 1 }

# If two data centers are down then for simplicity reasons just use the all_dc 
pool
default_backend www.foo.barall_dc

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: New feature request

[Pavlos Parissis]

On 05/29/2017 02:58 PM, John Dison wrote:
> Hello,
> 
> in ROADMAP I see:
> - spare servers : servers which are used in LB only when a minimum farm
> weight threshold is not satisfied anymore. Useful for inter-site LB with
> local pref by default.
> 
> 
> Is it possible to push this item priority to get it done for 1.8 please?  It 
> looks like it should not require major code refactoring, just another LB 
> scheme.
> 
> What I want to achieve is an ability to route request to "local" pool until 
> is get some
> pre-defined maximum load, and route extra request to "remote" pool of servers.
> 
> Thanks in advance.
> 


+1 as I also find it very useful. But I am afraid it is too late for 1.8.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

awesome article from Willy for seamless reload

[Pavlos Parissis]

Hi,

For those (I am on of them) who don't follow twitter/blogs/fb and all these 
noisy
stuff, check Willy's article about seamless reload here
https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks

I think section 4. Stopping and restarting HAProxy of management document should
be adjusted in order to reflect the new seamless reload functionality.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 06/05/2017 11:15 μμ, Pavlos Parissis wrote:
> On 04/05/2017 01:16 μμ, Olivier Houchard wrote:
>> On Thu, May 04, 2017 at 10:03:07AM +, Pierre Cheynier wrote:
>>> Hi Olivier,
>>>
>>> Many thanks for that ! As you know, we are very interested on this topic.
>>> We'll test your patches soon for sure.
>>>
>>> Pierre
>>
>> Hi Pierre :)
>>
>> Thanks ! I'm very interested in knowing how well it works for you.
>> Maybe we can talk about that around a beer sometime.
>>
>> Olivier
>>
> 
> Hi,
> 
> I finally managed to find time to perform some testing.
> 
> Fristly, let me explain environment.
> 
> Server and generator are on different servers (bare medal) with the same spec,
> network interrupts are pinned to all CPUs and irqbalancer daemon is disabled.
> Both nodes have 10GbE network interfaces.
> 
> I compared HAPEE with HAProxy using the following versions:
> 
> ### HAProxy
> The git SHA isn't mentioned in the output because I created the tarball
> with:
> 
> git archive --format=tar --prefix="haproxy-1.8.0/" HEAD | gzip -9 >
> haproxy-1.8.0.tar.gz
> 
> as I had to build the rpm using a tar ball, but I used the latest haproxy
> at f494977bc1a361c26f8cc0516366ef2662ac9502 commit.
> 
> /usr/sbin/haproxy -vv
> HA-Proxy version 1.8-dev1 2017/04/03
> Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -DMAX_HOSTNAME_LEN=42
>   OPTIONS = USE_LINUX_TPROXY=1 USE_CPU_AFFINITY=1 USE_REGPARM=1 USE_OPENSSL=1
> USE_PCRE=1 USE_PCRE_JIT=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
> Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with network namespace support.
> Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
> Compression algorithms supported : identity("identity")
> Encrypted password support via crypt(3): yes
> Built with PCRE version : 8.32 2012-11-30
> Running on PCRE version : 8.32 2012-11-30
> PCRE library supports JIT : yes
> 
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available filters :
> [SPOE] spoe
> [COMP] compression
> [TRACE] trace
> 
> ### HAPEE version
> /opt/hapee-1.7/sbin/hapee-lb -vv
> HA-Proxy version 1.7.0-1.0.0-163.180 2017/04/10
> Copyright 2000-2016 Willy Tarreau <wi...@haproxy.org>
> 
> Build options :
>   TARGET  = linux2628
>   CPU = generic
>   CC  = gcc
>   CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
> -DMAX_SESS_STKCTR=10 -DSTKTABLE_EXTRA_DATA_TYPES=10
>   OPTIONS = USE_MODULES=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_SLZ=1
> USE_CPU_AFFINITY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE= 
> USE_PCRE_JIT=1
> USE_NS=1
> 
> Default settings :
>   maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Encrypted password support via crypt(3): yes
> Built with libslz for stateless compression.
> Compression algorithms supported : identity("identity"), deflate("deflate"),
> raw-deflate("deflate"), gzip("gzip")
> Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
> Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
> OpenSSL library supports TLS extensions : yes
> OpenSSL library supports SNI : yes
> OpenSSL library supports prefer-server-ciphers : yes
> Built with PCRE version : 8.32 2012-11-30
> Running on PCRE version : 8.32 2012-11-30
> PCRE library supports JIT : yes
> Built with Lua version : Lua 5.3.3
> Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
> IP_FREEBIND
> Built with network namespace support
> 
> Available polling systems :
>   epoll : pref=300,  test result OK
>poll : pref=200,  test result OK
>  select : pref=150,  test result OK
> Total: 3 (3 usable), will use epoll.
> 
> Available filters :
> [COMP] compression
> [TRACE] trace
> [SPOE] spoe
> 
> 
> The configuration is the same and it is attached. As you can use I use nbproc 
> >1
> and each process is pinned to different CPU. We have 12

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 04/05/2017 01:16 μμ, Olivier Houchard wrote:
> On Thu, May 04, 2017 at 10:03:07AM +, Pierre Cheynier wrote:
>> Hi Olivier,
>>
>> Many thanks for that ! As you know, we are very interested on this topic.
>> We'll test your patches soon for sure.
>>
>> Pierre
> 
> Hi Pierre :)
> 
> Thanks ! I'm very interested in knowing how well it works for you.
> Maybe we can talk about that around a beer sometime.
> 
> Olivier
> 

Hi,

I finally managed to find time to perform some testing.

Fristly, let me explain environment.

Server and generator are on different servers (bare medal) with the same spec,
network interrupts are pinned to all CPUs and irqbalancer daemon is disabled.
Both nodes have 10GbE network interfaces.

I compared HAPEE with HAProxy using the following versions:

### HAProxy
The git SHA isn't mentioned in the output because I created the tarball
with:

git archive --format=tar --prefix="haproxy-1.8.0/" HEAD | gzip -9 >
haproxy-1.8.0.tar.gz

as I had to build the rpm using a tar ball, but I used the latest haproxy
at f494977bc1a361c26f8cc0516366ef2662ac9502 commit.

/usr/sbin/haproxy -vv
HA-Proxy version 1.8-dev1 2017/04/03
Copyright 2000-2017 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -DMAX_HOSTNAME_LEN=42
  OPTIONS = USE_LINUX_TPROXY=1 USE_CPU_AFFINITY=1 USE_REGPARM=1 USE_OPENSSL=1
USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with network namespace support.
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

### HAPEE version
/opt/hapee-1.7/sbin/hapee-lb -vv
HA-Proxy version 1.7.0-1.0.0-163.180 2017/04/10
Copyright 2000-2016 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-DMAX_SESS_STKCTR=10 -DSTKTABLE_EXTRA_DATA_TYPES=10
  OPTIONS = USE_MODULES=1 USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_SLZ=1
USE_CPU_AFFINITY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE= 
USE_PCRE_JIT=1
USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"),
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with network namespace support

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe


The configuration is the same and it is attached. As you can use I use nbproc >1
and each process is pinned to different CPU. We have 12 real CPUs as Intel hyper
threading is disabled, but we only use 10 CPUs for haproxy, the remaining two 
CPUs
are left for other daemons to use.

I experimented with wrk2 and httpress stress tools and decided to use wrk2 for
these tests. I didn't want to use the inject and other tools provided by haproxy
as I believe using different clients provides higher chances to spot problems.

In my tests I see that wrk2 reports higher read errors with HAProxy (3890) than
HAPEE (36). I don't know the meaning of the read error and it could be some
stupidity in the code of wrk2. I am saying this because two years ago we spent
four weeks stress testing HAPEE and found out that all open source http stress
tool sucks and some of the errors they report are client errors rather server.
But, in this case wrk2 was always reporting 

Re: Backport proposal, opinion needed

[Pavlos Parissis]

On 19/04/2017 12:13 μμ, Willy Tarreau wrote:
> Hi all,
> 
> Stephan (in Cc) reported me two nice segfaults in the config parser when
> feeding haproxy with some horribly fuzzed invalid configurations. To make
> it clear, it happens only when haproxy *fails* to start due to an error.
> But it's not a reason for failing the dirty way. Every time it was a
> problem in smp_resolve_args() which is used to resolve acl args.
> 
> The root cause of the issue is that there are certain types of errors
> where it's very tricky to unroll what has been started (eg: add multiple
> keywords to a list then you have to remove them and exactly them, taking
> care not to free a shared memory are if at least one remains because this
> one will be freed later), etc.
> 
> The first bug was a use-after-free causing all sort of random things like
> memory corruption or an infinite loop when trying to exit, which can be a
> problem for those aggregating configs from customers. The second one was
> a "more conventional" null dereference. I could fix both of them but I
> realized that the deeper reason is that we try to perform all the cross-
> reference checks after we've met such errors, which doesn't make sense
> and even causes some absurd errors to be reported. So I wrote the simple
> patch below for 1.8 and I think it would make sense to backport this into
> earlier versions to make everyone's life easier. It would also make the
> parser much more robust against such issues in the future.
> 
> Now the question is : this is not a bug fix but a small improvement which
> may have some impact on those being used to read error reports, so does
> anyone have any objection against this being backported (if so to regarding
> a specific version maybe) ?
> 


I also believe that it should be backported at least to 1.7 version[1].
It makes the output more clear and squeaks only the relevant bad config lines.

Cheers,
Pavlos


[1] IMHO: Users of 1.5 version should upgrade to 1.7, I don't see
any valid reason to stay on 1.5. From my personal experience I can tell
that 1.7 version is a rock solid release.



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 13/04/2017 06:18 μμ, Olivier Houchard wrote:
> On Thu, Apr 13, 2017 at 06:00:59PM +0200, Conrad Hoffmann wrote:
>> On 04/13/2017 05:10 PM, Olivier Houchard wrote:
>>> On Thu, Apr 13, 2017 at 04:59:26PM +0200, Conrad Hoffmann wrote:
 Sure, here it is ;P

 I now get a segfault (on reload):

 *** Error in `/usr/sbin/haproxy': corrupted double-linked list:
 0x05b511e0 ***

 Here is the backtrace, retrieved from the core file:

 (gdb) bt
 #0  0x7f4c92801067 in __GI_raise (sig=sig@entry=6) at
 ../nptl/sysdeps/unix/sysv/linux/raise.c:56
 #1  0x7f4c92802448 in __GI_abort () at abort.c:89
 #2  0x7f4c9283f1b4 in __libc_message (do_abort=do_abort@entry=1,
 fmt=fmt@entry=0x7f4c92934210 "*** Error in `%s': %s: 0x%s ***\n") at
 ../sysdeps/posix/libc_fatal.c:175
 #3  0x7f4c9284498e in malloc_printerr (action=1, str=0x7f4c929302ec
 "corrupted double-linked list", ptr=) at malloc.c:4996
 #4  0x7f4c92845923 in _int_free (av=0x7f4c92b71620 ,
 p=, have_lock=0) at malloc.c:3996
 #5  0x00485850 in tcp_find_compatible_fd (l=0xaaed20) at
 src/proto_tcp.c:812
 #6  tcp_bind_listener (listener=0xaaed20, errmsg=0x7ffccc774e10 "",
 errlen=100) at src/proto_tcp.c:878
 #7  0x00493ce1 in start_proxies (verbose=0) at src/proxy.c:793
 #8  0x004091ec in main (argc=21, argv=0x7ffccc775168) at
 src/haproxy.c:1942
>>>
>>> Ok, yet another stupid mistake, hopefully the attached patch fixes this :)
>>>
>>> Thanks !
>>>
>>> Olivier
>>
>>
>> It does indeed! Not only does it work now, the result is impressive! Not a
>> single dropped request even when aggressively reloading under substantial 
>> load!
>>
>> You certainly gave me an unexpected early easter present here :)
>>
>> I will now head out, but I am planning on installing a 1.8 version with
>> your patches on our canary pool (which receives a small amount of
>> production traffic to test changes) after the holidays. I will happily test
>> anything else that might be helpful for you. I will also set up a proper
>> load test inside our data center then, but I expect no surprises there (my
>> current tests were done over a VPN link, somewhat limiting the achievable
>> throughput).
>>
>> Once more, thank you so much! This will greatly simplify much of our
>> operations. If there is anything else we can help test, let me know :)
> 
> Pfew, at least :) Thanks a lot for your patience, and doing all that testing !
> 
> Olivier
> 


Joining this again a bit late, do you still want me to test it?
I would like to know if there are any performance impact, but I see that
Conrad Hoffmann has done all the hard work on this. So, we can conclude that we
don't expect any performance impact.

Once again thanks a lot for your hard work on this.
@Conrad Hoffmann, thanks a lot for testing this and checking if there is any
performance impact.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

HAProxy spins at 100 CPU....

[Pavlos Parissis]

Hi all,

Wait don't blame HAProxy yet, read this first,
https://blog.booking.com/troubleshooting-a-journey-into-the-unknown.html


I would like to thank Willy Tarreau for his support on this interesting system
and not HAProxy issue.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Multiple Config Files Use Leads to Lockout

[Pavlos Parissis]

On 08/04/2017 08:26 μμ, Coscend@HAProxy wrote:
> Hello Pavlos,
> 
> An update:  Following your guidance, we have successfully transitioned from 
> using 'system V init' to 'systemd' in order to start/stop/reload HAProxy. 
> Thank
> you for helping us advance with times.
> 
> We would appreciate your guidance on why using multiple HAProxy config files
> in a directory forces us to reset our password to our WebSocket product for us
> to login?

I don't believe the way configuration files are loaded has anything to do with 
the
behavior you observe. It sounds that the application demands user affinity as 
only
a single application server can serve the traffic from a single user.

Have you tried to use 'balance source' to stick requests from a single user to
a particular server?

This
https://blog.haproxy.com/2012/03/29/load-balancing-affinity-persistence-sticky-sessions-what-you-need-to-know/
could be a useful article to read.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 10/04/2017 11:52 μμ, Olivier Houchard wrote:
> On Mon, Apr 10, 2017 at 11:08:56PM +0200, Pavlos Parissis wrote:
>> On 10/04/2017 08:09 , Olivier Houchard wrote:
>>>
>>> Hi,
>>>
>>> On top of those patches, here a 3 more patches.
>>> The first one makes the systemd wrapper check for a HAPROXY_STATS_SOCKET
>>> environment variable. If set, it will use that as an argument to -x, when
>>> reloading the process.
>>
>> I see you want to introduce a specific environment variable for this 
>> functionality
>> and then fetched in the code with getenv(). This is a way to do it.
>>
>> IMHO: I prefer to pass a value to an argument, for instance -x. It is also
>> consistent with haproxy binary where someone uses -x argument as well.
>>
> 
> I'm not sure what you mean by this ?
> It is supposed to be directly the value given to -x as an argument.
> 

Ignore what I wrote above as I was under the wrong impression that we need to
pass -x to systemd wrapper during the reload, but the systemd wrapper is not
invoked during the reload.


>>> The second one sends listening unix sockets, as well as IPv4/v6 sockets. 
>>> I see no reason not to, and that means we no longer have to wait until
>>> the old process close the socket before being able to accept new connections
>>> on it.
>>
>>> The third one adds a new global optoin, nosockettransfer, if set, we assume
>>> we will never try to transfer listening sockets through the stats socket,
>>> and close any socket nout bound to our process, to save a few file
>>> descriptors.
>>>
>>
>> IMHO: a better name would be 'stats nounsedsockets', as it is referring to a
>> generic functionality of UNIX stats socket, rather to a very specific 
>> functionality.
>>
> 
> Well really it is a global setting, maybe my wording isn't great, but it
> makes haproxy close all sockets not bound to this process, as it used to,
> instead of keeping them around in case somebody asks for them. 
> 
>> I hope tomorrow I will find some time to test your patches.
>>
> 
> Thanks a lot ! This is greatly appreciated.
> 

Do you have a certain way to test it?
My plan is to have a stress environment where I fire-up X thousands of TCP
connections and produce a graph, which contain number of TCP RST received from
the client during a soft-reload of haproxy. I'll run this test against haproxy
1.8 with and without your patches. So, I will have a clear indication if your
patches solved the issue.


Any ideas?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 10/04/2017 11:48 μμ, Olivier Houchard wrote:
> On Mon, Apr 10, 2017 at 10:49:21PM +0200, Pavlos Parissis wrote:
>> On 07/04/2017 11:17 , Olivier Houchard wrote:
>>> On Fri, Apr 07, 2017 at 09:58:57PM +0200, Pavlos Parissis wrote:
>>>> On 06/04/2017 04:57 , Olivier Houchard wrote:
>>>>> On Thu, Apr 06, 2017 at 04:56:47PM +0200, Pavlos Parissis wrote:
>>>>>> On 06/04/2017 04:25 , Olivier Houchard wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> The attached patchset is the first cut at an attempt to work around the
>>>>>>> linux issues with SOREUSEPORT that makes haproxy refuse a few new 
>>>>>>> connections
>>>>>>> under heavy load.
>>>>>>> This works by transferring the existing sockets to the new process via 
>>>>>>> the
>>>>>>> stats socket. A new command-line flag has been added, -x, that takes the
>>>>>>> path to the unix socket as an argument, and if set, will attempt to 
>>>>>>> retrieve
>>>>>>> all the listening sockets;
>>>>>>> Right now, any error, either while connecting to the socket, or 
>>>>>>> retrieving
>>>>>>> the file descriptors, is fatal, but maybe it'd be better to just fall 
>>>>>>> back
>>>>>>> to the previous behavior instead of opening any missing socket ? I'm 
>>>>>>> still
>>>>>>> undecided about that.
>>>>>>>
>>>>>>> Any testing, comments, etc would be greatly appreciated.
>>>>>>>
>>>>>>
>>>>>> Does this patch set support HAProxy in multiprocess mode (nbproc > 1) ?
>>>>>>
>>>>>
>>>>> Hi Pavlos,
>>>>>
>>>>> If it does not, it's a bug :)
>>>>> In my few tests, it seemed to work.
>>>>>
>>>>> Olivier
>>>>>
>>>>
>>>>
>>>> I run systems with systemd and I can't see how I can test the seamless 
>>>> reload
>>>> functionality ( thanks for that) without a proper support for the UNIX 
>>>> socket
>>>> file argument (-x) in the haproxy-systemd-wrapper.
>>>>
>>>> I believe you need to modify the wrapper to accept -x argument for a single
>>>> UNIX socket file or -X for a directory path with multiple UNIX socket 
>>>> files,
>>>> when HAProxy runs in multi-process mode.
>>>>
>>>> What do you think?
>>>>
>>>> Cheers,
>>>> Pavlos
>>>>
>>>>
>>>>
>>>
>>>
>>> Hi Pavlos,
>>>
>>> I didn't consider systemd, so it may be I have to investigate there.
>>> You don't need to talk to all the processes to get the sockets, in the new
>>> world order, each process does have all the sockets, although it will accept
>>> connections only for those for which it is configured to do so (I plan to 
>>> add
>>> a configuration option to restore the old behavior, for those who don't 
>>> need 
>>> that, and want to save file descriptors).
>>> Reading the haproxy-systemd-wrapper code, it should be trivial.
>>> I just need to figure out how to properly provide the socket path to the
>>>  wrapper.
>>> I see that you already made use of a few environment variables in
>>> haproxy.service. Would that be reasonnable to add a new one, that could
>>> be set in haproxy.service.d/overwrite.conf ? I'm not super-used to systemd,
>>> and I'm not sure of how people are doing that kind of things.
>>>
>>
>> I believe you are referring to $CONFIG and PIDFILE environment variables. 
>> Those
>> two variables are passed to the two arguments, which were present but 
>> impossible
>> to adjust their input, switching to variables allowed people to overwrite 
>> their input.
>>
>> In this case, we are talking about a new functionality I guess the best 
>> approach
>> would be to have ExecStart using EXTRAOPTS:
>> ExecStart=/usr/sbin/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE $EXTRAOPTS
>>
>> This will allow users to set a value to the new argument and any other 
>> argument
>> they want
>> cat /etc/systemd/system/haproxy.service.d/overwrite.conf
>> [Service]
>> Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
>> "EXTRAOPTS=-x /foobar"
>>
>> or using default configuration file /etc/default/haproxy
>>
>> [Service]
>> Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
>> EnvironmentFile=-/etc/default/haproxy
>> ExecStart=/usr/sbin/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE $EXTRAOPTS
>>
>> cat /etc/default/haproxy
>> EXTRAOPTS="-x /foobar"
>>
>> I hope it helps,
>> Cheers,
>>
> 
> 
> 
> Hi Pavlos,
> 
> Yeah I see what you mean, it is certainly doable, though -x is a bit special,
> because you don't use it the first time you run haproxy, only for reloading,
> so that would mean the wrapper has special knowledge about it, and remove it
> from the user-supplied command line the first time it's called. I'm a bit
> uneasy about that, but if it's felt the best way to do it, I'll go ahead.
> 

I see, I forgot that the -x argument has only a meaning for the reload phase and
ExecReload uses haproxy and not the haproxy-systemd-wrapper.
So, it makes sense to pass the environment variable and let
haproxy-systemd-wrapper do its thing only on the reload.

Please ignore what I wrote above for EXTRAOPTS, sorry for the confusion.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 10/04/2017 08:09 μμ, Olivier Houchard wrote:
> 
> Hi,
> 
> On top of those patches, here a 3 more patches.
> The first one makes the systemd wrapper check for a HAPROXY_STATS_SOCKET
> environment variable. If set, it will use that as an argument to -x, when
> reloading the process.

I see you want to introduce a specific environment variable for this 
functionality
and then fetched in the code with getenv(). This is a way to do it.

IMHO: I prefer to pass a value to an argument, for instance -x. It is also
consistent with haproxy binary where someone uses -x argument as well.

> The second one sends listening unix sockets, as well as IPv4/v6 sockets. 
> I see no reason not to, and that means we no longer have to wait until
> the old process close the socket before being able to accept new connections
> on it.

> The third one adds a new global optoin, nosockettransfer, if set, we assume
> we will never try to transfer listening sockets through the stats socket,
> and close any socket nout bound to our process, to save a few file
> descriptors.
> 

IMHO: a better name would be 'stats nounsedsockets', as it is referring to a
generic functionality of UNIX stats socket, rather to a very specific 
functionality.

I hope tomorrow I will find some time to test your patches.

Thanks a lot for your work on this,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 07/04/2017 11:17 μμ, Olivier Houchard wrote:
> On Fri, Apr 07, 2017 at 09:58:57PM +0200, Pavlos Parissis wrote:
>> On 06/04/2017 04:57 , Olivier Houchard wrote:
>>> On Thu, Apr 06, 2017 at 04:56:47PM +0200, Pavlos Parissis wrote:
>>>> On 06/04/2017 04:25 , Olivier Houchard wrote:
>>>>> Hi,
>>>>>
>>>>> The attached patchset is the first cut at an attempt to work around the
>>>>> linux issues with SOREUSEPORT that makes haproxy refuse a few new 
>>>>> connections
>>>>> under heavy load.
>>>>> This works by transferring the existing sockets to the new process via the
>>>>> stats socket. A new command-line flag has been added, -x, that takes the
>>>>> path to the unix socket as an argument, and if set, will attempt to 
>>>>> retrieve
>>>>> all the listening sockets;
>>>>> Right now, any error, either while connecting to the socket, or retrieving
>>>>> the file descriptors, is fatal, but maybe it'd be better to just fall back
>>>>> to the previous behavior instead of opening any missing socket ? I'm still
>>>>> undecided about that.
>>>>>
>>>>> Any testing, comments, etc would be greatly appreciated.
>>>>>
>>>>
>>>> Does this patch set support HAProxy in multiprocess mode (nbproc > 1) ?
>>>>
>>>
>>> Hi Pavlos,
>>>
>>> If it does not, it's a bug :)
>>> In my few tests, it seemed to work.
>>>
>>> Olivier
>>>
>>
>>
>> I run systems with systemd and I can't see how I can test the seamless reload
>> functionality ( thanks for that) without a proper support for the UNIX socket
>> file argument (-x) in the haproxy-systemd-wrapper.
>>
>> I believe you need to modify the wrapper to accept -x argument for a single
>> UNIX socket file or -X for a directory path with multiple UNIX socket files,
>> when HAProxy runs in multi-process mode.
>>
>> What do you think?
>>
>> Cheers,
>> Pavlos
>>
>>
>>
> 
> 
> Hi Pavlos,
> 
> I didn't consider systemd, so it may be I have to investigate there.
> You don't need to talk to all the processes to get the sockets, in the new
> world order, each process does have all the sockets, although it will accept
> connections only for those for which it is configured to do so (I plan to add
> a configuration option to restore the old behavior, for those who don't need 
> that, and want to save file descriptors).
> Reading the haproxy-systemd-wrapper code, it should be trivial.
> I just need to figure out how to properly provide the socket path to the
>  wrapper.
> I see that you already made use of a few environment variables in
> haproxy.service. Would that be reasonnable to add a new one, that could
> be set in haproxy.service.d/overwrite.conf ? I'm not super-used to systemd,
> and I'm not sure of how people are doing that kind of things.
> 

I believe you are referring to $CONFIG and PIDFILE environment variables. Those
two variables are passed to the two arguments, which were present but impossible
to adjust their input, switching to variables allowed people to overwrite their 
input.

In this case, we are talking about a new functionality I guess the best approach
would be to have ExecStart using EXTRAOPTS:
ExecStart=/usr/sbin/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE $EXTRAOPTS

This will allow users to set a value to the new argument and any other argument
they want
cat /etc/systemd/system/haproxy.service.d/overwrite.conf
[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
"EXTRAOPTS=-x /foobar"

or using default configuration file /etc/default/haproxy

[Service]
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
EnvironmentFile=-/etc/default/haproxy
ExecStart=/usr/sbin/haproxy-systemd-wrapper -f $CONFIG -p $PIDFILE $EXTRAOPTS

cat /etc/default/haproxy
EXTRAOPTS="-x /foobar"

I hope it helps,
Cheers,



signature.asc
Description: OpenPGP digital signature

Re: Multiple Config Files Use Leads to Lockout

[Pavlos Parissis]

On 07/04/2017 06:00 μμ, Coscend@HAProxy wrote:
> Dear HAProxy Community,
> 
>  
> 
> We are seeking your guidance with regard to the following issue we are facing
> with our HAProxy use.
> 
>  
> 
> --
> 
> ISSUE
> 
> ---
> 
> When we use a single monolithic config file (haproxy.cfg), HAPRoxy launches 
> fine
> and we are able to access our products that use HTTP/TCP (WebSockets) and 
> RTMP. 
> 
>  
> 
> When we launch HAProxy using multiple config files in a directory, HAProxy
> launches fine.  However, our products do not allow administrator to login 
> (i.e.,
> locks out).  We have to reset the password to login.  Once the admin logs out,
> again it locks out the administrator and the password has to be reset to 
> login.
> 
>  
> 
> How is using multiple HAProxy config files in a directory connected to logging
> in to our WebSocket product?
> 
>  
> 
> 
> 
> DETAILS
> 
> 
> 
> We are running HAProxy as a daemon service.  We are using HAProxy init script
> from HAProxy’s Github source.  /etc/init.d/haproxy has the following lines:
> 

May I ask why you are using the SysVinit and not systemd to start haproxy? You
are using CentOS 7.2, which uses systemd, thus my question.

>  
> 
> Option 1:  When using SINGLE Config FILE
> 
> CFG=haproxy-1.7.5/conf/$BASENAME.cfg
> 
>  
> 
> Option 2:  When using a config DIRECTORY (conf.modular) with multiple files in
> alphanumerical order, namely global, defaults, frontend, backend, dns resolver
> and statistics:
> 
> CFG=haproxy-1.7.5/conf.modular
> 
>  

Without seeing the actual configuration, it would be quite difficult to get a
useful response. Can you share a sanitized version of your configuration?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [PATCH]: BUG/MINOR

[Pavlos Parissis]

On 07/04/2017 08:52 μμ, David CARLIER wrote:
> Hi all,
> 
> I was trying to compile the 1.8 branch under DragonflyBSD and went into a 
> build
> failure, thus
> this patch proposal.
> 
> Kindest regards.


I believe Steven Davidobitz has sent the same patch, see mail with subject
'[PATCH] BUILD: fix for non-transparent builds'.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 06/04/2017 04:57 μμ, Olivier Houchard wrote:
> On Thu, Apr 06, 2017 at 04:56:47PM +0200, Pavlos Parissis wrote:
>> On 06/04/2017 04:25 μμ, Olivier Houchard wrote:
>>> Hi,
>>>
>>> The attached patchset is the first cut at an attempt to work around the
>>> linux issues with SOREUSEPORT that makes haproxy refuse a few new 
>>> connections
>>> under heavy load.
>>> This works by transferring the existing sockets to the new process via the
>>> stats socket. A new command-line flag has been added, -x, that takes the
>>> path to the unix socket as an argument, and if set, will attempt to retrieve
>>> all the listening sockets;
>>> Right now, any error, either while connecting to the socket, or retrieving
>>> the file descriptors, is fatal, but maybe it'd be better to just fall back
>>> to the previous behavior instead of opening any missing socket ? I'm still
>>> undecided about that.
>>>
>>> Any testing, comments, etc would be greatly appreciated.
>>>
>>
>> Does this patch set support HAProxy in multiprocess mode (nbproc > 1) ?
>>
> 
> Hi Pavlos,
> 
> If it does not, it's a bug :)
> In my few tests, it seemed to work.
> 
> Olivier
> 


I run systems with systemd and I can't see how I can test the seamless reload
functionality ( thanks for that) without a proper support for the UNIX socket
file argument (-x) in the haproxy-systemd-wrapper.

I believe you need to modify the wrapper to accept -x argument for a single
UNIX socket file or -X for a directory path with multiple UNIX socket files,
when HAProxy runs in multi-process mode.

What do you think?

Cheers,
Pavlos





signature.asc
Description: OpenPGP digital signature

Re: [RFC][PATCHES] seamless reload

[Pavlos Parissis]

On 06/04/2017 04:25 μμ, Olivier Houchard wrote:
> Hi,
> 
> The attached patchset is the first cut at an attempt to work around the
> linux issues with SOREUSEPORT that makes haproxy refuse a few new connections
> under heavy load.
> This works by transferring the existing sockets to the new process via the
> stats socket. A new command-line flag has been added, -x, that takes the
> path to the unix socket as an argument, and if set, will attempt to retrieve
> all the listening sockets;
> Right now, any error, either while connecting to the socket, or retrieving
> the file descriptors, is fatal, but maybe it'd be better to just fall back
> to the previous behavior instead of opening any missing socket ? I'm still
> undecided about that.
> 
> Any testing, comments, etc would be greatly appreciated.
> 

Does this patch set support HAProxy in multiprocess mode (nbproc > 1) ?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: [ANNOUNCE] haproxy-1.6.12

[Pavlos Parissis]

On 04/04/2017 11:29 πμ, Willy Tarreau wrote:
> Hi,
> 
> HAProxy 1.6.12 was released on 2017/04/04. It added 46 new commits
> after version 1.6.11.
> 

You need to update the last version for 1.6 release on the site, it points to 
1.6.11.

Cheers,
Pavlos





signature.asc
Description: OpenPGP digital signature

Re: Problems with haproxy 1.7.3 on FreeBSD 11.0-p8

[Pavlos Parissis]

On 17/03/2017 05:57 μμ, Aleksandar Lazic wrote:
> Willy.
> 
> Am 14-03-2017 22:17, schrieb Willy Tarreau:
>> Matthias,
>>
>> I could finally track the problem down to a 5-year old bug in the
>> connection handler. It already used to affect Unix sockets but it
>> requires so rare a set of options and even then its occurrence rate
>> is so low that probably nobody noticed it yet.
>>
>> I'm attaching the patch to be applied on top of 1.7.3 which fixes it,
>> it will be merged into next version.
>>
>> Dmitry, you may prefer to take this one than to revert the previous
>> one from your ports, especially considering that a few connect()
>> immediately succeed over the loopback on FreeBSD and that it was
>> absolutely needed to trigger the bug (as well as the previously fixed
>> one, which had less impact).
>>
>> Or you may prefer to wait for 1.7.4. It's not planned yet given that
>> there are other fixes in the wild waiting for some feedback though.
>>
>> Thanks guys for the detailed feedback, it's now time to turn the page
>> and switch to less difficult ones!
> 
> I love your commit massages ;-).
> 

+1

> They are very detailed and sometimes bigger the the code change.
> 

Indeed. All commits, in any project, should be like this.

I personally follow the following rules:

1. short commit should be written, so that can be placed in the following 
sentence:
If you apply this commit it will 

2. Long commit message should clearly describe the problem, why it is a problem
and how the commit fixes the problem.

Having said, I have colleagues that not only follow the above rules but complain
to me about my very lengthy commit messages. I usually ignore them and redirect
their *rant* to /dev/null.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: Haproxy ssl with nginx backend problem.....

[Pavlos Parissis]

On 16/03/2017 09:21 πμ, 何智声 wrote:
> at first thank you read this email,and you help!
> i have problem for haproxy settings, 
> this is my haproxy.cfg settings
> 
> frontend
> 内嵌图片 3
> backend
> 内嵌图片 2
> 内嵌图片 1
> what happen on it and how to fix this error..thank you!!!


I am sorry but I can't clearly read the images you attached.
Could you please send the traces and config in cleat txt format?

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Force connection close after a haproxy reload

[Pavlos Parissis]

On 15/03/2017 11:48 πμ, Cyril Bonté wrote:
> Hi all,
> 
>> De: "Willy Tarreau"  À: "Robson Roberto Souza Peixoto"
>>  Cc: haproxy@formilux.org Envoyé: Mardi 14 Mars
>> 2017 13:20:46 Objet: Re: Force connection close after a haproxy reload
>> 
>> On Tue, Mar 14, 2017 at 11:16:26AM +, Robson Roberto Souza Peixoto
>> wrote:
>>> But will `-st` mode wait for current http requests finish? Or will 
>>> interrupt all connections without waiting for the responses?
>> 
>> It will interrupt them all but you said you were running TCP and not HTTP 
>> so with TCP there's no notion of end of request nor response.
> 
> As a reminder (to me), I sent a patch in december (just before the 1.7.0
> release), which immediately closes the HTTP keep-alived connections.
> Currently, during the soft stop, HTTP connections are only closed when a
> request is processed, it doesn't do anything on connections already in an
> idle state. I didn't spend more time on it but having a quick look at it, it
> may be ready to merge soon.
> 

What I have observed in production is that in HTTP mode the timeout settings
(client, http-requests) contribute to how long older process stay around. Longer
timeouts makes older processes to stay around longer time.
Having said this, it all depends to the traffic composition. For instance, when
you have radio streaming then older processes stay for days regardless to
timeout settings.


> About TCP connections, while I wrote the patch, I was thinking about a global
> "grace timeout", which will enforce haproxy exit if the soft stop takes too
> long (for example when tcp connections don't expire). Something like :
> 
> global grace 30s

This is a very interesting approach, as it solves the issue of old processes
staying around for days for TCP mode. +1 from me.


Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Some compilation SSL errors/warnings on debian testing

[Pavlos Parissis]

On 14/03/2017 10:20 μμ, Willy Tarreau wrote:
> On Tue, Mar 14, 2017 at 08:18:27PM +0100, Pavlos Parissis wrote:
>>>> On Debian testing with openssl 1.1.0e, I get the following warnings when I
>>>> compile 1.7 and 1.8:
>>>> https://gist.githubusercontent.com/unixsurfer/9c42361822f23cfe36f3b2169133b551/raw/4665476fdfb2a94d287814a2c8a36215cbebb465/gistfile1.txt
>>>
>>> Yes these ones are known and for now we don't have any workaround. It
>>> seems openssl 1.1 wants us to drop support for older TLS versions, but
>>> we definitely can't do that so we'll have to live with the warnings :-/
>>> I couldn't find a way to make them disappear.
>>>
>>
>> No worries, it compiles at the end and haproxy starts:-)
> 
> Ah that's how I test it before releasing... Just kidding, I don't verify
> that it starts :-)
> 
> (...)
>> I fully understand the situation, I will compile 1.6 against openssl 1.0.2 
>> version
>> on my Debian testing box. I am not using 1.6 version at all, too old :-), 
>> but I am
>> reshuffling code in haproxyadmin python lib and I want to make sure it
>> works with older versions of haproxy.
> 
> OK cool! Just out of curiosity, are there some features of 1.7 that you've
> already got used to and that prevent you from using 1.6, or is this just a
> matter of staying on something modern ?
> 

The latter, I prefer to use the latest stable version. I usually wait 1 month
before I switch to the new stable release[1]. For instance, I switched from 1.5 
to
1.6 when 1.6.3 was released. Switching to 1.7 takes more time because I have 
other
projects with higher priority.

[1] With the only exception of 1.5, I switched to 1.5.0 only a day after it was
released. Zero issues on production! But, I keep the config clean and very 
simple,
I hate unnecessary complexity.



signature.asc
Description: OpenPGP digital signature

Re: Some compilation SSL errors/warnings on debian testing

[Pavlos Parissis]

On 14/03/2017 05:24 μμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Tue, Mar 14, 2017 at 04:43:26PM +0100, Pavlos Parissis wrote:
>> Hi,
>>
>> On Debian testing with openssl 1.1.0e, I get the following warnings when I
>> compile 1.7 and 1.8:
>> https://gist.githubusercontent.com/unixsurfer/9c42361822f23cfe36f3b2169133b551/raw/4665476fdfb2a94d287814a2c8a36215cbebb465/gistfile1.txt
> 
> Yes these ones are known and for now we don't have any workaround. It
> seems openssl 1.1 wants us to drop support for older TLS versions, but
> we definitely can't do that so we'll have to live with the warnings :-/
> I couldn't find a way to make them disappear.
> 

No worries, it compiles at the end and haproxy starts:-)

>> When I compile 1.6 I get errors and compilation fails:
>> https://gist.githubusercontent.com/unixsurfer/4476410bbbaf2192af591123f4388850/raw/a733808a3028f0c9d7f53f4e699da6de3ae18969/gistfile1.txt
> 
> This is indeed expected, openssl 1.1's API is very different from 1.0.
> 
>> I compile it with:
>> make clean;make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
>> USE_PCRE_JIT=1 USE_TPROXY=1
>>
>> Am I the only seeing these warnings/errors? Searched on ML and someone 
>> mentioned
>> that haproxy 1.6 wont support 1.1.0 version of openssl, is this accurate? 
>> Having
>> openssl 1.0.2 and 1.1.0 on my personal development machine is fine, so zero
>> problems here if 1.6 does not support openssl 1.1.0 version.
> 
> Yes that's accurate. The risk of breakage is far too high for this to be
> backported to 1.6. With 1.7 not much different from 1.6, we'll have all
> people willing to explore openssl 1.1 users upgrade to haproxy 1.7 with
> very limited risks (and BTW some of the bugs currently affecting 1.7 are
> also on 1.6 and are in fact uncovered by some fixes for bugs that were
> hiding other ones).
> 

I fully understand the situation, I will compile 1.6 against openssl 1.0.2 
version
on my Debian testing box. I am not using 1.6 version at all, too old :-), but I 
am
reshuffling code in haproxyadmin python lib and I want to make sure it
works with older versions of haproxy.

Thanks for the reply,
Pavlos



signature.asc
Description: OpenPGP digital signature

Some compilation SSL errors/warnings on debian testing

[Pavlos Parissis]

Hi,

On Debian testing with openssl 1.1.0e, I get the following warnings when I
compile 1.7 and 1.8:
https://gist.githubusercontent.com/unixsurfer/9c42361822f23cfe36f3b2169133b551/raw/4665476fdfb2a94d287814a2c8a36215cbebb465/gistfile1.txt

When I compile 1.6 I get errors and compilation fails:
https://gist.githubusercontent.com/unixsurfer/4476410bbbaf2192af591123f4388850/raw/a733808a3028f0c9d7f53f4e699da6de3ae18969/gistfile1.txt

I compile it with:
make clean;make TARGET=linux2628 USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1
USE_PCRE_JIT=1 USE_TPROXY=1

Am I the only seeing these warnings/errors? Searched on ML and someone mentioned
that haproxy 1.6 wont support 1.1.0 version of openssl, is this accurate? Having
openssl 1.0.2 and 1.1.0 on my personal development machine is fine, so zero
problems here if 1.6 does not support openssl 1.1.0 version.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Feature request: routing a TCP stream based on Cipher Suites in a TLS ClientHello

[Pavlos Parissis]

On 23/02/2017 07:38 μμ, Lukas Tribus wrote:
> Hi,
> 
> Am 23.02.2017 um 04:02 schrieb James Brown:
>> Unfortunately, that feature only works with OpenSSL 1.0.2 (which,
>> incidentally, would be a good thing to note in the documentation)...
> 
> Good point; I did not remember this either ... we have to fix the docs.
> 
> 
> Lukas
> 

That means RedHat7, which comes with openssl 1.0.1, users can't use this
functionality! Is this because openssl 1.0.1 version doesn't support ECC
certificates?

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: multiproc ssl recommendations

[Pavlos Parissis]

On 9 December 2016 at 20:07, Apollon Oikonomopoulos <apoi...@debian.org> wrote:
> Hi Pavlos,
>
> On 17:31 Fri 09 Dec , Pavlos Parissis wrote:
>> On 09/12/2016 08:54 πμ, Apollon Oikonomopoulos wrote:
>> > Hi Willy, Elias,
>> >
>> > On 08:33 Fri 09 Dec , Willy Tarreau wrote:
>> >> On Thu, Dec 01, 2016 at 02:53:25PM +0100, Elias Abacioglu wrote:
>> >>> # Should I use core 0 on each CPU for backends (proc 1+15) or should
>> >>> I
>> >>> use core 1(proc 2+16)?
>> >>
>> >> Backends are processed on the same CPU as the frontend which passes them
>> >> the traffic, so the bind-process has no effect there. In fact bind-process
>> >> on a backend means "at least on these processes".
>> >>
>> >> That's why it's better to proceed like this (stupid numbers, just so that
>> >> you get the idea):
>> >>
>> >>listen ssl-offload
>> >>   bind-proess 2-50
>> >>   bind :443 ssl  process 2
>> >>   ...
>> >>   bind :443 ssl  process 50
>> >
>> > I wonder if a `per-process' keyword would make sense here. I find
>> >
>> >   bind :443 ssl  per-process
>> >
>> > more concise than 15 or 20 individual bind lines. This would have the
>> > same effect as N bind lines, one for each process in the bind-process
>> > list.
>>
>> If you have bind per process then all sockets are bound separately and you
>> get X listening sockets on port 443, which results to have one distinct 
>> socket
>> in each process with its own queue(SYN backlog queues and etc), and the 
>> kernel's
>> load balancing works much better.
>
> That's true, yes. However what I'm saying is that some syntactic sugar
> to have the parser auto-expand a single "bind" directive to create N
> sockets instead of one, would be nice.
>

Indeed, that would be nice. I guess it isn't big issue as most of the
people use a configuration
management tool, which does the expansion.

Cheers,
Pavlos

Re: multiproc ssl recommendations

[Pavlos Parissis]

On 09/12/2016 08:54 πμ, Apollon Oikonomopoulos wrote:
> Hi Willy, Elias,
> 
> On 08:33 Fri 09 Dec , Willy Tarreau wrote:
>> On Thu, Dec 01, 2016 at 02:53:25PM +0100, Elias Abacioglu wrote:
>>> # Should I use core 0 on each CPU for backends (proc 1+15) or should 
>>> I
>>> use core 1(proc 2+16)?
>>
>> Backends are processed on the same CPU as the frontend which passes them
>> the traffic, so the bind-process has no effect there. In fact bind-process
>> on a backend means "at least on these processes".
>>
>> That's why it's better to proceed like this (stupid numbers, just so that
>> you get the idea):
>>
>>listen ssl-offload
>>   bind-proess 2-50
>>   bind :443 ssl  process 2
>>   ...
>>   bind :443 ssl  process 50
> 
> I wonder if a `per-process' keyword would make sense here. I find
> 
>   bind :443 ssl  per-process
> 
> more concise than 15 or 20 individual bind lines. This would have the 
> same effect as N bind lines, one for each process in the bind-process 
> list.

If you have bind per process then all sockets are bound separately and you
get X listening sockets on port 443, which results to have one distinct socket
in each process with its own queue(SYN backlog queues and etc), and the kernel's
load balancing works much better.

So, haproxy performs better!

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Problem starting HaProxy after reboot

[Pavlos Parissis]

On 22/11/2016 10:09 πμ, Jürgen Haas wrote:
>> What do you mean by 'not operating at all'?
> 
> It doesn't respond to requests.
> 
>> Are you sure the IPs configured in frontends are available during boot?
> 
> Well, that's exactly what I don't know. Why shouldn't they?

I don't know, but I think it is better to find out. But, it could be
unrelated to the problem.

> It's a
> shared hosting environment like with the other HaProxy hosts as well,
> just that this one is running on Ubuntu 16 where the other run on Ubuntu 14.
> 


>> Can you share your config/version/systemd settings for haproxy?
> 
> This has happened last night with HaProxy 1.6.9 but I had seen that with
> a couple of earlier versions on that host too.
> 
> The config of HaProxy is nothing special and again equal to the other hosts.
> 
> For legacy reasons we are starting the process with initd like we did
> with the other OS versiopns for years before. Maybe we should switch to
> systemd here? COuld that cause something like this?
> 

If you use SysVinit then you don't need the systemd-wrapper.
If I understood you correctly you have the following setup
systemd->SysVinit init script->systemd-wrapper.

Am I right?

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: Problem starting HaProxy after reboot

[Pavlos Parissis]

On 22/11/2016 09:43 πμ, Jürgen Haas wrote:
> Hi all,
> 
> we do run HaProxy on 4 different hosts, all of which got configured by
> Ansible the same way and for 3 of them I don't have any issues. On one
> of them - which runs on Ubuntu 16.04 and the others on older version -
> starting HaProxy after reboot is failing.
> 
> Browsing the journal I found the logs below where I can see that HaProxy
> seems to be starting OK but then outputs some alerts. After that, there
> is no other message about HaProxy, but the result is that the process is
> running but HaProxy is not operating at all.

What do you mean by 'not operating at all'?
Are you sure the IPs configured in frontends are available during boot?
Can you share your config/version/systemd settings for haproxy?

 We do have to restart
> HaProxy and then everything is OK.
> 
> What could be causing this?
> 
> Thanks
> Jürgen
> 
> 
> Nov 21 23:35:40 ggate1 systemd[1]: Started /etc/rc.local Compatibility.
> Nov 21 23:35:40 ggate1 firewall[2618]: /etc/init.d/firewall: starting
> firewall
> Nov 21 23:35:40 ggate1 systemd[1]: netdata.service: PID file
> /var/run/netdata.pid not readable (yet?) after start: No such file or
> directory
> Nov 21 23:35:40 ggate1 systemd[1]: Started HAProxy Load Balancer.
> Nov 21 23:35:40 ggate1 systemd[1]: Started Linux real time system
> monitoring, done right.
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]:
> haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f
> /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
> Nov 21 23:35:40 ggate1 firewall[2618]: iptables: No chain/target/match
> by that name.
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> Nov 21 23:35:40 ggate1 haproxy-systemd-wrapper[2639]: [ALERT] 325/233540
> (2641) : sendmsg logger #1 failed: Operation not permitted (errno=1)
> lines 417-468
> 

I haven't seen the above. Under which user do you run haproxy?

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: HAProxy 1.5 vs 1.6

[Pavlos Parissis]

On 09/11/2016 09:20 μμ, Steven Le Roux wrote:
> Hi a first good coverage for a comparison between 1.5 and 1.6 would be
> http://blog.haproxy.com/2015/10/14/whats-new-in-haproxy-1-6/
> 
> 1.6 is perfectly considered stable and hasn't seen any maintenance
> release for more than 2 months. It's being widely used so I would be
> confident with it. It brings many improvements and features (libslz,
> lua, server states checkpointing,...) over 1.5
> 

Same story here. 1.6 is a rock solid release and works fine.

Cheers,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: http-reuse always, work quite well

[Pavlos Parissis]

On 22/10/2016 08:08 πμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Fri, Oct 21, 2016 at 03:01:52PM +0200, Pavlos Parissis wrote:
>>> I'm not surprized that always works better, but my point is that if it's
>>> much better it can be useful to stay with it, but if it's only 1% better
>>> it's not worth it.
>>>
>>
>> It is way better:-), see Marcin's response.
> 
> Ah sorry, I missed it. Indeed it looks much better, but we don't have
> the reference (no reuse) on this graph.

I will try to rerun the test tomorrow, which runs on production servers with
real user traffic:-)

> If the no reuse shows 10 times
> higher average times, then it means "safe" reuse brings a 10 times
> improvement and "always" brings 20 times so it's a matter of choice.
> However if safe does approximately the same as no reuse, for sure
> "always" is almost needed.
> 
>>>>> while "always" is optimal, strictly speaking it's
>>>>> not very clean if the clients are not always willing to retry a failed
>>>>> first request, and browsers typically fall into that category. A real
>>>>> world case can be a request dequeued to a connection that just closes.
>>>>
>>>> What is the response of HAProxy to clients in this case? HTTP 50N?
>>>
>>> No, the client-side connection will simply be aborted so that the client
>>> can decide whether to retry or not.
>>
>> Connection will be aborted by haproxy sending TCP RST?
> 
> As much as possible yes. The principle is to let the client retry the
> request (since it is the only one knowing whether it's safe or not).
> 
>>> I'd suggest a rule of thumb (maybe this should be added to the doc) : watch
>>> your logs over a long period. If you don't see queue timeouts, nor request
>>> timeouts, it's probably safe enough to use "always".
>>
>> Which field on the log do we need to watch? Tq?
> 
> Tw (time spent waiting in the queue), Tc (time spent getting a connection),
> and of course the termination flags, everything with a C or Q on the second
> char needs to be analysed.
> 

I looked at the logs for a period of 11hours and found zero occurrences of C or
Q. I also didn't noticed any change on Tw and Tc timers. I will keep an eye.

>>> Each time you notice
>>> one of them, there is a small risk of impacting another client. It's not
>>> rocket science but the risks depend on the same parameters.
>>
>>
>> Thanks a lot for yet another reach in information replies.
> 
> You're welcome. Please note that the reuse mechanism is not perfect and
> can still be improved. So do not hesitate to report any issue you find,
> we definitely need real-world feedback like this. I cannot promise that
> every issue will be fixed, but at least we need to consider them and see
> what can be done.
> 

Acked, I will report any issues we may find.

Thanks,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: HAProxy reloads lets old and outdated processes

[Pavlos Parissis]

On 25/10/2016 01:21 πμ, Willy Tarreau wrote:
> Hi guys,
> 
> On Tue, Oct 25, 2016 at 12:42:26AM +0200, Lukas Tribus wrote:
>> Not fixing *real world issues* because we don't agree with the use-case or
>> there is a design misconception somewhere else is dangerous. We don't have
>> to support every single obscure use-case out there, that's not what I am
>> saying, but systemd is a reality; as is docker and periodic reloads.
> (...)
> 
> Thank you both for your insights. There are indeed valid points in both
> of your arguments. I too am afraid of breaking things for people who do
> really abuse, but at the same time we cannot blame the users who get
> caught by systemd lying to them. I really don't care about people who
> would reload every 2 ms to be honnest, but I'm concerned about people
> shooting themselves in the foot because they use very large configs or
> because as you say Lukas, they accidently run the command twice. This
> is something we used to deal with in the past, it's hard to lose this
> robustness. I've seen configs taking minutes to start (300k backends),
> reduced to a few seconds after the backends were moved to a binary
> tree. But these seconds remain a period of uncertainty and that's not
> nice for the user.
> 
> I think the patch I sent this evening covers both of your concerns. It's
> quite simple, relies on a process *closing* a file descriptor, which also
> covers a dying/crashing process (because I never trust any design consisting
> in saying "I promise I will tell you when I die"). And it doesn't
> significantly change things. I'm really interested in feedback on it.
> Pavlos, please honnestly tell me if it really scares you and I'd like
> to address this (even if that means doing it differently). Let's consider
> that I want something backportable into HAPEE, you know that users there
> can be very demanding regarding reliability :-)
> 

Well, I have full confidence on the quality of your code (assuming you will
polish the patch to handle errors as you mentioned :-) ) and I am willing to
test it on our environment when arrives on HAPEE. But, we will never hit the
conditions which triggers this behavior as our configuration tool for haproxy
doesn't allow to reload very often, we allow 1 reload per min (this is
configurable of course). We did that in order to address also the case of too
many live processes for a cluster of haproxies which has a lot of long-lived TCP
connections [1].


> I'm really open to suggestions. I absolutely despise systemd and each time
> I have to work on the wrapper I feel like I'm going to throw up. So for me
> working on this crap is a huge pain each time. But I'm really fed up with
> seeing people having problems in this crazy environment because one clueless
> guy decided that he knew better than all others how a daemon should reload,
> so whatever we can do to make our users' lifes easier in captivity should
> at least be considered.
> 

Have you considered to report this to systemd? May be they have a solution, I
don't know.

To sum up, go ahead with the patch as it addresses real problems of users and
you can count on me testing HAPEE in our environment.

Cheers,
Pavlos

[1] I have mentioned before that we balance rsyslog traffic from 20K clients
and every time we reload haproxy we see old processes staying alive for days.
This is because frontend/backend runs in TCP mode and rsyslog daemon on clients
doesn't terminate the connection till it is restarted, it opens 1
long-lived TCP connection against frontend. haproxy can't close the connection
on shutdown as it does with HTTP mode as it doesn't understand the protocol and
tries to play nice and graceful with the clients, I will wait for you to close
the connection.



signature.asc
Description: OpenPGP digital signature

Re: HAProxy reloads lets old and outdated processes

[Pavlos Parissis]

Good morning,

Got my coffee ready before I read and reply:-)

On 25/10/2016 12:42 πμ, Lukas Tribus wrote:
> Hello,
> 
> 
> Am 24.10.2016 um 22:32 schrieb Pavlos Parissis:
>> 
>> IMHO: Ask the users to not perform reloads every 2miliseconds. It is
>> insane. You may spend X hours on this which will make the code a bot more
>> complex and cause possible breakages somewhere else.
> 
> Not fixing *real world issues* because we don't agree with the use-case or
> there is a design misconception somewhere else is dangerous. We don't have to
> support every single obscure use-case out there, that's not what I am saying,
> but systemd is a reality; as is docker and periodic reloads.
> 
> You are talking about 2 milliseconds, but that is just the testcase here,
> think about how long haproxy would need to start when it has to load
> thousands of certificates. Probably more than a few seconds (I don't have any
> clue), and it would be pretty easy to create a mess of processes, not because
> of docker/cloud orchestration/whatever, but in SSH by hitting reload two
> times in a row manually.
> 
> I don't want to be scared of hitting reload two times even if I'm on a
> systemd based box with heavy SSL traffic. In fact, I *do* wanna be able to
> reload haproxy every 2 ms, not because I need it, but because the alternative
> would mean I need to remember to be "always careful" about that "strange
> issue with systemd which is not our fault" and make sure my colleague is not
> doing the same thing I'm doing and we reload simultaneously. I don't want to
> run my infrastructure like a house of cards.
> 
> This is not limited to fancy new cloud orchestration technologies and it is
> not a minor issue either.
> 
> 

All valid points. The bottom line is to trust the reload process that wont cause
unexpected behavior regardless the frequency of reloads and wallclock of a
single reload.


> 
>> I am pretty sure 90% of the cases which require so often reload are the
>> ones which try to integrate HAProxy with docker stuff, where servers in the
>> pools are treated as ephemeral nodes, appear and disappear very often and
>> at high volume.
> 
> Not sure if I understand you here correctly, but this sounds like you are 
> implying that we shouldn't spend time fixing issues related to docker (and 
> similar technologies). I have to disagree.
> 

Οn the contrary, I have requested for ETA of DNS SRV functionality, which allows
to extend and shrink the backend without reload, and I have also requested for
the ability to add/remove servers via the socket. All these because I need to
support docker on my environments:-)

The high frequency of reloads on docker environment is the result of missing the
above 2 functionalities.

> 
> We may not like systemd and we may not like docker. But that doesn't mean
> its not worth looking into those issues.
> 

Οn the contrary, I *do* love systemd. I am not joking here.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: HAProxy reloads lets old and outdated processes

[Pavlos Parissis]

On 24/10/2016 09:13 μμ, Willy Tarreau wrote:
> Hi again,
> 
> On Mon, Oct 24, 2016 at 07:41:06PM +0200, Willy Tarreau wrote:
>> I don't know if this is something you're interested in experimenting
>> with. This is achieved using fcntl(F_SETLKW). It should be done in the
>> wrapper as well.
> 
> Finally I did it and it doesn't help at all. The signal-based asynchronous
> reload is fundamentally flawed. It's amazing to see how systemd managed to
> break something simple and robust in the sake of reliability, by introducing
> asynchronous signal delivery...
> 
> The problem is not even with overlapping writes (well, it very likely
> happens) but it is related to the fact that you never know whom you're
> sending your signals at all and that the children may not even be started
> yet, or may not have had the time to process the whole config file, etc.
> 
> So now I'm wondering what to do with all this mess. Declaring systemd
> misdesigned and born with some serious trauma will not help us progress
> on this, so we need to work around this pile of crap which tries to prevent
> us from dealing with a simple service.
> 
> Either we find a way to completely redesign the wrapper, even possibly the
> relation between the wrapper and the sub-processes, or we'll simply have
> to get rid of the reload action under systemd and reroute it to a restart.
> 
> I've thought about something which could possibly work though I'm far from
> being sure for now.
> 
> Let's say that the wrapper tries to take an exclusive lock on the pidfile
> upon receipt of SIGUSR2. It then keeps the file open and passes this FD to
> all the haproxy sub-processes. Ideally the FD num is passed as an argument
> to the child.
> 
> Once it fork()+exec(), it can simply close its fd. The exclusive lock is still
> maintained by the children so it's not lost. The benefit is that at this
> point, until the sub-processes have closed the pid file, there's no way for
> the wrapper to pick the same lock again. Thus it can *know* the processes
> have not finished booting. This will cause further SIGUSR2 processing to
> wait for the children processes to either start or die. Sort of a way to
> "pass" the lock to the sub-processes.
> 
> Here we don't even care if signals are sent in storm because only one of
> them will be used and will have to wait for the previous one to be dealt
> with.
> 
> The model is not perfect and ideally a lock file would be better than using
> the pidfile since the pidfile currently is opened late in haproxy and requires
> an unlinking in case of successful startup. But I suspect that using extra
> files will just make things worse. And I don't know if it's possible to flock
> something else (eg: a pipe).
> 
> BTW, that just makes me realize that we also have another possibility for this
> precisely using a pipe (which are more portable than mandatory locks). Let's
> see if that would work. The wrapper creates a pipe then forks. The child
> closes the read side, the parent the write side. Then the parent performs a
> read() on this fd and waits until it returns zero. The child execve() and
> calls the haproxy sub-processes. The FD is closed after the pidfile is updated
> (and in children). After the last close, the wrapper receives a zero on this
> pipe. If haproxy dies, the pipe is closed as well. We could even (ab)use it
> to let the wrapper know whether the process properly started or not, or pass
> the pids there (though that just needlessly complicates operations).
> 
> Any opinion on this ?
> 
> Willy
> 

IMHO: Ask the users to not perform reloads every 2miliseconds. It is insane.
You may spend X hours on this which will make the code a bot more complex and
cause possible breakages somewhere else.

I am pretty sure 90% of the cases which require so often reload are the ones 
which
try to integrate HAProxy with docker stuff, where servers in the pools are 
treated
as ephemeral nodes, appear and disappear very often and at high volume.

@Pieper, what is your user-case for so many reloads?


My 0.02cents,
Pavlos




signature.asc
Description: OpenPGP digital signature

Re: http-reuse always, work quite well

[Pavlos Parissis]

On 21/10/2016 08:14 πμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Wed, Oct 19, 2016 at 08:28:34AM +0200, Pavlos Parissis wrote:
>>> That's really great, thanks for the feedback. Have you tried the other
>>> http-reuse options ?
>>
>> A workmate did the experimentation on http-reuse and I only know that 
>> 'always'
>> worked better for us. I will ask him to provide some details.
> 
> I'm not surprized that always works better, but my point is that if it's
> much better it can be useful to stay with it, but if it's only 1% better
> it's not worth it.
> 

It is way better:-), see Marcin's response.

>>> while "always" is optimal, strictly speaking it's
>>> not very clean if the clients are not always willing to retry a failed
>>> first request, and browsers typically fall into that category. A real
>>> world case can be a request dequeued to a connection that just closes.
>>
>> What is the response of HAProxy to clients in this case? HTTP 50N?
> 
> No, the client-side connection will simply be aborted so that the client
> can decide whether to retry or not.

Connection will be aborted by haproxy sending TCP RST?

> Sometimes even the first request of
> the connection will benefit from a retry, but normally only subsequent
> requests are supposed to be retried.
> 
>>> In theory in your case, "aggressive" should do the same as "always",
>>> though since you know your applications it will not improve anything.
>>> However if "safe" works well enough (even if it causes a few more
>>> connections), you should instead use it. If the gains are minimal,
>>> then you'll have to compare :-)
>>>
>>> Oh, last thing, "always" is generally fine when connecting to a static
>>> server or a cache because you don't break end-user browsing session in
>>> the rare case an error happens.
>>>
>>
>> Fortunately, our applications don't suffer from this. Applications don't 
>> store
>> locally any session information. If a browser requests 10 HTTP requests over 
>> 4
>> TCP connection, those HTTP requests will be always delivered to different
>> servers. Furthermore, we use uWSGI which does also load balancing of 
>> requests to
>> a set of processes which do not share any information.
> 
> OK but what I meant is that if a request fails on the application side, it
> generally has some impact on the user's browsing session. A post which
> returns an error, some automatic filling of a list not being performed,
> etc. In browsers nowadays it's hard to force to retry a failed action
> and reloading the page is not always an option. For a static server if
> an image fails to load, that's just a minor issue and the user can always
> right-click on it, select "view image" and reload it.
> 
> I'd suggest a rule of thumb (maybe this should be added to the doc) : watch
> your logs over a long period. If you don't see queue timeouts, nor request
> timeouts, it's probably safe enough to use "always".

Which field on the log do we need to watch? Tq?

> Each time you notice
> one of them, there is a small risk of impacting another client. It's not
> rocket science but the risks depend on the same parameters.


Thanks a lot for yet another reach in information replies.
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: http-reuse always, work quite well

[Pavlos Parissis]

On 15/10/2016 09:31 πμ, Willy Tarreau wrote:
> Hi Pavlos,
> 
> On Fri, Oct 14, 2016 at 04:33:20PM +0200, Pavlos Parissis wrote:
>> Hi,
>>
>> I just want to drop a note and mention that http-reuse works very well
>> for us:
>>
>> % ss -t state established '( sport = :http )'|wc -l
>> 2113
>>
>> % ss -t state established '( dport = :http )'| wc -l
>> 408
>>
>> As, you can see connections established to backend servers are much less
>> than the connections from clients. In the attached graph you can see
>> 24-hour pattern doesn't influence that much the number of connections to
>> backend.
> 
> That's really great, thanks for the feedback. Have you tried the other
> http-reuse options ?

A workmate did the experimentation on http-reuse and I only know that 'always'
worked better for us. I will ask him to provide some details.

> while "always" is optimal, strictly speaking it's
> not very clean if the clients are not always willing to retry a failed
> first request, and browsers typically fall into that category. A real
> world case can be a request dequeued to a connection that just closes.

What is the response of HAProxy to clients in this case? HTTP 50N?

> Of course this very rarely happens, but "very rarely" is not something
> your end users will accept when they face it :-)
> 

True, very much true.

> In theory in your case, "aggressive" should do the same as "always",
> though since you know your applications it will not improve anything.
> However if "safe" works well enough (even if it causes a few more
> connections), you should instead use it. If the gains are minimal,
> then you'll have to compare :-)
> 
> Oh, last thing, "always" is generally fine when connecting to a static
> server or a cache because you don't break end-user browsing session in
> the rare case an error happens.
> 

Fortunately, our applications don't suffer from this. Applications don't store
locally any session information. If a browser requests 10 HTTP requests over 4
TCP connection, those HTTP requests will be always delivered to different
servers. Furthermore, we use uWSGI which does also load balancing of requests to
a set of processes which do not share any information.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: http-reuse always, work quite well

[Pavlos Parissis]

On 14/10/2016 08:49 μμ, Aleksandar Lazic wrote:
> Hi
> 
> Am 14-10-2016 16:33, schrieb Pavlos Parissis:
>> Hi,
>>
>> I just want to drop a note and mention that http-reuse works very well
>> for us:
>>
>> % ss -t state established '( sport = :http )'|wc -l
>> 2113
>>
>> % ss -t state established '( dport = :http )'| wc -l
>> 408
>>
>> As, you can see connections established to backend servers are much less
>> than the connections from clients. In the attached graph you can see
>> 24-hour pattern doesn't influence that much the number of connections to
>> backend.
> 
> Great ;-)
> 
> Just for my curiosity which great working version do you have in place?
> 

We use HAPEE 1.6 which is HAProxy 1.6.9 version plus some patches from 1.7.

You should expect the same behavior from HAProxy.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Dynamic backend changes without restarting

[Pavlos Parissis]

On 04/10/2016 01:11 μμ, Hayden James wrote:
> Was the concern that a better solution was possible that could fully
> repair the issue instead of a partial fix?
> 

I haven't seen that better solution, I may have missed it as I don't
read linux-netdev ML every day. May be Willy knows something more about it.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: nbproc best practices

[Pavlos Parissis]

On 03/10/2016 07:13 μμ, Mariusz Gronczewski wrote:
> Hi,
> 
> we've come to the point when we have to start using nbproc > 1 (mostly
> because going SSL-only in coming months) and as I understand I have
> to bind each process to separate admin socket and then repeat every
> command for each process, and in case of stats also sum up the
> counters.
> 
> Does any of that is planned to change in the upcoming (say 6-12
> months) release ( e.g. sharing a bit of mem to put stats of all
> processes in same place to access) ? 
> 

I don't know about this, as I haven't seen any commits in 1.7 (current
development release) which provide this functionality. We had a thread
about using JSON format for dumping stats and it was mentioned that
aggregation is something that HAProxy will do. No idea when that will be.

> I was planning on making a small daemon that just connects to socket
> and does required multiplexing and summarising over some text/REST API
> (and then probably push on Github if mgmt. doesn't complain) but I
> wouldn't want to do something that will be obsolete by next release
> 

You can also try to use https://github.com/unixsurfer/haproxytool.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature

Re: Dynamic backend changes without restarting

[Pavlos Parissis]

On 29/09/2016 07:13 μμ, Joseph Lynch wrote:
> You can always dynamically remove servers via the stats socket by 
> downing them. If your server pool is relatively well behaved you can 
> just pre-allocate and up and down as needed.
> 
> If you need to add new servers, afaik you have to reload, which
> won't drop existing connections but may drop new connections for a
> very brief moment (10-20ms). Old instances of HAProxy will hang
> around until all connections through them drain so you have to be
> careful if you're restarting a lot with long timeouts not to run out
> of memory.
> 
> Seehttps://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
>
> 

> for a deep dive into why reloading is a tricky problem. You'll note
> that existing connections are never dropped with -sf , but there are
> a very small number of new connections that are. The community has
> been working on it though!
> 
> Most notably within HAProxy the DNS work Baptiste is doing will 
> hopefully eventually allow complete dynamic re-assignment of the
> hosts and ports within a backend, meaning you just have to reload
> when your pool size is too high. I believe changing host is supported
> and port is in the works (double check me on this, might have
> changed).
> 
> Willy nearly had a Linux kernel patch in to 4.2 that fixed the 
> underlying problem with the Linux kernel, but then it got a bit
> derailed by the BPF changes that got merged. 
> Seehttps://marc.info/?t=14433140591=1=2 
>  for the story there.

Guys proposed to use BPF resolve the issue, but they didn't provide a
workable solution, which Willy's patch fixed the problem, at least
partially.

I know you can do a lot of interesting and useful things with BPF, like
stracing application in production without performance degradation, but
they could have accepted Willy's patch until a permanent solution arrives.

May be I am missing something here.

Cheers,
Pavlos



signature.asc
Description: OpenPGP digital signature