Re: Logging using %HP (path) produce different results with H1 and H2

[Ciprian Dorin Craciun]

On Wed, Aug 26, 2020 at 6:29 PM Willy Tarreau  wrote:
> > Then, still related to logging, might I add a feature request to just
> > use "raw" lines over UDP or TCP, instead of SysLog?  (Just as we have
> > now support for `stdout`, but with network support.)
>
> You already have it. I'm doing this:
>
> log 127.0.0.1:5514 format raw daemon
> log-format "%[uuid]"
>
> And netcat shows me only the uuid with no header nor anything. So
> technically speaking I don't think you're missing anything for this.


This is nice.  I'll have to give it a try.  (I thought that the `raw`
applied only to `stdout`.

However we are missing the TCP support.  :)

Ciprian.

Re: Logging using %HP (path) produce different results with H1 and H2

[Ciprian Dorin Craciun]

On Wed, Aug 26, 2020 at 1:08 PM Willy Tarreau  wrote:
> > Also what would be extra useful, especially for debugging and perhaps
> > security, is to add an `http-request|http-response log 
> >  if ` that would allow adding additional log
> > lines if a certain condition is met.
>
> We cannot realistically "add more lines" to a request, however deciding
> to emit one line at the instant the rule is processed shouldn't be hard
> at all, and I definitely see the value there, as you say, both for
> debugging and for security alerts.


Yes, you are right, I don't expect such a statement to add a new line
to the buffer, but to immediately format a log line and submit it.

(This also aids in debugging, because if the same log line is put in
multiple places of the `http-request` chain, one could see how a
certain header has changed, and it could also help in generating a log
line the moment the request was parsed, as opposed to waiting for the
response to be processed.)




> There's something I've been wanting for a while which would be log profiles
> (pre-defined and reusable log-format directives) that would significantly
> help with such situations as you probably don't want to duplicate the
> exact same format all the time. As such I'd rather first define how we
> want to reference them, then make sure that the suggestion above can be
> done in a forward compatible way.


Although `log-format` profiles might be useful, I would still keep the
option to just "paste" the same format everywhere it is needed.

(As a minor side-note:  the HAProxy configuration syntax is far from
trivial, and I see it more as an "assembler" than as a "high level
language";  therefore I've built myself a Python generator,
https://github.com/cipriancraciun/haproxy-configurator/tree/master/examples,
that just generates the matching HAProxy configuration file.  Moreover
I guess this is the way many other HAProxy integrators do it,
especially in the micro-services world.)


> These are good ideas that will fuel a long feature request :-)

Then, still related to logging, might I add a feature request to just
use "raw" lines over UDP or TCP, instead of SysLog?  (Just as we have
now support for `stdout`, but with network support.)

I know there is the issue of reliability, but that is not an issue,
given as now SysLog is also over UDP.

Thanks,
Ciprian.

Re: Logging using %HP (path) produce different results with H1 and H2

[Ciprian Dorin Craciun]

On Tue, Aug 25, 2020 at 3:56 PM Willy Tarreau  wrote:
> There was something important I've been wanting for a few versions, which
> was to have named log formats that we could declare in a central place and
> use everywhere. It would tremendously help here. I know it can be done
> using environment variables declared in the global section but I personally
> find this ugly.
>
> So I think it's the right place to open such a discussion (what we should
> log and whether or not it loses info by default or requires to duplicate
> some data while waiting for the response), so that we can reach a better
> and more modern solution. I'm open to proposals.


If we are to talk about logging, please make it easy to have valid
JSON log lines.

I know there is the `%[something(),json()]` that works as expected;
however there are many (especially old) log items (say `%ST`) that
sometimes expand to a number, and sometimes to a `-`, which basically
means one has to always treat everything as strings, and hope the
actual value doesn't contain a quote or JSON escape codes...  In fact,
to keep things simple, I would suggest just adding new functions
(fetch samples as they are called) that expose all items available in
the `Custom log format` section  (some are available, some are not
from what I remember);  these should expand to an empty string if they
are not available (as opposed to a `-`).

Also what would be extra useful, especially for debugging and perhaps
security, is to add an `http-request|http-response log 
 if ` that would allow adding additional log
lines if a certain condition is met.

Thanks,
Ciprian.

Re: Logging using %HP (path) produce different results with H1 and H2

[Ciprian Dorin Craciun]

On Fri, Aug 21, 2020 at 9:14 PM William Dauchy  wrote:
> On Fri, Aug 21, 2020 at 7:46 PM Pierre Cheynier  wrote:
> > We're running HAProxy 2.2.2.
> > It turns out logging requests paths using "%HP" var produce a different 
> > results on H1 vs. H2.
> >
> > H1: /path
> > H2: https://hostname.domain/path (< I consider this one buggy)


I've also hit this issue in May
(https://www.mail-archive.com/haproxy@formilux.org/msg37170.html), and
as many continue to upgrade to newer HAProxy versions, they might hit
that as well.  Personally I've opted into using variables set from
request / response rules, and then logging those in a custom log
format.

However the major issue, if I remember correctly, isn't that the
format has changed between HAProxy 2.0 and 2.1, but that in 2.1 the
same `%HP` produces different outputs based on the used protocol,
which will complicate log handling...

Perhaps a backward incompatible fix would be to always include the
protocol and `Host` header (or the IP if one is missing) in case of
HTTP/1.0 and HTTP/1.1, and thus have `%HP` always contain an URL.


> I believe this is expected; this behaviour has changed since v2.1 though.

It might be so, but the changelog for 2.1 says nothing about this change.

Ciprian.

The log variable `%HP` includes the host for HTTP/2 but not for HTTP/1.1 (backward incompatible change?)

[Ciprian Dorin Craciun]

I've upgraded from HAProxy 1.8 to 2.1 branch, and without changing the
configuration I've observed the following inconsistency:

The `%HP` log variable used to contain only the path for both HTTP/2
and HTTP/1.1 in HAProxy 1.8.

How with HAProxy 2.1.4 (as available on OpenSUSE 15.1) the `%HP` log
variable contains the full URI (without query) for HTTP/2, however for
HTTP/1.1 it contains only the path.

Is this an expected change, or is it a backward incompatible change?

Thanks,
Ciprian.

Strange issue: stalling while reading data from a backend server (happens on 2.0.10)

[Ciprian Dorin Craciun]

Sorry for bothering you, but I've searched in the release notes for
2.0.x, and the mailing list, and didn't find anything related to the
following issue.

So, I'm using HAProxy 2.0.10 as available on OpenSUSE 15.0, and
recently I've stumbled upon the following bug:
* I have a simple backend server that serves static files over plain
HTTP/1.1, without HTTP KeepAlive;  (nothing fancy);
* I have a HAProxy that sends certain requests to that backend server;
 (the config is quite complex, however as seen bellow I doubt the
issue is my configuration;)
* everything is done over loopback;  (both HAProxy, backend server and client;)
* for small requests (say 8, 16, and so up to 128K) everything works fine;
* for requests larger than 256K HAProxy "stalls" by apparently reading
the response headers, but not reading all the data;  (I see this from
`netstat` and `tcpdump`;)
* it happens with and without compression;  (compression is controlled
in HAProxy by MIME type and via `curl --compressed`;)
* this issue seems to happen on both HTTP/1.1 and HTTP/2, and in case
of HTTP/1.1 both with and without TLS;

I have downgraded HAProxy to 1.8.8 or upgraded to 2.1.4 and the issue
seems to disappear.

Did anyone encounter something like this?  Any pointers to what myight
trigger this?

Thanks,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Thu, Dec 19, 2019 at 7:08 AM Илья Шипицин  wrote:
>> Really I would have expected to find a lightweight and generic Python
>> or Ruby "framework" that allows one to write such tests in a more
>> "programmatic" manner...  :(
>
> python mechanize ?


Looking at the documentation, https://mechanize.readthedocs.io/, it
seems targeted towards HTML testing and automation, say something more
similar to Selenium.

But the direction is quite good, i.e. it includes programatic
definition, however I'm serching something targeting mainly the
HTTP-layer, and as added-value if it allows me to write "application"
tests the better.  :)


Aside from `mechanize`, there seems to be two categories of tools, at
opposite ends, without any overlapping:
* we have the web-application testers, like Postman, `mechanize` (from
what I've seen), a few other (especially NodeJS-based)
implementations, that focus exclusively on the application layer;
(i.e. HTML, JSON, forms, etc.)
* we have the low-level HTTP-layer load generating tools like `wrk` or
`ab`, that have almost no capability of checking the "application"
layer, and most of the time neither the HTTP layer;
* and we have `vtest` / `varnishtest` (and perhaps a few others that I
haven't found yet) that also focus on the HTTP-layer, but mainly only
target the proxy / cache aspect of it, and lack basic functionality
for any application testing;

I'm searching for something "in-between".  :)

Thanks,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Thu, Dec 19, 2019 at 9:10 AM Jarno Huuskonen  wrote:
> > So my question to the HAProxy community is if anyone knows / uses a
> > generic HTTP unit testing framework.
>
> Have you looked into varnishtest(vtest) ? There're examples in haproxy source
> reg-tests directory.


Interesting;  for reference I'll put the relevant links bellow:

  https://github.com/vtest/VTest -- the actual tool used by HAProxy
  https://github.com/haproxy/haproxy/tree/master/reg-tests
  https://varnish-cache.org/docs/trunk/reference/varnishtest.html
  https://varnish-cache.org/docs/trunk/reference/vtc.html#haproxy

I'll have a more closer look, however at a first glance the following
sticks out to me:
* (as a plus)  it seems to be quite competent and allows many
use-cases;  (even checking very low-level details such as TCP
window-size;)
* (as a minus)  the syntax is quite awkward;  therefore it will be
very hard for most to write checks;
* (as a minus)  it can't be extended into more "application-side"
testing;  (i.e. what Postman does best;)
* (as a minus)  I think it's targeted more at testing small,
individual, features, as for example I don't think it's able to
generate the test cases (including URLs, headers, etc.)
programatically, for example taking them from a file, etc.;

Thanks for the pointer, I'll take a closer look and perhaps I'm
mistaken about many of the above,
Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Wed, Dec 18, 2019 at 8:23 PM Илья Шипицин  wrote:
>> redirects are easy to test. using any framework.
>
> for example, jmeter (or any other http query tool)
>
>> * apply authentication for `admin`;
>> * force some caching headers for `web`, `static` and `media`;
>
> same here. assert in jmeter would serve caching headers check
>
>> * apply some "sanity" checks to requests / responses (i.e. except
>> admin, the rest should only serve `GET` requests);
>> * deny any request that doesn't match a set of domains
>
> same here. just fire random query using jmeter


JMeter, like Postman, don't seem very adequate tools for this purpose;
 for starters they are too centered around a UI (especially for
"editing" the tests).  (Although they have CLI runners, and perhaps
their "tests collections" can be serialized in a file and put in a Git
repository, I don't think one can easily use it with e.g. a custom
editor, etc., like any "normal" configuration.)

For example:
* https://jmeter.apache.org/demos/ -- an XML file for JMeter JMX files
used as "test plans";
* 
https://github.com/Blazemeter/taurus/blob/master/examples/functional/postman-sample-collection.json
-- a JSON file (random example) for Postman;


The closest I could find something that resembles "usability" is
Tavern, which at least has the advantage of YAML as opposed to "raw"
XML or JSON:
* https://github.com/taverntesting/tavern
* 
https://github.com/taverntesting/tavern/blob/master/example/simple/test_server.tavern.yaml


Really I would have expected to find a lightweight and generic Python
or Ruby "framework" that allows one to write such tests in a more
"programmatic" manner...  :(

Ciprian.

Re: How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

On Wed, Dec 18, 2019 at 6:47 PM Илья Шипицин  wrote:
> you are talking about testing ACL. can you provide some example ?


So let's assume I have a given HAProxy configuration, full of ACL's
and rules, that apply certain "firewalling", authentication /
authorization, "mangling" operations to the HTTP request (e.g. drop
"bad-bots", update headers, redirects, routing to various backends,
etc.).

Now how can I test that the HAProxy configuration actually
"implements" what it's proposes to?  I.e. how can I be sure that the
rules are in the proper order, that no ACL's are missing, etc.

My answer would be:  fire an HTTP request and see if it "does" what it
should.  (Perhaps expose as HTTP headers some "state" values to help
in checking things.)




My concrete example would be this:  I find HAProxy wonderful for any
non trivial HTTP deployment (and in fact anything "touched" by the
Internet);  unfortunately the configuration language (with it's flat
ACL's and request / response rules) is like "assembler" (as opposed to
say Python).  Therefore I've written myself a HAProxy "configurator"
in Python that based on simple Python code generates the full HAProxy
configuration.

For example:

  
https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/example-01.py
  
https://github.com/cipriancraciun/haproxy-configurator/blob/master/examples/_configs/example-01.cfg

, the Python script is (hopefully) readable and clearly shows the
intent of the resulting configuration:
* redirect everything via HTTPS;
* redirect `example.com` to `www.example.com`;
* redirect `/admin/*` to `admin.example.com/admin/*`, same for `/blog/*`;
* apply authentication for `admin`;
* force some caching headers for `web`, `static` and `media`;
* apply some "sanity" checks to requests / responses (i.e. except
admin, the rest should only serve `GET` requests);
* deny any request that doesn't match a set of domains

The resulting file is around 639 lines, and (given how I've chosen to
identify ACL's) is quite hard to "follow by hand".

So my question now is how do I test it...  Fire HTTP requests at it!  :)


I hope this gives everyone a glimpse into my use-case,
Ciprian.

How to "unit" test HAProxy configurations (and HTTP servers in general)

[Ciprian Dorin Craciun]

Hello all!

[First of all this question isn't only HAProxy specific, but can be
extended to any HTTP server (including Apache, Nginx, and any other
web application out there);  however I think it is especially
important for HAProxy given how many HTTP-routing / mangling
capabilities it has.]

I have a quite complex HAProxy configuration and I want to make sure
that while changing some ACL's and rules I don't break something else.

Therefore I set out to find a tool that allows me to "unit test" an
HTTP server (thus including HAProxy).  And to my surprise I didn't
find one...  Obviously there are many "framework" unit test platforms
out-there, each tailored for the underlying framework, such as Django,
Flask, Rails, Spring, Go-lang, etc.;  however nothing "generic" that
can test a web server by plain HTTP requests.


So my question to the HAProxy community is if anyone knows / uses a
generic HTTP unit testing framework.

(I have already written a few custom Bash scripts that given an URL
create a file on disk, and given it resides in a Git repository, I can
easily `git diff ./tests/responses` to see if anything changed, but
this is too "barbaric"...)  :)

Thanks,
Ciprian.

Re: MEDIUM: Adding upstream socks4 proxy support

[Ciprian Dorin Craciun]

On Thu, Jun 6, 2019 at 12:27 PM Igor Pav  wrote:
> Sorry to ask a not so related question here, I have a Linux gateway to
> redirect user's TCP traffic by using iptables like `iptables -t nat -A
> PREROUTING  -p tcp dst -j REDIRECT --to-ports 1000`, port 1000 is
> redsocks transparent tcp-to-socks proxy,
> since we have Alec's patch here, I wonder if that easy to modify the
> patch to meet my weird requirement like:
> ```
> listen tcps
>  bind: 1000 transparent
>  server x.x.x.x:2000 ssl
> ```
> x.x.x.x:2000 is a remote socks server with ssl wrap. I want user TCP
> traffic from port 1000 to redirect to remote socks server. I'm not so
> familiar in haproxy core architecture, can you guys point me which
> part in the source should I look at?


Hello Igor!

This is exactly the same use-case I've proposed (along others that
would fit in the same "pattern"), which was spawned into a different
thread with the subject:

  Discussion about "Upstream socks proxy support #82"

(Thus I would suggest to switch to that thread.)

At the moment, it is still debated if such a feature would be useful,
and unfortunately nobody has stepped-up to write a patch.  :)

Ciprian.

Re: Discussion about "Upstream socks proxy support #82"

[Ciprian Dorin Craciun]

On Mon, Jun 3, 2019 at 3:49 AM Aleksandar Lazic  wrote:
> nutinshell have another use case which is a `socks4-redirect`
> https://github.com/haproxy/haproxy/issues/82#issuecomment-498007739


Is there such a specification for `socks4-redirect`?  (I've looked in
the original SOCKS4 specification and didn't find any mention about
it:  https://www.openssh.com/txt/socks4.protocol .)




> I see at least this use cases from this thread.
>
> * client with explicit SOCKS support -> HAProxy SOCKS frontend -> normal 
> backend
> / servers -- useful for routing only specific domains through HAProxy
>
> The suggestion here is to handle socks in the same way as proxy protocol. As
> mentioned above I don't think that's that easy as the socks protocol is a
> request response system which proxy protocol isn't.


Based on the SOCKS4 specification, the protocol is "request-reply"
only in the first interaction (one request / one reply), and after
that it goes into tunnel mode:

https://www.openssh.com/txt/socks4.protocol
For a successful request,
the SOCKS server gets ready to relay traffic on both directions. This
enables the client to do I/O on its connection as if it were directly
connected to the application server.



> There are also not only "the" socks protocol as there is socks4,socks4a and
> socks5 with different feature sets and requirements. From what I seen in the
> thread is only socks4 with command **connect** expected. This means that the
> client sends the destination ip and port therefore haproxy does not need to
> resolve the destination.


Yes, that is the way I've proposed it:  SOCKS4 plain-and-simple, IPv4
only, no DNS, no other extensions.  (At least not in the first
iteration, although perhaps IPv6, without DNS resolutions, might also
be useful, as per https://tools.ietf.org/html/rfc3089 )




> From my point of view it is at least a similar option like `accept-proxy`
> required for example `accept-socks(4|4a|5)`
>
> One of the question which is open is make `accept-proxy` and
> `accept-socks(4|4a|5)` sense?
>
> @cipriancraciun and others opinions?


I think `accept-socks` would be enough (for the frontend), because
from what I gather the protocol version can be decoded from the actual
headers.  (Perhaps a new sample might be added to determine that SOCKS
is used, and which version, to then force certain versions.)

However, just like in case of `accept-proxy`, on the backend (better
said server) side an `send-socks(4|5)` would be needed to allow the
administrator to choose the downstream protocol.  (SOCKS4a from what I
gather added support for DNS resolution, which we don't actually
need.)




> * client (with / without) SOCKS support -> HAProxy frontend -> SOCKS enabled
> server -- useful for traffic filtering, and redirection to a proper SOCKS 
> proxy;
>
> The reason why the patch from alec isn't enough is that answer
>
> 
> From what I read @alec-liu implemented SOCKS4 in the backend only as a way to
> submit requests for to a "fixed" server IP through a "fixed" SOCKS4 proxy 
> server.
> 


Exactly, I underline once more, that the main difference with the
other proposed patch, is that in my proposal the server IP is the IP
of the SOCKS gateway, and that the sent IP in the SOCKS4 header is the
one in `src`, just as it happens in case of the (HA)PROXY protocol.)




> I think this feature would be nice in HAProxy but I also think it's a huge
> amount of work and increases the complexity to debug any errors.


In case of only supporting SOCKS4 as an alternative to the (HA)PROXY
header, the amount of work should be minimal, except that one also has
to read the initial reply from the SOCKS gateway.

Unfortunately I'm not that acquainted with the HAProxy internals, but
if one could pin-point me where in the "reply" path I should include
the response handling I could take a stab at it if I can find some
time.  Some time ago Willy provided me some pointers, but I bet given
the recent work to support HTTP/2, a lot has changed in the interim.

Ciprian.

Re: Small question regarding the sub-dir match (i.e. `-m dir`) with regard to "rooted" patterns

[Ciprian Dorin Craciun]

On Mon, Mar 11, 2019 at 1:22 PM Tim Düsterhus  wrote:
> > BTW, should I also open a feature request for an actual "subdir"
> > match?  (Perhaps if I have some time I could even come-up with a
> > patch...)
>
> I am not responsible for deciding what gets in and what doesn't. But: In
> my opinion there is no need for such a match as it can easily be
> "emulated" by combining `dir` and `beg` or `eq` and `beg` like you did.
> A regular expression such as ^/test(/|$) should work as well.


My main issue with the "alternatives" (and especially the regular
expression) is performance...  It is trivial for an `subdir` match to
do exactly what `beg` does, but check an extra character, than doing
two matches of exactly the same string...

Ciprian.


> If you consider doing a patch: Please read the CONTRIBUTING guide and
> send the patch to the list (i.e. not using a GitHub pull request, only
> the GitHub issue tracker is being used).

Good to know.

Ciprian.

Re: Small question regarding the sub-dir match (i.e. `-m dir`) with regard to "rooted" patterns

[Ciprian Dorin Craciun]

On Mon, Mar 11, 2019 at 1:12 PM Tim Düsterhus  wrote:
> I filed an issue to look into this:
> https://github.com/haproxy/haproxy/issues/61


Thanks.  (I didn't knew about the GitHub issues as being the official
channel to submit issues and requests.)

BTW, should I also open a feature request for an actual "subdir"
match?  (Perhaps if I have some time I could even come-up with a
patch...)

Thanks,
Ciprian.

Re: Small question regarding the sub-dir match (i.e. `-m dir`) with regard to "rooted" patterns

[Ciprian Dorin Craciun]

On Mon, Mar 11, 2019 at 12:58 PM Tim Düsterhus  wrote:
> The documentation only talks about a slash-delimited value, not about
> being at the beginning.

Technically yes, it works as documented, except the documentation is
misleading by using the work `subdir match` right in the beginning...

Moreover the name of the operator is `dir` which (my guess is that)
most would just mistakenly associate with a file-system, where they
would expect to match a prefix.

I would strongly suggest adding an explicit warning in the
documentation about this pitfall.


> The latter requirement would be redundant with
> the the prefix match: `-m beg`.

`-m beg` isn't exactly equivalent with an actual subdir match, because
both `/test123` and `/test/123` would be matched by `-m beg`, and only
the later is actually "under" the directory `/test`.

In the end I've used two ACL's, `-m dir /test` AND `-m beg /test`.
(Alternatively I could have used two like `-m eq /test` OR `-m beg
/test/`.)

Thanks,
Ciprian.

Re: Small question regarding the sub-dir match (i.e. `-m dir`) with regard to "rooted" patterns

[Ciprian Dorin Craciun]

[I'm re-sending this email as I guess it "fell through the cracks",
and I do believe that it is a bug.]


According to the HAProxy 1.8 documentation:
  http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.1


  - "dir"   : subdir match : check that a slash-delimited portion of the
  contents exactly matches one of the provided string patterns.
  This may be used with binary or string samples.
[...]
  - subdir match(-m dir) : the patterns are looked up inside the extracted
string, delimited with slashes ("/"), and the ACL matches if any of them
matches.


It is unclear if the math happens with the pattern "rooted" (i.e. the
pattern must also be a prefix) or not?

More exactly given an ACL of the form:

acl subdir path() -m dir -- /test


Which of the following would it match:
* `/test` -- I would expect yes;
* `/test/whatever` -- I would expect yes;
* `/whatever/test` -- I would expect no, however it does (at least in 1.8.14)...

Thanks,
Ciprian.

Re: nmap showing internal IP disclosure of ELB and not the HAPROXY of port 80

[Ciprian Dorin Craciun]

On Sat, Mar 9, 2019 at 10:45 AM DHAVAL JAISWAL  wrote:
> frontend loadbalancer_mycom
> bind 10.100.22.30:80
> mode http
>
> redirect scheme https if !{ ssl_fc }


If this line is the one that makes the redirect (and exposes the
internal IP in case of HTTP/1.0) then you can't fix it as it's part of
HAProxy internal code.

Perhaps you should use:

  redirect location https://mysite.com%[path] if !{ ssl_fc }


Additionally if you don't know beforehand the name of the domain you
could just deny all requests that don't have the `Host` header like so
(put before the `redirect` statement) (this basically breaks
HTTP/1.0):

  acl has_host req.hdr(Host) -m found
  http-request deny if !has_host
  redirect scheme https ...



[I've also included some other hints based on your config.]

> global
> maxconn 2 # Total Max Connections. This is dependent on ulimit
> [...]
> defaults
> maxconn 25000

The number from `defaults` should be less than the number in `global`
(I think;  double-check the documentation.)



> nbproc 8 # Number of processing cores. Dual Dual-core Opteron is 4 cores for 
> example.

Perhaps you should move to threads instead of processes, especially in
latest versions of HAProxy.



> log /dev/log local1 info
> log /dev/log local1 notice

Isn't the log duplicated?  (You can use only one line.)



> defaults
> timeout connect 120
> timeout client 120
> timeout server 120
> timeout http-request 120
> timeout tarpit 120


The timeouts are too "permissive" (2 minutes, especially for `client`
and `http-request`) and would easily allow an DoS attack by just
opening the connection and just idling for 2 minutes or slowly writing
the HTTP request for 2 minutes.



> backend my_cluster_mycom]
> [...]
> fullconn 1
> #server server0320 10.100.3.20:8080 weight 1 maxconn 512 check cookie t0320 
> check port 8080 inter 10s rise 3 fall 3 minconn 500 maxconn 3000 maxqueue 300 
> slowstart 15s
> #server server3320 10.100.33.20:8080 weight 1 maxconn 512 check cookie t3320 
> check port 8080 inter 10s rise 3 fall 3 minconn 500 maxconn 3000 maxqueue 300 
> slowstart 15s


You have two `maxconn` in the server lines.  (Although the servers
seem commented...)



> compression algo gzip
> compression type text/html text/plain text/css image/png image/gif image/jpeg 
> application/x-javascript text/xml application/xml application/xhtml+xml 
> application/x application/javascipt image/jpg


I think it's useless to compress images, especially JPEG (but also
perhaps PNG and GIF).



> timeout client 50


I think you are really opening yourself to an DoS attack.  :)  (~8
minutes is plenty to just eat your connections...)

If you set such large timeouts, then perhaps you should also try to
limit the number of connections per the same IP / network.  (Use stick
tables for this or `iptables` rules.)


Ciprian.

Re: nmap showing internal IP disclosure of ELB and not the HAPROXY of port 80

[Ciprian Dorin Craciun]

On Sat, Mar 9, 2019 at 9:06 AM DHAVAL JAISWAL  wrote:
> While doing network scan its showing internal IP disclosure vulnerability. 
> This internal IP is of ELB and not the HA proxy server.
>
> It is showing vulnerability on port 80. Following are the steps for reproduce.
>
> Can some one help me to fix this?


Based on the source code of that script the leak seems to be based on
the `Location` header used in redirecting HTTP to HTTPS:
  ( https://svn.nmap.org/nmap/scripts/http-internal-ip-disclosure.nse )


However that happens only when the client doesn't send the `Host` header:

  printf 'GET / HTTP/1.0\r\n\r\n' \
  | socat stdio tcp:viacom-214916319.ap-south-1.elb.amazonaws.com:80


Could you perhaps take a look at the HAProxy configuration where
you're doing HTTPS redirects?


Ciprian.

Is `ssl_fc` set when using `send-proxy-v2-ssl` and the original connection was over TLS? (HAProxy<->HAProxy setup)

[Ciprian Dorin Craciun]

Hello all!

As the subject anticipates, I have (on HAProxy 1.8.14) the following setup:
* an "upstream" HAProxy instance listening both on HTTP and HTTPS,
which sends to a beckend that has configured `send-proxy-v2-ssl`;
* a "downstream" HAProxy instance listening with `accept-proxy`;

Apparently the `ssl_fc` on the downstream HAProxy is always set to
false even though the original request originates over TLS, and the
PROXY v2 protocol seems to correctly contain this information as seen
from the network capture bellow.


  0d 0a 0d 0a 00 0d 0a 51  55 49 54 0a 21 11 00 23   ...Q UIT.!..#
0010  4f 76 0a b9 55 5a f4 f9  d9 ea 01 bb 01 00 02 68   Ov..UZ.. ...h
0020  32 20 00 0f 01 00 00 00  00 21 00 07 54 4c 53 76   2 .. .!..TLSv
0030  31 2e 32 47 45 54 20 2f  20 48 54 54 50 2f 31 2e   1.2GET /  HTTP/1.
0040  31 0d 0a 75 73 65 72 2d  61 67 65 6e 74 3a 20 63   1..user- agent: c
0050  75 72 6c 2f 37 2e 36 33  2e 30 0d 0a 61 63 63 65   url/7.63 .0


Thus my question is if `ssl_fc` is set to true only if the "current"
transport is actually over TLS?

And my second question, is there an ACL that evaluates to true if the
PROXY v2 protocol was used, and the TLS is present in the meta-data.

(I can obviously use the frontend port, which is correctly configured
to 443, to bypass `ssl_fc`, however I was looking for a more
"definitive" solution.)

Thanks,
Ciprian.

Re: Allowing more codes for `errorfile` (like 404) (that can be later re-used with `http-request deny deny_status 404`)

[Ciprian Dorin Craciun]

On Sun, Feb 10, 2019 at 1:00 PM Moemen MHEDHBI  wrote:
> For example, it is not the role of a reverse proxy to fetch a Web resource so 
> returning "404 Not found" won't make much sense and will make debug harder 
> when trying to identify where the 404 originated from.


For starters, if one assumes that HAProxy is (capable of) more than a
"reverse proxy", then you quickly start to all sorts of use-cases
especially in the realm of WAF (Web Application Firewalls).

For example it is notorious that many (less popular) frameworks are
too "leaky" in their error pages (especially in the 502 area), and
provide too much information that could be used by an attacker to
"scout" around.  Therefore I always have a rule in my HAProxy in which
if the backend doesn't respond with a "proper" code (in the 200 and
300 range), I just mask that with a `http-response deny`.

A similar approach I wanted to take with 404, in masking certain URL's
that are not actually allowed outside the "trusted" networks.


> That being said, it is still very useful to be able to make a reverse proxy 
> send any desired response. So you can do that by hard coding the error you 
> want to send in the errorfile. Example:
>
> errorfile 400 /tmp/400
> http-request deny deny_status 400 if { path_beg /test }
>
> Then in your /tmp/400:
>
> HTTP/1.1 404 Not Found


I know about the fact that in the `errorfile` you can put verbatim any
HTTP valid response, unfortunately the number of error codes is quite
limited.  (And 400 is not the same as 404.)


> HAProxy provides a cache, which was designed to perform cache on small 
> objects (favicon, css...). So this may be what you are looking for.

Unfortunately the feature I'm proposing is not as a way to "offload"
load from the actual web server (where the caching feature would
help), but instead of being able to serve some "operational" pages
straight out of HAProxy, without additional backend servers (or Lua)
code.

Again this use-case is geared more towards CDN custom error pages or
service routers.

Ciprian.

Supporting `http-response deny deny_status ` (just like for `http-request`)

[Ciprian Dorin Craciun]

Is there a reason why there isn't support for `deny_status ` in
`http-response` as it already exists in `http-request`?

Looking at the configuration code:

  
https://github.com/haproxy/haproxy/blob/06f5b6435ba99b7a6a034d27b56192e16249f6f0/src/http_rules.c#L83
  
https://github.com/haproxy/haproxy/blob/06f5b6435ba99b7a6a034d27b56192e16249f6f0/src/http_rules.c#L588

It seems like there just isn't the code necessary to configure that status code?



I've then tried to look into the actual handling code:

  
https://github.com/haproxy/haproxy/blob/21c741a665f4c7b354e961267824c81ec44f503f/src/proto_http.c#L1906
  
https://github.com/haproxy/haproxy/blob/afe57846bfb2edfb0db53f942b9500d090377c54/src/proto_htx.c#L3185

, and especially:

  
https://github.com/haproxy/haproxy/blob/21c741a665f4c7b354e961267824c81ec44f503f/src/proto_http.c#L4942
  
https://github.com/haproxy/haproxy/blob/afe57846bfb2edfb0db53f942b9500d090377c54/src/proto_htx.c#L2079

It seems that the code already exists there to handle any code (if
permitted by `errorfile` related structures), but it just defaults to
502.



Would a patch adding this feature be accepted?

Thanks,
Ciprian.

Allowing more codes for `errorfile` (like 404) (that can be later re-used with `http-request deny deny_status 404`)

[Ciprian Dorin Craciun]

First of all I understand that the `errorfile` (and related
`errorloc`) are for HAProxy's own generated errors.

However given how powerful the ACL system is, and the availability of
`http-request deny deny_status `, one can leverage all this and
implement a powerful WAF.

For example last week I tried to "hide" some URL's and pretend they
are 404's directly from HAProxy (as it seems that the backend server I
was using doesn't support this feature...)  However when I tried to
use `deny_status 404` it actually replied with a 403 (and the custom
page for that error).  Now this is not a big issue, however it might
give an "attacker" a hint that "something" is at that URL...



Therefore I took a look at the `errorfile` related code and how it is
currently implemented:


https://github.com/haproxy/haproxy/blob/61ea7dc005bc490ed3e5298ade3d932926fdb9f7/include/common/http.h#L82-L96

https://github.com/haproxy/haproxy/blob/61ea7dc005bc490ed3e5298ade3d932926fdb9f7/src/http.c#L218-L231

https://github.com/haproxy/haproxy/blob/61ea7dc005bc490ed3e5298ade3d932926fdb9f7/src/http.c#L233

https://github.com/haproxy/haproxy/blob/06f5b6435ba99b7a6a034d27b56192e16249f6f0/src/http_rules.c#L106-L111

https://github.com/haproxy/haproxy/blob/21c741a665f4c7b354e961267824c81ec44f503f/include/types/proxy.h#L408

At a first glance it implements a sort of "associative-array"
structure, of fixed `HTTP_ERR_SIZE`, with the actual codes being
statically mapped to indices via `http_err_codes`.

There is a global default array `http_err_msgs` (plus the
`http_err_chunks` counterpart) and a per-proxy embedded array
`proxy->errmsg`, both pre-allocated of size `HTTP_ERR_SIZE`.

Therefore adding support for a new status code is quite trivial, just
create the relevant entries in `http_err_codes` and `http_err_msgs`
and a new `HTTP_ERR_*` entry.  The downside is minimal, just a slight
increase of `sizeof (struct buffer)` bytes (32) globally and one for
each proxy, and almost no run-time impact in terms of CPU.

Thus I would suggest adding at least the following:
* 404 -- not-found -- with an obvious use-case;
* 402 -- payment-required -- which might be used as a "soft" 403
alternative, which suggests that although you are "authenticated" and
issued a valid request, you are not-allowed because of some
"accounting" reason;
* 406 -- not-acceptable -- although used for content negotiation, it
could be "abused" as a page indicating to the user that his "browser"
(or agent) is not "compatible" with the generated content;
* 409/410 -- conflict/gone -- alternatives for 403/404 if the user has
need for them;
* 451 -- unavailable-for-legal-reasons -- perhaps useful in these GDPR-days?
* 501 -- not-implemented -- useful perhaps to "hide" some
not-yet-published API endpoints, or other backend-related
"blacklisting";



However my proposal would be to allow "user-defined" error codes, the
main use-case being WAF/CDN custom status codes
(https://support.cloudflare.com/hc/en-us/sections/200820298-Error-Pages).

Such a change shouldn't be that involved, although for efficiency the
`proxy->errmsg` should be transformed from an embedded array to an
array pointer of variable size.

Would such a feature be accepted by the HAProxy developers?



Moreover, dare I say, this feature could be "abused" to serve a few
"static files" (like `favicon.ico` or `robots.txt`) directly from
HAProxy without requiring Lua.  In fact the most viewed topic on
HAProxy's forum is exactly about this:

  
https://discourse.haproxy.org/t/how-do-i-serve-a-single-static-file-from-haproxy/32

Ciprian.

Small question regarding the sub-dir match (i.e. `-m dir`) with regard to "rooted" patterns

[Ciprian Dorin Craciun]

According to the HAProxy 1.8 documentation:
  http://cbonte.github.io/haproxy-dconv/1.8/configuration.html#7.1


  - "dir"   : subdir match : check that a slash-delimited portion of the
  contents exactly matches one of the provided string patterns.
  This may be used with binary or string samples.
[...]
  - subdir match(-m dir) : the patterns are looked up inside the extracted
string, delimited with slashes ("/"), and the ACL matches if any of them
matches.


It is unclear if the math happens with the pattern "rooted" (i.e. the
pattern must also be a prefix) or not?

More exactly given an ACL of the form:

acl subdir path() -m dir -- /test


Which of the following would it match:
* `/test` -- I would expect yes;
* `/test/whatever` -- I would expect yes;
* `/whatever/test` -- I would expect no, however it does (at least in 1.8.14)...

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 2:22 PM Willy Tarreau  wrote:
> > As seen the timeout which I believe is the culprit is the `timeout
> > client 30s` which I guess is quite enough.
>
> I tend to consider that if the response starts to be sent,
> then the most expensive part was done and it'd better be completed
> otherwise the client will try again and inflict the same cost to the
> server again.


I prefer shorter timeout values because on the server side I have
uWSGI with Python, and with its default model (one process / request
at one time), having long outstanding connections could degrade the
user experience.


> You should probably increase this enough so that you
> don't see unexpected timeouts anymore, and rely on tcp-ut to cut early
> if a client doesn't read the data.


One question about this:  if the client gradually reads from the
(server side) buffer, but it doesn't completely clears it, having this
`TCP_USER_TIMEOUT` configured would consider this connection "live"?
More specifically, say there is 4MB in the server buffer and the
client "consumes" (i.e. acknowledges) only small parts of it, would
the timeout apply as:
(A) until the entire buffer is cleared, or
(B) until at least "some" amount of data is read;

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 12:12 PM Willy Tarreau  wrote:
> > If so then by not setting it the kernel should choose the default
> > value, which according to:
> > 
> > > sysctl net.ipv4.tcp_wmem
> > net.ipv4.tcp_wmem = 409616384   4194304
> > 
> > , should be 16384.
>
> No, it *starts* at 16384 then grows up to the configured limit depending
> on the ability to do so without losses and the available memory.


OK.  The Linux man-page eludes this part...  Good to know.  :)




> > Anyway, why am I trying to configure the sending buffer size:  if I
> > have large downloads and I have (some) slow clients, and as a
> > consequence HAProxy times out waiting for the kernel buffer to clear.
>
> Thus you might have very short timeouts! Usually it's not supposed to
> be an issue.

I wouldn't say they are "small":

timeout server 60s
timeout server-fin 6s
timeout client 30s
timeout client-fin 6s
timeout tunnel 180s
timeout connect 6s
timeout queue 30s
timeout check 6s
timeout tarpit 30s


As seen the timeout which I believe is the culprit is the `timeout
client 30s` which I guess is quite enough.


> > However if I configure the buffer size small enough it seems HAProxy
> > is "kept bussy" and nothing breaks.
>
> I see but then maybe you should simply lower the tcp_wmem max value a
> little bit, or increase your timeout ?

I'll try to experiment with `tcp_wmem max` as you've suggested.


> > Thus, is there a way to have both OK bandwidth for normal clients, and
> > not timeout for slow clients?
>
> That's exactly the role of the TCP stack. It measures RTT and losses and
> adjusts the send window accordingly. You must definitely let the TCP
> stack play its role there, you'll have much less problems. Even if you
> keep 4 MB as the max send window, for a 1 Mbps client that's rougly 40
> seconds of transfer. You can deal with this using much larger timeouts
> (1 or 2 minutes), and configure the tcp-ut value on the bind line to
> get rid of clients which do not ACK the data they're being sent at the
> TCP level.

I initially let the TCP "do its thing", but it got me into trouble
with poor wireless clients...

I'll also give `tcp-ut` as suggested.

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 11:41 AM Ciprian Dorin Craciun
 wrote:
> > - tune.sndbuf.client 16384 allows you to have 16384 bytes "on-the-fly", 
> > meaning unacknowlegded. 16384 / 0.16 sec = roughly 128 KB/s
> > - do the math with your value of 131072 and you will have get your ~800 
> > KB/s.


However, something bothers me...  Setting `tune.sndbuf.client`, is
used only to call `setsockopt (SO_SNDBUF)`, right?  It is not used by
HAProxy for any internal buffer size?

If so then by not setting it the kernel should choose the default
value, which according to:

> sysctl net.ipv4.tcp_wmem
net.ipv4.tcp_wmem = 409616384   4194304

, should be 16384.

Looking with `netstat` at the `Recv-Q` column, it seems that with no
`tune` setting the value even goes up to 5 MB.
However setting the `tune` parameter it always goes up to around 20 KB.




Anyway, why am I trying to configure the sending buffer size:  if I
have large downloads and I have (some) slow clients, and as a
consequence HAProxy times out waiting for the kernel buffer to clear.
However if I configure the buffer size small enough it seems HAProxy
is "kept bussy" and nothing breaks.

Thus, is there a way to have both OK bandwidth for normal clients, and
not timeout for slow clients?

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 11:33 AM Mathias Weiersmüller
 wrote:
> Sorry for the extremly brief answer:
> - you mentioned you have 160 ms latency.

Yes, I have mentioned this because I've read somewhere (not
remembering now where), that the `SO_SNDBUF` socket option also
impacts the TCP window size.


> - tune.sndbuf.client 16384 allows you to have 16384 bytes "on-the-fly", 
> meaning unacknowlegded. 16384 / 0.16 sec = roughly 128 KB/s
> - do the math with your value of 131072 and you will have get your ~800 KB/s.
> - no hidden voodoo happening here: read about BDP (Bandwidth Delay Product)

Please don't get me wrong:  I didn't imply any "voodoo".  :)

When I asked if there is some "hidden" consequence I didn't meant it
as "magic", but as a question for what other (unknown to me)
consequences there are.

And it seems that the `tune.sndbuf.client` also limits the TCP window size.

So my question is how can I (if at all possible) configure the buffer
size witout "breaking" the TCP window size?

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 10:35 AM Willy Tarreau  wrote:
> Note that these are not fragments but segments. And as Matti suggested,
> it's indeed due to GSO, you're seeing two TCP frames sent at once through
> the stack, and they will be segmented by the NIC.

I have disabled all offloading features:

tcp-segmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off


Now I see "as expected" Ethernet frames with `tcpdump` / `Wireshark`.
(There is indeed however a bump in kernel CPU usage.)


However the bandwidth behaviour is exactly the same:
* no `tune.sndbuf.client`, bandwidth goes up to 11 MB/s for a large download;
* with `tune.sndbuf.client 16384` it goes up to ~110 KB/s;
* with `tune.sndbuf.client 131072` it goes up to ~800 KB/s;
* with `tune.sndbuf.client 262144` it goes up to ~1400 KB/s;
(These are bandwidths obtained after the TCP window has "settled".)

It seems there is a liniar correlation between that tune parameter and
the bandwidth.


However due to the fact that I get the same behaviour both with and
without offloading, I wonder if there isn't somehow a "hidden"
consequence of setting this `tune.sndbuf.client` parameter?

Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 10:35 AM Willy Tarreau  wrote:
> On Sun, Sep 30, 2018 at 10:20:06AM +0300, Ciprian Dorin Craciun wrote:
> > I was just trying to replicate the issue I've seen yesterday, and for
> > a moment (in initial tests) I was able to.  However on repeated tests
> > it seems that the `tune.rcvbuf.*` (and related) have no impact, as I
> > constantly see TCP fragments (around 2842 bytes Ethernet frames).
>
> Note that these are not fragments but segments. And as Matti suggested,
> it's indeed due to GSO, you're seeing two TCP frames sent at once through
> the stack, and they will be segmented by the NIC.


[Just as info.]

So it seems I was able to reproduce the bandwith issue by only toying
with `tune.sndbuf.client`:
* with no value, downloading an 8 MB file I get decent bandwidth
around 4 MB/s;  (for larger files I even get up to 10 MB/s);  (a
typical Ethernet frame length as reported by `tcpdump` is around 59 KB
towards the end of the transfer;)
* with that tune parameter set to 128 KB, I get around 1 MB/s;  (a
typical Ethernet frame length is around 4 KB;)
* with that tune parameter set to 16 KB, I get around 100 KB/s;  (a
typical Ethernet frame lengh is around 2KB;)


By "typical Ethernet frame length" I meen a packet as reported by
`tcpdump` and viewed in Wrieshark looks like this (for the first one):

Frame 1078: 59750 bytes on wire (478000 bits), 59750 bytes captured
(478000 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Sep 30, 2018 10:26:58.667739000 EEST
[Time shift for this packet: 0.0 seconds]
Epoch Time: 1538292418.667739000 seconds
[Time delta from previous captured frame: 0.18000 seconds]
[Time delta from previous displayed frame: 0.18000 seconds]
[Time since reference or first frame: 1.901135000 seconds]
Frame Number: 1078
Frame Length: 59750 bytes (478000 bits)
Capture Length: 59750 bytes (478000 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:ethertype:ip:tcp:ssl:ssl]
[Coloring Rule Name: TCP]
[Coloring Rule String: tcp]
Ethernet II, Src: f2:3c:91:9f:51:b8 (f2:3c:91:9f:51:b8), Dst:
Cisco_9f:f0:0a (00:00:0c:9f:f0:0a)
Destination: Cisco_9f:f0:0a (00:00:0c:9f:f0:0a)
Source: f2:3c:91:9f:51:b8 (f2:3c:91:9f:51:b8)
Type: IPv4 (0x0800)
Internet Protocol Version 4, Src: XXX.XXX.XXX.XXX, Dst: XXX.XXX.XXX.XXX
0100  = Version: 4
 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
 00.. = Differentiated Services Codepoint: Default (0)
 ..00 = Explicit Congestion Notification: Not ECN-Capable
Transport (0)
Total Length: 59736
Identification: 0x8d7a (36218)
Flags: 0x4000, Don't fragment
0...    = Reserved bit: Not set
.1..    = Don't fragment: Set
..0.    = More fragments: Not set
...0    = Fragment offset: 0
Time to live: 64
Protocol: TCP (6)
Header checksum: 0x3054 [validation disabled]
[Header checksum status: Unverified]
Source: XXX.XXX.XXX.XXX
Destination: XXX.XXX.XXX.XXX
Transmission Control Protocol, Src Port: 443, Dst Port: 38150, Seq:
8271805, Ack: 471, Len: 59684
Source Port: 443
Destination Port: 38150
[Stream index: 0]
[TCP Segment Len: 59684]
Sequence number: 8271805(relative sequence number)
[Next sequence number: 8331489(relative sequence number)]
Acknowledgment number: 471(relative ack number)
1000  = Header Length: 32 bytes (8)
Flags: 0x010 (ACK)
Window size value: 234
[Calculated window size: 29952]
[Window size scaling factor: 128]
Checksum: 0x7d1c [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
[Timestamps]
TCP payload (59684 bytes)
TCP segment data (16095 bytes)
TCP segment data (10779 bytes)
[2 Reassembled TCP Segments (16405 bytes): #1073(310), #1078(16095)]
[Frame: 1073, payload: 0-309 (310 bytes)]
[Frame: 1078, payload: 310-16404 (16095 bytes)]
[Segment count: 2]
[Reassembled TCP length: 16405]
Secure Sockets Layer
Secure Sockets Layer


I'll try to disable offloading and see what happens.

I forgot to say that this is a paravirtualized VM running on Linode in
their Dallas datacenter.

Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 10:06 AM Mathias Weiersmüller
 wrote:
> I am pretty sure you have TCP segmentation offload enabled. The TCP/IP stack 
> therefore sends bigger-than-allowed TCP segments towards the NIC who in turn 
> takes care about the proper segmentation.

I was just trying to replicate the issue I've seen yesterday, and for
a moment (in initial tests) I was able to.  However on repeated tests
it seems that the `tune.rcvbuf.*` (and related) have no impact, as I
constantly see TCP fragments (around 2842 bytes Ethernet frames).


> You want to check the output of "ethtool -k eth0" and the values of:
> tcp-segmentation-offload
> generic-segmentation-offload

The output of `ethtool -k eth0` is bellow:

tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on


Thanks,
Ciprian.

Re: Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

On Sun, Sep 30, 2018 at 9:08 AM Willy Tarreau  wrote:
> > I've played with `tune.rcvbuf.server`, `tune.sndbuf.server`,
> > `tune.rcvbuf.client`, and `tune.sndbuf.client` and explicitly set them
> > to various values ranging from 4k to 256k.  Unfortunately in all cases
> > it seems that this generates too large TCP packets (larger than the
> > advertised and agreed MSS in both direction), which in turn leads to
> > TCP fragmentation and reassembly.  (Both client and server are Linux
> > >4.10.  The protocol used was HTTP 1.1 over TLS 1.2.)
>
> No no no, I'm sorry but this is not possible at all. You will never find
> a single TCP stack doing this! I'm pretty sure there is an issue somewhere
> in your capture or analysis.
>
> [...]
>
> However, if the problem you're experiencing is only with the listening
> side, there's an "mss" parameter that you can set on your "bind" lines
> to enforce a lower MSS, it may be a workaround in your case. I'm
> personally using it at home to reduce the latency over ADSL ;-)


I am also extreemly sckeptical that this is HAProxy's fault, however
the only change needed to eliminate this issue was commenting-out
these tune arguments.  I have also explicitly set the `mss` parameter
to `1400`.

The catpure was taken directly on the server on public interface.

I'll try to make a fresh catpure to see if I can replicate this.


> > The resulting bandwidth was around 10 MB.
>
> Please use correct units when reporting issues, in order to reduce the
> confusion. "10 MB" is not a bandwidth but a size (10 megabytes). Most
> likely you want to mean 10 megabytes per second (10 MB/s). But maybe
> you even mean 10 megabits per second (10 Mb/s or 10 Mbps), which equals
> 1.25 MB/s.

:)  Sorry for that.  (Thats the otucome of writing emails at 3 AM
after 4 hours of pocking into a production system.)  I completely
agree with you about the MB/Mb consistency, and I always hate that
some providers still use MB to mean mega-bits, like it's 2000.  :)

Yes, I meant 10 mega-bytes / second.  Sory again.

Ciprian.

Do `tune.rcvbuf.server` and `tune.sndbuf.server` (and their `tune.*.client` equivalents) lead to TCP fragmentation?

[Ciprian Dorin Craciun]

Hello all!

I've played with `tune.rcvbuf.server`, `tune.sndbuf.server`,
`tune.rcvbuf.client`, and `tune.sndbuf.client` and explicitly set them
to various values ranging from 4k to 256k.  Unfortunately in all cases
it seems that this generates too large TCP packets (larger than the
advertised and agreed MSS in both direction), which in turn leads to
TCP fragmentation and reassembly.  (Both client and server are Linux
>4.10.  The protocol used was HTTP 1.1 over TLS 1.2.)

The resulting outcome was a bandwidth of about 100 KB (for a
client-server latency of 160ms).

The only setting that din't have this effect was not to set them.  The
resulting bandwidth was around 10 MB.

(I have tested the backend server without HAProxy, in fact two
different webservers, both with and without HAProxy, and I would
exclude them as the issue.)


Thus I was wondering if anyone encountered similar issues and how
they've fixed it.  (My guess is that it's more due to the Linux TCP
implementation stack than from HAProxy.)


As a sidenote is the following interpretation correct:
* `tune.*buf.server` refers to the TCP sockets that the frontend binds
to and listens for actual clients;
* `tune.*buf.client` refers to the TCP sockets that the backend
creates and connects to the actual servers;

Thanks,
Ciprian.

Re: Show: haproxy-auth-request - HTTP access control using subrequests

[Ciprian Dorin Craciun]

On Sun, Jan 21, 2018 at 4:17 PM, Tim Düsterhus  wrote:
>> Quick question though:  does this script actually block HAproxy's
>> event loop while waiting for the response from the backend server?
>
> haproxy's Socket class is documented to be non-blocking, as explained
> here: https://www.arpalert.org/haproxy-lua.html#h204 ("Non blocking
> design"). Most of my article focused on getting the Socket class of
> haproxy to work, instead of using the native Socket class of Lua for
> that reason.


OK, so it seems that "non-blocking" in the context of Lua for HAProxy
it is actually understood "asynchronous", in the sense that if / when
the call would block, the HAProxy event loop takes over and restarts
the Lua code only when the blocking operation can be retried without
blocking.

So for the purpose of developing in Lua, the code seems to be
"blocking" and "synchronous", but behind the scenes HAProxy controls
the Lua VM to transform the Lua code exeguhiot into and
"asynchronous".

Thanks,
Ciprian.

Re: Show: haproxy-auth-request - HTTP access control using subrequests

[Ciprian Dorin Craciun]

On Fri, Jan 19, 2018 at 9:23 PM, Tim Düsterhus  wrote:
> https://github.com/TimWolla/haproxy-auth-request
>
> This Lua script reimplements the idea behind nginx'
> ngx_http_auth_request_module in haproxy: It allows you to decide whether
> an HTTP request should be allowed or not based on the result of an HTTP
> request to a backend service.



Nice little tool!

Quick question though:  does this script actually block HAproxy's
event loop while waiting for the response from the backend server?

(I've read quickly through your article describing the script, but
failed to get a clear answer to this question.  Moreover looking at
the script, unless HAProxy does something "funky" with the Lua
interpreter to give the impression of a synchronous API, I would say
it does block the HAProxy thread that calls this script.)

Thanks,
Ciprian.

Re: HAProxy support for SOCKS4 as alternative to PROXY protocol?

[Ciprian Dorin Craciun]

On Sun, Oct 22, 2017 at 11:11 PM, Aleksandar Lazic  wrote:
> Currently the socks protocol is not implemented in haproxy.


I was hoping someone had a patch "hidden".  :)




> What flow do you have in mind?


I have a couple of use-cases in mind, like for example:

* SOCKS4 in the backend, would allow HAProxy to route all backend
traffic through a proper SOCKS4 proxy;  this might be used as a
poor-man variant of a tunnel, like for example via SSH;  (if one makes
HAProxy into a transparent proxy, it could even serve as a layer-7
firewall;)

* SOCKS4 in the frontend, would allow HAProxy to act like a SOCKS4
proxy, and apply for example HTTP routing and filtering;  (for example
one configures HAProxy as a SOCKS4 proxy in a browser;)

Basically it allows HAProxy to interoperate with other SOCKS4 proxies
like SSH or Tor.

Ciprian.

HAProxy support for SOCKS4 as alternative to PROXY protocol?

[Ciprian Dorin Craciun]

After fiddling with some ideas revolving around HAProxy as a
transparent proxy, I wondered if HAProxy supports the SOCKS4 protocol
as an alternative to its own PROXY protocol.  Looking at the 1.7
documentation and source code it seems it doesn't (yet).

Thus, given how simple the SOCKS4 (not SOCKS4a or SOCKS5) protocol is,
I wanted to ask if anyone has coded a small patch to support it?




If not, any pointers on how to implement it?

Looking through the source code, I assume cloning `conn_recv_proxy`
should do the trick if only the SOCKS4 protocol wouldn't require that
the server talks back to the client.  Thus should I assume that it
should also borrow code from `conn_si_send_proxy`?

(The SOCKS4 handshake implies the client speaks first with a couple of
bytes, variable length but clearly determined, and as a result the
server sends back a couple of other bytes, exact length this time;
afterwards it just implies piping data from one end to the other.)

Thanks,
Ciprian.

Re: Issues with redirects for POST requests with payload

[Ciprian Dorin Craciun]

On Tue, May 9, 2017 at 9:47 PM, Willy Tarreau  wrote:
> On Tue, May 09, 2017 at 02:54:45PM +0300, Jarno Huuskonen wrote:
>> My firefox(52.1 on linux) was able to send 128k file,
>> but 800k file results in connection reset. My chrome sent 16k file, but
>> fails (ERR_CONNECTION_RESET) on 17k file (few times even the 17k file
>> worked).


My particular request is 17575 octets in total (headers + body), and
as Jarno observed, it sometimes works Firefox / Chromium, but most
times Chromium trips over it.

The "non-determinism" of the issue is triggered by the fact if the
browser manages to push all its payload over the network before the
response from HAProxy is received.  (If the network latency would be
zero, and the receive window on HAProxy size would be extremly small,
this issue would happen every time.)

Based on a few `tcpdump` captures the issue can be described in terms
of TCP as this:
* the client sends its headers, and starts sending the request body;
* HAProxy receives the headers, determines it's a redirect;
* HAProxy writes the response status line and headers, and closes the
connection via a reset packet;
* meanwhile the client while still writing to the socket its payload,
the reset packet is received which puts the client into an error state
(because it tries to write to a closed socket);
* thus the client just errors out without reading the response;

Sometimes the reset packet is received after the client has finished
writing, thus the condition is not encountered any more.

(On explicit request over private email, I can provide a small tcpdump
capture displaying the behaviour described herein.)




> Hmmm that sounds bad, it looks like we've broken something again.


I don't think HAProxy was "broken" this time, as we are using HAProxy
1.6.11 since last November, and only recently (since two-three weeks
ago) we started encountering this issue without having any major
changes to either the HAProxy configuration or the web application
where we encountered this issue.

In fact I think the browsers "broke" something, as only with recent
variants of Chromium and Firefox we encounter this.


However, since there are far less few HAProxy deployments than
browsers (say 50.000 to 1 in our deployment), I think the easiest
component to fix is HAProxy, by making it "drain" the entire
connection before closing it.  (Although I have no idea how
complicated is to do this.)




> What status code are you facing there ? I remember we've had such
> an issue in the past where the server timeout could expire during
> a long upload because it was not being refreshed during the upload
> (it would thus result in a 504).


This is the point, there is no "error" situation, as HAProxy just
issues the redirect status line and headers and then resets the
connection.

Thanks,
Ciprian.

Re: Issues with redirects for POST requests with payload

[Ciprian Dorin Craciun]

On Sat, May 6, 2017 at 11:13 AM, Ciprian Dorin Craciun
<ciprian.crac...@gmail.com> wrote:
> Hello all!
>
> In last weeks I've started encountering a problem, that for our
> particular use-case is seriously breaking some of our sites, namely:
>
> * a client makes a POST request which has a "largish" payload, one
> that does not manage to "push" it through before HAProxy has a chance
> to respond,
> * if HAProxy is configured to redirect such a request (like for
> example upgrading HTTP to HTTPS),
> * then HAProxy responds with the redirect, and closes the connection;
> but the client has not yet been able to push its POST body and
> receives a write error, and thus it aborts without trying to read the
> response from HAProxy;
>
>
> One can easily reproduce this with:
> (
>   printf -- 'POST /invalid HTTP/1.1\r\nHost: invalid.example.com\r\n\r\n'
>   dd if=/dev/urandom bs=1024 count=4 | base64
> ) \
> | socat -d -d -v tcp:127.0.0.1:80,sndbuf=16 stdio
>
> , which results in a connection reset, as `socat` is trying to push
> data to a closed socket.
>
> (Via private email I can give an actual `tcpdump` capture with production 
> data.)
>
>
>
>
> Unfortunately this issue doesn't impact a "random" client but recent
> versions of Firefox and Chrome, which just display a "connection
> reset" kind of message to the users.
>
>
> I've tried searching for a similar problem, and found this:
>
>   
> http://haproxy.formilux.narkive.com/9xhXJk4f/redirecting-on-a-large-post-without-reading-it-entirely
>   http://haproxy.formilux.narkive.com/gYztlqms/fwd-302-to-502-error
>
>
> But it's not clear to me if these issues were fixed since almost 8
> years ago, or how should I proceed in solving this issue myself.  (I'm
> open to applying patches and re-compiling HAProxy.)
>
>
> Increasing `tune.bufsize` to 128k doesn't seem to help either.
>
> (I am using HAProxy 1.6.11.)




Just wanted to "ping" this thread, as perhaps sending the original
email on weekend got it "forgotten".  :)

Or perhaps nobody has hit this issue in production (yet)?

Thanks,
Ciprian.

Re: Issues with redirects for POST requests with payload

[Ciprian Dorin Craciun]

Forgot to mention that it involves HAProxy 1.6.11.

Ciprian.


On Sat, May 6, 2017 at 11:13 AM, Ciprian Dorin Craciun
<ciprian.crac...@gmail.com> wrote:
> Hello all!
>
> In last weeks I've started encountering a problem, that for our
> particular use-case is seriously breaking some of our sites, namely:
>
> * a client makes a POST request which has a "largish" payload, one
> that does not manage to "push" it through before HAProxy has a chance
> to respond,
> * if HAProxy is configured to redirect such a request (like for
> example upgrading HTTP to HTTPS),
> * then HAProxy responds with the redirect, and closes the connection;
> but the client has not yet been able to push its POST body and
> receives a write error, and thus it aborts without trying to read the
> response from HAProxy;
>
>
> One can easily reproduce this with:
> (
>   printf -- 'POST /invalid HTTP/1.1\r\nHost: invalid.example.com\r\n\r\n'
>   dd if=/dev/urandom bs=1024 count=4 | base64
> ) \
> | socat -d -d -v tcp:127.0.0.1:80,sndbuf=16 stdio
>
> , which results in a connection reset, as `socat` is trying to push
> data to a closed socket.
>
> (Via private email I can give an actual `tcpdump` capture with production 
> data.)
>
>
>
>
> Unfortunately this issue doesn't impact a "random" client but recent
> versions of Firefox and Chrome, which just display a "connection
> reset" kind of message to the users.
>
>
> I've tried searching for a similar problem, and found this:
>
>   
> http://haproxy.formilux.narkive.com/9xhXJk4f/redirecting-on-a-large-post-without-reading-it-entirely
>   http://haproxy.formilux.narkive.com/gYztlqms/fwd-302-to-502-error
>
>
> But it's not clear to me if these issues were fixed since almost 8
> years ago, or how should I proceed in solving this issue myself.  (I'm
> open to applying patches and re-compiling HAProxy.)
>
>
> Increasing `tune.bufsize` to 128k doesn't seem to help either.
>
>
> Thanks,
> Ciprian.

Issues with redirects for POST requests with payload

[Ciprian Dorin Craciun]

Hello all!

In last weeks I've started encountering a problem, that for our
particular use-case is seriously breaking some of our sites, namely:

* a client makes a POST request which has a "largish" payload, one
that does not manage to "push" it through before HAProxy has a chance
to respond,
* if HAProxy is configured to redirect such a request (like for
example upgrading HTTP to HTTPS),
* then HAProxy responds with the redirect, and closes the connection;
but the client has not yet been able to push its POST body and
receives a write error, and thus it aborts without trying to read the
response from HAProxy;


One can easily reproduce this with:
(
  printf -- 'POST /invalid HTTP/1.1\r\nHost: invalid.example.com\r\n\r\n'
  dd if=/dev/urandom bs=1024 count=4 | base64
) \
| socat -d -d -v tcp:127.0.0.1:80,sndbuf=16 stdio

, which results in a connection reset, as `socat` is trying to push
data to a closed socket.

(Via private email I can give an actual `tcpdump` capture with production data.)




Unfortunately this issue doesn't impact a "random" client but recent
versions of Firefox and Chrome, which just display a "connection
reset" kind of message to the users.


I've tried searching for a similar problem, and found this:

  
http://haproxy.formilux.narkive.com/9xhXJk4f/redirecting-on-a-large-post-without-reading-it-entirely
  http://haproxy.formilux.narkive.com/gYztlqms/fwd-302-to-502-error


But it's not clear to me if these issues were fixed since almost 8
years ago, or how should I proceed in solving this issue myself.  (I'm
open to applying patches and re-compiling HAProxy.)


Increasing `tune.bufsize` to 128k doesn't seem to help either.


Thanks,
Ciprian.

Re: HAProxy for Centos 7

[Ciprian Dorin Craciun]

On Tue, Apr 25, 2017 at 6:39 PM, Manojkumar Gupta
 wrote:
> Please can you share the binary compiled copy of HAProxy for Centos 7, v1.6
> or above.
>
> I don’t have root access and when using generic platform there are multiple
> packages that I need as dependency and without root its hard to get it
> compiled from source.




I have v1.6.11 compiled for CentOS 7 (64 bit) available at the link bellow:

  
http://download.opensuse.org/repositories/home:/cipriancraciun:/centos-7-extras/CentOS_7/x86_64/haproxy-1.6.11-5.2.x86_64.rpm


It was built by using OpenSUSE's build platform (available also for
CentOS) and the builds details are found bellow:

  
https://build.opensuse.org/package/show?project=home%3Acipriancraciun%3Acentos-7-extras=haproxy


The package is built based on HAProxy's "vanilla" sources (i.e.
original sources), with a simple patch which was backported from 1.7.

But as Andrew observed, without root access, you won't be able to run
it as a service.  Moreover you'll have to extract the RPM manually.


Alternatively if this fails, you could also try download a version of
HAProxy I've extracted from Ubuntu's packages, available at this link:

  https://data.volution.ro/ciprian/a00ddf65afc4fac2069121305c6401b6/

You'll need `haproxy.elf-...` and `liblua5.3.so-...`, which you'll
have to rename without that suffix (which denotes their sources), and
when you run haproxy you'll have to also configure `LD_LIBRARY_PATH`
to the folder where you've placed `liblua5.3.so`.

(It is quite possible that you'll need to do a similar trick with the
CentOS variant also.)

Hope this helps,
Ciprian.

Re: unique-id-header and req.hdr

[Ciprian Dorin Craciun]

On Fri, Jan 27, 2017 at 10:24 PM, Patrick Hemmer
 wrote:
> Something that might satisfy both requests, why not just append to the
> existing request-id?
>
> unique-id-format %[req.hdr(X-Request-ID)],%{+X}o\
> %ci:%cp_%fi:%fp_%Ts_%rt:%pid
>
> This does result in a leading comma if X-Request-ID is unset. If that's
> unpleasant, you could do something like write tiny LUA sample converter to
> append a comma if the value is not empty.


However, just setting the `unique-id-format` is not enough, as we
should also send that ID to the backend, thus there is a need of
`http-request set-header X-Request-Id %[unique-id] if !...`.  (By not
using the `http-request`, we do get the ID from the header in the log,
but not to the backend.)

But now -- I can't say with certainty, but I remember trying various
variants -- I think the evaluation order of `unique-id-format` is
after all the `http-request` rules, thus the header will always be
empty (if not explicitly set in the request), although in the log we
would have a correct ID.


(This is why I settled with a less optimal solution of having two
headers, but with identical values, and working correctly in all
instances.)

Ciprian.

Re: unique-id-header and req.hdr

[Ciprian Dorin Craciun]

On Fri, Jan 27, 2017 at 9:01 PM, Cyril Bonté  wrote:
> Instead of using "unique-id-header" and temporary headers, you can use the
> "unique-id" fetch sample [1] :
>
> frontend public
> bind *:80
> unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
> default_backend ui
>
> backend ui
> http-request set-header X-Request-Id %[unique-id] unless {
> req.hdr(X-Request-Id) -m found }


Indeed this might be one version of ensuring that a `X-Request-Id`
exists, however it doesn't serve a second purpose (which I found most
important), namely of matching the HAProxy logs with the logs of a
downstream server (or an upstream client).

For example, suppose we have a client (an API library), which sets
`X-Request-Id` header to a random value, logs it, and makes the
request.  This request reaches HAProxy, which because it uses a custom
`unique-id-format` which disregards the `X-Request-Id` header, will
log a completely different request id, but pass the "original" header
to the downstream server.  The downstream server receives the request,
logs the request ID and gives back the response.

Now if we want to match all the three logs we can't.  The client and
server logs are in sync, but the HAProxy logs uses its own custom
request ID.


By using the variant I proposed (setting the `unique-id-format` to the
actual value of the `X-Request-Id` header), we can match all the three
logs.

Ciprian.


P.S.:  Obviously we can explicitly log the `X-Request-Id` header via a
capture, but it isn't as explicit (or easy to identify) as setting the
unique-id to the header value.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Wed, Jan 11, 2017 at 8:59 PM, Willy Tarreau  wrote:
>> [I can't speak with much confidence as this is the first time I see
>> the HAProxy code, but...]
>>
>>
>> >From what I see the main culprit for the connection close is the code:
>>
>>  [starting with line 4225 in `proto_http.c`] 
>> if (*location == '/' &&
>> (req->flags & HTTP_MSGF_XFER_LEN) &&
>> ((!(req->flags & HTTP_MSGF_TE_CHNK) && !req->body_len) ||
>> (req->msg_state == HTTP_MSG_DONE)) &&
>> ((txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_SCL ||
>>  (txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_KAL)) {
>> /* keep-alive possible */
>> 
>>
>>
>> Which might be rewrites just as:
>>
>>  [starting with line 4225 in `proto_http.c`] 
>> if (
>>(req->flags & HTTP_MSGF_XFER_LEN) &&
>> ((!(req->flags & HTTP_MSGF_TE_CHNK) && !req->body_len) ||
>> (req->msg_state == HTTP_MSG_DONE)) &&
>> ((txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_SCL ||
>>  (txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_KAL)) {
>> /* keep-alive possible */
>> 
>>
>>
>> I.e., just remove `*location == '/' &&`, and I assume not much will be
>> impacted, thus I guess no regressions should stem from this
>> correction.
>
> Absolutely. Feel free to provide a patch doing this (please check
> CONTRIBUTING for the format, the commit message and the subject line),
> tag it BUG/MINOR and I'll merge it.


No patch yet :) but I just wanted to confirm that this small change
seems to work just fine in production for the last two weeks.
(Granted I didn't make a thorough analysis of the traffic, but so far
no one complained, and the traffic isn't quite small.)

Perhaps later this week or next week I'll be back with a patch.

Ciprian.

Re: unique-id-header and req.hdr

[Ciprian Dorin Craciun]

On Fri, Jan 27, 2017 at 12:10 AM, sendmaildevnull
 wrote:
> I'm trying generate a unique-id-header only if one is not already provided
> in the request. If I provide the header in my request to haproxy I end up
> with duplicate headers, one with auto generated header and another with the
> user provided header. I attempted to use the technique mentioned here
> (http://discourse.haproxy.org/t/unique-id-adding-only-if-header-not-present/67/2)
> and listed below but it is not working for me. Basically, I'm unable to
> check/get value for unique-id-header (e.g. req.hdr(TMP-X-Request-Id)). Any
> ideas?


I have struggled with this one also, and I reached a simple (but
suboptimal) solution:
(A)  Set an `X-HA-Request-Id` header if it doesn't already exist;
(B)  Configure `unique-id-*` as follows:
  unique-id-format %[req.hdr(X-HA-Request-Id)]
  unique-id-header X-HA-Request-Id-2

(I.e. in the end there are two exact headers sent to the backend, with
equal values, either the original value of the `X-HA-Request-Id`
header, or a random one.)


BTW, I always set the request header (if it doesn't already hold a
value) as a random token which would yield 128 bits of entropy, and
doesn't provide any personal identifiable information.

  http-request set-header X-HA-Request-Id
%[rand(4294967295)].%[rand(4294967295)].%[rand(4294967295)].%[rand(4294967295)]
if ...


Hope it helps,
Ciprian.


P.S.:  It sure would be helpful to have some cryptographic
transformation functions like the SHA2 family, HMAC-based functions,
and a cryptographic random function.  :)

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Wed, Jan 11, 2017 at 8:27 PM, Lukas Tribus  wrote:
> But if we do remove those conditions, I guess we break a number of "old
> assumptions"
> and we will hit new code paths, so there is a potential for bugs :)


[I can't speak with much confidence as this is the first time I see
the HAProxy code, but...]


>From what I see the main culprit for the connection close is the code:
X-Bogosity: Ham, tests=bogofilter, spamicity=0.00, version=1.2.4

 [starting with line 4225 in `proto_http.c`] 
if (*location == '/' &&
(req->flags & HTTP_MSGF_XFER_LEN) &&
((!(req->flags & HTTP_MSGF_TE_CHNK) && !req->body_len) ||
(req->msg_state == HTTP_MSG_DONE)) &&
((txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_SCL ||
 (txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_KAL)) {
/* keep-alive possible */



Which might be rewrites just as:

 [starting with line 4225 in `proto_http.c`] 
if (
   (req->flags & HTTP_MSGF_XFER_LEN) &&
((!(req->flags & HTTP_MSGF_TE_CHNK) && !req->body_len) ||
(req->msg_state == HTTP_MSG_DONE)) &&
((txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_SCL ||
 (txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_KAL)) {
/* keep-alive possible */



I.e., just remove `*location == '/' &&`, and I assume not much will be
impacted, thus I guess no regressions should stem from this
correction.

Ciprian.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Wed, Jan 11, 2017 at 1:07 PM, Willy Tarreau  wrote:
> This one is obsolete, the format of the Location header field is
> described here :
>
>https://tools.ietf.org/html/rfc7231#page-68
>
> and it uses this format :
>
>https://tools.ietf.org/html/rfc3986#section-4.2
>
> As you can see, "//" authority path, path-absolute etc... are permitted.
>
> However while I understand it could solve your issue, I think it's an
> ugly trick in that you use it only to work around an undesirable haproxy
> behaviour. We'd rather ensure haproxy doesn't behave this way if not
> needed. Of course it could be a short-term workaround saving your from
> modifying the code.


Quick reply:  this would work only if we assume that we want to keep
the traffic on the same "transport", namely TLS or non-TLS.  But these
days, when TLS is starting to become widely deployed, I assume most
redirects are on the form of `https://canonical-domain.com`.

Ciprian.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Wed, Jan 11, 2017 at 1:02 PM, Willy Tarreau  wrote:
> Yes that's indeed the place. The reason for this behaviour is that
> (normally) when the client gets a redirect to another host or scheme,
> it will no longer use the current connection and will have to close
> it. The typical use case is when redirecting from HTTP to HTTPS, you
> definitely don't want to keep the connection active on port 80 since
> it will not be used anymore.


I guessed this was the main reason, and in general I think it is the
right behaviour, especially if this redirection is done only once when
the browser first contacts the server, and assuming that the resources
in the underlying site don't need to be redirected also.

However (see below)...




> But maybe your use case is different and that's something we should
> revisit or make a bit more configurable ?


However, in my particular use-case we are migrating quite a handful of
sites to be routed through HAProxy (around 100-200 different sites,
each with its own domain and aliases).  At the same time trying to
apply a policy of non-www prefixed domains (previously there wasn't
much consistency), redirecting aliases to the "canonical" domain,
etc.;  plus everything now should go over HTTPS.

Thus quite a handful of redirects of the form `https://subdomain.domain.com`.


Unfortunately a lot of these sites have hard-coded resources with the
`www` alternative domain and HTTP-only.  Therefore at least until we
rewrite those (which given their low-value might be delayed forever),
we'll end up with a lot of redirects (10-20 redirects first page
view), and because the close behaviour of HAProxy, we end up with a
lot of short-lived connections.  (This is not so much a load issue, as
it is a page load latency issue due to the TCP connection
establishment overhead.)

We have solved the HTTPS automatic upgrade by the browser by using the
`Content-Security-Policy: upgrade-insecure-requests` response header,
but nothing can be done for the non-`www` redirects.


What would help me in this situation:
* (best solution)  a "knob" to disable the close behaviour;
* (most likely solution)  I'll patch and build HAProxy myself to
remove that expression from that condition;
* (workarounds)  I described previously (an untested) solution
involving a secondary front-end / backend to handle the redirects;
alternatively I could write a small Go application that will issue the
redirects, but this would add another "moving part" which might
decrease the reliability;


Thanks,
Ciprian.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Tue, Jan 10, 2017 at 9:36 AM, Cyril Bonté  wrote:
> This is because haproxy behaves differently depending on the the Location
> URL :
> - beginning with /, it will allow HTTP keep-alived connections (Location:
> /redir/foo)
> - otherwise it unconditionnally won't, and there's no option to change this
> (Location: http://mysite/redir)




Quickly looking through the latest HAProxy code on the 1.7 branch, I
think I found the piece of code which is responsible for closing the
connections on redirects that do not start with `/`:


 [starting with line 4225] 
if (*location == '/' &&
(req->flags & HTTP_MSGF_XFER_LEN) &&
((!(req->flags & HTTP_MSGF_TE_CHNK) && !req->body_len) ||
(req->msg_state == HTTP_MSG_DONE)) &&
((txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_SCL ||
 (txn->flags & TX_CON_WANT_MSK) == TX_CON_WANT_KAL)) {
/* keep-alive possible */



(This also explains why protocol-relative URL's (i.e.
`//some-domain.com/...`) work.)

Thus if I were to patch my HAProxy build to eliminate the just the
`*location == '/'` expression, I would obtain keep-alive regardless of
where I redirect the browser.




Could someone more experienced in the HAProxy source code advise if
this is the only place I should give a look?

Thanks,
Ciprian.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Tue, Jan 10, 2017 at 11:00 PM, Bryan Talbot
 wrote:
> Whatever the reason for forcing the connection closed -- it only closes when 
> the scheme changes. Redirecting to a different host or port when using a 
> “scheme less” URI allows the connection to be kept open.
>
>
> listen http
> bind :8000
> http-request redirect location //127.0.0.2:8001/redir
>
>
[...]
>
>
> Maybe that will be useful to Ciprian to make the redirect to a new hostname 
> but keep the connection to the old host open if that’s what is needed.



Thanks for the suggestion, however there are two issues with this proposal:


(A)  I'm not sure how would browsers in general behave when faced with
a schema-less URL in the `Location` header.

A quick survey of the internet (i.e. the top search on Google points
to a StackOverflow question) suggests that browsers would handle these
"protocol-relative" URL's just fine when loading assets;  however the
HTTP standard seems to mandate "absolute" URL's with regard of the
`Location` header:

  https://tools.ietf.org/html/rfc2616#section-14.30


(B)  Moreover in my particular case it still doesn't fully solve the
issue, mainly because the organization I'm working for is migrating
quite a handful of sites and is applying two major changes:  moving to
TLS (via LetsEncrypt issued certificates) and making sure all sites
follow the policy of `www`-less prefix.  Thus the majority of our
redirects are of the form `https://subdomain.domain.com`.


Thanks,
Ciprian.

Re: HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

On Tue, Jan 10, 2017 at 9:36 AM, Cyril Bonté  wrote:
> This is because haproxy behaves differently depending on the the Location
> URL :
> - beginning with /, it will allow HTTP keep-alived connections (Location:
> /redir/foo)
> - otherwise it unconditionnally won't, and there's no option to change this
> (Location: http://mysite/redir)


First of all, sorry for providing an incomplete email by not
specifying the HAProxy version (which is 1.6.10), and a configuration
example.

Also thanks Cyril for pointing out to this behaviour, as in fact this
is exactly my use-case, redirecting a `www` site to its prefix-less
name.  (And perhaps towards `HTTPS`, but this could be solved by
specifying the `Content-Security-Policy:  ugrade-insecure-requests`.)
(Also my redirects are 307, i.e. temporary and instructing the browser
to use the same HTTP method.)


Might I ask why this behaviour?

I assume that it was presumed that these redirects will happen only
once for a particular client, and keeping alive that connection would
be useless, as the client would initiate a different one, given that
the host in the URL changed.




Perhaps can someone point me to the piece of code responsible for this
behaviour?  (I could write a patch and apply it only to my particular
deployment.)


However, might the following trick solve the problem:

* I create a frontend dedicated to make the redirects;
* I create a backend which uses as server the previously defined
frontend, and specifies the `http-server-close` behaviour;
* in the "main" frontend (where the redirects used to be) I just
forward (via `use-server`) the requests needed to be redirected to the
newly introduced backend;

Would this solve the connection closing on redirects?

(I'll have to test this sometime these days, and report back.)


Thanks,
Ciprian.

HTTP redirects while still allowing keep-alive

[Ciprian Dorin Craciun]

Quick question:  how can I configure HAProxy to redirect (via
`http-request redirect ...`) without HAProxy sending the `Connection:
close` header, thus still allowing keep-alive on this connection.

My use-case is the following:  I have a stubborn server that insists
on pointing to the "wrong" resource URL's, thus on a page load, I get
a storm of redirects, each with a different connection (due to the
`Connection: close` reply header).


I tried to skim the documentation and search the internet (and the
mailing list archives), but no such topic popped-up, thus I have the
feeling this is quite impossible as of now...

Thanks,
Ciprian.