Re: http-response set-header is unreliable

[Willy Tarreau]

Hi Tim,

On Tue, May 01, 2018 at 01:57:06AM +0200, Tim Düsterhus wrote:
> Willy,
> 
> Am 30.04.2018 um 23:06 schrieb Willy Tarreau:
> >> Anything I could do to help investigate this? I can apply patches with
> >> additional logging or I can send you the unredacted configuration in
> >> private if that would help.
> > 
> > OK, it's just that for now I can't propose anything, I'm context-switching
> > far too much between many many different stuff :-(
> 
> I fiddled a bit by myself: I found that the http_header_add_tail2 in
> http_res_get_intercept_rule, case ACT_HTTP_SET_HDR (proto_http.c:2934)
> returns -1 (indicating failure). This return value is being ignored.

OK, thanks for confirming this!

> Adding the following debug output in http_header_add_tail2:
> 
>   printf("Buffer Information: size(%d) i(%d)\n", msg->chn->buf->size,
> msg->chn->buf->i);
>   buffer_dump(stderr, msg->chn->buf, 0, msg->chn->buf->i);
>   printf("Add tail: %d: %s\n", bytes, text);
> 
> Yields:
> 
> Buffer Information: size(16384) i(16367)
> Dumping buffer 0x5654d8047960
> data=0x5654d8047974 o=0 i=16367 p=0x5654d8047974
> relative:   p=0x
> Dumping contents from byte 0 to byte 16367
> *snip*
> Add tail: 0: X-XSS-Protection: 1; mode=block
> 
> in failure cases. You can see that the buffer is filled to the top.
> 
> Decoding the dumped buffer reveals that the buffer contains
> significantly more of the HTTP *body* in failure cases than in success
> cases.

Yes, that's indeed expected.

> It might make sense to enlarge the rewrite buffer reservation by
> default.

We used to have this a long time ago, the maxrewrite value used to
default to half the buffer size. But it caused too many requests to
be rejected and became absurd when users configure large buffers.

> I can absolutely imagine that people put in a ton of
> information in the times of Content-Security-Policy, Expect-{CT,Staple}
> and whatnot.

Yes but even once you put this, you hardly reach 1 kB. Most requests are
way below 1 kB headers, at least for performance reasons.

> I don't know what issues a too-small buffer for non-rewrites would
> cause.

This limits the maximum size of accepted messages (request or response),
which results in *some* requests to be rejected due to (for example) very
large cookies. Overall, I'd say that the number of problem reports due to
rejected requests or reponses due to their size has dropped to almost zero
since we've limited the value to 1 kB. A long time ago it used to be a very
common report.

> Clearly the body must be able to span multiple buffers already,
> otherwise I would not be able to send bodies greater than 16kB.
> Will it need to allocate more buffers to do the same work, because every
> single one is smaller?

No, the body is simply streamed, not stored. If however you need to analyse
it (eg for a WAF) then you need to configure a larger buffer so that more
of the body can fit.

To give you an idea about how things currently work, when we perform an
recvfrom() call, we decide whether we read up to bufsize or up to
(bufsize-maxrewrite) depending on whether we are forwarding data or
still analysing the message. Thus when we receive a POST, we're not yet
forwarding, so up to 15 kB of headers+body are placed into the buffer,
leaving 1 kB available to add headers.

After the new native internal HTTP representation is implemented, I think
that the need for the maxrewrite will disappear, at the expense of using
one buffer for the headers and another one for the body, but we'll see.

Anyway we need to address the lack of error checking, and I really predict
some breakage here :-/

Willy

Re: http-response set-header is unreliable

[Tim Düsterhus]

Willy,

Am 30.04.2018 um 23:06 schrieb Willy Tarreau:
>> Anything I could do to help investigate this? I can apply patches with
>> additional logging or I can send you the unredacted configuration in
>> private if that would help.
> 
> OK, it's just that for now I can't propose anything, I'm context-switching
> far too much between many many different stuff :-(

I fiddled a bit by myself: I found that the http_header_add_tail2 in
http_res_get_intercept_rule, case ACT_HTTP_SET_HDR (proto_http.c:2934)
returns -1 (indicating failure). This return value is being ignored.

Adding the following debug output in http_header_add_tail2:

printf("Buffer Information: size(%d) i(%d)\n", msg->chn->buf->size,
msg->chn->buf->i);
buffer_dump(stderr, msg->chn->buf, 0, msg->chn->buf->i);
printf("Add tail: %d: %s\n", bytes, text);

Yields:

Buffer Information: size(16384) i(16367)
Dumping buffer 0x5654d8047960
data=0x5654d8047974 o=0 i=16367 p=0x5654d8047974
relative:   p=0x
Dumping contents from byte 0 to byte 16367
*snip*
Add tail: 0: X-XSS-Protection: 1; mode=block

in failure cases. You can see that the buffer is filled to the top.

Decoding the dumped buffer reveals that the buffer contains
significantly more of the HTTP *body* in failure cases than in success
cases.

>> But that does not mean that the current
>> documented behaviour is a good behaviour.
> 
> Absolutely. What I'm much more worried about now is that a number of
> deployments may already accidently be relying on this problem and may
> discoverr 400s or 502s the hard way after we fix it :-(  So we'll have
> to be very careful about this and enumerate what works and what fails
> to figure if there's a real risk that people rely on this or not. If so
> we may need to add an extra option to relax addition in case of failure
> to preserve the current behaviour, which I'd personally hate.
> 

I've outlined above what exactly fails where in my case (i.e. missing
return value checking).

It might make sense to enlarge the rewrite buffer reservation by
default. I can absolutely imagine that people put in a ton of
information in the times of Content-Security-Policy, Expect-{CT,Staple}
and whatnot.
I don't know what issues a too-small buffer for non-rewrites would
cause. Clearly the body must be able to span multiple buffers already,
otherwise I would not be able to send bodies greater than 16kB.
Will it need to allocate more buffers to do the same work, because every
single one is smaller?

Best regards
Tim Düsterhus

Re: http-response set-header is unreliable

[Willy Tarreau]

On Mon, Apr 30, 2018 at 09:06:16PM +0200, Tim Düsterhus wrote:
> Am 30.04.2018 um 15:48 schrieb Willy Tarreau:
> >> And why does it affect two headers at once? If the length is right below
> >> the limit intuitively only the very last header should be affected.
> > 
> > I really don't know, maybe the rules are aborted during their processing.
> 
> Anything I could do to help investigate this? I can apply patches with
> additional logging or I can send you the unredacted configuration in
> private if that would help.

OK, it's just that for now I can't propose anything, I'm context-switching
far too much between many many different stuff :-(

> tune.bufsize is documented to send a 502:
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#3.2-tune.bufsize
> 
> tune.maxrewrite is documented to "prevent addition of headers":
> https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#3.2-tune.maxrewrite

No, not "prevent addition of headers", in fact it's the exact opposite, it's
meant to ensure that a received message will not fill a header so that this
number of bytes will always remain available to permit addition of headers.
Now I can understand how it's possible to read the doc like this, but what
is meant there is that if it's too small, as a side effect it will indeed
prevent the addition of headers, but this will result in the message not to
be correctly processed and to fail (thus 400 or 502 depending on the
direction) since the resulting message becomes larger than tune.bufsize.

> So it works like documented.

No, really not. The doc explains a consequence, not a goal (though it may
not be that obvious).

> But that does not mean that the current
> documented behaviour is a good behaviour.

Absolutely. What I'm much more worried about now is that a number of
deployments may already accidently be relying on this problem and may
discoverr 400s or 502s the hard way after we fix it :-(  So we'll have
to be very careful about this and enumerate what works and what fails
to figure if there's a real risk that people rely on this or not. If so
we may need to add an extra option to relax addition in case of failure
to preserve the current behaviour, which I'd personally hate.

Regards,
Willy

LUA Converters should be global

[Patrick Hemmer]

Right now in LUA, all the HAProxy converters are accessible through the
`Converters` class. However this class is only accessible through the
TXN class. Can we get this changed so that the Converters class is a global?

My intent is to be able to call a builtin HAProxy converter from within
a custom LUA converter. However the LUA converter prototype does not
receive a TXN, so I have no access to the the Converters class.
Converters are supposed to be stateless, so I see no reason why they
should be accessible only from within the TXN class.

-Patrick

Lua http-request use-service issue

[Hessam Mirsadeghi]

Hi,

I'm using a Lua function with the applet class to respond to certain
requests directly from haproxy/Lua. For this, I have "http-request
use-service lua.foo if some_acl" in my frontend.
Everything works fine except when I add a default backend which itself has
a line like "http-request use-service lua.bar". In this case, even the
requests that match some_acl are still sent to the default backend and
handled by lua.bar.
As soon as I remove the Lua service call from the backend everything works
fine as expected.
I tested with both haproxy 1.7 and 1.8. In both cases, I'm using Lua 5.3.4.

Best,
Seyed

Re: http-response set-header is unreliable

[Tim Düsterhus]

Willy,

Am 30.04.2018 um 15:48 schrieb Willy Tarreau:
>> And why does it affect two headers at once? If the length is right below
>> the limit intuitively only the very last header should be affected.
> 
> I really don't know, maybe the rules are aborted during their processing.

Anything I could do to help investigate this? I can apply patches with
additional logging or I can send you the unredacted configuration in
private if that would help.

>> A last: Maybe it makes sense to create a log message if this limit is
>> hit instead of silently ignoring (security critical!) response headers?
> 
> Well, a long time ago we would get a 400 when trying to add or modify
> a header increasing the request size past the max buffer size, or a 502
> when doing this on the response. The single fact that it didn't occur
> in your case sounds like a bug to me. I suspect that along the
> implementation of all these http-request / http-response rulesets, we
> may have lost some failure checks in some gray areas (ie: when calling
> a Lua service or stuff like this, it's not always black or white). Thus
> I guess we need to recheck all of this to ensure we're not missing anything
> important because I totally agree with you that a request not fullfilling
> all the prerequisites for complete processing should not pass ; same for
> responses.
> 

tune.bufsize is documented to send a 502:
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#3.2-tune.bufsize

tune.maxrewrite is documented to "prevent addition of headers":
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#3.2-tune.maxrewrite

So it works like documented. But that does not mean that the current
documented behaviour is a good behaviour.

Best regards
Tim Düsterhus

Re: Switch from http mode to tcp mode at will

[Willy Tarreau]

Hi,

On Mon, Apr 30, 2018 at 04:53:58PM +0200, Hoggins! wrote:
> Hi,
> 
> Le 30/04/2018 à 16:35, Aleksandar Lazic a écrit :
> > Hi.
> >
> > Am 30.04.2018 um 16:11 schrieb Hoggins!:
> >> Hello list,
> >>
> >> I have a case where I would like HAProxy to "get out of the way" if a
> >> certain pattern is detected in a HTTP request. In that particular case,
> >> I would like to select a TCP backend over an HTTP one if the conditions
> >> are met.
> >> But I'm not sure if it's feasible or even if it's the correct practice
> >> for my case.
> > Something similar like websockets?
> >
> > https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/
> 
> I've tried that, actually, but HAProxy won't let me specify "mode tcp"
> in the selected backend, as I apparently can't mix a "mode http" in the
> frontend (or in globals) and a "mode tcp" in a backend selected by
> "use_backend". That would have been my best shot, actually.
> 
> I think I'm missing something else in my whole setup, some things are
> not clear, thanks for the help anyway !

In fact you're probably mixing several things here. First, blindly
switching from HTTP to TCP doesn't make much sense as the conditions
would be based on an already prepared HTTP context that you can't
destroy without first processing it.

However, HTTP already offers provisions for this :
  - establishing a tunnel via the CONNECT method : a client sends a CONNECT
request, the server responds "200 OK" and then the connection becomes a
fully bi-directional raw TCP connection.

  - upgrading the current connection (same as the upgrade mechanism used
by TLS in RFC2817, WebSocket in RFC6455 and HTTP/2 in RFC7540) : the
client sends a request, adds Upgrade: and Connection:
upgrade. If the server responds "101 switching protocols", then the
connection is upgraded to the protocol agreed between the client and
the server. In practice for most gateways (like haproxy), it simply
means switching to a plain TCP connection between the two ends so
nothing blocks whatever protocol the client and server want to talk.

These are the mechanisms offered by the standards, and they must definitely
be used whenever possible, for example if you're developing an application
which wants to make HTTP and your protocol coexist.

In additionto this haproxy can also detect HTTP protocol elements while
running in TCP mode. It requires to enable protocol inspection and it's
not always trivial, but it's sometimes quite convenient. And it allows
a TCP frontend to switch to an HTTP backend for HTTP requests, or to a
TCP backend for TCP requests. A typical use case is to distinguish HTTP
from HTTPS. You can for example proceed approximately like this (check
the doc to ensure I didn't mess up with some arguments but you should
get the overall idea) :

   frontend public
  bind :80
  tcp-request inspect-delay 10s  #10s max for an HTTP request to come
  tcp-request content-accept if HTTP
  use_backend http if HTTP
  default_backend tcp

   backend http
  mode http
  ...
  server srv1 1.1.1.1:80

   backend tcp
  mode tcp
  ...
  server srv1 1.1.1.1:443

There are a few limitations which are often acceptable in this context,
such as the fact that you cannot log HTTP elements in the frontend. But
such mechanisms can be used to distinguish multiple protocols over a
single port. Only banner protocols (those where the server talks first)
cannot be distinguished but can sometimes be deduced after a timeout.

Hoping this helps,
Willy

Logging Question

[UPPALAPATI, PRAVEEN]

Hi Willy/Oliver,

One small question:

When I capture the header it's returning .com in the log but when I perform 
Get on .com:1000 it is not matching the following configuration.

frontend http-1000
bind *:1000 
option httplog
capture request header Host len 20
acl is_east hdr(host) -i .com 
use_backend east_bk_1000_read if is_east

My question is how can I print o/p of hdr(host) & is_east  to log?

Appreciate your help.

Thanks,
Praveen.

stable-bot: NOTICE: 7 bug fixes in queue for next release

[stable-bot]

Hi,

This is a friendly bot that watches fixes pending for the next haproxy-stable 
release!  One such e-mail is sent every week once patches are waiting in the 
last maintenance branch, and an ideal release date is computed based on the 
severity of these fixes and their merge date.  Responses to this mail must be 
sent to the mailing list.

Last release 1.8.8 was issued on 2018/04/19.  There are currently 7 patches in 
the queue cut down this way:
- 1 MAJOR, first one merged on 2018/04/26
- 2 MEDIUM, first one merged on 2018/04/26
- 4 MINOR, first one merged on 2018/04/26

Thus the computed ideal release date for 1.8.9 would be 2018/05/10, which is in 
two weeks or less.

The current list of patches in the queue is:
- MAJOR   : channel: Fix crash when trying to read from a closed socket
- MEDIUM  : h2: implement missing support for chunked encoded uploads
- MEDIUM  : lua: Fix segmentation fault if a Lua task exits
- MINOR   : config: disable http-reuse on TCP proxies
- MINOR   : log: t_idle (%Ti) is not set for some requests
- MINOR   : pattern: Add a missing HA_SPIN_INIT() in pat_ref_newid()
- MINOR   : lua/threads: Make lua's tasks sticky to the current thread

---
The haproxy stable-bot is freely provided by HAProxy Technologies to help 
improve the quality of each HAProxy release.  If you have any issue with these 
emails or if you want to suggest some improvements, please post them on the 
list so that the solutions suiting the most users can be found.

Re: Switch from http mode to tcp mode at will

[Aleksandar Lazic]

Hi Hoggins.

It would help when you share some more informations.

haproxy -vv
Your config.

Best regards
Aleks


 Ursprüngliche Nachricht 
Von: Hoggins! 
Gesendet: 30. April 2018 16:53:58 MESZ
An: haproxy@formilux.org
Betreff: Re: Switch from http mode to tcp mode at will

Hi,

Le 30/04/2018 à 16:35, Aleksandar Lazic a écrit :
> Hi.
>
> Am 30.04.2018 um 16:11 schrieb Hoggins!:
>> Hello list,
>>
>> I have a case where I would like HAProxy to "get out of the way" if a
>> certain pattern is detected in a HTTP request. In that particular case,
>> I would like to select a TCP backend over an HTTP one if the conditions
>> are met.
>> But I'm not sure if it's feasible or even if it's the correct practice
>> for my case.
> Something similar like websockets?
>
> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/

I've tried that, actually, but HAProxy won't let me specify "mode tcp"
in the selected backend, as I apparently can't mix a "mode http" in the
frontend (or in globals) and a "mode tcp" in a backend selected by
"use_backend". That would have been my best shot, actually.

I think I'm missing something else in my whole setup, some things are
not clear, thanks for the help anyway !

>
>> Thanks !
>>
>>     Hoggins!
> Regards
> Aleks
>
>
>

Re: Switch from http mode to tcp mode at will

[Hoggins!]

Hi,

Le 30/04/2018 à 16:35, Aleksandar Lazic a écrit :
> Hi.
>
> Am 30.04.2018 um 16:11 schrieb Hoggins!:
>> Hello list,
>>
>> I have a case where I would like HAProxy to "get out of the way" if a
>> certain pattern is detected in a HTTP request. In that particular case,
>> I would like to select a TCP backend over an HTTP one if the conditions
>> are met.
>> But I'm not sure if it's feasible or even if it's the correct practice
>> for my case.
> Something similar like websockets?
>
> https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/

I've tried that, actually, but HAProxy won't let me specify "mode tcp"
in the selected backend, as I apparently can't mix a "mode http" in the
frontend (or in globals) and a "mode tcp" in a backend selected by
"use_backend". That would have been my best shot, actually.

I think I'm missing something else in my whole setup, some things are
not clear, thanks for the help anyway !

>
>> Thanks !
>>
>>     Hoggins!
> Regards
> Aleks
>
>
>




signature.asc
Description: OpenPGP digital signature

Re: Switch from http mode to tcp mode at will

[Aleksandar Lazic]

Hi.

Am 30.04.2018 um 16:11 schrieb Hoggins!:
> Hello list,
> 
> I have a case where I would like HAProxy to "get out of the way" if a
> certain pattern is detected in a HTTP request. In that particular case,
> I would like to select a TCP backend over an HTTP one if the conditions
> are met.
> But I'm not sure if it's feasible or even if it's the correct practice
> for my case.

Something similar like websockets?

https://www.haproxy.com/blog/websockets-load-balancing-with-haproxy/

> Thanks !
> 
>     Hoggins!

Regards
Aleks

Switch from http mode to tcp mode at will

[Hoggins!]

Hello list,

I have a case where I would like HAProxy to "get out of the way" if a
certain pattern is detected in a HTTP request. In that particular case,
I would like to select a TCP backend over an HTTP one if the conditions
are met.
But I'm not sure if it's feasible or even if it's the correct practice
for my case.

Thanks !

    Hoggins!



signature.asc
Description: OpenPGP digital signature

Re: http-response set-header is unreliable

[Willy Tarreau]

Hi Tim,

On Sun, Apr 29, 2018 at 09:36:13PM +0200, Tim Düsterhus wrote:
> Willy,
> 
> Am 28.04.2018 um 07:51 schrieb Willy Tarreau:
> > Not that many ideas. Could you retry by setting "tune.maxrewrite" to a
> > larger value ? It defaults to 1024, and maybe you're already adding 1kB
> > of response and there's no more room in the response buffer. It's just
> > a guess, I could be completely wrong.
> 
> I configured it as 1536 and it seems to have fixed the issue. Indeed I
> am settings a few very large headers: The Content-Security-Policy header
> I'm setting is 410 bytes on it's own. The Public-Key-Pins-Report-Only
> header is an additional 258 bytes.
> 
> I'm wondering why I am unable to reproduce the issue consistently,
> though. The only dynamic header I am setting with haproxy is:
> 
> http-response set-header X-Req-ID %[unique-id]
> 
> and that one does not differ in length. Either it should fit in the
> buffer all the time or it should never fit, no?
> 
> And why does it affect two headers at once? If the length is right below
> the limit intuitively only the very last header should be affected.

I really don't know, maybe the rules are aborted during their processing.

> A last: Maybe it makes sense to create a log message if this limit is
> hit instead of silently ignoring (security critical!) response headers?

Well, a long time ago we would get a 400 when trying to add or modify
a header increasing the request size past the max buffer size, or a 502
when doing this on the response. The single fact that it didn't occur
in your case sounds like a bug to me. I suspect that along the
implementation of all these http-request / http-response rulesets, we
may have lost some failure checks in some gray areas (ie: when calling
a Lua service or stuff like this, it's not always black or white). Thus
I guess we need to recheck all of this to ensure we're not missing anything
important because I totally agree with you that a request not fullfilling
all the prerequisites for complete processing should not pass ; same for
responses.

Thanks,
Willy

Re: Question on Caching.

[Willy Tarreau]

Hi Andrew,

On Mon, Apr 30, 2018 at 10:08:11AM +0100, Andrew Smalley wrote:
> Hi Willy
> 
> Thank you for you for your detailed reply explaining why you think only the
> favicon cache is sensible and that a full-blown cache within Haproxy
> is not the best of ideas although interesting.
> 
> I will continue the search for a viable yet small cache.

What are you looking for exactly ? What makes you think the small object
cache would not be suited to your use case, or that it would be desirable
to have a more complete cache inside the load balancer ? We didn't get
much feedback on the cache, so your opinion on this is obviously interesting.

Thanks,
Willy

Re: [PATCH 0/2] Re: Logging SSL pre-master-key

[Willy Tarreau]

On Sat, Apr 28, 2018 at 07:15:44PM -0400, Patrick Hemmer wrote:
> After much delay, I've addressed the requested changes as a new patch.

Both patches merged now (with SMP_F_CONST removed as noticed by Emeric).

Thanks!
Willy

Re: [ANNOUNCE] haproxy-1.7.11

[Aleksandar Lazic]

Am 30.04.2018 um 12:11 schrieb Christopher Faulet:
> Hi,
> 
> HAProxy 1.7.11 was released on 2018/04/30. It added 38 new commits after
> version 1.7.10.

New docker images also available.

https://hub.docker.com/r/me2digital/haproxy17/

Regards
Aleks

> It fixes a major issue when HAProxy is compiled with some GCC versions
> (<= 3.x and >= 5.x). Because of a typo in a if statement in the function
> bo_getline_nc(), HAProxy crashes when it tries to read from a closed
> socket. This issue may happen with Lua codes, when you try to read lines
> from a cosocket.
> 
> It also fixes an issue about stream-interfaces. Some activities on
> channel are lost when the stream is woken up leading to intermittent 504
> errors with the improbable termination state 'sR--'. Another issue fixes
> how client aborts are handled during redispatch.
> 
> Some other medium issues were addressed: A memory leak in str2ip2,
> possible segfaults on startup when a slowstart is used, the wrapping
> case in the function bo_putblk and the "tcp-checks connect".
> 
> Finally, among the minor fixes and doc updates, many are about the Lua
> part.
> 
> 
> Please find the usual URLs below :
>    Site index   : http://www.haproxy.org/
>    Discourse    : http://discourse.haproxy.org/
>    Sources  : http://www.haproxy.org/download/1.7/src/
>    Git repository   : http://git.haproxy.org/git/haproxy-1.7.git/
>    Git Web browsing : http://git.haproxy.org/?p=haproxy-1.7.git
>    Changelog    : http://www.haproxy.org/download/1.7/src/CHANGELOG
>    Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
> 
> ---
> Complete changelog :
> Aurélien Nephtali (2):
>   CLEANUP: ssl: Remove a duplicated #include
>   BUG/MINOR: cli: Fix a typo in the 'set rate-limit' usage
> 
> Baptiste Assmann (2):
>   BUG/MEDIUM: tcp-check: single connect rule can't detect DOWN servers
>   BUG/MINOR: tcp-check: use the server's service port as a fallback
> 
> Christopher Faulet (5):
>   BUG/MEDIUM: stream-int: Don't loss write's notifs when a stream is
> woken up
>   BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as
> earlier as possible
>   BUG/MEDIUM: buffer: Fix the wrapping case in bo_putblk
>   BUG/MINOR: email-alert: Set the mailer port during alert
> initialization
>   BUG/MINOR: http: Return an error in proxy mode when url2sa fails
> 
> Cyril Bonté (2):
>   BUG/MINOR: force-persist and ignore-persist only apply to backends
>   DOC: log: more than 2 log servers are allowed
> 
> Emeric Brun (1):
>   BUG/MINOR: session: Fix tcp-request session failure if handshake.
> 
> Jérôme Magnin (2):
>   DOC: clarify the scope of ssl_fc_is_resumed
>   DOC: Describe routing impact of using interface keyword on bind lines
> 
> Lukas Tribus (1):
>   DOC: don't suggest using http-server-close
> 
> Thierry FOURNIER (2):
>   BUG/MINOR: spoa-example: unexpected behavior for more than 127 args
>   BUG/MINOR: lua: return bad error messages
> 
> Thierry Fournier (3):
>   MINOR/BUILD: fix Lua build on Mac OS X
>   BUG/MINOR: lua: the function returns anything
>   BUG/MINOR: lua funtion hlua_socket_settimeout don't check negative
> values
> 
> Tim Duesterhus (9):
>   BUG/MINOR: lua: Fix default value for pattern in Socket.receive
>   DOC: lua: Fix typos in comments of hlua_socket_receive
>   BUG/MINOR: lua: Fix return value of Socket.settimeout
>   CLEANUP: sample: Fix comment encoding of sample.c
>   CLEANUP: sample: Fix outdated comment about sample casts functions
>   BUG/MINOR: sample: Fix output type of c_ipv62ip
>   CLEANUP: Fix typo in ARGT_MSK6 comment
>   BUG/MEDIUM: standard: Fix memory leak in str2ip2()
>   BUG/MAJOR: channel: Fix crash when trying to read from a closed
> socket
> 
> Willy Tarreau (9):
>   BUG/MEDIUM: stream: properly handle client aborts during redispatch
>   BUG/MEDIUM: srv-state: always ensure there's a warmup task before
> manipulating it
>   BUG/MINOR: poll: too large size allocation for FD events
>   BUG/MINOR: config: don't emit a warning when global stats is
> incompletely configured
>   BUILD/MINOR: fix Lua build on Mac OS X (again)
>   MINOR: log: stop emitting alerts when it's not possible to write
> on the socket
>   DOC: lua: update the links to the config and Lua API
>   BUILD/BUG: enable -fno-strict-overflow by default
>   BUG/MINOR: config: disable http-reuse on TCP proxies
> 

Re: [PATCH 2/2] MINOR: ssl: add fetch 'ssl_fc_session_key' and 'ssl_bc_session_key'

[Willy Tarreau]

Hi guys,

On Mon, Apr 30, 2018 at 11:13:13AM +0200, Emeric Brun wrote:
> Hi Patrick,
> 
> On 04/29/2018 01:15 AM, Patrick Hemmer wrote:
> > 
> > These fetches return the SSL master key of the front/back connection.
> > This is useful to decrypt traffic encrypted with ephemeral ciphers.
> > ---
> >  doc/configuration.txt | 13 +
> >  src/ssl_sock.c| 35 +++
> >  2 files changed, 48 insertions(+)
> > 
> > 
> 
> No comment, except that i think the flag SMP_F_CONST is not necessary in this 
> case

OK, no need to respin, I can get rid of it while merging.

Thanks!
Willy

Re: [PATCH 1/2] MINOR: ssl: disable SSL sample fetches when unsupported

[Patrick Hemmer]

On 2018/4/30 04:58, Emeric Brun wrote:
> Hi Patrick,
>
> On 04/29/2018 01:15 AM, Patrick Hemmer wrote:
>> Previously these fetches would return empty results when HAProxy was
>> compiled
>> without the requisite SSL support. This results in confusion and problem
>> reports from people who unexpectedly encounter the behavior.
>> ---
>>  src/ssl_sock.c | 27 +++
>>  1 file changed, 15 insertions(+), 12 deletions(-)
>>
>>
> Are you sure this path will not cause regression using boringssl or libressl 
> because i see some ifdef just based on openssl version.
>
>
> R,
> Emeric
Yes. All this change does is copy/move the ifdefs around. Previously the
ifdefs were inside the functions, so that if the functions were called,
they'd return empty. The change moves the ifdefs to remove the function
from the compiled code entirely. Yes it is possible that people who have
been calling these functions when they don't have support for them will
now get errors. But this is what was requested to happen by Willy.

-Patrick

[ANNOUNCE] haproxy-1.7.11

[Christopher Faulet]

Hi,

HAProxy 1.7.11 was released on 2018/04/30. It added 38 new commits after 
version 1.7.10.


It fixes a major issue when HAProxy is compiled with some GCC versions 
(<= 3.x and >= 5.x). Because of a typo in a if statement in the function 
bo_getline_nc(), HAProxy crashes when it tries to read from a closed 
socket. This issue may happen with Lua codes, when you try to read lines 
from a cosocket.


It also fixes an issue about stream-interfaces. Some activities on 
channel are lost when the stream is woken up leading to intermittent 504 
errors with the improbable termination state 'sR--'. Another issue fixes 
how client aborts are handled during redispatch.


Some other medium issues were addressed: A memory leak in str2ip2, 
possible segfaults on startup when a slowstart is used, the wrapping 
case in the function bo_putblk and the "tcp-checks connect".


Finally, among the minor fixes and doc updates, many are about the Lua part.


Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Sources  : http://www.haproxy.org/download/1.7/src/
   Git repository   : http://git.haproxy.org/git/haproxy-1.7.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-1.7.git
   Changelog: http://www.haproxy.org/download/1.7/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

---
Complete changelog :
Aurélien Nephtali (2):
  CLEANUP: ssl: Remove a duplicated #include
  BUG/MINOR: cli: Fix a typo in the 'set rate-limit' usage

Baptiste Assmann (2):
  BUG/MEDIUM: tcp-check: single connect rule can't detect DOWN servers
  BUG/MINOR: tcp-check: use the server's service port as a fallback

Christopher Faulet (5):
  BUG/MEDIUM: stream-int: Don't loss write's notifs when a stream 
is woken up
  BUG/MEDIUM: http: Switch the HTTP response in tunnel mode as 
earlier as possible

  BUG/MEDIUM: buffer: Fix the wrapping case in bo_putblk
  BUG/MINOR: email-alert: Set the mailer port during alert 
initialization

  BUG/MINOR: http: Return an error in proxy mode when url2sa fails

Cyril Bonté (2):
  BUG/MINOR: force-persist and ignore-persist only apply to backends
  DOC: log: more than 2 log servers are allowed

Emeric Brun (1):
  BUG/MINOR: session: Fix tcp-request session failure if handshake.

Jérôme Magnin (2):
  DOC: clarify the scope of ssl_fc_is_resumed
  DOC: Describe routing impact of using interface keyword on bind lines

Lukas Tribus (1):
  DOC: don't suggest using http-server-close

Thierry FOURNIER (2):
  BUG/MINOR: spoa-example: unexpected behavior for more than 127 args
  BUG/MINOR: lua: return bad error messages

Thierry Fournier (3):
  MINOR/BUILD: fix Lua build on Mac OS X
  BUG/MINOR: lua: the function returns anything
  BUG/MINOR: lua funtion hlua_socket_settimeout don't check 
negative values


Tim Duesterhus (9):
  BUG/MINOR: lua: Fix default value for pattern in Socket.receive
  DOC: lua: Fix typos in comments of hlua_socket_receive
  BUG/MINOR: lua: Fix return value of Socket.settimeout
  CLEANUP: sample: Fix comment encoding of sample.c
  CLEANUP: sample: Fix outdated comment about sample casts functions
  BUG/MINOR: sample: Fix output type of c_ipv62ip
  CLEANUP: Fix typo in ARGT_MSK6 comment
  BUG/MEDIUM: standard: Fix memory leak in str2ip2()
  BUG/MAJOR: channel: Fix crash when trying to read from a closed 
socket


Willy Tarreau (9):
  BUG/MEDIUM: stream: properly handle client aborts during redispatch
  BUG/MEDIUM: srv-state: always ensure there's a warmup task before 
manipulating it

  BUG/MINOR: poll: too large size allocation for FD events
  BUG/MINOR: config: don't emit a warning when global stats is 
incompletely configured

  BUILD/MINOR: fix Lua build on Mac OS X (again)
  MINOR: log: stop emitting alerts when it's not possible to write 
on the socket

  DOC: lua: update the links to the config and Lua API
  BUILD/BUG: enable -fno-strict-overflow by default
  BUG/MINOR: config: disable http-reuse on TCP proxies

--
Christopher

Re: Truly seamless reloads

[William Lallemand]

On Mon, Apr 30, 2018 at 10:35:37AM +0300, Veiko Kukk wrote:
> On 26/04/18 17:11, Veiko Kukk wrote:
> > Hi,
> > 
> > According to 
> > https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/
> >  
> > :
> > 
> > "The patchset has already been merged into the HAProxy 1.8 development 
> > branch and will soon be backported to HAProxy Enterprise Edition 1.7r1 
> > and possibly 1.6r2."
> > 
> > Has it been backported to 1.7 and/or 1.6?
> > 
> > If yes, then should seamless reload also work with multiprocess 
> > configurations? (nbproc > 1).
> 
> Can i assume the answer is no for both questions?
> 
> 
> Veiko
> 

Hello Veiko,

Indeed, the seamless reload is only available since HAProxy 1.8.

It supports multiprocess configuration.


-- 
William Lallemand

Re: [PATCH 2/2] MINOR: ssl: add fetch 'ssl_fc_session_key' and 'ssl_bc_session_key'

[Emeric Brun]

Hi Patrick,

On 04/29/2018 01:15 AM, Patrick Hemmer wrote:
> 
> These fetches return the SSL master key of the front/back connection.
> This is useful to decrypt traffic encrypted with ephemeral ciphers.
> ---
>  doc/configuration.txt | 13 +
>  src/ssl_sock.c| 35 +++
>  2 files changed, 48 insertions(+)
> 
> 

No comment, except that i think the flag SMP_F_CONST is not necessary in this 
case


R,
Emeric

Re: Question on Caching.

[Andrew Smalley]

Hi Willy

Thank you for you for your detailed reply explaining why you think only the
favicon cache is sensible and that a full-blown cache within Haproxy
is not the best of ideas although interesting.

I will continue the search for a viable yet small cache.



Andruw Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmal...@loadbalancer.org

Leave a Review | Deployment Guides | Blog


On 28 April 2018 at 06:48, Willy Tarreau  wrote:
> Hi Andrew,
>
> On Thu, Apr 26, 2018 at 10:06:00PM +0100, Andrew Smalley wrote:
>> Hello Haproxy mailing list
>>
>> I have been looking at caching technology and have found this
>>
>> https://github.com/jiangwenyuan/nuster/
>>
>> It claims to be a v1.7  / v1.8 branch fully compatible with haproxy
>> and indeed based on haproxy with the added capibility of having a
>> really fast cache as described here
>> https://github.com/jiangwenyuan/nuster/wiki/Web-cache-server-performance-benchmark:-nuster-vs-nginx-vs-varnish-vs-squid
>>
>> It looks interesting but I would love some feedback please
>
> It's indeed interesting. By the way it's only for 1.7 as the 1.8 branch also
> contains 1.7. First, he found that nginx's primary job is not to be a cache
> (just like haproxy is not), and that in the end, only squid and varnish are
> real caches.
>
> Second, he focuses on performance. It's not new for many of us that haproxy
> rocks here, being 3 times faster than nginx in single core and 3 times faster
> than varnish using 12 cores is easily expected since haproxy never makes any
> single I/O access. He could even have compared with the small object cache
> in 1.8.
>
> But there's an important point which is missed there : manageability.
> Varnish is a real cache and made for being manageable and flexible. It
> probably has its own shortcomings, but it does the job perfectly for those
> who need a fully manageable cache. Putting a full-blown cache into haproxy
> is not a good idea in my opinion. A load balancer must be mostly stateless
> so that it can be killed, rebooted or tweaked. Implementing a full-blown
> cache into it seriously affects this capacity. It may even require some
> reloads just to flush the cache, while a load balancer should never have
> to be touched for no reason, especially when it's shared between multiple
> customers.
>
> The reason I was OK with the "favicon cache" in haproxy is that I noticed
> that when placing haproxy in front of varnish, we wasted more CPU and time
> processing the connection between haproxy and varnish than delivering a
> very small object from memory. And others had noticed that before, seeing
> certain configs use dummy backends with "errorfile 503" to deliver very
> small objects. So I thought that a short-lived, tiny objects cache saving
> us from having to connect to varnish would benefit both components without
> adding any requirement for cache maintenance. It's really where I draw the
> line between what is acceptable in haproxy and what is not. The day someone
> asks here if we can implement a cache flush on the CLI will indicate we've
> gone too far already, and we purposely refrained from implementing it.
>
> With this said, I can understand why some people would like to have more,
> especially when seeing the performance numbers on the site above. Possibly
> that we should think how to make it easier for these people to maintain
> their code without having to rebase too much (eg they may need some extra
> register functions or hooks to avoid patching the core).
>
> Regards,
> Willy

Re: [PATCH 1/2] MINOR: ssl: disable SSL sample fetches when unsupported

[Emeric Brun]

Hi Patrick,

On 04/29/2018 01:15 AM, Patrick Hemmer wrote:
> 
> Previously these fetches would return empty results when HAProxy was
> compiled
> without the requisite SSL support. This results in confusion and problem
> reports from people who unexpectedly encounter the behavior.
> ---
>  src/ssl_sock.c | 27 +++
>  1 file changed, 15 insertions(+), 12 deletions(-)
> 
> 
Are you sure this path will not cause regression using boringssl or libressl 
because i see some ifdef just based on openssl version.


R,
Emeric

Another stable maintainer :-)

[Willy Tarreau]

Hi all,

since Christopher devotes a lot of time to fixing bugs and he often waits
for me to backport them, I figured it could make it more efficient for all
of us if he joined the stable team and could backport fixes and issue
releases himself. He agreed to this, so don't be surprised to see a new
release coming from him soon :-)

Welcome onboard Christopher!

Willy

Re: Truly seamless reloads

[Veiko Kukk]

On 26/04/18 17:11, Veiko Kukk wrote:

Hi,

According to 
https://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/ 
:


"The patchset has already been merged into the HAProxy 1.8 development 
branch and will soon be backported to HAProxy Enterprise Edition 1.7r1 
and possibly 1.6r2."


Has it been backported to 1.7 and/or 1.6?

If yes, then should seamless reload also work with multiprocess 
configurations? (nbproc > 1).


Can i assume the answer is no for both questions?


Veiko