Re: Simply adding a filter causes read error

2018-12-12 Thread flamesea12 
Hi,

It's 100% reproduciable in my centos 7 PC

### Good Config start #
global
    maxconn 100
    daemon
    nbproc 2

defaults
    retries 3
    option redispatch
    timeout client  60s
    timeout connect 60s
    timeout server  60s
    timeout http-request 60s
    timeout http-keep-alive 60s

frontend web
    bind *:8000

    mode http
    default_backend app
backend app
    mode http
    server nginx01 10.0.3.15:8080
### Good Config end #


And bad config


### Bad Config start #
global
    maxconn 100
    daemon
    nbproc 2

defaults
    retries 3
    option redispatch
    timeout client  60s
    timeout connect 60s
    timeout server  60s
    timeout http-request 60s
    timeout http-keep-alive 60s

frontend web
    bind *:8000

    mode http
    default_backend app
backend app
    mode http
    filter compression
    server nginx01 10.0.3.15:8080
### Bad Config end #


### Lua script used in wrk, a.lua: ###

local count = 0

request = function()
    local url = "/?count=" .. count
    count = count + 1
    return wrk.format(
    'GET',
    url
    )
end


### Test 1 #

wrk -c 1000 -s a.lua http://10.0.3.15:8000
Running 10s test @ http://10.0.3.15:8000
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    60.43ms   42.63ms   1.06s    91.54%
    Req/Sec     7.86k     1.40k   10.65k    67.54%
  157025 requests in 10.11s, 769.87MB read
  Socket errors: connect 0, read 20, write 0, timeout 0
Requests/sec:  15530.67
Transfer/sec:     76.14MB


### Test 2 ###

change

 filter compression

to ==>

 filter trace

And update flt_trace.c:
add `return 0;` in `trace_attach`
to avoid performance down since there are many print

static int
trace_attach(struct stream *s, struct filter *filter)
{
        struct trace_config *conf = FLT_CONF(filter);
        return 0; 


Running 10s test @ http://10.0.3.15:8000
  2 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    64.88ms   77.91ms   1.09s    98.26%
    Req/Sec     7.84k     1.47k   11.57k    67.71%
  155800 requests in 10.05s, 763.86MB read
  Socket errors: connect 0, read 21, write 0, timeout 0
Requests/sec:  15509.93
Transfer/sec:     76.04MB


- Original Message -
> From: Willy Tarreau 
> To: flamese...@yahoo.co.jp
> Cc: Aleksandar Lazic ; "haproxy@formilux.org" 
> 
> Date: 2018/12/13, Thu 16:16
> Subject: Re: Simply adding a filter causes read error
> 
> Hi,
> 
> On Thu, Dec 13, 2018 at 03:48:57PM +0900, flamese...@yahoo.co.jp wrote:
>>  Hi again
>>  I tested against v1.8.15, the error's persistent.
> 
> It's very unclear what type of problem you're experiencing. Do you have
> a working and a non-working config as a starting point, and a way to
> reproduce the problem ? Also, are you seeing errors or anything special
> in your logs when you are facing the problem ?
> 
> Thanks,
> Willy
>

Re: Simply adding a filter causes read error

2018-12-12 Thread Willy Tarreau 
Hi,

On Thu, Dec 13, 2018 at 03:48:57PM +0900, flamese...@yahoo.co.jp wrote:
> Hi again
> I tested against v1.8.15, the error's persistent.

It's very unclear what type of problem you're experiencing. Do you have
a working and a non-working config as a starting point, and a way to
reproduce the problem ? Also, are you seeing errors or anything special
in your logs when you are facing the problem ?

Thanks,
Willy

Re: Simply adding a filter causes read error

2018-12-12 Thread flamesea12 
Hi again
I tested against v1.8.15, the error's persistent.

 - Original Message -
 From: "flamese...@yahoo.co.jp" 
 To: Aleksandar Lazic ; "haproxy@formilux.org" 
 
 Date: 2018/12/7, Fri 22:59
 Subject: Re: Simply adding a filter causes read error
   
Hi
Thanks for the reply.
I have a test env with 3 identical servers( 8 core cpu and 32GB memory), one 
for wrk, one for nginx, and one for haproxy.
The network looks like wrk => haproxy => nginx. I have tuned OS settings like 
open file limits, etc.
And the test html file is default nginx index.html. There's no error when 
testing wrk => nginx, wrk => haproxy(no filter) => nginx.
Error began to appear if I add filter.
I've thought of performance affected by compression, but that's not true, 
because the request header sent by wrk does not accept compression.
I've even change the following code:
static inttrace_attach(struct stream *s, struct filter *filter){        struct 
trace_config *conf = FLT_CONF(filter);        return 0; // ignore this filter 
to avoid performance down since there are many print
And test with
    filter trace
This way I think there will be no performance affect, since the filter is 
ignored in the very beginning.
But still there are read errors.
Please let me know if you need more information.
Thanks,

 - Original Message -
 From: Aleksandar Lazic 
 To: flamese...@yahoo.co.jp; "haproxy@formilux.org"  
 Date: 2018/12/7, Fri 22:12
 Subject: Re: Simply adding a filter causes read error
   
Hi.

Am 07.12.2018 um 08:37 schrieb flamese...@yahoo.co.jp:
> Hi
> 
> I tested more, and found that even with option http-pretend-keepalive enabled,
> 
> if I increase the test duration , the read error still appear.

Please can you show us some logs when the error appears.
Can you also tell us some data about the Server on which haproxy, wrk and nginx
is running and how the network setup looks like.

maybe you reach some system limits as Compression requires some more os/hw
resources.

Regards
Aleks

> Running 3m test @ http://10.0.3.15:8000 
>   10 threads and 1000 connections
>   Thread Stats   Avg      Stdev     Max   +/- Stdev
>     Latency    19.84ms   56.36ms   1.34s    92.83%
>     Req/Sec    23.11k     2.55k   50.64k    87.10%
>   45986426 requests in 3.33m, 36.40GB read
>   Socket errors: connect 0, read 7046, write 0, timeout 0
> Requests/sec: 229817.63
> Transfer/sec:    186.30MB
> 
> thanks
> 
>    - Original Message -
>    *From:* "flamese...@yahoo.co.jp" 
>    *To:* Aleksandar Lazic ; "haproxy@formilux.org"
>    
>    *Date:* 2018/12/7, Fri 09:06
>    *Subject:* Re: Simply adding a filter causes read error
> 
>    Hi,
> 
>    Thanks for the reply, I thought the mail format is corrupted..
> 
>    I tried option http-pretend-keepalive, seems read error is gone, but 
>timeout
>    error raised(maybe its because the 1000 connections of wrk)
> 
>    Thanks
> 
>        - Original Message -
>        *From:* Aleksandar Lazic 
>        *To:* flamese...@yahoo.co.jp; "haproxy@formilux.org" 
>
>        *Date:* 2018/12/6, Thu 23:53
>        *Subject:* Re: Simply adding a filter causes read error
> 
>        Hi.
> 
>        Am 06.12.2018 um 15:20 schrieb flamese...@yahoo.co.jp
>        :
>        > Hi,
>        >
>        > I have a haproxy(v1.8.14) in front of several nginx backends,
>        everything works
>        > fine until I add compression in haproxy.
> 
>        There is a similar thread about this topic.
> 
>        https://www.mail-archive.com/haproxy@formilux.org/msg31897.html 
> 
>        Can you try to add this option in your config and see if the problem is
>        gone.
> 
>        option http-pretend-keepalive
> 
>        Regards
>        Aleks
> 
>        > My config looks like this:
>        >
>        > ### Config start #
>        > global
>        >     maxconn         100
>        >     daemon
>        >     nbproc 2
>        >
>        > defaults
>        >     retries 3
>        >     option redispatch
>        >     timeout client  60s
>        >     timeout connect 60s
>        >     timeout server  60s
>        >     timeout http-request 60s
>        >     timeout http-keep-alive 60s
>        >
>        > frontend web
>        >     bind *:8000
>        >
>        >     mode http
>        >     default_backend app
>        > backend app
>        >     mode http
>        >     #filter compression
>        >     #filter trace 
>        >     server nginx01 10.0.3.15:8080
>        > ### Config end #
>        >
>        >
>        > Lua script used in wrk:
>        > a.lua:
>        >
>        > local count = 0
>        >
>        > request = function()
>        >     local url = "/?count=" .. count
>        >     count = count + 1
>        >     return wrk.format(
>        >     'GET',
>        >     url
>        >     )
>        > end
>        >
>        >
>        > 01. wrk test against nginx: everything if OK
>        >
>        > wrk -c

[ANNOUNCE] haproxy-1.8.15

2018-12-12 Thread Willy Tarreau 
Hi,

HAProxy 1.8.15 was released on 2018/12/13. It added 69 new commits
after version 1.8.14.

Yes I know 1.8 has been lagging behind a little bit during these last few
months, but all the people able to emit a release were all totally booked
on finishing 1.9.

So here comes the long-expected 1.8.15 which fixes an assorted number of
issues.

The most visible bugs are a failure to properly configure the connection
window size in H2 which affects upload speed in 1.8, the HPACK encoding
of the accept-ranges header field in H2 responses which was replaced by
accept-language, an alignment issue on stick tables causing some strict
aligned architectures to crash when using stick tables, an improper
locking around crypt() which is not thread-safe, resulting in auth
requests to randomly fail in thread environments, a change on the way
401/407 are handled so that the last server preference is only applied
to non-deterministic algorithms (don't break hashing), a crash if someone
configures the cache size to be larger than 2047 MB, a risk of deadlocks
when using threads with queues or health checks state change depending
on the compiler's optimizations, an obscure bug in master-worker and
threads related to the handling of SIGUSR1 followed by SIGTERM, and
thread-safe Cur/CumSslConns counters (the current one could wrap in
either direction).

In addition, Rémi Gacogne found, reported, and fixed 5 bugs in the DNS
handling code which could be used to crash haproxy by spoofing response
packets from a server. I don't consider them dramatic since nobody
should make their LB rely on public, non-protected communication
channels to configure their farms, so I think that the DNS is always
in a safe area, but still we don't know. Rémi provided the fixes, and
Karol Babioch from SuSE obtained the following CVE IDs if that helps :

- CVE-2018-20102 -> out-of-bounds read in dns_validate_dns_response in
  dns.c

- CVE-2018-20103 -> infinite recursion by making the pointer point to
  itself in DNS reply

BTW, a quick point regarding the CVEs, I know that sometimes distros
want to have them to ease their backports. Doing this is crap. 100% of
the bugs cited above have much more likeliness to hit someone stronger
than these ones. The only way to correctly use CVEs is as an indicator
that's really time to emit a new version which contains 100% of the
other fixes as well.

Last but not least, Dirkjan Bussink brought the support for the new
ciphersuite option that really is mandatory to support TLSv1.3, so we
backported it to 1.8.

Finally most of Joseph Herlant's user-visible doc fixes were backported
as well.

I won't claim it's the last 1.8 of the year because someone will want
to prove me wrong. So let's say I'll just hope for it :-)

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Sources  : http://www.haproxy.org/download/1.8/src/
   Git repository   : http://git.haproxy.org/git/haproxy-1.8.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-1.8.git
   Changelog: http://www.haproxy.org/download/1.8/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Baptiste Assmann (1):
  BUG/MINOR: ssl: ssl_sock_parse_clienthello ignores session id

Bertrand Jacquin (1):
  DOC: Fix a few typos

Christopher Faulet (3):
  BUG/MINOR: config: Copy default error messages when parsing of a backend 
starts
  BUG/MINOR: cfgparse: Fix transition between 2 sections with the same name
  BUG/MINOR: cfgparse: Fix the call to post parser of the last sections 
parsed

Dirkjan Bussink (2):
  MEDIUM: ssl: add support for ciphersuites option for TLSv1.3
  CLEANUP: stick-tables: Remove unneeded double (()) around conditional 
clause

Emeric Brun (2):
  BUG/MEDIUM: Cur/CumSslConns counters not threadsafe.
  BUG/MEDIUM: mworker: segfault receiving SIGUSR1 followed by SIGTERM.

Frédéric Lécaille (4):
  BUG/MINOR: cache: Crashes with "total-max-size" > 2047(MB).
  BUG/MINOR: cache: Wrong usage of shctx_init().
  BUG/MINOR: ssl: Wrong usage of shctx_init().
  DOC: cache: Missing information about "total-max-size"

Ilya Shipitsin (1):
  BUG/MINOR: connection: avoid null pointer dereference in send-proxy-v2

Joseph Herlant (3):
  DOC: Fix typos in README and CONTRIBUTING
  DOC: Fix typos in different subsections of the documentation
  DOC: fix a few typos in the documentation

Jérôme Magnin (2):
  DOC: clarify that check-sni needs an argument.
  DOC: refer to check-sni in the documentation of sni

Lukas Tribus (5):
  DOC: clarify force-private-cache is an option
  DOC: fix reference to map files in MAINTAINERS
  BUG/MINOR: only mark connections private if NTLM is detected
  BUG/MINOR: only auto-prefer last server if lb-alg is non-deterministic
  DOC: restore note about "independant" typo

Moemen MHEDHBI (1):

Re: Sporadic problems with backend selection with H2.

2018-12-12 Thread Willy Tarreau 
On Wed, Dec 12, 2018 at 11:00:21AM -0700, Sean Reifschneider wrote:
> Thanks, that makes sense and I did a quick test with just the Host header
> based routing and that was looking good.  I don't recall where I got to
> both SNI and host selection, that might be something that fixed a problem 3
> years ago, or might have just been my misunderstanding.

It's very likely inherited from early configs where we didn't support SSL
but were able to inspect the SNI and route to an SSL server based on it.
Nevertheless, what matters is that 1) it's fixed for you, and 2) there is
no ugly bug to chase for me :-)

cheers,
Willy

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread PiBa-NL 

Hi Christopher,

Op 12-12-2018 om 13:49 schreef Christopher Faulet:

Le 12/12/2018 à 12:07, Pi Ba a écrit :
Found the issue on the 10th (I think commit 56b0348).. so yesterday's 
commit isn't the (only) problem.. tested with commit 0007d0a the 
issue also happens. Reverting only below mentioned commit I can't 
easily do atm. I'll check more closely this evening.




Hum, I don't understand, the commit 56b0348 fixes a bug in the H2 
multiplexer. you don't use it in your test-case. It should be totally 
unrelated.


It wasn't that specific commit that broke my test, its just that i 
picked that one because i could... It just that was they day i started 
to try 1.9-X on my production environment..


Having spend a bit more time checking/compiling various commits i found 
that this is the commit that 'broke' the testcase that was attached 
(some of the contents is misplaced/repeated..):

http://git.haproxy.org/?p=haproxy.git;a=commit;h=d247be0620c35ea0a43074fd88c6a520629c1823

p.s. I also send you off-list the full output of the test with the added 
configuration options: filter trace name BEFORE / filter compression / 
filter trace name AFTER .. (resulting in a 90MB log..)


Regards,

PiBa-NL (Pieter)

Re: Sporadic problems with backend selection with H2.

2018-12-12 Thread Sean Reifschneider 
Thanks, that makes sense and I did a quick test with just the Host header
based routing and that was looking good.  I don't recall where I got to
both SNI and host selection, that might be something that fixed a problem 3
years ago, or might have just been my misunderstanding.

Thanks!

Sean

On Tue, Dec 11, 2018 at 11:33 PM Willy Tarreau  wrote:

> Hi Sean,
>
> On Tue, Dec 11, 2018 at 04:22:53PM -0700, Sean Reifschneider wrote:
> > I've been trying to convert my haproxy setup (1.8.14) to by adding "alpn
> > h2,http/1.1" to my "bind" line in the frontend.  My haproxy config is
> north
> > of 300 lines, so I'll hold off on attaching it.
> >
> > My frontend selects backends using something like:
> >
> > acl aerial_acl hdr(host)  -m reg -i ^aerial[1-4].example.com
> > acl aerial_acl ssl_fc_sni -m reg -i ^aerial[1-4].example.com
> > use_backend aerial if aerial_acl
> >
> > acl wwwl_acl hdr(host)  -m reg -i ^www.example.com
> > acl www_acl ssl_fc_sni -m reg -i ^www.example.com
> > use_backend www if www_acl
> >
> > default_backend www
> >
> > And I have a bunch of those selecting different backends.
> >
> > The base haproxy config is quite battle-tested, it has been running on my
> > staging environment with H2 for ~6 months.  The production config is a
> > refinement (based off staging) that has been running for a few years in
> > production.
> >
> > But, when I enable H2 some small number of requests seem to be going to
> the
> > wrong backend.  I see something like a bunch of requests for map tiles on
> > one connection, then that connection gets a request that should be going
> to
> > the main web server, it has a header of "www.example.com" instead of "
> > aerial1.example.com", the backend server logs that the Host header was "
> > www.example.com", but it is the aerial backend.
> >
> > This happens infrequently, and if I take the "alpn h2,http/1.1" off the
> > "bind" line it does not seem to happen.  I have only small periods of
> time
> > when I've run it so I can't give an exhaustive list of user-agents, but I
> > definitely saw it happening with Chrome 70 on Windows 10, Chrome 71 on
> > Windows 7, Chrome 69 on Windows 10, Safari 12 on iOS 12.1, Chrome 68 on
> > Android...
> >
> > So, in summary:
> >
> > - Works fine without "alpn h2,http/1.1".
> > - Only some requests are mis-routed.
> > - It seems to be a keep-alive connection that opens to get map tiles,
> then
> > a few seconds or minutes later try to get something from the web server.
> > - The host header in the request seems to be correct, the backend is
> > logging it correct.
> > - The aerial tile and web servers have the same external IP address and
> are
> > routed based on the "Host" sent with the request.
>
> Do you log the host header on haproxy ? I'm asking because in 1.8, H2 first
> translates the request to H1, then processes it and passes it to the
> server.
> Thus if the server sees the correct host, haproxy should as well. And if
> haproxy fails to see the correct host header when using H2, it should also
> fail to see it in H1.
>
> Oh, wait a minute, you're using SNI to route the request. This is
> incorrect,
> you must always use the Host header for this. SNI is only used to tell the
> server (haproxy in this case) what certificate to present. But for a given
> connection, the client can send different requests for various hosts, which
> are indicated in the Host header. One example I have in mind would be when
> the server presents a certificate with alt-names (or a wildcard
> certificate)
> indicating to the client that the connection is still authoritative for
> other
> names. In this case nothing prevents the client from reusing the same
> connection for various hosts compatible with this cert. So you must
> absolutely rely exclusively on the Host header for all HTTP routing.
>
> Regards,
> Willy
>

Re: Uptick in 503s with HAProxy 1.8

2018-12-12 Thread Àbéjídé Àyodélé 
Hi,

Follow up from my previous email, we have new findings and the findings are as
below.

We observed endpoints that returned 400 with a termination reason of CH in 1.7
now sometimes return a 503 in 1.8 with a termination of CC. Find below
breakdown of CC and CH with their http response codes in the past 3 days.

"host","http_status_code","proxy_termination_state","_count"
"haproxy01","400","CH--","5610"
"haproxy01","503","CC--","1405"
"haproxy02","400","CH--","6967"
"haproxy03","400","CH--","7147"
"haproxy04","400","CH--","6910"
"haproxy05","400","CH--","6943"

haproxy01 is running 1.8.14 HAProxy while the rest are running 1.7.11

Note the servers are all behind the same L3/L4 load-balancer so the nature of
traffic is entirely similar.

Is there any change between 1.7 and 1.8 that will explain this new logging
behavior.

Thanks.

Abejide Ayodele
It always seems impossible until it's done. --Nelson Mandela
On Thu, Nov 1, 2018 at 10:25 AM Àbéjídé Àyodélé
 wrote:
>
> Hi Friends,
>
> We upgraded to HAProxy 1.8.14 from 1.7.11 and saw an uptick in the number of
> browser related 503 requests with termination reason CC. TC, TR and TW are
> consistent across these requests and are -1,-1,0 respectively, these requests
> are proxied to nginx, the only thing that has changed so far in our setup is
> upgrading from 1.7.11 to 1.8.14. Find below an example log entry:
>
> {"proxy_server_name":"http-in","proxy_tq":"826","proxy_tw":"0","proxy_tc":"-1","proxy_tr":"-1","proxy_tt":"826","http_status_code":"503","http_bytes_read":"0","http_captured_request_cookie":"-","http_captured_response_cookie":"-","proxy_termination_state":"CC--","proxy_active_connections":"131","proxy_frontend_connections":"104","proxy_backend_connections":"27","proxy_server_connections":"27","proxy_retries":"0","proxy_server_queue":"0","proxy_backend_queue":"0","http_method":"GET","http_request_url":"/favicon.ico","ssl_version":"TLSv1.2","ssl_cipher":"ECDHE-RSA-AES128-GCM-SHA256"}
>
> I can share our config if required, I am not including it in this email
> because of it's size and hope we can narrow down to specifics that will be
> easy to share.
>
> Do folks know if there any known things that could have changed between 1.7
> and 1.8 to have introduced this uptick?
>
> Thanks!
>
> Abejide Ayodele
> It always seems impossible until it's done. --Nelson Mandela

Re: [PATCH] ssl: factoring load cert/key and chains

2018-12-12 Thread Emmanuel Hocdet 
Hi Julien,

> Le 12 déc. 2018 à 14:28, Julien Laffaye  a écrit :
> 
> 
> On Wed, Dec 12, 2018 at 12:24 PM Emmanuel Hocdet  > wrote:
> 
> Hi,
> 
> I tried to improve the haproxy loading time with a lot of certificates, and 
> see a double file
> open for each certificate (one for private-key and one for the cert/chain).
> Multi-cert loading part have not this issue and is good candidate for sharing 
> code:
> patches is this work with factoring/cleanup/fix.
> 
> About speed: PEM file with private key in first position is far better.
> 
> If you can consider this patches?
> 
> ++
> Manu
> 
> 
> Hello,
> 
> I'm curious, do you have numbers concerning the open(2) optimization ?

Not much (a few seconds), because load cert and especially load key is the 
heaviest part,
but avoid syscall for reload  it's rather good when kernel has something else 
to do.

> And also the PEM ordering, I did not know it mattered.
> 
5 to 10% speedup in my tests.

I tested with pkcs12 format (with limited encryption) but it’s slower than pem.
The best gain is with EC certificates (x2 vs RSA-2048) (load key is really the 
heaviest part)

++
Manu

[PATCH] REGTEST: level 1 health-check test 2.

2018-12-12 Thread Frederic Lecaille 

Here is a new reg test for the health-check.

Sounds similar to h1.vtc but is more intensive with client 
connections to verify there is no connection consumption by the health

checks.

Also checks that only servers with "check" option are "health-check'ed".

Fred.
From f4a26125601cadce73535925c24f024058830b55 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Fr=C3=A9d=C3=A9ric=20L=C3=A9caille?= 
Date: Wed, 12 Dec 2018 10:51:14 +0100
Subject: [PATCH] REGTEST: level 1 health-check test 2.

With this test we check that the health-checks do not consume any connection on
the backend side.
---
 reg-tests/checks/h2.vtc | 677 
 1 file changed, 677 insertions(+)
 create mode 100644 reg-tests/checks/h2.vtc

diff --git a/reg-tests/checks/h2.vtc b/reg-tests/checks/h2.vtc
new file mode 100644
index ..7246
--- /dev/null
+++ b/reg-tests/checks/h2.vtc
@@ -0,0 +1,677 @@
+varnishtest "Health-checks"
+feature ignore_unknown_macro
+
+#REQUIRE_VERSION=1.8
+
+# This script start 40 servers named s0 up to s39.
+# For 0 <= i <= 19:
+#   - s(i) and s(i+1) belong to backend be(2*i+1),
+#   - fe(2*i+1) backend is configured to used be(2*i+1) backend.
+#   - only s(2*i+1) servers have health-checks enabled,
+#   - we start 20 clients named s(2*i+1) which connect to fe(2*i+1) frontend,
+#   - so that to ensure that health-checks do not consume any connection
+# (any varnishtest server without -repeat  with n > 1 accepts
+# only one connection).
+
+# Note that the first syslog message received is: "Proxy  started."
+syslog S1 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be1 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be1/srv1 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S3 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be3 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be3/srv3 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S5 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be5 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be5/srv5 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S7 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be7 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be7/srv7 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S9 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be9 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be9/srv9 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S11 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be11 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be11/srv11 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S13 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be13 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be13/srv13 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S15 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be15 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be15/srv15 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S17 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be17 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be17/srv17 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S19 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be19 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be19/srv19 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S21 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be21 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be21/srv21 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S23 -level notice {
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Proxy be23 started."
+recv
+expect ~ "[^:\\[ ]\\[${h1_pid}\\]: Health check for server be23/srv23 succeeded, reason: Layer4 check passed, check duration: [[:digit:]]+ms, status: 1/1 UP"
+} -start
+
+syslog S25

Re: [PATCH] ssl: factoring load cert/key and chains

2018-12-12 Thread Julien Laffaye 
On Wed, Dec 12, 2018 at 12:24 PM Emmanuel Hocdet  wrote:

>
> Hi,
>
> I tried to improve the haproxy loading time with a lot of certificates,
> and see a double file
> open for each certificate (one for private-key and one for the cert/chain).
> Multi-cert loading part have not this issue and is good candidate for
> sharing code:
> patches is this work with factoring/cleanup/fix.
>
> About speed: PEM file with private key in first position is far better.
>
> If you can consider this patches?
>
> ++
> Manu
>
>
Hello,

I'm curious, do you have numbers concerning the open(2) optimization ?
And also the PEM ordering, I did not know it mattered.

Regards,
Julien

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread Willy Tarreau 
On Wed, Dec 12, 2018 at 01:49:31PM +0100, Christopher Faulet wrote:
> Le 12/12/2018 à 12:07, Pi Ba a écrit :
> > Found the issue on the 10th (I think commit 56b0348).. so yesterday's
> > commit isn't the (only) problem.. tested with commit 0007d0a the issue
> > also happens. Reverting only below mentioned commit I can't easily do
> > atm. I'll check more closely this evening.
> > 
> 
> Hum, I don't understand, the commit 56b0348 fixes a bug in the H2
> multiplexer. you don't use it in your test-case. It should be totally
> unrelated.

At least I've merged your fix because it addresses a real breakage I
did last evening.

Thanks,
Willy

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread Christopher Faulet 

Le 12/12/2018 à 12:07, Pi Ba a écrit :
Found the issue on the 10th (I think commit 56b0348).. so yesterday's 
commit isn't the (only) problem.. tested with commit 0007d0a the issue 
also happens. Reverting only below mentioned commit I can't easily do 
atm. I'll check more closely this evening.




Hum, I don't understand, the commit 56b0348 fixes a bug in the H2 
multiplexer. you don't use it in your test-case. It should be totally 
unrelated.


--
Christopher

[PATCH] ssl: factoring load cert/key and chains

2018-12-12 Thread Emmanuel Hocdet 

Hi,

I tried to improve the haproxy loading time with a lot of certificates, and see 
a double file
open for each certificate (one for private-key and one for the cert/chain).
Multi-cert loading part have not this issue and is good candidate for sharing 
code:
patches is this work with factoring/cleanup/fix.

About speed: PEM file with private key in first position is far better.

If you can consider this patches?

++
Manu



0001-REORG-ssl-promote-cert_key_and_chain-handling.patch
Description: Binary data


0002-MINOR-ssl-use-STACK_OF-for-chain-certs.patch
Description: Binary data


0003-MINOR-ssl-SSL_CTX_set1_chain-compatibility.patch
Description: Binary data


0004-MINOR-ssl-used-cert_key_and_chain-func-in-load_cert_.patch
Description: Binary data


0005-BUG-MINOR-ssl-fix-ssl_sock_load_multi_cert-init-vars.patch
Description: Binary data


0006-CLEANUP-ssl-ssl_sock_load_crt_file_into_ckch.patch
Description: Binary data

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread Pi Ba 
Found the issue on the 10th (I think commit 56b0348).. so yesterday's
commit isn't the (only) problem.. tested with commit 0007d0a the issue also
happens. Reverting only below mentioned commit I can't easily do atm. I'll
check more closely this evening.

Op wo 12 dec. 2018 10:52 schreef Christopher Faulet <
christopher.fau...@capflam.org:

> Le 12/12/2018 à 02:08, PiBa-NL a écrit :
> > Hi List,
> >
> > Didn't have time yet to bisect when it went wrong. But attached testfile
> > produces the following output after 3 curl requests at different speeds,
> > this seems to trigger a problem as the hash of the downloaded content is
> > nolonger the same as it should be, (in my actual environment its a 2MB
> > javascript file that comes from a iis server behind haproxy.). Took
> > already a few hours more than desired to come up with a seemingly
> > reliable reproduction.
> > 1.9-dev10 is the first one i put on my production environment as i think
> > release is imminent so it 'should' be pretty stable ;), (yes i know..i
> > shouldn't assume..) before it was using 1.8.14.. So was quick to revert
> > to that 1.8 again :).
> >
> > Using these settings:
> >
> >   compression algo gzip
> >   compression type text/html text/plain application/json
> > application/javascript
> >   compression offload
> > When these compression settings are disabled, it completes successfully..
> >
> >  top   2.4 shell_cmd|  exit 1
> >  top   2.4 shell_cmd|fi
> >  top   2.5 shell_out|File1 all OK
> >  top   2.5 shell_out|File2 not OK 7798551c02a37ce89c77fc18fc415e5b
> >  top   2.5 shell_out|File3 not OK 3146c4c9fce4da750558bfd9387ffc3b
> >  top   2.5 shell_status = 0x0001
> >  top   2.5 shell_exit not as expected: got 0x0001 wanted 0x
> > *top   2.5 RESETTING after ./PB-TEST/ulticompres/b5.vtc
> > **   h12.5 Reset and free h1 haproxy 51853
> > **   h12.5 Wait
> > **   h12.5 Stop HAproxy pid=51853
> >  h12.5 STDOUT poll 0x11
> >  h12.5 Kill(2)=0: No error: 0
> > **   h12.6 WAIT4 pid=51853 status=0x0002 (user 0.253496 sys 0.00)
> > *top   2.6 TEST ./PB-TEST/ulticompres/b5.vtc FAILED
> > #top  TEST ./PB-TEST/ulticompres/b5.vtc FAILED (2.581) exit=2
> >
> > haproxy -v
> > HA-Proxy version 1.9-dev10-3815b22 2018/12/11
> > Copyright 2000-2018 Willy Tarreau 
> >
> > Can anyone confirm? Or perhaps even fix ;) Ill try and dig a little more
> > tomorrow evening :).
> >
> Hi Pieter,
>
> There is a bug with the commit 3815b22 when the message is chunked. I
> need to see with Willy how to fix it. BTW, if you revert this commit, it
> should work fine.
>
> Thanks,
> --
> Christopher
>

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread Christopher Faulet 

Le 12/12/2018 à 10:52, Christopher Faulet a écrit :

Le 12/12/2018 à 02:08, PiBa-NL a écrit :

Hi List,

Didn't have time yet to bisect when it went wrong. But attached testfile
produces the following output after 3 curl requests at different speeds,
this seems to trigger a problem as the hash of the downloaded content is
nolonger the same as it should be, (in my actual environment its a 2MB
javascript file that comes from a iis server behind haproxy.). Took
already a few hours more than desired to come up with a seemingly
reliable reproduction.
1.9-dev10 is the first one i put on my production environment as i think
release is imminent so it 'should' be pretty stable ;), (yes i know..i
shouldn't assume..) before it was using 1.8.14.. So was quick to revert
to that 1.8 again :).

Using these settings:

       compression algo gzip
       compression type text/html text/plain application/json
application/javascript
       compression offload
When these compression settings are disabled, it completes successfully..

 top   2.4 shell_cmd|  exit 1
 top   2.4 shell_cmd|    fi
 top   2.5 shell_out|File1 all OK
 top   2.5 shell_out|File2 not OK 7798551c02a37ce89c77fc18fc415e5b
 top   2.5 shell_out|File3 not OK 3146c4c9fce4da750558bfd9387ffc3b
 top   2.5 shell_status = 0x0001
 top   2.5 shell_exit not as expected: got 0x0001 wanted 0x
*    top   2.5 RESETTING after ./PB-TEST/ulticompres/b5.vtc
**   h1    2.5 Reset and free h1 haproxy 51853
**   h1    2.5 Wait
**   h1    2.5 Stop HAproxy pid=51853
 h1    2.5 STDOUT poll 0x11
 h1    2.5 Kill(2)=0: No error: 0
**   h1    2.6 WAIT4 pid=51853 status=0x0002 (user 0.253496 sys 0.00)
*    top   2.6 TEST ./PB-TEST/ulticompres/b5.vtc FAILED
#    top  TEST ./PB-TEST/ulticompres/b5.vtc FAILED (2.581) exit=2

haproxy -v
HA-Proxy version 1.9-dev10-3815b22 2018/12/11
Copyright 2000-2018 Willy Tarreau 

Can anyone confirm? Or perhaps even fix ;) Ill try and dig a little more
tomorrow evening :).


Hi Pieter,

There is a bug with the commit 3815b22 when the message is chunked. I
need to see with Willy how to fix it. BTW, if you revert this commit, it
should work fine.



Ok, Finally it was not too hard to fix (hoping it does not break 
anything else...). Here is a patch. Pieter, could you try it to see if 
it works for you ?


--
Christopher
>From 00ebd09575fd7451033b78c394f04476b79b9aa4 Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Wed, 12 Dec 2018 10:32:09 +0100
Subject: [PATCH] BUG/MEDIUM: mux-h1: Fix the zero-copy on output for chunked
 messages

The commit 3815b227f ("MEDIUM: mux-h1: implement true zero-copy of DATA blocks")
broke the output of chunked messages. When the zero-copy was performed on such
messages, no chunk size was emitted nor ending CRLF.

Now, the chunked envelope is added when necessary. We have at least the size of
the struct htx to emit it. So 40 bytes for now. It should be enough.
---
 src/mux_h1.c | 38 ++
 1 file changed, 38 insertions(+)

diff --git a/src/mux_h1.c b/src/mux_h1.c
index baaf0dfed..da5d5c17e 100644
--- a/src/mux_h1.c
+++ b/src/mux_h1.c
@@ -839,6 +839,34 @@ static void h1_capture_bad_message(struct h1c *h1c, struct h1s *h1s,
 			, h1_show_error_snapshot);
 }
 
+/* Emit the chunksize followed by a CRLF in front of data of the buffer
+ * . It goes backwards and starts with the byte before the buffer's
+ * head. The caller is responsible for ensuring there is enough room left before
+ * the buffer's head for the string.
+ */
+static void h1_emit_chunk_size(struct buffer *buf, size_t chksz)
+{
+	char *beg, *end;
+
+	beg = end = b_head(buf);
+	*--beg = '\n';
+	*--beg = '\r';
+	do {
+		*--beg = hextab[chksz & 0xF];
+	} while (chksz >>= 4);
+	buf->head -= (end - beg);
+	b_add(buf, end - beg);
+}
+
+/* Emit a CRLF after the data of the buffer . The caller is responsible for
+ * ensuring there is enough room left in the buffer for the string. */
+static void h1_emit_chunk_crlf(struct buffer *buf)
+{
+	*(b_peek(buf, b_data(buf))) = '\r';
+	*(b_peek(buf, b_data(buf) + 1)) = '\n';
+	b_add(buf, 2);
+}
+
 /*
  * Parse HTTP/1 headers. It returns the number of bytes parsed if > 0, or 0 if
  * it couldn't proceed. Parsing errors are reported by setting H1S_F_*_ERROR
@@ -1389,6 +1417,16 @@ static size_t h1_process_output(struct h1c *h1c, struct buffer *buf, size_t coun
 
 			buf->area = old_area;
 			buf->data = buf->head = 0;
+
+			/* The message is chunked. We need to emit the chunk
+			 * size. We have at least the size of the struct htx to
+			 * write the chunk envelope. It should be enough.
+			 */
+			if (h1m->flags & H1_MF_CHNK) {
+h1_emit_chunk_size(>obuf, count);
+h1_emit_chunk_crlf(>obuf);
+			}
+
 			total += count;
 			goto out;
 		}
-- 
2.19.2

Re: corruption of data with compression in 1.9-dev10

2018-12-12 Thread Christopher Faulet 

Le 12/12/2018 à 02:08, PiBa-NL a écrit :

Hi List,

Didn't have time yet to bisect when it went wrong. But attached testfile
produces the following output after 3 curl requests at different speeds,
this seems to trigger a problem as the hash of the downloaded content is
nolonger the same as it should be, (in my actual environment its a 2MB
javascript file that comes from a iis server behind haproxy.). Took
already a few hours more than desired to come up with a seemingly
reliable reproduction.
1.9-dev10 is the first one i put on my production environment as i think
release is imminent so it 'should' be pretty stable ;), (yes i know..i
shouldn't assume..) before it was using 1.8.14.. So was quick to revert
to that 1.8 again :).

Using these settings:

      compression algo gzip
      compression type text/html text/plain application/json
application/javascript
      compression offload
When these compression settings are disabled, it completes successfully..

 top   2.4 shell_cmd|  exit 1
 top   2.4 shell_cmd|    fi
 top   2.5 shell_out|File1 all OK
 top   2.5 shell_out|File2 not OK 7798551c02a37ce89c77fc18fc415e5b
 top   2.5 shell_out|File3 not OK 3146c4c9fce4da750558bfd9387ffc3b
 top   2.5 shell_status = 0x0001
 top   2.5 shell_exit not as expected: got 0x0001 wanted 0x
*    top   2.5 RESETTING after ./PB-TEST/ulticompres/b5.vtc
**   h1    2.5 Reset and free h1 haproxy 51853
**   h1    2.5 Wait
**   h1    2.5 Stop HAproxy pid=51853
 h1    2.5 STDOUT poll 0x11
 h1    2.5 Kill(2)=0: No error: 0
**   h1    2.6 WAIT4 pid=51853 status=0x0002 (user 0.253496 sys 0.00)
*    top   2.6 TEST ./PB-TEST/ulticompres/b5.vtc FAILED
#    top  TEST ./PB-TEST/ulticompres/b5.vtc FAILED (2.581) exit=2

haproxy -v
HA-Proxy version 1.9-dev10-3815b22 2018/12/11
Copyright 2000-2018 Willy Tarreau 

Can anyone confirm? Or perhaps even fix ;) Ill try and dig a little more
tomorrow evening :).


Hi Pieter,

There is a bug with the commit 3815b22 when the message is chunked. I 
need to see with Willy how to fix it. BTW, if you revert this commit, it 
should work fine.


Thanks,
--
Christopher