Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

Hi Pieter,

On Wed, Oct 14, 2015 at 01:43:40AM +0200, PiBa-NL wrote:
> Ok got some good news here :).. 1.6.0-release nolonger has the error i 
> encountered.
> 
> The commit below fixed the issue already.
> --
> CLEANUP: cli: ensure we can never double-free error messages
> http://git.haproxy.org/?p=haproxy.git;a=commit;h=6457d0fac304b7bba3e8af13501bf5ecf82bfa67
> --

Great news!

> I was still testing with 1.6-dev7 the fix above came the day after.. 
> Probably your testing with HEAD, which is why it doesn't happen for you. 
> Using snapshots or HEAD is not as easy as just following dev releases.. 
> So i usually stick to those unless i have reason to believe a newer 
> version might fix it already. I should have tested again sooner sorry.. 
> (I actually did test latest snapshot at the moment when i first reported 
> the issue..)

No problem, it's just that we weren't clear on our respective versions,
it's neither the first nor the last time.

> Anyway i burned some more hours on both your and my side than was 
> probably needed.
> One more issue gone :)

Please keep in mind that I got a few segfaults with pipelined requests
(the very large ones), so I'm pretty sure that a few bugs remain, though
less easy to reproduce than the one you were suffering from.

> Thanks for the support!

You're welcome, thanks for your feedback as well :-)

Willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Thierry FOURNIER]

On Tue, 13 Oct 2015 00:46:43 +0200
Willy Tarreau  wrote:

> On Tue, Oct 13, 2015 at 12:32:08AM +0200, PiBa-NL wrote:
> > >Yep so here rqh is in fact req->buf->i and as you noticed it's been
> > >decremented a second time.
> > >
> > >I'm seeing this which I find suspicious in hlua.c :
> > >
> > >   5909
> > >   5910  /* skip the requests bytes. */
> > >   5911  bo_skip(si_oc(si), strm->txn->req.eoh + 2);
> > >
> > >First I don't understand why "eoh+2", I suspect that it's for the CRLF
> > >in which case it's wrong since it can be a lone LF. Second, I'm not
> > >seeing sov being reset afterwards. Could you please just add this
> > >after this line :
> > >
> > >strm->txn->req.next -= strm->txn->req.sov;
> > >strm->txn->req.sov = 0;
> > This did not seem to resolve the issue.
> 
> OK thanks for testing at least!
> 
> > If you have any other idea where it might go wrong please let me know :)
> > Ill try and dig a little further tomorrow evening.
> 
> Unfortunately no I don't have any other idea. At this point I think
> I'll have to discuss with Thierry about this. We're in a situation
> where I know pretty well how HTTP forwarding works, while he knows
> very well how Lua works, but both of us have very unclear idea of
> the other one's part, so that doesn't help much :-/


I'm silently following the thread :) I can't reproducethe issue, I
supposed that is because it appears on FreeBSD, but I suppose that it
will be appear on Linux.

Westerday my laptop were out of battery, so it rebooted. I loose my
browsers with hundreds of internet, that is free some ram and I can
start a freebsd VM without swapping (I dont have a simple life).

I'm currently install a freebsd distrib, hoping that I reproduce the
problem.

Thierry 


> I'm still working on the management doc that I hoped to finish by today
> and seems to take longer, so the release might be deferred to tomorrow,
> maybe in the mean time we'll have the opportunity to find something odd,
> but I prefer not to put my nose there myself or it will further postpone
> the doc and the release which is waiting for it.
> 
> Thanks for your feedback!
> 
> Willy
>  
> 

Unsubscribe from list

[Ricardo]

Hello,

I have been trying to unsubscribe from list a few times but it didn't work.
Please, can somebody check if the account haproxy+unsubscr...@formilux.org is 
working as expected.

Thanks,   

Re: Unsubscribe from list

[Kobus Bensch]

Once on this list you can never leave.

On 13/10/2015 09:30, Ricardo wrote:

Hello,


I have been trying to unsubscribe from list a few times but it didn't 
work.


Please, can somebody check if the account 
haproxy+unsubscr...@formilux.org 
 is working as expected.



Thanks,


--
Kobus Bensch Trustpay Global LTD email signature Kobus Bensch
Senior Systems Administrator
Address:  22 & 24 | Frederick Sanger Road | Guildford | Surrey | GU2 7YD
DDI:  0207 871 3958
Tel:  0207 871 3890
Email: kobus.ben...@trustpayglobal.com 



--


Trustpay Global Limited is an authorised Electronic Money Institution 
regulated by the Financial Conduct Authority registration number 900043. 
Company No 07427913 Registered in England and Wales with registered address 
130 Wood Street, London, EC2V 6DL, United Kingdom.


For further details please visit our website at www.trustpayglobal.com.

The information in this email and any attachments are confidential and 
remain the property of Trustpay Global Ltd unless agreed by contract. It is 
intended solely for the person to whom or the entity to which it is 
addressed. If you are not the intended recipient you may not use, disclose, 
copy, distribute, print or rely on the content of this email or its 
attachments. If this email has been received by you in error please advise 
the sender and delete the email from your system. Trustpay Global Ltd does 
not accept any liability for any personal view expressed in this message.

Re: How to configure frontend/backend for SSL OR Non SSL Backend?

[Daren Sefcik]

I will need to look at your suggestion some more to understand how it will
pass 443 traffic to the backend server, I only see definitions for port 80.
Since I am using pfsense and the haproxy package I will also have to figure
out how to define peers as that does not seem to be available in the web
gui.

thanks

On Mon, Oct 12, 2015 at 8:22 AM, Baptiste  wrote:

> So basically, here is what you want to do:
> peers mypeers
>  # read the doc for the info to store here
>
> frontend ftapp
>  bind :80
>  bind :443
>  mode tcp
>  default_backend bkapp
>
> backend bkapp
>  mode tcp
>  stick-table type ip size 10k peers mypeers
>  stick on src
>  server s1 a.b.c.d check port 80
>  server s2 a.b.c.e check port 80
>
>
> Baptiste
>
>
> On Mon, Oct 12, 2015 at 4:40 PM, Daren Sefcik 
> wrote:
> > humm...not sure I know how to answer that...we have servers that require
> SSL
> > for some requests and not for others. I am not needing to do anything
> other
> > than pass the traffic along, not doing any inspection or verifying of
> cert
> > or anything. I tried to setup a frontend with 2 servers in the backend,
> one
> > with 443 and the other with 80 but that didn't seem to work, like it
> would
> > pick the wrong one to send to.
> >
> > On Mon, Oct 12, 2015 at 7:29 AM, Baptiste  wrote:
> >>
> >> Hi Daren,
> >>
> >> Do you want/need to decipher the traffic when using SSL?
> >>
> >> Baptiste
> >>
> >> On Mon, Oct 12, 2015 at 4:24 PM, Daren Sefcik  >
> >> wrote:
> >> > I am probably totally overlooking something but how do I configure a
> >> > frontend/backend to pass to the same server for both SSL and Non SSL
> >> > requests?  We have server that require ssl for some applications but
> >> > most of
> >> > the time not.
> >> >
> >
> >
>

Re: Unsubscribe from list

[Willy Tarreau]

Hello,

On Tue, Oct 13, 2015 at 10:30:04AM +0200, Ricardo wrote:
> Hello,
> 
> I have been trying to unsubscribe from list a few times but it didn't work.
> Please, can somebody check if the account haproxy+unsubscr...@formilux.org is 
> working as expected.

It definitely works since I'm seeing subscribptions and unsubscriptions
about once a week. Strange.

I've unsubscribed you manually, do not hesitate to ping me again when you
want to resubscribe if you experience the same problem.

Willy

Re: About maxconn and minconn

[Willy Tarreau]

Hi Dmitry,

sorry for the delay, I really didn't have time to analyse the config
you sent me.

A few points below :

On Wed, Oct 07, 2015 at 04:18:20PM +0300, Dmitry Sivachenko wrote:
> Oct  7 08:33:03 srv1 haproxy[77565]: unix:1 [07/Oct/2015:08:33:02.428] 
> MT-front MT_RU_EN-back/ 0/1000/-1/-1/1000 503 212 - - sQ-- 
> 125/124/108/0/0 0/28 "POST /some/url HTTP/1.1"
> (many similar at one moment)
> 
> Common part in these errors is "1000" in Tw and Tt, and "sQ--" termination 
> state.
> 
> Here is the relevant part on my config (I can post more if needed):
> 
> defaults
> balance roundrobin
> maxconn 1
> timeout queue 1s
> fullconn 3000
> default-server inter 5s downinter 1s fastinter 500ms fall 3 rise 1 
> slowstart 60s maxqueue 1 minconn 5 maxconn 150
> 
> backend MT_RU_EN-back
> mode http
> timeout server 30s
> server mt1-34 mt1-34:19016 track MT-back/mt1-34 weight 38
> server mt1-35 mt1-35:19016 track MT-back/mt1-35 weight 38
> 
> 
> So this error log indicates that request was sitting in the queue for timeout 
> queue==1s and his turn did not come.
> 
> In the stats web interface for MT_RU_EN-back backend I see the following 
> numbers:
> 
> Sessions: limit=3000, max=126 (for the whole backend)
> Limit=150, max=5 or 6 (for each server)
> 
> If I understand minconn/maxconn meaning right, each server should accept up 
> to min(150, 3000/18) connections
> 
> So according to stats the load were far from limits.

No, look, the log says there were 108 connections on the backend. This
is important since you're using minconn so you're using dynamic queueing.
This means that the effective limit when handling this request was around
maxconn*currconn/fullconn, which is 150*108/3000 = 5.4 so the limit was
at 5 connections. Thus the limit for this server was indeed reached.

Playing with minconn and fullconn is hard and strongly advised against,
unless you know exactly how to tune it. You must always ensure that a
normal load will be handled without queuing (or with a very small queue),
and that maxconn will quickly be reached to handle high traffic. I tend to
consider that an efficient fullconn is around 10% of the maximum load the
farm may have to deal with (which is the default value IIRC). Regarding
minconn, it's interesting not to set it too low. A good rule of thumb is
to estimate what would happen at 10% of fullconn (1% of the max load).
In your case, at 300 concurrent connections, your servers will accept
15 connections each. I have no idea whether this is enough or not to
handle the load. But let's say you have 4 servers, that's only 60
concurrent connections to process 300 front connections. While it can
be perfectly fine, you may need to increase the queue timeout so that
the requests can wait long enough for a slot. With a 5:1 overbooking and
your 1s queue timeout, that means you expect that the server's average
response time will not go above 200ms. That may be a bit short for some
applications, especially those sensitive to connection count.

Thus I'd suggest that you either lower fullconn or increase minconn, and
in any case that you also increase the queue timeout to cover the worst
overbooking situation with the average server's response time.

During the tuning phase, I'd suggest to *significantly* increase the queue
timeout so that you can observe the connection counts and even the average
response time per connection count, that will help you refine the tuning.

Hoping this helps,
Willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

Hi again :-)

On Tue, Oct 13, 2015 at 06:10:33PM +0200, Willy Tarreau wrote:
> I can't reproduce either unfortunately. I'm seeing some other minor
> issues related to how the closed input is handled and showing that
> pipelining doesn't work (only the first request is handled) but that's
> all I'm seeing I'm sorry.
> 
> I've tried injecting on stats in parallel to the other frontend, I've
> tried with close and keep-alive etc... I tried to change the poller
> just in case you would be facing a race condition, no way :-(
> 
> In general it's good to keep in mind that buffer_slow_realign() is
> called to realign wrapped requests, so that normally means that
> pipelining is needed. But even then for now I can't succeed.

As usual, sending an e-mail scares the bug and it starts to shake the
white flag :-)

So by configuring the buffer size to 1 and sending large 8kB requests,
I'm seeing a random behaviour. First, most of then time I end up with a
stuck session which never ends (no expiration timer set). And from time
to time it may crash. This time it was not in buffer_slow_realign() but
in buffer_insert_line2(), though the problem is the same :

(gdb) up
#2  0x0046e094 in http_header_add_tail2 (msg=0x7ce628, 
hdr_idx=0x7ce5c8, text=0x53b339 "Connection: close", len=17) at 
src/proto_http.c:595
595 bytes = buffer_insert_line2(msg->chn->buf, msg->chn->buf->p + 
msg->eoh, text, len);

(gdb) p msg->eoh
$6 = 8057
(gdb) p *msg->chn->buf
$7 = {p = 0x7f8e7b44bf9e "3456789.123456789\n", 'P' ..., 
size = 10008, i = 0, o = 8058, data = 0x7f8e7b44a024 "GET /1234567"}

(gdb) p msg->chn->buf->p - msg->chn->buf->data
$8 = 8058

As one may notice, since p is already 8kB from the beginning of the buffer
(hence 2kB from the end), writing at p + eoh is definitely wrong. Here we're
having a problem that msg->eoh is wrong or buf->p is wrong.

My opinion here is that buf->p is the wrong one, since we're dealing with a
8kB request, so it should definitely have been realigned. Or maybe it was
stripped and removed from the request buffer with HTTP processing still
enabled.

All this part is still totally unclear to me I'm afraid. I suggest that we
don't rush too fast on lua services and try to fix that during the stable
cycle. I don't want to postpone the release any further for something that
was added very recently and that is not causing any regression to existing
configs.

Best regards,
willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[Willy Tarreau]

Hi guys,

On Tue, Oct 13, 2015 at 10:07:33AM +0200, Thierry FOURNIER wrote:
> I'm silently following the thread :) I can't reproducethe issue, I
> supposed that is because it appears on FreeBSD, but I suppose that it
> will be appear on Linux.
> 
> Westerday my laptop were out of battery, so it rebooted. I loose my
> browsers with hundreds of internet, that is free some ram and I can
> start a freebsd VM without swapping (I dont have a simple life).
> 
> I'm currently install a freebsd distrib, hoping that I reproduce the
> problem.

I can't reproduce either unfortunately. I'm seeing some other minor
issues related to how the closed input is handled and showing that
pipelining doesn't work (only the first request is handled) but that's
all I'm seeing I'm sorry.

I've tried injecting on stats in parallel to the other frontend, I've
tried with close and keep-alive etc... I tried to change the poller
just in case you would be facing a race condition, no way :-(

In general it's good to keep in mind that buffer_slow_realign() is
called to realign wrapped requests, so that normally means that
pipelining is needed. But even then for now I can't succeed.

Willy

Re: [ANNOUNCE] haproxy-1.6.0 now released!

[Willy Tarreau]

On Tue, Oct 13, 2015 at 09:42:25PM +0200, Baptiste wrote:
> Great, amazing!
> Looking forward to 1.7!

Already online :

$ ./haproxy -v 
HA-Proxy version 1.7-dev0 2015/10/13

:-)

Willy

[PATCH] BUILD: install only relevant and existing documentation

[Vincent Bernat]

From: Vincent Bernat 

doc/haproxy-{en,fr}.txt have been removed recently but they were still
referenced in the Makefile. Many other documents have also been
added. Instead of hard-coding a list of documents to install, install
all those in doc/ with some exceptions:

 - coding-style.txt is more for developers
 - gpl.txt and lgpl.txt are usually present at other places (and I would
   have to remove them in the Debian packaging, less work for me)

The documentation in the subdirectories is not installed as it is more
targeted to developers.
---
 Makefile | 7 +--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index a7da80b56fa0..4f0aa8af705b 100644
--- a/Makefile
+++ b/Makefile
@@ -796,9 +796,12 @@ install-man:
install -d "$(DESTDIR)$(MANDIR)"/man1
install -m 644 doc/haproxy.1 "$(DESTDIR)$(MANDIR)"/man1
 
+EXCLUDE_DOCUMENTATION = lgpl gpl coding-style
+DOCUMENTATION = $(filter-out $(EXCLUDE_DOCUMENTATION),$(patsubst 
doc/%.txt,%,$(wildcard doc/*.txt)))
+
 install-doc:
install -d "$(DESTDIR)$(DOCDIR)"
-   for x in configuration architecture haproxy-en haproxy-fr; do \
+   for x in $(DOCUMENTATION); do \
install -m 644 doc/$$x.txt "$(DESTDIR)$(DOCDIR)" ; \
done
 
@@ -811,7 +814,7 @@ install: install-bin install-man install-doc
 
 uninstall:
rm -f "$(DESTDIR)$(MANDIR)"/man1/haproxy.1
-   for x in configuration architecture haproxy-en haproxy-fr; do \
+   for x in $(DOCUMENTATION); do \
rm -f "$(DESTDIR)$(DOCDIR)"/$$x.txt ; \
done
-rmdir "$(DESTDIR)$(DOCDIR)"
-- 
2.6.1

Re: [ANNOUNCE] haproxy-1.6.0 now released!

[Baptiste]

Great, amazing!
Looking forward to 1.7!

Baptiste

Re: HTTP Response Rewriting to Replace Internal IP with FQDN

[Susheel Jalali]

Dear Aleks,

Thank you for your continued help.  As you advised, we did the 
following.  We would appreciate any guidance you could give to solve 
this issue.


(1) We ran haproxy (1.5.14) in debug mode in two use cases for 
Product1.  The debug output for each case is posted below.
(2) Also, as requested, posted are the relevant parts of Tomcat web.xml 
and config.xml.
(3) We are able to access another product (Product2) with similar 
configuration, but the debug output does not have Location header.


Product1 Debug
Scenario 1:  Right external URL shows on the address bar, but the page 
gives error
When we use in backend:   rspirep ^Location:\ 
(https?://)([^:]*)(:[0-9]+)(/.*) Location:\ \4 if hdr_location


Result:
Google Chrome:  ERR_TOO_MANY_REDIRECTS.   The webpage at 
https://coscend.com:8443/Product1/ has resulted in too many redirects. 
Clearing your cookies for this site or allowing third-party cookies may 
fix the problem. If not, it is possibly a server configuration issue and 
not a problem with your computer.


Firefox:  Firefox has detected that the server is redirecting the 
request for this address in a way that will never complete.  This 
problem can sometimes be caused by disabling or refusing to accept cookies.


Scenario 2: Right Page is served, but shows http:// 
on the address bar


When we use in backend:  rspirep ^Location:\ 
(https?://)([^/]*)(:[0-9]+)(.*)$ Location:\ \1coscend.com\3\4 if 
hdr_location

Result:  It should be https://External_URL/Product1/signin?xyz

==
Debug output

Scenario 1:  Right external URL shows on the address bar, but the page 
gives error



Available polling systems :
  epoll : pref=300,  test result OK
  poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

:webapps-frontend.accept(0006)=0009 from [192.168.100.153:57322]
:webapps-frontend.clicls[0009:]
:webapps-frontend.closed[0009:]
0001:webapps-frontend.accept(0006)=0009 from [192.168.100.153:57323]
0001:webapps-frontend.clireq[0009:]: GET /Product1/ HTTP/1.1
0001:webapps-frontend.clihdr[0009:]: Host: coscend.com:8443
0001:webapps-frontend.clihdr[0009:]: Connection: keep-alive
0001:webapps-frontend.clihdr[0009:]: Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
0001:webapps-frontend.clihdr[0009:]: 
Upgrade-Insecure-Requests: 1
0001:webapps-frontend.clihdr[0009:]: User-Agent: Mozilla/5.0 
(Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/45.100.2444.201 Safari/537.36
0001:webapps-frontend.clihdr[0009:]: Accept-Encoding: gzip, 
deflate, sdch
0001:webapps-frontend.clihdr[0009:]: Accept-Language: 
en-US,en;q=0.8
0001:webapps-frontend.clihdr[0009:]: Cookie: 
JSESSIONID=6CD9160AEB4B2FC16AC392BE6479F3D9; 
express_sid=s%3A4wgQy_swupi3Nwxj8PS-Ly-ylGRU92iX.02vFzzzMQz51sLlz1A2A3Ecob28KL6mT79o0xS3Idmg

0001:subdomain_p1.srvrep[0009:000a]: HTTP/1.1 302 Found
0001:subdomain_p1.srvhdr[0009:000a]: Server: Apache-Coyote/1.1
0001:subdomain_p1.srvhdr[0009:000a]: Location: 
http:///Product1/

0001:subdomain_p1.srvhdr[0009:000a]: Transfer-Encoding: chunked
0001:subdomain_p1.srvhdr[0009:000a]: Date: Tue, 13 Oct 2015 19:10:19 GMT
0001:subdomain_p1.srvhdr[0009:000a]: Connection: close

+++
Scenario 2:
+++
Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

0002:webapps-frontend.accept(0006)=000b from [192.168.100.153:57304]
:webapps-frontend.accept(0006)=0009 from [192.168.100.153:57302]
0002:webapps-frontend.clicls[000b:]
0002:webapps-frontend.closed[000b:]
0001:webapps-frontend.accept(0006)=000a from [192.168.100.153:57303]
:webapps-frontend.clicls[0009:]
:webapps-frontend.closed[0009:]
0001:webapps-frontend.clicls[000a:]
0001:webapps-frontend.closed[000a:]
0003:webapps-frontend.accept(0006)=0009 from [192.168.100.153:57305]
0003:webapps-frontend.clireq[0009:]: GET /Product1/ HTTP/1.1
0003:webapps-frontend.clihdr[0009:]: Host: coscend.com:8443
0003:webapps-frontend.clihdr[0009:]: Connection: keep-alive
0003:webapps-frontend.clihdr[0009:]: Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
0003:webapps-frontend.clihdr[0009:]: 
Upgrade-Insecure-Requests: 1
0003:webapps-frontend.clihdr[0009:]: User-Agent: Mozilla/5.0 
(Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) 
Chrome/45.100.2444.201 Safari/537.36
0003:webapps-frontend.clihdr[0009:]: Accept-Encoding: gzip, 
deflate, sdch
0003:webapps-frontend.clihdr[0009:]: 

Re: OPTIM : IPv6 literal address parsing

[Mildis]

Le 2015-10-10 15:49, Willy Tarreau a écrit :

Hi,



@@ -856,11 +883,28 @@ struct sockaddr_storage *str2sa_range(const char
*str, int *low, int *high, char
else { /* IPv4 and IPv6 */
int use_fqdn = 0;

-   port1 = strrchr(str2, ':');
-   if (port1)
-   *port1++ = '\0';
-   else
+   /* IPv6 wildcard */
+   if (!strcmp(str2, "::")) {
+   port1 = "";
+   }

You're changing the parser here, because it now accepts "::" in a place
where a port was expected.


May I ask : isn’t the parser just a parser ? Making the caller’s 
responsibility to use or not the ports returned either in the 
sockaddr_storage or the 2 int depending on the context ? (‘bind’ 
requires a port but ‘server’’s port is optional)

Is str2sa_range aware of the context it’s called ?

Regards,
Mildis

Re: Unsubscribe from list

[Rich Vigorito]

Hi have had the same experience and was not unsubscribed. CAn you be 
unsubscribed manually as well please. 

From: Willy Tarreau 
Sent: Tuesday, October 13, 2015 7:22 AM
To: Ricardo
Cc: haproxy@formilux.org
Subject: Re: Unsubscribe from list

Hello,

On Tue, Oct 13, 2015 at 10:30:04AM +0200, Ricardo wrote:
> Hello,
>
> I have been trying to unsubscribe from list a few times but it didn't work.
> Please, can somebody check if the account haproxy+unsubscr...@formilux.org is 
> working as expected.

It definitely works since I'm seeing subscribptions and unsubscriptions
about once a week. Strange.

I've unsubscribed you manually, do not hesitate to ping me again when you
want to resubscribe if you experience the same problem.

Willy

[ANNOUNCE] haproxy-1.6.0 now released!

[Willy Tarreau]

Hi everyone,

Sixteen months after haproxy 1.5.0 was released, here comes 1.6.0.
We've done a much better job this time and despite some code being
pushed after the freeze (don't do that again or I'll bite), overall
the process has been doing quite well.

Recently for our products I had to walk over the whole changelog to
enumerate all the main features we created into 1.6 since 1.5.0, and I
was quite impressed in the end, and realized that I would be unable to
say what's new in 1.6 by citing just a few features as there are a lot.
Here's a simplified version, I won't go into details because it would
take a very long e-mail (and a lot of time, this evening I'm tired of
typing long sentences) :

  Resources management :
  - dynamic buffer allocation
  - automatic maxconn setting
  - peers disable
  - peers process binding

  Configuration management :
  - config support for quotes
  - config environment variables

  Notification / reporting :
  - stats clean encoding in CSV
  - mailers
  - log-tag
  - log-format tags %H*

  Server state management :
  - state keeping across reload
  - multiple redispatch
  - dns resolution
  - CLI server address change
  - external checker
  - tcp-check comment

  High-level processing / scripting :
  - lua
  - variables
  - gpt0
  - 64-bit integer in samples
  - more flexible sample management
  - declared captures
  - http-response redirect
  - http-request capture
  - {url,body}_param supports any param
  - device identification
  - option http-buffer-request
  - arithmetic fetches and convs
  - new fetch/conv (query, json, field, ...)
  - http-request set-{path,query,method,uri}
  - table lookups converters
  - header manipulation on status code 101

  Client-facing SSL/TLS :
  - ECDSA client support detection
  - SSL cert forgery on the fly
  - TLS cert transparency (SCTL)
  - TLS ticket key load from file/CLI
  - custom SSL DH params

  Server-facing SSL/TLS :
  - specify TLS sni to server
  - no-ssl-reuse

  Performance :
  - HTTP server connection sharing
  - use pcre-study by default
  - stateless gzip/deflate compression
  - compression of 201-203
  - pattern cache

  Reliability :
  - peers v2
  - TCP_USER_TIMEOUT

  Integration :
  - linux namespaces
  - http-request set-src
  - HTTP/0.9 disabled by default
  - RTSP basic compatibility
  - option http-ignore-probes
  - max syslog line length

  Documentation :
  - added more documentation (intro, management, lua)
  - removed obsolete and confusing docs
  - removed lots of obsolete config files

Just to illustrate how things went compared to version 1.5, in almost
16 months, we merged 1156 commits from 59 people, compared to 2463
commits from 76 people in 49 months. That's 50% more commits per month
and 138% more contributors per month. This proves the process is working
and that by having a faster rhythm we can avoid to bore contributors.

Now we have some official subsystem maintainers who have authority on
their parts and who are responsible for reviewing the code submitted in
their areas. That doesn't mean that I won't accept such patches anymore,
it just means that patches will take less time to get merged by avoiding
the back-and-forth exchanges we were doing because contributors didn't
know who to send them to nor how to proceed, and that if these persons
say "no" to some changes, I'll support them.

At the moment we have Thierry Fournier who's in charge of Lua and maps+
patterns, Emeric Brun for SSL and peers, Baptiste Assmann for the DNS
resolvers and tcp-checks, Simon Horman for mailers and external checks,
Cyril Bonté for everything related to the doc format and structure and
for stable backports as he's the stable co-maintainer with me, and David
Carlier for the DeviceAtlas device identification. Thanks to them! All of
them are doing this on their spare time. Please respect their code, their
time, and don't try to bypass them if you want to apply changes in their
areas, that will avoid a number of bugs.

I sincerely hope that this organisation, as well as the documentation that
we're progressively adding, will help get better quality contributions over
time and that code will take less time to be reviewed and will be merged
faster.

We got very good bug reports again during this cycle and the talented
people you're used to see jumping on any bug report have done an amazing
job again, so sincere thanks to them for this, because we'd have many
more bugs without their help!

By the way, I was hoping to be able to announce the 1000th subscriber on the
list by the release, but no, this morning we were 955 permanent subscribers
on this list, that's not bad anyway, especially considering that it's an open
list and that a number of people subscribe to ask a questionn and unsubscribe
afterwards :-)

Regarding the quality of this release, I think it is reasonably safe as long
as you don't blindly use the last-minute additions. Specifically we know that
the use-service feature is still not 100% reliable in HTTP 

Re: Unsubscribe from list

[Willy Tarreau]

On Tue, Oct 13, 2015 at 06:24:07PM +, Rich Vigorito wrote:
> Hi have had the same experience and was not unsubscribed. CAn you be
> unsubscribed manually as well please. 

Sure, done now.

Cheers,
Willy

Re: HTTP Response Rewriting to Replace Internal IP with FQDN

[Aleksandar Lazic]

Dear Susheel Jalali.

Am 13-10-2015 22:20, schrieb Susheel Jalali:

Dear Aleks,


[snipp]


++
Tomcat’s web.xml

In web.xml, the context parameter is:


globalScope
default


and the filter mapping for the application is:

Product1Application
/*


++
Tomcat’s config.xml
++


1937
8443
no
5081
http
none
Product1
Product1
/Product1/


Sorry but this could not be the full web.xml and server.xml!

Tomcat will not be able to start with this files!

Du you try to proxy the RED5 server?!

https://github.com/Red5/red5-server

BR
Aleks

Re: [ANNOUNCE] haproxy-1.6.0 now released!

[Aleksandar Lazic]

Hi.

Am 13-10-2015 21:50, schrieb Willy Tarreau:

On Tue, Oct 13, 2015 at 09:42:25PM +0200, Baptiste wrote:

Great, amazing!
Looking forward to 1.7!


Already online :

$ ./haproxy -v
HA-Proxy version 1.7-dev0 2015/10/13

:-)


As always fast, quite fast, flashy ;-)

BR Aleks

Re: [PATCH] BUILD: install only relevant and existing documentation

[Willy Tarreau]

On Tue, Oct 13, 2015 at 10:20:55PM +0200, Vincent Bernat wrote:
> From: Vincent Bernat 
> 
> doc/haproxy-{en,fr}.txt have been removed recently but they were still
> referenced in the Makefile. Many other documents have also been
> added. Instead of hard-coding a list of documents to install, install
> all those in doc/ with some exceptions:
> 
>  - coding-style.txt is more for developers
>  - gpl.txt and lgpl.txt are usually present at other places (and I would
>have to remove them in the Debian packaging, less work for me)
> 
> The documentation in the subdirectories is not installed as it is more
> targeted to developers.

Thank you Vincent. I once again got trapped by "git grep" which I always
believe looks in the whole project while it only looks in the current
sub-directory (I was in doc/), so I found the references in haproxy.spec
but not the Makefile.

Patch applied.

Willy

Re: HTTP Response Rewriting to Replace Internal IP with FQDN

[Aleksandar Lazic]

Am 13-10-2015 23:36, schrieb Aleksandar Lazic:

Dear Susheel Jalali.

Am 13-10-2015 22:20, schrieb Susheel Jalali:

Dear Aleks,


[snipp]


++
Tomcat’s web.xml

In web.xml, the context parameter is:


globalScope
default


and the filter mapping for the application is:

Product1Application
/*


++
Tomcat’s config.xml
++


1937
8443
no
5081
http
none
Product1
Product1
/Product1/


Sorry but this could not be the full web.xml and server.xml!

Tomcat will not be able to start with this files!

Du you try to proxy the RED5 server?!

https://github.com/Red5/red5-server


If yes what's in your red5.properties ?

https://github.com/Red5/red5-server/blob/master/src/main/server/conf/red5.properties

From my point of view it looks like you need to set the http.* variables 
to the right values.


https://github.com/Red5/red5-server/blob/3a6885433218ce2070b9064e195fc2ffde031c88/src/main/java/org/red5/server/net/servlet/RedirectHTTPServlet.java#L42

e. g.: https.port=1443 or what you want


BR
Aleks

[PATCH] DOC: specify that stats socket doc (section 9.2) is in management

[Kevin Decherf]

Commit 44aed90ce102c4136a5eda66d541f6fa79e141e8 moved the stats socket
documentation from config to management but the remaining references to
section 9.2 were not updated; improve it to be less confusing.

Signed-off-by: Kevin Decherf 
---
 doc/configuration.txt | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/doc/configuration.txt b/doc/configuration.txt
index b509238..be7e78f 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -859,7 +859,8 @@ stats socket [|] [param*]
   Binds a UNIX socket to  or a TCPv4/v6 address to .
   Connections to this socket will return various statistics outputs and even
   allow some commands to be issued to change some runtime settings. Please
-  consult section 9.2 "Unix Socket commands" for more details.
+  consult section 9.2 "Unix Socket commands" of Management Guide for more
+  details.
 
   All parameters supported by "bind" lines are supported, for instance to
   restrict access to some users or their access rights. Please consult
@@ -4233,7 +4234,7 @@ load-server-state-from-file { global | local | none }
 
   The format of the file is versionned and is very specific. To understand it,
   please read the documentation of the "show servers state" command (chapter
-  9.2).
+  9.2 of Management Guide).
 
   Arguments:
 global load the content of the file pointed by the global directive
-- 
2.5.1

Re: [PATCH] DOC: specify that stats socket doc (section 9.2) is in management

[Willy Tarreau]

On Tue, Oct 13, 2015 at 11:26:44PM +0200, Kevin Decherf wrote:
> Commit 44aed90ce102c4136a5eda66d541f6fa79e141e8 moved the stats socket
> documentation from config to management but the remaining references to
> section 9.2 were not updated; improve it to be less confusing.

Ah crap! I paid attention to this but still missed them :-(
Thanks Kevin, patch applied.

Willy

Re: core dump, lua service, 1.6-dev6 ss-20150930

[PiBa-NL]

Hi Willy, Thierry, others,

Op 13-10-2015 om 18:29 schreef Willy Tarreau:

Hi again :-)

On Tue, Oct 13, 2015 at 06:10:33PM +0200, Willy Tarreau wrote:

I can't reproduce either unfortunately. I'm seeing some other minor
issues related to how the closed input is handled and showing that
pipelining doesn't work (only the first request is handled) but that's
all I'm seeing I'm sorry.

I've tried injecting on stats in parallel to the other frontend, I've
tried with close and keep-alive etc... I tried to change the poller
just in case you would be facing a race condition, no way :-(

In general it's good to keep in mind that buffer_slow_realign() is
called to realign wrapped requests, so that normally means that
pipelining is needed. But even then for now I can't succeed.

As usual, sending an e-mail scares the bug and it starts to shake the
white flag :-)

So by configuring the buffer size to 1 and sending large 8kB requests,
I'm seeing a random behaviour. First, most of then time I end up with a
stuck session which never ends (no expiration timer set). And from time
to time it may crash. This time it was not in buffer_slow_realign() but
in buffer_insert_line2(), though the problem is the same :

(gdb) up
#2  0x0046e094 in http_header_add_tail2 (msg=0x7ce628, hdr_idx=0x7ce5c8, 
text=0x53b339 "Connection: close", len=17) at src/proto_http.c:595
595 bytes = buffer_insert_line2(msg->chn->buf, msg->chn->buf->p + 
msg->eoh, text, len);

(gdb) p msg->eoh
$6 = 8057
(gdb) p *msg->chn->buf
$7 = {p = 0x7f8e7b44bf9e "3456789.123456789\n", 'P' ..., size = 10008, 
i = 0, o = 8058, data = 0x7f8e7b44a024 "GET /1234567"}

(gdb) p msg->chn->buf->p - msg->chn->buf->data
$8 = 8058

As one may notice, since p is already 8kB from the beginning of the buffer
(hence 2kB from the end), writing at p + eoh is definitely wrong. Here we're
having a problem that msg->eoh is wrong or buf->p is wrong.

My opinion here is that buf->p is the wrong one, since we're dealing with a
8kB request, so it should definitely have been realigned. Or maybe it was
stripped and removed from the request buffer with HTTP processing still
enabled.

All this part is still totally unclear to me I'm afraid. I suggest that we
don't rush too fast on lua services and try to fix that during the stable
cycle. I don't want to postpone the release any further for something that
was added very recently and that is not causing any regression to existing
configs.

Best regards,
willy

Ok got some good news here :).. 1.6.0-release nolonger has the error i 
encountered.


The commit below fixed the issue already.
--
CLEANUP: cli: ensure we can never double-free error messages
http://git.haproxy.org/?p=haproxy.git;a=commit;h=6457d0fac304b7bba3e8af13501bf5ecf82bfa67
--

I was still testing with 1.6-dev7 the fix above came the day after.. 
Probably your testing with HEAD, which is why it doesn't happen for you. 
Using snapshots or HEAD is not as easy as just following dev releases.. 
So i usually stick to those unless i have reason to believe a newer 
version might fix it already. I should have tested again sooner sorry.. 
(I actually did test latest snapshot at the moment when i first reported 
the issue..)


Anyway i burned some more hours on both your and my side than was 
probably needed.

One more issue gone :)

Thanks for the support!

PiBa-NL

[SPAM] Mailbox Warning!!! (haproxy@formilux.org)

[e-Mailbox Administrator]

  
  This message notification was sent to haproxy@formilux.org
 
 The safety of your E-mail account (haproxy@formilux.org) is important. 
 
 Disk Usage: 95% 
   Your Mailbox haproxy@formilux.org is almost full.   
  Note!!! You wont be able to Receive or Send new messages.
 
 Pleaseclick here to increase your storage size automatically now. 
 
 Warning!!! please do not ignore message to avoid email account 
(haproxy@formilux.org) being closed.
 
 Thanks, 
 E-mail Administrator.