Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
Just for the few who have already downloaded it, I have re-uploaded
the snapshot with a fix (I failed my attempt at automatically renaming
it so it ended up with the same name).

There was a bug affecting the combination of accept-proxy + ssl which
I just fixed.

Regards,
Willy




Re: HAProxy with native SSL support !

2012-09-04 Thread Hervé COMMOWICK
What a great news !

Let's go testing on internal applications.

Congrats to the Exceliance team !

Hervé.

On 09/04/2012 08:12 AM, Willy Tarreau wrote:
 Just for the few who have already downloaded it, I have re-uploaded
 the snapshot with a fix (I failed my attempt at automatically renaming
 it so it ended up with the same name).
 
 There was a bug affecting the combination of accept-proxy + ssl which
 I just fixed.
 
 Regards,
 Willy
 
 

-- 
Hervé COMMOWICK
Ingénieur systèmes et réseaux.

http://www.rezulteo.com
by Lizeo Online Media Group http://www.lizeo-online-media-group.com/
42 quai Rambaud - 69002 Lyon (France) ⎮ ☎ +33 (0)4 63 05 95 30



Re: HAProxy with native SSL support !

2012-09-04 Thread Justin Karneges
On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote:
 After several months of efforts by the Exceliance team, we managed to
 rework all the buffer and connection layers in order to get SSL working
 on both sides of HAProxy.

Very cool.

Since HAProxy is event-driven, is anything done to avoid expensive crypto 
operations from blocking the event loop? Maybe that is what you intend for 
remote sockets ?

Justin

Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
On Mon, Sep 03, 2012 at 11:21:51PM -0700, Justin Karneges wrote:
 On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote:
  After several months of efforts by the Exceliance team, we managed to
  rework all the buffer and connection layers in order to get SSL working
  on both sides of HAProxy.
 
 Very cool.
 
 Since HAProxy is event-driven, is anything done to avoid expensive crypto 
 operations from blocking the event loop? Maybe that is what you intend for 
 remote sockets ?

Exactly !

That said, as I've always said, the problem is not the relative cost of
crypto but the computation time. On small devices, if an operation can
take 10ms, it clearly is a problem. However on larger devices, if no
operation takes more than 100 microseconds, then the added latency is
minimal compared to system latency. That does not mean it's not a problem,
just that when you're at 100% CPU, you might still have enough scheduling
capacity to process the rest of the traffic without too much impact.

But clearly when we can offload SSL to other processes, that will be much
better. This can already be done BTW, but you have to chain proxies. For
instance :

  global
 nbproc 4

  listen front
 bind-process 1,2
 bind :443 ssl haproxy.pem
 server http 127.0.0.1: send-proxy

  listen http
 bind-process 3
 bind 127.0.0.1: accept-proxy
 mode http
 server back 127.0.0.1:8443 send-proxy

  listen back
 bind-process 4
 bind 127.0.0.1:8443 accept-proxy
 server out 192.168.0.1:443 ssl

You see, https is accepted on processes 1 and 2, which decipher the
traffic and pass it to process 3, preserving IP information. This
process does the HTTP handling and passes the traffic to process 4
which will re-cipher it before going to the server.

Admittedly this is ugly and that's why we'd like to implement the
remote sockets soon, so that you can write something like this :

  listen http
 bind-process 3
 bind :443 ssl haproxy.pem process 1,2
 mode http
 server out 192.168.0.1:443 ssl process 4

Also, using socketpair() internally will be much faster and have a lower
overhead than TCP connect/accept.

Regards,
Willy




Re: HAProxy with native SSL support !

2012-09-04 Thread Aleksandar Lazic

Hi Willy,

congratulations to the whole Team.

Thanks for this feature, now the SSL-chain is much simpler ;-)

BR
Aleks

Am 04-09-2012 01:37, schrieb Willy Tarreau:

Hi all,

today is a great day (could say night considering the time I'm 
posting) !


After several months of efforts by the Exceliance team, we managed to
rework all the buffer and connection layers in order to get SSL 
working

bon both sides of HAProxy.

The code is still in preview, we can't break it anymore but 
considering
that we've fixed some bugs today, I'm sure that some still remain in 
the
100+ patches and 16000 lines of patches this work required (not 
counting

the many ones that were abandonned or re-merged multiple times).

The code is still going to change because we're getting closer to 
something
which will allow outgoing connections to be reused, resulting in 
keep-alive

on both sides. But not yet, be patient.

What's done right now ?

1) connections

Connections are independant entities which can be instanciated 
without
allocating a full session and its buffers. Connections are 
responsible
for handshakes and control, and pass data to buffers. 
Connection-level
TCP-request rules, the PROXY protocol and SSL handshakes are 
processed

at the connection level.

2) buffers

buffers have been split in three: channel (the tube where the data 
flows),
buffer (where data is temporarily stored for analysis or forwarding) 
and
optionally the pipe (stored in kernel area for forwarding only). New 
buffers
only handle data without consideration for what it's used for. Health 
checks

are currently being migrated to use this with connections.

3) data I/O

data I/O are now performed between a connection and a buffer. We have
two data-layer operations now : raw and ssl. It is very easy to add
new ones now, we're even wondering whether it would make sense to 
write

one dedicated to yassl in native mode (without the openssl API).

4) socket I/O

at the moment we only support normal sockets, but the design 
considered
remote sockets so that we could off-load heavy processing to 
external

processes (eg: HTTP on one process, SSL on two other). Remote sockets
have not been started yet but surely will. SHMs have also been 
considered

to emulate sockets.

5) configuration

Configuration has been extended to support the ssl keyword on 
bind lines

and on server lines. For both, the syntax is :

... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1]

cert.pem is a PEM file made by concatenating the .crt and the 
.key of a

certificate.

eg:   bind :443 ssl /etc/haproxy/pub.pem
  server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1

6) session management

SSL sessions are stored in a shared memory cache, allowing haproxy to 
run
with nbproc  1 and still work correctly. This is the session cache 
we
developped for stunnel then stud, it was time to adopt it in haproxy. 
It's
so fast that we don't use openssl's cache at all, since even at one 
single

process, it's at least as fast.

7) other

A lot remains to be done, mainly some of the aforementionned 
structres

are still included in other ones, which simplified the split Once all
the work is over, we should end up with less memory used per 
connection.

This is important to better handle DDoS.


At the moment, everything we could try seems to work fine. The SSL 
stacks
well on top of the PROXY protocol, which is very important to build 
SSL
offload farms (I'm sure Baptiste will want to write a blog article on 
the

subject of using sub-$1000 machines to build large 100k+tps farms).
Stats work over https too. Right now we're missing ACLs to match 
whether
the traffic was SSL or clear, as well as logs. Both can be worked 
around
by using distinct bind lines or even frontends. The doc is still 
clearly

lacking, but we think that the config will change a little bit.

Only the GNU makefile was updated, neither the BSD nor OSX were, 
they're
a little trickier. If someone with one of these systems wants to 
update

them, I'll happily accept the patches.

What else ? Ah yes, 4k. You're there wondering about the results. 
4000 SSL
connections per second and 300 Mbps is what we got out of a dual-core 
Atom
D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading 
was
enabled) :-) This is a bit more than stud and obviously much better 
than
stunnel (which doesn't scale to more than a few hundred connections 
before

the performance quickly drops).

And older tests seem to indicate that with YaSSL we can get 30-40% 
more,
maybe even more. We need to work with the YaSSL guys to slightly 
improve

their cache management before this can become a default build option.

Enough speaking, for those who want to test or even have the hardware 
to

run more interesting benchmarks, the code was merged into the master
branch and is in today's snapshot (20120904) here :

http://haproxy.1wt.eu/download/1.5/src/snapshot/

Build it by passing USE_OPENSSL=1 on the make command

Re: HAProxy with native SSL support !

2012-09-04 Thread Guillaume Castagnino
, for those who want to test or even have the hardware
 to run more interesting benchmarks, the code was merged into the
 master branch and is in today's snapshot (20120904) here :
 
 http://haproxy.1wt.eu/download/1.5/src/snapshot/
 
 Build it by passing USE_OPENSSL=1 on the make command line. You
 should also include support for linux-2.6 options for better results
 :
 
make TARGET=linux2628 USE_OPENSSL=1
 
 If all goes well by the end of the week, I'll issue -dev12, but I
 expect that we'll have some bugs to fix till then.
 
 BTW, be very careful, openssl is a memory monster. We counted about
 80kB per connection for haproxy+ssl, this is 800 MB for only 10k
 connections! And remember, this is still beta-quality code. Don't
 blindly put this in production (eventhough I did it on 1wt.eu :
 https://demo.1wt.eu/). You have been warned!
 
 Please use the links below :
 site index  : http://haproxy.1wt.eu/
 sources : http://haproxy.1wt.eu/download/1.5/src/snapshot/
 changelog   :
 http://haproxy.1wt.eu/download/1.5/src/snapshot/CHANGELOG Exceliance 
 : http://www.exceliance.fr/en/
 
 Have a lot of fun and please report your success/failures,
 Willy
-- 
Guillaume Castagnino
ca...@xwing.info / guilla...@castagnino.org




Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
Hi,

On Tue, Sep 04, 2012 at 09:12:53AM +0200, Guillaume Castagnino wrote:
 Hi,
 
 Great news !
 Just one question: is SNI support planned ? This would be great to allow 
 one certificate per named vhost.

Yes it's planned but not done yet. Emeric sees how to implement this but
we wanted to propose an early pre-release so that we can get feedback while
we're developing.

 I'm currently stuck with nginx for the SSL layer because of this feature 
 (I know that stunnel and stud recently get this feature, but not yet 
 tested). This would allow me to strip one layer from my setup ;)

I understand :-)

Cheers,
Willy




Re: HAProxy with native SSL support !

2012-09-04 Thread sami.djef...@advertstream.com
 the hardware
to run more interesting benchmarks, the code was merged into the
master branch and is in today's snapshot (20120904) here :

 http://haproxy.1wt.eu/download/1.5/src/snapshot/

Build it by passing USE_OPENSSL=1 on the make command line. You
should also include support for linux-2.6 options for better results
:

make TARGET=linux2628 USE_OPENSSL=1

If all goes well by the end of the week, I'll issue -dev12, but I
expect that we'll have some bugs to fix till then.

BTW, be very careful, openssl is a memory monster. We counted about
80kB per connection for haproxy+ssl, this is 800 MB for only 10k
connections! And remember, this is still beta-quality code. Don't
blindly put this in production (eventhough I did it on 1wt.eu :
https://demo.1wt.eu/). You have been warned!

Please use the links below :
 site index  : http://haproxy.1wt.eu/
 sources : http://haproxy.1wt.eu/download/1.5/src/snapshot/
changelog   :
http://haproxy.1wt.eu/download/1.5/src/snapshot/CHANGELOG Exceliance
 : http://www.exceliance.fr/en/

Have a lot of fun and please report your success/failures,
Willy


Great news willy,

Can't wait for testing and benchmarking the performance .
Great work and congratulation to the whole team.

Regards,

--

*Sami DJEFFAL*
Directeur du p�le Ing�nierie Syst�me Linux
http://www.adthinkmedia.com
Bourse de Paris : ALADM
Email : s...@advertstream.com mailto:s...@advertstream.com
*Tel : +33(0) 4 78 38 45 20*
Fax : +33(0) 4 78 42 25 54
79, rue Fran�ois MERMET
69160 TASSIN - FRANCE


/Groupe media 100% internet structuré autour 2 métiers complémentaires :
l'édition de sites et la monétisation d'audience/




Re: HAProxy with native SSL support !

2012-09-04 Thread Duncan Hall

On 04/09/12 09:37, Willy Tarreau wrote:


Have a lot of fun and please report your success/failures,
Willy




Small issue when compiling on CentOS 5.8 64bit against RPM versions of 
openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following:


make TARGET=linux2628 USE_OPENSSL=1
gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing 
-DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY 
-DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL 
-DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX 
-DCONFIG_HAPROXY_VERSION=\1.5-dev11\ 
-DCONFIG_HAPROXY_DATE=\2012/06/04\ \

-DBUILD_TARGET='linux2628' \
-DBUILD_ARCH='' \
-DBUILD_CPU='generic' \
-DBUILD_CC='gcc' \
-DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \
-DBUILD_OPTIONS='' \
-c -o src/haproxy.o src/haproxy.c
In file included from src/haproxy.c:83:
include/proto/proto_http.h:111: error: conflicting types for ‘error_message’
/usr/include/et/com_err.h:37: error: previous declaration of 
‘error_message’ was here

make: *** [src/haproxy.o] Error 1





Re: HAProxy with native SSL support !

2012-09-04 Thread Ricardo Fraile
Great!

Thanks Willy,




 De: Willy Tarreau w...@1wt.eu
Para: haproxy@formilux.org 
Enviado: Martes 4 de septiembre de 2012 1:37
Asunto: HAProxy with native SSL support !
 
Hi all,

today is a great day (could say night considering the time I'm posting) !

After several months of efforts by the Exceliance team, we managed to
rework all the buffer and connection layers in order to get SSL working
bon both sides of HAProxy.

The code is still in preview, we can't break it anymore but considering
that we've fixed some bugs today, I'm sure that some still remain in the
100+ patches and 16000 lines of patches this work required (not counting
the many ones that were abandonned or re-merged multiple times).

The code is still going to change because we're getting closer to something
which will allow outgoing connections to be reused, resulting in keep-alive
on both sides. But not yet, be patient.

What's done right now ?

1) connections

Connections are independant entities which can be instanciated without
allocating a full session and its buffers. Connections are responsible
for handshakes and control, and pass data to buffers. Connection-level
TCP-request rules, the PROXY protocol and SSL handshakes are processed
at the connection level.

2) buffers

buffers have been split in three: channel (the tube where the data flows),
buffer (where data is temporarily stored for analysis or forwarding) and
optionally the pipe (stored in kernel area for forwarding only). New buffers
only handle data without consideration for what it's used for. Health checks
are currently being migrated to use this with connections.

3) data I/O

data I/O are now performed between a connection and a buffer. We have
two data-layer operations now : raw and ssl. It is very easy to add
new ones now, we're even wondering whether it would make sense to write
one dedicated to yassl in native mode (without the openssl API).

4) socket I/O

at the moment we only support normal sockets, but the design considered
remote sockets so that we could off-load heavy processing to external
processes (eg: HTTP on one process, SSL on two other). Remote sockets
have not been started yet but surely will. SHMs have also been considered
to emulate sockets.

5) configuration

Configuration has been extended to support the ssl keyword on bind lines
and on server lines. For both, the syntax is :

    ... ssl cert.pem [ciphers suite] [nosslv3] [notlsv1]

    cert.pem is a PEM file made by concatenating the .crt and the .key of a
    certificate.

    eg:   bind :443 ssl /etc/haproxy/pub.pem
          server local 192.168.0.1:443 ssl ciphers EXPORT40 notlsv1

6) session management

SSL sessions are stored in a shared memory cache, allowing haproxy to run
with nbproc  1 and still work correctly. This is the session cache we
developped for stunnel then stud, it was time to adopt it in haproxy. It's
so fast that we don't use openssl's cache at all, since even at one single
process, it's at least as fast.

7) other

A lot remains to be done, mainly some of the aforementionned structres
are still included in other ones, which simplified the split Once all
the work is over, we should end up with less memory used per connection.
This is important to better handle DDoS.


At the moment, everything we could try seems to work fine. The SSL stacks
well on top of the PROXY protocol, which is very important to build SSL
offload farms (I'm sure Baptiste will want to write a blog article on the
subject of using sub-$1000 machines to build large 100k+tps farms).
Stats work over https too. Right now we're missing ACLs to match whether
the traffic was SSL or clear, as well as logs. Both can be worked around
by using distinct bind lines or even frontends. The doc is still clearly
lacking, but we think that the config will change a little bit.

Only the GNU makefile was updated, neither the BSD nor OSX were, they're
a little trickier. If someone with one of these systems wants to update
them, I'll happily accept the patches.

What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL
connections per second and 300 Mbps is what we got out of a dual-core Atom
D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was
enabled) :-) This is a bit more than stud and obviously much better than
stunnel (which doesn't scale to more than a few hundred connections before
the performance quickly drops).

And older tests seem to indicate that with YaSSL we can get 30-40% more,
maybe even more. We need to work with the YaSSL guys to slightly improve
their cache management before this can become a default build option.

Enough speaking, for those who want to test or even have the hardware to
run more interesting benchmarks, the code was merged into the master
branch and is in today's snapshot (20120904) here :

    http://haproxy.1wt.eu/download/1.5/src/snapshot/

Build it by passing USE_OPENSSL=1 on the make command line. You should
also

Re: HAProxy with native SSL support !

2012-09-04 Thread Baptiste
All,

A small howto to play with it can be found here:
http://blog.exceliance.fr/2012/09/04/howto-ssl-native-in-haproxy/

cheers



Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
On Tue, Sep 04, 2012 at 05:56:14PM +1000, Duncan Hall wrote:
 On 04/09/12 09:37, Willy Tarreau wrote:
 
 Have a lot of fun and please report your success/failures,
 Willy
 
 
 
 Small issue when compiling on CentOS 5.8 64bit against RPM versions of 
 openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following:
 
 make TARGET=linux2628 USE_OPENSSL=1
 gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing 
 -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY 
 -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL 
 -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX 
 -DCONFIG_HAPROXY_VERSION=\1.5-dev11\ 
 -DCONFIG_HAPROXY_DATE=\2012/06/04\ \
 -DBUILD_TARGET='linux2628' \
 -DBUILD_ARCH='' \
 -DBUILD_CPU='generic' \
 -DBUILD_CC='gcc' \
 -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \
 -DBUILD_OPTIONS='' \
 -c -o src/haproxy.o src/haproxy.c
 In file included from src/haproxy.c:83:
 include/proto/proto_http.h:111: error: conflicting types for ?error_message?
 /usr/include/et/com_err.h:37: error: previous declaration of 
 ?error_message? was here
 make: *** [src/haproxy.o] Error 1

This is very strange, this one is as old as 2006 ! I suspect that the
issue comes from new includes which uncovered this conflict. Just got
it too here, it indeed only happens with USE_OPENSSL=1.

Just found another build issue in futex which does not build on RHEL
due to missing u32 declaration. Fixed too. With this it builds for me.

I'm attaching the two patches.

Thanks for the report,
Willy

From 783f25800cc0b10793817afd1d78615125be3a36 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 4 Sep 2012 12:19:04 +0200
Subject: [PATCH 1/2] BUILD: http: rename error_message http_error_message to 
fix conflicts on RHEL

Duncan Hall reported a build issue on CentOS where error_message conflicts
with another system declaration when SSL is enabled. Rename the function.
---
 include/proto/proto_http.h |2 +-
 src/proto_http.c   |   56 ++--
 src/session.c  |2 +-
 3 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/include/proto/proto_http.h b/include/proto/proto_http.h
index 69d932c..e7ad8ff 100644
--- a/include/proto/proto_http.h
+++ b/include/proto/proto_http.h
@@ -108,7 +108,7 @@ void http_reset_txn(struct session *s);
 
 struct http_req_rule *parse_http_req_cond(const char **args, const char *file, 
int linenum, struct proxy *proxy);
 void free_http_req_rules(struct list *r);
-struct chunk *error_message(struct session *s, int msgnum);
+struct chunk *http_error_message(struct session *s, int msgnum);
 
 /* to be used when contents change in an HTTP message */
 #define http_msg_move_end(msg, bytes) do { \
diff --git a/src/proto_http.c b/src/proto_http.c
index 4faff52..d7208d2 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -669,7 +669,7 @@ static void http_server_error(struct session *t, struct 
stream_interface *si,
  * and message.
  */
 
-struct chunk *error_message(struct session *s, int msgnum)
+struct chunk *http_error_message(struct session *s, int msgnum)
 {
if (s-be-errmsg[msgnum].str)
return s-be-errmsg[msgnum];
@@ -841,25 +841,25 @@ void http_return_srv_error(struct session *s, struct 
stream_interface *si)
 
if (err_type  SI_ET_QUEUE_ABRT)
http_server_error(s, si, SN_ERR_CLICL, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type  SI_ET_CONN_ABRT)
http_server_error(s, si, SN_ERR_CLICL, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type  SI_ET_QUEUE_TO)
http_server_error(s, si, SN_ERR_SRVTO, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type  SI_ET_QUEUE_ERR)
http_server_error(s, si, SN_ERR_SRVCL, SN_FINST_Q,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type  SI_ET_CONN_TO)
http_server_error(s, si, SN_ERR_SRVTO, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else if (err_type  SI_ET_CONN_ERR)
http_server_error(s, si, SN_ERR_SRVCL, SN_FINST_C,
- 503, error_message(s, HTTP_ERR_503));
+ 503, http_error_message(s, HTTP_ERR_503));
else /* SI_ET_CONN_OTHER and others */
http_server_error(s, si, SN_ERR_INTERNAL, SN_FINST_C,
-   

Re: HAProxy with native SSL support !

2012-09-04 Thread Rahul Nair
Congratulations Willy and Team...

On Tue, Sep 4, 2012 at 3:59 PM, Willy Tarreau w...@1wt.eu wrote:

 On Tue, Sep 04, 2012 at 05:56:14PM +1000, Duncan Hall wrote:
  On 04/09/12 09:37, Willy Tarreau wrote:
  
  Have a lot of fun and please report your success/failures,
  Willy
  
  
 
  Small issue when compiling on CentOS 5.8 64bit against RPM versions of
  openssl-devel and e2fsprogs-devel-1.39-34.el5_8.1 I get the following:
 
  make TARGET=linux2628 USE_OPENSSL=1
  gcc -Iinclude -Iebtree -Wall -O2 -g -fno-strict-aliasing
  -DCONFIG_HAP_LINUX_SPLICE -DTPROXY -DCONFIG_HAP_LINUX_TPROXY
  -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_EPOLL -DENABLE_SEPOLL
  -DNETFILTER -DUSE_GETSOCKNAME -DUSE_OPENSSL -DUSE_SYSCALL_FUTEX
  -DCONFIG_HAPROXY_VERSION=\1.5-dev11\
  -DCONFIG_HAPROXY_DATE=\2012/06/04\ \
  -DBUILD_TARGET='linux2628' \
  -DBUILD_ARCH='' \
  -DBUILD_CPU='generic' \
  -DBUILD_CC='gcc' \
  -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \
  -DBUILD_OPTIONS='' \
  -c -o src/haproxy.o src/haproxy.c
  In file included from src/haproxy.c:83:
  include/proto/proto_http.h:111: error: conflicting types for
 ?error_message?
  /usr/include/et/com_err.h:37: error: previous declaration of
  ?error_message? was here
  make: *** [src/haproxy.o] Error 1

 This is very strange, this one is as old as 2006 ! I suspect that the
 issue comes from new includes which uncovered this conflict. Just got
 it too here, it indeed only happens with USE_OPENSSL=1.

 Just found another build issue in futex which does not build on RHEL
 due to missing u32 declaration. Fixed too. With this it builds for me.

 I'm attaching the two patches.

 Thanks for the report,
 Willy




-- 
-Rahul N.
IT Department
In2M Technologies Pvt Ltd. (Finicity)
Website: www.finicity.com/india


Re: HAProxy with native SSL support !

2012-09-04 Thread joris dedieu
Hi, Willy

Thanks for this long time expected feature !


 Have a lot of fun and please report your success/failures,

There is an include issue in this snapshot on FreeBSD (witch is not I
think ssl related) :

gmake TARGET=freebsd USE_OPENSSL=1
gcc -Iinclude -Iebtree -Wall  -O2 -g -fno-strict-aliasing
-DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE
-DUSE_OPENSSL  -DCONFIG_HAPROXY_VERSION=\1.5-dev11\
-DCONFIG_HAPROXY_DATE=\2012/06/04\ \
  -DBUILD_TARGET='freebsd' \
  -DBUILD_ARCH='' \
  -DBUILD_CPU='generic' \
  -DBUILD_CC='gcc' \
  -DBUILD_CFLAGS='-O2 -g -fno-strict-aliasing' \
  -DBUILD_OPTIONS='' \
   -c -o src/haproxy.o src/haproxy.c
gcc -Iinclude -Iebtree -Wall  -O2 -g -fno-strict-aliasing
-DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE
-DUSE_OPENSSL  -DCONFIG_HAPROXY_VERSION=\1.5-dev11\
-DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/sessionhash.o
src/sessionhash.c
gcc -Iinclude -Iebtree -Wall  -O2 -g -fno-strict-aliasing
-DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE
-DUSE_OPENSSL  -DCONFIG_HAPROXY_VERSION=\1.5-dev11\
-DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/base64.o src/base64.c
gcc -Iinclude -Iebtree -Wall  -O2 -g -fno-strict-aliasing
-DTPROXY -DCONFIG_HAP_CRYPT -DENABLE_POLL -DENABLE_KQUEUE
-DUSE_OPENSSL  -DCONFIG_HAPROXY_VERSION=\1.5-dev11\
-DCONFIG_HAPROXY_DATE=\2012/06/04\ -c -o src/protocols.o
src/protocols.c
In file included from src/protocols.c:20:
include/common/standard.h: In function 'is_addr':
include/common/standard.h:572: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:572: error: (Each undeclared identifier is
reported only once
include/common/standard.h:572: error: for each function it appears in.)
include/common/standard.h:574: error: 'AF_INET6' undeclared (first use
in this function)
include/common/standard.h: In function 'get_net_port':
include/common/standard.h:586: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:588: error: 'AF_INET6' undeclared (first use
in this function)
include/common/standard.h: In function 'get_host_port':
include/common/standard.h:598: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:600: error: 'AF_INET6' undeclared (first use
in this function)
include/common/standard.h: In function 'get_addr_len':
include/common/standard.h:610: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:612: error: 'AF_INET6' undeclared (first use
in this function)
include/common/standard.h:614: error: 'AF_UNIX' undeclared (first use
in this function)
include/common/standard.h: In function 'set_net_port':
include/common/standard.h:624: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:626: error: 'AF_INET6' undeclared (first use
in this function)
include/common/standard.h: In function 'set_host_port':
include/common/standard.h:636: error: 'AF_INET' undeclared (first use
in this function)
include/common/standard.h:638: error: 'AF_INET6' undeclared (first use
in this function)
gmake: *** [src/protocols.o] Erreur 1

A workaround is to include sys/socket.h in include/common/standard.h.

Once corrected, ssl support successfully build and run on FreeBSD 8.3.

Joris

 Willy





Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
Hi Joris,

On Tue, Sep 04, 2012 at 01:45:29PM +0200, joris dedieu wrote:
 Hi, Willy
 
 Thanks for this long time expected feature !
 
 
  Have a lot of fun and please report your success/failures,
 
 There is an include issue in this snapshot on FreeBSD (witch is not I
 think ssl related) :
(...)
 A workaround is to include sys/socket.h in include/common/standard.h.
 
 Once corrected, ssl support successfully build and run on FreeBSD 8.3.
 
 Joris

Thanks for the report, patch merged and attached.

Willy

From d50265aa0ed40969f2659c001e261e4ca78d011f Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 4 Sep 2012 14:18:33 +0200
Subject: [PATCH] BUILD: include sys/socket.h to fix build failure on FreeBSD

Joris Dedieu reported that include/common/standard.h needs this.
---
 include/common/standard.h |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/include/common/standard.h b/include/common/standard.h
index 511a1dd..93b9e36 100644
--- a/include/common/standard.h
+++ b/include/common/standard.h
@@ -26,6 +26,7 @@
 #include string.h
 #include time.h
 #include sys/types.h
+#include sys/socket.h
 #include sys/un.h
 #include netinet/in.h
 #include common/config.h
-- 
1.7.1



RE: HAProxy with native SSL support !

2012-09-04 Thread Lukas Tribus
 at least as fast.

 7) other

 A lot remains to be done, mainly some of the aforementionned structres
 are still included in other ones, which simplified the split Once all
 the work is over, we should end up with less memory used per connection.
 This is important to better handle DDoS.


 At the moment, everything we could try seems to work fine. The SSL stacks
 well on top of the PROXY protocol, which is very important to build SSL
 offload farms (I'm sure Baptiste will want to write a blog article on the
 subject of using sub-$1000 machines to build large 100k+tps farms).
 Stats work over https too. Right now we're missing ACLs to match whether
 the traffic was SSL or clear, as well as logs. Both can be worked around
 by using distinct bind lines or even frontends. The doc is still clearly
 lacking, but we think that the config will change a little bit.

 Only the GNU makefile was updated, neither the BSD nor OSX were, they're
 a little trickier. If someone with one of these systems wants to update
 them, I'll happily accept the patches.

 What else ? Ah yes, 4k. You're there wondering about the results. 4000 SSL
 connections per second and 300 Mbps is what we got out of a dual-core Atom
 D510 at 1.66 GHz, in SSLv3 running over 4 processes (hyperthreading was
 enabled) :-) This is a bit more than stud and obviously much better than
 stunnel (which doesn't scale to more than a few hundred connections before
 the performance quickly drops).

 And older tests seem to indicate that with YaSSL we can get 30-40% more,
 maybe even more. We need to work with the YaSSL guys to slightly improve
 their cache management before this can become a default build option.

 Enough speaking, for those who want to test or even have the hardware to
 run more interesting benchmarks, the code was merged into the master
 branch and is in today's snapshot (20120904) here :

 http://haproxy.1wt.eu/download/1.5/src/snapshot/

 Build it by passing USE_OPENSSL=1 on the make command line. You should
 also include support for linux-2.6 options for better results :

 make TARGET=linux2628 USE_OPENSSL=1

 If all goes well by the end of the week, I'll issue -dev12, but I expect
 that we'll have some bugs to fix till then.

 BTW, be very careful, openssl is a memory monster. We counted about 80kB
 per connection for haproxy+ssl, this is 800 MB for only 10k connections!
 And remember, this is still beta-quality code. Don't blindly put this in
 production (eventhough I did it on 1wt.eu : https://demo.1wt.eu/). You
 have been warned!

 Please use the links below :
 site index : http://haproxy.1wt.eu/
 sources : http://haproxy.1wt.eu/download/1.5/src/snapshot/
 changelog : http://haproxy.1wt.eu/download/1.5/src/snapshot/CHANGELOG
 Exceliance : http://www.exceliance.fr/en/

 Have a lot of fun and please report your success/failures,
 Willy


  


Re: HAProxy with native SSL support !

2012-09-04 Thread David BERARD
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

On 04/Sep - 01:37, Willy Tarreau w...@1wt.eu wrote:
| Have a lot of fun and please report your success/failures,
| Willy

Thanks a lot for this useful feature. It works well on a dual PPC64 Linux 
server.

I wrote a small path to add the SSL_OP_CIPHER_SERVER_PREFERENCE OpenSSL option
to frontend, if the 'prefer-server-ciphers' keyword is set.


https://0x1.fr/files/patchs/haproxy-ss-20120904_prefer_server_ciphers.patch

Example :

bind 10.11.12.13 ssl /etc/haproxy/ssl/cert.pem ciphers 
RC4:HIGH:!aNULL:!MD5 prefer-server-ciphers

This option mitigate the effect of the BEAST Attack (as I understand), and it
equivalent to : 
- Apache HTTPd SSLHonorCipherOrder option.
- Nginx ssl_prefer_server_ciphers option.

Maybe it can be useful to add OpenSSL option directly in the haproxy
configuration as the 'options' keyword in stunnel.

Best regards.
- -- 

David BERARD

contact(at)davidberard.fr
GPG|PGP KeyId 0xC8533354
GPG|PGP Key http://davidberard.fr/C8533354.gpgkey

*   No electrons were harmed in the transmission of this email *
-BEGIN PGP SIGNATURE-

iEYEARECAAYFAlBF/uAACgkQOL7fhchTM1Q0PQCgqbxmjbxKokJ2dFX28dbfjml4
KOcAnja+g7reSbHJVub8P4HYrcz1Q/TG
=PD86
-END PGP SIGNATURE-



Re: HAProxy with native SSL support ! = fix for ssl_cert

2012-09-04 Thread Willy Tarreau
Emeric reported that the build fails without USE_OPENSSL, which is caused
by a last-minute change I did yesterday evening. It shows up as ssl_cert
not being part of a structure.

If you get this, please use the attached patch.

Regards,
Willy

From ff9f7698fcefef66bceb1ec32a3da8b14947a594 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 4 Sep 2012 15:13:20 +0200
Subject: [PATCH] BUILD: fix build error without SSL (ssl_cert)

One last-minute optimization broke the build without SSL support.
Move ssl_cert out of the #ifdef/#endif and it's OK.
---
 include/types/protocols.h |2 +-
 src/haproxy.c |2 --
 2 files changed, 1 insertions(+), 3 deletions(-)

diff --git a/include/types/protocols.h b/include/types/protocols.h
index 4e40a67..b075ef6 100644
--- a/include/types/protocols.h
+++ b/include/types/protocols.h
@@ -130,8 +130,8 @@ struct listener {
char *interface;/* interface name or NULL */
int maxseg; /* for TCP, advertised MSS */
 
-#ifdef USE_OPENSSL
char *ssl_cert; /* ssl certificate */
+#ifdef USE_OPENSSL
struct {
SSL_CTX *ctx;
char *ciphers;  /* cipher suite to use if non-null */
diff --git a/src/haproxy.c b/src/haproxy.c
index adf2614..4e75080 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -996,9 +996,7 @@ void deinit(void)
l_next = l-next;
unbind_listener(l);
delete_listener(l);
-#ifdef USE_OPENSSL
free(l-ssl_cert);
-#endif
free(l-name);
free(l-counters);
free(l);
-- 
1.7.1



Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
Hi Lukas,

On Tue, Sep 04, 2012 at 03:05:14PM +0200, Lukas Tribus wrote:
 Willy, this is huge! Great, great work!
 
 A few comments/questions:
 
 - are you running latest and greatest openssl on demo.1wt.eu? I am asking
 because Secure Renegotiation doesn't seem to be supported [1]. Older
 (1.0.0?) releases seem to have a higher memory overhead as well, iirc.

Just checked and no, it's 0.9.8 only on this box. Anyway this is not
very important on this box since it gets very low load. Also this was
more for the test than anything else, so I expected that someone would
even complain about the self-signed cert :-)

 - I see you have implemented [nosslv3] [notlsv1] to limit the SSL/TLS
 protocols. I do prefer the nginx approach [2], positively specifying the
 protocols used. This is much more flexible. Even if today the 2 options are
 fine for generic HTTPS proxies in the Internet, there may be corner cases or
 future needs to specify the protocols in a more flexible manner, like nginx
 does.

Point taken. However, it's important to know that SSL uses the negative form,
which is why I preferred to use the same one. You have options to *disable*
use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on
the same logics because if openssl 1.2 comes with support for TLSv5, it will
be offered by default until we have an option to disable it. With the positive
approach, it will be offered by default until unknown, and once known, it will
suddenly be disabled for the same configs unless users explicitly ask for it.
I'm rather for long-term config compatibility you know :-)

 - about ssl ciphers, we should probably steal the default from nginx, by
 using HIGH:!aNULL:!MD5 [3], so we don't enable obsolete ciphers like DES by
 default [1].

OK, right now nothing is set by default and indeed it could be an idea.

 - any way to validate certificates of the backend? Perhaps a insecure/secure
 flag should be introduced. The exact host/domainname is needed as well to
 validate the backend certificate. Consider configurations where the backend
 server is not local, but can be contacted only over the big bad internet. I'm
 also thinking of a certificate validation in the health checks.

We're considering adding some verify options exactly for this, in order to
only accept some backend certificates as well as to validate some users. They
are not implemented yet, but this is planned.

 - probably not a much requested feature, but for the record: any plan to make
 client based certificates work? Could be interesting for example to
 authenticate haproxy on the backend servers. There are probably people with
 frontend client certificate use-cases as well. However general interest in
 this feature is probably low.

This was our primary concern, but in order to get these, we needed native
SSL :-)

So yes this will come, maybe soon, maybe later. We hope no later than the end
of the year depending on our available time, but maybe much sooner too. I also
want to be able to pass client cert info into HTTP headers, logs and ACLs.

 - you seem to be interested in supporting native yassl. Would you then drop
 openssl support or would the idea be to support both native yassl and the
 openssl api?

No, we wouldn't drop it, openssl is the de-facto standard. However if we see
a much higher performance level by using the native API, we'd probably write
a 3rd data layer dedicated to yassl, and would probably rename the current SSL
data layer so that we can choose the one we want at build time.

 Congratulations to this big step in haproxy development!

Thanks :-)

Regards,
willy




Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
Hi David,

On Tue, Sep 04, 2012 at 03:15:13PM +0200, David BERARD wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Hi,
 
 On 04/Sep - 01:37, Willy Tarreau w...@1wt.eu wrote:
 | Have a lot of fun and please report your success/failures,
 | Willy
 
 Thanks a lot for this useful feature. It works well on a dual PPC64 Linux 
 server.

Excellent, thanks for the report.

 I wrote a small path to add the SSL_OP_CIPHER_SERVER_PREFERENCE OpenSSL option
 to frontend, if the 'prefer-server-ciphers' keyword is set.
 
   
 https://0x1.fr/files/patchs/haproxy-ss-20120904_prefer_server_ciphers.patch
 
 Example :
 
   bind 10.11.12.13 ssl /etc/haproxy/ssl/cert.pem ciphers 
 RC4:HIGH:!aNULL:!MD5 prefer-server-ciphers
 
 This option mitigate the effect of the BEAST Attack (as I understand), and it
 equivalent to : 
   - Apache HTTPd SSLHonorCipherOrder option.
   - Nginx ssl_prefer_server_ciphers option.

OK I did not know there was such a workaround. Thanks for the patch, I'm
going to apply it now.

 Maybe it can be useful to add OpenSSL option directly in the haproxy
 configuration as the 'options' keyword in stunnel.

No, your current solution is better, as we'd like to get such options for
each bind line. So your patch is perfect :-)

Best regards,
Willy




RE: HAProxy with native SSL support !

2012-09-04 Thread Lukas Tribus

 Point taken. However, it's important to know that SSL uses the negative form,
 which is why I preferred to use the same one. You have options to *disable*
 use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on
 the same logics because if openssl 1.2 comes with support for TLSv5, it will
 be offered by default until we have an option to disable it. With the positive
 approach, it will be offered by default until unknown, and once known, it will
 suddenly be disabled for the same configs unless users explicitly ask for it.
 I'm rather for long-term config compatibility you know :-)

Right, I fully agree with you.


 However if we see a much higher performance level by using the native API,
 we'd probably write a 3rd data layer dedicated to yassl, and would probably
 rename the current SSL data layer so that we can choose the one we want at
 build time.

Ok, thats great.

I have another comment about yassl: It seems that CyaSSL, the ANSI C based
edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2
doesn't seem to be supported on yasll yet, while CyaSSL already supports it.
I do not have any programming knowlegde, so please execuse my probably stupid
question, but whats the reason to impement yasll instead of CyaSSL? Can't
a C-lib be implemented in a C++ application?


Regards,
Lukas

[1] http://www.yassl.com/yaSSL/Products-yassl.html





 Date: Tue, 4 Sep 2012 15:26:24 +0200
 From: w...@1wt.eu
 To: luky...@hotmail.com
 CC: haproxy@formilux.org
 Subject: Re: HAProxy with native SSL support !

 Hi Lukas,

 On Tue, Sep 04, 2012 at 03:05:14PM +0200, Lukas Tribus wrote:
  Willy, this is huge! Great, great work!
 
  A few comments/questions:
 
  - are you running latest and greatest openssl on demo.1wt.eu? I am asking
  because Secure Renegotiation doesn't seem to be supported [1]. Older
  (1.0.0?) releases seem to have a higher memory overhead as well, iirc.

 Just checked and no, it's 0.9.8 only on this box. Anyway this is not
 very important on this box since it gets very low load. Also this was
 more for the test than anything else, so I expected that someone would
 even complain about the self-signed cert :-)

  - I see you have implemented [nosslv3] [notlsv1] to limit the SSL/TLS
  protocols. I do prefer the nginx approach [2], positively specifying the
  protocols used. This is much more flexible. Even if today the 2 options are
  fine for generic HTTPS proxies in the Internet, there may be corner cases or
  future needs to specify the protocols in a more flexible manner, like nginx
  does.

 Point taken. However, it's important to know that SSL uses the negative form,
 which is why I preferred to use the same one. You have options to *disable*
 use of V2/V3/TLS, not to enable them. Thus I find it more durable to stay on
 the same logics because if openssl 1.2 comes with support for TLSv5, it will
 be offered by default until we have an option to disable it. With the positive
 approach, it will be offered by default until unknown, and once known, it will
 suddenly be disabled for the same configs unless users explicitly ask for it.
 I'm rather for long-term config compatibility you know :-)

  - about ssl ciphers, we should probably steal the default from nginx, by
  using HIGH:!aNULL:!MD5 [3], so we don't enable obsolete ciphers like DES 
  by
  default [1].

 OK, right now nothing is set by default and indeed it could be an idea.

  - any way to validate certificates of the backend? Perhaps a insecure/secure
  flag should be introduced. The exact host/domainname is needed as well to
  validate the backend certificate. Consider configurations where the backend
  server is not local, but can be contacted only over the big bad internet. 
  I'm
  also thinking of a certificate validation in the health checks.

 We're considering adding some verify options exactly for this, in order to
 only accept some backend certificates as well as to validate some users. They
 are not implemented yet, but this is planned.

  - probably not a much requested feature, but for the record: any plan to 
  make
  client based certificates work? Could be interesting for example to
  authenticate haproxy on the backend servers. There are probably people with
  frontend client certificate use-cases as well. However general interest in
  this feature is probably low.

 This was our primary concern, but in order to get these, we needed native
 SSL :-)

 So yes this will come, maybe soon, maybe later. We hope no later than the end
 of the year depending on our available time, but maybe much sooner too. I also
 want to be able to pass client cert info into HTTP headers, logs and ACLs.

  - you seem to be interested in supporting native yassl. Would you then drop
  openssl support or would the idea be to support both native yassl and the
  openssl api?

 No, we wouldn't drop it, openssl is the de-facto standard. However if we see
 a much higher 

Re: HAProxy with native SSL support !

2012-09-04 Thread Willy Tarreau
On Tue, Sep 04, 2012 at 04:12:43PM +0200, Lukas Tribus wrote:
  However if we see a much higher performance level by using the native API,
  we'd probably write a 3rd data layer dedicated to yassl, and would probably
  rename the current SSL data layer so that we can choose the one we want at
  build time.
 
 Ok, thats great.
 
 I have another comment about yassl: It seems that CyaSSL, the ANSI C based
 edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2
 doesn't seem to be supported on yasll yet, while CyaSSL already supports it.
 I do not have any programming knowlegde, so please execuse my probably stupid
 question, but whats the reason to impement yasll instead of CyaSSL? Can't
 a C-lib be implemented in a C++ application?

In fact when I say yassl, I really mean CyaSSL.

Regards,
Willy




Re: HAProxy with native SSL support !

2012-09-04 Thread Falco Schmutz
Great ! Thanks to the team ! :-)


2012/9/4 Willy Tarreau w...@1wt.eu

 On Tue, Sep 04, 2012 at 04:12:43PM +0200, Lukas Tribus wrote:
   However if we see a much higher performance level by using the native
 API,
   we'd probably write a 3rd data layer dedicated to yassl, and would
 probably
   rename the current SSL data layer so that we can choose the one we
 want at
   build time.
 
  Ok, thats great.
 
  I have another comment about yassl: It seems that CyaSSL, the ANSI C
 based
  edition of yaSSL is recommended over yasll (see [1]). For example TLS1.2
  doesn't seem to be supported on yasll yet, while CyaSSL already supports
 it.
  I do not have any programming knowlegde, so please execuse my probably
 stupid
  question, but whats the reason to impement yasll instead of CyaSSL? Can't
  a C-lib be implemented in a C++ application?

 In fact when I say yassl, I really mean CyaSSL.

 Regards,
 Willy





-- 


Falco SCHMUTZ

 http://fr.linkedin.com/in/fschmutz

E-mail : fschm...@premaccess.com
Mobile : +33 6 80 22 00 29
Fixe Suisse : +41 44 586 77 74
Fax Suisse : +41 26 347 28 90
Fax France : +33 9 72 12 19 25

premaccess sàrl
Route des Arsenaux 41 - CP 132
CH-1705 Fribourg, Suisse

http://www.premaccess.com



Attention : le présent message et toutes les pièces jointes sont
confidentiels et établis a l'attention exclusive du ou des destinataire(s)
indique(s). Toute autre diffusion ou utilisation non autorisée est
interdite. Si vous recevez ce message par erreur, veuillez immédiatement en
avertir l'expéditeur par e-mail en retour, détruire le message et vous
abstenir de toute référence aux informations qui y figurent afin d'éviter
les sanctions attachées a la divulgation et a l'utilisation d'informations
confidentielles. Les messages électroniques sont susceptibles d'altération.
Nous déclinons toute responsabilité en cas d'altération ou de falsification
du présent message.

Warning : this e-mail and any files attached are confidential and intended
solely to the named addressee(s). Any unauthorised dissemination or use is
strictly prohibited. If you received this e-mail in error, please
immediately notify the sender by reply e-mail and then delete the e-mail
from your system. Please do not copy, use or make reference to it for any
purpose, or disclose its contents to any person : to do so could expose you
to sanctions. E-mails can be altered or falsified. We shall not be liable
for any alteration or falsification on this e-mail.


Re: HAProxy with native SSL support !

2012-09-04 Thread Justin Karneges
On Tuesday, September 04, 2012 08:41:44 AM Willy Tarreau wrote:
 On Mon, Sep 03, 2012 at 11:21:51PM -0700, Justin Karneges wrote:
  On Tuesday, September 04, 2012 01:37:17 AM Willy Tarreau wrote:
   After several months of efforts by the Exceliance team, we managed to
   rework all the buffer and connection layers in order to get SSL working
   on both sides of HAProxy.
  
  Very cool.
  
  Since HAProxy is event-driven, is anything done to avoid expensive crypto
  operations from blocking the event loop? Maybe that is what you intend for
  remote sockets ?
 
 Exactly !
 
 That said, as I've always said, the problem is not the relative cost of
 crypto but the computation time. On small devices, if an operation can
 take 10ms, it clearly is a problem. However on larger devices, if no
 operation takes more than 100 microseconds, then the added latency is
 minimal compared to system latency. That does not mean it's not a problem,
 just that when you're at 100% CPU, you might still have enough scheduling
 capacity to process the rest of the traffic without too much impact.
 
 But clearly when we can offload SSL to other processes, that will be much
 better. This can already be done BTW, but you have to chain proxies. For
 instance :
 
   global
  nbproc 4
 
   listen front
  bind-process 1,2
  bind :443 ssl haproxy.pem
  server http 127.0.0.1: send-proxy
 
   listen http
  bind-process 3
  bind 127.0.0.1: accept-proxy
  mode http
  server back 127.0.0.1:8443 send-proxy
 
   listen back
  bind-process 4
  bind 127.0.0.1:8443 accept-proxy
  server out 192.168.0.1:443 ssl
 
 You see, https is accepted on processes 1 and 2, which decipher the
 traffic and pass it to process 3, preserving IP information. This
 process does the HTTP handling and passes the traffic to process 4
 which will re-cipher it before going to the server.
 
 Admittedly this is ugly and that's why we'd like to implement the
 remote sockets soon, so that you can write something like this :
 
   listen http
  bind-process 3
  bind :443 ssl haproxy.pem process 1,2
  mode http
  server out 192.168.0.1:443 ssl process 4
 
 Also, using socketpair() internally will be much faster and have a lower
 overhead than TCP connect/accept.

Usually, the most expensive operations in TLS are the public key ones at the 
start of a negotiation (and possibly a renegotiation, though I'm not sure of 
protocol details there). However, pretty much all other TLS traffic should be 
fast to execute. I wonder if there is a way to differentiate these steps? This 
is how you'd ensure that a bunch of new https connections do not cause 
slowdown of existing https connections.

I have tried in the past to use OpenSSL in an event-driven application in a 
way that does not block the event loop. This was for a local desktop 
application that used smart cards, where an OpenSSL call might block for 5 
seconds while it hits the hardware. Since the expected number of TLS sessions 
in the application was small, I simply made one thread per TLS session, and 
funneled all results back to the event-driven main thread. Of course a thread 
per connection approach would not work if you want to handle thousands of 
connections like HAProxy. One day I wanted to go back and rewrite my code such 
that I could send operations to separate threads or keep them in the main 
thread based on a given TLS session state. This way, I could make it so only 
the TLS handshakes (with the expensive RSA operations and such) would need 
concurrency. Then the number of threads needed would be the number of expected 
simultaneous handshakes, which would be a far more manageable number (20?) 
than expected simultaneous sessions (thousands).

This is not an answer for you, but just a brain dump of my thoughts on the 
subject.

Justin

RE: HAProxy with native SSL support !

2012-09-04 Thread Lukas Tribus

Hi,

 In fact when I say yassl, I really mean CyaSSL.

Ok, great.



A few more comments about (C)yassl:

-    development of new features is obviously not as fast as in OpenSSL. For
    example TLS SNI is not supported yet (ETA: next release) [1]. This feature
    was introduced in 2007 (0.9.8f) in the openssl implementation, and it was
    enabled by default in 2009 (0.9.8j), [2]. That doesn't mean yassl isn't
    suited for haproxy, one just needs to be aware that yassl is NOT a faster
    openssl with lower memory overhead. Its a different implementation with
    different goals and OpenSSL will remain the full-featured reference, with
    company's like Google introducing new features and providing bugfixes
    for it.

-    (C)yassl doesn't support - by design - renegotiation. They also don't
    implement RFC4756 (secure renegotiation), see [3]. While this is not
    a security problem (from a server point of view), it will become an
    interoperability problem sooner or later, once browser vendors make
    the switch, and threat non-RFC4756 capable servers as broken [4],
    [5], [6]. I wonder how this is going to be fixed, if at all.

-    I also believe that most of the haproxy maintainers will compile haproxy
    with openssl, and yassl will be more or less available only to those
    compiling haproxy on their own. In fact, libyassl/libcyassl is not even
    available in debian for example [7], and they don't like libs linked
    statically to their packages.


Anyway, time will tell and the mentioned issues are no showstoppers.

Regards,
Lukas


[1] 
http://www.yassl.com/yaSSL/Blog/Entries/2012/8/15_CyaSSL_to_include_SNI_%28Server_Name_Indication%29_in_Upcoming_Release.html
[2] http://www.openssl.org/news/changelog.html
[3] http://yassl.com/yaSSL/Docs-cyassl-manual-9-library-design.html
[4] 
https://wiki.mozilla.org/Security:Renegotiation#security.ssl.treat_unsafe_negotiation_as_broken
[5] 
https://wiki.mozilla.org/Security:Renegotiation#security.ssl.require_safe_negotiation
[6] 
http://my.opera.com/securitygroup/blog/2010/11/04/a-few-results-from-the-tls-prober
[7] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=664553