Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am Thu, 4 May 2017 08:57:41 +0200
schrieb Baptiste :

> On Thu, May 4, 2017 at 8:50 AM, Aleksandar Lazic 
> wrote:
> 
> > Hi.
> >
> > awsome timing because nginx just released his WAF also ;-).
> >
> > https://www.nginx.com/blog/modsecurity-waf-released/
>
> Well, you can't compare a proprietary product with an open source one!
> And here, we see the benefits of the community behind such product.

Well I see the benefit in the same requirements.
Both solutin need a independent modsecurity => libmodsecurity.

> Baptiste
> 
Aleks

Re: ModSecurity: First integration patches

[Baptiste]

On Thu, May 4, 2017 at 8:50 AM, Aleksandar Lazic  wrote:

> Hi.
>
> awsome timing because nginx just released his WAF also ;-).
>
> https://www.nginx.com/blog/modsecurity-waf-released/
>
>
Well, you can't compare a proprietary product with an open source one!
And here, we see the benefits of the community behind such product.

Baptiste

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi.

awsome timing because nginx just released his WAF also ;-).

https://www.nginx.com/blog/modsecurity-waf-released/

Now we only need to give Willy the time and chance to work on
HTTP/2 to have this feature also in haproxy ;-)

Regards
Aleks

Am Thu, 27 Apr 2017 23:10:15 +0200
schrieb Thierry Fournier :

> > On 27 Apr 2017, at 18:53, Aleksandar Lazic 
> > wrote:
> > 
> > Hi Willy.
> > 
> > Am 27-04-2017 12:05, schrieb Willy Tarreau:  
> >> Hi Thierry,
> >> On Thu, Apr 20, 2017 at 03:05:35PM +0200, Thierry Fournier wrote:  
> >>> Hi,
> >>> After a quick private brainstorm, Willy propose to me a new
> >>> binary encoding for the headers. It is useless to give the
> >>> numbers of headers contained in the block, so the end of headers
> >>> is marked by an empty header (header name and value have a length
> >>> of 0). The new patches in attachment (first patch is 0002).  
> >> OK I finally merged your patchset, but I fixed a few typos and a
> >> bit of confusing wording in the doc.  
> > 
> > I'm not Thierry but I say, thanks ;-).  
> 
> 
> I’m Thierry and I say thanks too ;-)
> 
> 
> > This make my docker setup easier.
> > 
> > I will write a blog entry about this, I will share it here.  
> 
> 
> great, share the link !
> 
> 
> >   
> >> Thanks,
> >> Willy  
> > 
> > Very best regards
> > Aleks  
> 

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 27 Apr 2017, at 18:53, Aleksandar Lazic  wrote:
> 
> Hi Willy.
> 
> Am 27-04-2017 12:05, schrieb Willy Tarreau:
>> Hi Thierry,
>> On Thu, Apr 20, 2017 at 03:05:35PM +0200, Thierry Fournier wrote:
>>> Hi,
>>> After a quick private brainstorm, Willy propose to me a new binary encoding
>>> for the headers. It is useless to give the numbers of headers contained in
>>> the block, so the end of headers is marked by an empty header (header name
>>> and value have a length of 0).
>>> The new patches in attachment (first patch is 0002).
>> OK I finally merged your patchset, but I fixed a few typos and a bit of
>> confusing wording in the doc.
> 
> I'm not Thierry but I say, thanks ;-).


I’m Thierry and I say thanks too ;-)


> This make my docker setup easier.
> 
> I will write a blog entry about this, I will share it here.


great, share the link !


> 
>> Thanks,
>> Willy
> 
> Very best regards
> Aleks

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi Willy.

Am 27-04-2017 12:05, schrieb Willy Tarreau:

Hi Thierry,

On Thu, Apr 20, 2017 at 03:05:35PM +0200, Thierry Fournier wrote:

Hi,

After a quick private brainstorm, Willy propose to me a new binary 
encoding
for the headers. It is useless to give the numbers of headers 
contained in
the block, so the end of headers is marked by an empty header (header 
name

and value have a length of 0).

The new patches in attachment (first patch is 0002).


OK I finally merged your patchset, but I fixed a few typos and a bit of
confusing wording in the doc.


I'm not Thierry but I say, thanks ;-).
This make my docker setup easier.

I will write a blog entry about this, I will share it here.


Thanks,
Willy


Very best regards
Aleks

Re: ModSecurity: First integration patches

[Willy Tarreau]

Hi Thierry,

On Thu, Apr 20, 2017 at 03:05:35PM +0200, Thierry Fournier wrote:
> Hi,
> 
> After a quick private brainstorm, Willy propose to me a new binary encoding
> for the headers. It is useless to give the numbers of headers contained in
> the block, so the end of headers is marked by an empty header (header name
> and value have a length of 0).
> 
> The new patches in attachment (first patch is 0002).

OK I finally merged your patchset, but I fixed a few typos and a bit of
confusing wording in the doc.

Thanks,
Willy

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 23 Apr 2017, at 15:19, Aleksandar Lazic  wrote:
> 
> Hi Thierry
> 
> Am 20-04-2017 15:05, schrieb Thierry Fournier:
>> Hi,
>> After a quick private brainstorm, Willy propose to me a new binary encoding
>> for the headers. It is useless to give the numbers of headers contained in
>> the block, so the end of headers is marked by an empty header (header name
>> and value have a length of 0).
>> The new patches in attachment (first patch is 0002).
> 
> I have now created a docker image
> 
> https://hub.docker.com/r/me2digital/haproxy-waf/
> 
> The build was successfully and haproxy starts.
> 
> I have also try to check if the modsecurity starts and yeah it starts ;-)
> 
> ###
> 
> 1492953204.290643 [00] ModSecurity for nginx (STABLE)/2.9.1 
> (http://www.modsecurity.org/) configured.
> 1492953204.290705 [00] ModSecurity: APR compiled version="1.4.8"; loaded 
> version="1.4.8"
> 1492953204.290730 [00] ModSecurity: PCRE compiled version="8.32 "; loaded 
> version="8.32 2012-11-30"
> 1492953204.290739 [00] ModSecurity: YAJL compiled version="2.0.4"
> 1492953204.290743 [00] ModSecurity: LIBXML compiled version="2.9.1"
> 1492953204.290747 [00] ModSecurity: Status engine is currently disabled, 
> enable it by set SecStatusEngine to On.
> 1492953209.298799 [03] 0 clients connected
> 1492953209.298937 [04] 0 clients connected
> ...
> ###
> 
> Now we can go the next step and create a complete solution ;-)
> 
> It would be interesting to go the same way as mod_sec & nginx is going.
> 
> https://www.trustwave.com/Resources/SpiderLabs-Blog/Announcing-the-availability-of-ModSecurity-extension-for-Nginx/
> 


Hi aleksander,

Thanks for the news. I integrated ModSec using the same way than NGinx, so it is
absolutely a good idea to follow the same way than ModSec/NGinx. I will write an
howto and publish benchmark when I found a lot of time.

A feedback on your test is welcome.

Thierry

>> Thierry
> 
> Regards
> Aleks
> 
>>> On 19 Apr 2017, at 15:34, thierry.fourn...@arpalert.org wrote:
>>> Hi,
>>> There is a new lot of patches for the spoa/modescurity contrib.
>>> Thierry
>>> On Wed, 19 Apr 2017 11:24:36 +0200
>>> Thierry Fournier  wrote:
> On 19 Apr 2017, at 09:16, Aleksandar Lazic  wrote:
> Am 19-04-2017 05:51, schrieb Willy Tarreau:
>> On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:
>>> Why not reuse the upcoming http/2 format.
>>> HTTP/2 is *easy* to parse and the implementations of servers are 
>>> growing?
>> Are you kidding ? I mean you want everyone to have to implement HPACK 
>> etc ?
> Well there are a currently lot of http/2 servers out there, but I 
> understand your concerns.
>>> I know this is still on the todo list but I think this is worth to go
>>> instead to add another protocol.
>>> Or we can take a look into grpc, which is on top of http/2 with 
>>> protobuffer.
>>> http://www.grpc.io/
>>> As I take more time to think in this direction why not add a grpc filter
>>> like the spoa filter?
>>> Opinions?
>> Code generators are always a pain to deal with. It reminds me the old 
>> days
>> when I was using rpcgen. And grpc doesn't provide a C implementation so 
>> that
>> kills it :-)
> Yeah I have also seen that they prefer c++, c#, go and other languages 
> but not offer a c lib.
> This is strange just because the core is written in c, but I don't need 
> to understand everything ;-)
>> Thierry told me he's going to implement a simple TLV-like list which will
>> be much easier to implement and easy to code on both sides.
> @Thierry:
> Ah something like netstrings ?
> https://cr.yp.to/proto/netstrings.txt
 Hi thanks for the link. I do not like this encoding because it is not easy 
 to
 parse. I must read ascii representation of numbers and convert-it. I prefer
 simple binary format, with length (encoding in the same way than the SPOE
 protocol) and value.
 Hi, the main goal is providing:
 - a format easy to decode
 - not adding lot of dependencies to haproxy
 So, after brainstorm, I propose:
 - remove all data already accessible via existing sample-fetches from the 
 new
  sample-fetch introduced by my previous patch. These data are:
  method, path, query string, http-version and the body. All of these data 
 are
  available in some sample fetches and it will be great to reuse it. Note 
 that
  the only data which keep are the headers.
 - For the headers I propose two format: The first format is for general 
 use. It
  is simply the header block in the same format that we received (with the 
 final
  \r\n for detecting truncated header bloc)
 - For header I propose also a binary format like this:
  
  [
 
 
  ]…
  This format is easy to decode because each length is encoded 

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi Thierry

Am 20-04-2017 15:05, schrieb Thierry Fournier:

Hi,

After a quick private brainstorm, Willy propose to me a new binary 
encoding
for the headers. It is useless to give the numbers of headers contained 
in
the block, so the end of headers is marked by an empty header (header 
name

and value have a length of 0).

The new patches in attachment (first patch is 0002).


I have now created a docker image

https://hub.docker.com/r/me2digital/haproxy-waf/

The build was successfully and haproxy starts.

I have also try to check if the modsecurity starts and yeah it starts 
;-)


###

1492953204.290643 [00] ModSecurity for nginx (STABLE)/2.9.1 
(http://www.modsecurity.org/) configured.
1492953204.290705 [00] ModSecurity: APR compiled version="1.4.8"; loaded 
version="1.4.8"
1492953204.290730 [00] ModSecurity: PCRE compiled version="8.32 "; 
loaded version="8.32 2012-11-30"

1492953204.290739 [00] ModSecurity: YAJL compiled version="2.0.4"
1492953204.290743 [00] ModSecurity: LIBXML compiled version="2.9.1"
1492953204.290747 [00] ModSecurity: Status engine is currently disabled, 
enable it by set SecStatusEngine to On.

1492953209.298799 [03] 0 clients connected
1492953209.298937 [04] 0 clients connected
...
###

Now we can go the next step and create a complete solution ;-)

It would be interesting to go the same way as mod_sec & nginx is going.

https://www.trustwave.com/Resources/SpiderLabs-Blog/Announcing-the-availability-of-ModSecurity-extension-for-Nginx/


Thierry


Regards
Aleks






On 19 Apr 2017, at 15:34, thierry.fourn...@arpalert.org wrote:

Hi,

There is a new lot of patches for the spoa/modescurity contrib.

Thierry


On Wed, 19 Apr 2017 11:24:36 +0200
Thierry Fournier  wrote:



On 19 Apr 2017, at 09:16, Aleksandar Lazic  
wrote:




Am 19-04-2017 05:51, schrieb Willy Tarreau:

On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:

Why not reuse the upcoming http/2 format.
HTTP/2 is *easy* to parse and the implementations of servers are 
growing?
Are you kidding ? I mean you want everyone to have to implement 
HPACK etc ?


Well there are a currently lot of http/2 servers out there, but I 
understand your concerns.


I know this is still on the todo list but I think this is worth to 
go

instead to add another protocol.
Or we can take a look into grpc, which is on top of http/2 with 
protobuffer.

http://www.grpc.io/
As I take more time to think in this direction why not add a grpc 
filter

like the spoa filter?
Opinions?
Code generators are always a pain to deal with. It reminds me the 
old days
when I was using rpcgen. And grpc doesn't provide a C 
implementation so that

kills it :-)


Yeah I have also seen that they prefer c++, c#, go and other 
languages but not offer a c lib.
This is strange just because the core is written in c, but I don't 
need to understand everything ;-)


Thierry told me he's going to implement a simple TLV-like list 
which will

be much easier to implement and easy to code on both sides.


@Thierry:
Ah something like netstrings ?
https://cr.yp.to/proto/netstrings.txt


Hi thanks for the link. I do not like this encoding because it is not 
easy to
parse. I must read ascii representation of numbers and convert-it. I 
prefer
simple binary format, with length (encoding in the same way than the 
SPOE

protocol) and value.

Hi, the main goal is providing:
- a format easy to decode
- not adding lot of dependencies to haproxy

So, after brainstorm, I propose:

- remove all data already accessible via existing sample-fetches from 
the new

  sample-fetch introduced by my previous patch. These data are:
  method, path, query string, http-version and the body. All of these 
data are
  available in some sample fetches and it will be great to reuse it. 
Note that

  the only data which keep are the headers.

- For the headers I propose two format: The first format is for 
general use. It
  is simply the header block in the same format that we received 
(with the final

  \r\n for detecting truncated header bloc)

- For header I propose also a binary format like this:
  
  [
 
 
  ]…
  This format is easy to decode because each length is encoded with 
the SPOE
  numeric values encoding method, so it is always embedded in SPOE 
client.


The SPOE message description will become

  spoe-message check-request
 uniqueid args method path query req.ver req.hdrs_bin 
req.body_size req.body


uniqueid is a string which allow to do matching between the mod 
security logs and
the haproxy logs. I propose the uniqueid, but this string can 
be another
think like an header containing a uniqueid previously 
generated.


req.hrds_bin is the header bloc in binary format.

req.body_size is the advertised size of the body and it is used to 
determine if the

 body is full or truncated.

Thierry


I personally
think that adding dependencies on tons of libs to implement simple 
stuff
is more 

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 19-04-2017 11:24, schrieb Thierry Fournier:

On 19 Apr 2017, at 09:16, Aleksandar Lazic  wrote:



Am 19-04-2017 05:51, schrieb Willy Tarreau:

On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:

Why not reuse the upcoming http/2 format.
HTTP/2 is *easy* to parse and the implementations of servers are 
growing?
Are you kidding ? I mean you want everyone to have to implement HPACK 
etc ?


Well there are a currently lot of http/2 servers out there, but I 
understand your concerns.


I know this is still on the todo list but I think this is worth to 
go

instead to add another protocol.
Or we can take a look into grpc, which is on top of http/2 with 
protobuffer.

http://www.grpc.io/
As I take more time to think in this direction why not add a grpc 
filter

like the spoa filter?
Opinions?
Code generators are always a pain to deal with. It reminds me the old 
days
when I was using rpcgen. And grpc doesn't provide a C implementation 
so that

kills it :-)


Yeah I have also seen that they prefer c++, c#, go and other languages 
but not offer a c lib.
This is strange just because the core is written in c, but I don't 
need to understand everything ;-)


Thierry told me he's going to implement a simple TLV-like list which 
will

be much easier to implement and easy to code on both sides.


@Thierry:
Ah something like netstrings ?
https://cr.yp.to/proto/netstrings.txt


Hi thanks for the link. I do not like this encoding because it is not 
easy to
parse. I must read ascii representation of numbers and convert-it. I 
prefer
simple binary format, with length (encoding in the same way than the 
SPOE

protocol) and value.


Thanks for clarification.


Hi, the main goal is providing:
 - a format easy to decode
 - not adding lot of dependencies to haproxy


That's very polite.


So, after brainstorm, I propose:

 - remove all data already accessible via existing sample-fetches from 
the new

   sample-fetch introduced by my previous patch. These data are:
   method, path, query string, http-version and the body. All of these 
data are
   available in some sample fetches and it will be great to reuse it. 
Note that

   the only data which keep are the headers.

 - For the headers I propose two format: The first format is for 
general use. It

   is simply the header block in the same format that we received
(with the final
   \r\n for detecting truncated header bloc)

 - For header I propose also a binary format like this:
   
   [
  
  
   ]…
   This format is easy to decode because each length is encoded with 
the SPOE
   numeric values encoding method, so it is always embedded in SPOE 
client.


The SPOE message description will become

   spoe-message check-request
  uniqueid args method path query req.ver req.hdrs_bin
req.body_size req.body

uniqueid is a string which allow to do matching between the mod
security logs and
 the haproxy logs. I propose the uniqueid, but this string can
be another
 think like an header containing a uniqueid previously 
generated.


req.hrds_bin is the header bloc in binary format.

req.body_size is the advertised size of the body and it is used to
determine if the
  body is full or truncated.


Looks good to me.

Regards
Aleks



Thierry


I personally
think that adding dependencies on tons of libs to implement simple 
stuff
is more of a problem than an help even for implementors, because when 
you
start to enter some dependency hell or version incompatibilities, 
it's a

real pain, especially when it was done only to find how to implement
just a small list. These things are useful if you want to implement 
heavy

inter-application communications but it's not really the case here.


I got you.


Cheers,
Willy


Regards
Aleks

Re: ModSecurity: First integration patches

[thierry . fournier]

Hi,

There is a new lot of patches for the spoa/modescurity contrib.

Thierry


On Wed, 19 Apr 2017 11:24:36 +0200
Thierry Fournier  wrote:

> 
> > On 19 Apr 2017, at 09:16, Aleksandar Lazic  wrote:
> > 
> > 
> > 
> > Am 19-04-2017 05:51, schrieb Willy Tarreau:
> >> On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:
> >>> Why not reuse the upcoming http/2 format.
> >>> HTTP/2 is *easy* to parse and the implementations of servers are growing?
> >> Are you kidding ? I mean you want everyone to have to implement HPACK etc ?
> > 
> > Well there are a currently lot of http/2 servers out there, but I 
> > understand your concerns.
> > 
> >>> I know this is still on the todo list but I think this is worth to go
> >>> instead to add another protocol.
> >>> Or we can take a look into grpc, which is on top of http/2 with 
> >>> protobuffer.
> >>> http://www.grpc.io/
> >>> As I take more time to think in this direction why not add a grpc filter
> >>> like the spoa filter?
> >>> Opinions?
> >> Code generators are always a pain to deal with. It reminds me the old days
> >> when I was using rpcgen. And grpc doesn't provide a C implementation so 
> >> that
> >> kills it :-)
> > 
> > Yeah I have also seen that they prefer c++, c#, go and other languages but 
> > not offer a c lib.
> > This is strange just because the core is written in c, but I don't need to 
> > understand everything ;-)
> > 
> >> Thierry told me he's going to implement a simple TLV-like list which will
> >> be much easier to implement and easy to code on both sides.
> > 
> > @Thierry:
> > Ah something like netstrings ?
> > https://cr.yp.to/proto/netstrings.txt
> 
> Hi thanks for the link. I do not like this encoding because it is not easy to
> parse. I must read ascii representation of numbers and convert-it. I prefer
> simple binary format, with length (encoding in the same way than the SPOE
> protocol) and value.
> 
> Hi, the main goal is providing:
>  - a format easy to decode
>  - not adding lot of dependencies to haproxy
> 
> So, after brainstorm, I propose:
> 
>  - remove all data already accessible via existing sample-fetches from the new
>sample-fetch introduced by my previous patch. These data are:
>method, path, query string, http-version and the body. All of these data 
> are
>available in some sample fetches and it will be great to reuse it. Note 
> that
>the only data which keep are the headers.
> 
>  - For the headers I propose two format: The first format is for general use. 
> It
>is simply the header block in the same format that we received (with the 
> final
>\r\n for detecting truncated header bloc)
> 
>  - For header I propose also a binary format like this:
>
>[
>   
>   
>]…
>This format is easy to decode because each length is encoded with the SPOE
>numeric values encoding method, so it is always embedded in SPOE client.
> 
> The SPOE message description will become
> 
>spoe-message check-request
>   uniqueid args method path query req.ver req.hdrs_bin req.body_size 
> req.body
> 
> uniqueid is a string which allow to do matching between the mod security logs 
> and
>  the haproxy logs. I propose the uniqueid, but this string can be 
> another
>  think like an header containing a uniqueid previously generated.
> 
> req.hrds_bin is the header bloc in binary format.
> 
> req.body_size is the advertised size of the body and it is used to determine 
> if the
>   body is full or truncated.
> 
> Thierry
> 
> >> I personally
> >> think that adding dependencies on tons of libs to implement simple stuff
> >> is more of a problem than an help even for implementors, because when you
> >> start to enter some dependency hell or version incompatibilities, it's a
> >> real pain, especially when it was done only to find how to implement
> >> just a small list. These things are useful if you want to implement heavy
> >> inter-application communications but it's not really the case here.
> > 
> > I got you.
> > 
> >> Cheers,
> >> Willy
> > 
> > Regards
> > Aleks
> 
> 
>From 2759c1584f08f397ae11672fa11d260970033406 Mon Sep 17 00:00:00 2001
From: Thierry FOURNIER 
Date: Sun, 9 Apr 2017 05:41:27 +0200
Subject: [PATCH 1/5] BUG/MINOR: change header-declared function to static
 inline

When we include the header proto/spoe.h in other files in the same
project, the compilator claim that the symbol have multiple definitions:

   src/flt_spoe.o: In function `spoe_encode_varint':
   ~/git/haproxy/include/proto/spoe.h:45: multiple definition of `spoe_encode_varint'
   src/proto_http.o:~/git/haproxy/include/proto/spoe.h:45: first defined here
---
 include/proto/spoe.h |   16 
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/include/proto/spoe.h b/include/proto/spoe.h
index 06fb52d..da19db1 100644
--- a/include/proto/spoe.h
+++ b/include/proto/spoe.h
@@ 

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 19 Apr 2017, at 09:16, Aleksandar Lazic  wrote:
> 
> 
> 
> Am 19-04-2017 05:51, schrieb Willy Tarreau:
>> On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:
>>> Why not reuse the upcoming http/2 format.
>>> HTTP/2 is *easy* to parse and the implementations of servers are growing?
>> Are you kidding ? I mean you want everyone to have to implement HPACK etc ?
> 
> Well there are a currently lot of http/2 servers out there, but I understand 
> your concerns.
> 
>>> I know this is still on the todo list but I think this is worth to go
>>> instead to add another protocol.
>>> Or we can take a look into grpc, which is on top of http/2 with protobuffer.
>>> http://www.grpc.io/
>>> As I take more time to think in this direction why not add a grpc filter
>>> like the spoa filter?
>>> Opinions?
>> Code generators are always a pain to deal with. It reminds me the old days
>> when I was using rpcgen. And grpc doesn't provide a C implementation so that
>> kills it :-)
> 
> Yeah I have also seen that they prefer c++, c#, go and other languages but 
> not offer a c lib.
> This is strange just because the core is written in c, but I don't need to 
> understand everything ;-)
> 
>> Thierry told me he's going to implement a simple TLV-like list which will
>> be much easier to implement and easy to code on both sides.
> 
> @Thierry:
> Ah something like netstrings ?
> https://cr.yp.to/proto/netstrings.txt

Hi thanks for the link. I do not like this encoding because it is not easy to
parse. I must read ascii representation of numbers and convert-it. I prefer
simple binary format, with length (encoding in the same way than the SPOE
protocol) and value.

Hi, the main goal is providing:
 - a format easy to decode
 - not adding lot of dependencies to haproxy

So, after brainstorm, I propose:

 - remove all data already accessible via existing sample-fetches from the new
   sample-fetch introduced by my previous patch. These data are:
   method, path, query string, http-version and the body. All of these data are
   available in some sample fetches and it will be great to reuse it. Note that
   the only data which keep are the headers.

 - For the headers I propose two format: The first format is for general use. It
   is simply the header block in the same format that we received (with the 
final
   \r\n for detecting truncated header bloc)

 - For header I propose also a binary format like this:
   
   [
  
  
   ]…
   This format is easy to decode because each length is encoded with the SPOE
   numeric values encoding method, so it is always embedded in SPOE client.

The SPOE message description will become

   spoe-message check-request
  uniqueid args method path query req.ver req.hdrs_bin req.body_size 
req.body

uniqueid is a string which allow to do matching between the mod security logs 
and
 the haproxy logs. I propose the uniqueid, but this string can be 
another
 think like an header containing a uniqueid previously generated.

req.hrds_bin is the header bloc in binary format.

req.body_size is the advertised size of the body and it is used to determine if 
the
  body is full or truncated.

Thierry

>> I personally
>> think that adding dependencies on tons of libs to implement simple stuff
>> is more of a problem than an help even for implementors, because when you
>> start to enter some dependency hell or version incompatibilities, it's a
>> real pain, especially when it was done only to find how to implement
>> just a small list. These things are useful if you want to implement heavy
>> inter-application communications but it's not really the case here.
> 
> I got you.
> 
>> Cheers,
>> Willy
> 
> Regards
> Aleks

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 19-04-2017 05:51, schrieb Willy Tarreau:

On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:

Why not reuse the upcoming http/2 format.
HTTP/2 is *easy* to parse and the implementations of servers are 
growing?


Are you kidding ? I mean you want everyone to have to implement HPACK 
etc ?


Well there are a currently lot of http/2 servers out there, but I 
understand your concerns.



I know this is still on the todo list but I think this is worth to go
instead to add another protocol.

Or we can take a look into grpc, which is on top of http/2 with 
protobuffer.

http://www.grpc.io/

As I take more time to think in this direction why not add a grpc 
filter

like the spoa filter?
Opinions?


Code generators are always a pain to deal with. It reminds me the old 
days
when I was using rpcgen. And grpc doesn't provide a C implementation so 
that

kills it :-)


Yeah I have also seen that they prefer c++, c#, go and other languages 
but not offer a c lib.
This is strange just because the core is written in c, but I don't need 
to understand everything ;-)


Thierry told me he's going to implement a simple TLV-like list which 
will

be much easier to implement and easy to code on both sides.


@Thierry:
Ah something like netstrings ?
https://cr.yp.to/proto/netstrings.txt


I personally
think that adding dependencies on tons of libs to implement simple 
stuff
is more of a problem than an help even for implementors, because when 
you
start to enter some dependency hell or version incompatibilities, it's 
a

real pain, especially when it was done only to find how to implement
just a small list. These things are useful if you want to implement 
heavy

inter-application communications but it's not really the case here.


I got you.


Cheers,
Willy


Regards
Aleks

Re: ModSecurity: First integration patches

[Willy Tarreau]

On Tue, Apr 18, 2017 at 11:55:46PM +0200, Aleksandar Lazic wrote:
> Why not reuse the upcoming http/2 format.
> HTTP/2 is *easy* to parse and the implementations of servers are growing?

Are you kidding ? I mean you want everyone to have to implement HPACK etc ?

> I know this is still on the todo list but I think this is worth to go
> instead to add another protocol.
> 
> Or we can take a look into grpc, which is on top of http/2 with protobuffer.
> http://www.grpc.io/
> 
> As I take more time to think in this direction why not add a grpc filter
> like the spoa filter?
> Opinions?

Code generators are always a pain to deal with. It reminds me the old days
when I was using rpcgen. And grpc doesn't provide a C implementation so that
kills it :-)

Thierry told me he's going to implement a simple TLV-like list which will
be much easier to implement and easy to code on both sides. I personally
think that adding dependencies on tons of libs to implement simple stuff
is more of a problem than an help even for implementors, because when you
start to enter some dependency hell or version incompatibilities, it's a
real pain, especially when it was done only to find how to implement
just a small list. These things are useful if you want to implement heavy
inter-application communications but it's not really the case here.

Cheers,
Willy

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 18-04-2017 15:10, schrieb Willy Tarreau:

On Tue, Apr 18, 2017 at 02:59:20PM +0200, Christopher Faulet wrote:

Le 18/04/2017 à 14:40, Willy Tarreau a écrit :
> On Tue, Apr 18, 2017 at 12:15:20PM +0200, Christopher Faulet wrote:
> > I finally took the time to review your patches, mainly the second one, about
> > the sample fetch. I think it would be pity to introduced such complex sample
> > fetch. All parts, except the HTTP headers, are already available in
> > dedicated sample fetches. It could be better to only add a sample fetch to
> > get HTTP headers (req.hdrs and res.hdrs, something like that). Because a
> > sample fetch cannot return a list, we can probably encode it into a binary
> > buffer using a \0 as separator. something like:
> >
> >
> > 
\0\0\0\0...
> >
> > This way, the sample fetch does not depend on the SPOE and can be used in
> > another context. And concerning your SPOA, this will be quite easy to parse
> > it.
>
> Actually I'm not quite sure about this one, such an encoding could in fact
> make it harder to process while most use cases will either want to log it
> (thus copy the whole string as-is) or decode it as received (and might have
> to rebuild the initial header block with CRLF in between).
>

My first idea was to avoid the headers parsing in the SPOA. Because it 
was
already done by HAProxy, it could be cool to not redo it. Maybe the 
sample
fetch can take an argument to choose the format (raw or encoded). The 
format
is open. \0 may not be the better separator. This is just a trick to 
deal

with a list, like req.hdr_names.


I think there are different needs. I totally agree that if we can build 
a
list to avoid SPOAs having to parse headers it would be cool, but at 
the
same time we need to keep in mind that some components there will 
expect
to receive the request as-is and in this case having to transform it 
twice
needlessly adds some complexity. So in the end I'm all for having a 
req.hdrs
and res.hdrs which contain the exact block as received, just like we 
have
for the body, and on top of this we can have some variants where an 
easy to
use serialization and encoding is applied. Maybe we can even think 
about
header factorization to avoid sending multiple times the same header 
name

with different values. Just an idea.


Why not reuse the upcoming http/2 format.
HTTP/2 is *easy* to parse and the implementations of servers are 
growing?
I know this is still on the todo list but I think this is worth to go 
instead to add another protocol.


Or we can take a look into grpc, which is on top of http/2 with 
protobuffer.

http://www.grpc.io/

As I take more time to think in this direction why not add a grpc filter 
like the spoa filter?

Opinions?
I can take a look to implement it.


Willy


Regards
aleks

Re: ModSecurity: First integration patches

[Willy Tarreau]

On Tue, Apr 18, 2017 at 02:59:20PM +0200, Christopher Faulet wrote:
> Le 18/04/2017 à 14:40, Willy Tarreau a écrit :
> > On Tue, Apr 18, 2017 at 12:15:20PM +0200, Christopher Faulet wrote:
> > > I finally took the time to review your patches, mainly the second one, 
> > > about
> > > the sample fetch. I think it would be pity to introduced such complex 
> > > sample
> > > fetch. All parts, except the HTTP headers, are already available in
> > > dedicated sample fetches. It could be better to only add a sample fetch to
> > > get HTTP headers (req.hdrs and res.hdrs, something like that). Because a
> > > sample fetch cannot return a list, we can probably encode it into a binary
> > > buffer using a \0 as separator. something like:
> > > 
> > > 
> > > \0\0\0\0...
> > > 
> > > This way, the sample fetch does not depend on the SPOE and can be used in
> > > another context. And concerning your SPOA, this will be quite easy to 
> > > parse
> > > it.
> > 
> > Actually I'm not quite sure about this one, such an encoding could in fact
> > make it harder to process while most use cases will either want to log it
> > (thus copy the whole string as-is) or decode it as received (and might have
> > to rebuild the initial header block with CRLF in between).
> > 
> 
> My first idea was to avoid the headers parsing in the SPOA. Because it was
> already done by HAProxy, it could be cool to not redo it. Maybe the sample
> fetch can take an argument to choose the format (raw or encoded). The format
> is open. \0 may not be the better separator. This is just a trick to deal
> with a list, like req.hdr_names.

I think there are different needs. I totally agree that if we can build a
list to avoid SPOAs having to parse headers it would be cool, but at the
same time we need to keep in mind that some components there will expect
to receive the request as-is and in this case having to transform it twice
needlessly adds some complexity. So in the end I'm all for having a req.hdrs
and res.hdrs which contain the exact block as received, just like we have
for the body, and on top of this we can have some variants where an easy to
use serialization and encoding is applied. Maybe we can even think about
header factorization to avoid sending multiple times the same header name
with different values. Just an idea.

Willy

Re: ModSecurity: First integration patches

[Christopher Faulet]

Le 18/04/2017 à 14:40, Willy Tarreau a écrit :

On Tue, Apr 18, 2017 at 12:15:20PM +0200, Christopher Faulet wrote:

I finally took the time to review your patches, mainly the second one, about
the sample fetch. I think it would be pity to introduced such complex sample
fetch. All parts, except the HTTP headers, are already available in
dedicated sample fetches. It could be better to only add a sample fetch to
get HTTP headers (req.hdrs and res.hdrs, something like that). Because a
sample fetch cannot return a list, we can probably encode it into a binary
buffer using a \0 as separator. something like:


\0\0\0\0...

This way, the sample fetch does not depend on the SPOE and can be used in
another context. And concerning your SPOA, this will be quite easy to parse
it.


Actually I'm not quite sure about this one, such an encoding could in fact
make it harder to process while most use cases will either want to log it
(thus copy the whole string as-is) or decode it as received (and might have
to rebuild the initial header block with CRLF in between).



My first idea was to avoid the headers parsing in the SPOA. Because it 
was already done by HAProxy, it could be cool to not redo it. Maybe the 
sample fetch can take an argument to choose the format (raw or encoded). 
The format is open. \0 may not be the better separator. This is just a 
trick to deal with a list, like req.hdr_names.



However I agree that having something like req.hdrs / res.hdrs could be
useful instead of having the whole block containing all the buffer. We
already have "req.body" for the body as a binary block, so it totally
makes sense to have "req.hdrs" for the same purpose, and let agents deal
with either one or the other or both.



--
Christopher Faulet

Re: ModSecurity: First integration patches

[Willy Tarreau]

On Tue, Apr 18, 2017 at 12:15:20PM +0200, Christopher Faulet wrote:
> I finally took the time to review your patches, mainly the second one, about
> the sample fetch. I think it would be pity to introduced such complex sample
> fetch. All parts, except the HTTP headers, are already available in
> dedicated sample fetches. It could be better to only add a sample fetch to
> get HTTP headers (req.hdrs and res.hdrs, something like that). Because a
> sample fetch cannot return a list, we can probably encode it into a binary
> buffer using a \0 as separator. something like:
> 
> 
> \0\0\0\0...
> 
> This way, the sample fetch does not depend on the SPOE and can be used in
> another context. And concerning your SPOA, this will be quite easy to parse
> it.

Actually I'm not quite sure about this one, such an encoding could in fact
make it harder to process while most use cases will either want to log it
(thus copy the whole string as-is) or decode it as received (and might have
to rebuild the initial header block with CRLF in between).

However I agree that having something like req.hdrs / res.hdrs could be
useful instead of having the whole block containing all the buffer. We
already have "req.body" for the body as a binary block, so it totally
makes sense to have "req.hdrs" for the same purpose, and let agents deal
with either one or the other or both.

Willy

Re: ModSecurity: First integration patches

[Christopher Faulet]

Le 12/04/2017 à 10:49, Christopher Faulet a écrit :

Le 11/04/2017 à 10:49, Thierry Fournier a écrit :

Hi list

I join one usage of HAProxy / SPOE, it is WAF offloading.

These patches are a first version, it have some limitations describe
in the README file in the directory contrib/modsecurity.

  - Christopher, please check the patch "BUG/MINOR", it is about spoe
functions.

  - The exemple of ModSecurity compilation can be improved. It is based
on my local distro.

The feedback are welcome.



Hi Thierry,

Really nice ! I'll take a look at it soon. Glad to see the first service
that uses the SPOE ! Good job.



Hi Thierry,

I finally took the time to review your patches, mainly the second one, 
about the sample fetch. I think it would be pity to introduced such 
complex sample fetch. All parts, except the HTTP headers, are already 
available in dedicated sample fetches. It could be better to only add a 
sample fetch to get HTTP headers (req.hdrs and res.hdrs, something like 
that). Because a sample fetch cannot return a list, we can probably 
encode it into a binary buffer using a \0 as separator. something like:



\0\0\0\0...

This way, the sample fetch does not depend on the SPOE and can be used 
in another context. And concerning your SPOA, this will be quite easy to 
parse it.


About the SPOA, it seems to be ok. The server part is based on the SPOA 
example so it should be ok (or you can blame me for all bugs :) For the 
mod_security part, I blindly trust you.


--
Christopher Faulet

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi Willy.

Am 14-04-2017 11:41, schrieb Willy Tarreau:

Hi Aleks,

On Fri, Apr 14, 2017 at 10:25:56AM +0200, Aleksandar Lazic wrote:

Willy can you please commit this patches.


I'm fine with this, but I prefer to give some time to Christopher to
review this, as Thierry asked. It can take quite some time since the
purpse of cutting the development cycle into 3 phases precisely was
to let developers stop spending time reviewing code and focus on their
code instead :-)


;-)


But here it's going to affect only third-party code in contrib/ so the
risks for haproxy's stability don't exist and we can just take a look.
I just want not to interrupt Chris right now.


Perfect thanks.


Cheers,
Willy


cheers
Aleks

Re: ModSecurity: First integration patches

[Willy Tarreau]

Hi Aleks,

On Fri, Apr 14, 2017 at 10:25:56AM +0200, Aleksandar Lazic wrote:
> Willy can you please commit this patches.

I'm fine with this, but I prefer to give some time to Christopher to
review this, as Thierry asked. It can take quite some time since the
purpse of cutting the development cycle into 3 phases precisely was
to let developers stop spending time reviewing code and focus on their
code instead :-)

But here it's going to affect only third-party code in contrib/ so the
risks for haproxy's stability don't exist and we can just take a look.
I just want not to interrupt Chris right now.

Cheers,
Willy

Re: ModSecurity: First integration patches

[Willy Tarreau]

On Thu, Apr 13, 2017 at 01:17:07PM +0200, Christopher Faulet wrote:
> The hello-handshake is done only once, when a new connection with a SPOA is
> opened. But we can improve the SPOP by adding a new frame type to handle
> admin tasks (like graceful stop). This way, for a specific connection, it
> would be possible to wait for last ACK frames without sending new frames to
> the SPOA. Then stopping the SPOA listeners to let the SPOP health check
> failed should do the trick, I guess.

For this you need a "closing" state for your connections, where you know
they are not eligible to new requests. That's pretty much similar to what
we do by setting a server weight to zero in HTTP load balancing.

But such transient states for connections can quickly become tricky to
handle. The first thing is that you want to be sure that the replacement
process is ready or the agent has reported a zero weight before starting
to notify about the closing state, otherwise you'll end up with single-
request connections that quickly pile up and can even take CPU if they're
made over SSL. And it's very important not to conflate the agent state
with the connection state and to avoid reporting the agent state inside
existing connections. The problem when doing this precisely is during
reloads where both old and new connections coexist and report conflicting
information. That's why health checks and/or agent checks are much better
to get the active SPOA health. 

For example the agent's health and willingness to accept new connections
could be reported during the handshake. This way a health check can be
developped for this and there's no risk that such information could be
misreported later and cause trouble.

Willy

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi.

Am 13-04-2017 02:06, schrieb Aleksandar Lazic:

Am 12-04-2017 23:33, schrieb Aleksandar Lazic:

Am 12-04-2017 21:28, schrieb thierry.fourn...@arpalert.org:

On Wed, 12 Apr 2017 21:21:58 +0200
Aleksandar Lazic  wrote:


[snipp]


Do you have the patches as files where I can download it?
It's easier for docker to call a 'curl -vLO ...' then to go across a
mail body ;-)


Not sure to understand. I given the patches as file. Note that I'm
testing new email client. So I put the patches here:

http://www.arpalert.org/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
http://www.arpalert.org/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
http://www.arpalert.org/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch


[snipp]

Willy can you please commit this patches.

Thank you very much.

Cheers.
Aleks

Re: ModSecurity: First integration patches

[Christopher Faulet]

Le 13/04/2017 à 12:53, Thierry Fournier a écrit :



On 13 Apr 2017, at 12:28, Willy Tarreau  wrote:

On Thu, Apr 13, 2017 at 12:21:20PM +0200, Thierry Fournier wrote:

.) the patches apply only on haproxy 1.8 because some files does not exists on 
1.7 ( e. g. include/proto/spoe.h )



Ok. I think that SPOE was introduced in 1.7, obviously I'm wrong.


No, it was introduced in 1.7 but there were some improvements later
(like pipelining etc).

(...)

.) How can the rule-set be reloaded? stop & start || gracefully?



I do not process this part. Today, you must stop and start the process. The 
graceful doesn't exists.
I guess than the graceful can be implemented easily. You can ensure the 
availability of the
SPOA Modsec using the properties of the HAProxy backend.


Actually that's a very good point. I think it would even be possible to
ensure a graceful shutdown using disable-on-404 or using an agent so
that you can roll the restart over multiple WAF nodes.



Interesting. I think about a system which (on SPOA side) stop listeners
and wait for the end of processing current requests. By this way, the SPOA
doesn’t accept requests, and HAProxy send requests on the other process.
Another way is using the CLI and set one spoa/modsec in graceful mode.

Adding a special check is the best way, but the daemon speaks SPOP and not
HTTP. Maybe a thread which listen on specific port dedicated to this
function ? Or improving the SPOP for asking graceful mode in the agent-hello
response message ? (it seems that haproxy send periodically haproxy-hello
messages, but maybe I’m wrong)



The hello-handshake is done only once, when a new connection with a SPOA 
is opened. But we can improve the SPOP by adding a new frame type to 
handle admin tasks (like graceful stop). This way, for a specific 
connection, it would be possible to wait for last ACK frames without 
sending new frames to the SPOA. Then stopping the SPOA listeners to let 
the SPOP health check failed should do the trick, I guess.


--
Christopher Faulet

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 13 Apr 2017, at 12:28, Willy Tarreau  wrote:
> 
> On Thu, Apr 13, 2017 at 12:21:20PM +0200, Thierry Fournier wrote:
>>> .) the patches apply only on haproxy 1.8 because some files does not exists 
>>> on 1.7 ( e. g. include/proto/spoe.h )
>> 
>> 
>> Ok. I think that SPOE was introduced in 1.7, obviously I'm wrong.
> 
> No, it was introduced in 1.7 but there were some improvements later
> (like pipelining etc).
> 
> (...)
>>> .) How can the rule-set be reloaded? stop & start || gracefully?
>> 
>> 
>> I do not process this part. Today, you must stop and start the process. The 
>> graceful doesn't exists.
>> I guess than the graceful can be implemented easily. You can ensure the 
>> availability of the
>> SPOA Modsec using the properties of the HAProxy backend.
> 
> Actually that's a very good point. I think it would even be possible to
> ensure a graceful shutdown using disable-on-404 or using an agent so
> that you can roll the restart over multiple WAF nodes.


Interesting. I think about a system which (on SPOA side) stop listeners
and wait for the end of processing current requests. By this way, the SPOA
doesn’t accept requests, and HAProxy send requests on the other process.
Another way is using the CLI and set one spoa/modsec in graceful mode.

Adding a special check is the best way, but the daemon speaks SPOP and not
HTTP. Maybe a thread which listen on specific port dedicated to this
function ? Or improving the SPOP for asking graceful mode in the agent-hello
response message ? (it seems that haproxy send periodically haproxy-hello
messages, but maybe I’m wrong)

Thierry


> Willy

Re: ModSecurity: First integration patches

[Willy Tarreau]

On Thu, Apr 13, 2017 at 12:21:20PM +0200, Thierry Fournier wrote:
> > .) the patches apply only on haproxy 1.8 because some files does not exists 
> > on 1.7 ( e. g. include/proto/spoe.h )
> 
> 
> Ok. I think that SPOE was introduced in 1.7, obviously I'm wrong.

No, it was introduced in 1.7 but there were some improvements later
(like pipelining etc).

(...)
> > .) How can the rule-set be reloaded? stop & start || gracefully?
> 
> 
> I do not process this part. Today, you must stop and start the process. The 
> graceful doesn't exists.
> I guess than the graceful can be implemented easily. You can ensure the 
> availability of the
> SPOA Modsec using the properties of the HAProxy backend.

Actually that's a very good point. I think it would even be possible to
ensure a graceful shutdown using disable-on-404 or using an agent so
that you can roll the restart over multiple WAF nodes.

Willy

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 13 Apr 2017, at 02:06, Aleksandar Lazic  wrote:
> 
> 
> 
> Am 12-04-2017 23:33, schrieb Aleksandar Lazic:
>> Am 12-04-2017 21:28, schrieb thierry.fourn...@arpalert.org:
>>> On Wed, 12 Apr 2017 21:21:58 +0200
>>> Aleksandar Lazic  wrote:
> 
> [snipp]
> 
 Do you have the patches as files where I can download it?
 It's easier for docker to call a 'curl -vLO ...' then to go across a
 mail body ;-)
>>> Not sure to understand. I given the patches as file. Note that I'm
>>> testing new email client. So I put the patches here:
>>> http://www.arpalert.org/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
>>> http://www.arpalert.org/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
>>> http://www.arpalert.org/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch
>> I'm so sorry for the rush. :-(
>> I have seen to late that you have send the patches to the list.
>> Thanks for the links. I will take more care in the future.
> 
> I have now build the haproxy with modsecurity on centos 7.3 ;-)
> 
> I have used this file for modsecurity.
> https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/v3.0/master/crs-setup.conf.example
> 
> ###
> /usr/local/bin/modsecurity -f crs-setup.conf.example
> 1492041223.145110 [00] ModSecurity for nginx (STABLE)/2.9.1 
> (http://www.modsecurity.org/) configured.
> 1492041223.145159 [00] ModSecurity: APR compiled version="1.4.8"; loaded 
> version="1.4.8"
> 1492041223.145193 [00] ModSecurity: PCRE compiled version="8.32 "; loaded 
> version="8.32 2012-11-30"
> 1492041223.145197 [00] ModSecurity: LIBXML compiled version="2.9.1"
> 1492041223.145200 [00] ModSecurity: Status engine is currently disabled, 
> enable it by set SecStatusEngine to On.
> 1492041228.152877 [01] 0 clients connected
> 1492041228.153037 [02] 0 clients connected
> 1492041228.153069 [03] 0 clients connected
> ...
> ###
> 
> It was a little bit challenging.
> 
> .) the patches apply only on haproxy 1.8 because some files does not exists 
> on 1.7 ( e. g. include/proto/spoe.h )


Ok. I think that SPOE was introduced in 1.7, obviously I’m wrong.


> git clone http://git.haproxy.org/git/haproxy.git/
> 
> patch -d haproxy -p 1 -i 
> /usr/src/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
> patch -d haproxy -p 1 -i 
> /usr/src/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
> patch -d haproxy -p 1 -i 
> /usr/src/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch
> 
> .) you will need a lot of devel packages inclusive some httpd one.
> 
> yum install -y apr-devel apr-util-devel gcc make libevent-devel libxml2-devel 
> libcurl-devel httpd-devel pcre-devel yajl-devel


Yes Modsecurity is linked designed for apache and needs Apache libraries (APR), 
libevent is for
the SPOA. libcurl and yajl are used for the Modsecurity “mlogc” function.


> .) I will figure out which runtime packages will be necessary.
> .) I have started a Dockerfile which you can find at github.
> 
> https://github.com/git001/haproxy-waf/blob/master/Dockerfile
> 
> Open questions for me.


Note: I swapped the order of your questions


> .) How big can a content be? Where can we define some limits?


ModSecurity analyses an Haproxy buffer. (don’t forget the directive “option 
http-buffer-request”)
For my own usage, the HAProxy buffer are configured as 1MB. When the buffer is 
full or when
the http request is receive, all the data are offloaded towards ModSecurity.


> .) How is the transfer-encoding handled (a. k. a. streaming)?


The stream is not processed, just the first buffer containing the header 
request and a maximum
of the body it is.


> .) How can the rule-set be reloaded? stop & start || gracefully?


I do not process this part. Today, you must stop and start the process. The 
graceful doesn’t exists.
I guess than the graceful can be implemented easily. You can ensure the 
availability of the
SPOA Modsec using the properties of the HAProxy backend.


> Again thanks Thierry for your work this looks very good.


Thanks for testing.

Thierry


> Regards
> Aleks

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 12-04-2017 23:33, schrieb Aleksandar Lazic:

Am 12-04-2017 21:28, schrieb thierry.fourn...@arpalert.org:

On Wed, 12 Apr 2017 21:21:58 +0200
Aleksandar Lazic  wrote:


[snipp]


Do you have the patches as files where I can download it?
It's easier for docker to call a 'curl -vLO ...' then to go across a
mail body ;-)


Not sure to understand. I given the patches as file. Note that I'm
testing new email client. So I put the patches here:

http://www.arpalert.org/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
http://www.arpalert.org/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
http://www.arpalert.org/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch


I'm so sorry for the rush. :-(

I have seen to late that you have send the patches to the list.

Thanks for the links. I will take more care in the future.


I have now build the haproxy with modsecurity on centos 7.3 ;-)

I have used this file for modsecurity.
https://github.com/SpiderLabs/owasp-modsecurity-crs/blob/v3.0/master/crs-setup.conf.example

###
/usr/local/bin/modsecurity -f crs-setup.conf.example
1492041223.145110 [00] ModSecurity for nginx (STABLE)/2.9.1 
(http://www.modsecurity.org/) configured.
1492041223.145159 [00] ModSecurity: APR compiled version="1.4.8"; loaded 
version="1.4.8"
1492041223.145193 [00] ModSecurity: PCRE compiled version="8.32 "; 
loaded version="8.32 2012-11-30"

1492041223.145197 [00] ModSecurity: LIBXML compiled version="2.9.1"
1492041223.145200 [00] ModSecurity: Status engine is currently disabled, 
enable it by set SecStatusEngine to On.

1492041228.152877 [01] 0 clients connected
1492041228.153037 [02] 0 clients connected
1492041228.153069 [03] 0 clients connected
...
###

It was a little bit challenging.

.) the patches apply only on haproxy 1.8 because some files does not 
exists on 1.7 ( e. g. include/proto/spoe.h )


git clone http://git.haproxy.org/git/haproxy.git/

patch -d haproxy -p 1 -i 
/usr/src/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
patch -d haproxy -p 1 -i 
/usr/src/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
patch -d haproxy -p 1 -i 
/usr/src/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch


.) you will need a lot of devel packages inclusive some httpd one.

yum install -y apr-devel apr-util-devel gcc make libevent-devel 
libxml2-devel libcurl-devel httpd-devel pcre-devel yajl-devel


.) I will figure out which runtime packages will be necessary.
.) I have started a Dockerfile which you can find at github.

https://github.com/git001/haproxy-waf/blob/master/Dockerfile

Open questions for me.

.) How is the transfer-encoding handled (a. k. a. streaming)?
.) How big can a content be? Where can we define some limits?
.) How can the rule-set be reloaded? stop & start || gracefully?

Again thanks Thierry for your work this looks very good.

Regards
Aleks

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 12-04-2017 21:28, schrieb thierry.fourn...@arpalert.org:

On Wed, 12 Apr 2017 21:21:58 +0200
Aleksandar Lazic  wrote:


Hi.

Am 12-04-2017 10:08, schrieb Thierry Fournier:
>> On 12 Apr 2017, at 09:57, Aleksandar Lazic  wrote:
>>
>>
>>
>> Am 11-04-2017 10:49, schrieb Thierry Fournier:
>>> Hi list
>>> I join one usage of HAProxy / SPOE, it is WAF offloading.
>>> These patches are a first version, it have some limitations describe
>>> in the README file in the directory contrib/modsecurity.
>>> - Christopher, please check the patch "BUG/MINOR", it is about spoe
>>>   functions.
>>> - The exemple of ModSecurity compilation can be improved. It is based
>>>   on my local distro.
>>> The feedback are welcome.
>>
>> I agree with Olivier that's really great stuff ;-)
>
>
> thanks.
>
>
>> To which branch (I assume master) can I apply this patch?
>
>
> You’re right. The core haproxy patch is very light, I guess that the
> patch will be applied on any branch from 1.7.0 (needs SPOE)
>
>
>> I will then create a docker image for testing ;-)
>
>
> Great !

Do you have the patches as files where I can download it?
It's easier for docker to call a 'curl -vLO ...' then to go across a
mail body ;-)



Not sure to understand. I given the patches as file. Note that I'm
testing new email client. So I put the patches here:

http://www.arpalert.org/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
http://www.arpalert.org/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
http://www.arpalert.org/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch


I'm so sorry for the rush. :-(

I have seen to late that you have send the patches to the list.

Thanks for the links. I will take more care in the future.


Thierry


Aleks

Re: ModSecurity: First integration patches

[thierry . fournier]

On Wed, 12 Apr 2017 21:21:58 +0200
Aleksandar Lazic  wrote:

> Hi.
> 
> Am 12-04-2017 10:08, schrieb Thierry Fournier:
> >> On 12 Apr 2017, at 09:57, Aleksandar Lazic  wrote:
> >> 
> >> 
> >> 
> >> Am 11-04-2017 10:49, schrieb Thierry Fournier:
> >>> Hi list
> >>> I join one usage of HAProxy / SPOE, it is WAF offloading.
> >>> These patches are a first version, it have some limitations describe
> >>> in the README file in the directory contrib/modsecurity.
> >>> - Christopher, please check the patch "BUG/MINOR", it is about spoe
> >>>   functions.
> >>> - The exemple of ModSecurity compilation can be improved. It is based
> >>>   on my local distro.
> >>> The feedback are welcome.
> >> 
> >> I agree with Olivier that's really great stuff ;-)
> > 
> > 
> > thanks.
> > 
> > 
> >> To which branch (I assume master) can I apply this patch?
> > 
> > 
> > You’re right. The core haproxy patch is very light, I guess that the
> > patch will be applied on any branch from 1.7.0 (needs SPOE)
> > 
> > 
> >> I will then create a docker image for testing ;-)
> > 
> > 
> > Great !
> 
> Do you have the patches as files where I can download it?
> It's easier for docker to call a 'curl -vLO ...' then to go across a 
> mail body ;-)


Not sure to understand. I given the patches as file. Note that I'm
testing new email client. So I put the patches here:

   
http://www.arpalert.org/0001-BUG-MINOR-change-header-declared-function-to-static-.patch
   
http://www.arpalert.org/0002-MINOR-Add-binary-encoding-request-sample-fetch.patch
   http://www.arpalert.org/0003-MINOR-Add-ModSecurity-wrapper-as-contrib.patch

Thierry

> >>> Thierry
> >> 
> >> Aleks
> 

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Hi.

Am 12-04-2017 10:08, schrieb Thierry Fournier:

On 12 Apr 2017, at 09:57, Aleksandar Lazic  wrote:



Am 11-04-2017 10:49, schrieb Thierry Fournier:

Hi list
I join one usage of HAProxy / SPOE, it is WAF offloading.
These patches are a first version, it have some limitations describe
in the README file in the directory contrib/modsecurity.
- Christopher, please check the patch "BUG/MINOR", it is about spoe
  functions.
- The exemple of ModSecurity compilation can be improved. It is based
  on my local distro.
The feedback are welcome.


I agree with Olivier that's really great stuff ;-)



thanks.



To which branch (I assume master) can I apply this patch?



You’re right. The core haproxy patch is very light, I guess that the
patch will be applied on any branch from 1.7.0 (needs SPOE)



I will then create a docker image for testing ;-)



Great !


Do you have the patches as files where I can download it?
It's easier for docker to call a 'curl -vLO ...' then to go across a 
mail body ;-)



Thierry


Aleks

Re: ModSecurity: First integration patches

[Christopher Faulet]

Le 11/04/2017 à 10:49, Thierry Fournier a écrit :

Hi list

I join one usage of HAProxy / SPOE, it is WAF offloading.

These patches are a first version, it have some limitations describe
in the README file in the directory contrib/modsecurity.

 - Christopher, please check the patch "BUG/MINOR", it is about spoe
   functions.

 - The exemple of ModSecurity compilation can be improved. It is based
   on my local distro.

The feedback are welcome.



Hi Thierry,

Really nice ! I'll take a look at it soon. Glad to see the first service 
that uses the SPOE ! Good job.


--
Christopher Faulet

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 12 Apr 2017, at 09:57, Aleksandar Lazic  wrote:
> 
> 
> 
> Am 11-04-2017 10:49, schrieb Thierry Fournier:
>> Hi list
>> I join one usage of HAProxy / SPOE, it is WAF offloading.
>> These patches are a first version, it have some limitations describe
>> in the README file in the directory contrib/modsecurity.
>> - Christopher, please check the patch "BUG/MINOR", it is about spoe
>>   functions.
>> - The exemple of ModSecurity compilation can be improved. It is based
>>   on my local distro.
>> The feedback are welcome.
> 
> I agree with Olivier that's really great stuff ;-)


thanks.


> To which branch (I assume master) can I apply this patch?


You’re right. The core haproxy patch is very light, I guess that the patch will 
be applied on any branch from 1.7.0 (needs SPOE)


> I will then create a docker image for testing ;-)


Great !


>> Thierry
> 
> Aleks

Re: ModSecurity: First integration patches

[Aleksandar Lazic]

Am 11-04-2017 10:49, schrieb Thierry Fournier:

Hi list

I join one usage of HAProxy / SPOE, it is WAF offloading.

These patches are a first version, it have some limitations describe
in the README file in the directory contrib/modsecurity.

 - Christopher, please check the patch "BUG/MINOR", it is about spoe
   functions.

 - The exemple of ModSecurity compilation can be improved. It is based
   on my local distro.

The feedback are welcome.


I agree with Olivier that's really great stuff ;-)

To which branch (I assume master) can I apply this patch?

I will then create a docker image for testing ;-)


Thierry


Aleks

Re: ModSecurity: First integration patches

[Thierry Fournier]

> On 11 Apr 2017, at 11:24, Olivier Doucet  wrote:
> 
> Hi Thierry,
> 
> 
> 
> 2017-04-11 10:49 GMT+02:00 Thierry Fournier :
> Hi list
> 
> I join one usage of HAProxy / SPOE, it is WAF offloading.
> 
> These patches are a first version, it have some limitations describe
> in the README file in the directory contrib/modsecurity.
> 
>  - Christopher, please check the patch "BUG/MINOR", it is about spoe
>functions.
> 
>  - The exemple of ModSecurity compilation can be improved. It is based
>on my local distro.
> 
> The feedback are welcome.
> 
> Nice work for a first version ! I definitely see what great production usage 
> could be done with it. 


thanks.


> Your README file referenced several issues in ModSecurity itself, this is not 
> something I would expect from this highly used software …


If you’re taking about the section ModSecurity bug, you must keep in mind that
ModSecurity is written for Apache. The integration in HAProxy or NGINX is not
“natural” for these soft. (Hi hope that the V3 will embed a more compatible 
API).

The first bug is maybe due to my build system. In first time, I compile my own
version of APR, and maybe it is caused by a bad compile parameter. In second 
time, I used APR from system, and I have the same bug.

I have found some posts on Internet about this bug, but ant way to fix it.
I suppose that this bug is not available with Apache. The configuration way to
avoir this bug is using this conf: “SecAuditLogType Concurrent"

The second bug is averred only with HAProxy and NGINX. ModSecurity with
Apache doesn’t have this bug. I will be avoided without using wildcards. So, 
you can
lists each included configuration file.


> What happened if for some reason modSecurity does not answer in the timeout 
> defined ?


Good question. I suppose that the answer is around the SPOE timeouts.


> What happened if modsecurity throws an error ?


The variable “txn.modsec.code” is not set. So this variable is set to 0 if all 
is good,
not set if an error occurs, or it contains the recommended HTTP return code.

Thierry


> Olivier
> 

Re: ModSecurity: First integration patches

[Olivier Doucet]

Hi Thierry,



2017-04-11 10:49 GMT+02:00 Thierry Fournier :

> Hi list
>
> I join one usage of HAProxy / SPOE, it is WAF offloading.
>
> These patches are a first version, it have some limitations describe
> in the README file in the directory contrib/modsecurity.
>
>  - Christopher, please check the patch "BUG/MINOR", it is about spoe
>functions.
>
>  - The exemple of ModSecurity compilation can be improved. It is based
>on my local distro.
>
> The feedback are welcome.
>

Nice work for a first version ! I definitely see what great production
usage could be done with it.

Your README file referenced several issues in ModSecurity itself, this is
not something I would expect from this highly used software ...

What happened if for some reason modSecurity does not answer in the timeout
defined ? What happened if modsecurity throws an error ?


Olivier