Re: [tcp|http]-check expect status explained

2020-05-06 Thread Christopher Faulet

Le 07/05/2020 à 00:06, Aleksandar Lazic a écrit :

On 07.05.20 00:02, Lukas Tribus wrote:

On Wed, 6 May 2020 at 23:33, Aleksandar Lazic  wrote:


Hi.

The doc for [tcp|http]-check expect have some *-status arguments like "L7OK", 
"L7OKC","L6OK" and "L4OK" and so on.

In the whole documentation are this states not explained.
I'm not sure in which chapter this states fit's, quick reminder HTTP,global, 
logging, new chapter?
My suggestion is to add "1.2." HTTP in HAProxy and explain how the htx works 
and what layer 4+6+7 means, opinions.


It's not in the configuration documentation, it's in the management doc:

https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.1


Thanks, I don't look very often there but I should.


Hi Aleks,

You're right, it is not obvious. These status are reported in the stats and are 
described in the management guide as Lukas said. But it is probably a good idea 
to be more explicit. I slightly updated the configuration documentation to not 
rely on internal names. I added a short description for each status instead.


Thanks,
--
Christopher Faulet



Re: [tcp|http]-check expect status explained

2020-05-06 Thread Aleksandar Lazic
On 07.05.20 00:02, Lukas Tribus wrote:
> On Wed, 6 May 2020 at 23:33, Aleksandar Lazic  wrote:
>>
>> Hi.
>>
>> The doc for [tcp|http]-check expect have some *-status arguments like 
>> "L7OK", "L7OKC","L6OK" and "L4OK" and so on.
>>
>> In the whole documentation are this states not explained.
>> I'm not sure in which chapter this states fit's, quick reminder HTTP,global, 
>> logging, new chapter?
>> My suggestion is to add "1.2." HTTP in HAProxy and explain how the htx works 
>> and what layer 4+6+7 means, opinions.
> 
> It's not in the configuration documentation, it's in the management doc:
> 
> https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.1

Thanks, I don't look very often there but I should.

> Lukas

Regards
Aleks



Re: [tcp|http]-check expect status explained

2020-05-06 Thread Lukas Tribus
On Wed, 6 May 2020 at 23:33, Aleksandar Lazic  wrote:
>
> Hi.
>
> The doc for [tcp|http]-check expect have some *-status arguments like "L7OK", 
> "L7OKC","L6OK" and "L4OK" and so on.
>
> In the whole documentation are this states not explained.
> I'm not sure in which chapter this states fit's, quick reminder HTTP,global, 
> logging, new chapter?
> My suggestion is to add "1.2." HTTP in HAProxy and explain how the htx works 
> and what layer 4+6+7 means, opinions.

It's not in the configuration documentation, it's in the management doc:

https://cbonte.github.io/haproxy-dconv/2.0/management.html#9.1


Lukas



Re: about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread Lukas Tribus
Hello,

On Wed, 6 May 2020 at 20:25, William Lallemand  wrote:
> > As such I think it's about time we change the default value to 2048 and
> > get rid of this annoying warning before 2.2 gets released (and at the
> > same time 86% of the users will be able to remove one cryptic line in
> > their config). This way those who don't know/need it will be more
> > secure by default and those who need it will still be able to.
> >
> > Does anyone have any objection or alternate recommendation ?
> >
> > Thanks,
> > Willy
>
>
> I'm fine with that, most people use at least a value of 2048 because of
> the warning, their modern distribution will probably deny a lower value,
> and we add this warning a long time ago.

I agree, we should default to 2048 and remove warnings.


Lukas



[tcp|http]-check expect status explained

2020-05-06 Thread Aleksandar Lazic
Hi.

The doc for [tcp|http]-check expect have some *-status arguments like "L7OK", 
"L7OKC","L6OK" and "L4OK" and so on.

In the whole documentation are this states not explained.
I'm not sure in which chapter this states fit's, quick reminder HTTP,global, 
logging, new chapter?
My suggestion is to add "1.2." HTTP in HAProxy and explain how the htx works 
and what layer 4+6+7 means, opinions.

I ask because I would like to create a table form the source to the 
documentation.

http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/checks.c;h=d5306defe05f77039562ba30fd1c1fcfec4b8f62;hb=HEAD#l106

http://git.haproxy.org/?p=haproxy.git;a=blob;f=src/checks.c;h=d5306defe05f77039562ba30fd1c1fcfec4b8f62;hb=HEAD#l139

Regards

Aleks



Re: about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread William Lallemand
On Wed, May 06, 2020 at 08:25:06PM +0200, William Lallemand wrote:
> I recall a discussion where the default openssl.cnf in some distribution
> was denying a DH lower than 2048. You probably think about this one.
>

Found the commit related to this:
https://github.com/haproxy/haproxy/commit/a9363eb6a58b3057d7d75ced4c8c711cf093c364


-- 
William Lallemand



Re: about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread William Lallemand
On Wed, May 06, 2020 at 07:59:55PM +0200, Willy Tarreau wrote:
> Hi all,
> 
> while running on a trivial test config in which I had enabled
> "zero-warning", my process refused to start due to the good old
> warning "Setting tune.ssl.default-dh-param to 1024 blah blah".
> 
> I was almost certain we discussed about switching the default value
> to 2048 for 2.0 or 2.1 but couldn't find any trace of this, so I must
> have dreamed or discussed it in person. I've run a quick check on the
> configs shared on the list over the last two years and found this:
> 

I recall a discussion where the default openssl.cnf in some distribution
was denying a DH lower than 2048. You probably think about this one.

> $ tail  -c80m Mail/lists/haproxy-ml | grep -o 'tune.ssl.default-dh-param[ 
> ]\+[0-9]\+' | awk '{print $1,$2}' |sort|uniq -c|sort -n
>   1 tune.ssl.default-dh-param 4096
>  13 tune.ssl.default-dh-param 1024
>  86 tune.ssl.default-dh-param 2048
> 
> Thus it seems that the vast majority of users (exactly 86%) prefer to
> use 2048 which is also the one recommended in the warning. All I found
> on the subject was in fact added to the doc by Rémi who implemented the
> tunable 6 years ago (commit f46cd6e4ec), and he warned:
> 
>values greater than 1024 bits are not supported by Java 7 and
>earlier clients
> 
> Do we still really care given how old this is now and that users can
> still force the value if they absolutely need it ?
> 
> As such I think it's about time we change the default value to 2048 and
> get rid of this annoying warning before 2.2 gets released (and at the
> same time 86% of the users will be able to remove one cryptic line in
> their config). This way those who don't know/need it will be more
> secure by default and those who need it will still be able to.
> 
> Does anyone have any objection or alternate recommendation ?
> 
> Thanks,
> Willy


I'm fine with that, most people use at least a value of 2048 because of
the warning, their modern distribution will probably deny a lower value,
and we add this warning a long time ago.

-- 
William Lallemand



Re: about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread Willy Tarreau
On Wed, May 06, 2020 at 06:10:26PM +, Branitsky, Norman wrote:
> New RHEL 8 Crypto Configuration mentioned in:
> 
> 
> 
> From:  ??? 
> 
> Sent: Wednesday, May 6, 2020 5:34 AM
> 
> To: HAProxy 
> 
> Subject: running haproxy with predefined security policies on RHEL8 ?
> 
> 
> 
> Hello,
> 
> do we have any experience of 
> https://www.redhat.com/en/blog/consistent-security-crypto-policies-red-hat-enterprise-linux-8
> 
> 
> 
> defines FUTURE configuration as:
> 
> no SHA-1 signatures
> 
> DH and RSA parameters minimum 3072

Thank you Norman, so that would actually encourage to go into this
direction as well.

Willy



RE: about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread Branitsky, Norman
New RHEL 8 Crypto Configuration mentioned in:



From: Илья Шипицин 

Sent: Wednesday, May 6, 2020 5:34 AM

To: HAProxy 

Subject: running haproxy with predefined security policies on RHEL8 ?



Hello,

do we have any experience of 
https://www.redhat.com/en/blog/consistent-security-crypto-policies-red-hat-enterprise-linux-8



defines FUTURE configuration as:

no SHA-1 signatures

DH and RSA parameters minimum 3072



Norman Branitsky

Senior Cloud Architect

P: 416-916-1752



-Original Message-
From: Willy Tarreau 
Sent: Wednesday, May 6, 2020 2:00 PM
To: HAProxy 
Cc: eb...@haproxy.com; wlallem...@haproxy.com; remi.gaco...@powerdns.com
Subject: about Warning: Setting tune.ssl.default-dh-param to 1024



Hi all,



while running on a trivial test config in which I had enabled "zero-warning", 
my process refused to start due to the good old warning "Setting 
tune.ssl.default-dh-param to 1024 blah blah".



I was almost certain we discussed about switching the default value to 2048 for 
2.0 or 2.1 but couldn't find any trace of this, so I must have dreamed or 
discussed it in person. I've run a quick check on the configs shared on the 
list over the last two years and found this:



$ tail  -c80m Mail/lists/haproxy-ml | grep -o 'tune.ssl.default-dh-param[ 
]\+[0-9]\+' | awk '{print $1,$2}' |sort|uniq -c|sort -n

  1 tune.ssl.default-dh-param 4096

 13 tune.ssl.default-dh-param 1024

 86 tune.ssl.default-dh-param 2048



Thus it seems that the vast majority of users (exactly 86%) prefer to use 2048 
which is also the one recommended in the warning. All I found on the subject 
was in fact added to the doc by Rémi who implemented the tunable 6 years ago 
(commit f46cd6e4ec), and he warned:



   values greater than 1024 bits are not supported by Java 7 and

   earlier clients



Do we still really care given how old this is now and that users can still 
force the value if they absolutely need it ?



As such I think it's about time we change the default value to 2048 and get rid 
of this annoying warning before 2.2 gets released (and at the same time 86% of 
the users will be able to remove one cryptic line in their config). This way 
those who don't know/need it will be more secure by default and those who need 
it will still be able to.



Does anyone have any objection or alternate recommendation ?



Thanks,

Willy




about Warning: Setting tune.ssl.default-dh-param to 1024

2020-05-06 Thread Willy Tarreau
Hi all,

while running on a trivial test config in which I had enabled
"zero-warning", my process refused to start due to the good old
warning "Setting tune.ssl.default-dh-param to 1024 blah blah".

I was almost certain we discussed about switching the default value
to 2048 for 2.0 or 2.1 but couldn't find any trace of this, so I must
have dreamed or discussed it in person. I've run a quick check on the
configs shared on the list over the last two years and found this:

$ tail  -c80m Mail/lists/haproxy-ml | grep -o 'tune.ssl.default-dh-param[ 
]\+[0-9]\+' | awk '{print $1,$2}' |sort|uniq -c|sort -n
  1 tune.ssl.default-dh-param 4096
 13 tune.ssl.default-dh-param 1024
 86 tune.ssl.default-dh-param 2048

Thus it seems that the vast majority of users (exactly 86%) prefer to
use 2048 which is also the one recommended in the warning. All I found
on the subject was in fact added to the doc by Rémi who implemented the
tunable 6 years ago (commit f46cd6e4ec), and he warned:

   values greater than 1024 bits are not supported by Java 7 and
   earlier clients

Do we still really care given how old this is now and that users can
still force the value if they absolutely need it ?

As such I think it's about time we change the default value to 2048 and
get rid of this annoying warning before 2.2 gets released (and at the
same time 86% of the users will be able to remove one cryptic line in
their config). This way those who don't know/need it will be more
secure by default and those who need it will still be able to.

Does anyone have any objection or alternate recommendation ?

Thanks,
Willy



Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4

2020-05-06 Thread Olivier D
Hi again,

Le mer. 6 mai 2020 à 17:47, Willy Tarreau  a écrit :

> Hi Olivier,
>
> On Wed, May 06, 2020 at 05:29:59PM +0200, Olivier D wrote:
> > > Try applying this commit:
> > >
> > >
> https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94
> >
> >
> > So this patch is not working for me, with or without patching Apache2
> with
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
> >
> > But "good news" : reverting 7f26391bc51 did the trick.
>
> This is sad. So this means that we've trained external components to
> get used to our bugs and consider them to be the default behavior
> despite what the doc says.
>
> > To make sure we are talking about the same things, I've attached both
> > commits as patch files.
> > - applying 7f26391bc.patch did not fix the issue
> > - reverting 02c88036a.patch fixed the issue
>
> Then this is confusing because 7f2639 was already applied to your tree
> and is supposed to be the one causing you the issue, and 02c88 was not
> yet so you couldn't revert it. That's why I'd like to have a precise
> description of the starting state and what operations you did which
> worked and those which didn't.
>

My bad, I've inverted  7f26391bc and  02c88036a ... I should have used
named patches instead of dealing with commit number.
So to be clear :
I'm using 2.0.14 source code. In this version, patch 7f26391bc is already
applied and 02c88036a is not.
So applying 02c88036a did nothing (well, it triggers two different
non-working behaviour with Apache 2.4 patched, and a single behaviour
without the patch on remoteip).
And with clean source again, reverting 7f26391bc did the trick (on both
Apache 2.4 versions).



> > How safe is it to use 02c88036a reverted in production ?
>
> Either choice is safe for a given component. It's just that we send
> wrong information on health checks, forcing other implementations to
> implement bugs (see bug #511), that at least Dovecot doesn't support
> a "relaxed" approach combining LOCAL with an address, and that from
> what you're saying, Apache2 doesn't support LOCAL without an address.
>
> At this point I guess we'll have to revert all this from older branches
> and provide a config option for 2.2+ to re-enable the old behavior for
> compatibility with servers that got it wrong the first time.
>

Yes I understand. That's the price of fame :) HAProxy is now so widely used
that softwares are implemented based on what HAProxy does instead of what
the specs says.

Olivier


Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4

2020-05-06 Thread Willy Tarreau
Hi Olivier,

On Wed, May 06, 2020 at 05:29:59PM +0200, Olivier D wrote:
> > Try applying this commit:
> >
> > https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94
> 
> 
> So this patch is not working for me, with or without patching Apache2 with
> https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
> 
> But "good news" : reverting 7f26391bc51 did the trick.

This is sad. So this means that we've trained external components to
get used to our bugs and consider them to be the default behavior
despite what the doc says.

> To make sure we are talking about the same things, I've attached both
> commits as patch files.
> - applying 7f26391bc.patch did not fix the issue
> - reverting 02c88036a.patch fixed the issue

Then this is confusing because 7f2639 was already applied to your tree
and is supposed to be the one causing you the issue, and 02c88 was not
yet so you couldn't revert it. That's why I'd like to have a precise
description of the starting state and what operations you did which
worked and those which didn't.

> How safe is it to use  02c88036a reverted in production ?

Either choice is safe for a given component. It's just that we send
wrong information on health checks, forcing other implementations to
implement bugs (see bug #511), that at least Dovecot doesn't support
a "relaxed" approach combining LOCAL with an address, and that from
what you're saying, Apache2 doesn't support LOCAL without an address.

At this point I guess we'll have to revert all this from older branches
and provide a config option for 2.2+ to re-enable the old behavior for
compatibility with servers that got it wrong the first time.

Willy



Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4

2020-05-06 Thread Olivier D
Hello,

Le mer. 6 mai 2020 à 15:30, Tim Düsterhus  a écrit :

> Olivier,
>
> > I was not aware there were any change in the way HAProxy was doing its
> > checks over proxy-protocol in 2.0.14 ... any hint ?
>
> This sounds like this issue we've seen with Dovecot:
> https://www.mail-archive.com/haproxy@formilux.org/msg36890.html
>
> Try applying this commit:
>
> https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94


So this patch is not working for me, with or without patching Apache2 with
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893

But "good news" : reverting 7f26391bc51 did the trick.

To make sure we are talking about the same things, I've attached both
commits as patch files.
- applying 7f26391bc.patch did not fix the issue
- reverting 02c88036a.patch fixed the issue

How safe is it to use  02c88036a reverted in production ?

Olivier
--- src/connection.c
+++ src/connection.c
@@ -1247,6 +1247,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct 
server *srv, struct connec
/* At least one of src or dst is not of AF_INET or AF_INET6 */
if (  !src
   || !dst
+  || conn_is_back(remote)
   || (src->ss_family != AF_INET && src->ss_family != AF_INET6)
   || (dst->ss_family != AF_INET && dst->ss_family != AF_INET6)) {
if (buf_len < PP2_HDR_LEN_UNSPEC)
@@ -1256,14 +1257,7 @@ int make_proxy_line_v2(char *buf, int buf_len, struct 
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
-   /* Note: due to historic compatibility with V1 which required
-* to send "PROXY" with local addresses for local connections,
-* we can end up here with the remote in fact being our outgoing
-* connection. We still want to send real addresses and LOCAL on
-* it.
-*/
-   hdr->ver_cmd = PP2_VERSION;
-   hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL : 
PP2_CMD_PROXY;
+   hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)

--- src/connection.c
+++ src/connection.c
@@ -1318,11 +1318,18 @@ int make_proxy_line_v2(char *buf, int buf_len, struct 
server *srv, struct connec
ret = PP2_HDR_LEN_UNSPEC;
}
else {
+   /* Note: due to historic compatibility with V1 which required
+* to send "PROXY" with local addresses for local connections,
+* we can end up here with the remote in fact being our outgoing
+* connection. We still want to send real addresses and LOCAL on
+* it.
+*/
+   hdr->ver_cmd = PP2_VERSION;
+   hdr->ver_cmd |= conn_is_back(remote) ? PP2_CMD_LOCAL : 
PP2_CMD_PROXY;
/* IPv4 for both src and dst */
if (src->ss_family == AF_INET && dst->ss_family == AF_INET) {
if (buf_len < PP2_HDR_LEN_INET)
return 0;
-   hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET | PP2_TRANS_STREAM;
hdr->addr.ip4.src_addr = ((struct sockaddr_in 
*)src)->sin_addr.s_addr;
hdr->addr.ip4.src_port = ((struct sockaddr_in 
*)src)->sin_port;
@@ -1336,7 +1343,6 @@ int make_proxy_line_v2(char *buf, int buf_len, struct 
server *srv, struct connec

if (buf_len < PP2_HDR_LEN_INET6)
return 0;
-   hdr->ver_cmd = PP2_VERSION | PP2_CMD_PROXY;
hdr->fam = PP2_FAM_INET6 | PP2_TRANS_STREAM;
if (src->ss_family == AF_INET) {
v4tov6(, &((struct sockaddr_in 
*)src)->sin_addr);


Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3

2020-05-06 Thread Gilchrist DADAGLO
Hi,
It was a manual test.
I used the default example and adapted it with the parameters in "Test
settings".
"Test result" is just a copy from the logs to stdio.

I was just to make sure we exchange all data types supported by the
protocol.

This could be automated indeed but I didn't do anything for that purpose.

Gilchrist

On Wed, May 6, 2020, 16:13 Илья Шипицин  wrote:

> How did you get "test result"?
> Should we add automated test for that? For example, once a week
>
> On Wed, May 6, 2020, 5:28 PM Gilchrist Dadaglo  wrote:
>
>>
>> Background:
>> Python 2 is no longer supported since January, 1st 2020 as per
>> https://www.python.org/doc/sunset-python-2/
>> The purpose of this change is to make the spoa_server contrib
>> library
>> compatible with Python 3 to allow transition to Python 3.
>>
>> Test Settings:
>> ps_python.py:
>> ...
>> spoa.set_var_null("null", spoa.scope_txn)
>> spoa.set_var_boolean("boolean", spoa.scope_txn, True)
>> spoa.set_var_int32("int32", spoa.scope_txn, 1234)
>> spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
>> spoa.set_var_int64("int64", spoa.scope_txn, 1234)
>> spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
>> spoa.set_var_ipv4("ipv4", spoa.scope_txn,
>> ipaddress.IPv4Address(u"127.0.0.1"))
>> spoa.set_var_ipv6("ipv6", spoa.scope_txn,
>> ipaddress.IPv6Address(u"1::f"))
>> spoa.set_var_str("str", spoa.scope_txn, "1::f")
>> spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x42f\x63\x63")
>> spoa.set_var_str("python_version", spoa.scope_sess,
>> str(sys.version_info))
>> ...
>> haproxy.cfg:
>> ...
>> http-request capture var(txn.verb.null),debug len 1
>> http-request capture var(txn.verb.boolean),debug len 1
>> http-request capture var(txn.verb.int32),debug len 4
>> http-request capture var(txn.verb.uint32),debug len 4
>> http-request capture var(txn.verb.int64),debug len 4
>> http-request capture var(txn.verb.uint64),debug len 4
>> http-request capture var(txn.verb.ipv4),debug len 16
>> http-request capture var(txn.verb.ipv6),debug len 45
>> http-request capture var(txn.verb.str),debug len 32
>> http-request capture var(txn.verb.bin),debug len 32
>> http-request capture var(sess.verb.python_version),debug len 100
>> ...
>>
>> Test result:
>> Python 3.8:
>> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=8, micro=1, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>> Python 3.7:
>> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=7, micro=6, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>> Python 3.6:
>> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
>> minor=6, micro=10, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>> Python 2.7:
>> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
>> 1/1/0/0/0 0/0
>> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=2,
>> minor=7, micro=17, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>>
>> Not tested:
>> Python <2.7
>> ---
>>  haproxy/contrib/spoa_server/Makefile|  37 +
>>  haproxy/contrib/spoa_server/README  |  10 +-
>>  haproxy/contrib/spoa_server/ps_python.c | 179 +++-
>>  haproxy/contrib/spoa_server/ps_python.h |  52 +++
>>  4 files changed, 241 insertions(+), 37 deletions(-)
>>  create mode 100644 haproxy/contrib/spoa_server/ps_python.h
>>
>>


Re: [PATCH] MAJOR: contrib: porting spoa_server to support python3

2020-05-06 Thread Илья Шипицин
How did you get "test result"?
Should we add automated test for that? For example, once a week

On Wed, May 6, 2020, 5:28 PM Gilchrist Dadaglo  wrote:

>
> Background:
> Python 2 is no longer supported since January, 1st 2020 as per
> https://www.python.org/doc/sunset-python-2/
> The purpose of this change is to make the spoa_server contrib
> library
> compatible with Python 3 to allow transition to Python 3.
>
> Test Settings:
> ps_python.py:
> ...
> spoa.set_var_null("null", spoa.scope_txn)
> spoa.set_var_boolean("boolean", spoa.scope_txn, True)
> spoa.set_var_int32("int32", spoa.scope_txn, 1234)
> spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
> spoa.set_var_int64("int64", spoa.scope_txn, 1234)
> spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
> spoa.set_var_ipv4("ipv4", spoa.scope_txn,
> ipaddress.IPv4Address(u"127.0.0.1"))
> spoa.set_var_ipv6("ipv6", spoa.scope_txn,
> ipaddress.IPv6Address(u"1::f"))
> spoa.set_var_str("str", spoa.scope_txn, "1::f")
> spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x42f\x63\x63")
> spoa.set_var_str("python_version", spoa.scope_sess,
> str(sys.version_info))
> ...
> haproxy.cfg:
> ...
> http-request capture var(txn.verb.null),debug len 1
> http-request capture var(txn.verb.boolean),debug len 1
> http-request capture var(txn.verb.int32),debug len 4
> http-request capture var(txn.verb.uint32),debug len 4
> http-request capture var(txn.verb.int64),debug len 4
> http-request capture var(txn.verb.uint64),debug len 4
> http-request capture var(txn.verb.ipv4),debug len 16
> http-request capture var(txn.verb.ipv6),debug len 45
> http-request capture var(txn.verb.str),debug len 32
> http-request capture var(txn.verb.bin),debug len 32
> http-request capture var(sess.verb.python_version),debug len 100
> ...
>
> Test result:
> Python 3.8:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=8, micro=1, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 3.7:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=7, micro=6, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 3.6:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
> minor=6, micro=10, releaselevel='final', serial=0)} "POST / HTTP/1.1"
> Python 2.7:
> ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR--
> 1/1/0/0/0 0/0
> {|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=2,
> minor=7, micro=17, releaselevel='final', serial=0)} "POST / HTTP/1.1"
>
> Not tested:
> Python <2.7
> ---
>  haproxy/contrib/spoa_server/Makefile|  37 +
>  haproxy/contrib/spoa_server/README  |  10 +-
>  haproxy/contrib/spoa_server/ps_python.c | 179 +++-
>  haproxy/contrib/spoa_server/ps_python.h |  52 +++
>  4 files changed, 241 insertions(+), 37 deletions(-)
>  create mode 100644 haproxy/contrib/spoa_server/ps_python.h
>
>


Re: [PATCH] fix errored ARM64 builds in travis-ci

2020-05-06 Thread Илья Шипицин
btw, I'm going to enable arm64 builds back soon :)


but your efforts are awesome.

ср, 6 мая 2020 г. в 17:18, Martin Grigorov :

> Hi,
>
> I've just created a PR (https://github.com/haproxy/haproxy/pull/617/files)
> that introduces testing on ARM64/AARCH64 at GitHub Actions.
> It almost works! There are few tests that fail. Any help finding the
> reason is very welcome!
>
> Martin
>
> On Mon, Mar 23, 2020 at 11:12 AM Martin Grigorov 
> wrote:
>
>> Hi Илья,
>>
>> On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин 
>> wrote:
>>
>>> Martin,
>>>
>>> as the one of the most interested in ARM64 builds, I've got news for you
>>>
>>>
>>> can you try
>>>
>>> travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' ||
>>> (cat build-ssl.log && exit 1)
>>>
>>> in travis ? (please not "travis_wait 30" instead of "travis_wait")
>>>
>>
>> it is running at the moment here:
>> https://travis-ci.org/github/martin-g/haproxy/builds/665770469
>>
>>
>>>
>>>
>>> also, it might be important to clear travis cache from time to time.
>>> as for myself, "travis_wait 30" helped me to resolve similar issue on
>>> another project (in my own fork haproxy on arm64 builds just fine)
>>>
>>> ср, 18 мар. 2020 г. в 23:35, Илья Шипицин :
>>>
 well, there are several topics on travis-ci forum related to "output on
 ARM64 got truncated in the mid of ..."
 Let us disable ARM64 travis-ci builds for few months.

 Martin, I'll play with hosted github runner in order to find a way how
 we can limit its builds to allowed only.

 ср, 18 мар. 2020 г. в 18:57, Martin Grigorov :

>
> Current master's build passed the problematic point in my TravisCI
> project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359
> Note: I use TravisCI .org while HAProxy's official project is at .com:
> https://travis-ci.com/github/haproxy/haproxy
> I also think this is a problem on TravisCI's end.
>
> Martin
>
> On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин 
> wrote:
>
>> I will disable PR builds.
>>
>> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau  wrote:
>>
>>> On Wed, Mar 18, 2020 at 06:21:15PM +0500,  ??? wrote:
>>> > let us calm down a bit :)
>>>
>>> Agreed, especially since the build on PRs already happens and already
>>> adds noise.
>>>
>>> > yes, I still believe it is because of buffering. I might have
>>> missed
>>> > something.
>>> > unless I will repair it, I'll drop arm64 support on travis (and we
>>> will
>>> > switch to self hosted github action runner)
>>>
>>> OK.
>>>
>>> Willy
>>>
>>


Re: Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4

2020-05-06 Thread Tim Düsterhus
Olivier,

Am 06.05.20 um 15:15 schrieb Olivier D:
> This morning I tried to upgrade HAProxy 2.0.13 to 2.0.14 but had to
> rollback immediately : some backends checks started to fail.
> Error reported was : SOCKERR - SSL handshake failure
> 
> The backends failing have a specific configuration as follows (I removed
> anything unnecessary to trigger the issue)
> listen webtruc:443
> mode tcp
> bind X.X.X.X:443
> server xxx X.X.X.X:443 check weight 5 send-proxy-v2-ssl-cn check-ssl
> verify none
> 
> Backend is an Apache 2.4 with "RemoteIPProxyProtocol On".
> In apache logs I have :
> [remoteip:error] [pid 1067 [client :26847] AH03507:
> RemoteIPProxyProtocol: unsupported command 20
> 
> I can link this error to this bugreport :
> https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
> So I applied this patch to Apache 2.4 and then get this error :
> HAproxy side: L7STS Bad request
> Apache side : RemoteIPProxyProtocol data is missing, but required! Aborting
> request.
> 
> I was not aware there were any change in the way HAProxy was doing its
> checks over proxy-protocol in 2.0.14 ... any hint ?

This sounds like this issue we've seen with Dovecot:
https://www.mail-archive.com/haproxy@formilux.org/msg36890.html

Try applying this commit:
https://github.com/haproxy/haproxy/commit/02c88036a61e09d0676a2b6b4086af677b023b94

Best regards
Tim Düsterhus



Version 2.0.14 breaking change vs 2.0.13 with send-proxy-v2-ssl-cn + Apache 2.4

2020-05-06 Thread Olivier D
Hello,

This morning I tried to upgrade HAProxy 2.0.13 to 2.0.14 but had to
rollback immediately : some backends checks started to fail.
Error reported was : SOCKERR - SSL handshake failure

The backends failing have a specific configuration as follows (I removed
anything unnecessary to trigger the issue)
listen webtruc:443
mode tcp
bind X.X.X.X:443
server xxx X.X.X.X:443 check weight 5 send-proxy-v2-ssl-cn check-ssl
verify none

Backend is an Apache 2.4 with "RemoteIPProxyProtocol On".
In apache logs I have :
[remoteip:error] [pid 1067 [client :26847] AH03507:
RemoteIPProxyProtocol: unsupported command 20

I can link this error to this bugreport :
https://bz.apache.org/bugzilla/show_bug.cgi?id=63893
So I applied this patch to Apache 2.4 and then get this error :
HAproxy side: L7STS Bad request
Apache side : RemoteIPProxyProtocol data is missing, but required! Aborting
request.

I was not aware there were any change in the way HAProxy was doing its
checks over proxy-protocol in 2.0.14 ... any hint ?


HAProxy -vv output :
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
-fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered
-Wno-missing-field-initializers -Wno-implicit-fallthrough
-Wno-stringop-overflow -Wtype-limits -Wshift-negative-value
-Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_THREAD=0 USE_STATIC_PCRE=1 USE_OPENSSL=1 USE_LUA=1
USE_ZLIB=1 USE_NS=

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE
-PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED
-REGPARM +STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE
+LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4
-MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO -NS +DL +RT -DEVICEATLAS
-51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=20).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTXside=FE|BE mux=H2
  h2 : mode=HTTP   side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP   side=FE|BE mux=PASS

Available services : none

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace


[PATCH] MAJOR: contrib: porting spoa_server to support python3

2020-05-06 Thread Gilchrist Dadaglo

Background:
Python 2 is no longer supported since January, 1st 2020 as per 
https://www.python.org/doc/sunset-python-2/
The purpose of this change is to make the spoa_server contrib library
compatible with Python 3 to allow transition to Python 3.

Test Settings:
ps_python.py:
...
spoa.set_var_null("null", spoa.scope_txn)
spoa.set_var_boolean("boolean", spoa.scope_txn, True)
spoa.set_var_int32("int32", spoa.scope_txn, 1234)
spoa.set_var_uint32("uint32", spoa.scope_txn, 1234)
spoa.set_var_int64("int64", spoa.scope_txn, 1234)
spoa.set_var_uint64("uint64", spoa.scope_txn, 1234)
spoa.set_var_ipv4("ipv4", spoa.scope_txn, 
ipaddress.IPv4Address(u"127.0.0.1"))
spoa.set_var_ipv6("ipv6", spoa.scope_txn, 
ipaddress.IPv6Address(u"1::f"))
spoa.set_var_str("str", spoa.scope_txn, "1::f")
spoa.set_var_bin("bin", spoa.scope_txn, "1:\x01:\x42f\x63\x63")
spoa.set_var_str("python_version", spoa.scope_sess, 
str(sys.version_info))
...
haproxy.cfg:
...
http-request capture var(txn.verb.null),debug len 1
http-request capture var(txn.verb.boolean),debug len 1
http-request capture var(txn.verb.int32),debug len 4
http-request capture var(txn.verb.uint32),debug len 4
http-request capture var(txn.verb.int64),debug len 4
http-request capture var(txn.verb.uint64),debug len 4
http-request capture var(txn.verb.ipv4),debug len 16
http-request capture var(txn.verb.ipv6),debug len 45
http-request capture var(txn.verb.str),debug len 32
http-request capture var(txn.verb.bin),debug len 32
http-request capture var(sess.verb.python_version),debug len 100
...

Test result:
Python 3.8:
ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR-- 1/1/0/0/0 
0/0 
{|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
 minor=8, micro=1, releaselevel='final', serial=0)} "POST / HTTP/1.1"
Python 3.7:
ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR-- 1/1/0/0/0 
0/0 
{|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
 minor=7, micro=6, releaselevel='final', serial=0)} "POST / HTTP/1.1"
Python 3.6:
ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR-- 1/1/0/0/0 
0/0 
{|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=3,
 minor=6, micro=10, releaselevel='final', serial=0)} "POST / HTTP/1.1"
Python 2.7:
ft_public ft_public/ 0/-1/-1/-1/0 403 212 - - PR-- 1/1/0/0/0 
0/0 
{|1|1234|1234|1234|1234|127.0.0.1|1::f|1::f|1:#01:Bfcc|sys.version_info(major=2,
 minor=7, micro=17, releaselevel='final', serial=0)} "POST / HTTP/1.1"

Not tested:
Python <2.7
---
 haproxy/contrib/spoa_server/Makefile|  37 +
 haproxy/contrib/spoa_server/README  |  10 +-
 haproxy/contrib/spoa_server/ps_python.c | 179 +++-
 haproxy/contrib/spoa_server/ps_python.h |  52 +++
 4 files changed, 241 insertions(+), 37 deletions(-)
 create mode 100644 haproxy/contrib/spoa_server/ps_python.h

diff --git a/haproxy/contrib/spoa_server/Makefile b/haproxy/contrib/spoa_server/Makefile
index f075282..e7b20db 100644
--- a/haproxy/contrib/spoa_server/Makefile
+++ b/haproxy/contrib/spoa_server/Makefile
@@ -23,10 +23,47 @@ endif
 
 ifneq ($(USE_PYTHON),)
 OBJS += ps_python.o
+
+# "--embed" flag is supported (and required) only from python 3.8+
+check_python_config := $(shell if python3-config --embed; then echo "python3.8+"; \
+elif hash python3-config; then echo "python3"; \
+elif hash python-config; then echo "python2"; fi)
+
+ifeq ($(check_python_config), python3.8+)
+PYTHON_DEFAULT_INC := $(shell python3-config --includes)
+PYTHON_DEFAULT_LIB := $(shell python3-config --libs --embed)
+else ifeq ($(check_python_config), python3)
+PYTHON_DEFAULT_INC := $(shell python3-config --includes)
+PYTHON_DEFAULT_LIB := $(shell python3-config --libs)
+else ifeq ($(check_python_config), python2)
+PYTHON_DEFAULT_INC := $(shell python-config --includes)
+PYTHON_DEFAULT_LIB := $(shell python-config --libs)
+endif
+
+
+# Add default path
+ifneq ($(PYTHON_DEFAULT_INC),)
+CFLAGS += $(PYTHON_DEFAULT_INC)
+else
 CFLAGS += -I/usr/include/python2.7
+endif
+ifneq ($(PYTHON_DEFAULT_LIB),)
+LDLIBS += $(PYTHON_DEFAULT_LIB)
+else
 LDLIBS += -lpython2.7
 endif
 
+# Add user additional paths if any
+ifneq ($(PYTHON_INC),)
+CFLAGS += -I$(PYTHON_INC)
+endif
+ifneq ($(PYTHON_LIB),)
+LDLIBS += -L$(PYTHON_LIB)
+endif
+
+LDLIBS +=-Wl,--export-dynamic
+endif
+
 spoa: $(OBJS)
 	$(LD) $(LDFLAGS) -o $@ $^ $(LDLIBS)
 
diff --git a/haproxy/contrib/spoa_server/README b/haproxy/contrib/spoa_server/README
index 341f5f9..f654a23 100644
--- a/haproxy/contrib/spoa_server/README
+++ b/haproxy/contrib/spoa_server/README
@@ -14,9 +14,10 @@ is done.
 You have to install the development packages, either from the
 

[PR] Build and test on AARCH64 at GitHub actions

2020-05-06 Thread PR Bot
Dear list!

Author: Martin Tzvetanov Grigorov 
Number of patches: 21

This is an automated relay of the Github pull request:
   Build and test on AARCH64 at GitHub actions

Patch title(s): 
   Add ARM64 testing
   Fix YAML
   Do not use '-it' for 'docker run'
   Create a custom Dockerfile where the build & test will be executed
   Checkout the project
   Checkout VTest and install dependencies in Docker
   Use Ubuntu:20.04 instead of Centos:8
   Extract the Shell script into a file
   Export CC for `make vtest`
   Add dependency to zlib1g-dev
   Add gcc to general dependencies
   Install wget to be able to download OpenSSL
   Disable LeakSanitizer because it fails with:
   Do not build OpenSSL. It is too slow with QEMU
   Remove USE_OPENSSL=0
   Remove testing of WURFL
   Re-enable OpenSSL because without it some tests fail
   Rename arm64.yml to aarch64.yml for consistency
   Use checkout@v2
   Add a README explaining how the testing on AARCH64 works
   Re-enable testing for Windows

Link:
   https://github.com/haproxy/haproxy/pull/617

Edit locally:
   wget https://github.com/haproxy/haproxy/pull/617.patch && vi 617.patch

Apply locally:
   curl https://github.com/haproxy/haproxy/pull/617.patch | git am -

Description:
   Since testing on ARM64/AARCH64 has been [disabled](https://github.com/
   haproxy/haproxy/commit/18b303e9f9b38e3a35b05cfe54d57ce3c81599c0#diff-3
   54f30a63fb0907d4ad57269548329e3) at TravisCI because there were some
   infrastructure related problems I propose a new way to test at GitHub
   Actions.
   The new approach is based on QEMU and uses Docker images
   to simplify the setup of QEMU.
   
   All is nice and shiny but
   there is one big problem - few tests fail consistently:
   
   ```
   ## Starting vtest ##
   Testing with haproxy version: 2.2-dev0-bbd4d68-993
   #top  TEST
   reg-tests/ssl/wrong_ctx_storage.vtc FAILED (0.280) exit=2
   #top
   TEST reg-tests/connection/proxy_protocol_random_fail.vtc FAILED
   (0.306) exit=2
   #top  TEST reg-tests/seamless-
   reload/abns_socket.vtc FAILED (0.711) exit=2
   #top  TEST reg-
   tests/http-rules/map_regm_with_backref.vtc FAILED (0.642) exit=2
   #
   top  TEST reg-tests/http-
   rules/converters_ipmask_concat_strcmp_field_word.vtc FAILED (0.778)
   exit=2
   5 tests failed, 0 tests skipped, 48 tests passed
   ## Gathering results
   ##
   ## Test case: reg-
   tests/connection/proxy_protocol_random_fail.vtc ##
   ## test
   results in:
   "/tmp/haregtests-2020-05-06_11-59-19.fNnEdV/vtc.7747.3fdec118"
    top  shell_exit not as expected: got 0x007f wanted 0x
   ## Test case: reg-tests/http-
   rules/converters_ipmask_concat_strcmp_field_word.vtc ##
   ##
   test results in:
   "/tmp/haregtests-2020-05-06_11-59-19.fNnEdV/vtc.7747.4f6aaeeb"
    c1   HTTP header is incomplete
   ## Test case: reg-
   tests/http-rules/map_regm_with_backref.vtc ##
   ## test results
   in: "/tmp/haregtests-2020-05-06_11-59-19.fNnEdV/vtc.7747.2a6adc02"
    c1   HTTP header is incomplete
    s1   HTTP rx failed (fd:6
   read: Connection reset by peer)
   ## Test case: reg-
   tests/ssl/wrong_ctx_storage.vtc ##
   ## test results in:
   "/tmp/haregtests-2020-05-06_11-59-19.fNnEdV/vtc.7747.6fe1dd3f"
    top  shell_exit not as expected: got 0x007f wanted 0x
   ## Test case: reg-tests/seamless-reload/abns_socket.vtc ##
   ## test results in:
   "/tmp/haregtests-2020-05-06_11-59-19.fNnEdV/vtc.7747.09b9aabe"
    c1   Failed to open 127.0.0.1 39405: (null)
   make: ***
   [Makefile:995: reg-tests] Error 1
   ```
   
   I've explained how
   to run the setup locally in `.github/workflows/aarch64/README.md`.

Instructions:
   This github pull request will be closed automatically; patch should be
   reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is
   invited to comment, even the patch's author. Please keep the author and
   list CCed in replies. Please note that in absence of any response this
   pull request will be lost.



Re: [PATCH] fix errored ARM64 builds in travis-ci

2020-05-06 Thread Martin Grigorov
Hi,

I've just created a PR (https://github.com/haproxy/haproxy/pull/617/files)
that introduces testing on ARM64/AARCH64 at GitHub Actions.
It almost works! There are few tests that fail. Any help finding the reason
is very welcome!

Martin

On Mon, Mar 23, 2020 at 11:12 AM Martin Grigorov 
wrote:

> Hi Илья,
>
> On Sun, Mar 22, 2020 at 2:46 PM Илья Шипицин  wrote:
>
>> Martin,
>>
>> as the one of the most interested in ARM64 builds, I've got news for you
>>
>>
>> can you try
>>
>> travis_wait 30 bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat
>> build-ssl.log && exit 1)
>>
>> in travis ? (please not "travis_wait 30" instead of "travis_wait")
>>
>
> it is running at the moment here:
> https://travis-ci.org/github/martin-g/haproxy/builds/665770469
>
>
>>
>>
>> also, it might be important to clear travis cache from time to time.
>> as for myself, "travis_wait 30" helped me to resolve similar issue on
>> another project (in my own fork haproxy on arm64 builds just fine)
>>
>> ср, 18 мар. 2020 г. в 23:35, Илья Шипицин :
>>
>>> well, there are several topics on travis-ci forum related to "output on
>>> ARM64 got truncated in the mid of ..."
>>> Let us disable ARM64 travis-ci builds for few months.
>>>
>>> Martin, I'll play with hosted github runner in order to find a way how
>>> we can limit its builds to allowed only.
>>>
>>> ср, 18 мар. 2020 г. в 18:57, Martin Grigorov :
>>>

 Current master's build passed the problematic point in my TravisCI
 project: https://travis-ci.org/github/martin-g/haproxy/jobs/663953359
 Note: I use TravisCI .org while HAProxy's official project is at .com:
 https://travis-ci.com/github/haproxy/haproxy
 I also think this is a problem on TravisCI's end.

 Martin

 On Wed, Mar 18, 2020 at 3:43 PM Илья Шипицин 
 wrote:

> I will disable PR builds.
>
> On Wed, Mar 18, 2020, 6:27 PM Willy Tarreau  wrote:
>
>> On Wed, Mar 18, 2020 at 06:21:15PM +0500,  ??? wrote:
>> > let us calm down a bit :)
>>
>> Agreed, especially since the build on PRs already happens and already
>> adds noise.
>>
>> > yes, I still believe it is because of buffering. I might have missed
>> > something.
>> > unless I will repair it, I'll drop arm64 support on travis (and we
>> will
>> > switch to self hosted github action runner)
>>
>> OK.
>>
>> Willy
>>
>


[PATCH] MINOR: server: adds slowstart parameter

2020-05-06 Thread Scheglmann, Stefan
Builds new models with added slowstart server parameter.

>From 7638bf05e36f3ae890e4d023acbfeb60bee4b802 Mon Sep 17 00:00:00 2001
From: Stefan Scheglmann 
Date: Thu, 30 Apr 2020 13:05:49 +0200
Subject: [PATCH] MINOR: server: adds slowstart parameter

Newly generated models, adding support for slowstart paramter in
servers. Generated from updated haproxy-spec.yml from updated
dataplaneapi-specification.
---
 server.go | 21 +
 1 file changed, 21 insertions(+)

diff --git a/server.go b/server.go
index ae99d90..5fbd5c7 100644
--- a/server.go
+++ b/server.go
@@ -153,6 +153,10 @@ type Server struct {
// Enum: [enabled disabled]
SendProxyV2 string `json:"send-proxy-v2,omitempty"`

+   // slowstart
+   // Pattern: ^[1-9][0-9]*(us|ms|s|m|h|d)$
+   Slowstart string `json:"slowstart,omitempty"`
+
// sni
// Pattern: ^[^\s]+$
Sni string `json:"sni,omitempty"`
@@ -277,6 +281,10 @@ func (m *Server) Validate(formats strfmt.Registry) error {
res = append(res, err)
}

+   if err := m.validateSlowstart(formats); err != nil {
+   res = append(res, err)
+   }
+
if err := m.validateSni(formats); err != nil {
res = append(res, err)
}
@@ -918,6 +926,19 @@ func (m *Server) validateSendProxyV2(formats 
strfmt.Registry) error {
return nil
 }

+func (m *Server) validateSlowstart(formats strfmt.Registry) error {
+
+   if swag.IsZero(m.Slowstart) { // not required
+   return nil
+   }
+
+   if err := validate.Pattern("slowstart", "body", string(m.Slowstart), 
`^[1-9][0-9]*(us|ms|s|m|h|d)$`); err != nil {
+   return err
+   }
+
+   return nil
+}
+
 func (m *Server) validateSni(formats strfmt.Registry) error {

if swag.IsZero(m.Sni) { // not required
--
2.20.1 (Apple Git-117)

--

Kind regards,

Stefan Scheglmann
PAAS Developer

E-mail   scheglm...@strato.de
Website  www.strato.com

STRATO AG | Pascalstraße 10 | 10587 Berlin | Germany

The mandatory information can be found here 
https://www.strato-hosting.co.uk/imprint/



[PATCH] MINOR: server: adds slowstart parameter

2020-05-06 Thread Scheglmann, Stefan

Adds slowstart parameter to dataplaneapi-specification. These patches add 
slowstart to dataplaneapi-specification. 
Sry for duplicate, this replaces previous mail with one mail per patch and 
patch included.

>From a02f7bef4ae6624eeaa916294e896c7af73b451a Mon Sep 17 00:00:00 2001
From: Stefan Scheglmann 
Date: Thu, 30 Apr 2020 12:51:42 +0200
Subject: [PATCH] MINOR: servers: adds slowstart parameter

---
 build/haproxy_spec.yaml   | 3 +++
 models/configuration.yaml | 3 +++
 2 files changed, 6 insertions(+)

diff --git a/build/haproxy_spec.yaml b/build/haproxy_spec.yaml
index ec84626..b5d2c8d 100644
--- a/build/haproxy_spec.yaml
+++ b/build/haproxy_spec.yaml
@@ -1050,6 +1050,9 @@ definitions:
   - enabled
   - disabled
   type: string
+slowstart:
+  pattern: ^[1-9][0-9]*(us|ms|s|m|h|d)$
+  type: string
 sni:
   pattern: ^[^\s]+$
   type: string
diff --git a/models/configuration.yaml b/models/configuration.yaml
index 9e6c238..3fbaf8e 100644
--- a/models/configuration.yaml
+++ b/models/configuration.yaml
@@ -798,6 +798,9 @@ server:
 send-proxy-v2:
   type: string
   enum: [enabled, disabled]
+slowstart:
+  type: string
+  pattern: '^[1-9][0-9]*(us|ms|s|m|h|d)$'
 sni:
   type: string
   pattern: '^[^\s]+$'
--
2.20.1 (Apple Git-117)

--

Kind regards,

Stefan Scheglmann
PAAS Developer

E-mail   scheglm...@strato.de
Website  www.strato.com

STRATO AG | Pascalstraße 10 | 10587 Berlin | Germany

The mandatory information can be found here 
https://www.strato-hosting.co.uk/imprint/



[PATCH] MINOR: server: adds slowstart parameter

2020-05-06 Thread Scheglmann, Stefan
Add optional slowstart parameter to dataplaneapi-specification and build new 
models from it. There two followup patches pending, one on client-native and 
one on dataplaneapi, waiting for those to be merged.


--

Kind regards,

Stefan Scheglmann
SWH Dev

Phone  + 49 30 88615 3358
Fax  + 49 30 88615 3358
E-Mail scheglm...@strato.de
Website  www.strato.com

STRATO AG  |  Pascalstraße 10  |  10587 Berlin  | Germany

Supervisory Board: René Obermann (Chairman)
Management Board: Dr. Christian Böing (Chairman),
Christoph Steffens, René Wienholtz
Commercial register: Municipal court Berlin-Charlottenburg HRB 79450


0001-MINOR-server-adds-slowstart-parameter_models.patch
Description: 0001-MINOR-server-adds-slowstart-parameter_models.patch


0001-MINOR-servers-adds-slowstart-parameter_specification.patch
Description: 0001-MINOR-servers-adds-slowstart-parameter_specification.patch


Understanding rate-limit sessions

2020-05-06 Thread Olivier D
Hello,

I was creating counter-measures against a DOS attack, but I failed to
understand some numbers I received.
I'm using HAProxy 2.0.14

My (expurged) frontend config is :

listen test
bind X.X.X.X:443
maxconn 65536
rate-limit sessions 128

But during the attack, the following numbers are reported on HAProxy status
page :
FRONTEND:
Session rate: max=1821 ; limit=128
Putting cursor on "max" shows :
max connection rate: 2368/s
max session rate: 1821/s
max request rate: 3901/s

I wondered why session-rate exceeded my maximal number of 128 I set on
config file. I'm probably not understanding something correctly here. The
documentation seems quite clear : "Since the session rate is measured every
millisecond, it is extremely accurate"

Any clue ?

Unfortunately I was not quick enough to record the traffic received, or
dump internal HAProxy counters when this happens :(

Thank you !

Olivier


running haproxy with predefined security policies on RHEL8 ?

2020-05-06 Thread Илья Шипицин
Hello,

do we have any experience of
https://www.redhat.com/en/blog/consistent-security-crypto-policies-red-hat-enterprise-linux-8
?

Cheers,
Ilya Shipitcin


Re: Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Илья Шипицин
thank you, I will have a look!

ср, 6 мая 2020 г. в 14:27, Martin Grigorov :

> Hi Илья,
>
> On Wed, May 6, 2020 at 11:59 AM Илья Шипицин  wrote:
>
>> do you run tests on GH arm64 agents ? is it dedicated (your own) agents
>> attached to your repo ? can you give a link ?
>>
>
> I use Docker + QEMU with GH hosted runner.
> You can see the current diff at
> https://github.com/haproxy/haproxy/compare/master...martin-g:feature/test-aarch64-on-github-actions
> 'windows-latest.yml` is temporary disabled until I'm ready with my
> experiments.
>
> As explained at
> https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories
>  using
> self-hosted runner is not secure if they are used for Pull Requests.
> They can be used for pushes to `master` branch though since this is
> supposed to be a reviewed code!
>
> But the tests failures I reported below fail on my local machine (x86_64)
> and my ARM64 VM. I do not use Docker and/or QEMU here.
>
> Martin
>
>
>> ср, 6 мая 2020 г. в 13:22, Martin Grigorov :
>>
>>> Hello HAProxy team,
>>>
>>> While working on a PR to build & test HAProxy on AARCH64 at GitHub
>>> Actions I've noticed a strange behavior for some of the tests.
>>>
>>> To reduce the time of the build I've removed USE_OPENSSL=1 from the
>>> FLAGS [1] passed to "make".
>>> The build passed successfully, some of the tests are skipped because
>>> they depend on SSL library, e.g.:
>>>
>>> ...
>>> Add test: reg-tests/lua/txn_get_priv.vtc
>>>   Add test: reg-tests/lua/h_txn_get_priv.vtc
>>>   Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not
>>> compiled with the required option OPENSSL
>>>   Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
>>> with the required option OPENSSL
>>>   Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not
>>> compiled with the required option OPENSSL
>>>   Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled
>>> with the required option OPENSSL
>>> ...
>>>
>>> but few tests just fail:
>>>
>>> Testing with haproxy version: 2.2-dev7
>>> #top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
>>> #top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill
>>> -9)
>>> #top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
>>> signal=9
>>> 2 tests failed, 0 tests skipped, 64 tests passed
>>> ## Gathering results ##
>>> ## Test case: reg-tests/compression/lua_validation.vtc ##
>>> ## test results in:
>>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
>>> ## Test case: reg-tests/lua/txn_get_priv.vtc ##
>>> ## test results in:
>>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
>>>  c0   HTTP rx timeout (fd:6 5000 ms)
>>>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
>>> Makefile:995: recipe for target 'reg-tests' failed
>>> make: *** [reg-tests] Error 1
>>>
>>> These tests fail both on AARCH64 and x86_64 consistently until I add
>>> back USE_OPENSSL=1
>>> to the flags.
>>>
>>> Is there an issue with these tests ?
>>>
>>> 1.
>>> https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14
>>>
>>> Regards,
>>> Martin
>>>
>>


Re: Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Martin Grigorov
Hi Илья,

On Wed, May 6, 2020 at 11:59 AM Илья Шипицин  wrote:

> do you run tests on GH arm64 agents ? is it dedicated (your own) agents
> attached to your repo ? can you give a link ?
>

I use Docker + QEMU with GH hosted runner.
You can see the current diff at
https://github.com/haproxy/haproxy/compare/master...martin-g:feature/test-aarch64-on-github-actions
'windows-latest.yml` is temporary disabled until I'm ready with my
experiments.

As explained at
https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories
using
self-hosted runner is not secure if they are used for Pull Requests.
They can be used for pushes to `master` branch though since this is
supposed to be a reviewed code!

But the tests failures I reported below fail on my local machine (x86_64)
and my ARM64 VM. I do not use Docker and/or QEMU here.

Martin


> ср, 6 мая 2020 г. в 13:22, Martin Grigorov :
>
>> Hello HAProxy team,
>>
>> While working on a PR to build & test HAProxy on AARCH64 at GitHub
>> Actions I've noticed a strange behavior for some of the tests.
>>
>> To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS
>> [1] passed to "make".
>> The build passed successfully, some of the tests are skipped because they
>> depend on SSL library, e.g.:
>>
>> ...
>> Add test: reg-tests/lua/txn_get_priv.vtc
>>   Add test: reg-tests/lua/h_txn_get_priv.vtc
>>   Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not
>> compiled with the required option OPENSSL
>>   Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
>> with the required option OPENSSL
>>   Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled
>> with the required option OPENSSL
>>   Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled
>> with the required option OPENSSL
>> ...
>>
>> but few tests just fail:
>>
>> Testing with haproxy version: 2.2-dev7
>> #top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
>> #top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill
>> -9)
>> #top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
>> signal=9
>> 2 tests failed, 0 tests skipped, 64 tests passed
>> ## Gathering results ##
>> ## Test case: reg-tests/compression/lua_validation.vtc ##
>> ## test results in:
>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
>> ## Test case: reg-tests/lua/txn_get_priv.vtc ##
>> ## test results in:
>> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
>>  c0   HTTP rx timeout (fd:6 5000 ms)
>>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
>> Makefile:995: recipe for target 'reg-tests' failed
>> make: *** [reg-tests] Error 1
>>
>> These tests fail both on AARCH64 and x86_64 consistently until I add back
>> USE_OPENSSL=1
>> to the flags.
>>
>> Is there an issue with these tests ?
>>
>> 1.
>> https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14
>>
>> Regards,
>> Martin
>>
>


Re: Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Илья Шипицин
do you run tests on GH arm64 agents ? is it dedicated (your own) agents
attached to your repo ? can you give a link ?

ср, 6 мая 2020 г. в 13:22, Martin Grigorov :

> Hello HAProxy team,
>
> While working on a PR to build & test HAProxy on AARCH64 at GitHub Actions
> I've noticed a strange behavior for some of the tests.
>
> To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS
> [1] passed to "make".
> The build passed successfully, some of the tests are skipped because they
> depend on SSL library, e.g.:
>
> ...
> Add test: reg-tests/lua/txn_get_priv.vtc
>   Add test: reg-tests/lua/h_txn_get_priv.vtc
>   Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not compiled
> with the required option OPENSSL
>   Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
> with the required option OPENSSL
>   Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled
> with the required option OPENSSL
>   Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled with
> the required option OPENSSL
> ...
>
> but few tests just fail:
>
> Testing with haproxy version: 2.2-dev7
> #top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
> #top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill -9)
> #top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
> signal=9
> 2 tests failed, 0 tests skipped, 64 tests passed
> ## Gathering results ##
> ## Test case: reg-tests/compression/lua_validation.vtc ##
> ## test results in:
> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
> ## Test case: reg-tests/lua/txn_get_priv.vtc ##
> ## test results in:
> "/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
>  c0   HTTP rx timeout (fd:6 5000 ms)
>  h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
> Makefile:995: recipe for target 'reg-tests' failed
> make: *** [reg-tests] Error 1
>
> These tests fail both on AARCH64 and x86_64 consistently until I add back
> USE_OPENSSL=1
> to the flags.
>
> Is there an issue with these tests ?
>
> 1.
> https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14
>
> Regards,
> Martin
>


Failing tests if USE_OPENSSL=1 is omitted in the FLAGS

2020-05-06 Thread Martin Grigorov
Hello HAProxy team,

While working on a PR to build & test HAProxy on AARCH64 at GitHub Actions
I've noticed a strange behavior for some of the tests.

To reduce the time of the build I've removed USE_OPENSSL=1 from the FLAGS
[1] passed to "make".
The build passed successfully, some of the tests are skipped because they
depend on SSL library, e.g.:

...
Add test: reg-tests/lua/txn_get_priv.vtc
  Add test: reg-tests/lua/h_txn_get_priv.vtc
  Skip reg-tests/ssl/wrong_ctx_storage.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/ssl_client_auth.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/add_ssl_crt-list.vtc because haproxy is not compiled
with the required option OPENSSL
  Skip reg-tests/ssl/set_ssl_cert.vtc because haproxy is not compiled with
the required option OPENSSL
...

but few tests just fail:

Testing with haproxy version: 2.2-dev7
#top  TEST reg-tests/lua/txn_get_priv.vtc FAILED (5.008) exit=2
#top  TEST reg-tests/compression/lua_validation.vtc TIMED OUT (kill -9)
#top  TEST reg-tests/compression/lua_validation.vtc FAILED (10.009)
signal=9
2 tests failed, 0 tests skipped, 64 tests passed
## Gathering results ##
## Test case: reg-tests/compression/lua_validation.vtc ##
## test results in:
"/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.7fffd08a"
## Test case: reg-tests/lua/txn_get_priv.vtc ##
## test results in:
"/tmp/haregtests-2020-05-06_09-08-49.WqBiWC/vtc.14212.065f2677"
 c0   HTTP rx timeout (fd:6 5000 ms)
 h1   Bad exit status: 0x0100 exit 0x1 signal 0 core 0
Makefile:995: recipe for target 'reg-tests' failed
make: *** [reg-tests] Error 1

These tests fail both on AARCH64 and x86_64 consistently until I add back
USE_OPENSSL=1
to the flags.

Is there an issue with these tests ?

1.
https://github.com/haproxy/haproxy/blob/fafa13dd6549ee431f41dc3c1857855974d79bea/.travis.yml#L14

Regards,
Martin