SSL Labs says my server isn't doing ssl session resumption

2021-06-11 Thread Shawn Heisey 

I'm fiddling with ssl labs to see how I can improve my TLS setup.

Here's what they say about a site I have behind haproxy with TLS:

https://www.elyograg.org/foo/haproxy-ssllabs-session-resumption-not-working.png

They claim that session resumption isn't working.  I'm hoping that I've 
just done something wrong... which will be bad for my ego but great for 
getting problems fixed.  I did have the option to disable tls tickets, 
but when I took it out, my ssl labs grade didn't go down, so it's still 
not there.


This is what I have in the global section:

global
log 127.0.0.1   len 65535 format rfc5424 local0
log 127.0.0.1   len 65535 format rfc5424 local1 notice
maxconn 4096
daemon
spread-checks   2
tune.bufsize65536
tune.http.logurilen 49152
tune.ssl.cachesize 10
tune.ssl.lifetime   900
ssl-server-verify   none
tune.ssl.default-dh-param   2048
ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-server-ciphers  
RC4-MD5:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES128-SHA:AES256-SHA:AES256-SHA256

stats socket /etc/haproxy/stats.socket

I don't think there's anything else in the config that could affect 
this, but if there is something that would help diagnose, let me know.


Here's info about haproxy, which I compiled from source:

root@smeagol:~# haproxy -vv
HAProxy version 2.4.0-6cbbecf 2021/05/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 
2026.

Known bugs: http://www.haproxy.org/bugs/bugs-2.4.0.html
Running on: Linux 5.8.0-55-generic #62~20.04.1-Ubuntu SMP Wed Jun 2 
08:55:04 UTC 2021 x86_64

Build options :
  TARGET  = linux-glibc
  CPU = native
  CC  = cc
  CFLAGS  = -O2 -march=native -g -Wall -Wextra 
-Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered 
-Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond 
-Wnull-dereference

  OPTIONS = USE_PCRE_JIT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE +PCRE_JIT -PCRE2 
-PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE 
-STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT 
+CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -CLOSEFROM +ZLIB 
-SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL 
+SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX 
-MEMORY_PROFILING


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND

Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.3.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' 
keyword)
  h2 : mode=HTTP   side=FE|BE mux=H2   
flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
fcgi : mode=HTTP   side=BEmux=FCGI 
flags=HTX|HOL_RISK|NO_UPG

: mode=HTTP   side=FE|BE mux=H1   flags=HTX
  h1 : mode=HTTP   side=FE|BE mux=H1   
flags=HTX|NO_UPG

: mode=TCPside=FE|BE mux=PASS flags=
none : mode=TCPside=FE|BE mux=PASS 
flags=NO_UPG


Available services : none

Available filters :
[SPOE] spoe
[CACHE] cache
[FCGI] fcgi-app
[COMP] compression
[TRACE] trace

Re: http-response set-header and redirect

2021-06-11 Thread James Brown 
Thanks!

On Fri, Jun 11, 2021 at 11:36 AM Tim Düsterhus  wrote:

> James,
>
> On 6/11/21 8:28 PM, James Brown wrote:
> > Is there any reason (performance or otherwise) to use http-response
> instead
> > of just turning everything into http-after-response?
>
> There is a difference: If a http-response rule fails [1] then a standard
> error page will be emitted. For this error page the http-after-response
> rules will need be evaluated. They might fail as well, aborting the
> processing and causing a very simple 500 Internal Server Error to be
> emitted. This will suppress any other error (e.g. 503, 403, …).
>
> So complex http-after-response rules might cause additional (debugging)
> issues in error situations.
>
> I recommend using them for the most essential stuff only. In my case
> that is the Strict-Transport-Security header and a request ID response
> header.
>
> Best regards
> Tim Düsterhus
>
> [1] e.g. if there's insufficient memory to add the header.
>


-- 
James Brown
Engineer

HAProxyConf 2021 - Call for papers

2021-06-11 Thread Willy Tarreau 
Hi all,

some of you have probably already noticed the announce in [1], the
2021 edition of the HAProxyConf will take place on November 16-17 as
a "virtual event" (I personally prefer to say "online" as "virtual"
always makes me feel that I'm losing something important).

For sure we'll all miss the ability to chat around a beer or at a
restaurant, but on the other hand it opens a great opportunity to
welcome among our speakers some who possibly don't want to travel
half of the world for 2 days. And while the 2019 edition was
awesome and the talks of great quality, we can't deny that speakers
and attendees mostly came from North America and Western Europe, and
I would be very surprised if there weren't other very interesting
use cases with different challenges in other regions.

As such, I encourage all those who think their use case is somewhat
unique, or who figured a great way to optimize their infrastructure
or their work and who want to share it with others and get their
feedback, to consider submitting a proposal. All the details for
the submissions are in [2], and the deadline for submissions is
June 25th. Please note that just like 2 years ago, you don't need
to send the presentation now, it's just a quick summary of what
you'd like to present and why you think it could interest others.

The talks will be 20-25mn so they won't require an extreme amount
of preparation, and this time excuses such as "it's too far",
"I'll be jet-lagged", "I don't fly" or "I don't have the time to
punch a 2-days hole in my schedule" won't stand :-)

The full details are in [3] below.

If you're looking for an idea of something to present, you can have a
look at suggestions in [1], but my personal advice would be to just
open your haproxy.cfg, look at it for a while and remember what problem
you solved with these strange few lines you've added to work around
a big outage, a DDoS, fix a bogus application, smoothly integrate
two mutually incompatible components, or save money by reducing the
number of servers. And think again about last time you said "when
I have some time to kill I'll need to document this, because it's
a great trick that can serve others". Now you have the opportunity
to do it, don't miss it!

Willy
--
[1] https://www.haproxy.com/blog/haproxyconf-2021-call-for-papers/
[2] https://www.haproxyconf.com/call-for-papers/
[3] https://www.haproxyconf.com/

Re: http-response set-header and redirect

2021-06-11 Thread Tim Düsterhus 

James,

On 6/11/21 8:28 PM, James Brown wrote:

Is there any reason (performance or otherwise) to use http-response instead
of just turning everything into http-after-response?


There is a difference: If a http-response rule fails [1] then a standard 
error page will be emitted. For this error page the http-after-response 
rules will need be evaluated. They might fail as well, aborting the 
processing and causing a very simple 500 Internal Server Error to be 
emitted. This will suppress any other error (e.g. 503, 403, …).


So complex http-after-response rules might cause additional (debugging) 
issues in error situations.


I recommend using them for the most essential stuff only. In my case 
that is the Strict-Transport-Security header and a request ID response 
header.


Best regards
Tim Düsterhus

[1] e.g. if there's insufficient memory to add the header.

Re: http-response set-header and redirect

2021-06-11 Thread James Brown 
Is there any reason (performance or otherwise) to use http-response instead
of just turning everything into http-after-response?

On Fri, Jun 11, 2021 at 11:07 AM Tim Düsterhus  wrote:

> James,
>
> On 6/11/21 8:03 PM, James Brown wrote:
> > Is there any way to set a HTTP header on a redirect being emitted by
> > haproxy?
>
> To also match HAProxy generated responses (including redirects and error
> pages) you will need to use 'http-after-response':
>
>
> https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#http-after-response
>
> Best regards
> Tim Düsterhus
>


-- 
James Brown
Engineer

Re: Weird behavior of spoe between http and https requests

2021-06-11 Thread Aleksandar Lazic 

Hi.

On 11.06.21 18:07, Aleksandar Lazic wrote:

Hi.

I use haproxy 2.4 with this fe config.

```
global
     log stdout format raw daemon
     daemon
     maxconn 2
     stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd 
listeners
     stats timeout 30s

     tune.ssl.default-dh-param 2048

     # Default SSL material locations
     ca-base /etc/ssl/certs
     crt-base /etc/ssl/private


     # See 
https://ssl-config.mozilla.org/#server=haproxy=2.1=old=1.1.1d=5.4
     ssl-default-bind-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
     ssl-default-bind-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
     ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.0

     ssl-default-server-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
     ssl-default-server-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
     ssl-default-server-options no-tls-tickets ssl-min-ver TLSv1.0


defaults http
   log global
   mode http
   retry-on all-retryable-errors
   option forwardfor
   option redispatch
   option http-ignore-probes
   option httplog
   option dontlognull
   option log-health-checks
   option socket-stats
   timeout connect 5s
   timeout client  50s
   timeout server  50s
   http-reuse safe
   errorfile 400 /etc/haproxy/errors/400.http
   errorfile 403 /etc/haproxy/errors/403.http
   errorfile 408 /etc/haproxy/errors/408.http
   errorfile 500 /etc/haproxy/errors/500.http
   errorfile 502 /etc/haproxy/errors/502.http
   errorfile 503 /etc/haproxy/errors/503.http
   errorfile 504 /etc/haproxy/errors/504.http

frontend http-in
   bind *:80
   mode http

   unique-id-format %rt
   http-request set-var(sess.my_fe_path) path
   http-request set-var(sess.my_fe_src) src
   http-request set-var(sess.my_fe_referer) req.hdr(Referer)
   http-request set-var(sess.my_fe_requestedhost) req.hdr(Host)

   # define the spoe agents
   filter spoe engine agent-on-http-req config /etc/haproxy/spoe-url.conf
   filter spoe engine agent-on-http-res config /etc/haproxy/spoe-url.conf

frontend https-in

   bind :::443 v4v6 alpn h2,http/1.1 ssl ca-file 
/etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/

   unique-id-format %rt
   http-request set-var(sess.my_fe_path) path
   http-request set-var(sess.my_fe_src) src
   http-request set-var(sess.my_fe_referer) req.hdr(Referer)
   http-request set-var(sess.my_fe_requestedhost) req.hdr(Host)

   # define the spoe agents
   filter spoe engine agent-on-http-req config /etc/haproxy/spoe-url.conf
   filter spoe engine agent-on-http-res config /etc/haproxy/spoe-url.conf
```

And with this spoe config.
```
[agent-on-http-req]
spoe-agent agent-on-http-req

     log global

     messages agent-on-http-req

     option var-prefix feevents

     timeout hello  2s
     timeout idle   2m
     timeout processing 1s

     use-backend agent-on-http-req

spoe-message agent-on-http-req
     args my_path=path my_src=src my_referer=req.hdr(Referer) my_sid=unique-id 
my_req_host=req.hdr(Host)
     event on-frontend-http-request

[agent-on-http-res]
spoe-agent agent-on-http-res

     log global

     messages agent-on-http-res

     option var-prefix feevents

     timeout hello  2s
     timeout idle   2m
     timeout processing 1s

     use-backend agent-on-http-res

spoe-message agent-on-http-res
     args my_path=var(sess.my_fe_path) my_src=src 
my_referer=var(sess.my_fe_referer) my_sid=unique-id 
my_req_host=var(sess.my_fe_requestedhost)
     event on-http-response
```

Now when I make a http request I get all values and args.
```
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Msg Name  
:agent-on-http-req:
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Msg Count :5:
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name  
:my_path:
Jun 11

Re: [PATCH 0/4] Use 'feature cmd' in regtests

2021-06-11 Thread Илья Шипицин 
haproxy/regression-testing.txt at master · haproxy/haproxy (github.com)


can it be converted into md/rst ?? please please ?

пт, 11 июн. 2021 г. в 23:19, Илья Шипицин :

> there's reg-test documentation for beginners.
> should it be updated as well ?
>
> пт, 11 июн. 2021 г. в 22:56, Tim Duesterhus :
>
>> Hi!
>>
>> I hope I added all the active developers that touch the reg-tests to the
>> 'CC'
>> list :-)
>>
>> This series updates the regtests to make use of VTest's 'feature cmd'
>> syntax
>> to skip tests that are not supported in the current environment.
>>
>> In the long run this will should result in much cleaner tests, because
>> they
>> don't need to be parsed by run-regtests.sh to extract these marker
>> comments. It
>> also allows to easily add test specific requirements without needing to
>> invent
>> an entirely new syntax. An example might be the recently added tests that
>> are
>> not supported on BoringSSL. These should be able to get a condition like:
>>
>> feature cmd "$HAPROXY_PROGRAM -vv |grep -vq BoringSSL"
>>
>> and then they won't run for BoringSSL (the example is untested).
>>
>> After this series is applied the output of 'make reg-tests' will change.
>> Tests
>> that are excluded using 'feature cmd' will still be listed as "Add test"
>> in
>> the test gathering section. However the final line will show that a few
>> tests
>> have been skipped:
>>
>> 0 tests failed, 4 tests skipped, 105 tests passed
>>
>> I don't think this is going to be an issue. But if it is, please complain!
>>
>> If all of you are happy with the series then it might be merged. Just
>> remember
>> to use 'feature cmd' after it is applied :-)
>>
>> Best regards
>>
>> Tim Düsterhus (4):
>>   REGTESTS: Replace REQUIRE_VERSION=2.5 with 'haproxy -cc'
>>   REGTESTS: Replace REQUIRE_OPTIONS with 'haproxy -cc' for 2.5+ tests
>>   REGTESTS: Replace REQUIRE_BINARIES with 'command -v'
>>   REGTESTS: Remove support for REQUIRE_BINARIES
>>
>>  reg-tests/http-messaging/http_abortonclose.vtc |  2 +-
>>  reg-tests/mcli/mcli_start_progs.vtc|  2 +-
>>  reg-tests/ssl/add_ssl_crt-list.vtc |  2 +-
>>  reg-tests/ssl/new_del_ssl_cafile.vtc   |  6 +++---
>>  reg-tests/ssl/new_del_ssl_crlfile.vtc  |  6 +++---
>>  reg-tests/ssl/set_ssl_cafile.vtc   |  6 +++---
>>  reg-tests/ssl/set_ssl_cert.vtc |  2 +-
>>  reg-tests/ssl/set_ssl_cert_bundle.vtc  |  2 +-
>>  reg-tests/ssl/set_ssl_cert_noext.vtc   |  2 +-
>>  reg-tests/ssl/set_ssl_crlfile.vtc  |  6 +++---
>>  reg-tests/ssl/set_ssl_server_cert.vtc  |  2 +-
>>  reg-tests/ssl/show_ssl_ocspresponse.vtc|  6 +++---
>>  reg-tests/startup/check_condition.vtc  |  1 -
>>  scripts/run-regtests.sh| 12 
>>  14 files changed, 22 insertions(+), 35 deletions(-)
>>
>> --
>> 2.32.0
>>
>>

Re: [PATCH 0/4] Use 'feature cmd' in regtests

2021-06-11 Thread Илья Шипицин 
there's reg-test documentation for beginners.
should it be updated as well ?

пт, 11 июн. 2021 г. в 22:56, Tim Duesterhus :

> Hi!
>
> I hope I added all the active developers that touch the reg-tests to the
> 'CC'
> list :-)
>
> This series updates the regtests to make use of VTest's 'feature cmd'
> syntax
> to skip tests that are not supported in the current environment.
>
> In the long run this will should result in much cleaner tests, because they
> don't need to be parsed by run-regtests.sh to extract these marker
> comments. It
> also allows to easily add test specific requirements without needing to
> invent
> an entirely new syntax. An example might be the recently added tests that
> are
> not supported on BoringSSL. These should be able to get a condition like:
>
> feature cmd "$HAPROXY_PROGRAM -vv |grep -vq BoringSSL"
>
> and then they won't run for BoringSSL (the example is untested).
>
> After this series is applied the output of 'make reg-tests' will change.
> Tests
> that are excluded using 'feature cmd' will still be listed as "Add test" in
> the test gathering section. However the final line will show that a few
> tests
> have been skipped:
>
> 0 tests failed, 4 tests skipped, 105 tests passed
>
> I don't think this is going to be an issue. But if it is, please complain!
>
> If all of you are happy with the series then it might be merged. Just
> remember
> to use 'feature cmd' after it is applied :-)
>
> Best regards
>
> Tim Düsterhus (4):
>   REGTESTS: Replace REQUIRE_VERSION=2.5 with 'haproxy -cc'
>   REGTESTS: Replace REQUIRE_OPTIONS with 'haproxy -cc' for 2.5+ tests
>   REGTESTS: Replace REQUIRE_BINARIES with 'command -v'
>   REGTESTS: Remove support for REQUIRE_BINARIES
>
>  reg-tests/http-messaging/http_abortonclose.vtc |  2 +-
>  reg-tests/mcli/mcli_start_progs.vtc|  2 +-
>  reg-tests/ssl/add_ssl_crt-list.vtc |  2 +-
>  reg-tests/ssl/new_del_ssl_cafile.vtc   |  6 +++---
>  reg-tests/ssl/new_del_ssl_crlfile.vtc  |  6 +++---
>  reg-tests/ssl/set_ssl_cafile.vtc   |  6 +++---
>  reg-tests/ssl/set_ssl_cert.vtc |  2 +-
>  reg-tests/ssl/set_ssl_cert_bundle.vtc  |  2 +-
>  reg-tests/ssl/set_ssl_cert_noext.vtc   |  2 +-
>  reg-tests/ssl/set_ssl_crlfile.vtc  |  6 +++---
>  reg-tests/ssl/set_ssl_server_cert.vtc  |  2 +-
>  reg-tests/ssl/show_ssl_ocspresponse.vtc|  6 +++---
>  reg-tests/startup/check_condition.vtc  |  1 -
>  scripts/run-regtests.sh| 12 
>  14 files changed, 22 insertions(+), 35 deletions(-)
>
> --
> 2.32.0
>
>

Re: http-response set-header and redirect

2021-06-11 Thread Tim Düsterhus 

James,

On 6/11/21 8:03 PM, James Brown wrote:

Is there any way to set a HTTP header on a redirect being emitted by
haproxy?


To also match HAProxy generated responses (including redirects and error 
pages) you will need to use 'http-after-response':


https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#http-after-response

Best regards
Tim Düsterhus

http-response set-header and redirect

2021-06-11 Thread James Brown 
Is there any way to set a HTTP header on a redirect being emitted by
haproxy?

Given the following simplified config:

global
log stdout user

defaults
log global
timeout client 9s
timeout server 10s
timeout connect 1s

frontend test_fe
mode http
http-response set-header Foo Bar
bind localhost:
redirect prefix https://www.example.com

It appears that the Foo header is not set when the redirect is emitted. Is
there any way to configure HAproxy to process `http-response` statements on
a redirect?

-- 
James Brown
Engineer

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
it works :)

oops · chipitsine/haproxy@2ce9681 (github.com)


I'll polish it a bit and will send final patch tomorrow

пт, 11 июн. 2021 г. в 20:42, Илья Шипицин :

>
>
> пт, 11 июн. 2021 г. в 20:34, William Lallemand :
>
>> On Fri, Jun 11, 2021 at 08:14:49PM +0500, Илья Шипицин wrote:
>> > I've found ubuntu musl package, so we can just link to it in CI, for
>> > example (I'll try)
>> >
>>
>>
>> Well, that won't give you the same environnement as a docker image,
>> with the same versions. I'll honestly prefer if we could do it with the
>> alpine image.
>>
>
> there's always gap between "what is tested" and "what is packaged".
> package maintainers add untested selinux patches and link against
> various libs.
>
> I agree with you that we should prefer to test on alpine natively.
>
>
>>
>> I you want to build with the musl-gcc wrapper you will need to link the
>> linux headers in the musl headers directory otherwise it won't work the
>> way their package is done.
>>
>
>
>
>
>
>>
>> --
>> William Lallemand
>>
>

[PATCH 1/4] REGTESTS: Replace REQUIRE_VERSION=2.5 with 'haproxy -cc'

2021-06-11 Thread Tim Duesterhus 
This is safe, because running `haproxy -cc 'version_atleast(2.5-dev0)'` on
HAProxy 2.4 will also result in an exit code of 1.
---
 reg-tests/http-messaging/http_abortonclose.vtc | 2 +-
 reg-tests/ssl/new_del_ssl_cafile.vtc   | 2 +-
 reg-tests/ssl/new_del_ssl_crlfile.vtc  | 2 +-
 reg-tests/ssl/set_ssl_cafile.vtc   | 2 +-
 reg-tests/ssl/set_ssl_crlfile.vtc  | 2 +-
 reg-tests/ssl/show_ssl_ocspresponse.vtc| 2 +-
 reg-tests/startup/check_condition.vtc  | 1 -
 7 files changed, 6 insertions(+), 7 deletions(-)

diff --git a/reg-tests/http-messaging/http_abortonclose.vtc 
b/reg-tests/http-messaging/http_abortonclose.vtc
index 49cdaeadd..c083f29f1 100644
--- a/reg-tests/http-messaging/http_abortonclose.vtc
+++ b/reg-tests/http-messaging/http_abortonclose.vtc
@@ -4,7 +4,7 @@ feature ignore_unknown_macro
 # NOTE : This test may fail if too many vtest are running in parallel because
 #the port reserved for closed s1 server may be reused by another vtest
 
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REGTEST_TYPE=slow
 
 # b1 : Don't send /c4 before /c3 was received by s2 server
diff --git a/reg-tests/ssl/new_del_ssl_cafile.vtc 
b/reg-tests/ssl/new_del_ssl_cafile.vtc
index 613e12866..f943a6632 100644
--- a/reg-tests/ssl/new_del_ssl_cafile.vtc
+++ b/reg-tests/ssl/new_del_ssl_cafile.vtc
@@ -9,7 +9,7 @@
 # - Check that you have socat
 
 varnishtest "Test the 'new ssl ca-file' and 'del ssl ca-file' commands of the 
CLI"
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REQUIRE_OPTIONS=OPENSSL
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
diff --git a/reg-tests/ssl/new_del_ssl_crlfile.vtc 
b/reg-tests/ssl/new_del_ssl_crlfile.vtc
index 0d82200e7..027f00096 100644
--- a/reg-tests/ssl/new_del_ssl_crlfile.vtc
+++ b/reg-tests/ssl/new_del_ssl_crlfile.vtc
@@ -9,7 +9,7 @@
 # - Check that you have socat
 
 varnishtest "Test the 'new ssl crl-file' and 'del ssl crl-file' commands of 
the CLI"
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REQUIRE_OPTIONS=OPENSSL
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
diff --git a/reg-tests/ssl/set_ssl_cafile.vtc b/reg-tests/ssl/set_ssl_cafile.vtc
index eb625639b..2f4ef2e8a 100644
--- a/reg-tests/ssl/set_ssl_cafile.vtc
+++ b/reg-tests/ssl/set_ssl_cafile.vtc
@@ -15,7 +15,7 @@
 # - Check that you have socat
 
 varnishtest "Test the 'set ssl ca-file' feature of the CLI"
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REQUIRE_OPTIONS=OPENSSL
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
diff --git a/reg-tests/ssl/set_ssl_crlfile.vtc 
b/reg-tests/ssl/set_ssl_crlfile.vtc
index 7060a1477..d3aad9065 100644
--- a/reg-tests/ssl/set_ssl_crlfile.vtc
+++ b/reg-tests/ssl/set_ssl_crlfile.vtc
@@ -18,7 +18,7 @@
 # - Check that you have socat
 
 varnishtest "Test the 'set ssl crl-file' feature of the CLI"
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REQUIRE_OPTIONS=OPENSSL
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
diff --git a/reg-tests/ssl/show_ssl_ocspresponse.vtc 
b/reg-tests/ssl/show_ssl_ocspresponse.vtc
index b7a2d5105..a942f2c47 100644
--- a/reg-tests/ssl/show_ssl_ocspresponse.vtc
+++ b/reg-tests/ssl/show_ssl_ocspresponse.vtc
@@ -19,7 +19,7 @@
 # - Check that you have socat
 
 varnishtest "Test the 'show ssl ocsp-response' and 'show ssl cert 
foo.pem.ocsp' features of the CLI"
-#REQUIRE_VERSION=2.5
+feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 #REQUIRE_OPTIONS=OPENSSL
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
diff --git a/reg-tests/startup/check_condition.vtc 
b/reg-tests/startup/check_condition.vtc
index 7d3324bfe..d56d73fde 100644
--- a/reg-tests/startup/check_condition.vtc
+++ b/reg-tests/startup/check_condition.vtc
@@ -1,6 +1,5 @@
 varnishtest "Tests the -cc argument"
 
-#REQUIRE_VERSION=2.5
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 
 shell {
-- 
2.32.0

[PATCH 0/4] Use 'feature cmd' in regtests

2021-06-11 Thread Tim Duesterhus 
Hi!

I hope I added all the active developers that touch the reg-tests to the 'CC'
list :-)

This series updates the regtests to make use of VTest's 'feature cmd' syntax
to skip tests that are not supported in the current environment.

In the long run this will should result in much cleaner tests, because they
don't need to be parsed by run-regtests.sh to extract these marker comments. It
also allows to easily add test specific requirements without needing to invent
an entirely new syntax. An example might be the recently added tests that are
not supported on BoringSSL. These should be able to get a condition like:

feature cmd "$HAPROXY_PROGRAM -vv |grep -vq BoringSSL"

and then they won't run for BoringSSL (the example is untested).

After this series is applied the output of 'make reg-tests' will change. Tests
that are excluded using 'feature cmd' will still be listed as "Add test" in
the test gathering section. However the final line will show that a few tests
have been skipped:

0 tests failed, 4 tests skipped, 105 tests passed

I don't think this is going to be an issue. But if it is, please complain!

If all of you are happy with the series then it might be merged. Just remember
to use 'feature cmd' after it is applied :-)

Best regards

Tim Düsterhus (4):
  REGTESTS: Replace REQUIRE_VERSION=2.5 with 'haproxy -cc'
  REGTESTS: Replace REQUIRE_OPTIONS with 'haproxy -cc' for 2.5+ tests
  REGTESTS: Replace REQUIRE_BINARIES with 'command -v'
  REGTESTS: Remove support for REQUIRE_BINARIES

 reg-tests/http-messaging/http_abortonclose.vtc |  2 +-
 reg-tests/mcli/mcli_start_progs.vtc|  2 +-
 reg-tests/ssl/add_ssl_crt-list.vtc |  2 +-
 reg-tests/ssl/new_del_ssl_cafile.vtc   |  6 +++---
 reg-tests/ssl/new_del_ssl_crlfile.vtc  |  6 +++---
 reg-tests/ssl/set_ssl_cafile.vtc   |  6 +++---
 reg-tests/ssl/set_ssl_cert.vtc |  2 +-
 reg-tests/ssl/set_ssl_cert_bundle.vtc  |  2 +-
 reg-tests/ssl/set_ssl_cert_noext.vtc   |  2 +-
 reg-tests/ssl/set_ssl_crlfile.vtc  |  6 +++---
 reg-tests/ssl/set_ssl_server_cert.vtc  |  2 +-
 reg-tests/ssl/show_ssl_ocspresponse.vtc|  6 +++---
 reg-tests/startup/check_condition.vtc  |  1 -
 scripts/run-regtests.sh| 12 
 14 files changed, 22 insertions(+), 35 deletions(-)

-- 
2.32.0

[PATCH 3/4] REGTESTS: Replace REQUIRE_BINARIES with 'command -v'

2021-06-11 Thread Tim Duesterhus 
This migrates the tests to the native `feature cmd` functionality of VTest.
---
 reg-tests/mcli/mcli_start_progs.vtc | 2 +-
 reg-tests/ssl/add_ssl_crt-list.vtc  | 2 +-
 reg-tests/ssl/new_del_ssl_cafile.vtc| 2 +-
 reg-tests/ssl/new_del_ssl_crlfile.vtc   | 2 +-
 reg-tests/ssl/set_ssl_cafile.vtc| 2 +-
 reg-tests/ssl/set_ssl_cert.vtc  | 2 +-
 reg-tests/ssl/set_ssl_cert_bundle.vtc   | 2 +-
 reg-tests/ssl/set_ssl_cert_noext.vtc| 2 +-
 reg-tests/ssl/set_ssl_crlfile.vtc   | 2 +-
 reg-tests/ssl/set_ssl_server_cert.vtc   | 2 +-
 reg-tests/ssl/show_ssl_ocspresponse.vtc | 2 +-
 11 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/reg-tests/mcli/mcli_start_progs.vtc 
b/reg-tests/mcli/mcli_start_progs.vtc
index 08de157f1..eb6f63502 100644
--- a/reg-tests/mcli/mcli_start_progs.vtc
+++ b/reg-tests/mcli/mcli_start_progs.vtc
@@ -1,7 +1,7 @@
 varnishtest "Try to start a master CLI with 2 programs"
 #REGTEST_TYPE=bug
 #REQUIRE_VERSION=2.0
-#REQUIRE_BINARIES=sleep
+feature cmd "command -v sleep"
 
 feature ignore_unknown_macro
 
diff --git a/reg-tests/ssl/add_ssl_crt-list.vtc 
b/reg-tests/ssl/add_ssl_crt-list.vtc
index ca5228501..7aae2338a 100644
--- a/reg-tests/ssl/add_ssl_crt-list.vtc
+++ b/reg-tests/ssl/add_ssl_crt-list.vtc
@@ -13,7 +13,7 @@
 varnishtest "Test the 'add ssl crt-list' feature of the CLI"
 #REQUIRE_VERSION=2.2
 #REQUIRE_OPTIONS=OPENSSL
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 2 {
diff --git a/reg-tests/ssl/new_del_ssl_cafile.vtc 
b/reg-tests/ssl/new_del_ssl_cafile.vtc
index 536db5007..1b5bef1a4 100644
--- a/reg-tests/ssl/new_del_ssl_cafile.vtc
+++ b/reg-tests/ssl/new_del_ssl_cafile.vtc
@@ -11,7 +11,7 @@
 varnishtest "Test the 'new ssl ca-file' and 'del ssl ca-file' commands of the 
CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 2 {
diff --git a/reg-tests/ssl/new_del_ssl_crlfile.vtc 
b/reg-tests/ssl/new_del_ssl_crlfile.vtc
index eeed09caf..54bbdc239 100644
--- a/reg-tests/ssl/new_del_ssl_crlfile.vtc
+++ b/reg-tests/ssl/new_del_ssl_crlfile.vtc
@@ -11,7 +11,7 @@
 varnishtest "Test the 'new ssl crl-file' and 'del ssl crl-file' commands of 
the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 3 {
diff --git a/reg-tests/ssl/set_ssl_cafile.vtc b/reg-tests/ssl/set_ssl_cafile.vtc
index 5dcfaf989..72ce3e6dc 100644
--- a/reg-tests/ssl/set_ssl_cafile.vtc
+++ b/reg-tests/ssl/set_ssl_cafile.vtc
@@ -17,7 +17,7 @@
 varnishtest "Test the 'set ssl ca-file' feature of the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 4 {
diff --git a/reg-tests/ssl/set_ssl_cert.vtc b/reg-tests/ssl/set_ssl_cert.vtc
index c4d088306..85684bc3e 100644
--- a/reg-tests/ssl/set_ssl_cert.vtc
+++ b/reg-tests/ssl/set_ssl_cert.vtc
@@ -22,7 +22,7 @@
 varnishtest "Test the 'set ssl cert' feature of the CLI"
 #REQUIRE_VERSION=2.2
 #REQUIRE_OPTIONS=OPENSSL
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 9 {
diff --git a/reg-tests/ssl/set_ssl_cert_bundle.vtc 
b/reg-tests/ssl/set_ssl_cert_bundle.vtc
index aaec89dda..218f7bfb4 100644
--- a/reg-tests/ssl/set_ssl_cert_bundle.vtc
+++ b/reg-tests/ssl/set_ssl_cert_bundle.vtc
@@ -17,7 +17,7 @@
 varnishtest "Test the 'set ssl cert' feature of the CLI with bundles"
 #REQUIRE_VERSION=2.3
 #REQUIRE_OPTIONS=OPENSSL
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 9 {
diff --git a/reg-tests/ssl/set_ssl_cert_noext.vtc 
b/reg-tests/ssl/set_ssl_cert_noext.vtc
index f1c42ff84..b7bafa8a3 100644
--- a/reg-tests/ssl/set_ssl_cert_noext.vtc
+++ b/reg-tests/ssl/set_ssl_cert_noext.vtc
@@ -14,7 +14,7 @@
 varnishtest "Test the 'set ssl cert' feature of the CLI with separate key and 
crt"
 #REQUIRE_VERSION=2.2
 #REQUIRE_OPTIONS=OPENSSL
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 3 {
diff --git a/reg-tests/ssl/set_ssl_crlfile.vtc 
b/reg-tests/ssl/set_ssl_crlfile.vtc
index 4ee7c1225..f6d97ce6b 100644
--- a/reg-tests/ssl/set_ssl_crlfile.vtc
+++ b/reg-tests/ssl/set_ssl_crlfile.vtc
@@ -20,7 +20,7 @@
 varnishtest "Test the 'set ssl crl-file' feature of the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
 feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
-#REQUIRE_BINARIES=socat
+feature cmd "command -v socat"
 feature ignore_unknown_macro
 
 server s1 -repeat 4 {
diff --git

[PATCH 2/4] REGTESTS: Replace REQUIRE_OPTIONS with 'haproxy -cc' for 2.5+ tests

2021-06-11 Thread Tim Duesterhus 
This migrates the tests for HAProxy versions that support '-cc' to the native
VTest functionality.
---
 reg-tests/ssl/new_del_ssl_cafile.vtc| 2 +-
 reg-tests/ssl/new_del_ssl_crlfile.vtc   | 2 +-
 reg-tests/ssl/set_ssl_cafile.vtc| 2 +-
 reg-tests/ssl/set_ssl_crlfile.vtc   | 2 +-
 reg-tests/ssl/show_ssl_ocspresponse.vtc | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/reg-tests/ssl/new_del_ssl_cafile.vtc 
b/reg-tests/ssl/new_del_ssl_cafile.vtc
index f943a6632..536db5007 100644
--- a/reg-tests/ssl/new_del_ssl_cafile.vtc
+++ b/reg-tests/ssl/new_del_ssl_cafile.vtc
@@ -10,7 +10,7 @@
 
 varnishtest "Test the 'new ssl ca-file' and 'del ssl ca-file' commands of the 
CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
-#REQUIRE_OPTIONS=OPENSSL
+feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
 
diff --git a/reg-tests/ssl/new_del_ssl_crlfile.vtc 
b/reg-tests/ssl/new_del_ssl_crlfile.vtc
index 027f00096..eeed09caf 100644
--- a/reg-tests/ssl/new_del_ssl_crlfile.vtc
+++ b/reg-tests/ssl/new_del_ssl_crlfile.vtc
@@ -10,7 +10,7 @@
 
 varnishtest "Test the 'new ssl crl-file' and 'del ssl crl-file' commands of 
the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
-#REQUIRE_OPTIONS=OPENSSL
+feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
 
diff --git a/reg-tests/ssl/set_ssl_cafile.vtc b/reg-tests/ssl/set_ssl_cafile.vtc
index 2f4ef2e8a..5dcfaf989 100644
--- a/reg-tests/ssl/set_ssl_cafile.vtc
+++ b/reg-tests/ssl/set_ssl_cafile.vtc
@@ -16,7 +16,7 @@
 
 varnishtest "Test the 'set ssl ca-file' feature of the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
-#REQUIRE_OPTIONS=OPENSSL
+feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
 
diff --git a/reg-tests/ssl/set_ssl_crlfile.vtc 
b/reg-tests/ssl/set_ssl_crlfile.vtc
index d3aad9065..4ee7c1225 100644
--- a/reg-tests/ssl/set_ssl_crlfile.vtc
+++ b/reg-tests/ssl/set_ssl_crlfile.vtc
@@ -19,7 +19,7 @@
 
 varnishtest "Test the 'set ssl crl-file' feature of the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
-#REQUIRE_OPTIONS=OPENSSL
+feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
 
diff --git a/reg-tests/ssl/show_ssl_ocspresponse.vtc 
b/reg-tests/ssl/show_ssl_ocspresponse.vtc
index a942f2c47..29673a525 100644
--- a/reg-tests/ssl/show_ssl_ocspresponse.vtc
+++ b/reg-tests/ssl/show_ssl_ocspresponse.vtc
@@ -20,7 +20,7 @@
 
 varnishtest "Test the 'show ssl ocsp-response' and 'show ssl cert 
foo.pem.ocsp' features of the CLI"
 feature cmd "$HAPROXY_PROGRAM -cc 'version_atleast(2.5-dev0)'"
-#REQUIRE_OPTIONS=OPENSSL
+feature cmd "$HAPROXY_PROGRAM -cc 'feature(OPENSSL)'"
 #REQUIRE_BINARIES=socat
 feature ignore_unknown_macro
 
-- 
2.32.0

[PATCH 4/4] REGTESTS: Remove support for REQUIRE_BINARIES

2021-06-11 Thread Tim Duesterhus 
This is no longer used since the migration to the native `feature cmd`
functionality.
---
 scripts/run-regtests.sh | 12 
 1 file changed, 12 deletions(-)

diff --git a/scripts/run-regtests.sh b/scripts/run-regtests.sh
index b542f24f8..6eadc06c7 100755
--- a/scripts/run-regtests.sh
+++ b/scripts/run-regtests.sh
@@ -56,9 +56,6 @@ _help()
 #REQUIRE_VERSION=0.0
 #REQUIRE_VERSION_BELOW=99.9
 
-# To define required binaries for a test:
-#REQUIRE_BINARIES=socat,curl
-
   Configure environment variables to set the haproxy and vtest binaries to use
 setenv HAPROXY_PROGRAM /usr/local/sbin/haproxy
 setenv VTEST_PROGRAM /usr/local/bin/vtest
@@ -128,7 +125,6 @@ _findtests() {
 require_options="$(sed -ne 's/^#REQUIRE_OPTIONS=//p' "$i" | sed  -e 's/,/ 
/g')"
 require_services="$(sed -ne 's/^#REQUIRE_SERVICES=//p' "$i" | sed  -e 
's/,/ /g')"
 exclude_targets="$(sed -ne 's/^#EXCLUDE_TARGETS=//p' "$i" | sed  -e 's/,/ 
/g')"
-require_binaries="$(sed -ne 's/^#REQUIRE_BINARIES=//p' "$i" | sed  -e 
's/,/ /g')"
 if [ $any_test -ne 1 ] ; then
 regtest_type="$(sed -ne 's/^#REGTEST_TYPE=//p' "$i")"
 if [ -z $regtest_type ] ; then
@@ -205,14 +201,6 @@ _findtests() {
   fi
 done
 
-for requiredbin in $require_binaries; do
-  if ! command -v $requiredbin >/dev/null 2>&1
-  then
-echo "  Skip $i because '"$requiredbin"' is not installed"
-skiptest=1
-  fi
-done
-
 if [ -z $skiptest ]; then
   echo "  Add test: $i"
   testlist="$testlist $i"
-- 
2.32.0

Re: [PATCH 1/2] REGTESTS: Remove REQUIRE_VERSION=1.6 from all tests

2021-06-11 Thread Willy Tarreau 
On Fri, Jun 11, 2021 at 06:16:24PM +0200, Tim Duesterhus wrote:
> HAProxy 1.6 is EOL, thus this always matches.
(...)
Both applied, thanks Tim!
willy

[PATCH 1/2] REGTESTS: Remove REQUIRE_VERSION=1.6 from all tests

2021-06-11 Thread Tim Duesterhus 
HAProxy 1.6 is EOL, thus this always matches.
---
 reg-tests/compression/lua_validation.vtc  | 1 -
 reg-tests/converter/json.vtc  | 1 -
 reg-tests/converter/url_dec.vtc   | 1 -
 reg-tests/http-messaging/h1_to_h1.vtc | 1 -
 reg-tests/http-messaging/http_request_buffer.vtc  | 1 -
 reg-tests/http-rules/h1_to_h1c.vtc| 1 -
 reg-tests/http-rules/map_redirect.vtc | 1 -
 reg-tests/lua/lua_socket.vtc  | 1 -
 reg-tests/mailers/healthcheckmail.vtc | 1 -
 reg-tests/ssl/ssl_client_auth.vtc | 1 -
 reg-tests/webstats/webstats-scope-and-post-change.vtc | 1 -
 11 files changed, 11 deletions(-)

diff --git a/reg-tests/compression/lua_validation.vtc 
b/reg-tests/compression/lua_validation.vtc
index d001c2d32..b10cbd93e 100644
--- a/reg-tests/compression/lua_validation.vtc
+++ b/reg-tests/compression/lua_validation.vtc
@@ -1,7 +1,6 @@
 # Checks that compression doesn't cause corruption..
 
 varnishtest "Compression validation"
-#REQUIRE_VERSION=1.6
 #REQUIRE_OPTIONS=ZLIB|SLZ,LUA,OPENSSL
 #REGTEST_TYPE=slow
 
diff --git a/reg-tests/converter/json.vtc b/reg-tests/converter/json.vtc
index efac8f622..b1c5f3800 100644
--- a/reg-tests/converter/json.vtc
+++ b/reg-tests/converter/json.vtc
@@ -1,6 +1,5 @@
 varnishtest "json converter test"
 
-#REQUIRE_VERSION=1.6
 
 feature ignore_unknown_macro
 
diff --git a/reg-tests/converter/url_dec.vtc b/reg-tests/converter/url_dec.vtc
index 464c35a27..9db3b64ef 100644
--- a/reg-tests/converter/url_dec.vtc
+++ b/reg-tests/converter/url_dec.vtc
@@ -1,6 +1,5 @@
 varnishtest "url_dec converter Test"
 
-#REQUIRE_VERSION=1.6
 
 feature ignore_unknown_macro
 
diff --git a/reg-tests/http-messaging/h1_to_h1.vtc 
b/reg-tests/http-messaging/h1_to_h1.vtc
index 4a442c79d..8d784d681 100644
--- a/reg-tests/http-messaging/h1_to_h1.vtc
+++ b/reg-tests/http-messaging/h1_to_h1.vtc
@@ -1,5 +1,4 @@
 varnishtest "HTTP request tests: H1 to H1 (HTX mode supported only for HAProxy 
>= 1.9)"
-#REQUIRE_VERSION=1.6
 
 # Run it with HAPROXY_PROGRAM=$PWD/haproxy varnishtest -l -k -t 1 "$1"
 
diff --git a/reg-tests/http-messaging/http_request_buffer.vtc 
b/reg-tests/http-messaging/http_request_buffer.vtc
index 1ac04c7ac..d9cc6aec8 100644
--- a/reg-tests/http-messaging/http_request_buffer.vtc
+++ b/reg-tests/http-messaging/http_request_buffer.vtc
@@ -1,7 +1,6 @@
 varnishtest "A test for http-request-buffer option"
 feature ignore_unknown_macro
 
-#REQUIRE_VERSION=1.6
 
 # This test checks HTTP request buffering feature.
 # We run one server s1 which can serve only one client (no -repeat argument 
here).
diff --git a/reg-tests/http-rules/h1_to_h1c.vtc 
b/reg-tests/http-rules/h1_to_h1c.vtc
index aedb41ff7..5ae1f9335 100644
--- a/reg-tests/http-rules/h1_to_h1c.vtc
+++ b/reg-tests/http-rules/h1_to_h1c.vtc
@@ -1,5 +1,4 @@
 varnishtest "Composite HTTP manipulation test (H1 clear to H1 clear)"
-#REQUIRE_VERSION=1.6
 #REQUIRE_VERSION_BELOW=1.9
 
 # This config tests several http-request features and their interactions.
diff --git a/reg-tests/http-rules/map_redirect.vtc 
b/reg-tests/http-rules/map_redirect.vtc
index f6a0eeb2d..77e9b0dc3 100644
--- a/reg-tests/http-rules/map_redirect.vtc
+++ b/reg-tests/http-rules/map_redirect.vtc
@@ -2,7 +2,6 @@ varnishtest "haproxy host header: map / redirect tests"
 #REQUIRE_OPTIONS=PCRE|PCRE2
 feature ignore_unknown_macro
 
-#REQUIRE_VERSION=1.6
 
 server s1 {
rxreq
diff --git a/reg-tests/lua/lua_socket.vtc b/reg-tests/lua/lua_socket.vtc
index 2e126f5c1..83e06a63d 100644
--- a/reg-tests/lua/lua_socket.vtc
+++ b/reg-tests/lua/lua_socket.vtc
@@ -2,7 +2,6 @@ varnishtest "Lua: check socket functionality from a lua-task"
 feature ignore_unknown_macro
 
 #REQUIRE_OPTIONS=LUA
-#REQUIRE_VERSION=1.6
 
 server s1 {
 rxreq
diff --git a/reg-tests/mailers/healthcheckmail.vtc 
b/reg-tests/mailers/healthcheckmail.vtc
index ce3335f9b..75325080b 100644
--- a/reg-tests/mailers/healthcheckmail.vtc
+++ b/reg-tests/mailers/healthcheckmail.vtc
@@ -1,6 +1,5 @@
 varnishtest "Lua: txn:get_priv() scope"
 #REQUIRE_OPTIONS=LUA
-#REQUIRE_VERSION=1.6
 #REGTEST_TYPE=broken
 
 feature ignore_unknown_macro
diff --git a/reg-tests/ssl/ssl_client_auth.vtc 
b/reg-tests/ssl/ssl_client_auth.vtc
index f82be7b16..885302e47 100644
--- a/reg-tests/ssl/ssl_client_auth.vtc
+++ b/reg-tests/ssl/ssl_client_auth.vtc
@@ -15,7 +15,6 @@
 # 
https://www.haproxy.com/blog/ssl-client-certificate-management-at-application-level/
 
 varnishtest "Test the client auth"
-#REQUIRE_VERSION=1.6
 #REQUIRE_OPTIONS=OPENSSL
 feature ignore_unknown_macro
 
diff --git a/reg-tests/webstats/webstats-scope-and-post-change.vtc 
b/reg-tests/webstats/webstats-scope-and-post-change.vtc
index b26356fef..b88287a7a 100644
--- a/reg-tests/webstats/webstats-scope-and-post-change.vtc
+++ b/reg-tests/webstats/webstats-scope-and-post-change.vtc
@@ -1,5 +1,4 @@
 varnishtest "Webgui stats page check

[PATCH 2/2] REGTESTS: Remove REQUIRE_VERSION=1.7 from all tests

2021-06-11 Thread Tim Duesterhus 
HAProxy 1.7 is the lowest supported version, thus this always matches.
---
 reg-tests/http-rules/map_regm_with_backref.vtc | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reg-tests/http-rules/map_regm_with_backref.vtc 
b/reg-tests/http-rules/map_regm_with_backref.vtc
index 7a5b879d2..78af44721 100644
--- a/reg-tests/http-rules/map_regm_with_backref.vtc
+++ b/reg-tests/http-rules/map_regm_with_backref.vtc
@@ -10,7 +10,6 @@
 varnishtest "map_regm get_trash_chunk test"
 feature ignore_unknown_macro
 
-#REQUIRE_VERSION=1.7
 #REGTEST_TYPE=bug
 
 syslog S1 -level notice {
-- 
2.32.0

Weird behavior of spoe between http and https requests

2021-06-11 Thread Aleksandar Lazic 

Hi.

I use haproxy 2.4 with this fe config.

```
global
log stdout format raw daemon
daemon
maxconn 2
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd 
listeners
stats timeout 30s

tune.ssl.default-dh-param 2048

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private


# See 
https://ssl-config.mozilla.org/#server=haproxy=2.1=old=1.1.1d=5.4
ssl-default-bind-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
ssl-default-bind-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.0

ssl-default-server-ciphers 
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA
ssl-default-server-ciphersuites 
TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options no-tls-tickets ssl-min-ver TLSv1.0


defaults http
  log global
  mode http
  retry-on all-retryable-errors
  option forwardfor
  option redispatch
  option http-ignore-probes
  option httplog
  option dontlognull
  option log-health-checks
  option socket-stats
  timeout connect 5s
  timeout client  50s
  timeout server  50s
  http-reuse safe
  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

frontend http-in
  bind *:80
  mode http

  unique-id-format %rt
  http-request set-var(sess.my_fe_path) path
  http-request set-var(sess.my_fe_src) src
  http-request set-var(sess.my_fe_referer) req.hdr(Referer)
  http-request set-var(sess.my_fe_requestedhost) req.hdr(Host)

  # define the spoe agents
  filter spoe engine agent-on-http-req config /etc/haproxy/spoe-url.conf
  filter spoe engine agent-on-http-res config /etc/haproxy/spoe-url.conf

frontend https-in

  bind :::443 v4v6 alpn h2,http/1.1 ssl ca-file 
/etc/haproxy/letsencryptauthorityx3.pem crt /etc/ssl/haproxy/

  unique-id-format %rt
  http-request set-var(sess.my_fe_path) path
  http-request set-var(sess.my_fe_src) src
  http-request set-var(sess.my_fe_referer) req.hdr(Referer)
  http-request set-var(sess.my_fe_requestedhost) req.hdr(Host)

  # define the spoe agents
  filter spoe engine agent-on-http-req config /etc/haproxy/spoe-url.conf
  filter spoe engine agent-on-http-res config /etc/haproxy/spoe-url.conf
```

And with this spoe config.
```
[agent-on-http-req]
spoe-agent agent-on-http-req

log global

messages agent-on-http-req

option var-prefix feevents

timeout hello  2s
timeout idle   2m
timeout processing 1s

use-backend agent-on-http-req

spoe-message agent-on-http-req
args my_path=path my_src=src my_referer=req.hdr(Referer) my_sid=unique-id 
my_req_host=req.hdr(Host)
event on-frontend-http-request

[agent-on-http-res]
spoe-agent agent-on-http-res

log global

messages agent-on-http-res

option var-prefix feevents

timeout hello  2s
timeout idle   2m
timeout processing 1s

use-backend agent-on-http-res

spoe-message agent-on-http-res
args my_path=var(sess.my_fe_path) my_src=src 
my_referer=var(sess.my_fe_referer) my_sid=unique-id 
my_req_host=var(sess.my_fe_requestedhost)
event on-http-response
```

Now when I make a http request I get all values and args.
```
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Msg Name  
:agent-on-http-req:
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Msg Count :5:
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name  
:my_path:
Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value 
:/test:
Jun 11 16:01:01 reggata-001 spoe-url[112969]:

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
пт, 11 июн. 2021 г. в 20:34, William Lallemand :

> On Fri, Jun 11, 2021 at 08:14:49PM +0500, Илья Шипицин wrote:
> > I've found ubuntu musl package, so we can just link to it in CI, for
> > example (I'll try)
> >
>
>
> Well, that won't give you the same environnement as a docker image,
> with the same versions. I'll honestly prefer if we could do it with the
> alpine image.
>

there's always gap between "what is tested" and "what is packaged".
package maintainers add untested selinux patches and link against
various libs.

I agree with you that we should prefer to test on alpine natively.


>
> I you want to build with the musl-gcc wrapper you will need to link the
> linux headers in the musl headers directory otherwise it won't work the
> way their package is done.
>





>
> --
> William Lallemand
>

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand 
On Fri, Jun 11, 2021 at 08:14:49PM +0500, Илья Шипицин wrote:
> I've found ubuntu musl package, so we can just link to it in CI, for
> example (I'll try)
> 


Well, that won't give you the same environnement as a docker image,
with the same versions. I'll honestly prefer if we could do it with the
alpine image.

I you want to build with the musl-gcc wrapper you will need to link the
linux headers in the musl headers directory otherwise it won't work the
way their package is done.

-- 
William Lallemand

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
пт, 11 июн. 2021 г. в 20:18, Willy Tarreau :

> On Fri, Jun 11, 2021 at 08:14:49PM +0500,  ??? wrote:
> > @Willy Tarreau  , do you think it is good idea to display libc
> > variant in "haproxy -vv" ?
>
> If needed we can (for those that are detectable), but I'm not convinced
> of the benefits. If it's in order to exclude some tests, I'd rather
> merge this into new config predicates and use Tim's "-cc" instead. This
> will give us way more flexibility over the long term.
>


same as we display openssl variant in "haproxy -vv".
visibility only



>
> Cheers,
> Willy
>

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Willy Tarreau 
On Fri, Jun 11, 2021 at 08:14:49PM +0500,  ??? wrote:
> @Willy Tarreau  , do you think it is good idea to display libc
> variant in "haproxy -vv" ?

If needed we can (for those that are detectable), but I'm not convinced
of the benefits. If it's in order to exclude some tests, I'd rather
merge this into new config predicates and use Tim's "-cc" instead. This
will give us way more flexibility over the long term.

Cheers,
Willy

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
@Willy Tarreau  , do you think it is good idea to display libc
variant in "haproxy -vv" ?
(not sure actually whether musl represent itself in recognizable way)

I've found ubuntu musl package, so we can just link to it in CI, for
example (I'll try)


пт, 11 июн. 2021 г. в 20:03, Илья Шипицин :

>
>
> пт, 11 июн. 2021 г. в 19:43, William Lallemand :
>
>> On Fri, Jun 11, 2021 at 07:09:14PM +0500, Илья Шипицин wrote:
>> > I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
>> > afraid they will not stay for long time.
>> > using custom images in github actions is straightforward, have a look
>> >
>> > centos 6 · chipitsine/haproxy@20fabcd (github.com)
>> > <
>> https://github.com/chipitsine/haproxy/commit/20fabcd005dc9e3bac54a84bf44631f177fa79c2
>> >
>> >
>> >
>> > the same way you can specify either alpine or even special prepared
>> image.
>> > as I recall last time, we decided not to add alpine because "it is
>> tested
>> > anyway when docker images are created"
>> >
>>
>> As far as I know, it is only built, not tested. We encountered a few
>> problems that could have been caught before being built by Docker.
>> And that will prevent us to release a version that doesn't work
>> correctly with docker before they are built on the docker hub.
>>
>>
>> > also, there's small caveat, github actions runs agent inside docker
>> > container, it might have issues with older libc (or musl).
>> > but it worth a try
>> >
>>
>> Let's hope it works in this case.
>>
>
> ok, you made my weekend :)
> I will try tomorrow.
>
> also, I can contact github, no idea why they do not link they agents
> statically.
>
>
>>
>>
>> --
>> William Lallemand
>>
>

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
пт, 11 июн. 2021 г. в 19:43, William Lallemand :

> On Fri, Jun 11, 2021 at 07:09:14PM +0500, Илья Шипицин wrote:
> > I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
> > afraid they will not stay for long time.
> > using custom images in github actions is straightforward, have a look
> >
> > centos 6 · chipitsine/haproxy@20fabcd (github.com)
> > <
> https://github.com/chipitsine/haproxy/commit/20fabcd005dc9e3bac54a84bf44631f177fa79c2
> >
> >
> >
> > the same way you can specify either alpine or even special prepared
> image.
> > as I recall last time, we decided not to add alpine because "it is tested
> > anyway when docker images are created"
> >
>
> As far as I know, it is only built, not tested. We encountered a few
> problems that could have been caught before being built by Docker.
> And that will prevent us to release a version that doesn't work
> correctly with docker before they are built on the docker hub.
>
>
> > also, there's small caveat, github actions runs agent inside docker
> > container, it might have issues with older libc (or musl).
> > but it worth a try
> >
>
> Let's hope it works in this case.
>

ok, you made my weekend :)
I will try tomorrow.

also, I can contact github, no idea why they do not link they agents
statically.


>
>
> --
> William Lallemand
>

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand 
On Fri, Jun 11, 2021 at 07:09:14PM +0500, Илья Шипицин wrote:
> I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
> afraid they will not stay for long time.
> using custom images in github actions is straightforward, have a look
> 
> centos 6 · chipitsine/haproxy@20fabcd (github.com)
> 
> 
> 
> the same way you can specify either alpine or even special prepared image.
> as I recall last time, we decided not to add alpine because "it is tested
> anyway when docker images are created"
>

As far as I know, it is only built, not tested. We encountered a few
problems that could have been caught before being built by Docker.
And that will prevent us to release a version that doesn't work
correctly with docker before they are built on the docker hub. 


> also, there's small caveat, github actions runs agent inside docker
> container, it might have issues with older libc (or musl).
> but it worth a try
> 

Let's hope it works in this case.


-- 
William Lallemand

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand 
On Fri, Jun 11, 2021 at 07:19:51PM +0500, Илья Шипицин wrote:
> William, if you do not have a time, I can try to create github action based
> on your cirrus patch ... tomorrow ?
> 

I tried quickly but like Tim I couldn't make it work.

I can't spend much time on this, if you are able to make this work I
prefer to run it from github actions, otherwise we'll go with cirrus.

Thanks,

-- 
William Lallemand

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
William, if you do not have a time, I can try to create github action based
on your cirrus patch ... tomorrow ?

пт, 11 июн. 2021 г. в 19:09, Илья Шипицин :

> I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
> afraid they will not stay for long time.
> using custom images in github actions is straightforward, have a look
>
> centos 6 · chipitsine/haproxy@20fabcd (github.com)
> 
>
>
> the same way you can specify either alpine or even special prepared image.
> as I recall last time, we decided not to add alpine because "it is tested
> anyway when docker images are created"
>
> also, there's small caveat, github actions runs agent inside docker
> container, it might have issues with older libc (or musl).
> but it worth a try
>
>
> пт, 11 июн. 2021 г. в 19:02, William Lallemand :
>
>> This commit adds a CI job to cirrus-ci which builds HAProxy on Alpine
>> Linux, allowing to build and test HAProxy with musl.
>>
>> OpenSSL, PCRE2, Lua 5.3 as well as the prometheus exporter are enabled.
>>
>> GNU grep was purposely installed to run the reg-test script.
>> ---
>>  .cirrus.yml | 13 +
>>  1 file changed, 13 insertions(+)
>>
>> diff --git a/.cirrus.yml b/.cirrus.yml
>> index 9b83e6169..392a3abc5 100644
>> --- a/.cirrus.yml
>> +++ b/.cirrus.yml
>> @@ -11,3 +11,16 @@ FreeBSD_task:
>>  - ./haproxy -vv
>>  - ldd haproxy
>>  - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests
>> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
>> cat $folder/INFO $folder/LOG; done && exit 1)
>> +
>> +alpine_task:
>> +  container:
>> +image: alpine:latest
>> +  only_if: $CIRRUS_BRANCH =~ 'master|next'
>> +  script:
>> +- apk add gcc make tar git python3 libc-dev linux-headers pcre-dev
>> pcre2-dev openssl-dev lua5.3-dev grep socat curl
>> +- git clone https://github.com/VTest/VTest.git ../vtest
>> +- make -C ../vtest FLAGS="-O2 -s -Wall"
>> +- make CC=cc V=1 TARGET=linux-musl USE_LUA=1
>> LUA_INC=/usr/include/lua5.3 LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1
>> USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
>> +- ./haproxy -vv
>> +- ldd haproxy
>> +- env VTEST_PROGRAM=../vtest/vtest make reg-tests
>> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
>> cat $folder/INFO $folder/LOG; done && exit 1)
>> --
>> 2.17.1
>>
>>

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Илья Шипицин 
I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
afraid they will not stay for long time.
using custom images in github actions is straightforward, have a look

centos 6 · chipitsine/haproxy@20fabcd (github.com)



the same way you can specify either alpine or even special prepared image.
as I recall last time, we decided not to add alpine because "it is tested
anyway when docker images are created"

also, there's small caveat, github actions runs agent inside docker
container, it might have issues with older libc (or musl).
but it worth a try


пт, 11 июн. 2021 г. в 19:02, William Lallemand :

> This commit adds a CI job to cirrus-ci which builds HAProxy on Alpine
> Linux, allowing to build and test HAProxy with musl.
>
> OpenSSL, PCRE2, Lua 5.3 as well as the prometheus exporter are enabled.
>
> GNU grep was purposely installed to run the reg-test script.
> ---
>  .cirrus.yml | 13 +
>  1 file changed, 13 insertions(+)
>
> diff --git a/.cirrus.yml b/.cirrus.yml
> index 9b83e6169..392a3abc5 100644
> --- a/.cirrus.yml
> +++ b/.cirrus.yml
> @@ -11,3 +11,16 @@ FreeBSD_task:
>  - ./haproxy -vv
>  - ldd haproxy
>  - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests
> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
> cat $folder/INFO $folder/LOG; done && exit 1)
> +
> +alpine_task:
> +  container:
> +image: alpine:latest
> +  only_if: $CIRRUS_BRANCH =~ 'master|next'
> +  script:
> +- apk add gcc make tar git python3 libc-dev linux-headers pcre-dev
> pcre2-dev openssl-dev lua5.3-dev grep socat curl
> +- git clone https://github.com/VTest/VTest.git ../vtest
> +- make -C ../vtest FLAGS="-O2 -s -Wall"
> +- make CC=cc V=1 TARGET=linux-musl USE_LUA=1
> LUA_INC=/usr/include/lua5.3 LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1
> USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
> +- ./haproxy -vv
> +- ldd haproxy
> +- env VTEST_PROGRAM=../vtest/vtest make reg-tests
> REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do
> cat $folder/INFO $folder/LOG; done && exit 1)
> --
> 2.17.1
>
>

Re: add alpine linux to the CI

2021-06-11 Thread Tim Düsterhus 

William

On 6/11/21 4:01 PM, William Lallemand wrote:

I couldn't find a way to launch an alpine job easily with github actions so
instead I wrote one for cirrus-ci, It will help debugging Docker images
and musl problems.


I believe GitHub Actions also supports running using a Docker Image. I 
attempted to build something for Fedora Rawhide, but did not succeed 
either. We already use Cirrus, so if Cirrus works, then Cirrus it is.


Best regards
Tim Düsterhus

Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread Tim Düsterhus 

William,

On 6/11/21 4:01 PM, William Lallemand wrote:

This commit adds a CI job to cirrus-ci which builds HAProxy on Alpine
Linux, allowing to build and test HAProxy with musl.

OpenSSL, PCRE2, Lua 5.3 as well as the prometheus exporter are enabled.

GNU grep was purposely installed to run the reg-test script.


LGTM

Best regards
Tim Düsterhus

[PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand 
This commit adds a CI job to cirrus-ci which builds HAProxy on Alpine
Linux, allowing to build and test HAProxy with musl.

OpenSSL, PCRE2, Lua 5.3 as well as the prometheus exporter are enabled.

GNU grep was purposely installed to run the reg-test script.
---
 .cirrus.yml | 13 +
 1 file changed, 13 insertions(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index 9b83e6169..392a3abc5 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -11,3 +11,16 @@ FreeBSD_task:
 - ./haproxy -vv
 - ldd haproxy
 - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests 
REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do cat 
$folder/INFO $folder/LOG; done && exit 1)
+
+alpine_task:
+  container:
+image: alpine:latest
+  only_if: $CIRRUS_BRANCH =~ 'master|next'
+  script:
+- apk add gcc make tar git python3 libc-dev linux-headers pcre-dev 
pcre2-dev openssl-dev lua5.3-dev grep socat curl
+- git clone https://github.com/VTest/VTest.git ../vtest
+- make -C ../vtest FLAGS="-O2 -s -Wall"
+- make CC=cc V=1 TARGET=linux-musl USE_LUA=1 LUA_INC=/usr/include/lua5.3 
LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
+- ./haproxy -vv
+- ldd haproxy
+- env VTEST_PROGRAM=../vtest/vtest make reg-tests 
REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do cat 
$folder/INFO $folder/LOG; done && exit 1)
-- 
2.17.1

add alpine linux to the CI

2021-06-11 Thread William Lallemand 
Hello guys,

I couldn't find a way to launch an alpine job easily with github actions so
instead I wrote one for cirrus-ci, It will help debugging Docker images
and musl problems.

Example of the run here: https://cirrus-ci.com/task/5985082050609152

I'll push it in the master if that's fine with you.

Re: how to write to a file safely in haproxy

2021-06-11 Thread Willy Tarreau 
Hello,

On Fri, Jun 11, 2021 at 11:38:38AM +0530, reshma r wrote:
> Hello all,
> I Had a follow up query, sorry if it is an obvious question. once I
> implement the socket call as a sidecar process, say as a lua script that
> reads the new configuration from portal into a variable, how would I then
> update this variable within haproxy to be used by lua script loaded using
> lua_load directive? I understand we can do it using stats socket via CLI
> and socat but is there a way to do it from within lua script (the one
> running as sidecar process) itself so that I check and update portal config
> variable every 5 min automatically?

There are various ways. Your Lua script could periodically connect outside
to retrieve some data and set local variables for example. If you'd prefer
to control this from outside you could also write an HTTP service in Lua
that receives a certain URL after appropriate controls and extracts info
from query string, payload or headers, and sets variables as well. I don't
have much example to point you to, though.

Hoping this helps,
Willy

Re: how to write to a file safely in haproxy

2021-06-11 Thread reshma r 
Hello all,
I Had a follow up query, sorry if it is an obvious question. once I
implement the socket call as a sidecar process, say as a lua script that
reads the new configuration from portal into a variable, how would I then
update this variable within haproxy to be used by lua script loaded using
lua_load directive? I understand we can do it using stats socket via CLI
and socat but is there a way to do it from within lua script (the one
running as sidecar process) itself so that I check and update portal config
variable every 5 min automatically?


Thank you

On Thu, 27 May 2021, 17:37 reshma r,  wrote:

> Hi, thank you for the detailed and informative reply! It definitely helped
> clarify things.
> Indeed I had disabled chroot to read the files at runtime, but have since
> switched to read during init phase . Thank you for the tips on the
> architecture aspect as well. I will study and explore these options.
>
> Thanks,
> Reshma
>
> On Thu, May 27, 2021 at 11:50 AM Willy Tarreau  wrote:
>
>> Hi,
>>
>> On Wed, May 26, 2021 at 10:43:17PM +0530, reshma r wrote:
>> > Hi Tim thanks a lot for the reply. I am not familiar with what a sidecar
>> > process is. I will look into it. If it is specific to haproxy, if you
>> could
>> > point to some relevant documentation that would be helpful.
>>
>> It's not specific to haproxy, it's a general principle consisting in
>> having another process deal with certain tasks. See it as an assistant
>> if you want. We can do a parallel at lower layers so that it might be
>> clearer. Your kernel deals with routing tables, yet the kernel never
>> manipulates files by itself nor does it stop processing packets to
>> dump a routing table update into a file. Instead it's up to separate
>> processes to perform such slow tasks, and keep it in sync.
>>
>> > >I am making a
>> > > socket call which periodically checks whether the portal has been
>> changed
>> > > (from within haproxy action).
>> >
>> > Leaving aside the writing to file bit for a moment, is it otherwise
>> okay to
>> > do to the above within Haproxy alone and read the config fetched from
>> > portal into a global variable instead of saving to file? Or is it not an
>> > advisable solution? Actually this is what I am doing at present and I
>> have
>> > not observed any issues with performance...
>>
>> What you must never ever do is to read/write files at run time, as this
>> is extremely slow and will pause your traffic. It can be supported to
>> load a file during boot from Lua for example since at this point there
>> is no network processing in progress. We only slightly discourage from
>> doing so because most often the code starts by reading during init and
>> two months later it ends up being done at runtime (and people start to
>> disable chroots and permission drops in order to do this).
>>
>> It's possible to read/set process-wide variables from the CLI, so maybe
>> you can send some events there. Also as Tim mentioned, it's possible to
>> read/set maps from the CLI, that are also readable from Lua. That may
>> be another option to pass live info between an external process and your
>> haproxy config or Lua code. In fact nowadays plenty of people are
>> (ab)using maps to use them as dynamic routing tables or to store dynamic
>> thresholds. What is convenient with them is that they're loaded during
>> boot and you can feed the whole file over the CLI at runtime to pass
>> updates. Maybe that can match your needs.
>>
>> Last point, as a general architecture rule, as soon as you're using
>> multiple components (portal, LB, agents, etc), it's critically important
>> to define who's authoritary over the other ones. Once you do that you
>> need to make sure that the other ones can be sacrified and restarted.
>> In your case I suspect the authority is the portal and that the rest
>> can be killed and restarted at any time. This means that the trust you
>> put in such components must always be lower than the trust you put in
>> the authority (portal I presume).
>>
>> Thus these components must not play games like dumping files by
>> themselves.
>> In the best case they could be consulted to retrieve a current state
>> to be reused in case of a reload. But your portal should be the one
>> imposing its desired state on others. For example, imagine you face a
>> bug, a crash, an out-of-memory or whatever situation where your haproxy
>> dies in the middle of a dump to this file. Your file is ruined and you
>> cannot reuse it. Possibly you can't even restart the service anymore
>> because your corrupted file causes startup errors.
>>
>> This means you'll necessarily have to make sure that a fresh new copy
>> can be instantly delivered by the portal just to cover this unlikely
>> case. If you implement this capability in your portal, then it should
>> become the standard way to produce that file (you don't want the file
>> to come from two different sources, do you?). Then you can simply have
>> a sidecar process dedicated to