Re: Reproducible CI build with OpenSSL and "latest" keyword

2022-12-06 Thread William Lallemand
Tim,

On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote:
> > What I suggest is to stop using "latest" for the "git push" CI, but
> > using it only in a separate CI (once a day/week I don't know). And only
> > use fixed version of the libraries on the CI so builds are not broken by
> > external components. Because in my opinion the "git push" CI is to test
> > our code, not the libraries.
> > 
> 
> I don't even think such a weekly job is necessary [1].

> Add an item to the release checklist "check if any new SSL versions
> are available and add them to matrix.py" and this should be fine, all
> SSL versions will then be updated every 6 months and can also be
> updated on demand for important releases. 

Well, I don't want to see the CI fail just for testing this, having the
weekly one gives you the status before integration and is also a
reminder.

> It's similar to how I simply rerun the Coccinelle 
> patches from time to time to fix whatever crept in since the last release.
> 

I disagree, porting to a new API is not something you would do just
before a release, you need to do it progressively if possible, because
it could introduce heavy development and sometimes discussions with the
library developers and unfortunately that could take time.

That would be too bad to postpone support for a new version because
nobody looked at this during the development cycle, and the changes are
too heavy to be integrated.


-- 
William Lallemand



Re: Reproducible CI build with OpenSSL and "latest" keyword

2022-12-06 Thread William Lallemand
On Tue, Dec 06, 2022 at 07:54:33PM +0500, Илья Шипицин wrote:
> I recall I even promised to do something, but I did not :-)
> 
> automatically determine "which is latest 3.0.x" does not make much sense,
> it is stable branch, very conservative. we can stick to 3.0.7, for example.
> I do not expect any breaking change between 3.0.7 and 3.0.8

I recall a discussion like this indeed, couldn't find the previous
thread. There was cases of broken minor release for LibreSSL for
example, so it's better to stick to a release for the push CI  and
upgrade once the weekly CI passed.

> we can move "latest" to weekly, np. as for stable branches CI, I think them
> do not have to follow the same rules as development branch, we can have
> different matrix for stable and development.

I think we could it this way:

- weekly "latest" for libreSSL, OpenSSL, whateverSSL for HAProxy master
  branch
- git push CI with fixed version in HAProxy master 


> I think I got the idea.
> looks like you use the same github actions for stable branches.
>

Indeed, at some point the master branch become a stable branch, so we
inherit all of this.

> either I will manage to make them different or I will stick to
> 3.0.something.  hopefully tomorrow

IMHO it should never be "latest" anywhere else than the weekly, we don't
want to be disturbed by this when pushing our code. 

I don't think we need a weekly for the stable branch, so it could be
conditionned by a check on the version, for example only started if
there is '-dev' in the version.

So we should probably:

- Revert "latest" to "3.0.7" in the master, and backport the patch in the
previous supported branches. (as far as 2.4 I think)

- Introduce "3.1.0-alpha1" to master

- Introduce "latest" to weekly master

-- 
William Lallemand



Reproducible CI build with OpenSSL and "latest" keyword

2022-12-06 Thread William Lallemand
Hello,

We are experiencing difficulties with the way the CI matrix is
generated with the SSL libraries.

As I already mentionned, I don't really like the "latest" keyword for
the OpenSSL version as it prevent us to have reproducible builds.
It updates versions without warning, even major ones.

Since OpenSSL 3.1.0-aplha1 was released we are affected by the problem,
we stopped building with 3.0.x without noticing, and our internal CI for
the stable branches start failing because of that. Majour versions must
never change in the previous branches.

What I suggest is to stop using "latest" for the "git push" CI, but
using it only in a separate CI (once a day/week I don't know). And only
use fixed version of the libraries on the CI so builds are not broken by
external components. Because in my opinion the "git push" CI is to test
our code, not the libraries.

What do you guys think?

-- 
William Lallemand



Re: Certificate picking order

2022-11-21 Thread William Lallemand
Hi,

On Sat, Nov 19, 2022 at 05:45:43PM +0100, William Edwards wrote:
> Hello,
> 
> When multiple SSL certificates exist for a given domain, which one does 
> HAProxy pick?
> 
> I'm specifically interested in knowing what happens when:
> 
> - Multiple certificates with the exact same set of common names exist

The SNI and CN are registered in a tree and the match will depend on the
declaration order.

But using this order is not convenient in some cases, instead you could
use the crt-list keyword that let you redefine which certificate should
serve which SNI/CN.

> - A more specific certificate exists, e.g. one wildcard certificate 
> (*.example.com) and one covering only a subdomain (foo.example.com)

It first look for an exact match, then look for a wildcard.
In case of a crt-list it also look for a negative entry.
For example your could have in your crt-list:

cert1.pem *.example.com !foo2.example.com !foo3.example.com
cert2.pem foo2.example.com
cert3.pem foo3.example.com

> ... and if the order of .pem files matters in a `crt` directory.
> 

They are registered in alphabetical order so it does. 


> I am unable to find this in the documentation. But I'm pretty sure I've 
> seen it in there before...
> 

Regards,

-- 
William Lallemand



Re: [PATCH] fix spelling "choosen" --> "chosen"

2022-11-02 Thread William Lallemand
On Tue, Nov 01, 2022 at 12:16:16PM +0100, Willy Tarreau wrote:
> Hi Ilya,
> 
> On Tue, Nov 01, 2022 at 03:49:18PM +0500,  ??? wrote:
> > Hello,
> > 
> > I'm not sure how good is idea to fix variable names.
> > if we want to keep as is, I'd setup spelling exclusion.
> 
> Interesting. I'm CCing the relevant maintainers, they're the best placed
> to know if they're willing to fix the code now (hint: better sooner than
> later :-)).
> 
> Note before applying the patch, one "choosen" in a comment mistakenly
> ended up as "chooen" instead of "chosen", so let's fix it before applying.
> 
> Thanks!
> Willy
> 
> > -   if (!tp->choosen)
> > +   if (!tp->chosen)
> > return;
> >  
> > -   chunk_appendf(b, "\n\tversion_information:(choosen=0x%08x", 
> > tp->choosen);
> > +   chunk_appendf(b, "\n\tversion_information:(chosen=0x%08x", tp->coosen);
 

I don't think it will even compile this way. 
> 

-- 
William Lallemand



Re: [PR] prelim-wolfSSL updates

2022-10-25 Thread William Lallemand
Thanks for the pull request, we are going to take a look and test it.

On Sun, Oct 23, 2022 at 12:23:02AM +0200, PR Bot wrote:
> Dear list!
> 
> Author: Uriah Pollock 
> Number of patches: 1
> 
> This is an automated relay of the Github pull request:
>prelim-wolfSSL updates
> 
> Patch title(s): 
>prelim-wolfSSL updates
> 
> Link:
>https://github.com/haproxy/haproxy/pull/1908
> 
> Edit locally:
>wget https://github.com/haproxy/haproxy/pull/1908.patch && vi 1908.patch
> 
> Apply locally:
>curl https://github.com/haproxy/haproxy/pull/1908.patch | git am -
> 
> Description:
> 
> 
> Instructions:
>This github pull request will be closed automatically; patch should be
>reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is
>invited to comment, even the patch's author. Please keep the author and
>list CCed in replies. Please note that in absence of any response this
>pull request will be lost.
> 

-- 
William Lallemand



Re: coredump and traceback on the CI

2022-10-20 Thread William Lallemand
On Thu, Oct 20, 2022 at 10:51:23PM +0500, Илья Шипицин wrote:
> I would suggest to display vtest result failure only if vtest failed,
> haproxy/vtest.yml at master · haproxy/haproxy (github.com)
> <https://github.com/haproxy/haproxy/blob/master/.github/workflows/vtest.yml#L139>
> 
> I doubt if there could be coredump together with successful vtest 
>

Indeed that make sense, I'll do that.

> just curious, was it only reproduced on CI ? not reproduced locally on
> alpine ?
> 

I tested on a local container but it didn't reproduce all the parameters
to make it fail, my alpine version wasn't the right one in my container,
even if I used :latest it wasn't the same, and the number of threads
wasn't right either.

Once I had the coredump I identified what was missing but I spend some
time trying to change multiple parameters.

-- 
William Lallemand



coredump and traceback on the CI

2022-10-20 Thread William Lallemand
Hello List,

I had some difficulties today to reproduce a bug which was only
visible on the alpine container CI. So I found some way to investigate
directly from the CI.

The attached patch enables the coredumps and does a "gdb -ex 'thread
apply all bt full'" for the alpine/musl job.

I think this will help to debug a lot, I know there is also the ability
to get an artefact with the coredump, which could also be interesting,
but having the traceback on the CI page is easy.

If no one complain, I'll push the patch.

Cheers,

-- 
William Lallemand
>From 4465fe8c77aa2ce664ebfc46b41d20c4440d10cc Mon Sep 17 00:00:00 2001
From: William Lallemand 
Date: Thu, 20 Oct 2022 15:01:01 +0200
Subject: [PATCH] CI: github: dump the backtrace of coredumps in the alpine
 container

This patch allows to show the backtrace of a coredump produced in the
alpine/musl jobs.

It activates some option required by the containers to allow the
production of coredump, set a shared directory so the kernel could dump
the coredump within the container. Some debug packages were also added.
---
 .github/workflows/musl.yml | 21 +++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/musl.yml b/.github/workflows/musl.yml
index 5a6b46a7b..ff1a0f780 100644
--- a/.github/workflows/musl.yml
+++ b/.github/workflows/musl.yml
@@ -12,14 +12,21 @@ jobs:
   runs-on: ubuntu-latest
   container:
 image: alpine:latest
+options: --privileged --ulimit core=-1 --security-opt seccomp=unconfined
+volumes:
+  - /tmp/core:/tmp/core
   steps:
+  - name: coredump setup
+run: |
+  ulimit -c unlimited
+  echo '/tmp/core/core.%h.%e.%t' > /proc/sys/kernel/core_pattern
   - uses: actions/checkout@v3
   - name: Install dependencies
-run: apk add gcc make tar git python3 libc-dev linux-headers pcre-dev pcre2-dev openssl-dev lua5.3-dev grep socat curl
+run: apk add gcc gdb make tar git python3 libc-dev linux-headers pcre-dev pcre2-dev openssl-dev lua5.3-dev grep socat curl musl-dbg lua5.3-dbg
   - name: Install VTest
 run: scripts/build-vtest.sh
   - name: Build
-run: make -j$(nproc) CC=cc V=1 TARGET=linux-musl USE_LUA=1 LUA_INC=/usr/include/lua5.3 LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
+run: make -j$(nproc) TARGET=linux-musl DEBUG_CFLAGS='-ggdb3' CC=cc V=1 USE_LUA=1 LUA_INC=/usr/include/lua5.3 LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
   - name: Show version
 run: ./haproxy -vv
   - name: Show linked libraries
@@ -30,6 +37,15 @@ jobs:
   - name: Run VTest
 id: vtest
 run: make reg-tests VTEST_PROGRAM=../vtest/vtest REGTESTS_TYPES=default,bug,devel
+  - name: Show coredumps
+if: ${{ failure() }}
+run: |
+  ls /tmp/core/
+  for file in /tmp/core/core.*; do
+printf "::group::"
+gdb -ex 'thread apply all bt full' ./haproxy $file
+echo "::endgroup::"
+  done
   - name: Show results
 if: ${{ failure() }}
 run: |
@@ -40,3 +56,4 @@ jobs:
 echo "::endgroup::"
   done
   shopt -s nullglob
+
-- 
2.34.1



Re: [PATCH] CI: use proper version generating when {OPENSSL,LIBRESSL}_VERSION=latest semantic is used

2022-10-18 Thread William Lallemand
On Tue, Oct 18, 2022 at 03:10:07PM +0500, Илья Шипицин wrote:
> > Sorry I didn't see the first commit that introduced this behavior.  I'm
> > not sure we would want to replace the version automatically in the CI
> > for OpenSSL.
> >
> 
> it was supposed behaviour for OPENSSL_VERSION=latest
> 

Yes, that's exactly why I said that I missed the introduction of that
change :-)

> > Currently we are testing the stock OpenSSL of the ubuntu which is 1.1.1,
> > the 1.0.2u and the "latest". The latest is currently a 3.0.x but once
> > the 3.1.x is released we would still need the 3.0 branch.
> >
> 
> I think we should review our approach to "stock" and "latest" when
> ubuntu-latest
> will be 22.04 (it is shipped with 3.0.X)
> 
> I'll have a look at your point as well


Sure, that's not urgent, we will need to add a 1.1.1 separately in this
case.

> > I think we need something to test the latest release of a branch, and
> > not the latest version of all branches. Maybe we could specify "3.0.x"
> > to get the latest 3.0?
> >
> 
> maybe-maybe-maybe.
> 
> we can introduce "latest in 3.0.x" I guess. not much to code.


I think that's the most logical behavior we need for testing HAProxy
without breaking the CI because of a major change in the library.
What we need is to test our code for each of our commit, if the CI break
on our commit because we changed the lib version it's difficult to know
where the issue is.

Maybe we could have a separated CI job with the "latest" version, so we ensure
that this is running correctly before integrating the version to the
"per-commit" CI jobs. What do you think about that? It seems like a good
compromise to me.


-- 
William Lallemand



Re: [PATCH] CI: use proper version generating when {OPENSSL,LIBRESSL}_VERSION=latest semantic is used

2022-10-18 Thread William Lallemand
On Thu, Oct 13, 2022 at 08:54:38AM +0200, Willy Tarreau wrote:
> Hi Ilya,
> 
> On Tue, Oct 11, 2022 at 12:18:40PM +0500,  ??? wrote:
> > split patches attached.
> 
> Sorry for the delay. Both applied now, thank you!
> Willy
> 

Hello,

Sorry I didn't see the first commit that introduced this behavior.  I'm
not sure we would want to replace the version automatically in the CI
for OpenSSL.

Currently we are testing the stock OpenSSL of the ubuntu which is 1.1.1,
the 1.0.2u and the "latest". The latest is currently a 3.0.x but once
the 3.1.x is released we would still need the 3.0 branch.

I think we need something to test the latest release of a branch, and
not the latest version of all branches. Maybe we could specify "3.0.x"
to get the latest 3.0?

Regards,

-- 
William Lallemand



Re: LibreSSL 3.6.0 QUIC support with HAProxy 2.7

2022-10-06 Thread William Lallemand
On Thu, Oct 06, 2022 at 08:46:08AM +0500, Илья Шипицин wrote:
> libressl-3.6.0  was released yesterday
> 
> [image: image.png]
> 
> 
> hopefully, github pipeline will pick it on the next build (it tries to pick
> latest available).

I'm confused, the CI is switching major branches automatically?


> we can modify github pipeline to use quic for libressl builds
> 

I think that's a good idea, indeed.



-- 
William Lallemand



Re: LibreSSL 3.6.0 QUIC support with HAProxy 2.7

2022-09-15 Thread William Lallemand
On Thu, Sep 15, 2022 at 01:06:25AM +0200, Aleksandar Lazic wrote:
> Hi William.
>
> [...]
> How about to change this to something like
> 
> Built with SSL Library version
> Running on SSL Library version
> SSL library supports ...
> 
> Because it's confusing :-)
> 
> Built with OpenSSL version : LibreSSL 3.6.0
> 
> I thought also something like
> 
> Built with (OpenSSL|LibreSSL) version : LibreSSL 3.6.0
> 
> But this looks ugly to me.
> 
> 

I get your point, but this is still a library from the OpenSSL family, a
fork which uses most of the OpenSSL API, you still have to build with
USE_OPENSSL=1. It's the same for OpenSSL, LibreSSL, quicTLS, BoringSSL.

At some point if we add a whole new API, for example gnuTLS or wolfssl,
this would be a whole new API, and we would have to rename the defines
and probably this line in haproxy -vv.

-- 
William Lallemand



LibreSSL 3.6.0 QUIC support with HAProxy 2.7

2022-09-14 Thread William Lallemand
Hello List,

We've just finished the portage of HAProxy for the next libreSSL
version which implements the quicTLS API.

For those interested this is how you are supposed to compile everything:

The libreSSL library:

$ git clone https://github.com/libressl-portable/portable libressl
$ cd libressl
$ ./autogen.sh

// The QUIC API is not public and not available in the shared
// library for now, you have to link with the .a
$ ./configure --prefix=/opt/libressl-quic/ --disable-shared 
CFLAGS=-DLIBRESSL_HAS_QUIC
$ make V=1
$ sudo make install

HAProxy:

$ git clone http://git.haproxy.org/git/haproxy.git/
$ cd haproxy
$ make TARGET=linux-glibc USE_OPENSSL=1 USE_QUIC=1 
SSL_INC=/opt/libressl-quic/include/ \
   SSL_LIB=/opt/libressl-quic/lib/ DEFINE='-DLIBRESSL_HAS_QUIC'


$ ./haproxy -vv
HAProxy version 2.7-dev5-7eeef9-91 2022/09/14 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-47-generic #51-Ubuntu SMP Thu Aug 11 07:51:15 
UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = cc
  CFLAGS  = -O2 -ggdb3 -Wall -Wextra -Wundef 
-Wdeclaration-after-statement -Wfatal-errors -Wtype-limits 
-Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference 
-fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare 
-Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers 
-Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment 
-DLIBRESSL_HAS_QUIC
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1 
USE_QUIC=1
  DEBUG   = -DDEBUG_MEMORY_POOLS -DDEBUG_STRICT

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE -PCRE_JIT -PCRE2 
-PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE 
-STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE 
+GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY +TFO 
+NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL 
-PROCCTL +THREAD_DUMP -EVPORTS -OT +QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, 
default=8).
Built with OpenSSL version : LibreSSL 3.6.0
Running on OpenSSL version : LibreSSL 3.6.0
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.3
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), 
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT 
IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.2.0

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' 
keyword)
   quic : mode=HTTP  side=FE mux=QUIC  flags=HTX|NO_UPG|FRAMED
 h2 : mode=HTTP  side=FE|BE  mux=H2flags=HTX|HOL_RISK|NO_UPG
   fcgi : mode=HTTP  side=BE mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
   : mode=HTTP  side=FE|BE  mux=H1flags=HTX
 h1 : mode=HTTP  side=FE|BE  mux=H1flags=HTX|NO_UPG
   : mode=TCP   side=FE|BE  mux=PASS  flags=
   none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
    [TRACE] trace



Regards,


-- 
William Lallemand



Re: most probably next LibreSSL release will come with ... QUIC

2022-08-31 Thread William Lallemand
On Mon, Aug 29, 2022 at 11:20:29PM +0500, Илья Шипицин wrote:
> Hello,
> 
> Provide the remaining QUIC API. · libressl-portable/openbsd@635aa39
> (github.com)
> <https://github.com/libressl-portable/openbsd/commit/635aa39dc9ac8ec2d6b57b6c789c75646a55a5da>
> 
> 

That's good to read! It didn't make it to libressl-portable for now but
we will definitively try it once it's available.
-- 
William Lallemand



Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

2022-08-23 Thread William Lallemand
On Mon, Aug 22, 2022 at 04:33:51PM +, Bren wrote:
> --- Original Message ---
> On Monday, August 22nd, 2022 at 7:03 AM, William Lallemand 
>  wrote:
> 
> > I'm going to issue a 2.6.4 today with the patch.
> 
> Just rolled out 2.6.4 and everything appears to be working as expected now. 
> Thanks!
> 

Great! Thanks for the feedback!

-- 
William Lallemand



[ANNOUNCE] haproxy-2.6.4

2022-08-22 Thread William Lallemand
Hi,

HAProxy 2.6.4 was released on 2022/08/22. It added 2 new commits
after version 2.6.3.

This version was quickly released after the 2.6.3 because it introduced a
regression which could cause an infinite loop.

When in mworker mode, and without any master CLI configured (-S), after forking
the workers, and re-executing itself, the master will try to iterate the
proxies list but this one is empty, causing a infinite loop. The master is left
in a unusable state.

With systemd, the process is killed since the READY=1 signal is never sent.

This problem does not occur with the default systemd service which provides the
-S argument.


Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Documentation: http://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : http://www.haproxy.org/download/2.6/src/
   Git repository   : http://git.haproxy.org/git/haproxy-2.6.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-2.6.git
   Changelog: http://www.haproxy.org/download/2.6/src/CHANGELOG
   Pending bugs : http://www.haproxy.org/l/pending-bugs
   Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
   Code reports : http://www.haproxy.org/l/code-reports
   Latest builds: http://www.haproxy.org/l/dev-packages


---
Complete changelog :
Emeric Brun (1):
  BUG/MAJOR: mworker: fix infinite loop on master with no proxies.

William Lallemand (1):
  BUG/MINOR: ssl/cli: error when the ca-file is empty

---

-- 
William Lallemand



Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

2022-08-22 Thread William Lallemand
On Sat, Aug 20, 2022 at 09:56:40PM +0200, Willy Tarreau wrote:
> On Sat, Aug 20, 2022 at 09:36:21PM +0200, Ionel GARDAIS wrote:
> > That was it :
> > - remove the EXTRAOPTS from /etc/default/haproxy
> > - stop the running process referencing -x /run/haproxy/admin.sock on the CLI
> > - upgrade
> > 
> > All is OK.
> > First processes do not list -x on the CLI and a reload spawn a process with 
> > -x sockpair@
> 
> Very interesting, thank you Ionel.
> 
> There is this fix in 2.6.3 which seems in part related to this:
> 
>   commit ae1e5a8c72ced4c9527f05bd5b14916d1b9d92c8
>   Author: William Lallemand 
>   Date:   Mon Jul 25 15:51:30 2022 +0200
> 
> BUG/MINOR: sockpair: wrong return value for fd_send_uxst()
> 
> The fd_send_uxst() function which is used to send a socket over the
> socketpair returns 1 upon error instead of -1, which means the error
> case of the sendmsg() is never catched correctly.
> 
> Must be backported as far as 1.9.
> 
> (cherry picked from commit f67e8fb92c795808f60b2406ae395ebc0ca180c5)
> Signed-off-by: Christopher Faulet 
> 
> However I'm confused by that part in the code because it seems to try
> to send FDs in the function that's supposed to retrieve them, so I'm
> missing something related to the context where this is used. But if
> the caller was not able to properly handle an error, it could possibly
> explain (at least in part) why it failed for you after the upgrade when
> trying to connect if there's an issue with this socket. I'll have to
> check with William on Monday to clarify all this and try to reproduce. 
> 
> Thanks,
> Willy

It looked like we found the culprit, the patch 3b68b60 ("BUG/MAJOR: log-forward:
Fix log-forward proxies not fully initialized") was not handling the
case where no proxies are in the list.

In master-worker mode, the process is exec() again without any
configuration in order to free the memory, when -S is used to create a
master CLI the list is not empty and everything is good... however
without the -S argument, the list is empty and the master will
indefinitely loop.

So the workaround for this bug is to keep the -S in EXTRAOPTS... which
is the default in haproxy.service. If you rewrote EXTRAOPTS in
/etc/default/haproxy, don't forgot to put it there.

Emeric provided a new patch to fix the problem,
https://github.com/haproxy/haproxy/commit/8032a276ce8007020366d18ebd7400ad5dedc4f4

I'm going to issue a 2.6.4 today with the patch.

-- 
William Lallemand



Re: [*EXT*] [ANNOUNCE] haproxy-2.6.3

2022-08-22 Thread William Lallemand
On Sat, Aug 20, 2022 at 08:50:25PM +, Bren wrote:
> --- Original Message ---
> On Saturday, August 20th, 2022 at 3:43 PM, Vincent Bernat  
> wrote:
> 
> > Do you have something here too?
> 
> Nope. In fact I just removed that from the build.
> 
> > This does not match the file shipped by HAProxy, but this may explain
> > why you also run into this bug.
> 
> What ships with the source is:
> 
> Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" 
> "EXTRAOPTS=-S /run/haproxy-master.sock"
> 
> I'm using the config for this:
> 
> stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 
> level admin
> 
> So I probably removed that last part.

Hello Bren,

Do you have any error into your logs at startup? Could you share them?

Thanks.

-- 
William Lallemand



Re: Buffer limits when adding a large number of CA certs into one ca-file via socket

2022-08-16 Thread William Lallemand
On Tue, Aug 16, 2022 at 11:16:43AM +, Lais, Alexander wrote:
> Hi William,
> 
> Thank you! I figured you were on holidays. A lot of our team are as well.
> 
> Do you see this being back ported to 2.5 / 2.6 (LTS) as well?

Unfortunately we usually don't backport this kind of features as this is
an API change and could break things.
The stable branches are meant to be maintenance only.

Also it will probably need some adjustments and new keywords to remove a
specific index in the file and that kind of things.

-- 
William Lallemand



Re: Buffer limits when adding a large number of CA certs into one ca-file via socket

2022-08-16 Thread William Lallemand
On Thu, Aug 04, 2022 at 11:57:16AM +, Lais, Alexander wrote:
> Hi William,
> 
> thanks again for the PoC you referenced in the GitHub issue.
> This would solve the use case for us and would fix the ca-cert editing / 
> updating feature introduced in HAProxy 2.5.
> 
> Can we support further with the development, be it with code or testing, to 
> get from this PoC to a full fix in one of next release streams?
> 
> Thanks and kind regards,
> Alex
> 
Hello Alex,

Sorry for the late reply, I was in vacation for a few days. 

I'm going to finish the development and tests for the feature so this
could be integrated for the next 2.7 major version.

Regards,

-- 
William Lallemand



Re: Buffer limits when adding a large number of CA certs into one ca-file via socket

2022-07-29 Thread William Lallemand



On Tue, Jul 26, 2022 at 03:04:41PM +, Lais, Alexander wrote:
> Dear all,
> 
> We are now using the new feature of adding CA files dynamically via the stats 
> / admin socket.
> 
> Assuming that the CA file does not exist yet, our understanding is that we:
> 
> 1. Create a CA file (new ssl ca-file customer-cas.pem)
> 
> 2. Set the content of the CA file with payload notation;
> "set ssl ca-file customer-cas.pem <<\n[a bunch of PEM blocks]\n”
> 
> 3. Commit the CA file (commit ssl ca-file customer-cas.pem)
> 
> In step 2 we are reaching the limit of the global buffer size (defined via 
> tune.bufsize, ours is tuned to ca. 71k, allowing for a comfortable 64k of 
> headers).
> Some of the CA files that we want to add are larger than this buffer and are 
> not properly processed by the CLI.
> 
> It is understandable that the CLI socket needs some buffer and that this 
> buffer is limited.
> That said, reading the CA files data from disk does not pose any 
> (perceivable) size limit. We recently implemented a dynamic update to avoid 
> having to reload the HAProxy process whenever there was a change, and ran 
> into this issue.
> 
> We’ve added a feature request on GitHub: 
> https://github.com/haproxy/haproxy/issues/1805
> 
> This e-mail is to ask whether maybe we have overlooked something in terms of 
> configuration possibilities, either for the socket or on how to use the CLI 
> for creating ca-files?
> 

You are indeed reaching a limitation of the current system, I'll reply
directly on your feature request.

Thanks,

-- 
William Lallemand



Re: "Success" logs in HTTP frontends

2022-07-29 Thread William Lallemand
On Fri, Jul 29, 2022 at 11:10:32AM +0200, Christopher Faulet wrote:
> Le 7/29/22 à 10:13, Christian Ruppert a écrit :
> > Hi list,
> > 
> > so I noticed on my private HAProxy I have 2 of those logs within the
> > past ~1-2 months:
> > haproxy[28669]: 1.2.3.4:48596 [17/Jun/2022:13:55:18.530] public/HTTPSv4:
> > Success
> > 
> > So that's nothing so far but still no idea what that means.
> > At work, of 250 mio log entries per day, there are about 600k of those
> > "Success" ones.
> > haproxy[27892]: 192.168.70.102:7904 [29/May/2022:00:13:37.316]
> > genfrontend_35310-foobar/3: Success
> > 
> > I'm not sure what it means by "3". Is it the third bind?
> > 
> > I couldn't trigger those "Success" logs by either restarting or
> > reloading. What is it for / where does it come from?
> > 
> 
> Hi Christian,
> 
> What is your version ? At first glance, I can't find such log message in the 
> code. It could come from a lua module.
> 
> In fact, I found something. It is probably because an "embryonic" session is 
> killed with no connection/ssl error. For instance, an SSL connection rejected 
> because of a "tcp-request session" rule (so after the SSL handshake). The 
> same 
> may happen with a listener using the PROXY protocol.
> 
> Regards,


Could be something like that indeed, the "Success" message is the string
for CO_ER_NONE in the fc_err_str fetch. (The default error string)

Maybe we lack some intermediate state, or we could just change the
string ?

It is only the string for the handshake status so this is confusing when
used as an error.

-- 
William Lallemand



Re: Thoughts on QUIC/HTTP3

2022-07-09 Thread William Lallemand
On Fri, Jul 08, 2022 at 09:11:02AM -0600, Shawn Heisey wrote:
> 
> The openssl that haproxy is compiled against is in /opt/quictls/ssl ... 
> but there is a distribution-provided openssl package in /usr/lib/ssl as 
> well.  Both locations contain "certs".
> 

But is there any certificates in the /opt/quictls/ssl/certs/ directory ?

> Setting either environment variable that you have mentioned does not 
> eliminate the warning.

It should already be set to /opt/quictls/ssl/certs/, if you specified
the openssldir at /opt/quictls/ssl/ during the build of your library.

> root@bilbo:~# SSL_CERT_DIR=/opt/quictls/ssl/certs haproxy -c -f 
> /etc/haproxy/haproxy.cfg
> [NOTICE]   (2379692) : haproxy version is 2.6.1
> [NOTICE]   (2379692) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2379692) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> root@bilbo:~# OPENSSLDIR=/opt/quictls/ssl haproxy -c -f 
> /etc/haproxy/haproxy.cfg
> [NOTICE]   (2379701) : haproxy version is 2.6.1
> [NOTICE]   (2379701) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2379701) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> 
> My setup has no need to verify certificates, so the warning doesn't 
> actually matter for me.  But it could be a problem for someone else.
> 

In fact there is a warning because you might want to use the httpclient
at runtime, and the httpclient is using the CAs, so they are loaded at
startup.

I supposed you don't have anything in this directory, or it failed to
load for some reason and this could be a bug we need to fix.

The message should have been about the httpclient, I'll look into this
to clarify it.

> I did figure out the correct way to run the "version -d" command you 
> mentioned on the quictls install:
> 
> elyograg@smeagol:~$ LD_LIBRARY_PATH=/opt/quictls/lib64 
> /opt/quictls/bin/openssl version -d
> OPENSSLDIR: "/opt/quictls/ssl"
 
You had to use LD_LIBRARY_PATH because you didn't use the rpath when
compiling, this is necessary if you don't install the library in
/usr/lib/.
You only need to add -Wl,-rpath=/opt/quictls/lib64 to your ./config line.
https://wiki.openssl.org/index.php/Compilation_and_Installation#Using_RPATHs


> My install does quic/http3 correctly, so I know it is finding and using 
> quictls.
> 

Ok, you can always check with ldd if you have some doubts.

-- 
William Lallemand



Re: Thoughts on QUIC/HTTP3

2022-07-08 Thread William Lallemand
On Thu, Jul 07, 2022 at 07:53:24AM -0600, Shawn Heisey wrote:
> On 7/6/22 09:50, Илья Шипицин wrote:
> > haproxy is built in CI against latest quictls, for example quictls-3.0.5
> >
> > https://github.com/haproxy/haproxy/runs/721404?check_suite_focus=true
> >
> > please open an issue on github with failure details, no known build 
> > failures so far
> 
> Shortly after I saw this message, I tried the build again.  My script 
> does "git pull" on the repo.  There were a bunch of updates to the 
> quictls repo, and now haproxy compiles and runs.
> 
> I am getting a new config warning, though:
> 
> elyograg@bilbo:/usr/local/src$ sudo haproxy -c -f /etc/haproxy/haproxy.cfg
> [NOTICE]   (2080586) : haproxy version is 2.6.1
> [NOTICE]   (2080586) : path to executable is /usr/local/sbin/haproxy
> [WARNING]  (2080586) : config : ca-file: 0 CA were loaded from '@system-ca'
> Warnings were found.
> Configuration file is valid
> 
 
HAProxy uses the ca-certificates provided by OpenSSL.
The SSL_CERT_DIR by default is set to the "certs" directory inside your
openssldir. You can check your openssldir by using the "openssl" binary
you compiled with your library (not the one of your distribution).

  $ openssl version -d
  OPENSSLDIR: "/usr/lib/ssl"

So you might want to set the SSL_CERT_DIR environment variable before
starting HAProxy or doing a symlink from your openssldir to the real
path of your ca-certificates ( /etc/ssl/certs ? )

This warning is emitted when trying to load the ca-certificates into the
httpclient at startup with an empty directory. (Which is not supposed to
happen on the openssl build of your distribution)

-- 
William Lallemand



Re: running SECLEVEL=2 for OpenSSL-3.0 tests ?

2022-07-05 Thread William Lallemand
On Tue, Jul 05, 2022 at 12:06:14PM +0500, Илья Шипицин wrote:
> вт, 5 июл. 2022 г. в 11:56, William Lallemand :
> 
> > On Tue, Jul 05, 2022 at 11:15:25AM +0500, Илья Шипицин wrote:
> > > I tried to run on Ubuntu 22.04, it is shipped with OpenSSL-3.0 and
> > > SECLEVEL=2 by default (probably it is correct for RedHat 9 as well ?)
> > >
> > > test · chipitsine/haproxy@1d69992 (github.com)
> > > <
> > https://github.com/chipitsine/haproxy/runs/7163834085?check_suite_focus=true#step:16:602
> > >
> > >
> > > ssl - What could cause "dh key too small" error? - Stack Overflow
> > > <
> > https://stackoverflow.com/questions/61626206/what-could-cause-dh-key-too-small-error
> > >
> > >
> > > if nobody minds, I'll add SECLEVEL=2 to CI.
> > > shall we run *only* SECLEVEL=2 or shall we expand build matrix ?
> > >
> >
> > That's not a good idea, this is supposed to be the default in a lot of
> > distribution and this could hide a lot of problems. HAProxy must works
> > with this default settings, the failing reg-test must be fixed instead.
> >
> 
> I mean "what to do after reg-test fix" (no question on that).
> in order to prevent regression...
> 

Sorry I didn't get correctly what you wanted to do.

Maybe we could add at least a 22.04 to the build.

We could convert the whole matrix to 22.04 later, but we still need to
test with OpenSSL 1.1.1, so we need reg-tests with the 1.1.1 built
manually.

-- 
William Lallemand



Re: running SECLEVEL=2 for OpenSSL-3.0 tests ?

2022-07-05 Thread William Lallemand
On Tue, Jul 05, 2022 at 11:15:25AM +0500, Илья Шипицин wrote:
> I tried to run on Ubuntu 22.04, it is shipped with OpenSSL-3.0 and
> SECLEVEL=2 by default (probably it is correct for RedHat 9 as well ?)
> 
> test · chipitsine/haproxy@1d69992 (github.com)
> <https://github.com/chipitsine/haproxy/runs/7163834085?check_suite_focus=true#step:16:602>
> 
> ssl - What could cause "dh key too small" error? - Stack Overflow
> <https://stackoverflow.com/questions/61626206/what-could-cause-dh-key-too-small-error>
> 
> if nobody minds, I'll add SECLEVEL=2 to CI.
> shall we run *only* SECLEVEL=2 or shall we expand build matrix ?
>

That's not a good idea, this is supposed to be the default in a lot of
distribution and this could hide a lot of problems. HAProxy must works
with this default settings, the failing reg-test must be fixed instead.

-- 
William Lallemand



Re: lua: Add missed lua 5.4 references

2022-07-04 Thread William Lallemand
On Tue, Jun 28, 2022 at 10:11:32AM +0200, Christian Ruppert wrote:
> On 2022-06-22 04:54, Willy Tarreau wrote:
> > Hi Christian,
> > 
> > On Tue, Jun 21, 2022 at 11:05:09PM +0200, Christian Ruppert wrote:
> >> Hey guys,
> >> 
> >> is there any news on this or got this one just lost? I couldn't find a
> >> response to it so I assume it just got lost.
> >> Or is there anything against it?
> >> To bad forwarding doesn't work and since this mail is quite old 
> >> already:
> >> https://www.mail-archive.com/haproxy@formilux.org/msg39689.html
> > 
> > Oh it definitely got lost in the noise. Could someone please repost it
> > with a slightly clearer description and fixing wrapping issues so that
> > we can apply it ?
> > 
> > Thanks!
> > Willy
> 
> Not sure if he's still around so I think it's ok if we do that.
> Attached the patch, based on his one.
> Should be backported to at least 2.6, 2.5 and 2.4 IMHO.
> Perhaps even 2.2? IIRC it got 5.4 support as well.
> 
> Tested against the current 2.7 master:
> ./haproxy -vv | grep -i lua
>OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT= USE_THREAD=1 USE_LIBCRYPT=1 
> USE_OPENSSL=1 USE_LUA=1 USE_ZLIB= USE_SLZ=1 USE_NS=1 USE_51DEGREES= 
> USE_WURFL= USE_SYSTEMD= USE_PROMEX=
> Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 
> -PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY 
> +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO 
> +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL 
> +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL 
> -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING
> Built with Lua version : Lua 5.4.2
> 

Thanks, applied!

-- 
William Lallemand



Re: [PATCH] CI: enable gcc asan builds

2022-07-04 Thread William Lallemand
On Sat, Jul 02, 2022 at 01:40:30PM +0200, Tim Düsterhus wrote:
> Hi
> 
> On 7/2/22 08:03, Илья Шипицин wrote:
> > let us run asan for gcc as well.
> 
> This patch appears reasonable to me.
> 
> Best regards
> Tim Düsterhus
> 
Thanks, applied!

-- 
William Lallemand



rhel+quic packages available

2022-06-30 Thread William Lallemand
On Wed, Jun 08, 2022 at 03:01:05PM +0200, William Lallemand wrote:
> On Wed, Jun 01, 2022 at 07:17:33PM +0500, Илья Шипицин wrote:
> > spec files work under centos 8 as well, but IUS currently builds only
> > centos 7, I haven't figured out how to add centous 8 yet
> > 
> 
> IUS seems to only support CentOS 7 unfortunately. :(
> 
> We had a talk with the guys from zenetys.com which are haproxy users for
> a long time and they told us that they are maintaining packages for RHEL
> 6/7/8.
> 
> https://packages.zenetys.com/latest/redhat/
> https://github.com/zenetys/rpm-haproxy
> 
> I added the links on https://github.com/haproxy/wiki/wiki/Packages .
> 

Just a little update about this, zenetys updated their packages and now
provides an haproxy 2.6+quic package which is available for thel 6, 7
and 8.

https://github.com/zenetys/rpm-haproxy/commits/haproxy26z+quic

-- 
William Lallemand



Re: HttpClient in Lua

2022-06-21 Thread William Lallemand
On Tue, Jun 21, 2022 at 08:33:04AM +1000, Philip Young wrote:
> I don’t mind the idea, it would reduce having a separate service/
> proxy. If creating it inside HAProxy then wouldn’t that mess with the
> threading and it blocking? 
> 

That's just a proxy in your haproxy configuration so there is no reason
it will block.

-- 
William Lallemand



Re: HttpClient in Lua

2022-06-20 Thread William Lallemand
On Mon, Jun 20, 2022 at 08:27:22PM +1000, Philip Young wrote:
> Thanks for the answer William, it is very much appreciated. It is good to get 
> some clarification and will stop me continuing to spend time trying to get it 
> to work.
> 
>  In the meantime, I am working around the problem by calling out to a local 
> python service running on the same machine as HAProxy over http, which is 
> then making the authorisation request with a client certificate.  Not ideal, 
> but will switch out the hack once it is supported in HAProxy. 
> 
> Thanks again
> Phil
> 
If you want to take the hackish road, you can just simply create a proxy
in your haproxy which does this, with an SSL server and a crt. This way
you can still use the httpclient or the socket API directly with this
proxy.

-- 
William Lallemand



Re: HttpClient in Lua

2022-06-20 Thread William Lallemand
On Wed, Jun 15, 2022 at 11:33:27PM +1000, Philip Young wrote:
> Hi
> I am currently writing a LUA module to make authorisation decisions on 
> whether a request is allowed, by calling out to another service to make the 
> authorisation decision. 
> In the Lua module, I am using Socket.connect_ssl() to connect to the 
> authorisation service but I am struggling to work out how to set the path to 
> the certificate I want to use to connect to the authorisation service. 
> Does anybody know how to set the path to the certificate that is used when 
> using Socket.connect_ssl() 
> Is it possible to do this using the httpclient?
> I have tried asking the Slack chat channel and on the commercial site but no 
> one knows. 
> 
> Cheers Phil

Hello Phil,

Unfortunately the Socket and the HTTPClient lua class are not able to
use a client certificate right now. This should evolve in the future but
the current architecture is not able to do it.

-- 
William Lallemand



Re: To upgrade from 2.5 to 2.6 on ubuntu

2022-06-20 Thread William Lallemand
On Fri, Jun 17, 2022 at 10:55:19PM +, Henning Svane wrote:
> Hi
> To upgrade to HAproxy 2.6 will I have to add this repository?
> 
> sudo add-apt-repository ppa:vbernat/haproxy-2.6
> Or is there another way I should use?
> 
> I use Ubuntu 20.04 and would also like to upgrade to 22.04. Any know problems 
> with 22.04?
> 
> Regards
> Henning
> 

Probably the best way to do it indeed.

I don't know any problem regarding 22.04 at the moment, there are some
changes with the algorithms used since the migration to openssl 3.0, if
you need old algorithms you might want to reenable them.

-- 
William Lallemand



Re: grooming IUS haproxy packages

2022-06-08 Thread William Lallemand
On Wed, Jun 01, 2022 at 07:17:33PM +0500, Илья Шипицин wrote:
> spec files work under centos 8 as well, but IUS currently builds only
> centos 7, I haven't figured out how to add centous 8 yet
> 

IUS seems to only support CentOS 7 unfortunately. :(

We had a talk with the guys from zenetys.com which are haproxy users for
a long time and they told us that they are maintaining packages for RHEL
6/7/8.

https://packages.zenetys.com/latest/redhat/
https://github.com/zenetys/rpm-haproxy

I added the links on https://github.com/haproxy/wiki/wiki/Packages .

-- 
William Lallemand



Re: grooming IUS haproxy packages

2022-06-01 Thread William Lallemand
On Wed, Jun 01, 2022 at 09:50:20AM +0500, Илья Шипицин wrote:
> Hello,
> 
> I created couple of PRs
> 
> HAProxy 2.0.29 by chipitsine · Pull Request #18 · iusrepo/haproxy20
> (github.com) <https://github.com/iusrepo/haproxy20/pull/18>
> HAProxy 2.2.24 by chipitsine · Pull Request #21 · iusrepo/haproxy22
> (github.com) <https://github.com/iusrepo/haproxy22/pull/21>
> 
> 2.0 and 2.2 are updated to their latest versions.
> while I'm working on 2.4, 2.6, I would like to ask you to review current
> packages
> 

Thanks Ilya! 

> 1) which USE_XXX to add/remove

Some of the flags could be removed since they are already in the
linux-glibc target: USE_CRYPT_H, USE_GETADDRINFO.

Starting with 2.4 we can do some changes:

- don't use EXTRA_OBJS for the prometheus exporter, but USE_PROMEX=1
- use USE_SLZ=1 instead of USE_ZLIB=1 

A possible update could be lua-5.4, but that's not a requirement.


> 2) improving build process

No idea for now.

> 3) so on
>

I'm still confused about something, it looks like it only provides
packages for RHEL/CentOS 7 or am I missing the other versions somewhere?


> also, some of IUS packages do have READMEs in github, I'm looking for ideas
> what to put to README.
>
I have no clue, that's probably not important.

-- 
William Lallemand



Re: how to install on RHEL7 and 8

2022-05-31 Thread William Lallemand
Hello Ryan,

On Thu, May 26, 2022 at 01:28:58PM -0500, Ryan O'Hara wrote:
> 
> I am the maintainer for all the Red Hat and Fedora packages. Feel free to
> ask questions here on the mailing list or email me directly.
> 
> I try to keep Fedora up to date with latest upstream, but once a release
> goes into a specific Fedora release (eg. haproxy-2.4 in Fedora 35) I don't
> update to haproxy-2.5 in that same release. I have in the past and I get
> angry emails about rebasing to a newer release. I've spoken to Willy about
> this in the past and we seem to be in agreement on this.
> 
> RHEL is different. We almost never rebase to a later major release for the
> lifetime of RHEL. The one exception was when we added haproxy-1.8 to RHSCL
> (software collections) in RHEL7 since the base RHEL7 had haproxy-1.5 and
> there were significant features added to the 1.8 release.
> 
> I get this complaint often for haproxy in RHEL. Keep in mind that RHEL is
> focused on consistency and stability over a long period of time. I can't
> stress this enough - it is extremely rare to rebase to a new, major release
> of haproxy (or anything else) in a major RHEL release. For example, RHEL9
> has haproxy-2.4 and will likely always have that version.

I understand your point, indeed RHEL is focused on stability and it
seems normal that the packages maintained inside RHEL does not jump from
one major version to another. 

> I do often rebase to newer minor release to pick up bug fixes (eg.
> haproxy-2.4.8 will be updated to haproxy-2.4.17, but very unlikely to
> be anything beyond the latest 2.4 release). I understand this is not
> for everybody.
> 

That's the right way to do it in my opinion, a stable distribution
just needs to follow the minor releases which basically contains the
bugfixes.

But what we are trying to offer is a way for users to chose another
branch so they could benefits from new features. Since RHEL has a long
life cycle and HAProxy has 2 majors releases per year, RHEL can't
provide them and that's normal.

Some users actually need up to date versions because the needs evolve,
the tools, and the protocols too. For example someone who want to use
the latest dynamic features with their kubernetes, or who wants to test
http/3.

Debian had the same problem as RHEL, but it was solved with
haproxy.debian.net which provides multiple HAProxy branches for multiple
debian versions. It would be great if we can achieve something like this
with COPR or anything else.

> As mentioned elsewhere, COPR is likely the best place for this. It had been
> awhile since I've used it, but there have been times I did special,
> unsupported builds in COPR for others to use.
> 
> Hope this helps.
> 
> Ryan

Thanks!

-- 
William Lallemand



Re: how to install on RHEL7 and 8

2022-05-26 Thread William Lallemand
Ilya,

On Thu, May 26, 2022 at 03:13:54PM +0500, Илья Шипицин wrote:
> I'll try to focus on redhat packaging (I'm somewhat familiar with Fedora
> COPR, and I can try OBS).
>

I don't think OBS is relevant for this case, the documentation is poor
and it's complicated to contribute to a package.

I don't know about COPR, but most of the work seems to have been done in
IUS, and it's easy to contribute just by doing a pull request on their
repository. I'm not sure there is any advantage to creating another
repository, but I might be wrong.


> if I will not come back in next couple of weeks, that means I did not find
> a time.
> 

Well be careful then, because I'm talking about long-term maintenance,
not just another package not being updated after a few months like all
haproxy RPM that we can find out there. It really takes time and
dedication.

Regards,

-- 
William Lallemand



Re: how to install on RHEL7 and 8

2022-05-25 Thread William Lallemand
On Tue, May 24, 2022 at 08:56:14PM +, Alford, Mark wrote:
> Do you have instruction on the exact library needed to fo the full install on 
> RHEL 7 and RHEL 8
> 
> I read the INSTALL doc in the tar ball and the did the make command and it 
> failed because of LUA but lua.2.5.3 is installed
> 
> Please help
> 
> 
Hello,

I'm using this thread to launch a call for help about the redhat
packaging.

We try to document the list of available packages here:
https://github.com/haproxy/wiki/wiki/Packages

The IUS repository is know to work but only provides packages as far as
2.2. no 2.3, 2.4 or 2.5 are there but I'm seeing an open ticket for
the 2.4 here: https://github.com/iusrepo/wishlist/issues/303

Unfortunately nobody ever step up to maintain constantly the upstream
releases for redhat/centos like its done for ubuntu/debian on
haproxy.debian.net.

Maybe it could be done with IUS, its as simple as a pull request on
their github for each new release, but someone need to be involve.

I'm not a redhat user, but from time to time someone is asking for a
redhat package and nothing is really available and maintained outside of
the official redhat one.

Regards,

-- 
William Lallemand



Re: Backporting "MEDIUM: mworker: reexec in waitpid mode after successful loading" to 2.4

2022-05-13 Thread William Lallemand
On Tue, May 10, 2022 at 02:21:42PM +0200, William Lallemand wrote:
> On Tue, May 10, 2022 at 12:09:59PM +0200, Christian Ruppert wrote:
> > 
> > It even just happened when running with gdb, without a reload.
> > 
> 
> What the patch does is re-executing the master in wait-mode once the
> worker was started in order to free the master memory of huge data
> (maps, SSL certificates etc).
> 
> I thought your crash was unrelated but indeed what is weird is that you
> experienced a watchdog crash in the master... which is is really
> surprising since the master does not do much.
> 
> > Does that help? You mworker commit *seems*, at least at the first 
> > glance, to fix that. Without I have multiple coredumps within 24 hours. 
> > Often I can trigger some by just reloading/restarting. With your commit 
> > I couldn't for almost 24 hours + doing 100 reloads with 10s sleep 
> > between each.
> > Let me know if you want me to turn on some debug flags or something 
> > else. Or do you want a dump? I'd share it off-list then.
> 
> That does help indeed, but I will need a full coredump with the binaries
> to analyze what provoked this watchdog in the master!
> 
> Is it a problem you have since a while or did it happens with an update?
> It's not impossible that a fix provoked this.
> 

We had some exchanges with Christian in private about this issue, it
resulted in this fix for the watchdog.

I pushed a fix in the master repository for the watchdog issue:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=ae053b30da4db588f7fabe09e5f85cbebdc421ad

-- 
William Lallemand



Re: Backporting "MEDIUM: mworker: reexec in waitpid mode after successful loading" to 2.4

2022-05-10 Thread William Lallemand
On Tue, May 10, 2022 at 12:09:59PM +0200, Christian Ruppert wrote:
> 
> It even just happened when running with gdb, without a reload.
> 

What the patch does is re-executing the master in wait-mode once the
worker was started in order to free the master memory of huge data
(maps, SSL certificates etc).

I thought your crash was unrelated but indeed what is weird is that you
experienced a watchdog crash in the master... which is is really
surprising since the master does not do much.

> Does that help? You mworker commit *seems*, at least at the first 
> glance, to fix that. Without I have multiple coredumps within 24 hours. 
> Often I can trigger some by just reloading/restarting. With your commit 
> I couldn't for almost 24 hours + doing 100 reloads with 10s sleep 
> between each.
> Let me know if you want me to turn on some debug flags or something 
> else. Or do you want a dump? I'd share it off-list then.

That does help indeed, but I will need a full coredump with the binaries
to analyze what provoked this watchdog in the master!

Is it a problem you have since a while or did it happens with an update?
It's not impossible that a fix provoked this.

-- 
William Lallemand



Re: Backporting "MEDIUM: mworker: reexec in waitpid mode after successful loading" to 2.4

2022-05-10 Thread William Lallemand
On Tue, May 10, 2022 at 11:05:01AM +0200, Christian Ruppert wrote:
> Hi guys, William,
> 
> can we please get that "MEDIUM: mworker: reexec in waitpid mode after 
> successful loading" - fab0fdce981149a4e3172f2b81113f505f415595 
> backported to 2.4?
> I seem to run into it, at least on one of our 40 LBs. This one is a VM 
> though. It sometimes crashes after each reload. Running 2.5 with 
> fab0fdce981149a4e3172f2b81113f505f415595 seems to fix the issue for me.
> 
> https://github.com/haproxy/haproxy/commit/fab0fdce981149a4e3172f2b81113f505f415595
> 

Hello Christian,

Honestly we run into a lot of issues and bugfixes after this patch was
pushed, I don't think it's even possible to backport this without
breaking the 2.4, there are a lot of corner cases and I don't want to
break this branch which is pretty stable now.

2.5 already runs with this architecture for a while in some places which
make it more robust but it was not easy to get there. Also the next LTS
version which is 2.6 is almost there!

What kind of crashes are you experimenting? It's supposed to help with
the possible OOM on reload when too much memory was consumed by the
master.

-- 
William Lallemand



Re: [PATCH] move missing function definition to openssl-compat.h

2022-04-25 Thread William Lallemand
On Sat, Apr 23, 2022 at 11:22:24PM +0500, Илья Шипицин wrote:
> Hello,
> 
> small cleanup patch.
> 
> Ilya

> From 637f02dc75a68bf40d30cb78d4e021551d323d90 Mon Sep 17 00:00:00 2001
> From: Ilya Shipitsin 
> Date: Sat, 23 Apr 2022 23:07:26 +0500
> Subject: [PATCH] CLEANUP: move ssl_sock_load_ocsp definition to
>  openssl-compat.h
> 
> literally it removes one "ifdef" and moves missing function definition
> to openssl-compat.h

Hello,

openssl-compat.h is not suited for that, it's mostly used to implement
the missing OpenSSL accessors in the previous versions or in the forks.

The ssl_sock functions must rest in ssl_sock.c. As you can see in this
file there is a openssl counterpart to that function above.

Cheers,

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.6-dev6

2022-04-19 Thread William Lallemand
On Sat, Apr 16, 2022 at 08:26:33PM +0200, Willy Tarreau wrote:
> On Sat, Apr 16, 2022 at 11:12:41PM +0500,  ??? wrote:
> > > > > William has also set up a build system that's triggered by the CI and
> > > that
> > > > > produces packages of the latest development version for various
> > > distros.
> > > > > The goal is to help users deploy development versions to participate 
> > > > > to
> > > > > the testing and benefit early from new features, as we know that till
> > > now
> > > > > it used to require particular efforts and that not everyone has enough
> > > > > time to think about rebuilding packages often. I'll let William expand
> > > on
> > > > > this point regarding what's covered and how to use this.
> > > > >
> > > >
> > > > that's interesting.
> > > > any links?
> > >
> > > As I said he will share the details :-)
> > >
> > 
> > I've something ...
> > wlallemand/haproxy-nightly-build (github.com)
> > <https://github.com/wlallemand/haproxy-nightly-build>
> 
> Ah I forgot, he updated the links in the wiki:
> 
>  
> https://github.com/haproxy/wiki/wiki/Packages#community-maintained-repositories
> 
> But he knows the current status and the next steps if any.
> 
> Willy
> 

Indeed,

It uses the Open Build System from OpenSuse to generate packages on
debian and ubuntu. Each time we push some code on the git repository,
github trigger the build by doing an HTTP request on the service.


The status of the latest build is here: 
https://build.opensuse.org/project/show/home:wlallemand

At the moment it builds packages for Debian 10/11/Unstable and Ubuntu
20.04 with multiple architecture depending on what's available.
More distributions are available, like redhat/centos but I will need to
spend some time on this.

The packages are available here: 
https://software.opensuse.org/download/package?package=haproxy=home%3Awlallemand

You can grab them individually or install the repository, which is
really convenient if you want to update on a daily basis.


The version of the package is generated this way:

   haproxy_   2.6-dev6  .0.   git20220416  .a8b1065b6  -0+52.2_amd64.deb
 ^   ^ ^   ^  ^ 
 |   | |   |  + Some obscure OBS 
sub-version 
 |   | |   |
 |   | |   + Git Commit ID
 |   | |
     |   | + Date of the day
 |   |
 |   + Number of commits after the tag
 |
 + Latest tag

I hope this will be useful for users that want to deploy the development
version for testing purposes.

Regards,

-- 
William Lallemand



Re: [PATCH]: BUILD/MINOR: ssl openssl 3 warning fix

2022-04-07 Thread William Lallemand
On Thu, Apr 07, 2022 at 11:07:41AM +0500, Илья Шипицин wrote:
> ср, 6 апр. 2022 г. в 14:08, William Lallemand :
> 
> > On Wed, Apr 06, 2022 at 09:45:02AM +0100, David CARLIER wrote:
> > > > I recall there is a openssl3 port ongoing perhaps ?
> > >
> > > I was trying to see if the said 3.x portage work is close to be merged
> > > to master then yes my patch is useless.
> > > Otherwise
> >
> > In fact everything but the engine to provider port was merged! So you
> > should only have warning about that.
> >
> > We will probably keep the engine support in the code and disabled it by
> > default. But what could be done is adding
> > -DOPENSSL_API_COMPAT=0x1010L to the compiler command when activating
> > USE_ENGINE.
> >
> 
> I'd suggest to use implicit "#if !defined(OPENSSL_NO_ENGINE)" which is
> available for both openssl-1.1.1 and 3.0
> instead of explicit USE_ENGINE
> 

We already use OPENSSL_NO_ENGINE, which only checks if the engine
support is available in openssl. This is not the actual problem, most of
the time the engine support is built within OpenSSL, the problem is to
be able to build without the deprecated API which means without the engine
feature in HAProxy by default.

-- 
William Lallemand



Re: [PATCH]: BUILD/MINOR: ssl openssl 3 warning fix

2022-04-06 Thread William Lallemand
On Wed, Apr 06, 2022 at 09:45:02AM +0100, David CARLIER wrote:
> > I recall there is a openssl3 port ongoing perhaps ?
> 
> I was trying to see if the said 3.x portage work is close to be merged
> to master then yes my patch is useless.
> Otherwise

In fact everything but the engine to provider port was merged! So you
should only have warning about that.

We will probably keep the engine support in the code and disabled it by
default. But what could be done is adding
-DOPENSSL_API_COMPAT=0x1010L to the compiler command when activating
USE_ENGINE.
> 
> > but here a little "band-aid" proposal for the actual master branch.
> 
> my patch is just a temporary fix.
> 

At the moment we prefer to keep track of the warning because we need to
fix them properly, but on the CI we are actually doing:
make DEFINE="-DOPENSSL_API_COMPAT=0x1010L -DOPENSSL_NO_DEPRECATED" 
> 

Regards,

-- 
William Lallemand



Re: [PATCH]: BUILD/MINOR: ssl openssl 3 warning fix

2022-04-06 Thread William Lallemand
Hello David,

On Tue, Apr 05, 2022 at 07:07:47PM +0100, David CARLIER wrote:
> Hi, I recall there is a openssl3 port ongoing perhaps ?
> 
> but here a little "band-aid" proposal for the actual master branch.
>
> From 2d76cb9f249b9519d2b102133b224716965f3f02 Mon Sep 17 00:00:00 2001
> From: David Carlier 
> Date: Tue, 5 Apr 2022 19:03:05 +0100
> Subject: [PATCH] BUID/MINOR: ssl fix warning build with openssl 3
> 
> fix warning build from various deprecated apis from the 3.x release.

I'm not sure what deprecated warning you are seeing, I only have the one about
the Engine API, that will be disabled by default later.

> ---
>  include/haproxy/openssl-compat.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/haproxy/openssl-compat.h 
> b/include/haproxy/openssl-compat.h
> index 87bb5109c..8a252aef9 100644
> --- a/include/haproxy/openssl-compat.h
> +++ b/include/haproxy/openssl-compat.h
> @@ -2,6 +2,7 @@
>  #define _HAPROXY_OPENSSL_COMPAT_H
>  #ifdef USE_OPENSSL
>  
> +#define OPENSSL_API_COMPAT 0x1010L
>  #include 
>  #include 
>  #include 
> -- 
> 2.34.1

That is not a good idea in my opinion, the goal is a real portage to the 3.0
API, once it's done it is supposed to compile with OPENSSL_NO_DEPRECATED
defined.

-- 
William Lallemand



Re: Re: CI caching improvement

2022-03-22 Thread William Lallemand
On Mon, Mar 21, 2022 at 04:19:38PM +0500, Илья Шипицин wrote:
> 
> I think we can adjust build-ssl.sh script to download tagged quictls (and
> cache it in the way we do cache openssl itself)
> Tags · quictls/openssl (github.com)
> <https://github.com/quictls/openssl/tags>
> 

Unfortunately that won't work, those tags are inherited from the OpenSSL
repository and does not contain any QUIC code. They didn't make any tag
for quictls, probably because they don't want to do an active
maintenance.

However it looks like they are maintaining a branch with their quic
patches for each openssl subversions. And those branches are not
modified too much once they pushed it.

Maybe we could just take the latest commit of the "openssl-3.0.2+quic"
branch for now, and update it from time to time.

The other solution I considered for VTest was to cache one build
per day. Maybe we could do that for quictls, because the project is not
that active and most of the time the people developing on QUIC test
everything on their computer. That could be the best compromise between
low maintenance and automatic update of the quictls code.
The only requirement would be to display the quictls commit ID
somewehere in the log to be certain of the version we are linking with.


-- 
William Lallemand



Re: [PATCH] CI: switch to LibreSSL-3.5.1

2022-03-18 Thread William Lallemand
On Wed, Mar 16, 2022 at 12:29:41PM +0500, Илья Шипицин wrote:
> Hello,
> 
> as LibreSSL-3.5.1 is released, let us switch to the most recent release.
> 
> thanks,
> Ilya

Thanks, applied.

-- 
William Lallemand



Re: CI caching improvement

2022-03-18 Thread William Lallemand
On Wed, Mar 16, 2022 at 09:31:56AM +0100, Tim Düsterhus wrote:
> Willy,
> 
> On 3/8/22 20:43, Tim Düsterhus wrote:
> >> Yes my point was about VTest. However you made me think about a very good
> >> reason for caching haproxy builds as well :-)  Very commonly, some VTest
> >> randomly fails. Timing etc are involved. And at the moment, it's impossible
> >> to restart the tests without rebuilding everything. And it happens to me to
> >> click "restart all jobs" sometimes up to 2-3 times in a row in order to end
> > 
> > I've looked up that roadmap entry I was thinking about: A "restart this
> > job" button apparently is planned for Q1 2022.
> > 
> > see https://github.com/github/roadmap/issues/271 "any individual job"
> > 
> > Caching the HAProxy binary really is something I strongly advice against
> > based on my experience with GitHub Actions and CI in general.
> > 
> > I think the restart of the individual job sufficiently solves the issue
> > of flaky builds (until they are fixed properly).
> > 
> 
> In one of my repositories I noticed that this button is now there. One 
> can now re-run individual jobs and also all failed jobs. See screenshots 
> attached.
> 

Hello Tim,

It looks like it is available as well on our repositories, I just test
it and it works correctly.

Honestly I really don't like the dependency to another repository with a
format specific to github.

I agree that a cleaner integration with github with their specific tools
is nice, but I don't want us to be locked with github, we are still
using cirrus, travis, sometimes gitlab, and also running some of the
scripts by hand. 

We also try to avoid the dependencies to other projects and its much
simplier to have few shell scripts and a CI configuration in the
repository. And typescript is not a language we would want to depend on
if we need to debug it for example.

Giving that github is offering the job restart feature, we could skip
the VTest caching, since it's a little bit ugly. Only the quictls cache
need to be fixed.

Regards,

-- 
William Lallemand



Re: [EXTERNAL] Re: CI caching improvement

2022-03-08 Thread William Lallemand
On Tue, Mar 08, 2022 at 08:05:31PM +0100, Tim Düsterhus wrote:
> 
> Let me make a competing proposal that:
> 
> 1. Keeps the complexity out of the GitHub workflow configuration in 
> haproxy/haproxy.
> 2. Still allows VTest caching.
> 
> For my https://github.com/TimWolla/haproxy-auth-request repository I 
> have created a reusable GitHub Action to perform the HAProxy 
> installation similar to 'actions/checkout':
> 
> https://github.com/TimWolla/action-install-haproxy/
> 
> I just spent a bit of time to fork that action and to make it VTest 
> specific:
> 
> https://github.com/TimWolla/action-install-vtest/
> 
> The action receives the VTest branch or commit as the input and will 
> handle all the heavy lifting of downloading, compiling and caching VTest.
> 
> The necessary changes to HAProxy then look like this:
> 
> https://github.com/TimWolla/haproxy/commit/78af831402e354f22d67682be0f323dec9c26a52
> 
> This basically replaces the use of 'scripts/build-vtest.sh' by 
> 'timwolla/action-install-vtest@main', so the configuration in the 
> haproxy/haproxy repository is not getting any more complicated, as all 
> the heavy lifting is done in the action which can be independently 
> tested and maintained.
> 
> If this proposal sounds good to you, then I'd like to suggest the following:
> 
> 1. Willy creates a new haproxy/action-install-vtest repository in the 
> haproxy organization.
> 2. Willy creates a new GitHub team with direct push access to that 
> repository.
> 3. Willy adds me to that team, so that I can maintain that repository 
> going forward (e.g. to handle the Dependabot pull requests that keep the 
> JavaScript dependencies up to date).
> 
> If that repository is properly set up, I'll send a patch to switch over 
> haproxy/haproxy to make use of that action.
> 
> Best regards
> Tim Düsterhus

Tim,

Honestly I'm confused, it is overcomplicated in my opinion :(

I don't really see the benefits in creating a whole new repository
instead of the few lines in the yaml file.

We are talking about doing a new project for just the equivalent of a 5
lines shell script... which really don't need to be tested and
maintained outside of the project.

I feel like I'm missing something with my simple implementation, we are
already downloading all the SSL libraries, should we stop doing it this
way? What could be the problems with this?

It seems like you want to do this in a strict github way, which is
probably convenient for a lot of usecase, but it just look really more
complicated that my first proposal.

-- 
William Lallemand



Re: CI caching improvement

2022-03-08 Thread William Lallemand
On Tue, Mar 08, 2022 at 04:17:00PM +0100, Tim Düsterhus wrote:
> William
> 
> On 3/8/22 16:06, William Lallemand wrote:
> > Also, I'm wondering if we could also cache the build of HAProxy, you
> > could think that weird, but in fact it will help relaunch the tests when
> > one is failing, without rebuilding the whole thing.
> 
> Please don't. You always want to run a clean build, otherwise you're 
> going to get super-hard-to-debug failures if some object file is 
> accidentally taken from the cache instead of a clean build.
> 

The cache is supposed to be done once the build is valid, if the build
are not reproducible I think we have a bigger problem here, we are not
supposed to trigger random behavior of the compiler.

> I've heard that redoing a single failed build instead of the full matrix 
> is already on GitHub's roadmap, so the problem will solve itself.
> 
Maybe but that will still build HAProxy for nothing, if you have an
example of an unreproduicible build case with HAProxy I'm fine with this
statement but honestly nothing come to my mind. Even if one of the
distribution component is broken during a lapse of time, the problem
will reappear without the cache.

> > Let me know if we can improve the attached patch, otherwise I'll merge
> > it.
> > 
> 
> I don't like it. As you say: It's ugly and introduces complexity for a 
> mere 8 second gain. Sure, we should avoid burning the planet by wasting 
> CPU cycles in CI, but there's a point we're the additional complexity 
> results in a maintenance nightmare.

I don't really see where there is complexity, it's just a curl to get an
ID, and I don't have the impression that the github cache is not working
corretly since we are using it with the ssl libraries.

The build is conditioned by a unique if statement and a simple sh
oneliner.

It's not really 8s, it's 8s per job, and someting not everything is run in
parallel, without mentionning private repositories where the time is
accounted. Also that was also a way that could be used for quictls which
still takes a lot of time.

-- 
William Lallemand



Re: CI caching improvement

2022-03-08 Thread William Lallemand
On Tue, Mar 08, 2022 at 08:38:00PM +0500, Илья Шипицин wrote:
> 
> I'm fine with swapping "vtest" <--> "haproxy" order.
> 

Ok, I can do that.

> also, I do not think current patch is ugly, it is acceptable for me (if we
> agree to save 8 sec).

Honestly I don't see the value in building the same binary which never
change again and again, and it does not add much complexity so I think
that's fine.


> I'm afraid that current patch require some fix,
> because GitHub uses cache in exclusive way, i.e.
> you need unique cache key per job, current cache key is not job dependent
> (but the rest looks fine)
> 

I don't think I get that, the key is a combination of the VTest commit +
the hash per job.

key: vtest-${{ steps.vtest-id.outputs.key }}-${{ 
steps.generate-cache-key.outputs.key }}


Thanks,

-- 
William Lallemand



CI caching improvement

2022-03-08 Thread William Lallemand
Hello,

The attached patch implements a somehow ugly way to cache the VTest
binary, basically it gets the commit ID by doing a curl of the
master.patch on the github URL.

It allows to save ~8s per matrix row, which is around 160s in total. I
know there is a small window where the curl and the git clone won't have
the same ID but that will be rebuild anyway for the next build, so
that's fine in my opinion.

We could probably use the same approach to cache quictls or anything
that uses a git repository.

Also, I'm wondering if we could also cache the build of HAProxy, you
could think that weird, but in fact it will help relaunch the tests when
one is failing, without rebuilding the whole thing.

Let me know if we can improve the attached patch, otherwise I'll merge
it.

Regards,

-- 
William Lallemand
>From 34649ae5549a73d0f43530794f47861fb679510e Mon Sep 17 00:00:00 2001
From: William Lallemand 
Date: Tue, 8 Mar 2022 14:49:25 +0100
Subject: [PATCH] CI: github: add VTest to the github actions cache

Get the latest master commit ID from VTest and use it as a key for the
cache. VTest takes around 8s to build per matrix row, we save around
160s of CI with this.
---
 .github/workflows/vtest.yml | 18 +-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
index a9e86b6a22..38099093cd 100644
--- a/.github/workflows/vtest.yml
+++ b/.github/workflows/vtest.yml
@@ -55,6 +55,11 @@ jobs:
   run: |
 echo "::set-output name=key::$(echo ${{ matrix.name }} | sha256sum | awk '{print $1}')"
 
+- name: Get VTest master commit ID
+  id: vtest-id
+  run: |
+echo "::set-output name=key::$(curl -s https://github.com/vtest/VTest/commit/master.patch 2>/dev/null | head -n1 | awk '{print $2}')"
+
 - name: Cache SSL libs
   if: ${{ matrix.ssl && matrix.ssl != 'stock' && matrix.ssl != 'BORINGSSL=yes' && matrix.ssl != 'QUICTLS=yes' }}
   id: cache_ssl
@@ -70,6 +75,14 @@ jobs:
   with:
 path: '~/opt-ot/'
 key: ot-${{ matrix.CC }}-${{ env.OT_CPP_VERSION }}-${{ contains(matrix.name, 'ASAN') }}
+
+- name: Cache VTest binary
+  id: cache_vtest
+  uses: actions/cache@v2
+  with:
+path: '~/vtest/'
+key: vtest-${{ steps.vtest-id.outputs.key }}-${{ steps.generate-cache-key.outputs.key }}
+
 - name: Install apt dependencies
   if: ${{ startsWith(matrix.os, 'ubuntu-') }}
   run: |
@@ -86,8 +99,11 @@ jobs:
 brew install socat
 brew install lua
 - name: Install VTest
+  if: ${{ steps.cache_vtest.outputs.cache-hit != 'true' }}
   run: |
 scripts/build-vtest.sh
+mkdir ~/vtest/
+cp ../vtest/vtest ~/vtest/
 - name: Install SSL ${{ matrix.ssl }}
   if: ${{ matrix.ssl && matrix.ssl != 'stock' && steps.cache_ssl.outputs.cache-hit != 'true' }}
   run: env ${{ matrix.ssl }} scripts/build-ssl.sh
@@ -134,7 +150,7 @@ jobs:
 # This is required for macOS which does not actually allow to increase
 # the '-n' soft limit to the hard limit, thus failing to run.
 ulimit -n 5000
-make reg-tests VTEST_PROGRAM=../vtest/vtest REGTESTS_TYPES=default,bug,devel
+make reg-tests VTEST_PROGRAM=~/vtest/vtest REGTESTS_TYPES=default,bug,devel
 - name: Show VTest results
   if: ${{ failure() && steps.vtest.outcome == 'failure' }}
   run: |
-- 
2.32.0



Re: [PATCH] BUILD ssl: another build warning on LIBRESSL_VERSION_NUMBER

2022-03-01 Thread William Lallemand
On Mon, Feb 28, 2022 at 10:13:31PM +0100, Julien Thomas wrote:
> Hi William,
> 
> Please find attached a patch I used in order to remove some
> LIBRESSL_VERSION_NUMBER warnings when building haproxy 2.5.4 with an old
> version of OpenSSL.
> 
> Cheers,
> Julien

Thanks Julien, I just pushed it into master.

-- 
William Lallemand



[ANNOUNCE] haproxy-2.5.3

2022-02-18 Thread William Lallemand
Hi,

HAProxy 2.5.3 was released on 2022/02/18. It added 17 new commits
after version 2.5.2.

This version was released shortly after the 2.5.2 because some pending patches
were missing. This version fixes the pending issues that were identified in
2.5.1.

Some bugs in the httpclient were found by Baptiste and Willy, one in the lua
implementation that was preventing to use the same httpclient context correctly
for several requests and the other one was causing a crash with the CLI
implementation when the destination buffer had not enough space.

In 2.5 we spent time migrating the master-worker mode to a new model that
automatically switch to the wait mode after loading the configuration, helping
to free the memory of the master. This new behavior revealed some issues in the
wait mode that existed for a long time, but were only present upon a loading
failure. We fixed some FD leaks related to this, that could have resulted in FD
exhaustion after a lot of reloads and failures.

Willy fixed some alignement problems that were found when using gcc-11 + RHEL8,
resulting in instant crashes on startup.

Lukas fixed a problem an issue with multiline ESMTP response in the
mailer code.

Rémi fixed some SSL issues reported by coverity.

As usual, people using the 2.5 branch are encouraged to migrate to this version.

Please find the usual URLs below :
   Site index   : http://www.haproxy.org/
   Discourse: http://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Wiki : https://github.com/haproxy/wiki/wiki
   Sources  : http://www.haproxy.org/download/2.5/src/
   Git repository   : http://git.haproxy.org/git/haproxy-2.5.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git
   Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/


---
Complete changelog :
Christopher Faulet (3):
  BUG/MINOR: sink: Use the right field in appctx context in release callback
  BUG/MEDIUM: resolvers: Really ignore trailing dot in domain names
  MINOR: httpclient: Don't limit data transfer to 1024 bytes

Lukas Tribus (1):
  BUG/MINOR: mailers: negotiate SMTP, not ESMTP

Remi Tricot-Le Breton (3):
  BUG/MINOR: ssl: Add missing return value check in ssl_ocsp_response_print
  BUG/MINOR: ssl: Fix leak in "show ssl ocsp-response" CLI command
  BUG/MINOR: ssl: Missing return value check in ssl_ocsp_response_print

William Lallemand (3):
  BUG/MINOR: mworker: fix a FD leak of a sockpair upon a failed reload
  BUG/MINOR: httpclient: reinit flags in httpclient_start()
  BUG/MINOR: tools: url2sa reads ipv4 too far

Willy Tarreau (7):
  MINOR: sock: move the unused socket cleaning code into its own function
  BUG/MEDIUM: mworker: close unused transferred FDs on load failure
  BUG/MEDIUM: fd: always align fdtab[] to 64 bytes
  BUG/MAJOR: compiler: relax alignment constraints on certain structures
  CLEANUP: httpclient/cli: fix indentation alignment of the help message
  BUG/MEDIUM: httpclient: limit transfers to the maximum available room
  DEBUG: buffer: check in __b_put_blk() whether the buffer room is respected

---

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.5.2

2022-02-17 Thread William Lallemand
Hi,

On Thu, Feb 17, 2022 at 08:25:39AM +0100, Willy Tarreau wrote:
> By the way Vincent, William figured that I missed a few patches during my
> long backport session yesterday. I'll have to check with him if they
> warrant another release or if they can wait for next one. Hence no need
> to rush on new packages yet ;-) I'll keep you updated whatever the
> outcome.
> 

I'll probably emit a 2.5.3 this evening or tomorrow, some of the
forgotten fixes could be bothersome for people trying to migrate in 2.5.

-- 
William Lallemand



Re: [PATCH] fix guarding when OPENSSL_NO_DH is set

2022-02-14 Thread William Lallemand


Hello Ilya,

> Subject: [PATCH 1/2] BUILD: SSL: fix guarding when OpenSSL is built with
>  OPENSSL_NO_DH
> 
> some parts of the code support OPENSSL_NO_DH macro, but other do not.
> let us add wherever appropriate


I can't apply this one, part of the DH code changed during OpenSSLv3
port, could you check if this is still relevant?

Thanks,

-- 
William Lallemand



Re: [EXTERNAL] Re: Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 11:54:05PM +0500, Илья Шипицин wrote:
>
> as you already suggested "best effort" support policy, it should not
> require your time.
> am I correct ?
> 

Don't worry I will still review and merge patches :-)

-- 
William Lallemand



Re: Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 07:46:44PM +0100, William Lallemand wrote:
>
> On Fri, Feb 04, 2022 at 11:02:24PM +0500, Илья Шипицин wrote:
> > пт, 4 февр. 2022 г. в 19:16, William Lallemand :
> > 
> > > On Fri, Feb 04, 2022 at 11:52:06AM +0100, William Lallemand wrote:
> > > >
> > > > I just tried to build with the latest boringSSL version, the problem is
> > > > on our side:
> > > >
> > > > We are defining X509_OBJECT_get0_X509_CRL() because it does not exist in
> > > > boringSSL, and inside it we are accessing the members of the X509_OBJECT
> > > > and it can't work since it's opaque.
> > > >
> > > > We need to use the accessors instead, or find an alternative API.
> > > >
> > >
> > > So, basically, we need to extract some X509_CRL from an X509_STORE which
> > > could be done in OpenSSL  with the X509_OBJECT API by using
> > > X509_OBJECT_get0_X509_CRL() which is not available in boringSSL
> > >
> > > We are kind of stuck there because this is supposed to be the low
> > > level API, now everything is opaque and we can't do much.
> > >
> > > The alternative would be to stop using the X509_STORE for storing the
> > > CRL, and use a STACK_OF(X509_CRL)... But we use the store because it
> > > could be either a X509_CRL or a X509... so it would be kind of
> > > redefining a X509_STORE API... and I honestly don't want to do that.
> > >
> > > In my opinion there is a problem in their API because there is no
> > > accessor for the X509_CRL in a X509_OBJECT. Which is kind of weird
> > > because they have a X509_OBJECT_get0_X509() accessor, they probably just
> > > missed it.
> > >
> > 
> > we already use "#ifdef X509_V_FLAG_CRL_CHECK" for enabling those functions.
> > in theory we have 2 choices
> > 
> > 1) adopt X509_OBJECT_get0_X509_CRL to BoringSSL in some way
> > 2) exclude CRL manipulations under BoringSSL by adjusting ifdef
> 
> I can't spend more time on this, feel free to open a bug report on their
> bugtracker, this is probably better for everyone.
> 
> You can adjust the ifdef as well.
> 

Just to be more precise about the ifdef: this won't work with boringSSL,
OpenSSL supports the CRLs, they define X509_V_FLAG_CRL_CHECK, the problem
is the lack of accessor, not the support of CRL in boringSSL.

-- 
William Lallemand



Re: Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 11:02:24PM +0500, Илья Шипицин wrote:
> пт, 4 февр. 2022 г. в 19:16, William Lallemand :
> 
> > On Fri, Feb 04, 2022 at 11:52:06AM +0100, William Lallemand wrote:
> > >
> > > I just tried to build with the latest boringSSL version, the problem is
> > > on our side:
> > >
> > > We are defining X509_OBJECT_get0_X509_CRL() because it does not exist in
> > > boringSSL, and inside it we are accessing the members of the X509_OBJECT
> > > and it can't work since it's opaque.
> > >
> > > We need to use the accessors instead, or find an alternative API.
> > >
> >
> > So, basically, we need to extract some X509_CRL from an X509_STORE which
> > could be done in OpenSSL  with the X509_OBJECT API by using
> > X509_OBJECT_get0_X509_CRL() which is not available in boringSSL
> >
> > We are kind of stuck there because this is supposed to be the low
> > level API, now everything is opaque and we can't do much.
> >
> > The alternative would be to stop using the X509_STORE for storing the
> > CRL, and use a STACK_OF(X509_CRL)... But we use the store because it
> > could be either a X509_CRL or a X509... so it would be kind of
> > redefining a X509_STORE API... and I honestly don't want to do that.
> >
> > In my opinion there is a problem in their API because there is no
> > accessor for the X509_CRL in a X509_OBJECT. Which is kind of weird
> > because they have a X509_OBJECT_get0_X509() accessor, they probably just
> > missed it.
> >
> 
> we already use "#ifdef X509_V_FLAG_CRL_CHECK" for enabling those functions.
> in theory we have 2 choices
> 
> 1) adopt X509_OBJECT_get0_X509_CRL to BoringSSL in some way
> 2) exclude CRL manipulations under BoringSSL by adjusting ifdef

I can't spend more time on this, feel free to open a bug report on their
bugtracker, this is probably better for everyone.

You can adjust the ifdef as well.

-- 
William Lallemand



Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 11:52:06AM +0100, William Lallemand wrote:
> 
> I just tried to build with the latest boringSSL version, the problem is
> on our side:
> 
> We are defining X509_OBJECT_get0_X509_CRL() because it does not exist in
> boringSSL, and inside it we are accessing the members of the X509_OBJECT
> and it can't work since it's opaque.
> 
> We need to use the accessors instead, or find an alternative API.
> 

So, basically, we need to extract some X509_CRL from an X509_STORE which
could be done in OpenSSL  with the X509_OBJECT API by using
X509_OBJECT_get0_X509_CRL() which is not available in boringSSL

We are kind of stuck there because this is supposed to be the low
level API, now everything is opaque and we can't do much.

The alternative would be to stop using the X509_STORE for storing the
CRL, and use a STACK_OF(X509_CRL)... But we use the store because it
could be either a X509_CRL or a X509... so it would be kind of
redefining a X509_STORE API... and I honestly don't want to do that.

In my opinion there is a problem in their API because there is no
accessor for the X509_CRL in a X509_OBJECT. Which is kind of weird
because they have a X509_OBJECT_get0_X509() accessor, they probably just
missed it.

-- 
William Lallemand



Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 11:18:50AM +0100, William Lallemand wrote:
> On Fri, Feb 04, 2022 at 09:57:25AM +0100, Remi Tricot-Le Breton wrote:
> >
> > 
> > On 02/02/2022 17:49, William Lallemand wrote:
> > >
> > >> Subject: [PATCH 2/7] BUILD: SSL: define X509_OBJECT for BoringSSL
> > >>
> > >> X509_OBJECT is opaque in BonringSSL, since we still use it, let us move 
> > >> it to openssl-compat.h
> > >>
> > >> from 
> > >> https://boringssl.googlesource.com/boringssl/+/refs/heads/2924/include/openssl/x509_vfy.h#120
> > > I'm not really fond of this kind of declaration, most of the time we
> > > added helpers that were available in recent version of OpenSSL in this
> > > file. But in this case, adding a whole structure that was removed...
> > > with no guarantee that this will continue to work it's not a good idea.
> > >
> > >  From what I get they aligned the opaque structures with the OpenSSL API,
> > > so we probably will have the same problem with OpenSSL v3 without the
> > > obsolete API. And we are currently in the process of porting it to
> > > HAProxy. We probably need to change the code that uses X509_OBJECT.
> > > So I suppose it will start to work during this portage.
> > >
> > X509_OBJECT and the APIs working on this structure were not marked as 
> > deprecated in OpenSSLv3, we are facing yet another place where BoringSSL 
> > seems a bit excessive in what they want to keep hidden.
> > Managing BoringSSL would still be much more expensive than managing 
> > OpenSSLv3 if this kind of problem happens on many structures.
> > 
> 
> Thanks for the clarification Rémi, I now remember having this
> conversation :-)
> 
> But when checking the commits they didn't make anything deprecated in
> fact, they just made it opaque.
> 
> https://boringssl.googlesource.com/boringssl/+/dddb60eb9700110835ff6e2b429de40a17006429
> 
> In this commit they pretend aligning with OpenSSL, which might be the
> case, if you take a look at openssl/x509.h, they still define:
> 
> - DEFINE_STACK_OF(X509_OBJECT)
> - OPENSSL_EXPORT STACK_OF(X509_OBJECT) *X509_STORE_get0_objects(X509_STORE 
> *st);
> 
> So either there is an export problem of the X509_OBJECT or we are
> missing a include or something else is wrongly done.
> 
> I'll made some test to check what's going on.
> 

I just tried to build with the latest boringSSL version, the problem is
on our side:

We are defining X509_OBJECT_get0_X509_CRL() because it does not exist in
boringSSL, and inside it we are accessing the members of the X509_OBJECT
and it can't work since it's opaque.

We need to use the accessors instead, or find an alternative API.

-- 
William Lallemand



Re: Re: [PATCH] get BoringSSL back to the game

2022-02-04 Thread William Lallemand
On Fri, Feb 04, 2022 at 09:57:25AM +0100, Remi Tricot-Le Breton wrote:
>
> 
> On 02/02/2022 17:49, William Lallemand wrote:
> >
> >> Subject: [PATCH 2/7] BUILD: SSL: define X509_OBJECT for BoringSSL
> >>
> >> X509_OBJECT is opaque in BonringSSL, since we still use it, let us move it 
> >> to openssl-compat.h
> >>
> >> from 
> >> https://boringssl.googlesource.com/boringssl/+/refs/heads/2924/include/openssl/x509_vfy.h#120
> > I'm not really fond of this kind of declaration, most of the time we
> > added helpers that were available in recent version of OpenSSL in this
> > file. But in this case, adding a whole structure that was removed...
> > with no guarantee that this will continue to work it's not a good idea.
> >
> >  From what I get they aligned the opaque structures with the OpenSSL API,
> > so we probably will have the same problem with OpenSSL v3 without the
> > obsolete API. And we are currently in the process of porting it to
> > HAProxy. We probably need to change the code that uses X509_OBJECT.
> > So I suppose it will start to work during this portage.
> >
> X509_OBJECT and the APIs working on this structure were not marked as 
> deprecated in OpenSSLv3, we are facing yet another place where BoringSSL 
> seems a bit excessive in what they want to keep hidden.
> Managing BoringSSL would still be much more expensive than managing 
> OpenSSLv3 if this kind of problem happens on many structures.
> 

Thanks for the clarification Rémi, I now remember having this
conversation :-)

But when checking the commits they didn't make anything deprecated in
fact, they just made it opaque.

https://boringssl.googlesource.com/boringssl/+/dddb60eb9700110835ff6e2b429de40a17006429

In this commit they pretend aligning with OpenSSL, which might be the
case, if you take a look at openssl/x509.h, they still define:

- DEFINE_STACK_OF(X509_OBJECT)
- OPENSSL_EXPORT STACK_OF(X509_OBJECT) *X509_STORE_get0_objects(X509_STORE *st);

So either there is an export problem of the X509_OBJECT or we are
missing a include or something else is wrongly done.

I'll made some test to check what's going on.

-- 
William Lallemand



Re: [PATCH] get BoringSSL back to the game

2022-02-02 Thread William Lallemand
Ilya,

Adding Fred to the thread, so he can gives his opinion about the QUIC
part.

On Mon, Jan 31, 2022 at 10:22:01AM +0500, Илья Шипицин wrote:
> 0001 ..  0003 are "pre QUIC" patches
> 0007  is very simple

Regarding the first patches:


> Subject: [PATCH 3/7] REGTESTS: skip show_ssl_ocspresponse.vtc when BoringSSL 
> is used
> 
> OCSP stapling implementation is not compatible with BoringSSL, test
> is broken in BoringSSL

Merged.

> Subject: [PATCH 2/7] BUILD: SSL: define X509_OBJECT for BoringSSL
> 
> X509_OBJECT is opaque in BonringSSL, since we still use it, let us move it to 
> openssl-compat.h
> 
> from 
> https://boringssl.googlesource.com/boringssl/+/refs/heads/2924/include/openssl/x509_vfy.h#120

I'm not really fond of this kind of declaration, most of the time we
added helpers that were available in recent version of OpenSSL in this
file. But in this case, adding a whole structure that was removed...
with no guarantee that this will continue to work it's not a good idea. 

>From what I get they aligned the opaque structures with the OpenSSL API,
so we probably will have the same problem with OpenSSL v3 without the
obsolete API. And we are currently in the process of porting it to
HAProxy. We probably need to change the code that uses X509_OBJECT.
So I suppose it will start to work during this portage.

> Subject: [PATCH 1/7] BUILD: SSL: adjust guard for X509_get_X509_PUBKEY(x)
> 
> BoringSSL defines that function since
> https://boringssl.googlesource.com/boringssl/+/33f8d33af0dcb083610e978baad5a8b6e1cfee82

Merged.


-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.6-dev1

2022-02-01 Thread William Lallemand
On Tue, Feb 01, 2022 at 07:41:08PM +0100, Willy Tarreau wrote:
>   - master-worker: some simplifications were applied to the way the new
> process connects to the old one to retrieve listening sockets. In the
> past it would rely on "-x" on the command line, or would try to connect
> to the first socket that had the "expose-fd listener" statement. Now
> the new process will simply reuse the existing worker socket to
> retrieve the old socket FDs so that it will work even without a stats
> socket. As such "-x" is no more used in master mode.

To be a bit more precise, the "-x" option is still used internally, it
is now able to use the internal socketpairs which are used
between the master and the workers. So after a reload you would see
something like this:

68684 pts/14   S+ 0:00 ./haproxy -W -f haproxy.cfg
68703 pts/14   Sl+0:00 ./haproxy -sf 68686 -x sockpair@3 -W -f 
haproxy.cfg

The biggest benefit is that you don't have to configure anything
anymore to do a hitless reload.

-- 
William Lallemand



Re: [PATCH] get BoringSSL back to the game

2022-02-01 Thread William Lallemand
On Mon, Jan 31, 2022 at 10:22:01AM +0500, Илья Шипицин wrote:
>
> Hello,
>

Hello Ilya,

> 0001 ..  0003 are "pre QUIC" patches
> 0004 ..  0006 are most questionable QUIC part
> 0007  is very simple
> 
> 
> we can discuss whether BoringSSL should be
> 1) dropped completely
> 2) supported, but no QUIC
> 3) supported for QUIC as well
> 
> as for "3)" I've checked current state of QUICTLS, looks like its future is
> not clear, it is not updated since mid december 2021, also it is not clear
> whether OpenSSL is going to accept it or not.
> 
> thanks,
> Ilya


BoringSSL is a burden in term of maintenance, since they only work in a
rolling release mode, we can't offer a real support in maintenancecc
branches.

The current development team also won't have time to support fully
BoringSSL, the only library we are fully supporting is OpenSSL.

The other libs are supported as a best effort, but not all features are
available.

I don't mind merging some patches for improving boringSSL support, but
what's bothering me is building the master with the boringSSL HEAD in
the CI.  Because API changes and can broke the build without us doing
anything.

So if we don't want to be bothered to much we could activate the
boringSSL build weekly or something like that. But I don't want a
reminder each time I pushed that boringSSL broke something.

Regarding the QUIC patches, I'll let the guys in charge of that decides,
but the development of QUIC in HAProxy is made with quictls currently.

-- 
William Lallemand



Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with set server ssl

2022-01-19 Thread William Lallemand
On Wed, Jan 19, 2022 at 03:32:35PM +0100, Willy Tarreau wrote:
> Subject: Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with 
> set server ssl
>
> On Wed, Jan 19, 2022 at 03:24:44PM +0100, William Lallemand wrote:
> > On Tue, Jan 18, 2022 at 12:07:21PM +0100, Willy Tarreau wrote:
> > >
> > > On Mon, Jan 17, 2022 at 07:46:24PM +0100, William Dauchy wrote:
> > > > Hello Christopher,
> > > > 
> > > > On Wed, Jan 12, 2022 at 12:45 PM William Dauchy  
> > > > wrote:
> > > > > my approach was to say:
> > > > > - remove the implicit behavior
> > > > > - then work on the missing commands for the health checks
> > > > 
> > > > Do you think we can conclude on it?
> > > 
> > > Just merged after our discussion on it :-)
> > > 
> > 
> > Can we also mark it as deprecated in 2.5? patch attached
> 
> Sure, works for me!
> 
> Willy
> 

Thanks, pushed in master.


-- 
William Lallemand



Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with set server ssl

2022-01-19 Thread William Lallemand
On Tue, Jan 18, 2022 at 12:07:21PM +0100, Willy Tarreau wrote:
>
> On Mon, Jan 17, 2022 at 07:46:24PM +0100, William Dauchy wrote:
> > Hello Christopher,
> > 
> > On Wed, Jan 12, 2022 at 12:45 PM William Dauchy  wrote:
> > > my approach was to say:
> > > - remove the implicit behavior
> > > - then work on the missing commands for the health checks
> > 
> > Do you think we can conclude on it?
> 
> Just merged after our discussion on it :-)
> 

Can we also mark it as deprecated in 2.5? patch attached

-- 
William Lallemand
>From 9998a33d3a027ff6863eab71bcc2f2d7158319b4 Mon Sep 17 00:00:00 2001
From: William Lallemand 
Date: Wed, 19 Jan 2022 15:17:08 +0100
Subject: [PATCH] DOC: management: mark "set server ssl" as deprecated

This command was integrated in 2.4 when it was not possible to handle
SSL with dynamic servers, this is now possible so we should prefer this
way.

Must be backported in 2.5.
---
 doc/management.txt | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/doc/management.txt b/doc/management.txt
index b96cbc8789..b9b7ebdfb7 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -2185,11 +2185,14 @@ set server / fqdn 
   Change a server's FQDN to the value passed in argument. This requires the
   internal run-time DNS resolver to be configured and enabled for this server.
 
-set server / ssl [ on | off ]
+set server / ssl [ on | off ]  (deprecated)
   This option configures SSL ciphering on outgoing connections to the server.
   When switch off, all traffic becomes plain text; health check path is not
   changed.
 
+  This command is deprecated, create a new server dynamically with or without
+  SSL instead, using the "add server" command.
+
 set severity-output [ none | number | string ]
   Change the severity output format of the stats socket connected to for the
   duration of the current session.
-- 
2.32.0



Re: changes in 2.5

2022-01-18 Thread William Lallemand
On Mon, Jan 17, 2022 at 10:21:44PM -0300, Joao Morais wrote:
> 
> Hello list, I have a consumer of the master socket’s `show proc`
> output and I observed that 2.5 changed its lay out, and this change
> lead me to two doubts:

> - Is there a release notes or something with all the backward
> compatibility changes between minor versions? I observed that 2.5 now
> requires that var() has a match type, but maybe there are other
> changes that I’m missing and I should take care;

We try to limite breakage when we create a new branch, but sometimes
things were poorly designed and invalid configuration could break, we
describe these changes in the changelog mails most of the time.

You can also try a git log on doc/management.txt and
doc/configuration.txt it will show the changes.

> - Do you suggest me a good way to identify the proxy version so I can
> use a distinct parser for the show proc output? I’m currently using
> the last field of the second line, but might be a better option out
> there that I’m not aware of.

For the `show proc` output, I specifically added a "show version"
command in 2.5, which was backported in 2.4.10, 2.3.17, 2.2.20, in the
master of the soon to be released 2.0.27.

The change in `show proc` was made to remove the relative PID because
HAProxy is not multi-process anymore.

But if you parse the output by splitting the spaces and keeping the
right field with the header you shouldn't have a parsing problem.

-- 
William Lallemand



Re: [EXTERNAL] Re: [PATCH] BUILD: opentracing: display warning in case of using OT_USE_VARS at compile time

2021-12-28 Thread William Lallemand
On Tue, Dec 28, 2021 at 12:14:37PM +0100, Miroslav Zagorac wrote:
> 
> Hello William,
> 
> I think that this commit can be applied to branches 2.5 and 2.6-dev.
> 
> 
> Best regards.
> 

Thanks, I added the information about the backport in the patch and I
pushed it in master!

-- 
William Lallemand



Re: [PATCH] BUILD: opentracing: display warning in case of using OT_USE_VARS at compile time

2021-12-28 Thread William Lallemand
On Mon, Dec 27, 2021 at 01:22:57PM +0100, Miroslav Zagorac wrote:
>
> On 12/27/2021 01:16 PM, Miroslav Zagorac wrote:
> > Hello all,
> >
> > to avoid misunderstandings (like say github issue #1493) when compiling
> > HAproxy source with the OpenTracing filter enabled, a warning has been
> > added if the OT_USE_VARS variable is used.
> >
> > If appropriate, please apply this commit.
> >
> > Best regards.
> >
> 
> sorry, i forgot the patch.  :(
> 

Hi Miroslav, In which versions this patch should be backported?

Thanks

-- 
William Lallemand



Re: [PATCH] BUILD: unbreak the build with newer libressl

2021-12-15 Thread William Lallemand
On Tue, Dec 07, 2021 at 08:34:39PM -0500, Daniel Jakots wrote:
> Hi,
> 
> Here's the file inline generated with `git format-patch -1`. Is it ok?
> 
> I'm not subscribed to the mailing list, please keep me in Cc:.
> 
> Thanks,
> Daniel
> 
> From bc44099cb32a95d3a8895a6232b5b0ce5c9cb5c0 Mon Sep 17 00:00:00 2001

Thanks Daniel, I merged it into master.

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.5.0

2021-12-14 Thread William Lallemand
On Tue, Nov 23, 2021 at 05:18:37PM +0100, Willy Tarreau wrote:
>
> Hi,
> 
> HAProxy 2.5.0 was released on 2021/11/23. It added 9 new commits after
> version 2.5-dev15, fixing minor last-minute details (bind warnings
> that turned to errors, and an incorrect free in the backend SSL cache).
> 

Hi Thierry,

Could you update the lua documentation at 
http://www.arpalert.org/haproxy-api.html?

It looks like neither the 2.4 version nor the 2.5 were published.

Also the 2.4-dev link seems to be the master, maybe you could rename
"2.4dev" into "master" directly?

Thanks,

-- 
William Lallemand



Re: [PATC H] adjust vtc for cert revocation check

2021-12-10 Thread William Lallemand
On Sat, Dec 04, 2021 at 02:45:15PM +0500, Илья Шипицин wrote:
> Subject: [PATC H] adjust vtc for cert revocation check
>
> hello,
> 
> breaking behaviour was introduced on LibreSSL side.
> more details: https://github.com/libressl-portable/portable/issues/697
> 
> in short, currently vtc expects 21, but some openssl variations return 20
> 
> X509_V_ERR_UNABLE_TO_VERIFY_LEAF_SIGNATURE   = 21
> X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY = 20
> 
> cheers,
> Ilya

Thanks, applied.

-- 
William Lallemand



Re: OCSP with dynamic SSL storage

2021-11-22 Thread William Lallemand
On Fri, Nov 05, 2021 at 01:30:53PM +0100, Marco Corte wrote:
> Subject: Re: OCSP with dynamic SSL storage
>
> Il 2021-11-05 13:11 Marco Corte ha scritto:
> > Hi all.
> > 
> > I have a bind section that contains
> > ... ssl crt ZZZ.pem ...
> > 
> > where ZZZ.pem is actually a full path.
> > 
> > If I upload a new certificate/key to ZZZ.pem and a corresponding OCSP
> > response to ZZZ.pem.ocsp and do a
> > 
> > # systemctl reload haproxy.service
> > 
> > 
> > then the certificate and the OCSP stapling are correct.
> > Moreover I can update the OCSP, when needed
> > 
> > # printf "set ssl ocsp-response <<\n$(base64 ZZZ.pem.ocsp)\n\n" |
> > socat /run/haproxy/admin.sock stdio
> > OCSP Response updated!
> > 
> > 
> > 
> > If, after updating the files, I use the following procedure, I am not
> > able to update the OCSP response
> > 
> > # printf "set ssl cert ZZZ.pem <<\n$(cat ZZZ.pem\n\ncommit ssl cert
> > ZZZ.pem\n" | socat /run/haproxy/admin.sock stdio
> > Transaction created for certificate ZZZ.pem!
> > 
> > Committing ZZZ.pem..
> > Success!
> > 
> > # printf "set ssl ocsp-response <<\n$(base64 ZZZ.pem.ocsp)\n\n" |
> > socat /run/haproxy/admin.sock stdio
> > OCSP single response: Certificate ID does not match any certificate or 
> > issuer.
> > 
> > 
> > Since the two files ZZZ.pem and ZZZ.pem.ocsp are always the same, I
> > suspect that I am doing something wrong.
> > Am I skipping any step?
> > 
> > Thank you
> > Ciao!
> > 
> > .marcoc
> > 
> > Please note that I may have messed up with some commands while
> > anonymizing them in this email.
> 
> I forgot to mention the version: haproxy v2.4.8 on Ubuntu 18.04
> 

Hello,

Sorry for the late reply, when updating a certificate dynamically it is
recommended to update its .ocsp at the same time before committing, so
it could add again the Certificate ID in the OCSP tree. It's the only
HAProxy can know that OCSP was activated.

Once its done, you can use the "set ssl ocsp-response", like you were
using before.

Look at the example in the documentation:

https://cbonte.github.io/haproxy-dconv/2.4/management.html#9.3-set%20ssl%20cert

Regards,

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.5-dev15

2021-11-20 Thread William Lallemand
On Fri, Nov 19, 2021 at 08:03:22PM +0100, Willy Tarreau wrote:
>   - since TLS early-data support was added, resumed connections could
> cause a confusingly incorrect error to be reported if the strict-sni
> was used or changed, because the session would still be accepted. This
> affects 1.8 and above.

Not exactly, every non-matching SNI with strict-sni activated were
causing a accidental "handshake failure" instead of a "unrecognized
name". Because the clientHello callback was returning with a success
code. The error was generated after the callback because openSSL
couldn't finish the handshake.

However, in the case of a resume, no error was reported, but openSSL
didn't had any handshake to do, so the connection was still accepted
even though the SNI wasn't matching.

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.5-dev10

2021-10-18 Thread William Lallemand
On Sat, Oct 16, 2021 at 04:22:29PM +0200, Willy Tarreau wrote:
>
> Hi,
> 
> HAProxy 2.5-dev10 was released on 2021/10/16. It added 75 new commits
> after version 2.5-dev9.
> 
> The smoke is progressively being blown away and we're starting to see
> clearer what final 2.5 will look like.
> 
> In completely random order, here are the main changes I noticed in this
> release:
> 
>   - some fixes for OpenSSL 3.0.0 support from Rémi and William; regression
> tests were fixed as well and the version in the CI was upgraded from
> alpha17 to 3.0.0

What's important to mention is that we don't have broken SSL reg-tests
anymore, which is probably the first time since we introduced the
reg-tests, lot of them was broken because of incompatibilities with
previous OpenSSL version, or LibreSSL/boringSSL.

Regarding the whole reg-tests directory, only 3 remains broken:

reg-tests$ git grep REGTEST_TYPE=broken .
connection/proxy_protocol_random_fail.vtc:#REGTEST_TYPE=broken
mailers/healthcheckmail.vtc:#REGTEST_TYPE=broken
seamless-reload/abns_socket.vtc:#REGTEST_TYPE=broken

> 
>   - William added a new config predicate "ssllib_name_startswith" to
> detect the type of SSL library in "-cc" rules.

Actually it's Rémi's patches. It completes the "openssl_version_atleast"
predicate that was previously done and allow us to be very precise in
the test selection, which was not the case before.

-- 
William Lallemand



Re: question: ExecStartPre removal from systemd unit file

2021-08-19 Thread William Lallemand
Hi Tim,

On Thu, Aug 19, 2021 at 12:22:25PM +0200, Tim Düsterhus wrote:
> 
> The config check should prevent HAProxy from going into wait mode when 
> the config is bad on a reload. If I am not mistaken it's not possible to 
> recover from wait mode without a full restart, no?
> 

Well, this line is not used for the reload, but only for the start. For
the reload the first ExecReload line is used:

ExecReload=@SBINDIR@/haproxy -Ws -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
 
The purpose of this reload line is only to return a status code to
systemd during a reload, because kill can't achieve that. It's not
really a problem to be in "wait" mode, if you do a reload again with a
working configuration it will be in a normal state. The wait mode is
just a state where the master only supervise the previous workers and
couldn't fork new workers.

-- 
William Lallemand



question: ExecStartPre removal from systemd unit file

2021-08-19 Thread William Lallemand
Hi List,

I realized yesterday that we have this line in the systemd unit file:

ExecStartPre=@SBINDIR@/haproxy -f $CONFIG -c -q $EXTRAOPTS

Which does not make any sense to me since starting HAProxy itself
checks the configuration, so it slows the start of the service for
nothing.

I'm going to remove this line.

Is there anyone against it, or did I miss a particular usecase?

Thanks,

-- 
William Lallemand



Re: [ANNOUNCE] haproxy-2.5-dev3

2021-08-02 Thread William Lallemand
Hello,

On Sun, Aug 01, 2021 at 06:28:27PM +0200, Willy Tarreau wrote:
>   - a new option "httpslog" was added to complement "httplog". It aims at
> providing some info about the TLS frontend connection by default, such
> as the ciphers used and errors met etc. It is also possible to disable
> low-level SSL error reports to only use these ones (and this should be
> the long-term direction to take). A few sample fetch functions were
> added to extract the SSL-level info. I'm aware that the thread on this
> subject is still active, and any feedback is welcome if that helps to
> further improve the situation for users.
> 

We need feedback about this, it will probably change in the future, the
github thread is available here:

https://github.com/haproxy/haproxy/issues/693

Don't hesitate to report your problems or needs in the ticket.

-- 
William Lallemand



Re: no-stop keyword proposal

2021-07-27 Thread William Lallemand
On Tue, Jul 20, 2021 at 12:18:18PM -0300, Joao Morais wrote:
> Subject: no-stop keyword proposal
>
> 
> Hello list, the diff below is a proposal to add a bind keyword used to
> flag LI_O_NOSTOP option in the bind’s listener.
> 
> Regarding the use case: I need the ability to reach a stopping, but
> still running haproxy instance to, at least:
> 1) fairly distribute
> shutdown sessions of long running connections (usually websockets)
> before hard-stop-after timeouts and kicks all the remaining
> connections at the same time[1]; 2) collect some relevant metrics from
> a stopping instance, e.g. current sessions and rps, which would be
> otherwise lost when these metrics are collected only from the current
> instance.
> 

This is already possible using the master CLI, it was done to access all
processes, old and new. It was designed to do that kind of things.


> Regarding the patch: it’s just the changes I needed to make and
> confirm that it works like I was expecting, provided that the
> listening socket is changed before reloading haproxy into a new
> instance. Please let me know if such improvement can be made and also
> if I’m in the right path.
> 
> ~jm
> 
> [1] https://www.mail-archive.com/haproxy@formilux.org/msg40916.html
> 
> diff --git a/src/cli.c b/src/cli.c
> index b3132191d..4285e5a72 100644
> --- a/src/cli.c
> +++ b/src/cli.c
> @@ -1841,6 +1841,19 @@ static int bind_parse_level(char **args, int cur_arg, 
> struct proxy *px, struct b
>   return 0;
>  }
> 
> +/* parse the "no-stop" bind keyword */
> +static int bind_parse_no_stop(char **args, int cur_arg, struct proxy *px, 
> struct bind_conf *conf, char **err)
> +{
> + struct listener *l;
> +
> + list_for_each_entry(l, >listeners, by_bind) {
> + l->options |= LI_O_NOSTOP;
> + HA_ATOMIC_INC(_jobs);
> + }
> +
> + return 0;
> +}
> +
>  static int bind_parse_severity_output(char **args, int cur_arg, struct proxy 
> *px, struct bind_conf *conf, char **err)
>  {
>   if (!*args[cur_arg + 1]) {
> @@ -2984,6 +2997,7 @@ INITCALL1(STG_REGISTER, cfg_register_keywords, 
> _kws);
>  static struct bind_kw_list bind_kws = { "STAT", { }, {
>   { "level", bind_parse_level,1 }, /* set the unix socket admin 
> level */
>   { "expose-fd", bind_parse_expose_fd, 1 }, /* set the unix socket expose 
> fd rights */
> + { "no-stop",   bind_parse_no_stop, 0 }, /* flag LI_O_NOSTOP in the 
> listener options */
>   { "severity-output", bind_parse_severity_output, 1 }, /* set the 
> severity output format */
>   { NULL, NULL, 0 },
>  }};
> 
> 

That's not a good idea in my opinion, it's an internal state that should
be used only in the case of communication between the master and the
workers, if we expose this to users we will probably have a lot of
corner cases to handle.
This keyword is only meant to say to a worker that it must keep the
communication with the master even if it's trying to exit, so we could
do some maintenance or debugging over the master CLI.

-- 
William Lallemand



Re: Proposal about new default SSL log format

2021-07-08 Thread William Lallemand
On Thu, Jul 08, 2021 at 02:48:32PM +0200, Willy Tarreau wrote:
> On Thu, Jul 08, 2021 at 02:18:32PM +0200, William Lallemand wrote:
> > I saw that you hesitated between "conn_status" and "conn_err_code", the
> > "conn_" prefix could be confusing at some point once you try to have
> > errors on the frontend and the backend side in the same log-format, I
> > think something starting by "fc_conn_" would be more understandable.
> 
> Indeed, and more consistent with what we already have. fc_* is for the
> front connection, bc_* is for the back connection. By the way if we're
> focusing on SSL it should even be ssl_fc_* (we already have a ton of
> them, nobody will find the new one if it doesn't use the same prefix).
>

That's what Rémi implemented for the SSL fetches, but the connection
error one is more generic and does not focus on SSL at all.

> > That seems good to me, we only need frontend info IMHO. People who need
> > the SSL backend connection are not the most common case so they could
> > make their own log-format with it.
> 
> I tend to think that if we're focusing on https vs http, it makes sense
> to consider the frontend only as well for the standard logs.
>

I agree.

> Also some background on the log format, originally we used to place the
> URI at the end of the line because most loggers used to truncate logs
> at 1024 bytes and the tail of the URI was far less important than other
> fields. This explains why we've started to insert certain fields at
> certain places before this. 20 years later I think it is perfectly
> reasonable to consider appending fields *after* the URI, which is also
> a great way of applying minimal changes to existing log parsers, and
> to allow http/https log lines to be easily compared when aligned.
> 

I agree, this way it's easily readable without having to realign
mentally the fields in common between an http line and an https one.


-- 
William Lallemand



Re: Proposal about new default SSL log format

2021-07-08 Thread William Lallemand
Hello,

On Wed, Jul 07, 2021 at 04:43:53PM +0200, Remi Tricot-Le Breton wrote:
> I was indeed more focused on logging SSL related information only, with 
> the idea that an SSL log line could be displayed after a completed 
> handshake, hence the lack of upper layer information in the log line. 
> But it would indeed bring a new level of complexity in the log because 
> correlating the SSL and HTTP log lines of a specific session might be a 
> pain. After talking with Willy, an https log was deemed more useful. It 
> would only be an extension of the existing HTTP log with SSL specific 
> information added. This log format would concern every log line raised 
> by the frontends using this log format (one line per HTTP response of 
> the SSL session).

Yes, what would be great is a "option httpslog" which would provide a
default log line for HTTP over SSL.

> A quick recap of the topics raised by the multiple conversations had in 
> this thread :
> - we won't used tabs as separators in order to remain consistent with 
> the existing log format (especially since this new format will only be 
> an extension of the HTTP one)

I agree, using tab doesn't not seems to be something we would want, it's
the same problem with people that would want json in their log, they
need a new format, not the default one.

> - the date format won't change right now, it is a whole new subject
> - the logged lines will all have the same "date+process_name+pid" prefix 
> as the already existing logs
> - the HTTP log format won't be touched yet, changing it would be a whole 
> new subject as well

Agreed.

> 
> 
> The log would then be as follows :
> 
> 
>      >>> Jul  1 18:11:31 haproxy[143338]: 127.0.0.1:37740 
> [01/Jul/2021:18:11:31.517] \
>    ssl_frontend~ ssl_frontend/s2 0/0/0/7/+7 200 2750 - -  \
>    1/1/1/1/0 0/0 {1wt.eu} {} "GET /index.html HTTP/1.1 \
>    0/0/0/0 TLSv1.3/TLS_AES_256_GCM_SHA384
> 
>    Field   Format    Extract from the 
> example above
>    1   process_name '[' pid ']:' haproxy[143338]:
>    2   client_ip ':' client_port 127.0.0.1:37740
>    3   '[' request_date ']' [01/Jul/2021:18:11:31.517]
>    4   frontend_name ssl_frontend~
>    5   backend_name '/' server_name ssl_frontend/s2
>    6   TR '/' Tw '/' Tc '/' Tr '/' Ta*   
> 0/0/0/7/+7
>    7 status_code 200
>    8 bytes_read* 2750
>    9 captured_request_cookie -
>   10 captured_response_cookie -
>   11 termination_state 
>   12   actconn '/' feconn '/' beconn '/' srv_conn '/' retries*    
> 1/1/1/1/0
>   13   srv_queue '/' 
> backend_queue  0/0
>   14   '{' captured_request_headers* '}' {haproxy.1wt.eu}
>   15   '{' captured_response_headers* 
> '}'    {}
>   16   '"' http_request '"'  "GET /index.html 
> HTTP/1.1"
>   17 *conn_status '/' SSL fe hsk error '/' SSL vfy '/' SSL CA vfy*  
> 0/0/0/0
>   18 *ssl_version '/' ssl_ciphers* TLSv1.3/TLS_AES_256_GCM_SHA384
> 
> and the corresponding log-format :
> 
>      %ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC \
>      %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r \
> *%[conn_err_code]/%[ssl_fc_hsk_err]/%[ssl_c_err]/%[ssl_c_ca_err]* \
> *%sslv/%sslc*
> 

I saw that you hesitated between "conn_status" and "conn_err_code", the
"conn_" prefix could be confusing at some point once you try to have
errors on the frontend and the backend side in the same log-format, I
think something starting by "fc_conn_" would be more understandable.

That seems good to me, we only need frontend info IMHO. People who need
the SSL backend connection are not the most common case so they could
make their own log-format with it.


-- 
William Lallemand



Re: Speeding up opentracing build in CI ?

2021-06-17 Thread William Lallemand
On Thu, Jun 17, 2021 at 03:53:10PM +0200, Willy Tarreau wrote:
> Subject: Re: Speeding up opentracing build in CI ?
>
> On Thu, Jun 17, 2021 at 03:29:58PM +0200, Willy Tarreau wrote:
> > On Thu, Jun 17, 2021 at 03:24:19PM +0200, Willy Tarreau wrote:
> > > On Thu, Jun 10, 2021 at 08:55:13PM +0500,  ??? wrote:
> > > > I was mistaken. LibreSSL does not like parallel install
> > > > 
> > > > libressl fails on `make -j4 install` · Issue #461 ·
> > > > libressl-portable/portable (github.com)
> > > > <https://github.com/libressl-portable/portable/issues/461>
> > > > 
> > > > 
> > > > anyway, if CI works, I'm ok with changes
> > > 
> > > OK I've applied the change to use build_sw on openssl only (libressl
> > > doesn't have it), and -j$(nproc) for this phase for openssl on linux
> > > (will likely be 2). Let's see.
> > 
> > Hmmm it fails with 1.0.2, there's no build_sw there. In addition I do
> > remember that 1.0.2 used to require significant patches to be reliable
> > under make -j.
> > 
> > Let's wait for the remaining tests to conclude.
> 
> OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
> I've just excluded versions 1.x from both the parallel build and the
> build_sw target and that's good now.
> 
> Willy

Great improvement, thanks!

-- 
William Lallemand



Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-14 Thread William Lallemand
On Mon, Jun 14, 2021 at 12:01:11PM +0200, Tim Düsterhus wrote:
> We only run Travis once weekly, because of the limited credits we have. 
> Thus only the newest commit at time of running will have a Travis CI 
> Status integrated into GitHub.
> 
> The most recent one is this: 
> https://github.com/haproxy/haproxy/commit/1b095cac9468d0c3eeb157e9b1a2947487bd3c83


I thought it disappeared completely from the interface, good to know!

Thanks

-- 
William Lallemand



Re: [PATCH 0/4] Use 'feature cmd' in regtests

2021-06-14 Thread William Lallemand
On Mon, Jun 14, 2021 at 02:03:55PM +0200, Willy Tarreau wrote:
> On Mon, Jun 14, 2021 at 03:49:05PM +0500,  ??? wrote:
> > I believe William means conditions like "openssl is 1.1.0 or higher", but
> > that's possible using grep
>

I agree, there are cases when you would want (OPENSSL_VERSION > 0.9.8 &&
OPENSSL_VERSION < 1.1.0) for example, we were limited in the past by
that. Maybe that could be done with grep but honestly that's
over-complicated and poorly readable.

It's not really the case anymore because we stopped to test with
CentOS 6 on the CI but we could have that kind of problem again. 

> And anyway I do want to add config predicates that will provide this.
> As I said when I added a few of them for ".if", the main use will be
> vtc, and we should not waste our time, and just add what is missing.
> 

Looks like a good idea imho, it could even be used to provide several
kind of regex depending of which regex library you use for example.

-- 
William Lallemand



Re: [PATCH 0/4] Use 'feature cmd' in regtests

2021-06-14 Thread William Lallemand
On Fri, Jun 11, 2021 at 07:56:14PM +0200, Tim Duesterhus wrote:
> Hi!
> 

Hello Tim,

> I hope I added all the active developers that touch the reg-tests to the 'CC'
> list :-)
> 
> This series updates the regtests to make use of VTest's 'feature cmd' syntax
> to skip tests that are not supported in the current environment.
> 
> In the long run this will should result in much cleaner tests, because they
> don't need to be parsed by run-regtests.sh to extract these marker comments. 
> It
> also allows to easily add test specific requirements without needing to invent
> an entirely new syntax. An example might be the recently added tests that are
> not supported on BoringSSL. These should be able to get a condition like:
> 
> feature cmd "$HAPROXY_PROGRAM -vv |grep -vq BoringSSL"
> 
> and then they won't run for BoringSSL (the example is untested).
> 

That's very interesting, I had a discussion with Willy last week about
this kind of problems. I think that's clearly the cleanest soluton to
the problem.

In my opinion it only lacks a way to check the OpenSSL version from
'haproxy -cc' so we can enable ssl/set_ssl_cert_bundle.vtc again.


> After this series is applied the output of 'make reg-tests' will change. Tests
> that are excluded using 'feature cmd' will still be listed as "Add test" in
> the test gathering section. However the final line will show that a few tests
> have been skipped:
> 
> 0 tests failed, 4 tests skipped, 105 tests passed
> 
> I don't think this is going to be an issue. But if it is, please complain!
>

Hm the only problem I have with this, is that we won't be able to see why a
test was excluded.

-- 
William Lallemand



Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-14 Thread William Lallemand
On Sat, Jun 12, 2021 at 03:47:46PM +0500, Илья Шипицин wrote:
>
> final apline/musl patch attached
> 

Thanks Ilya, I thought it will be more hackish but it's clean and
simple!

Also it looks like we can build/test several architectures by combining
the same technique  with the qemu docker image (
https://github.com/docker/setup-qemu-action )

I just remembered that we had travis-ci to achieve that in the past, but
it looks like it's not integrated in the github interface, I don't
remember why. https://travis-ci.com/github/haproxy/haproxy It looks like
it's still running though.

According to their documentation they are running the CI
on the actual CPU not a emulated one, so still beter than qemu.

-- 
William Lallemand



Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand
On Fri, Jun 11, 2021 at 08:14:49PM +0500, Илья Шипицин wrote:
> I've found ubuntu musl package, so we can just link to it in CI, for
> example (I'll try)
> 


Well, that won't give you the same environnement as a docker image,
with the same versions. I'll honestly prefer if we could do it with the
alpine image.

I you want to build with the musl-gcc wrapper you will need to link the
linux headers in the musl headers directory otherwise it won't work the
way their package is done.

-- 
William Lallemand



Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand
On Fri, Jun 11, 2021 at 07:09:14PM +0500, Илья Шипицин wrote:
> I'm against expanding cirrus matrix. cirrus is overloaded already, I'm
> afraid they will not stay for long time.
> using custom images in github actions is straightforward, have a look
> 
> centos 6 · chipitsine/haproxy@20fabcd (github.com)
> <https://github.com/chipitsine/haproxy/commit/20fabcd005dc9e3bac54a84bf44631f177fa79c2>
> 
> 
> the same way you can specify either alpine or even special prepared image.
> as I recall last time, we decided not to add alpine because "it is tested
> anyway when docker images are created"
>

As far as I know, it is only built, not tested. We encountered a few
problems that could have been caught before being built by Docker.
And that will prevent us to release a version that doesn't work
correctly with docker before they are built on the docker hub. 


> also, there's small caveat, github actions runs agent inside docker
> container, it might have issues with older libc (or musl).
> but it worth a try
> 

Let's hope it works in this case.


-- 
William Lallemand



Re: [PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand
On Fri, Jun 11, 2021 at 07:19:51PM +0500, Илья Шипицин wrote:
> William, if you do not have a time, I can try to create github action based
> on your cirrus patch ... tomorrow ?
> 

I tried quickly but like Tim I couldn't make it work.

I can't spend much time on this, if you are able to make this work I
prefer to run it from github actions, otherwise we'll go with cirrus.

Thanks,

-- 
William Lallemand



[PATCH] CI: cirrus: add alpine linux to the jobs

2021-06-11 Thread William Lallemand
This commit adds a CI job to cirrus-ci which builds HAProxy on Alpine
Linux, allowing to build and test HAProxy with musl.

OpenSSL, PCRE2, Lua 5.3 as well as the prometheus exporter are enabled.

GNU grep was purposely installed to run the reg-test script.
---
 .cirrus.yml | 13 +
 1 file changed, 13 insertions(+)

diff --git a/.cirrus.yml b/.cirrus.yml
index 9b83e6169..392a3abc5 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -11,3 +11,16 @@ FreeBSD_task:
 - ./haproxy -vv
 - ldd haproxy
 - env VTEST_PROGRAM=../vtest/vtest gmake reg-tests 
REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do cat 
$folder/INFO $folder/LOG; done && exit 1)
+
+alpine_task:
+  container:
+image: alpine:latest
+  only_if: $CIRRUS_BRANCH =~ 'master|next'
+  script:
+- apk add gcc make tar git python3 libc-dev linux-headers pcre-dev 
pcre2-dev openssl-dev lua5.3-dev grep socat curl
+- git clone https://github.com/VTest/VTest.git ../vtest
+- make -C ../vtest FLAGS="-O2 -s -Wall"
+- make CC=cc V=1 TARGET=linux-musl USE_LUA=1 LUA_INC=/usr/include/lua5.3 
LUA_LIB=/usr/lib/lua5.3 USE_OPENSSL=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_PROMEX=1
+- ./haproxy -vv
+- ldd haproxy
+- env VTEST_PROGRAM=../vtest/vtest make reg-tests 
REGTESTS_TYPES=default,bug,devel || (for folder in /tmp/*regtest*/vtc.*; do cat 
$folder/INFO $folder/LOG; done && exit 1)
-- 
2.17.1




add alpine linux to the CI

2021-06-11 Thread William Lallemand
Hello guys,

I couldn't find a way to launch an alpine job easily with github actions so
instead I wrote one for cirrus-ci, It will help debugging Docker images
and musl problems.

Example of the run here: https://cirrus-ci.com/task/5985082050609152

I'll push it in the master if that's fine with you.





Re: Speeding up opentracing build in CI ?

2021-06-10 Thread William Lallemand
On Thu, Jun 10, 2021 at 07:52:23AM +0200, Willy Tarreau wrote:
> Subject: Re: Speeding up opentracing build in CI ?
>
> On Thu, Jun 10, 2021 at 07:19:37AM +0200, Willy Tarreau wrote:
> > On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> > > OT takes about 30 sec (it is built with almost everything disabled). the
> > > biggest time eater is openssl-3.0.0
> > 
> > Maybe that one could be sped up too, I haven't checked if it uses parallel
> > builds.
> 
> So I checked. Good news, it wasn't parallel either, and this alone:
> 
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) all
> +make install_sw
>  )
>  }
> 
> Is enough to drop from 4:52 to 1:28 on my machine. About 1/4 of this time
> is used to build man and HTML pages that we don't use. Instead of the "all"
> target, we should use "build_sw"
> 
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) build_sw
> +make install_sw
>  )
>  }
> 
> this further downs the time to 1:9, hence more than 4 times faster than
> the initial one. It should probably be tested on macos to be certain it's
> OK there as well, and I don't know how to get the CPU count there (or
> maybe we could just force it to a low value like 2 or 4).
> 
> Willy
> 

Looks fine to me, but from what I remember when debugging some reg-tests
there was only one CPU available, I hope I'm wrong.

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-07 Thread William Lallemand
On Mon, Jun 07, 2021 at 02:17:24PM +0200, William Lallemand wrote:
>
> On Mon, Jun 07, 2021 at 02:09:33PM +0200, Tim Düsterhus wrote:
> >
> > William,
> > 
> > On 6/7/21 1:30 PM, William Lallemand wrote:
> > > On Mon, Jun 07, 2021 at 04:02:00PM +0500, Илья Шипицин wrote:
> > >> sorry, I do not have much spare time to implement that in short time
> > >> perspective.
> > >> I think of 2-3 month timeframe.
> > >>
> > > 
> > > Isn't it possible to pass a make argument for one specific CI job?
> > > 
> > > This way we could just do something like:
> > > 
> > >make DEBUG_CFLAGS="-g -Wno-deprecated-declarations"
> > > 
> > 
> > Yes, this would be possible. It's already done to enable QUIC for BoringSSL:
> > 
> > https://github.com/haproxy/haproxy/blob/a9334df5a9bee2bf17b22f965b08df8d47f2f63e/.github/matrix.py#L116-L117
> > 
> > It would need an extra case for OpenSSL 3. You can test whether it works 
> > locally using:
> > 
> > $ python3 .github/matrix.py push
> > 
> > Best regards
> > Tim Düsterhus
> 
> Looks easy enough, I'll try that, thanks.
> 

It worked:

https://github.com/haproxy/haproxy/runs/2764512334

Thanks,

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-07 Thread William Lallemand
On Mon, Jun 07, 2021 at 02:09:33PM +0200, Tim Düsterhus wrote:
> Subject: Re: [PATCH] CI: enable openssl-3.0.0 builds
>
> William,
> 
> On 6/7/21 1:30 PM, William Lallemand wrote:
> > On Mon, Jun 07, 2021 at 04:02:00PM +0500, Илья Шипицин wrote:
> >> sorry, I do not have much spare time to implement that in short time
> >> perspective.
> >> I think of 2-3 month timeframe.
> >>
> > 
> > Isn't it possible to pass a make argument for one specific CI job?
> > 
> > This way we could just do something like:
> > 
> >make DEBUG_CFLAGS="-g -Wno-deprecated-declarations"
> > 
> 
> Yes, this would be possible. It's already done to enable QUIC for BoringSSL:
> 
> https://github.com/haproxy/haproxy/blob/a9334df5a9bee2bf17b22f965b08df8d47f2f63e/.github/matrix.py#L116-L117
> 
> It would need an extra case for OpenSSL 3. You can test whether it works 
> locally using:
> 
> $ python3 .github/matrix.py push
> 
> Best regards
> Tim Düsterhus

Looks easy enough, I'll try that, thanks.

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-07 Thread William Lallemand
On Mon, Jun 07, 2021 at 05:08:32PM +0500, Илья Шипицин wrote:
> пн, 7 июн. 2021 г. в 16:31, William Lallemand :
> 
> > On Mon, Jun 07, 2021 at 04:02:00PM +0500, Илья Шипицин wrote:
> > > sorry, I do not have much spare time to implement that in short time
> > > perspective.
> > > I think of 2-3 month timeframe.
> > >
> >
> > Isn't it possible to pass a make argument for one specific CI job?
> >
> > This way we could just do something like:
> >
> >   make DEBUG_CFLAGS="-g -Wno-deprecated-declarations"
> >
> 
> please send a patch
> 

Sorry if I wasn't clear but this was a question. I have no idea how this
works and you are the more experienced with Tim on this subject :-)

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-07 Thread William Lallemand
On Mon, Jun 07, 2021 at 04:02:00PM +0500, Илья Шипицин wrote:
> sorry, I do not have much spare time to implement that in short time
> perspective.
> I think of 2-3 month timeframe.
> 

Isn't it possible to pass a make argument for one specific CI job?

This way we could just do something like:

  make DEBUG_CFLAGS="-g -Wno-deprecated-declarations"

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-07 Thread William Lallemand
On Sun, Jun 06, 2021 at 05:36:45AM +0200, Willy Tarreau wrote:
> On Sun, Jun 06, 2021 at 12:51:53AM +0200, Tim Düsterhus wrote:
> > Ilya,
> > 
> > On 6/5/21 5:10 AM,  ??? wrote:
> > > here are two patches:
> > > 
> > > - deprecated warnings suppressed
> > > - openssl-3.0.0 enabled
> > > 
> > 
> > In the second patch you forgot the 'CI:' prefix in the commit message.
> > 
> > Other than that the patches LGTM with regard to the CI changes.
> 
> OK thanks Tim, I'll take it and adjust the subject.
> 
> Willy
> 

Hello,

I don't think that's a good idea to do it this way, the first intent was
to be able to build OpenSSL 3.0.0, but the -Wdeprecated-declarations was
preventing to build with -Werror.

With this patch we are hiding all these warnings for all users and they
can be relevant at some point, not only for OpenSSL, but for the other
libs that are linked with haproxy.

In my opinion we should only disable them for this specific build of
OpenSSL 3.0.0 on the CI, not for everyone in the Makefile.

-- 
William Lallemand



Re: [PATCH] CI: enable openssl-3.0.0 builds

2021-06-02 Thread William Lallemand
On Wed, Jun 02, 2021 at 04:34:16PM +0500, Илья Шипицин wrote:
> Subject: Re: [PATCH] CI: enable openssl-3.0.0 builds
>
> ср, 2 июн. 2021 г. в 16:27, Tim Düsterhus :
> 
> > Ilya,
> >
> > On 6/2/21 12:58 PM, Илья Шипицин wrote:
> > > as openssl-3.0.0 is getting close to release, let us add it to build
> > matrix.
> > >
> >
> > I dislike that this is going to report all commits in red, until the
> > build failures with OpenSSL 3 are fixed.

I agree, but in my opinion we should do it in two steps:

* fix the *errors* and build without -Werror in order to see the
  -Wdeprecated-declarations warnings.

* port haproxy to the new API (long term goal) to be able to build
  with openssl 3.0.0 with -Werror.


> 
> @William Lallemand   has an appetite to make it
> green ;)
> 

I'll fix what I can to be able to build with
-Wno-deprecated-declarations.

> I did a quick research, whether
> > some job could be marked as "experimental" / "allowed failure" with
> > GitHub Actions, but could not find anything.
> >

I don't think we need this, fixing the build errors shouldn't be long,
Emeric already made some tests with openssl3 few weeks ago and it was
working fine.

-- 
William Lallemand



  1   2   3   4   5   6   >