Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
On Thu, Jul 30, 2020 at 08:25:16AM +0500, ??? wrote:
> > EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
> > > +name: libressl-2.9.2 | prometheus-exporte
> >
> > I guess this was "prometheus-exporter" above (missing "r") ?
> >
>
> oops.
> can you please fix it during merge ?
No problem, now fixed.
> > > - - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> > $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS"
> > ADDLIB="-Wl,-rpath,$SSL_LIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC"
> > EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
> > > + - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> > $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z
> > ${SSL_LIB+x} ] && echo ADDLIB="-Wl,-rpath,$SSL_LIB")
> > 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
> >
> > I did not notice this "-j3", shouldn't we use "-j$(nproc)" instead so
> > that we use all CPUs made available to us ?
> >
>
> it was a long ago, "-j3" appeared to be faster in my tests than $(nproc)
> (travis uses 2 cores).
> I do not have particular opinion. we can switch to $(nproc)
OK, I left it unchanged then.
Both patches merged, thank you!
Willy
Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
чт, 30 июл. 2020 г. в 07:48, Willy Tarreau :
> Hi Ilya,
>
> thanks for this. Two minor comments (just let me know, I can adjust
> the patches before merging them if you confirm):
>
> >- os: linux
> > if: type == cron
> > compiler: clang
> > env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2
> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
> > +name: libressl-2.9.2 | prometheus-exporte
>
> I guess this was "prometheus-exporter" above (missing "r") ?
>
oops.
can you please fix it during merge ?
>
> > - - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS"
> ADDLIB="-Wl,-rpath,$SSL_LIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC"
> EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
> > + - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z
> ${SSL_LIB+x} ] && echo ADDLIB="-Wl,-rpath,$SSL_LIB")
> 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
>
> I did not notice this "-j3", shouldn't we use "-j$(nproc)" instead so
> that we use all CPUs made available to us ?
>
it was a long ago, "-j3" appeared to be faster in my tests than $(nproc)
(travis uses 2 cores).
I do not have particular opinion. we can switch to $(nproc)
>
> Willy
>
Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
Hi Ilya,
thanks for this. Two minor comments (just let me know, I can adjust
the patches before merging them if you confirm):
>- os: linux
> if: type == cron
> compiler: clang
> env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2
> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
> +name: libressl-2.9.2 | prometheus-exporte
I guess this was "prometheus-exporter" above (missing "r") ?
> - - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET $FLAGS
> DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" ADDLIB="-Wl,-rpath,$SSL_LIB"
> 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
> + - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET $FLAGS
> DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z ${SSL_LIB+x} ] &&
> echo ADDLIB="-Wl,-rpath,$SSL_LIB") 51DEGREES_SRC="$FIFTYONEDEGREES_SRC"
> EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
I did not notice this "-j3", shouldn't we use "-j$(nproc)" instead so
that we use all CPUs made available to us ?
Willy
[PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
Hello,
I did not like SSL_LIB, SSL_INC after arm64 switched to stock lib (in an
ugly way).
let us make stock lib first class citizen.
Cheers,
Ilya Shipitcin
From 6ff056e7ded7784fcb4020ea8f37b12b23c855b6 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Thu, 30 Jul 2020 01:47:55 +0500
Subject: [PATCH 2/2] CI: travis-ci : switch to stock openssl when
openssl-1.1.1 is used
instead of building openssl-1.1.1 let us use stock package and save
some electricity
---
.travis.yml | 19 ---
1 file changed, 16 insertions(+), 3 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 3ee52f613..89ab24397 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -33,56 +33,67 @@ matrix:
if: type == cron
compiler: gcc
env: TARGET=linux-glibc OPENSSL_VERSION=1.0.2u
+name: openssl-1.0.2
- os: linux
arch: amd64
if: type == push
compiler: clang
-env: TARGET=linux-glibc OPENSSL_VERSION=1.1.1f CC=clang-9
+env: TARGET=linux-glibc CC=clang-9
+name: openssl-1.1.1
- os: linux
arch: arm64
if: type == push
compiler: clang
env: TARGET=linux-glibc CC=clang-9
+name: openssl-1.1.1
- os: linux
arch: s390x
if: type == push
compiler: gcc
-env: TARGET=linux-glibc OPENSSL_VERSION=1.1.1f
+env: TARGET=linux-glibc
+name: openssl-1.1.1
- os: linux
if: type == cron
compiler: clang
-env: TARGET=linux-glibc OPENSSL_VERSION=1.1.1f COVERITY_SCAN_PROJECT_NAME="Haproxy" COVERITY_SCAN_BRANCH_PATTERN="*" COVERITY_SCAN_NOTIFICATION_EMAIL="chipits...@gmail.com" COVERITY_SCAN_BUILD_COMMAND="make CC=clang TARGET=$TARGET $FLAGS 51DEGREES_SRC=$FIFTYONEDEGREES_SRC"
+env: TARGET=linux-glibc COVERITY_SCAN_PROJECT_NAME="Haproxy" COVERITY_SCAN_BRANCH_PATTERN="*" COVERITY_SCAN_NOTIFICATION_EMAIL="chipits...@gmail.com" COVERITY_SCAN_BUILD_COMMAND="make CC=clang TARGET=$TARGET $FLAGS 51DEGREES_SRC=$FIFTYONEDEGREES_SRC"
script:
- |
if [ ! -z ${COVERITY_SCAN_TOKEN+x} ]; then
make -C contrib/wurfl
curl -s "https://scan.coverity.com/scripts/travisci_build_coverity_scan.sh; | bash || true
fi
+name: openssl-1.1.1
- os: linux
if: type == cron
compiler: clang
env: TARGET=linux-glibc OPENSSL_VERSION=1.1.0l FIFTYONEDEGREES_SRC="contrib/51d/src/trie" CC=clang-9
+name: openssl-1.1.1 | 51d trie
- os: linux
if: type == push
compiler: clang
env: TARGET=linux-glibc LIBRESSL_VERSION=3.1.1 CC=clang-9
+name: libressl-3.1.1
- os: linux
env: DEBUG_OPTIONS=""
if: type == cron
compiler: clang
env: TARGET=linux-glibc LIBRESSL_VERSION=3.0.2 CC=clang-9
+name: libressl-3.0.2 | ERR=
- os: linux
if: type == cron
compiler: clang
env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2 EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
+name: libressl-2.9.2 | prometheus-exporte
- os: linux
if: type == cron
compiler: clang
env: TARGET=linux-glibc BORINGSSL=yes
+name: boringssl
- os: linux
if: type == push
compiler: clang
env: TARGET=linux-glibc FLAGS= CC=clang-9
+name: FLAGS=
- os: osx
osx_image: xcode12
if: type == push
@@ -91,6 +102,7 @@ matrix:
- echo 'brew "socat"' > brew.bundle
- brew bundle --file=brew.bundle
env: TARGET=osx FLAGS="USE_OPENSSL=1" OPENSSL_VERSION=1.1.1f
+name: openssl-1.1.1
- os: linux
if: type == cron
compiler: clang
@@ -98,6 +110,7 @@ matrix:
before_script:
- git clone https://github.com/wtarreau/libslz
- cd libslz && make && make PREFIX=${HOME}/opt install && cd ..
+name: openssl-1.1.1 | slz | pcre2
allow_failures:
- os: linux
arch: arm64
--
2.26.2
From 00b0146b626c66a0fde8f90967db7fd8d83c79e9 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Thu, 30 Jul 2020 01:39:24 +0500
Subject: [PATCH 1/2] CI : travis-ci : prepare for using stock OpenSSL
initially SSL_LIB and SSL_INC were set globally and we assumed
that any OpenSSL variant is supposed to be built using "script/build-ssl.sh".
starting with ARM64 build we use stock openssl, also it makes sense
to use stock openssl for 1.1.1 builds for velocity sake.
Let us make stock openssl lib first class citizen.
SSL_LIB and SSL_INC are only set when custom openssl variant
is built.
---
.travis.yml | 17 +++--
1 file changed, 7 insertions(+), 10 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index cf227b3a5..3ee52f613 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,8 +12,6 @@ branches:
env:
global:
- FLAGS="USE_ZLIB=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_LUA=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_WURFL=1 WURFL_INC=contrib/wurfl WURFL_LIB=contrib/wurfl USE_DEVICEATLAS=1 DEVICEATLAS_SRC=contrib/deviceatlas USE_51DEGREES=1"
-- SSL_LIB=${HOME}/opt/lib
-- SSL_INC=${HOME}/opt/include
- TMPDIR=/tmp
- FIFTYONEDEGREES_SRC="contrib/51d/src/pattern"
-
Re: [PATCH] CI: Expand use of GitHub Actions for CI
On Wed, Jul 29, 2020 at 10:22:33PM +0500, Илья Шипицин wrote: > Tim, > > there's also gitlab official mirror (ran by William) > https://gitlab.com/haproxy.org/haproxy > > > in theory we can use gitlab-ci as well :) > I don't want this to be considered an "official" mirror, I deliberately wrote "unofficial mirror" everywhere in the description because it was used for test purposes at some point, but we decided to use github instead. -- William Lallemand
Re: [PATCH] CI: Expand use of GitHub Actions for CI
ср, 29 июл. 2020 г. в 22:33, Tim Düsterhus : > Ilya, > > Am 29.07.20 um 19:05 schrieb Илья Шипицин: > >> Known issues: > >> - Apparently the git commit is not properly detected during build. The > HEAD > >> currently shows as 2.3-dev1. > >> > > > > there's something wrong with git checkout (or the way we use it) > > > > have a look (do not ask me why, I've no idea why it is 2.0-dev...) > > > > Run ./haproxy -vv > > HA-Proxy version 2.0-dev2-f7c257-3122 2020/07/29 - https://haproxy.org/ > > Status: development branch - not safe for use in production. > > > > > https://github.com/chipitsine/haproxy/runs/924210892?check_suite_focus=true > > > > That's interesting. I am suspecting that the shallow clones are at fault > here. That's why I attempted to increase the fetch-depth to 100. Travis > uses 50. Unfortunately that did not help. We'll have to investigate this. > the same "h2spec" in "haproxy/haproxy" runs against current master. I think it might be something "cached" (??) or "actions/checkout@v2" (why do we use v1 ??) or maybe https://github.com/actions/checkout#Checkout-HEAD ?? as for migrating to GH actions in general, I suggest the following 1) I will have a look at red builds (I did not expect red LibreSSL builds) 2) I'll run arm64/s390/ppc64le for big matrix to see if there are red builds 3) can we track "pro et contra" of travis <--> GH in dedicated GH issue ? issue is much like mailing list, but it is pinned > > Or maybe someone knows the necessary git internals on top of their head > and can chime in :-) > > Best regards > Tim Düsterhus >
Re: SLZ vs ZLIB
Hello, On Wed, 29 Jul 2020 at 19:19, Илья Шипицин wrote: > however, ZLIB is enabled by default in many distros and docker images. > any idea why ZLIB is chosen by default ? Because zlib is known, packaged and used everywhere and by everyone while slz is a niche library. It would need a marketing campaign I guess. lukas
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, Am 29.07.20 um 19:05 schrieb Илья Шипицин: >> Known issues: >> - Apparently the git commit is not properly detected during build. The HEAD >> currently shows as 2.3-dev1. >> > > there's something wrong with git checkout (or the way we use it) > > have a look (do not ask me why, I've no idea why it is 2.0-dev...) > > Run ./haproxy -vv > HA-Proxy version 2.0-dev2-f7c257-3122 2020/07/29 - https://haproxy.org/ > Status: development branch - not safe for use in production. > > https://github.com/chipitsine/haproxy/runs/924210892?check_suite_focus=true > That's interesting. I am suspecting that the shallow clones are at fault here. That's why I attempted to increase the fetch-depth to 100. Travis uses 50. Unfortunately that did not help. We'll have to investigate this. Or maybe someone knows the necessary git internals on top of their head and can chime in :-) Best regards Tim Düsterhus
Re: [PATCH] CI: Expand use of GitHub Actions for CI
ср, 29 июл. 2020 г. в 22:27, Tim Düsterhus : > Ilya, > > Am 29.07.20 um 19:22 schrieb Илья Шипицин: > > there's also gitlab official mirror (ran by William) > > https://gitlab.com/haproxy.org/haproxy > > > > > > in theory we can use gitlab-ci as well :) > > > > Please no more external services! I've specifically reimplemented the > Travis tests in GitHub Actions, because they are native to GitHub and > pretty awesome because of the dynamic matrix generation support. I am > hoping that we can turn off Travis sooner rather than later. > travis offers arm64, ppc64le and s390. GH offers only amd64 and osx. > > Best regards > Tim Düsterhus >
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya,
Am 29.07.20 um 18:55 schrieb Илья Шипицин:
>> +for ssl in [
>> + "stock",
>> + "OPENSSL_VERSION=1.0.2u",
>> + "OPENSSL_VERSION=1.1.0l",
>> + "OPENSSL_VERSION=1.1.1f",
>> + "LIBRESSL_VERSION=2.9.2",
>> + "LIBRESSL_VERSION=3.0.2",
>> + "LIBRESSL_VERSION=3.1.1",
>> +]:
>> + flags = ["USE_OPENSSL=1"]
>> + if ssl != "stock":
>> +flags.append("SSL_LIB=${HOME}/opt/lib")
>> +flags.append("SSL_INC=${HOME}/opt/include")
>>
>
> I do not like that polluting assignment. I know where it came from, it was
> me :)
> I'm going to get rid of it, hope in 1-2 weeks.
>
Yes, I copied that from Travis, because the workflow aims to be a Travis
CI replacement.
Best regards
Tim Düsterhus
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, Am 29.07.20 um 19:22 schrieb Илья Шипицин: > there's also gitlab official mirror (ran by William) > https://gitlab.com/haproxy.org/haproxy > > > in theory we can use gitlab-ci as well :) > Please no more external services! I've specifically reimplemented the Travis tests in GitHub Actions, because they are native to GitHub and pretty awesome because of the dynamic matrix generation support. I am hoping that we can turn off Travis sooner rather than later. Best regards Tim Düsterhus
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, Am 29.07.20 um 18:58 schrieb Илья Шипицин: > generally, there was an idea of keeping "developer velocity" as high as > possible. > i.e. to perform check in 3-5 min after the push. > > Github Actions allows 4 parallel builds (same as Travis CI), I thought of > moving some builds to GH actions. > also, that was a reason to move some rare configurations to "cron" trigger. > > Big matrix takes 30 min. It is not acceptable. > GitHub Actions allows 20 concurrent jobs: https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#usage-limits And with self hosted runners (maybe HAProxy Technologies is willing to sponsor some?) there is no limit. Anyhow: The Workflow can be optimized for sure. It currently performs no caching at all. But it still gets 1:45 minutes for a simple gcc build with all feature flags and 3:09 minutes for gcc with a custom OpenSSL. That's super fast. The Mac OS one is pretty slow, because all the VTest tests keep timing out for some reason. It will be faster if the tests pass. Overall the whole workflow shows as 8:10 minutes for me which is in "3 to 5 minute" territory if some optimizations are performed. Best regards Tim Düsterhus
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Tim,
there's also gitlab official mirror (ran by William)
https://gitlab.com/haproxy.org/haproxy
in theory we can use gitlab-ci as well :)
ср, 29 июл. 2020 г. в 21:49, Tim Duesterhus :
> Travis is becoming overall increasingly unreliable lately. We've already
> seen that the timing sensitive tests regularly fail and thus they were
> disabled.
>
> GitHub Actions VMs are working well, possibly allowing to use custom
> runners
> for special tasks in the future.
>
> In addition to this better performance its workflow configuration language
> is more expressive compared to the Travis CI one. Specifically the build
> matrix does not need to be specified in YAML. Instead it can be generated
> ad-hoc using a script. This allows us to cleanly define the various build
> configurations without having an unreadable 80 line mess where the flags
> are inconsistently activated. As an example in the current Travis CI
> configuration the prometheus exporter is tested together with LibreSSL
> 2.9.2
> for whatever reason.
>
> In addition to all the previous points the UI of Travis is not that nice.
> On GitHub you are just seeing that "Travis failed" without any details
> which
> exact job failed. This requires you to visit the slow Travis page and look
> up the details there. GitHub Actions creates a single entry for each
> configuration that is tested, allowing you to see the details without
> needing
> to leave GitHub.
>
> This new GitHub Actions workflow aims to reproduce the configurations
> tested
> in Travis. It comes close, but is not completely there, yet. Consider this
> patch a proof of concept that will evolve in the future, ideally with
> Ilya's
> expertise.
>
> The current configurations are as follows. Each one is tested with both gcc
> and clang.
> - All features disabled (no USE flags)
> - All features enabled (all USE flags)
> - Standalone test of each of the supported compression libraries:
> - USE_ZLIB=1
> - USE_SLZ=1
> - Standalone test of various SSL libraries:
> - stock (the SSL on the VM)
> - OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
> - LibreSSL 2.9.2, 3.0.2, 3.1.1
>
> Future additions of new tests should take care to not test unrelated stuff.
> Instead a distinct configuration should be added.
>
> Additionally there is a Mac OS test with clang and all features disabled.
>
> Known issues:
> - Apparently the git commit is not properly detected during build. The HEAD
> currently shows as 2.3-dev1.
> - Mac OS currently fails.
> - The detailed logs are not printed for failing tests.
> ---
> .github/workflows/vtest.yml | 175
> 1 file changed, 175 insertions(+)
> create mode 100644 .github/workflows/vtest.yml
>
> diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
> new file mode 100644
> index 0..d94322474
> --- /dev/null
> +++ b/.github/workflows/vtest.yml
> @@ -0,0 +1,175 @@
> +# Copyright 2019 Ilya Shipitsin
> +# Copyright 2020 Tim Duesterhus
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License
> +# as published by the Free Software Foundation; either version
> +# 2 of the License, or (at your option) any later version.
> +
> +name: VTest
> +
> +on:
> + push:
> +
> +jobs:
> + generate-matrix:
> +name: Generate Build Matrix
> +runs-on: ubuntu-latest
> +outputs:
> + matrix: ${{ steps.set-matrix.outputs.matrix }}
> +steps:
> + - name: Generate Build Matrix
> +id: set-matrix
> +shell: python
> +run: |
> + import json
> +
> + matrix = []
> + os = "ubuntu-latest"
> + TARGET = "linux-glibc"
> + for CC in [ "gcc", "clang" ]:
> +matrix.append({
> + "name": "{} with no features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": ""
> +})
> +
> +matrix.append({
> + "name": "{} with all features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": " ".join([
> +"USE_ZLIB=1",
> +"USE_PCRE=1",
> +"USE_PCRE_JIT=1",
> +"USE_LUA=1",
> +"USE_OPENSSL=1",
> +"USE_SYSTEMD=1",
> + # "USE_WURFL=1",
> + # "WURFL_INC=contrib/wurfl",
> + # "WURFL_LIB=contrib/wurfl",
> + # "USE_DEVICEATLAS=1",
> + # "DEVICEATLAS_SRC=contrib/deviceatlas",
> + # "USE_51DEGREES=1",
> + # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern",
> + ])
> +})
> +
> +for compression in [
> + "USE_SLZ=1",
> + "USE_ZLIB=1"
> +]:
> + matrix.append({
> +"name": "{} with {} on
SLZ vs ZLIB
Hello, I am running pretty big LB installation with http compression and SSL termination enabled. I was very surprised that 70-80% of cpu is spent on "gzip". Also, I found that SLZ is much better than ZLIB in terms of cpu usage. however, ZLIB is enabled by default in many distros and docker images. any idea why ZLIB is chosen by default ? cheers, Ilya Shipitcin
Re: [PATCH] CI: Expand use of GitHub Actions for CI
ср, 29 июл. 2020 г. в 21:49, Tim Duesterhus :
> Travis is becoming overall increasingly unreliable lately. We've already
> seen that the timing sensitive tests regularly fail and thus they were
> disabled.
>
> GitHub Actions VMs are working well, possibly allowing to use custom
> runners
> for special tasks in the future.
>
> In addition to this better performance its workflow configuration language
> is more expressive compared to the Travis CI one. Specifically the build
> matrix does not need to be specified in YAML. Instead it can be generated
> ad-hoc using a script. This allows us to cleanly define the various build
> configurations without having an unreadable 80 line mess where the flags
> are inconsistently activated. As an example in the current Travis CI
> configuration the prometheus exporter is tested together with LibreSSL
> 2.9.2
> for whatever reason.
>
> In addition to all the previous points the UI of Travis is not that nice.
> On GitHub you are just seeing that "Travis failed" without any details
> which
> exact job failed. This requires you to visit the slow Travis page and look
> up the details there. GitHub Actions creates a single entry for each
> configuration that is tested, allowing you to see the details without
> needing
> to leave GitHub.
>
> This new GitHub Actions workflow aims to reproduce the configurations
> tested
> in Travis. It comes close, but is not completely there, yet. Consider this
> patch a proof of concept that will evolve in the future, ideally with
> Ilya's
> expertise.
>
> The current configurations are as follows. Each one is tested with both gcc
> and clang.
> - All features disabled (no USE flags)
> - All features enabled (all USE flags)
> - Standalone test of each of the supported compression libraries:
> - USE_ZLIB=1
> - USE_SLZ=1
> - Standalone test of various SSL libraries:
> - stock (the SSL on the VM)
> - OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
> - LibreSSL 2.9.2, 3.0.2, 3.1.1
>
> Future additions of new tests should take care to not test unrelated stuff.
> Instead a distinct configuration should be added.
>
> Additionally there is a Mac OS test with clang and all features disabled.
>
> Known issues:
> - Apparently the git commit is not properly detected during build. The HEAD
> currently shows as 2.3-dev1.
>
there's something wrong with git checkout (or the way we use it)
have a look (do not ask me why, I've no idea why it is 2.0-dev...)
Run ./haproxy -vv
HA-Proxy version 2.0-dev2-f7c257-3122 2020/07/29 - https://haproxy.org/
Status: development branch - not safe for use in production.
https://github.com/chipitsine/haproxy/runs/924210892?check_suite_focus=true
> - Mac OS currently fails.
> - The detailed logs are not printed for failing tests.
> ---
> .github/workflows/vtest.yml | 175
> 1 file changed, 175 insertions(+)
> create mode 100644 .github/workflows/vtest.yml
>
> diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
> new file mode 100644
> index 0..d94322474
> --- /dev/null
> +++ b/.github/workflows/vtest.yml
> @@ -0,0 +1,175 @@
> +# Copyright 2019 Ilya Shipitsin
> +# Copyright 2020 Tim Duesterhus
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License
> +# as published by the Free Software Foundation; either version
> +# 2 of the License, or (at your option) any later version.
> +
> +name: VTest
> +
> +on:
> + push:
> +
> +jobs:
> + generate-matrix:
> +name: Generate Build Matrix
> +runs-on: ubuntu-latest
> +outputs:
> + matrix: ${{ steps.set-matrix.outputs.matrix }}
> +steps:
> + - name: Generate Build Matrix
> +id: set-matrix
> +shell: python
> +run: |
> + import json
> +
> + matrix = []
> + os = "ubuntu-latest"
> + TARGET = "linux-glibc"
> + for CC in [ "gcc", "clang" ]:
> +matrix.append({
> + "name": "{} with no features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": ""
> +})
> +
> +matrix.append({
> + "name": "{} with all features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": " ".join([
> +"USE_ZLIB=1",
> +"USE_PCRE=1",
> +"USE_PCRE_JIT=1",
> +"USE_LUA=1",
> +"USE_OPENSSL=1",
> +"USE_SYSTEMD=1",
> + # "USE_WURFL=1",
> + # "WURFL_INC=contrib/wurfl",
> + # "WURFL_LIB=contrib/wurfl",
> + # "USE_DEVICEATLAS=1",
> + # "DEVICEATLAS_SRC=contrib/deviceatlas",
> + # "USE_51DEGREES=1",
> + # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern",
> +
Re: [PATCH] CI: Expand use of GitHub Actions for CI
generally, there was an idea of keeping "developer velocity" as high as possible. i.e. to perform check in 3-5 min after the push. Github Actions allows 4 parallel builds (same as Travis CI), I thought of moving some builds to GH actions. also, that was a reason to move some rare configurations to "cron" trigger. Big matrix takes 30 min. It is not acceptable. ср, 29 июл. 2020 г. в 21:53, Tim Düsterhus : > Ilya, > List, > > Am 29.07.20 um 18:49 schrieb Tim Duesterhus: > > [long snip] > > You can find an example run of that GitHub Action workflow here: > > https://github.com/TimWolla/haproxy/actions/runs/187396816 > > I've attached a screenshot of how nice it looks in the commit overview > compared to Travis. > > The build output itself also is much cleaner, because it clearly > separates the steps and even allows to collapse substeps (e.g. the ldd > output): > > https://github.com/TimWolla/haproxy/runs/924096838?check_suite_focus=true > > Ilya, feel free to adjust around in that workflow file. But please take > care to keep it as clean as possible. I hope that the Python support > helps here. The Travis one is a small mess to be honest :-) > > Best regards > Tim Düsterhus >
Re: [PATCH] CI: Expand use of GitHub Actions for CI
ср, 29 июл. 2020 г. в 21:49, Tim Duesterhus :
> Travis is becoming overall increasingly unreliable lately. We've already
> seen that the timing sensitive tests regularly fail and thus they were
> disabled.
>
> GitHub Actions VMs are working well, possibly allowing to use custom
> runners
> for special tasks in the future.
>
> In addition to this better performance its workflow configuration language
> is more expressive compared to the Travis CI one. Specifically the build
> matrix does not need to be specified in YAML. Instead it can be generated
> ad-hoc using a script. This allows us to cleanly define the various build
> configurations without having an unreadable 80 line mess where the flags
> are inconsistently activated. As an example in the current Travis CI
> configuration the prometheus exporter is tested together with LibreSSL
> 2.9.2
> for whatever reason.
>
> In addition to all the previous points the UI of Travis is not that nice.
> On GitHub you are just seeing that "Travis failed" without any details
> which
> exact job failed. This requires you to visit the slow Travis page and look
> up the details there. GitHub Actions creates a single entry for each
> configuration that is tested, allowing you to see the details without
> needing
> to leave GitHub.
>
> This new GitHub Actions workflow aims to reproduce the configurations
> tested
> in Travis. It comes close, but is not completely there, yet. Consider this
> patch a proof of concept that will evolve in the future, ideally with
> Ilya's
> expertise.
>
> The current configurations are as follows. Each one is tested with both gcc
> and clang.
> - All features disabled (no USE flags)
> - All features enabled (all USE flags)
> - Standalone test of each of the supported compression libraries:
> - USE_ZLIB=1
> - USE_SLZ=1
> - Standalone test of various SSL libraries:
> - stock (the SSL on the VM)
> - OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
> - LibreSSL 2.9.2, 3.0.2, 3.1.1
>
> Future additions of new tests should take care to not test unrelated stuff.
> Instead a distinct configuration should be added.
>
> Additionally there is a Mac OS test with clang and all features disabled.
>
> Known issues:
> - Apparently the git commit is not properly detected during build. The HEAD
> currently shows as 2.3-dev1.
> - Mac OS currently fails.
> - The detailed logs are not printed for failing tests.
> ---
> .github/workflows/vtest.yml | 175
> 1 file changed, 175 insertions(+)
> create mode 100644 .github/workflows/vtest.yml
>
> diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
> new file mode 100644
> index 0..d94322474
> --- /dev/null
> +++ b/.github/workflows/vtest.yml
> @@ -0,0 +1,175 @@
> +# Copyright 2019 Ilya Shipitsin
> +# Copyright 2020 Tim Duesterhus
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License
> +# as published by the Free Software Foundation; either version
> +# 2 of the License, or (at your option) any later version.
> +
> +name: VTest
> +
> +on:
> + push:
> +
> +jobs:
> + generate-matrix:
> +name: Generate Build Matrix
> +runs-on: ubuntu-latest
> +outputs:
> + matrix: ${{ steps.set-matrix.outputs.matrix }}
> +steps:
> + - name: Generate Build Matrix
> +id: set-matrix
> +shell: python
> +run: |
> + import json
> +
> + matrix = []
> + os = "ubuntu-latest"
> + TARGET = "linux-glibc"
> + for CC in [ "gcc", "clang" ]:
> +matrix.append({
> + "name": "{} with no features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": ""
> +})
> +
> +matrix.append({
> + "name": "{} with all features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": " ".join([
> +"USE_ZLIB=1",
> +"USE_PCRE=1",
> +"USE_PCRE_JIT=1",
> +"USE_LUA=1",
> +"USE_OPENSSL=1",
> +"USE_SYSTEMD=1",
> + # "USE_WURFL=1",
> + # "WURFL_INC=contrib/wurfl",
> + # "WURFL_LIB=contrib/wurfl",
> + # "USE_DEVICEATLAS=1",
> + # "DEVICEATLAS_SRC=contrib/deviceatlas",
> + # "USE_51DEGREES=1",
> + # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern",
> + ])
> +})
> +
> +for compression in [
> + "USE_SLZ=1",
> + "USE_ZLIB=1"
> +]:
> + matrix.append({
> +"name": "{} with {} on {}".format(CC, compression, os),
> +"os": os,
> +"TARGET": TARGET,
> +"CC": CC,
> +
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, List, Am 29.07.20 um 18:49 schrieb Tim Duesterhus: > [long snip] You can find an example run of that GitHub Action workflow here: https://github.com/TimWolla/haproxy/actions/runs/187396816 I've attached a screenshot of how nice it looks in the commit overview compared to Travis. The build output itself also is much cleaner, because it clearly separates the steps and even allows to collapse substeps (e.g. the ldd output): https://github.com/TimWolla/haproxy/runs/924096838?check_suite_focus=true Ilya, feel free to adjust around in that workflow file. But please take care to keep it as clean as possible. I hope that the Python support helps here. The Travis one is a small mess to be honest :-) Best regards Tim Düsterhus
[PATCH] CI: Expand use of GitHub Actions for CI
Travis is becoming overall increasingly unreliable lately. We've already
seen that the timing sensitive tests regularly fail and thus they were
disabled.
GitHub Actions VMs are working well, possibly allowing to use custom runners
for special tasks in the future.
In addition to this better performance its workflow configuration language
is more expressive compared to the Travis CI one. Specifically the build
matrix does not need to be specified in YAML. Instead it can be generated
ad-hoc using a script. This allows us to cleanly define the various build
configurations without having an unreadable 80 line mess where the flags
are inconsistently activated. As an example in the current Travis CI
configuration the prometheus exporter is tested together with LibreSSL 2.9.2
for whatever reason.
In addition to all the previous points the UI of Travis is not that nice.
On GitHub you are just seeing that "Travis failed" without any details which
exact job failed. This requires you to visit the slow Travis page and look
up the details there. GitHub Actions creates a single entry for each
configuration that is tested, allowing you to see the details without needing
to leave GitHub.
This new GitHub Actions workflow aims to reproduce the configurations tested
in Travis. It comes close, but is not completely there, yet. Consider this
patch a proof of concept that will evolve in the future, ideally with Ilya's
expertise.
The current configurations are as follows. Each one is tested with both gcc
and clang.
- All features disabled (no USE flags)
- All features enabled (all USE flags)
- Standalone test of each of the supported compression libraries:
- USE_ZLIB=1
- USE_SLZ=1
- Standalone test of various SSL libraries:
- stock (the SSL on the VM)
- OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
- LibreSSL 2.9.2, 3.0.2, 3.1.1
Future additions of new tests should take care to not test unrelated stuff.
Instead a distinct configuration should be added.
Additionally there is a Mac OS test with clang and all features disabled.
Known issues:
- Apparently the git commit is not properly detected during build. The HEAD
currently shows as 2.3-dev1.
- Mac OS currently fails.
- The detailed logs are not printed for failing tests.
---
.github/workflows/vtest.yml | 175
1 file changed, 175 insertions(+)
create mode 100644 .github/workflows/vtest.yml
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
new file mode 100644
index 0..d94322474
--- /dev/null
+++ b/.github/workflows/vtest.yml
@@ -0,0 +1,175 @@
+# Copyright 2019 Ilya Shipitsin
+# Copyright 2020 Tim Duesterhus
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License
+# as published by the Free Software Foundation; either version
+# 2 of the License, or (at your option) any later version.
+
+name: VTest
+
+on:
+ push:
+
+jobs:
+ generate-matrix:
+name: Generate Build Matrix
+runs-on: ubuntu-latest
+outputs:
+ matrix: ${{ steps.set-matrix.outputs.matrix }}
+steps:
+ - name: Generate Build Matrix
+id: set-matrix
+shell: python
+run: |
+ import json
+
+ matrix = []
+ os = "ubuntu-latest"
+ TARGET = "linux-glibc"
+ for CC in [ "gcc", "clang" ]:
+matrix.append({
+ "name": "{} with no features enabled on {}".format(CC, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "FLAGS": ""
+})
+
+matrix.append({
+ "name": "{} with all features enabled on {}".format(CC, os),
+ "os": os,
+ "TARGET": TARGET,
+ "CC": CC,
+ "FLAGS": " ".join([
+"USE_ZLIB=1",
+"USE_PCRE=1",
+"USE_PCRE_JIT=1",
+"USE_LUA=1",
+"USE_OPENSSL=1",
+"USE_SYSTEMD=1",
+ # "USE_WURFL=1",
+ # "WURFL_INC=contrib/wurfl",
+ # "WURFL_LIB=contrib/wurfl",
+ # "USE_DEVICEATLAS=1",
+ # "DEVICEATLAS_SRC=contrib/deviceatlas",
+ # "USE_51DEGREES=1",
+ # "FIFTYONEDEGREES_SRC=contrib/51d/src/pattern",
+ ])
+})
+
+for compression in [
+ "USE_SLZ=1",
+ "USE_ZLIB=1"
+]:
+ matrix.append({
+"name": "{} with {} on {}".format(CC, compression, os),
+"os": os,
+"TARGET": TARGET,
+"CC": CC,
+"compression": compression,
+"FLAGS": compression
+ })
+
+for ssl in [
+ "stock",
+ "OPENSSL_VERSION=1.0.2u",
+ "OPENSSL_VERSION=1.1.0l",
+ "OPENSSL_VERSION=1.1.1f",
+
Re: Several CVEs in Lua 5.4
Hello, On Wed, 29 Jul 2020 at 11:16, Froehlich, Dominik wrote: > > Hi Lukas, > > Thanks for the reply. > My query goes along the lines of which Lua version is compatible with HAproxy > and contains fixes to those CVEs. > I could not find a specific instruction as to which Lua version can be used > to build HAproxy / has been tested for production use. Currently LUA 5.3 is supported, patches will be committed soon for LUA 5.4 support: https://github.com/haproxy/haproxy/issues/730#issuecomment-664555213 But the way to fix this is not to rush to a new major LUA release, but to backport the fixes to LUA 5.3 instead. Lukas
Re: [PATCH] BUG/MAJOR: dns: fix null pointer dereference in snr_update_srv_status
On Tue, Jul 28, 2020 at 02:58:57PM +0200, Baptiste wrote: > Hi Jerome, > > Thanks a lot for the debugging and the fix. > This is all good and can be applied. Great, patch applied then. Thanks guys! Willy
Re: SRV records resolution failure if Authority section is present
Patch applied, thanks guys! Willy
Re: Several CVEs in Lua 5.4
On 7/29/20 11:16 AM, Froehlich, Dominik wrote: Hi Lukas, Thanks for the reply. My query goes along the lines of which Lua version is compatible with HAproxy and contains fixes to those CVEs. I could not find a specific instruction as to which Lua version can be used to build HAproxy / has been tested for production use. We are consuming a bundled version (currently HAproxy 1.9.15 with Lua 5.3.5) but I don't know if it is safe to bump the Lua version only. I don't think HAProxy works with Lua 5.4 (yet), there should be another recent thread about it. So you are stuck at 5.3.x branch for now. Best regards, Adis
Re: Several CVEs in Lua 5.4
Hi Lukas, Thanks for the reply. My query goes along the lines of which Lua version is compatible with HAproxy and contains fixes to those CVEs. I could not find a specific instruction as to which Lua version can be used to build HAproxy / has been tested for production use. We are consuming a bundled version (currently HAproxy 1.9.15 with Lua 5.3.5) but I don't know if it is safe to bump the Lua version only. Thanks and regards, D On 29.07.20, 11:06, "Lukas Tribus" wrote: Hello, On Wed, 29 Jul 2020 at 10:23, Froehlich, Dominik wrote: > > Hello everyone, > > Not sure if this is already addressed. Today I got a CVE report of several issues with Lua 5.3.5 up to 5.4. > > I believe Lua 5.4 is currently recommended to build with HAproxy 2.x? > > Before I open an issue on github I would like to ask if these are already known / addressed: I don't understand, specifically what are you asking us to do here? It's not like we ship LUA ... Lukas
Re: Several CVEs in Lua 5.4
Hello, On Wed, 29 Jul 2020 at 10:23, Froehlich, Dominik wrote: > > Hello everyone, > > Not sure if this is already addressed. Today I got a CVE report of several > issues with Lua 5.3.5 up to 5.4. > > I believe Lua 5.4 is currently recommended to build with HAproxy 2.x? > > Before I open an issue on github I would like to ask if these are already > known / addressed: I don't understand, specifically what are you asking us to do here? It's not like we ship LUA ... Lukas
Web Design & Mobile Apps Development Proposal for haproxy.com
Hello haproxy.com We are a web development company, with goals in harnessing our client's online presence by making well structured and up to date websites and mobile applications for them to explore more into the online business world. If you have a website, we would like to do a first free analysis and help you understand the pros and cons of your website. *Major services we provide:*1) Web Development 2) Web redesign 3) Website Maintenance 4) Hosting 5) Mobile apps (Ios & Android) etc. During this time of the pandemic, where everything is in shambles, we would be glad to be your online partners and help you have a better online presence at a *very affordable price*. *Best Regards* Sylvia Lewis ** *NOTE:* If you are *interested*, kindly get back to us with your most convenient number and a good date and time for us to reach out to you over a quick call so that we can have a clear discussion and be on the same page.
Several CVEs in Lua 5.4
Hello everyone, Not sure if this is already addressed. Today I got a CVE report of several issues with Lua 5.3.5 up to 5.4. I believe Lua 5.4 is currently recommended to build with HAproxy 2.x? Before I open an issue on github I would like to ask if these are already known / addressed: Lua 5.3.5 has a use-after-free in lua_upvaluejoin in lapi.c. https://nvd.nist.gov/vuln/detail/CVE-2019-6706 Lua through 5.4.0 mishandles the interaction between stack resizes and garbage collection. https://nvd.nist.gov/vuln/detail/CVE-2020-15888 Lua through 5.4.0 has a getobjname heap-based buffer over-read because youngcollection in lgc.c uses markold for an insufficient number of list members. https://nvd.nist.gov/vuln/detail/CVE-2020-15889 Best regards, D
