Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
On Fri, Jul 31, 2020 at 02:13:14AM +0500, ??? wrote: > Hello, > > I repaired SLZ build. > > it used to work earlier because of SSL_LIB, SSL_INC variables. Makes sense indeed, applied now, thanks! Willy
Re: HAProxy 1.9 Snapshot generation broken?
Hi Tim, On Thu, Jul 30, 2020 at 10:37:15PM +0200, Tim Düsterhus wrote: > Hi List, > Willy, > > either I am dumb or it appears that the generation of HAProxy 1.9 > snapshots is broken. In > http://www.haproxy.org/download/1.9/src/snapshot/ the newest tarball is > from Jun, 12th. Wow, seems you're right! However I really don't understand what's broken: when I start the script by hand it works, and the *only* difference with other working versions is exactly the branch number in the BRANCH variable :-( And permissions are the same everywhere. I'll check logs in case I find anything. Thanks! Willy
Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
Hello,
I repaired SLZ build.
it used to work earlier because of SSL_LIB, SSL_INC variables.
чт, 30 июл. 2020 г. в 16:14, Илья Шипицин :
> SLZ also copied lib and headers to ${HOME}/opt
>
>
> I do not want to use SSL_INC, SSL_LIB for SLZ.
> I will send patch tomorrow (or earlier)
>
> чт, 30 июл. 2020 г. в 07:48, Willy Tarreau :
>
>> Hi Ilya,
>>
>> thanks for this. Two minor comments (just let me know, I can adjust
>> the patches before merging them if you confirm):
>>
>> >- os: linux
>> > if: type == cron
>> > compiler: clang
>> > env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2
>> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
>> > +name: libressl-2.9.2 | prometheus-exporte
>>
>> I guess this was "prometheus-exporter" above (missing "r") ?
>>
>> > - - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
>> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS"
>> ADDLIB="-Wl,-rpath,$SSL_LIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC"
>> EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
>> > + - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
>> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z
>> ${SSL_LIB+x} ] && echo ADDLIB="-Wl,-rpath,$SSL_LIB")
>> 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
>>
>> I did not notice this "-j3", shouldn't we use "-j$(nproc)" instead so
>> that we use all CPUs made available to us ?
>>
>> Willy
>>
>
From 01cdfc7756377624af7c3f04da5e758481f03515 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Fri, 31 Jul 2020 02:05:55 +0500
Subject: [PATCH 1/2] CI: travis-ci: use better name for Coverity scan job
---
.travis.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index 7ff035472..2d2a027ce 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -62,7 +62,7 @@ matrix:
make -C contrib/wurfl
curl -s "https://scan.coverity.com/scripts/travisci_build_coverity_scan.sh; | bash || true
fi
-name: openssl-1.1.1
+name: Coverity scan | openssl-1.1.1
- os: linux
if: type == cron
compiler: clang
--
2.26.2
From 787278ea33b715f15a930fc2ab019e52fcdb2003 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Fri, 31 Jul 2020 02:08:13 +0500
Subject: [PATCH 2/2] CI: travis-ci: use proper linking flags for SLZ build
previously SSL_INC and SSL_INC were set for all builds, and SLZ lib
was linked because of those flags. After we switched SLZ build to
stock openssl lib, SSL_INC, SSL_LIB are not set anymore.
Good time to set SLZ_INC, SLZ_LIB for such builds
---
.travis.yml | 5 -
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/.travis.yml b/.travis.yml
index 2d2a027ce..ca6073ee7 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -110,6 +110,8 @@ matrix:
before_script:
- git clone https://github.com/wtarreau/libslz
- cd libslz && make && make PREFIX=${HOME}/opt install && cd ..
+ - export SLZ_INC=${HOME}/opt/include SLZ_LIB=${HOME}/opt/lib
+ - export ADDLIB="-Wl,-rpath,$SLZ_LIB"
name: openssl-1.1.1 | slz | pcre2
allow_failures:
- os: linux
@@ -125,12 +127,13 @@ install:
if [ ! -z ${OPENSSL_VERSION+x} ] || [ ! -z ${LIBRESSL_VERSION+x} ] || [ ! -z ${BORINGSSL+x} ]; then
travis_wait bash -c 'scripts/build-ssl.sh >build-ssl.log 2>&1' || (cat build-ssl.log && exit 1)
export SSL_LIB=${HOME}/opt/lib SSL_INC=${HOME}/opt/include
+ export ADDLIB="-Wl,-rpath,$SSL_LIB"
fi
script:
- if [ "${CC%-*}" = "clang" ]; then export FLAGS="$FLAGS USE_OBSOLETE_LINKER=1" DEBUG_CFLAGS="-g -fsanitize=address" LDFLAGS="-fsanitize=address"; fi
- make -C contrib/wurfl
- - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z ${SSL_LIB+x} ] && echo ADDLIB="-Wl,-rpath,$SSL_LIB") 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
+ - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" ADDLIB="$ADDLIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
- ./haproxy -vv
- if [ "${TRAVIS_OS_NAME}" = "linux" ]; then ldd haproxy; fi
- if [ "${TRAVIS_OS_NAME}" = "osx" ]; then otool -L haproxy; fi
--
2.26.2
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, Am 29.07.20 um 21:15 schrieb Илья Шипицин: > as for migrating to GH actions in general, I suggest the following > > 1) I will have a look at red builds (I did not expect red LibreSSL builds) They look like a fluke to me, the test for the field converter failed. But I did not bother to investigate for a PoC patch. > 2) I'll run arm64/s390/ppc64le for big matrix to see if there are red builds I would be fine with leaving those architectures on Travis for the time being. amd64 is by far and large the most important one and we can benefit from the improved GitHub Actions there. For the other arches a basic smoke test (does it compile?) should catch most of the issues. > 3) can we track "pro et contra" of travis <--> GH in dedicated GH issue ? > issue is much like mailing list, but it is pinned Feel free to create an issue if you feel it is helpful. I don't particularly care either way (mailing list vs issue). Best regards Tim Düsterhus
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Ilya, Am 30.07.20 um 08:50 schrieb Илья Шипицин: > as for failing osx builds, I think it might be due to long folder name > > https://github.com/TimWolla/haproxy/runs/924097118?check_suite_focus=true#step:10:361 > > it is 100+ characters. there's also unix socket in that dir, which has > limitation on filename length. > I use TMPDIR=/tmp in travis to make path shorter Ah, yeah. That makes sense, I remember that issue from Travis. > (but it would be nice to output logs on failure to see that error in log) Yes, I acknowledged that as "Known Issues". My patch is a very rough first version that will need some love. But I wanted to get some feedback as it is in an acceptable state now. Best regards Tim Düsterhus
Re: Can I help with the 2.1 release?
On Thu, Jul 30, 2020 at 10:37 PM Julien Pivotto wrote: > I'm with Lukas on this. 2.1 is a strong release, and we should be > grateful for everyone which is using that release, as their feedback is > valuable for the building the next releases of HAProxy. My apologies if the message sounded ungrateful, for rolling out new changes and testing. As the latest 2.2.0 release did show just recently, there is great benefit in people running upcoming (new) changes. On Thu, Jul 30, 2020 at 10:29 PM Lukas Tribus wrote: > 2.1 is not a technical preview, it's a proper release train with full > support. Support for it will cease in 2021-Q1, but I don't think you > can conclude that that means it's getting less love now. My "technical preview" wording and the release ramp-down expectation was somewhat based on past release lines, such as the 2.1.0 ANNOUNCE saying "2.1 is a stable branch that will be maintained till around Q1 2021, and is mostly aimed at experienced users, just like 1.9 was" and the 2.0.0 ANNOUNCE saying "As most of you know, 1.9 will not be maintained for a long time and should mostly be seen as a technological preview or technical foundation for 2.0." I do recognize the 2021Q1 commitment of maintenance. If the release velocity is indeed to be expected from the team for the 2.1 line then apologies for my doubt on the priorities/time allocation. On Thu, Jul 30, 2020 at 10:37 PM Julien Pivotto wrote: > I am not yet confident to run 2.2 in prod yet, but I will roll out 2.2 > in non-prod env soon. On Thu, Jul 30, 2020 at 10:29 PM Lukas Tribus wrote: > I would be reluctant to suggest upgrading mission-critical setups to > 2.2, it's not even a month old at this point. Unless you expect to run > into bugs and have time and resources to troubleshoot it. Everyone should, of course, evaluate their upgrade strategies themselves. I did not intend that to be a general advisory to "upgrade all the things". Instead I was attempting to pose a legitimate question out of interest as to whether there are any blockers for a 2.2 LTS migration from 2.1, considering they had already upgraded from the 2.0 LTS.
Re: Can I help with the 2.1 release?
On 30 Jul 21:29, Lukas Tribus wrote: > Hello, > > On Thu, 30 Jul 2020 at 20:49, Valter Jansons wrote: > > > > On Thu, Jul 30, 2020 at 6:44 PM Harris Kaufmann > > wrote: > > > my company really needs the next 2.1 release but we want to avoid > > > deploying a custom, self compiled version. > > > > > > Is there something I can do to help with the release? I guess there > > > are no blocking issues left? > > > > For mission-critical setups you should be running the LTS release > > lines. The 2.1 release line was more of a technical preview line for > > the following 2.2 LTS release, to keep changes flowing, and you should > > not expect regular new release tags on the 2.1 line considering the > > 2.2 line has shipped. I am not involved in the release process but I > > would assume the team will push a new 2.1 tag some day however I do > > not see that being a high priority for them in any way. > > > > As a result, I would instead rephrase the question in the other > > direction: Are there any blockers for you to upgrade to 2.2? > > I'm not sure I agree. > > I would be reluctant to suggest upgrading mission-critical setups to > 2.2, it's not even a month old at this point. Unless you expect to run > into bugs and have time and resources to troubleshoot it. > > 2.1 is not a technical preview, it's a proper release train with full > support. Support for it will cease in 2021-Q1, but I don't think you > can conclude that that means it's getting less love now. > > > Lukas > I'm with Lukas on this. 2.1 is a strong release, and we should be grateful for everyone which is using that release, as their feedback is valuable for the building the next releases of HAProxy. I am not yet confident to run 2.2 in prod yet, but I will roll out 2.2 in non-prod env soon. -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Re: Can I help with the 2.1 release?
Hello, On Thu, 30 Jul 2020 at 20:49, Valter Jansons wrote: > > On Thu, Jul 30, 2020 at 6:44 PM Harris Kaufmann > wrote: > > my company really needs the next 2.1 release but we want to avoid > > deploying a custom, self compiled version. > > > > Is there something I can do to help with the release? I guess there > > are no blocking issues left? > > For mission-critical setups you should be running the LTS release > lines. The 2.1 release line was more of a technical preview line for > the following 2.2 LTS release, to keep changes flowing, and you should > not expect regular new release tags on the 2.1 line considering the > 2.2 line has shipped. I am not involved in the release process but I > would assume the team will push a new 2.1 tag some day however I do > not see that being a high priority for them in any way. > > As a result, I would instead rephrase the question in the other > direction: Are there any blockers for you to upgrade to 2.2? I'm not sure I agree. I would be reluctant to suggest upgrading mission-critical setups to 2.2, it's not even a month old at this point. Unless you expect to run into bugs and have time and resources to troubleshoot it. 2.1 is not a technical preview, it's a proper release train with full support. Support for it will cease in 2021-Q1, but I don't think you can conclude that that means it's getting less love now. Lukas
Re: Can I help with the 2.1 release?
On Thu, Jul 30, 2020 at 6:44 PM Harris Kaufmann wrote: > my company really needs the next 2.1 release but we want to avoid > deploying a custom, self compiled version. > > Is there something I can do to help with the release? I guess there > are no blocking issues left? For mission-critical setups you should be running the LTS release lines. The 2.1 release line was more of a technical preview line for the following 2.2 LTS release, to keep changes flowing, and you should not expect regular new release tags on the 2.1 line considering the 2.2 line has shipped. I am not involved in the release process but I would assume the team will push a new 2.1 tag some day however I do not see that being a high priority for them in any way. As a result, I would instead rephrase the question in the other direction: Are there any blockers for you to upgrade to 2.2?
Can I help with the 2.1 release?
Hi, my company really needs the next 2.1 release but we want to avoid deploying a custom, self compiled version. Is there something I can do to help with the release? I guess there are no blocking issues left? Best regards, Harris
Re: [PATCH] CI: rework SSL_LIB, SSL_INC in more elegant way, improve build speed by switching to stock lib for openssl-1.1.1 builds
SLZ also copied lib and headers to ${HOME}/opt
I do not want to use SSL_INC, SSL_LIB for SLZ.
I will send patch tomorrow (or earlier)
чт, 30 июл. 2020 г. в 07:48, Willy Tarreau :
> Hi Ilya,
>
> thanks for this. Two minor comments (just let me know, I can adjust
> the patches before merging them if you confirm):
>
> >- os: linux
> > if: type == cron
> > compiler: clang
> > env: TARGET=linux-glibc LIBRESSL_VERSION=2.9.2
> EXTRA_OBJS="contrib/prometheus-exporter/service-prometheus.o" CC=clang-9
> > +name: libressl-2.9.2 | prometheus-exporte
>
> I guess this was "prometheus-exporter" above (missing "r") ?
>
> > - - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS"
> ADDLIB="-Wl,-rpath,$SSL_LIB" 51DEGREES_SRC="$FIFTYONEDEGREES_SRC"
> EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
> > + - make -j3 CC=$CC CPU_CFLAGS.generic="-O1" V=1 ERR=1 TARGET=$TARGET
> $FLAGS DEBUG_CFLAGS="$DEBUG_CFLAGS" LDFLAGS="$LDFLAGS" $([ ! -z
> ${SSL_LIB+x} ] && echo ADDLIB="-Wl,-rpath,$SSL_LIB")
> 51DEGREES_SRC="$FIFTYONEDEGREES_SRC" EXTRA_OBJS="$EXTRA_OBJS" $DEBUG_OPTIONS
>
> I did not notice this "-j3", shouldn't we use "-j$(nproc)" instead so
> that we use all CPUs made available to us ?
>
> Willy
>
Re: ssl-skip-self-issued-ca option does not work
On Fri, Jul 17, 2020 at 04:47:10PM +0900, Sehoon Kim wrote: > Hi, > > ssl-skip-self-issued-ca option does not work. > commit "83cb2cb7" seems to remove this option. > > > Best regards, > Seri I suppose you were talking about commit f187ce6. There is indeed a problem there, because a previous patch was breaking the compatibility with the bundles... I'm not sure how to fix this since we can't skip the the Root CA with SSL_CTX_set1_chain. Maybe we could move it from the chain and move it to the issuer. -- William Lallemand
Re: [PATCH] CI: Expand use of GitHub Actions for CI
Tim, let us move this activity to GH issue ?
as for failing osx builds, I think it might be due to long folder name
https://github.com/TimWolla/haproxy/runs/924097118?check_suite_focus=true#step:10:361
it is 100+ characters. there's also unix socket in that dir, which has
limitation on filename length.
I use TMPDIR=/tmp in travis to make path shorter
(but it would be nice to output logs on failure to see that error in log)
ср, 29 июл. 2020 г. в 21:49, Tim Duesterhus :
> Travis is becoming overall increasingly unreliable lately. We've already
> seen that the timing sensitive tests regularly fail and thus they were
> disabled.
>
> GitHub Actions VMs are working well, possibly allowing to use custom
> runners
> for special tasks in the future.
>
> In addition to this better performance its workflow configuration language
> is more expressive compared to the Travis CI one. Specifically the build
> matrix does not need to be specified in YAML. Instead it can be generated
> ad-hoc using a script. This allows us to cleanly define the various build
> configurations without having an unreadable 80 line mess where the flags
> are inconsistently activated. As an example in the current Travis CI
> configuration the prometheus exporter is tested together with LibreSSL
> 2.9.2
> for whatever reason.
>
> In addition to all the previous points the UI of Travis is not that nice.
> On GitHub you are just seeing that "Travis failed" without any details
> which
> exact job failed. This requires you to visit the slow Travis page and look
> up the details there. GitHub Actions creates a single entry for each
> configuration that is tested, allowing you to see the details without
> needing
> to leave GitHub.
>
> This new GitHub Actions workflow aims to reproduce the configurations
> tested
> in Travis. It comes close, but is not completely there, yet. Consider this
> patch a proof of concept that will evolve in the future, ideally with
> Ilya's
> expertise.
>
> The current configurations are as follows. Each one is tested with both gcc
> and clang.
> - All features disabled (no USE flags)
> - All features enabled (all USE flags)
> - Standalone test of each of the supported compression libraries:
> - USE_ZLIB=1
> - USE_SLZ=1
> - Standalone test of various SSL libraries:
> - stock (the SSL on the VM)
> - OpenSSL 1.0.2u, 1.1.0l, 1.1.1f
> - LibreSSL 2.9.2, 3.0.2, 3.1.1
>
> Future additions of new tests should take care to not test unrelated stuff.
> Instead a distinct configuration should be added.
>
> Additionally there is a Mac OS test with clang and all features disabled.
>
> Known issues:
> - Apparently the git commit is not properly detected during build. The HEAD
> currently shows as 2.3-dev1.
> - Mac OS currently fails.
> - The detailed logs are not printed for failing tests.
> ---
> .github/workflows/vtest.yml | 175
> 1 file changed, 175 insertions(+)
> create mode 100644 .github/workflows/vtest.yml
>
> diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
> new file mode 100644
> index 0..d94322474
> --- /dev/null
> +++ b/.github/workflows/vtest.yml
> @@ -0,0 +1,175 @@
> +# Copyright 2019 Ilya Shipitsin
> +# Copyright 2020 Tim Duesterhus
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License
> +# as published by the Free Software Foundation; either version
> +# 2 of the License, or (at your option) any later version.
> +
> +name: VTest
> +
> +on:
> + push:
> +
> +jobs:
> + generate-matrix:
> +name: Generate Build Matrix
> +runs-on: ubuntu-latest
> +outputs:
> + matrix: ${{ steps.set-matrix.outputs.matrix }}
> +steps:
> + - name: Generate Build Matrix
> +id: set-matrix
> +shell: python
> +run: |
> + import json
> +
> + matrix = []
> + os = "ubuntu-latest"
> + TARGET = "linux-glibc"
> + for CC in [ "gcc", "clang" ]:
> +matrix.append({
> + "name": "{} with no features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": ""
> +})
> +
> +matrix.append({
> + "name": "{} with all features enabled on {}".format(CC, os),
> + "os": os,
> + "TARGET": TARGET,
> + "CC": CC,
> + "FLAGS": " ".join([
> +"USE_ZLIB=1",
> +"USE_PCRE=1",
> +"USE_PCRE_JIT=1",
> +"USE_LUA=1",
> +"USE_OPENSSL=1",
> +"USE_SYSTEMD=1",
> + # "USE_WURFL=1",
> + # "WURFL_INC=contrib/wurfl",
> + # "WURFL_LIB=contrib/wurfl",
> + # "USE_DEVICEATLAS=1",
> + # "DEVICEATLAS_SRC=contrib/deviceatlas",
> + # "USE_51DEGREES=1",
> +
