Re: Reproducible CI build with OpenSSL and "latest" keyword
as for reporting "what is ubuntu-latest" and "what is ssl=stock", I did not have much success yet. github does not expose that information in easy way. actually, there's build step where image version is reported, but it is collapsed [image: image.png] ср, 14 дек. 2022 г. в 19:55, Илья Шипицин : > > > ср, 14 дек. 2022 г. в 19:23, William Lallemand : > >> On Wed, Dec 14, 2022 at 06:34:26PM +0500, Илья Шипицин wrote: >> > I am attaching another patch, i.e. using "ubuntu-latest" and >> "macos-latest" >> > for development branches and fixed images for stable branches. >> > >> >> Thank you, that make sense, I'll backport it to 2.6 as well. >> >> We just need to be careful every 2 years when the ubuntu version change >> and an HAProxy release is done, not to be stuck in 22.04 :-) >> > > it is what you have asked for, if versions are fixed, someone has to keep > an eye on it :) > > >> >> -- >> William Lallemand >> >
Re: Reproducible CI build with OpenSSL and "latest" keyword
ср, 14 дек. 2022 г. в 19:23, William Lallemand : > On Wed, Dec 14, 2022 at 06:34:26PM +0500, Илья Шипицин wrote: > > I am attaching another patch, i.e. using "ubuntu-latest" and > "macos-latest" > > for development branches and fixed images for stable branches. > > > > Thank you, that make sense, I'll backport it to 2.6 as well. > > We just need to be careful every 2 years when the ubuntu version change > and an HAProxy release is done, not to be stuck in 22.04 :-) > it is what you have asked for, if versions are fixed, someone has to keep an eye on it :) > > -- > William Lallemand >
Re: Reproducible CI build with OpenSSL and "latest" keyword
On Wed, Dec 14, 2022 at 06:34:26PM +0500, Илья Шипицин wrote: > I am attaching another patch, i.e. using "ubuntu-latest" and "macos-latest" > for development branches and fixed images for stable branches. > Thank you, that make sense, I'll backport it to 2.6 as well. We just need to be careful every 2 years when the ubuntu version change and an HAProxy release is done, not to be stuck in 22.04 :-) -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
I am attaching another patch, i.e. using "ubuntu-latest" and "macos-latest"
for development branches and fixed images for stable branches.
пн, 12 дек. 2022 г. в 19:57, William Lallemand :
> On Mon, Dec 12, 2022 at 07:27:59PM +0500, Илья Шипицин wrote:
> > I attached a patch.
> >
>
> Thanks!
>
> > btw, we only build for the latest LibreSSL. are we ok to skip LibreSSL
> for
> > stable branches ?
> >
>
> In <= 2.5 we are still building with 3.5.3,
>
> http://git.haproxy.org/?p=haproxy-2.5.git;a=blob;f=.github/matrix.py;hb=HEAD#l132
>
> Ideally it would be better to still build libreSSL in stable.
>
> In my opinion there should be at least one version + the latest for this
> method to work, but if the latest is equal to an already built version
> that doesn't make sens to build it again.
>
> --
> William Lallemand
>
From 2d2679d3b18cd0d2c6e9f21cf38e9f15d1ba5cec Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Wed, 14 Dec 2022 18:18:20 +0500
Subject: [PATCH 1/2] CI: github: remove redundant ASAN loop
it was there because we only ran ASAN for clang, now no need to separate loop
---
.github/matrix.py | 67 ++-
1 file changed, 32 insertions(+), 35 deletions(-)
diff --git a/.github/matrix.py b/.github/matrix.py
index fd9491aee..93ecd3c80 100755
--- a/.github/matrix.py
+++ b/.github/matrix.py
@@ -112,6 +112,38 @@ for CC in ["gcc", "clang"]:
}
)
+# ASAN
+
+matrix.append(
+{
+"name": "{}, {}, ASAN, all features".format(clean_os(os), CC),
+"os": os,
+"TARGET": TARGET,
+"CC": CC,
+"FLAGS": get_asan_flags(CC)
++ [
+"USE_ZLIB=1",
+"USE_OT=1",
+"OT_INC=${HOME}/opt-ot/include",
+"OT_LIB=${HOME}/opt-ot/lib",
+"OT_RUNPATH=1",
+"USE_PCRE=1",
+"USE_PCRE_JIT=1",
+"USE_LUA=1",
+"USE_OPENSSL=1",
+"USE_SYSTEMD=1",
+"USE_WURFL=1",
+"WURFL_INC=addons/wurfl/dummy",
+"WURFL_LIB=addons/wurfl/dummy",
+"USE_DEVICEATLAS=1",
+"DEVICEATLAS_SRC=addons/deviceatlas/dummy",
+"USE_PROMEX=1",
+"USE_51DEGREES=1",
+"51DEGREES_SRC=addons/51degrees/dummy/pattern",
+],
+}
+)
+
for compression in ["USE_ZLIB=1"]:
matrix.append(
{
@@ -154,41 +186,6 @@ for CC in ["gcc", "clang"]:
}
)
-# ASAN
-
-os = "ubuntu-latest"
-TARGET = "linux-glibc"
-for CC in ["gcc","clang"]:
-matrix.append(
-{
-"name": "{}, {}, ASAN, all features".format(clean_os(os), CC),
-"os": os,
-"TARGET": TARGET,
-"CC": CC,
-"FLAGS": get_asan_flags(CC)
-+ [
-"USE_ZLIB=1",
-"USE_OT=1",
-"OT_INC=${HOME}/opt-ot/include",
-"OT_LIB=${HOME}/opt-ot/lib",
-"OT_RUNPATH=1",
-"USE_PCRE=1",
-"USE_PCRE_JIT=1",
-"USE_LUA=1",
-"USE_OPENSSL=1",
-"USE_SYSTEMD=1",
-"USE_WURFL=1",
-"WURFL_INC=addons/wurfl/dummy",
-"WURFL_LIB=addons/wurfl/dummy",
-"USE_DEVICEATLAS=1",
-"DEVICEATLAS_SRC=addons/deviceatlas/dummy",
-"USE_PROMEX=1",
-"USE_51DEGREES=1",
-"51DEGREES_SRC=addons/51degrees/dummy/pattern",
-],
-}
-)
-
# macOS
os = "macos-latest"
--
2.38.1
From e45cc5e13cc8559695d02ca7de7c93e85456e925 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Wed, 14 Dec 2022 18:23:24 +0500
Subject: [PATCH 2/2] CI: github: split matrix for development and stable
branches
ML ref: https://www.mail-archive.com/haproxy@formilux.org/msg42934.html
we agreed to use "latest" images for development branches and fixed
images for stable branches
---
.github/matrix.py | 25 -
1 file changed, 8 insertions(+), 17 deletions(-)
diff --git a/.github/matrix.py b/.github/matrix.py
index 93ecd3c80..ffc3414b9 100755
--- a/.github/matrix.py
+++ b/.github/matrix.py
@@ -22,15 +22,6 @@ else:
print("Generating matrix for type '{}'.".format(ref_name))
-
-def clean_os(os):
-if os == "ubuntu-latest":
-return "Ubuntu"
-elif os == "macos-latest":
-return "macOS"
-return os.replace("-latest", "")
-
-
def clean_ssl(ssl):
return ssl.replace("_VERSION", "").lower()
@@ -70,12 +61,12 @@ matrix = []
# Ubuntu
-os = "ubuntu-latest"
+os = "ubuntu-latest" if "haproxy-" not in ref_name else "ubuntu-22.04"
TARGET = "linux-glibc"
for CC in ["gcc", "clang"]:
matrix.append(
{
-"name": "{}, {}, no features".format(clean_os(os), CC),
+"name": "{}, {}, no
Re: Reproducible CI build with OpenSSL and "latest" keyword
On Mon, Dec 12, 2022 at 07:27:59PM +0500, Илья Шипицин wrote: > I attached a patch. > Thanks! > btw, we only build for the latest LibreSSL. are we ok to skip LibreSSL for > stable branches ? > In <= 2.5 we are still building with 3.5.3, http://git.haproxy.org/?p=haproxy-2.5.git;a=blob;f=.github/matrix.py;hb=HEAD#l132 Ideally it would be better to still build libreSSL in stable. In my opinion there should be at least one version + the latest for this method to work, but if the latest is equal to an already built version that doesn't make sens to build it again. -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
I attached a patch.
btw, we only build for the latest LibreSSL. are we ok to skip LibreSSL for
stable branches ?
the remaining feature requests might be addressed later, I hope
пн, 12 дек. 2022 г. в 13:03, William Lallemand :
> On Mon, Dec 12, 2022 at 08:48:06AM +0100, William Lallemand wrote:
> > Hi Ilya !
> >
> > On Mon, Dec 12, 2022 at 10:56:11AM +0500, Илья Шипицин wrote:
> > > hello,
> > >
> > > I made some prototype of I meant:
> > >
> > >
> https://github.com/chipitsine/haproxy/commit/c95955ecfd1a5b514c235b0f155bfa71178b51d5
> > >
> >
> > - We don't often use "dev" in our branches so we should build everything
> > when it's not a stable branch.
> >
> > - We don't want to build "3.0" OR latest, in fact we only need to
> > condition the "latest" build, because the other one will always be
> > built.
> >
> > So once the "3.1" is released we could add an entry for it to
> > the file and "latest" will be another version. This way we could
> > backport the "3.1" in previous branches if we want to support it.
> >
> > > I;m not sure how stable branches are named in private github ci. If
> you can
> > > enlighten me, I'll try to adopt.
> > > currently, I did the following, if branch name is either master or
> contains
> > > "dev", so "latest" semantic is chosen, fixed versions are used
> otherwise.
> > >
> >
> > The stable branches are named "haproxy-X.Y", so in my opinion we should
> > build the "latest" for anything which is not a stable branch.
> >
> > > also, I know that the same ci is used for
> > >
> > > https://github.com/haproxytech/quic-dev
> > >
> > >
> > > @Frederic Lecaille , which behaviour would
> you like
> > > for that repo ? what is branch naming convention ?
> > >
> > The same as the master branch IMHO.
> >
> > Also, the problem is uglier than I thought, we are not testing 1.1.1
> > anymore since "ubuntu-latest" was upgraded to 22.04 a few weeks ago
> > without us noticing. "ssl=stock" is now a 3.0 branch. It brokes all
> > stable branches below 2.6 because they need the deprecated SSL API.
> > I changed "ubuntu-latest" to "ubuntu-20.04" for those branches so it
> > works as earlier. I'm going to reintroduce "1.1.1" for master to 2.6 so
> > it is correctly tested again.
> >
> > In my opinion we need a similar mecanism for the distribution than for
> > the ssl libs. Maybe using "latest" only in dev branches and a fixed
> > version for stable branches will be enough.
> >
> > Regards,
> >
>
> Just thought about something, is it possible to have the versions in the
> job names ? So we don't have surprises. For example the Ubuntu version
> which was resolved by "ubuntu-latest" and the SSL version of
> "ssl=stock", we could easily see the changes this way.
>
> --
> William Lallemand
>
From d3056da0e532914fca7ff0936be34d3df3e94602 Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Mon, 12 Dec 2022 19:15:22 +0500
Subject: [PATCH] CI: split ssl lib selection based on git branch
when *SSL_VERSION="latest" behaviour was introduced, it seems to be fine
for development branches, but too intrusive for stable branches.
let us limit "latest" semantic only for development builds, if branch name
contains "haproxy-" it is supposed to be stable branch, no latest openssl
should be taken
---
.github/matrix.py | 10 --
.github/workflows/vtest.yml | 2 +-
2 files changed, 5 insertions(+), 7 deletions(-)
diff --git a/.github/matrix.py b/.github/matrix.py
index 98d0a1f2a..fd9491aee 100755
--- a/.github/matrix.py
+++ b/.github/matrix.py
@@ -15,12 +15,12 @@ import re
from os import environ
if len(sys.argv) == 2:
-build_type = sys.argv[1]
+ref_name = sys.argv[1]
else:
-print("Usage: {} ".format(sys.argv[0]), file=sys.stderr)
+print("Usage: {} ".format(sys.argv[0]), file=sys.stderr)
sys.exit(1)
-print("Generating matrix for type '{}'.".format(build_type))
+print("Generating matrix for type '{}'.".format(ref_name))
def clean_os(os):
@@ -129,11 +129,9 @@ for CC in ["gcc", "clang"]:
"stock",
"OPENSSL_VERSION=1.0.2u",
"OPENSSL_VERSION=1.1.1s",
-"OPENSSL_VERSION=latest",
-"LIBRESSL_VERSION=latest",
"QUICTLS=yes",
#"BORINGSSL=yes",
-]:
+] + (["OPENSSL_VERSION=latest", "LIBRESSL_VERSION=latest"] if "haproxy-" not in ref_name else []):
flags = ["USE_OPENSSL=1"]
if ssl == "BORINGSSL=yes" or ssl == "QUICTLS=yes" or "LIBRESSL" in ssl:
flags.append("USE_QUIC=1")
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
index fb7b1d968..a7cdcc514 100644
--- a/.github/workflows/vtest.yml
+++ b/.github/workflows/vtest.yml
@@ -26,7 +26,7 @@ jobs:
- uses: actions/checkout@v3
- name: Generate Build Matrix
id: set-matrix
-run: python3 .github/matrix.py "${{ github.event_name }}"
+run: python3 .github/matrix.py "${{ github.ref_name }}"
# The Test job actually runs the tests.
Test:
--
2.38.1
Re: Reproducible CI build with OpenSSL and "latest" keyword
On Mon, Dec 12, 2022 at 08:48:06AM +0100, William Lallemand wrote: > Hi Ilya ! > > On Mon, Dec 12, 2022 at 10:56:11AM +0500, Илья Шипицин wrote: > > hello, > > > > I made some prototype of I meant: > > > > https://github.com/chipitsine/haproxy/commit/c95955ecfd1a5b514c235b0f155bfa71178b51d5 > > > > - We don't often use "dev" in our branches so we should build everything > when it's not a stable branch. > > - We don't want to build "3.0" OR latest, in fact we only need to > condition the "latest" build, because the other one will always be > built. > > So once the "3.1" is released we could add an entry for it to > the file and "latest" will be another version. This way we could > backport the "3.1" in previous branches if we want to support it. > > > I;m not sure how stable branches are named in private github ci. If you can > > enlighten me, I'll try to adopt. > > currently, I did the following, if branch name is either master or contains > > "dev", so "latest" semantic is chosen, fixed versions are used otherwise. > > > > The stable branches are named "haproxy-X.Y", so in my opinion we should > build the "latest" for anything which is not a stable branch. > > > also, I know that the same ci is used for > > > > https://github.com/haproxytech/quic-dev > > > > > > @Frederic Lecaille , which behaviour would you like > > for that repo ? what is branch naming convention ? > > > The same as the master branch IMHO. > > Also, the problem is uglier than I thought, we are not testing 1.1.1 > anymore since "ubuntu-latest" was upgraded to 22.04 a few weeks ago > without us noticing. "ssl=stock" is now a 3.0 branch. It brokes all > stable branches below 2.6 because they need the deprecated SSL API. > I changed "ubuntu-latest" to "ubuntu-20.04" for those branches so it > works as earlier. I'm going to reintroduce "1.1.1" for master to 2.6 so > it is correctly tested again. > > In my opinion we need a similar mecanism for the distribution than for > the ssl libs. Maybe using "latest" only in dev branches and a fixed > version for stable branches will be enough. > > Regards, > Just thought about something, is it possible to have the versions in the job names ? So we don't have surprises. For example the Ubuntu version which was resolved by "ubuntu-latest" and the SSL version of "ssl=stock", we could easily see the changes this way. -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
OK, I will send v2 today (I may also reintroduce 1.1.1) On Mon, Dec 12, 2022, 1:48 PM William Lallemand wrote: > Hi Ilya ! > > On Mon, Dec 12, 2022 at 10:56:11AM +0500, Илья Шипицин wrote: > > hello, > > > > I made some prototype of I meant: > > > > > https://github.com/chipitsine/haproxy/commit/c95955ecfd1a5b514c235b0f155bfa71178b51d5 > > > > - We don't often use "dev" in our branches so we should build everything > when it's not a stable branch. > > - We don't want to build "3.0" OR latest, in fact we only need to > condition the "latest" build, because the other one will always be > built. > > So once the "3.1" is released we could add an entry for it to > the file and "latest" will be another version. This way we could > backport the "3.1" in previous branches if we want to support it. > > > I;m not sure how stable branches are named in private github ci. If you > can > > enlighten me, I'll try to adopt. > > currently, I did the following, if branch name is either master or > contains > > "dev", so "latest" semantic is chosen, fixed versions are used otherwise. > > > > The stable branches are named "haproxy-X.Y", so in my opinion we should > build the "latest" for anything which is not a stable branch. > > > also, I know that the same ci is used for > > > > https://github.com/haproxytech/quic-dev > > > > > > @Frederic Lecaille , which behaviour would you > like > > for that repo ? what is branch naming convention ? > > > The same as the master branch IMHO. > > Also, the problem is uglier than I thought, we are not testing 1.1.1 > anymore since "ubuntu-latest" was upgraded to 22.04 a few weeks ago > without us noticing. "ssl=stock" is now a 3.0 branch. It brokes all > stable branches below 2.6 because they need the deprecated SSL API. > I changed "ubuntu-latest" to "ubuntu-20.04" for those branches so it > works as earlier. I'm going to reintroduce "1.1.1" for master to 2.6 so > it is correctly tested again. > > In my opinion we need a similar mecanism for the distribution than for > the ssl libs. Maybe using "latest" only in dev branches and a fixed > version for stable branches will be enough. > > Regards, > > -- > William Lallemand >
Re: Reproducible CI build with OpenSSL and "latest" keyword
Hi Ilya ! On Mon, Dec 12, 2022 at 10:56:11AM +0500, Илья Шипицин wrote: > hello, > > I made some prototype of I meant: > > https://github.com/chipitsine/haproxy/commit/c95955ecfd1a5b514c235b0f155bfa71178b51d5 > - We don't often use "dev" in our branches so we should build everything when it's not a stable branch. - We don't want to build "3.0" OR latest, in fact we only need to condition the "latest" build, because the other one will always be built. So once the "3.1" is released we could add an entry for it to the file and "latest" will be another version. This way we could backport the "3.1" in previous branches if we want to support it. > I;m not sure how stable branches are named in private github ci. If you can > enlighten me, I'll try to adopt. > currently, I did the following, if branch name is either master or contains > "dev", so "latest" semantic is chosen, fixed versions are used otherwise. > The stable branches are named "haproxy-X.Y", so in my opinion we should build the "latest" for anything which is not a stable branch. > also, I know that the same ci is used for > > https://github.com/haproxytech/quic-dev > > > @Frederic Lecaille , which behaviour would you like > for that repo ? what is branch naming convention ? > The same as the master branch IMHO. Also, the problem is uglier than I thought, we are not testing 1.1.1 anymore since "ubuntu-latest" was upgraded to 22.04 a few weeks ago without us noticing. "ssl=stock" is now a 3.0 branch. It brokes all stable branches below 2.6 because they need the deprecated SSL API. I changed "ubuntu-latest" to "ubuntu-20.04" for those branches so it works as earlier. I'm going to reintroduce "1.1.1" for master to 2.6 so it is correctly tested again. In my opinion we need a similar mecanism for the distribution than for the ssl libs. Maybe using "latest" only in dev branches and a fixed version for stable branches will be enough. Regards, -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
hello, I made some prototype of I meant: https://github.com/chipitsine/haproxy/commit/c95955ecfd1a5b514c235b0f155bfa71178b51d5 I;m not sure how stable branches are named in private github ci. If you can enlighten me, I'll try to adopt. currently, I did the following, if branch name is either master or contains "dev", so "latest" semantic is chosen, fixed versions are used otherwise. also, I know that the same ci is used for https://github.com/haproxytech/quic-dev @Frederic Lecaille , which behaviour would you like for that repo ? what is branch naming convention ? cheers, Ilya ср, 7 дек. 2022 г. в 09:16, Илья Шипицин : > > > вт, 6 дек. 2022 г. в 23:29, Willy Tarreau : > >> On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote: >> > William, >> > >> > On 12/6/22 15:37, William Lallemand wrote: >> > > As I already mentionned, I don't really like the "latest" keyword for >> > > the OpenSSL version as it prevent us to have reproducible builds. >> > > It updates versions without warning, even major ones. >> > >> > I agree and also was not really happy with the 'latest' back when it was >> > introduced in the first place, but didn't care strongly enough to speak >> up >> > then. >> >> No problem, it's always easier to see what we've missed after it reaches >> its limits :-) >> >> > > What I suggest is to stop using "latest" for the "git push" CI, but >> > > using it only in a separate CI (once a day/week I don't know). And >> only >> > > use fixed version of the libraries on the CI so builds are not broken >> by >> > > external components. Because in my opinion the "git push" CI is to >> test >> > > our code, not the libraries. >> > > >> > >> > I don't even think such a weekly job is necessary [1]. Add an item to >> the >> > release checklist "check if any new SSL versions are available and add >> them >> > to matrix.py" and this should be fine, all SSL versions will then be >> updated >> > every 6 months and can also be updated on demand for important releases. >> > It's similar to how I simply rerun the Coccinelle patches from time to >> time >> > to fix whatever crept in since the last release. >> >> I'm seeing a slightly different approach here. We need to keep in mind >> that: >> - dev must be as forward-thinking as possible; as such, being aware >> that something is going to break soon is useful, particularly when >> it comes to forthcoming changes in third party dependencies. I.e. >> openssl 3 breakage was reported upfront and that was fortunate >> because a lot of changes were required to adapt to it, long before >> it was even released. So I don't know if the weekly job is the best >> option or not, but right now, watching form time to time how it goes >> with openssl 3.1-dev is useful. Here I'm speaking about a branch, it >> thus means that we're following what significant changes may bother >> us. It's purely integration testing, and we could decide that one >> major upgrade will not be the target for the current version because >> it requires too much work. But if we focus on such a target we should >> be as up-to-date as possible (no need to alarm on past breakage if >> the issue is since fixed). That's where following a -latest from a >> lib can be useful. >> >> - dev must also tell us if we're going to cause breakage on what we >> expect to support. E.g. once we see that 3.1-dev looks good and we >> decide to enable it, we'll also need to be able to quickly follow it, >> because until it's released, it may revert some changes or change its >> API and cause breakage. It's not like a stable version we can trust, >> and we need to know as well if we finally prefer to give up because >> going back-and-forth indicates a third-party dependency is not stable >> enough for example, or we could mention before the release that it's >> only experimental for now. >> >> - even when adopting a stable release, we'll suffer from bugs in third >> party dependencies just like we suffer from our own bugs. If OpenSSL >> 3.1.0 is released and we decide to follow its stable branch, maybe >> some tests will randomly fail from time to time. And we probably >> don't want to manually change its version every week to follow the >> first post-release fixes that we may need to stabilize the CI. In >> this case again, following the latest from a given branch would be >> useful, but it becomes less critical. Indeed, generally the most >> annoying issues should stabilize during the very first versions, and >> we don't need to care as much about updating the lib once the CI works >> fine again. >> >> - however, once we release a version, the most important is to make sure >> there are as few moving parts as possible. We should not change any >> dependency automatically for the CI. Triggering the build on the same >> release two weeks apart should execute the same
Re: Reproducible CI build with OpenSSL and "latest" keyword
вт, 6 дек. 2022 г. в 23:29, Willy Tarreau : > On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote: > > William, > > > > On 12/6/22 15:37, William Lallemand wrote: > > > As I already mentionned, I don't really like the "latest" keyword for > > > the OpenSSL version as it prevent us to have reproducible builds. > > > It updates versions without warning, even major ones. > > > > I agree and also was not really happy with the 'latest' back when it was > > introduced in the first place, but didn't care strongly enough to speak > up > > then. > > No problem, it's always easier to see what we've missed after it reaches > its limits :-) > > > > What I suggest is to stop using "latest" for the "git push" CI, but > > > using it only in a separate CI (once a day/week I don't know). And only > > > use fixed version of the libraries on the CI so builds are not broken > by > > > external components. Because in my opinion the "git push" CI is to test > > > our code, not the libraries. > > > > > > > I don't even think such a weekly job is necessary [1]. Add an item to the > > release checklist "check if any new SSL versions are available and add > them > > to matrix.py" and this should be fine, all SSL versions will then be > updated > > every 6 months and can also be updated on demand for important releases. > > It's similar to how I simply rerun the Coccinelle patches from time to > time > > to fix whatever crept in since the last release. > > I'm seeing a slightly different approach here. We need to keep in mind > that: > - dev must be as forward-thinking as possible; as such, being aware > that something is going to break soon is useful, particularly when > it comes to forthcoming changes in third party dependencies. I.e. > openssl 3 breakage was reported upfront and that was fortunate > because a lot of changes were required to adapt to it, long before > it was even released. So I don't know if the weekly job is the best > option or not, but right now, watching form time to time how it goes > with openssl 3.1-dev is useful. Here I'm speaking about a branch, it > thus means that we're following what significant changes may bother > us. It's purely integration testing, and we could decide that one > major upgrade will not be the target for the current version because > it requires too much work. But if we focus on such a target we should > be as up-to-date as possible (no need to alarm on past breakage if > the issue is since fixed). That's where following a -latest from a > lib can be useful. > > - dev must also tell us if we're going to cause breakage on what we > expect to support. E.g. once we see that 3.1-dev looks good and we > decide to enable it, we'll also need to be able to quickly follow it, > because until it's released, it may revert some changes or change its > API and cause breakage. It's not like a stable version we can trust, > and we need to know as well if we finally prefer to give up because > going back-and-forth indicates a third-party dependency is not stable > enough for example, or we could mention before the release that it's > only experimental for now. > > - even when adopting a stable release, we'll suffer from bugs in third > party dependencies just like we suffer from our own bugs. If OpenSSL > 3.1.0 is released and we decide to follow its stable branch, maybe > some tests will randomly fail from time to time. And we probably > don't want to manually change its version every week to follow the > first post-release fixes that we may need to stabilize the CI. In > this case again, following the latest from a given branch would be > useful, but it becomes less critical. Indeed, generally the most > annoying issues should stabilize during the very first versions, and > we don't need to care as much about updating the lib once the CI works > fine again. > > - however, once we release a version, the most important is to make sure > there are as few moving parts as possible. We should not change any > dependency automatically for the CI. Triggering the build on the same > release two weeks apart should execute the same tests, otherwise we > can think that we broke something during a backport while it's not > the case (that's in fact how we first noticed the problem). > > Thus I think that: > - for dev, adopting the latest version of a chosen branch (released or > dev) if possible would be desirable so that we're informed ASAP that > the branch we're expecting to support is experiencing/causing some > trouble ; > > - for dev, watching weekly or so the level of support of future versions > that we have not yet planed to support would help prepare internals > where needed ; > > - for stable, we'd just stick to the version that was in effect that the > moment of the release (i.e. the version the tests
Re: Reproducible CI build with OpenSSL and "latest" keyword
Tim, On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote: > > What I suggest is to stop using "latest" for the "git push" CI, but > > using it only in a separate CI (once a day/week I don't know). And only > > use fixed version of the libraries on the CI so builds are not broken by > > external components. Because in my opinion the "git push" CI is to test > > our code, not the libraries. > > > > I don't even think such a weekly job is necessary [1]. > Add an item to the release checklist "check if any new SSL versions > are available and add them to matrix.py" and this should be fine, all > SSL versions will then be updated every 6 months and can also be > updated on demand for important releases. Well, I don't want to see the CI fail just for testing this, having the weekly one gives you the status before integration and is also a reminder. > It's similar to how I simply rerun the Coccinelle > patches from time to time to fix whatever crept in since the last release. > I disagree, porting to a new API is not something you would do just before a release, you need to do it progressively if possible, because it could introduce heavy development and sometimes discussions with the library developers and unfortunately that could take time. That would be too bad to postpone support for a new version because nobody looked at this during the development cycle, and the changes are too heavy to be integrated. -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
On Tue, Dec 06, 2022 at 06:59:30PM +0100, Tim Düsterhus wrote: > William, > > On 12/6/22 15:37, William Lallemand wrote: > > As I already mentionned, I don't really like the "latest" keyword for > > the OpenSSL version as it prevent us to have reproducible builds. > > It updates versions without warning, even major ones. > > I agree and also was not really happy with the 'latest' back when it was > introduced in the first place, but didn't care strongly enough to speak up > then. No problem, it's always easier to see what we've missed after it reaches its limits :-) > > What I suggest is to stop using "latest" for the "git push" CI, but > > using it only in a separate CI (once a day/week I don't know). And only > > use fixed version of the libraries on the CI so builds are not broken by > > external components. Because in my opinion the "git push" CI is to test > > our code, not the libraries. > > > > I don't even think such a weekly job is necessary [1]. Add an item to the > release checklist "check if any new SSL versions are available and add them > to matrix.py" and this should be fine, all SSL versions will then be updated > every 6 months and can also be updated on demand for important releases. > It's similar to how I simply rerun the Coccinelle patches from time to time > to fix whatever crept in since the last release. I'm seeing a slightly different approach here. We need to keep in mind that: - dev must be as forward-thinking as possible; as such, being aware that something is going to break soon is useful, particularly when it comes to forthcoming changes in third party dependencies. I.e. openssl 3 breakage was reported upfront and that was fortunate because a lot of changes were required to adapt to it, long before it was even released. So I don't know if the weekly job is the best option or not, but right now, watching form time to time how it goes with openssl 3.1-dev is useful. Here I'm speaking about a branch, it thus means that we're following what significant changes may bother us. It's purely integration testing, and we could decide that one major upgrade will not be the target for the current version because it requires too much work. But if we focus on such a target we should be as up-to-date as possible (no need to alarm on past breakage if the issue is since fixed). That's where following a -latest from a lib can be useful. - dev must also tell us if we're going to cause breakage on what we expect to support. E.g. once we see that 3.1-dev looks good and we decide to enable it, we'll also need to be able to quickly follow it, because until it's released, it may revert some changes or change its API and cause breakage. It's not like a stable version we can trust, and we need to know as well if we finally prefer to give up because going back-and-forth indicates a third-party dependency is not stable enough for example, or we could mention before the release that it's only experimental for now. - even when adopting a stable release, we'll suffer from bugs in third party dependencies just like we suffer from our own bugs. If OpenSSL 3.1.0 is released and we decide to follow its stable branch, maybe some tests will randomly fail from time to time. And we probably don't want to manually change its version every week to follow the first post-release fixes that we may need to stabilize the CI. In this case again, following the latest from a given branch would be useful, but it becomes less critical. Indeed, generally the most annoying issues should stabilize during the very first versions, and we don't need to care as much about updating the lib once the CI works fine again. - however, once we release a version, the most important is to make sure there are as few moving parts as possible. We should not change any dependency automatically for the CI. Triggering the build on the same release two weeks apart should execute the same tests, otherwise we can think that we broke something during a backport while it's not the case (that's in fact how we first noticed the problem). Thus I think that: - for dev, adopting the latest version of a chosen branch (released or dev) if possible would be desirable so that we're informed ASAP that the branch we're expecting to support is experiencing/causing some trouble ; - for dev, watching weekly or so the level of support of future versions that we have not yet planed to support would help prepare internals where needed ; - for stable, we'd just stick to the version that was in effect that the moment of the release (i.e. the version the tests were run on during dev). And that would make sense considering that a release is nothing more than freezing changes between dev and stable. I'm particularly conscious that there may be technical limitations for the
Re: Reproducible CI build with OpenSSL and "latest" keyword
William, On 12/6/22 15:37, William Lallemand wrote: As I already mentionned, I don't really like the "latest" keyword for the OpenSSL version as it prevent us to have reproducible builds. It updates versions without warning, even major ones. I agree and also was not really happy with the 'latest' back when it was introduced in the first place, but didn't care strongly enough to speak up then. What I suggest is to stop using "latest" for the "git push" CI, but using it only in a separate CI (once a day/week I don't know). And only use fixed version of the libraries on the CI so builds are not broken by external components. Because in my opinion the "git push" CI is to test our code, not the libraries. I don't even think such a weekly job is necessary [1]. Add an item to the release checklist "check if any new SSL versions are available and add them to matrix.py" and this should be fine, all SSL versions will then be updated every 6 months and can also be updated on demand for important releases. It's similar to how I simply rerun the Coccinelle patches from time to time to fix whatever crept in since the last release. Best regards Tim Düsterhus
Re: Reproducible CI build with OpenSSL and "latest" keyword
вт, 6 дек. 2022 г. в 21:22, William Lallemand : > On Tue, Dec 06, 2022 at 07:54:33PM +0500, Илья Шипицин wrote: > > I recall I even promised to do something, but I did not :-) > > > > automatically determine "which is latest 3.0.x" does not make much sense, > > it is stable branch, very conservative. we can stick to 3.0.7, for > example. > > I do not expect any breaking change between 3.0.7 and 3.0.8 > > I recall a discussion like this indeed, couldn't find the previous > thread. There was cases of broken minor release for LibreSSL for > example, so it's better to stick to a release for the push CI and > upgrade once the weekly CI passed. > > > we can move "latest" to weekly, np. as for stable branches CI, I think > them > > do not have to follow the same rules as development branch, we can have > > different matrix for stable and development. > > I think we could it this way: > > - weekly "latest" for libreSSL, OpenSSL, whateverSSL for HAProxy master > branch > - git push CI with fixed version in HAProxy master > > > > I think I got the idea. > > looks like you use the same github actions for stable branches. > > > > Indeed, at some point the master branch become a stable branch, so we > inherit all of this. > > > either I will manage to make them different or I will stick to > > 3.0.something. hopefully tomorrow > > IMHO it should never be "latest" anywhere else than the weekly, we don't > want to be disturbed by this when pushing our code. > > I don't think we need a weekly for the stable branch, so it could be > conditionned by a check on the version, for example only started if > there is '-dev' in the version. > > So we should probably: > > - Revert "latest" to "3.0.7" in the master, and backport the patch in the > previous supported branches. (as far as 2.4 I think) > I want to check how github actions are good on branch filtering. if they are, it adds flexibility. if they are not, ok, we will maintain the same actions for all branches. > > - Introduce "3.1.0-alpha1" to master > > - Introduce "latest" to weekly master > > -- > William Lallemand > >
Re: Reproducible CI build with OpenSSL and "latest" keyword
On Tue, Dec 06, 2022 at 07:54:33PM +0500, Илья Шипицин wrote: > I recall I even promised to do something, but I did not :-) > > automatically determine "which is latest 3.0.x" does not make much sense, > it is stable branch, very conservative. we can stick to 3.0.7, for example. > I do not expect any breaking change between 3.0.7 and 3.0.8 I recall a discussion like this indeed, couldn't find the previous thread. There was cases of broken minor release for LibreSSL for example, so it's better to stick to a release for the push CI and upgrade once the weekly CI passed. > we can move "latest" to weekly, np. as for stable branches CI, I think them > do not have to follow the same rules as development branch, we can have > different matrix for stable and development. I think we could it this way: - weekly "latest" for libreSSL, OpenSSL, whateverSSL for HAProxy master branch - git push CI with fixed version in HAProxy master > I think I got the idea. > looks like you use the same github actions for stable branches. > Indeed, at some point the master branch become a stable branch, so we inherit all of this. > either I will manage to make them different or I will stick to > 3.0.something. hopefully tomorrow IMHO it should never be "latest" anywhere else than the weekly, we don't want to be disturbed by this when pushing our code. I don't think we need a weekly for the stable branch, so it could be conditionned by a check on the version, for example only started if there is '-dev' in the version. So we should probably: - Revert "latest" to "3.0.7" in the master, and backport the patch in the previous supported branches. (as far as 2.4 I think) - Introduce "3.1.0-alpha1" to master - Introduce "latest" to weekly master -- William Lallemand
Re: Reproducible CI build with OpenSSL and "latest" keyword
I think I got the idea. looks like you use the same github actions for stable branches. either I will manage to make them different or I will stick to 3.0.something. hopefully tomorrow вт, 6 дек. 2022 г. в 19:54, Илья Шипицин : > I recall I even promised to do something, but I did not :-) > > automatically determine "which is latest 3.0.x" does not make much sense, > it is stable branch, very conservative. we can stick to 3.0.7, for example. > I do not expect any breaking change between 3.0.7 and 3.0.8 > > we can move "latest" to weekly, np. as for stable branches CI, I think > them do not have to follow the same rules as development branch, we can > have different matrix for stable and development. > > вт, 6 дек. 2022 г. в 19:37, William Lallemand : > >> Hello, >> >> We are experiencing difficulties with the way the CI matrix is >> generated with the SSL libraries. >> >> As I already mentionned, I don't really like the "latest" keyword for >> the OpenSSL version as it prevent us to have reproducible builds. >> It updates versions without warning, even major ones. >> >> Since OpenSSL 3.1.0-aplha1 was released we are affected by the problem, >> we stopped building with 3.0.x without noticing, and our internal CI for >> the stable branches start failing because of that. Majour versions must >> never change in the previous branches. >> >> What I suggest is to stop using "latest" for the "git push" CI, but >> using it only in a separate CI (once a day/week I don't know). And only >> use fixed version of the libraries on the CI so builds are not broken by >> external components. Because in my opinion the "git push" CI is to test >> our code, not the libraries. >> >> What do you guys think? >> >> -- >> William Lallemand >> >
Re: Reproducible CI build with OpenSSL and "latest" keyword
I recall I even promised to do something, but I did not :-) automatically determine "which is latest 3.0.x" does not make much sense, it is stable branch, very conservative. we can stick to 3.0.7, for example. I do not expect any breaking change between 3.0.7 and 3.0.8 we can move "latest" to weekly, np. as for stable branches CI, I think them do not have to follow the same rules as development branch, we can have different matrix for stable and development. вт, 6 дек. 2022 г. в 19:37, William Lallemand : > Hello, > > We are experiencing difficulties with the way the CI matrix is > generated with the SSL libraries. > > As I already mentionned, I don't really like the "latest" keyword for > the OpenSSL version as it prevent us to have reproducible builds. > It updates versions without warning, even major ones. > > Since OpenSSL 3.1.0-aplha1 was released we are affected by the problem, > we stopped building with 3.0.x without noticing, and our internal CI for > the stable branches start failing because of that. Majour versions must > never change in the previous branches. > > What I suggest is to stop using "latest" for the "git push" CI, but > using it only in a separate CI (once a day/week I don't know). And only > use fixed version of the libraries on the CI so builds are not broken by > external components. Because in my opinion the "git push" CI is to test > our code, not the libraries. > > What do you guys think? > > -- > William Lallemand >
Reproducible CI build with OpenSSL and "latest" keyword
Hello, We are experiencing difficulties with the way the CI matrix is generated with the SSL libraries. As I already mentionned, I don't really like the "latest" keyword for the OpenSSL version as it prevent us to have reproducible builds. It updates versions without warning, even major ones. Since OpenSSL 3.1.0-aplha1 was released we are affected by the problem, we stopped building with 3.0.x without noticing, and our internal CI for the stable branches start failing because of that. Majour versions must never change in the previous branches. What I suggest is to stop using "latest" for the "git push" CI, but using it only in a separate CI (once a day/week I don't know). And only use fixed version of the libraries on the CI so builds are not broken by external components. Because in my opinion the "git push" CI is to test our code, not the libraries. What do you guys think? -- William Lallemand
