Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 17, 2021 at 04:48:45PM +0200, Tim Düsterhus wrote:
> To clarify: This specific timing also includes the wait times until the
> build VMs are available. It's basically the wall time elapsed between the
> push happening and the *last* build reporting the status. So the time the
> developer has to wait for all the results to be available :-) It's not the
> duration of the slowest job.

Ah OK, probably less variations then!

> I can confirm that OpenSSL 3 is much faster, though. But you only see that
> when directly looking into a specific run.

Yes, that's how I initially noticed it :-)

Whatever we can do to keep build times low will be a benefit both for us
in terms of feedback, and for the shared CI infrastructure.

Willy

Re: Speeding up opentracing build in CI ?

[Tim Düsterhus]

Willy,

On 6/17/21 4:39 PM, Willy Tarreau wrote:

Ah indeed. Well, it even suffers variations that are larger than the
potential savings, ranging from 10'45 to 15'00 over the last few builds.
So we don't really know how much we're saving, we can only hope it saves
a little bit.



To clarify: This specific timing also includes the wait times until the 
build VMs are available. It's basically the wall time elapsed between 
the push happening and the *last* build reporting the status. So the 
time the developer has to wait for all the results to be available :-) 
It's not the duration of the slowest job.


I can confirm that OpenSSL 3 is much faster, though. But you only see 
that when directly looking into a specific run.


Best regards
Tim Düsterhus

Re: Speeding up opentracing build in CI ?

[Илья Шипицин]

чт, 17 июн. 2021 г. в 19:39, Willy Tarreau :

> On Thu, Jun 17, 2021 at 04:31:57PM +0200, Tim Düsterhus wrote:
> > Willy, William,
> >
> > On 6/17/21 3:55 PM, William Lallemand wrote:
> > > > OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
> > > > I've just excluded versions 1.x from both the parallel build and the
> > > > build_sw target and that's good now.
> > > >
> > > > Willy
> > >
> > > Great improvement, thanks!
> > >
> >
> > Here's a full filtered overview:
> >
> >
> https://github.com/haproxy/haproxy/actions/workflows/vtest.yml?query=branch%3Amaster
> >
> > It's down from ~11 minutes until the last build finishes to ~9 minutes.
>
> Ah indeed. Well, it even suffers variations that are larger than the
> potential savings, ranging from 10'45 to 15'00 over the last few builds.
> So we don't really know how much we're saving, we can only hope it saves
> a little bit.
>

we are in control of that.
what if we'll add "sleep 30" and remove it after while ?
nobody will tell it is "a little bit speedup"


>
> Thanks!
> Willy
>

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 17, 2021 at 04:31:57PM +0200, Tim Düsterhus wrote:
> Willy, William,
> 
> On 6/17/21 3:55 PM, William Lallemand wrote:
> > > OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
> > > I've just excluded versions 1.x from both the parallel build and the
> > > build_sw target and that's good now.
> > > 
> > > Willy
> > 
> > Great improvement, thanks!
> > 
> 
> Here's a full filtered overview:
> 
> https://github.com/haproxy/haproxy/actions/workflows/vtest.yml?query=branch%3Amaster
> 
> It's down from ~11 minutes until the last build finishes to ~9 minutes.

Ah indeed. Well, it even suffers variations that are larger than the
potential savings, ranging from 10'45 to 15'00 over the last few builds.
So we don't really know how much we're saving, we can only hope it saves
a little bit.

Thanks!
Willy

Re: Speeding up opentracing build in CI ?

[Tim Düsterhus]

Willy, William,

On 6/17/21 3:55 PM, William Lallemand wrote:

OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
I've just excluded versions 1.x from both the parallel build and the
build_sw target and that's good now.

Willy


Great improvement, thanks!



Here's a full filtered overview:

https://github.com/haproxy/haproxy/actions/workflows/vtest.yml?query=branch%3Amaster

It's down from ~11 minutes until the last build finishes to ~9 minutes.

Best regards
Tim Düsterhus

Re: Speeding up opentracing build in CI ?

[William Lallemand]

On Thu, Jun 17, 2021 at 03:53:10PM +0200, Willy Tarreau wrote:
> Subject: Re: Speeding up opentracing build in CI ?
>
> On Thu, Jun 17, 2021 at 03:29:58PM +0200, Willy Tarreau wrote:
> > On Thu, Jun 17, 2021 at 03:24:19PM +0200, Willy Tarreau wrote:
> > > On Thu, Jun 10, 2021 at 08:55:13PM +0500,  ??? wrote:
> > > > I was mistaken. LibreSSL does not like parallel install
> > > > 
> > > > libressl fails on `make -j4 install` · Issue #461 ·
> > > > libressl-portable/portable (github.com)
> > > > <https://github.com/libressl-portable/portable/issues/461>
> > > > 
> > > > 
> > > > anyway, if CI works, I'm ok with changes
> > > 
> > > OK I've applied the change to use build_sw on openssl only (libressl
> > > doesn't have it), and -j$(nproc) for this phase for openssl on linux
> > > (will likely be 2). Let's see.
> > 
> > Hmmm it fails with 1.0.2, there's no build_sw there. In addition I do
> > remember that 1.0.2 used to require significant patches to be reliable
> > under make -j.
> > 
> > Let's wait for the remaining tests to conclude.
> 
> OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
> I've just excluded versions 1.x from both the parallel build and the
> build_sw target and that's good now.
> 
> Willy

Great improvement, thanks!

-- 
William Lallemand

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 17, 2021 at 03:29:58PM +0200, Willy Tarreau wrote:
> On Thu, Jun 17, 2021 at 03:24:19PM +0200, Willy Tarreau wrote:
> > On Thu, Jun 10, 2021 at 08:55:13PM +0500,  ??? wrote:
> > > I was mistaken. LibreSSL does not like parallel install
> > > 
> > > libressl fails on `make -j4 install` · Issue #461 ·
> > > libressl-portable/portable (github.com)
> > > 
> > > 
> > > 
> > > anyway, if CI works, I'm ok with changes
> > 
> > OK I've applied the change to use build_sw on openssl only (libressl
> > doesn't have it), and -j$(nproc) for this phase for openssl on linux
> > (will likely be 2). Let's see.
> 
> Hmmm it fails with 1.0.2, there's no build_sw there. In addition I do
> remember that 1.0.2 used to require significant patches to be reliable
> under make -j.
> 
> Let's wait for the remaining tests to conclude.

OK that's a net win, openssl-3.0.0-alpha17 dropped from 8'29 to 2'55.
I've just excluded versions 1.x from both the parallel build and the
build_sw target and that's good now.

Willy

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 17, 2021 at 03:24:19PM +0200, Willy Tarreau wrote:
> On Thu, Jun 10, 2021 at 08:55:13PM +0500,  ??? wrote:
> > I was mistaken. LibreSSL does not like parallel install
> > 
> > libressl fails on `make -j4 install` · Issue #461 ·
> > libressl-portable/portable (github.com)
> > 
> > 
> > 
> > anyway, if CI works, I'm ok with changes
> 
> OK I've applied the change to use build_sw on openssl only (libressl
> doesn't have it), and -j$(nproc) for this phase for openssl on linux
> (will likely be 2). Let's see.

Hmmm it fails with 1.0.2, there's no build_sw there. In addition I do
remember that 1.0.2 used to require significant patches to be reliable
under make -j.

Let's wait for the remaining tests to conclude.

Willy

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 10, 2021 at 08:55:13PM +0500,  ??? wrote:
> I was mistaken. LibreSSL does not like parallel install
> 
> libressl fails on `make -j4 install` · Issue #461 ·
> libressl-portable/portable (github.com)
> 
> 
> 
> anyway, if CI works, I'm ok with changes

OK I've applied the change to use build_sw on openssl only (libressl
doesn't have it), and -j$(nproc) for this phase for openssl on linux
(will likely be 2). Let's see.

Willy

Re: Speeding up opentracing build in CI ?

[Tim Düsterhus]

William,

On 6/10/21 5:48 PM, William Lallemand wrote:

Looks fine to me, but from what I remember when debugging some reg-tests
there was only one CPU available, I hope I'm wrong.



GitHub-provided Action runners are 2-core VMs:

https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Best regards
Tim Düsterhus

Re: Speeding up opentracing build in CI ?

[Илья Шипицин]

I was mistaken. LibreSSL does not like parallel install

libressl fails on `make -j4 install` · Issue #461 ·
libressl-portable/portable (github.com)
<https://github.com/libressl-portable/portable/issues/461>


anyway, if CI works, I'm ok with changes

чт, 10 июн. 2021 г. в 20:49, William Lallemand :

> On Thu, Jun 10, 2021 at 07:52:23AM +0200, Willy Tarreau wrote:
> > Subject: Re: Speeding up opentracing build in CI ?
> >
> > On Thu, Jun 10, 2021 at 07:19:37AM +0200, Willy Tarreau wrote:
> > > On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> > > > OT takes about 30 sec (it is built with almost everything disabled).
> the
> > > > biggest time eater is openssl-3.0.0
> > >
> > > Maybe that one could be sped up too, I haven't checked if it uses
> parallel
> > > builds.
> >
> > So I checked. Good news, it wasn't parallel either, and this alone:
> >
> > --- a/scripts/build-ssl.sh
> > +++ b/scripts/build-ssl.sh
> > @@ -21,7 +21,8 @@ build_openssl_linux () {
> >  (
> >  cd "openssl-${OPENSSL_VERSION}/"
> >  ./config shared --prefix="${HOME}/opt"
> --openssldir="${HOME}/opt" -DPURIFY
> > -make all install_sw
> > +make -j$(nproc) all
> > +make install_sw
> >  )
> >  }
> >
> > Is enough to drop from 4:52 to 1:28 on my machine. About 1/4 of this time
> > is used to build man and HTML pages that we don't use. Instead of the
> "all"
> > target, we should use "build_sw"
> >
> > --- a/scripts/build-ssl.sh
> > +++ b/scripts/build-ssl.sh
> > @@ -21,7 +21,8 @@ build_openssl_linux () {
> >  (
> >  cd "openssl-${OPENSSL_VERSION}/"
> >  ./config shared --prefix="${HOME}/opt"
> --openssldir="${HOME}/opt" -DPURIFY
> > -make all install_sw
> > +make -j$(nproc) build_sw
> > +make install_sw
> >  )
> >  }
> >
> > this further downs the time to 1:9, hence more than 4 times faster than
> > the initial one. It should probably be tested on macos to be certain it's
> > OK there as well, and I don't know how to get the CPU count there (or
> > maybe we could just force it to a low value like 2 or 4).
> >
> > Willy
> >
>
> Looks fine to me, but from what I remember when debugging some reg-tests
> there was only one CPU available, I hope I'm wrong.
>
> --
> William Lallemand
>

Re: Speeding up opentracing build in CI ?

[William Lallemand]

On Thu, Jun 10, 2021 at 07:52:23AM +0200, Willy Tarreau wrote:
> Subject: Re: Speeding up opentracing build in CI ?
>
> On Thu, Jun 10, 2021 at 07:19:37AM +0200, Willy Tarreau wrote:
> > On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> > > OT takes about 30 sec (it is built with almost everything disabled). the
> > > biggest time eater is openssl-3.0.0
> > 
> > Maybe that one could be sped up too, I haven't checked if it uses parallel
> > builds.
> 
> So I checked. Good news, it wasn't parallel either, and this alone:
> 
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) all
> +make install_sw
>  )
>  }
> 
> Is enough to drop from 4:52 to 1:28 on my machine. About 1/4 of this time
> is used to build man and HTML pages that we don't use. Instead of the "all"
> target, we should use "build_sw"
> 
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) build_sw
> +make install_sw
>  )
>  }
> 
> this further downs the time to 1:9, hence more than 4 times faster than
> the initial one. It should probably be tested on macos to be certain it's
> OK there as well, and I don't know how to get the CPU count there (or
> maybe we could just force it to a low value like 2 or 4).
> 
> Willy
> 

Looks fine to me, but from what I remember when debugging some reg-tests
there was only one CPU available, I hope I'm wrong.

-- 
William Lallemand

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 10, 2021 at 09:00:32AM +0300,  ??? wrote:
> openssl does not support -j for make install
> I filed a bug on them, they told me "OK, just don't use it"

I don't care about install, which I didn't touch,  but about "all".
I never install anything with -j anyway. The savings are significant
enough to be used.

And regardless we should definitely avoid building all that useless
doc that takes more than one minute.

Willy

Re: Speeding up opentracing build in CI ?

[Илья Шипицин]

openssl does not support -j for make install

I filed a bug on them, they told me "OK, just don't use it"

On Thu, Jun 10, 2021, 8:52 AM Willy Tarreau  wrote:

> On Thu, Jun 10, 2021 at 07:19:37AM +0200, Willy Tarreau wrote:
> > On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> > > OT takes about 30 sec (it is built with almost everything disabled).
> the
> > > biggest time eater is openssl-3.0.0
> >
> > Maybe that one could be sped up too, I haven't checked if it uses
> parallel
> > builds.
>
> So I checked. Good news, it wasn't parallel either, and this alone:
>
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt"
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) all
> +make install_sw
>  )
>  }
>
> Is enough to drop from 4:52 to 1:28 on my machine. About 1/4 of this time
> is used to build man and HTML pages that we don't use. Instead of the "all"
> target, we should use "build_sw"
>
> --- a/scripts/build-ssl.sh
> +++ b/scripts/build-ssl.sh
> @@ -21,7 +21,8 @@ build_openssl_linux () {
>  (
>  cd "openssl-${OPENSSL_VERSION}/"
>  ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt"
> -DPURIFY
> -make all install_sw
> +make -j$(nproc) build_sw
> +make install_sw
>  )
>  }
>
> this further downs the time to 1:9, hence more than 4 times faster than
> the initial one. It should probably be tested on macos to be certain it's
> OK there as well, and I don't know how to get the CPU count there (or
> maybe we could just force it to a low value like 2 or 4).
>
> Willy
>

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 10, 2021 at 07:19:37AM +0200, Willy Tarreau wrote:
> On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> > OT takes about 30 sec (it is built with almost everything disabled). the
> > biggest time eater is openssl-3.0.0
> 
> Maybe that one could be sped up too, I haven't checked if it uses parallel
> builds.

So I checked. Good news, it wasn't parallel either, and this alone:

--- a/scripts/build-ssl.sh
+++ b/scripts/build-ssl.sh
@@ -21,7 +21,8 @@ build_openssl_linux () {
 (
 cd "openssl-${OPENSSL_VERSION}/"
 ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
-DPURIFY
-make all install_sw
+make -j$(nproc) all
+make install_sw
 )
 }

Is enough to drop from 4:52 to 1:28 on my machine. About 1/4 of this time
is used to build man and HTML pages that we don't use. Instead of the "all"
target, we should use "build_sw"

--- a/scripts/build-ssl.sh
+++ b/scripts/build-ssl.sh
@@ -21,7 +21,8 @@ build_openssl_linux () {
 (
 cd "openssl-${OPENSSL_VERSION}/"
 ./config shared --prefix="${HOME}/opt" --openssldir="${HOME}/opt" 
-DPURIFY
-make all install_sw
+make -j$(nproc) build_sw
+make install_sw
 )
 }

this further downs the time to 1:9, hence more than 4 times faster than
the initial one. It should probably be tested on macos to be certain it's
OK there as well, and I don't know how to get the CPU count there (or
maybe we could just force it to a low value like 2 or 4).

Willy

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 10, 2021 at 10:15:46AM +0500,  ??? wrote:
> OT takes about 30 sec (it is built with almost everything disabled). the
> biggest time eater is openssl-3.0.0

Maybe that one could be sped up too, I haven't checked if it uses parallel
builds.

> hopefully, OT will speed up to 10 sec by using parallel builds

That's still better than nothing!

Thanks,
Willy

Re: Speeding up opentracing build in CI ?

[Илья Шипицин]

OT takes about 30 sec (it is built with almost everything disabled). the
biggest time eater is openssl-3.0.0

hopefully, OT will speed up to 10 sec by using parallel builds

чт, 10 июн. 2021 г. в 10:05, Willy Tarreau :

> On Thu, Jun 10, 2021 at 07:55:17AM +0300,  ??? wrote:
> > First one is supposed to be cached one day together with "opt" folder.
> >
> > However, we can indeed use parallel builds until cache is enabled for
> > github actions
>
> OK. The other use case I saw was that it's more convenient for developers
> than having to go through each component's readme to figure how to build
> that :-)
>
> I'll make a patch then, thank you!
> Willy
>

Re: Speeding up opentracing build in CI ?

[Willy Tarreau]

On Thu, Jun 10, 2021 at 07:55:17AM +0300,  ??? wrote:
> First one is supposed to be cached one day together with "opt" folder.
> 
> However, we can indeed use parallel builds until cache is enabled for
> github actions

OK. The other use case I saw was that it's more convenient for developers
than having to go through each component's readme to figure how to build
that :-)

I'll make a patch then, thank you!
Willy

Re: Speeding up opentracing build in CI ?

[Илья Шипицин]

First one is supposed to be cached one day together with "opt" folder.

However, we can indeed use parallel builds until cache is enabled for
github actions

On Thu, Jun 10, 2021, 7:49 AM Willy Tarreau  wrote:

> Tim, Ilya,
>
> while testing Miroslav's fix, I found the opentracing build to be quite
> slow and figured it doesn't use parallel builds. Do you have any objection
> against patching the script like this ?
>
> diff --git a/scripts/build-ot.sh b/scripts/build-ot.sh
> index 59d6af587..1c296b64b 100755
> --- a/scripts/build-ot.sh
> +++ b/scripts/build-ot.sh
> @@ -19,7 +19,7 @@ if [ "$(cat ${HOME}/opt/.ot-cpp-version)" !=
> "${OT_CPP_VERSION}" ]; then
>  mkdir build
>  cd build
>  cmake -DCMAKE_INSTALL_PREFIX=${HOME}/opt -DBUILD_STATIC_LIBS=OFF
> -DBUILD_MOCKTRACER=OFF -DBUILD_TESTING=OFF >
> -make
> +make -j$(nproc)
>  make install
>  echo "${OT_CPP_VERSION}" > "${HOME}/opt/.ot-cpp-version"
>  fi
> @@ -28,7 +28,7 @@ git clone
> https://github.com/haproxytech/opentracing-c-wrapper.git
>  cd opentracing-c-wrapper
>   ./scripts/bootstrap
>   ./configure --prefix=${HOME}/opt --with-opentracing=${HOME}/opt
> - make
> + make -j$(nproc)
>   make install
>
> I'm assuming it's already made to build only in environments we support
> so we shouldn't care about the presence of the nproc utility there.
>
> Thanks,
> Willy
>