Re: [PATCH] move VTest installation to scripts/build-vtest.sh
On Tue, May 18, 2021 at 12:07:17PM +0200, Tim Düsterhus wrote: > Ilya, > > On 5/18/21 11:50 AM, ??? wrote: > > I wonder if we can benefit from dependent steps, i.e. build VTest just once > > and deliver binary to other containers (instead of building it every time). > > > > The complexity is not worth it to shave of another half second or so. > Installing VTest is down to 6 seconds for GitHub Actions. That's certainly > good enough. I agree, and caches tend to add more uncertainty when trying to diagnose trouble. This one is really fast. Willy
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
Ilya, On 5/18/21 11:50 AM, Илья Шипицин wrote: I wonder if we can benefit from dependent steps, i.e. build VTest just once and deliver binary to other containers (instead of building it every time). The complexity is not worth it to shave of another half second or so. Installing VTest is down to 6 seconds for GitHub Actions. That's certainly good enough. Best regards Tim Düsterhus
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
вт, 18 мая 2021 г. в 13:54, Willy Tarreau : > On Sat, May 15, 2021 at 05:53:57PM +0200, Tim Düsterhus wrote: > > Willy, > > Ilya, > > > > On 5/15/21 8:58 AM, ??? wrote: > > > I attached a patch that uses "curl". on a distance it seems to be > faster > > > for 50% > > > > > > > This one looks good to me. > > Now merged, thanks guys. > > A quick comment about this, since we're trying to be fast, the following: > > curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o > VTest.tar.gz > mkdir ../vtest > tar xvf VTest.tar.gz -C ../vtest --strip-components=1 > > Can be more optimally done this way by paralleling the download and the > extraction: > > mkdir ../vtest > curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz | \ > tar -C ../vtest -zxf - --strip-components=1 > > However due to vtest being very small (180 kB) it will not make any > difference on this one. > I wonder if we can benefit from dependent steps, i.e. build VTest just once and deliver binary to other containers (instead of building it every time). > > Cheers, > Willy >
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
On Sat, May 15, 2021 at 05:53:57PM +0200, Tim Düsterhus wrote: > Willy, > Ilya, > > On 5/15/21 8:58 AM, ??? wrote: > > I attached a patch that uses "curl". on a distance it seems to be faster > > for 50% > > > > This one looks good to me. Now merged, thanks guys. A quick comment about this, since we're trying to be fast, the following: curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o VTest.tar.gz mkdir ../vtest tar xvf VTest.tar.gz -C ../vtest --strip-components=1 Can be more optimally done this way by paralleling the download and the extraction: mkdir ../vtest curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz | \ tar -C ../vtest -zxf - --strip-components=1 However due to vtest being very small (180 kB) it will not make any difference on this one. Cheers, Willy
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
Willy, Ilya, On 5/15/21 8:58 AM, Илья Шипицин wrote: I attached a patch that uses "curl". on a distance it seems to be faster for 50% This one looks good to me. Best regards Tim Düsterhus
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
сб, 15 мая 2021 г. в 01:14, Илья Шипицин :
>
>
> сб, 15 мая 2021 г. в 00:56, Nicolas CARPi :
>
>> > I compared several times, git clone is faster
>> In my hands curl is the fastest option compared to git clone.
>>
>
> ok, I'll make more comparison tomorrow
>
I attached a patch that uses "curl". on a distance it seems to be faster
for 50%
>
>
>>
>> Wouldn't "git clone --depth 1" be a good option to consider here? It
>> avoids getting the full history.
>>
>> ~Nico
>>
>
From 7d933d88dc03782bab2985081bb9629d3a3389fd Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin
Date: Sat, 15 May 2021 11:46:15 +0500
Subject: [PATCH] CI: introduce scripts/build-vtest.sh for installing VTest
we install VTest for all CI systems, let us unify instalation
---
.cirrus.yml| 3 +--
.github/workflows/openssl-nodeprecated.yml | 3 +--
.github/workflows/vtest.yml| 8 ++--
.travis.yml| 4 +---
scripts/build-vtest.sh | 10 ++
5 files changed, 15 insertions(+), 13 deletions(-)
create mode 100755 scripts/build-vtest.sh
diff --git a/.cirrus.yml b/.cirrus.yml
index fdabfdcdd..9b83e6169 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -6,8 +6,7 @@ FreeBSD_task:
install_script:
- pkg update -f && pkg upgrade -y && pkg install -y openssl git gmake lua53 socat pcre
script:
-- git clone https://github.com/VTest/VTest.git ../vtest
-- make -C ../vtest
+- scripts/build-vtest.sh
- gmake CC=clang V=1 ERR=1 TARGET=freebsd USE_ZLIB=1 USE_PCRE=1 USE_OPENSSL=1 USE_LUA=1 LUA_INC=/usr/local/include/lua53 LUA_LIB=/usr/local/lib LUA_LIB_NAME=lua-5.3
- ./haproxy -vv
- ldd haproxy
diff --git a/.github/workflows/openssl-nodeprecated.yml b/.github/workflows/openssl-nodeprecated.yml
index b853fe233..6833911e4 100644
--- a/.github/workflows/openssl-nodeprecated.yml
+++ b/.github/workflows/openssl-nodeprecated.yml
@@ -23,8 +23,7 @@ jobs:
- uses: actions/checkout@v1
- name: prepare VTest
run: |
-git clone https://github.com/VTest/VTest.git ../vtest
-make -C ../vtest FLAGS="-O2 -s -Wall"
+scripts/build-vtest.sh
- name: build haproxy
run: |
make DEFINE="-DOPENSSL_API_COMPAT=0x1010L -DOPENSSL_NO_DEPRECATED" -j3 CC=gcc ERR=1 TARGET=linux-glibc USE_OPENSSL=1
diff --git a/.github/workflows/vtest.yml b/.github/workflows/vtest.yml
index cb52f27d6..f7f34d720 100644
--- a/.github/workflows/vtest.yml
+++ b/.github/workflows/vtest.yml
@@ -60,11 +60,7 @@ jobs:
brew install lua
- name: Install VTest
run: |
-curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o VTest.tar.gz
-mkdir VTest
-tar xvf VTest.tar.gz -C VTest --strip-components=1
-make -C VTest -j$(nproc) FLAGS="-O2 -s -Wall"
-sudo install -m755 VTest/vtest /usr/local/bin/vtest
+scripts/build-vtest.sh
- name: Install SSL ${{ matrix.ssl }}
if: ${{ matrix.ssl && matrix.ssl != 'stock' }}
run: env ${{ matrix.ssl }} scripts/build-ssl.sh
@@ -101,7 +97,7 @@ jobs:
- name: Run VTest for HAProxy ${{ steps.show-version.outputs.version }}
id: vtest
# sudo is required, because macOS fails due to an open files limit.
- run: sudo make reg-tests REGTESTS_TYPES=default,bug,devel
+ run: sudo make reg-tests VTEST_PROGRAM=../vtest/vtest REGTESTS_TYPES=default,bug,devel
- name: Show results
if: ${{ failure() }}
# The chmod / sudo is necessary due to the `sudo` while running the tests.
diff --git a/.travis.yml b/.travis.yml
index 1aa415aa8..3083e302c 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -38,9 +38,7 @@ matrix:
if: type == cron
install:
- - git clone https://github.com/VTest/VTest.git ../vtest
- # Special flags due to: https://github.com/vtest/VTest/issues/12
- - make -C ../vtest FLAGS="-O2 -s -Wall"
+ - scripts/build-vtest.sh
script:
- make -j$(nproc) ERR=1 TARGET=linux-glibc CC=$CC DEBUG=-DDEBUG_STRICT=1 $FLAGS
diff --git a/scripts/build-vtest.sh b/scripts/build-vtest.sh
new file mode 100755
index 0..4db35d6ee
--- /dev/null
+++ b/scripts/build-vtest.sh
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+set -eux
+
+curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o VTest.tar.gz
+mkdir ../vtest
+tar xvf VTest.tar.gz -C ../vtest --strip-components=1
+# Special flags due to: https://github.com/vtest/VTest/issues/12
+make -C ../vtest FLAGS="-O2 -s -Wall"
+
--
2.31.1
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
сб, 15 мая 2021 г. в 00:56, Nicolas CARPi : > > I compared several times, git clone is faster > In my hands curl is the fastest option compared to git clone. > ok, I'll make more comparison tomorrow > > Wouldn't "git clone --depth 1" be a good option to consider here? It > avoids getting the full history. > > ~Nico >
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
> I compared several times, git clone is faster In my hands curl is the fastest option compared to git clone. Wouldn't "git clone --depth 1" be a good option to consider here? It avoids getting the full history. ~Nico
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
сб, 15 мая 2021 г. в 00:20, Tim Düsterhus : > Ilya, > > On 5/14/21 9:02 PM, Илья Шипицин wrote: > > let us unify VTest installation magic. > > I disagree with using 'git clone' here, cloning the repository with full > history is wasteful. Please use the tarball approach from vtest.yml > I think, if we download from github releases, indeed those are pre-packaged. Master seems to be created on the fly. I compared several times, git clone is faster [ilia@localhost temp]$ time git clone https://github.com/VTest/VTest.git Cloning into 'VTest'... remote: Enumerating objects: 1072, done. remote: Counting objects: 100% (320/320), done. remote: Compressing objects: 100% (171/171), done. remote: Total 1072 (delta 213), reused 241 (delta 149), pack-reused 752 Receiving objects: 100% (1072/1072), 588.23 KiB | 223.00 KiB/s, done. Resolving deltas: 100% (740/740), done. real 0m6.045s user 0m0.287s sys 0m0.087s [ilia@localhost temp]$ time curl -fsSL https://github.com/vtest/VTest/archive/master.tar.gz -o VTest.tar.gz real 0m8.185s user 0m0.071s sys 0m0.020s [ilia@localhost temp]$ I can swich to "curl" ifthere's more justification of that > > Best regards > Tim Düsterhus >
Re: [PATCH] move VTest installation to scripts/build-vtest.sh
Ilya, On 5/14/21 9:02 PM, Илья Шипицин wrote: let us unify VTest installation magic. I disagree with using 'git clone' here, cloning the repository with full history is wasteful. Please use the tarball approach from vtest.yml Best regards Tim Düsterhus
