Re: OSX builds in Travis

2020-07-15 Thread Willy Tarreau
On Sat, Jul 11, 2020 at 10:03:29PM +0500,  ??? wrote:
> I decided to switch to the latest osx available (list:
> https://docs.travis-ci.com/user/reference/osx/#macos-version )
> I attached updated patch

OK applied now, thank you Ilya! We'll see how it performs :-)

Willy



Re: OSX builds in Travis

2020-07-11 Thread Илья Шипицин
I decided to switch to the latest osx available (list:
https://docs.travis-ci.com/user/reference/osx/#macos-version )
I attached updated patch

сб, 11 июл. 2020 г. в 14:37, Vincent Bernat :

>  ❦ 11 juillet 2020 12:45 +05, Илья Шипицин:
>
> >> > he-he, brew bundle is deprecated (does not work)
> >> >
> >> >
> >>
> https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle
> >>
> >> It's very old. It has been added back at some point. Here is upstream
> >> recommending its use: https://github.com/Homebrew/brew/issues/2491
> >
> >
> > https://travis-ci.com/github/chipitsine/haproxy/jobs/359964175#L133
>
> It seems brew shipped by Travis is quite old. A `brew update` would fix
> this but then it adds 10 minutes to the build... So, back to square one.
> --
> Nothing so needs reforming as other people's habits.
> -- Mark Twain, "Pudd'nhead Wilson's Calendar"
>
From 16cf456954698a465b5bf6d2e937cd0224f6a6ec Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin 
Date: Sat, 11 Jul 2020 21:51:37 +0500
Subject: [PATCH] CI: travis-ci: speed up osx build by running brew scripted,
 switch to latest osx image

we used to use travis-ci brew plugin to install "socat", travis-ci brew
plugin works predictable in "all update" mode. sometimes it might take 12 minutes.

let us improve developer velocity by running brew from command line. It takes 2 minutes
instead of 12 minutes

latest osx seems to have more stable brew, let us switch to latest osx available.
osx images list: https://docs.travis-ci.com/user/reference/osx/#macos-version
---
 .travis.yml | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 39b52ce2c..cf227b3a5 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,9 +22,6 @@ addons:
   apt:
 update: true
 packages: [ liblua5.3-dev, libsystemd-dev, libpcre2-dev, clang-9, socat, ninja-build ]
-  homebrew:
-update: true
-packages: [ socat ]
 
 cache:
   directories:
@@ -94,8 +91,12 @@ matrix:
 compiler: clang
 env: TARGET=linux-glibc FLAGS= CC=clang-9
   - os: osx
+osx_image: xcode12
 if: type == push
 compiler: clang
+before_script:
+  - echo 'brew "socat"' > brew.bundle
+  - brew bundle --file=brew.bundle
 env: TARGET=osx FLAGS="USE_OPENSSL=1" OPENSSL_VERSION=1.1.1f
   - os: linux
 if: type == cron
-- 
2.26.2



Re: OSX builds in Travis

2020-07-11 Thread Vincent Bernat
 ❦ 11 juillet 2020 12:45 +05, Илья Шипицин:

>> > he-he, brew bundle is deprecated (does not work)
>> >
>> >
>> https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle
>>
>> It's very old. It has been added back at some point. Here is upstream
>> recommending its use: https://github.com/Homebrew/brew/issues/2491
>
>
> https://travis-ci.com/github/chipitsine/haproxy/jobs/359964175#L133

It seems brew shipped by Travis is quite old. A `brew update` would fix
this but then it adds 10 minutes to the build... So, back to square one.
-- 
Nothing so needs reforming as other people's habits.
-- Mark Twain, "Pudd'nhead Wilson's Calendar"



Re: OSX builds in Travis

2020-07-11 Thread Илья Шипицин
сб, 11 июл. 2020 г. в 10:06, Vincent Bernat :

>  ❦ 11 juillet 2020 00:48 +05, Илья Шипицин:
>
> > he-he, brew bundle is deprecated (does not work)
> >
> >
> https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle
>
> It's very old. It has been added back at some point. Here is upstream
> recommending its use: https://github.com/Homebrew/brew/issues/2491


https://travis-ci.com/github/chipitsine/haproxy/jobs/359964175#L133

>
>
> However, as long as socat is not part of the packages already installed,
> just `brew install socat` should always work.
> --
> It usually takes more than three weeks to prepare a good impromptu speech.
> -- Mark Twain
>


Re: OSX builds in Travis

2020-07-10 Thread Vincent Bernat
 ❦ 11 juillet 2020 00:48 +05, Илья Шипицин:

> he-he, brew bundle is deprecated (does not work)
>
> https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle

It's very old. It has been added back at some point. Here is upstream
recommending its use: https://github.com/Homebrew/brew/issues/2491

However, as long as socat is not part of the packages already installed,
just `brew install socat` should always work.
-- 
It usually takes more than three weeks to prepare a good impromptu speech.
-- Mark Twain



Re: OSX builds in Travis

2020-07-10 Thread Илья Шипицин
I attached a patch which speeds up osx builds by 10 minutes,

сб, 11 июл. 2020 г. в 00:48, Илья Шипицин :

> he-he, brew bundle is deprecated (does not work)
>
>
> https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle
>
> пт, 10 июл. 2020 г. в 14:55, Илья Шипицин :
>
>>
>>
>> пт, 10 июл. 2020 г. в 14:21, Vincent Bernat :
>>
>>>  ❦  9 juillet 2020 13:12 +05, Илья Шипицин:
>>>
>>> > do you think does it make sense to use scripted brew instead of travis
>>> > plugin ?
>>> >
>>> > if so, we can try to "brew instal blah-blah-blah || ok, we failed,
>>> lets'
>>> > update and install one more time"
>>>
>>> I have also hit the problem several time. Brew upstream says to use
>>> "brew bundle" instead:
>>>
>>> #v+
>>> -brew install libtool libxml2 check
>>> +brew bundle --file=- <<-EOS
>>> +brew "libtool"
>>> +brew "libxml2"
>>> +brew "check"
>>> +EOS
>>> #v-
>>>
>>> This way, brew doesn't complain if a requested package is already
>>> installed but not at the latest version.
>>>
>>
>> that's nice.
>>
>> I'll try to
>>
>> 1) get rid of socat
>> 2) use brew command line
>> 3) switch osx to Github Actions (thus we probably will run 4 builds in
>> parallel in travis and 4 builds in Github Actions)
>>
>>
>>
>>> --
>>> It is a wise father that knows his own child.
>>> -- William Shakespeare, "The Merchant of Venice"
>>>
>>
From 49063133cf8a8e8d6d35a2bda1117ea6609910fd Mon Sep 17 00:00:00 2001
From: Ilya Shipitsin 
Date: Sat, 11 Jul 2020 01:15:43 +0500
Subject: [PATCH] CI: travis-ci: speed up osx build by running brew scripted

we used to use travis-ci brew plugin to install "socat", travis-ci brew
plugin works predictable in "all update" mode. sometimes it might take 12 minutes.

let us improve developer velocity by running brew from command line. It takes 2 minutes
instead of 12 minutes,
---
 .travis.yml | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 39b52ce2c..e455c3c77 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,9 +22,6 @@ addons:
   apt:
 update: true
 packages: [ liblua5.3-dev, libsystemd-dev, libpcre2-dev, clang-9, socat, ninja-build ]
-  homebrew:
-update: true
-packages: [ socat ]
 
 cache:
   directories:
@@ -96,6 +93,8 @@ matrix:
   - os: osx
 if: type == push
 compiler: clang
+before_script:
+  - HOMEBREW_NO_AUTO_UPDATE=1 brew install socat
 env: TARGET=osx FLAGS="USE_OPENSSL=1" OPENSSL_VERSION=1.1.1f
   - os: linux
 if: type == cron
-- 
2.26.2



Re: OSX builds in Travis

2020-07-10 Thread Илья Шипицин
he-he, brew bundle is deprecated (does not work)

https://apple.stackexchange.com/questions/148454/brew-bundle-reporting-error-unknown-command-bundle

пт, 10 июл. 2020 г. в 14:55, Илья Шипицин :

>
>
> пт, 10 июл. 2020 г. в 14:21, Vincent Bernat :
>
>>  ❦  9 juillet 2020 13:12 +05, Илья Шипицин:
>>
>> > do you think does it make sense to use scripted brew instead of travis
>> > plugin ?
>> >
>> > if so, we can try to "brew instal blah-blah-blah || ok, we failed, lets'
>> > update and install one more time"
>>
>> I have also hit the problem several time. Brew upstream says to use
>> "brew bundle" instead:
>>
>> #v+
>> -brew install libtool libxml2 check
>> +brew bundle --file=- <<-EOS
>> +brew "libtool"
>> +brew "libxml2"
>> +brew "check"
>> +EOS
>> #v-
>>
>> This way, brew doesn't complain if a requested package is already
>> installed but not at the latest version.
>>
>
> that's nice.
>
> I'll try to
>
> 1) get rid of socat
> 2) use brew command line
> 3) switch osx to Github Actions (thus we probably will run 4 builds in
> parallel in travis and 4 builds in Github Actions)
>
>
>
>> --
>> It is a wise father that knows his own child.
>> -- William Shakespeare, "The Merchant of Venice"
>>
>


Re: OSX builds in Travis

2020-07-10 Thread Илья Шипицин
пт, 10 июл. 2020 г. в 14:21, Vincent Bernat :

>  ❦  9 juillet 2020 13:12 +05, Илья Шипицин:
>
> > do you think does it make sense to use scripted brew instead of travis
> > plugin ?
> >
> > if so, we can try to "brew instal blah-blah-blah || ok, we failed, lets'
> > update and install one more time"
>
> I have also hit the problem several time. Brew upstream says to use
> "brew bundle" instead:
>
> #v+
> -brew install libtool libxml2 check
> +brew bundle --file=- <<-EOS
> +brew "libtool"
> +brew "libxml2"
> +brew "check"
> +EOS
> #v-
>
> This way, brew doesn't complain if a requested package is already
> installed but not at the latest version.
>

that's nice.

I'll try to

1) get rid of socat
2) use brew command line
3) switch osx to Github Actions (thus we probably will run 4 builds in
parallel in travis and 4 builds in Github Actions)



> --
> It is a wise father that knows his own child.
> -- William Shakespeare, "The Merchant of Venice"
>


Re: OSX builds in Travis

2020-07-10 Thread Vincent Bernat
 ❦  9 juillet 2020 13:12 +05, Илья Шипицин:

> do you think does it make sense to use scripted brew instead of travis
> plugin ?
>
> if so, we can try to "brew instal blah-blah-blah || ok, we failed, lets'
> update and install one more time"

I have also hit the problem several time. Brew upstream says to use
"brew bundle" instead:

#v+
-brew install libtool libxml2 check
+brew bundle --file=- <<-EOS
+brew "libtool"
+brew "libxml2"
+brew "check"
+EOS
#v-

This way, brew doesn't complain if a requested package is already
installed but not at the latest version.
-- 
It is a wise father that knows his own child.
-- William Shakespeare, "The Merchant of Venice"



Re: OSX builds in Travis

2020-07-09 Thread Willy Tarreau
On Thu, Jul 09, 2020 at 01:35:57PM +0500,  ??? wrote:
> let it settle down a bit/
> 
> 
> I'll have a look in few days

Great, thanks!
Willy



Re: OSX builds in Travis

2020-07-09 Thread Илья Шипицин
let it settle down a bit/


I'll have a look in few days

чт, 9 июл. 2020 г. в 13:29, Willy Tarreau :

> On Thu, Jul 09, 2020 at 12:19:32PM +0500,  ??? wrote:
> > We install socat, because it is (or was?) needed for some tests. OSX
> > requires to update whole brew for that. Otherwise it works unstable
>
> Wow, so the we'd rather build and install socat ourselves from sources,
> because quite frankly here it looks like a full OS upgrade!
>
> Willy
>


Re: OSX builds in Travis

2020-07-09 Thread Willy Tarreau
On Thu, Jul 09, 2020 at 12:19:32PM +0500,  ??? wrote:
> We install socat, because it is (or was?) needed for some tests. OSX
> requires to update whole brew for that. Otherwise it works unstable

Wow, so the we'd rather build and install socat ourselves from sources,
because quite frankly here it looks like a full OS upgrade!

Willy



Re: OSX builds in Travis

2020-07-09 Thread Dinko Korunic
Hi Илья,

I think that Travis’ Homebrew plugin is just fine, but I would definitely avoid 
updating/upgrading Homebrew as that’s certainly going to make builds much 
slower.

Do you have any sample logs of the situation where socat failed to install with 
non-updated Homebrew? It doesn’t make sense that socat has been failing to 
install, as there are binary downloads available for almost every OSX and even 
more so, Formula/socat.rb is being changed really infrequently (once per year 
at most, according to git history for socat.rb).


> On 9 Jul 2020, at 10:07, Илья Шипицин  wrote:
> 
> we have homebrew --> update --> true
> 
> https://github.com/haproxy/haproxy/blob/master/.travis.yml#L26 
> 
> 
> 
> if we remove it, brew will not get updated.
> 
> 
> most of the time it is not an issue and we can install socat. but under some 
> circumstances socat refuses to install without brew update
> 
> чт, 9 июл. 2020 г. в 12:42, Dinko Korunic  >:
> I would suggest using HOMEBREW_NO_AUTO_UPDATE environment variable to avoid 
> Brew auto-updating where it’s not really needed, for instance as in:
> 
> HOMEBREW_NO_AUTO_UPDATE=1 brew install socat
> 
> If that doesn’t work (but I think it should), pinning will cause none of 
> dependancies to be installed automatically and only through manual install:
> 
> brew list | xargs brew pin
> brew install socat
> 
> 
>> On 9 Jul 2020, at 09:19, Илья Шипицин > > wrote:
>> 
>> We install socat, because it is (or was?) needed for some tests. OSX 
>> requires to update whole brew for that. Otherwise it works unstable
>> 
>> On Thu, Jul 9, 2020, 9:16 AM Willy Tarreau > > wrote:
>> Hi Ilya,
>> 
>> is it normal that the OSX build procedure in travis pulls gigabytes of
>> ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
>> and whatnot for many minutes ?
>> 
>>  https://travis-ci.com/github/haproxy/haproxy/jobs/359124175 
>> 
>> 
>> It's been doing this for more than 12 minutes now without even starting
>> to build haproxy. I'm suspecting that it's reinstalling a full-featured
>> desktop operating system, which seems awkward and counter productive at
>> best.
>> 
>> I don't know if that's automatically done based on broken dependencies
>> or if it is caused by a preliminary upgrade of the whole system, but I
>> think we need to address this as it's quite not efficient in this form.
>> 
>> Thanks!
>> Willy
> 
> -- 
> Dinko Korunic   ** Standard disclaimer applies **
> Sent from OSF1 osf1v4b V4.0 564 alpha
> 

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha



Re: OSX builds in Travis

2020-07-09 Thread Илья Шипицин
Dinko,

do you think does it make sense to use scripted brew instead of travis
plugin ?

if so, we can try to "brew instal blah-blah-blah || ok, we failed, lets'
update and install one more time"

чт, 9 июл. 2020 г. в 13:07, Илья Шипицин :

> we have homebrew --> update --> true
>
> https://github.com/haproxy/haproxy/blob/master/.travis.yml#L26
>
>
> if we remove it, brew will not get updated.
>
>
> most of the time it is not an issue and we can install socat. but under
> some circumstances socat refuses to install without brew update
>
> чт, 9 июл. 2020 г. в 12:42, Dinko Korunic :
>
>> I would suggest using HOMEBREW_NO_AUTO_UPDATE environment variable to
>> avoid Brew auto-updating where it’s not really needed, for instance as in:
>>
>> HOMEBREW_NO_AUTO_UPDATE=1 brew install socat
>>
>> If that doesn’t work (but I think it should), pinning will cause none of
>> dependancies to be installed automatically and only through manual install:
>>
>> brew list | xargs brew pin
>> brew install socat
>>
>>
>> On 9 Jul 2020, at 09:19, Илья Шипицин  wrote:
>>
>> We install socat, because it is (or was?) needed for some tests. OSX
>> requires to update whole brew for that. Otherwise it works unstable
>>
>> On Thu, Jul 9, 2020, 9:16 AM Willy Tarreau  wrote:
>>
>>> Hi Ilya,
>>>
>>> is it normal that the OSX build procedure in travis pulls gigabytes of
>>> ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
>>> and whatnot for many minutes ?
>>>
>>>  https://travis-ci.com/github/haproxy/haproxy/jobs/359124175
>>>
>>> It's been doing this for more than 12 minutes now without even starting
>>> to build haproxy. I'm suspecting that it's reinstalling a full-featured
>>> desktop operating system, which seems awkward and counter productive at
>>> best.
>>>
>>> I don't know if that's automatically done based on broken dependencies
>>> or if it is caused by a preliminary upgrade of the whole system, but I
>>> think we need to address this as it's quite not efficient in this form.
>>>
>>> Thanks!
>>> Willy
>>>
>>
>> --
>> Dinko Korunic   ** Standard disclaimer applies **
>> Sent from OSF1 osf1v4b V4.0 564 alpha
>>
>>


Re: OSX builds in Travis

2020-07-09 Thread Илья Шипицин
we have homebrew --> update --> true

https://github.com/haproxy/haproxy/blob/master/.travis.yml#L26


if we remove it, brew will not get updated.


most of the time it is not an issue and we can install socat. but under
some circumstances socat refuses to install without brew update

чт, 9 июл. 2020 г. в 12:42, Dinko Korunic :

> I would suggest using HOMEBREW_NO_AUTO_UPDATE environment variable to
> avoid Brew auto-updating where it’s not really needed, for instance as in:
>
> HOMEBREW_NO_AUTO_UPDATE=1 brew install socat
>
> If that doesn’t work (but I think it should), pinning will cause none of
> dependancies to be installed automatically and only through manual install:
>
> brew list | xargs brew pin
> brew install socat
>
>
> On 9 Jul 2020, at 09:19, Илья Шипицин  wrote:
>
> We install socat, because it is (or was?) needed for some tests. OSX
> requires to update whole brew for that. Otherwise it works unstable
>
> On Thu, Jul 9, 2020, 9:16 AM Willy Tarreau  wrote:
>
>> Hi Ilya,
>>
>> is it normal that the OSX build procedure in travis pulls gigabytes of
>> ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
>> and whatnot for many minutes ?
>>
>>  https://travis-ci.com/github/haproxy/haproxy/jobs/359124175
>>
>> It's been doing this for more than 12 minutes now without even starting
>> to build haproxy. I'm suspecting that it's reinstalling a full-featured
>> desktop operating system, which seems awkward and counter productive at
>> best.
>>
>> I don't know if that's automatically done based on broken dependencies
>> or if it is caused by a preliminary upgrade of the whole system, but I
>> think we need to address this as it's quite not efficient in this form.
>>
>> Thanks!
>> Willy
>>
>
> --
> Dinko Korunic   ** Standard disclaimer applies **
> Sent from OSF1 osf1v4b V4.0 564 alpha
>
>


Re: OSX builds in Travis

2020-07-09 Thread Dinko Korunic
I would suggest using HOMEBREW_NO_AUTO_UPDATE environment variable to avoid 
Brew auto-updating where it’s not really needed, for instance as in:

HOMEBREW_NO_AUTO_UPDATE=1 brew install socat

If that doesn’t work (but I think it should), pinning will cause none of 
dependancies to be installed automatically and only through manual install:

brew list | xargs brew pin
brew install socat


> On 9 Jul 2020, at 09:19, Илья Шипицин  wrote:
> 
> We install socat, because it is (or was?) needed for some tests. OSX requires 
> to update whole brew for that. Otherwise it works unstable
> 
> On Thu, Jul 9, 2020, 9:16 AM Willy Tarreau mailto:w...@1wt.eu>> 
> wrote:
> Hi Ilya,
> 
> is it normal that the OSX build procedure in travis pulls gigabytes of
> ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
> and whatnot for many minutes ?
> 
>  https://travis-ci.com/github/haproxy/haproxy/jobs/359124175 
> 
> 
> It's been doing this for more than 12 minutes now without even starting
> to build haproxy. I'm suspecting that it's reinstalling a full-featured
> desktop operating system, which seems awkward and counter productive at
> best.
> 
> I don't know if that's automatically done based on broken dependencies
> or if it is caused by a preliminary upgrade of the whole system, but I
> think we need to address this as it's quite not efficient in this form.
> 
> Thanks!
> Willy

-- 
Dinko Korunic   ** Standard disclaimer applies **
Sent from OSF1 osf1v4b V4.0 564 alpha



Re: OSX builds in Travis

2020-07-09 Thread Илья Шипицин
We install socat, because it is (or was?) needed for some tests. OSX
requires to update whole brew for that. Otherwise it works unstable

On Thu, Jul 9, 2020, 9:16 AM Willy Tarreau  wrote:

> Hi Ilya,
>
> is it normal that the OSX build procedure in travis pulls gigabytes of
> ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
> and whatnot for many minutes ?
>
>  https://travis-ci.com/github/haproxy/haproxy/jobs/359124175
>
> It's been doing this for more than 12 minutes now without even starting
> to build haproxy. I'm suspecting that it's reinstalling a full-featured
> desktop operating system, which seems awkward and counter productive at
> best.
>
> I don't know if that's automatically done based on broken dependencies
> or if it is caused by a preliminary upgrade of the whole system, but I
> think we need to address this as it's quite not efficient in this form.
>
> Thanks!
> Willy
>


OSX builds in Travis

2020-07-08 Thread Willy Tarreau
Hi Ilya,

is it normal that the OSX build procedure in travis pulls gigabytes of
ruby and python crap, including fonts, libgpg, gnutls, qt, postgresql
and whatnot for many minutes ?

 https://travis-ci.com/github/haproxy/haproxy/jobs/359124175

It's been doing this for more than 12 minutes now without even starting
to build haproxy. I'm suspecting that it's reinstalling a full-featured
desktop operating system, which seems awkward and counter productive at
best.

I don't know if that's automatically done based on broken dependencies
or if it is caused by a preliminary upgrade of the whole system, but I
think we need to address this as it's quite not efficient in this form.

Thanks!
Willy