Re: Travis CI timeouts for MacPorts builds

[Ryan Schmidt]

On Sep 9, 2020, at 06:31, Zero King wrote:

> However, GitHub Actions does not provide a list of IP addresses
> AFAIK, so integration with paste.macports.org would be difficult. Maybe
> I should just use the other pastebin and hardcode the URL into mpbot-ci
> to avoid it being inherited in macports-ports forks, but that would
> make paste.macports.org useless (at least for CI).
> 

We should use our own paste.macports.org service, not a third-party service.

We need the list of IP addresses also so that we can allow access to private 
archives. As I did for Azure Pipelines, I can examine the CDN access logs to 
try to determine the IP addresses and add them to the whitelist.

Re: Travis CI timeouts for MacPorts builds

[Zero King]

On Tue, Sep 08, 2020 at 10:12:08PM +0200, Mojca Miklavec wrote:

On Tue, 8 Sep 2020 at 20:49, Zero King wrote:


> I forgot why we never switched from Azure to GitHub Actions.

Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
remove Travis CI and Azure Pipelines soon.


Awesome, thank you very much!

I'm just confused about one thing. The docs says that only 10.15 is supported:
   
https://docs.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners
while the github actions lists 10.14. Is the truth somewhere inbetween?


GitHub Actions and Azure Pipelines share the same infrastructure, so in
fact both 10.14 and 10.15 are supported.

https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops

I have disabled Travis failures, so PRs failing on Travis will not cause
errored checks anymore. macOS 10.15 is added to the Actions workflow,
and I will remove the Azure Pipelines integration if it is proven stable
and gets pastebin integration.

GitHub Actions can trigger webhook events, and go-github have added
support for Actions related APIs, so it should be easier to implement
the CI result comment with GitHub Actions than Azure Pipelines or
Travis.

However, GitHub Actions does not provide a list of IP addresses
AFAIK, so integration with paste.macports.org would be difficult. Maybe
I should just use the other pastebin and hardcode the URL into mpbot-ci
to avoid it being inherited in macports-ports forks, but that would
make paste.macports.org useless (at least for CI).

[1] 
https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads#workflow_run
[2] https://github.com/google/go-github

On Tue, Sep 08, 2020 at 10:42:57PM +0200, Clemens Lang wrote:

GitHub Actions' IPs should be added to the pastebin allowlist. Logs
are uploaded as artifacts, but they are zipped[1], so uploading to
paste.macports.org is still preferred.


Is there a documented IP block used by GitHub Actions?


Unfortunately, I have not found any documentation about that in their
docs, but the Azure Pipelines docs did mention that they were
investigating options.

"Our Mac IP ranges are not included in the Azure IPs above, though we
are investigating options to publish these in the future."

On Tue, Sep 08, 2020 at 10:51:31PM +0200, Mojca Miklavec wrote:

On Tue, 8 Sep 2020 at 22:43, Clemens Lang wrote:

On Tue, Sep 08, 2020 at 06:48:58PM +, Zero King wrote:
> On Tue, Sep 08, 2020 at 06:39:41PM +0200, Mojca Miklavec wrote:
> > After a while Azure came around with 6 hours of timeout which was a
> > lot more useful, and the same solution was ported from Travis to
> > Azure. I forgot why we never switched from Azure to GitHub Actions.
>
> Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
> remove Travis CI and Azure Pipelines soon.

I'm not sure this was a good choice. GitHub recently announced stricter
limits on CPU minutes available though GitHub actions.


I was only aware of GitLab decreasing the limits. According to
https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions
we have 200 free minutes per month? That's literally useless if true.


The 2,000 minutes limitation only applies to private repositories.


(It's not clear to me whether Azure will stay free forever. We should
probably be looking into setting up our own builders in the long run.)


That's true, but we need a volunteer and hardware resources.


Mojca

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Tue, 8 Sep 2020 at 19:25, Ralph Seichter wrote:
>
> * Mojca Miklavec:
>
> >> If I can assist, let me know, but I don't think it likely at this
> >> point.
> >
> > Because you lack time or for some other reason?
>
> I would make the time for MacPorts. My assumption is just that I still
> know to little about how the MacPorts builds work to be of assistance,
> and that "you guys" have matters in hand.

Our manpower is pretty limited.
There are many contributors to ports, but not so many who contribute
to the base & infrastructure. It's a very small group of people with
full-time jobs and other hobbies. To the best of my knowledge nobody
is actively working on solving this particular challenge to better
address building PRs (it's doable, it just requires some time
investment).

So any helping hand would be extremely welcome.

> If you think that I can
> actually help, and if there is documentation or source code available,
> please let me know. I'm ready for some reading. ;-)

I definitely believe that new contributors can bring in a lot of
useful stuff, and if you are willing to tackle this challenge, we
would certainly be very very grateful.

I can provide some pointers to the existing code, but I'm pretty sure
that it would be more of a hindrance than of any actual help. To
clarify: you can either decide to start improving the existing tools
(and in that case having a better understanding of them is a must),
but proof-of-concept for disposable images for building ports in PRs
doesn't require any special knowledge about the current codebase.

There are the following parts that need to be addressed:

(1.)
  (a) prepare a set of images with different macOS versions (for as
far back in history as it gets, but start with one)
  (b) for later: find the easiest way to keep updating MacPorts and
macOS inside the base image on regular bases
(2.)
  (a) figure out how to fire up a disposable image, build a
hello_world.cpp on it, report build success/failure and destroy the VM
  (b) figure out how to do that using either
http://docs.buildbot.net/latest/manual/configuration/workers-libvirt.html
or something equivalent

The repository for infrastructure is mostly here (again, no need to
understand any of it right now unless you want to contribute there
directly with other improvements):
https://github.com/macports/macports-infrastructure
I'm not sure if we have some really good pieces of documentation about
how it all works (there is some internal documentation for those who
run the server), but feel free to ask if there is any part of the
project where you would like to contribute to in particular.

Ryan is currently working on a rewrite of buildbot setup, so the above
link is already "old" in a sense.

Mojca

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Tue, 8 Sep 2020 at 22:43, Clemens Lang wrote:
> On Tue, Sep 08, 2020 at 06:48:58PM +, Zero King wrote:
> > On Tue, Sep 08, 2020 at 06:39:41PM +0200, Mojca Miklavec wrote:
> > > After a while Azure came around with 6 hours of timeout which was a
> > > lot more useful, and the same solution was ported from Travis to
> > > Azure. I forgot why we never switched from Azure to GitHub Actions.
> >
> > Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
> > remove Travis CI and Azure Pipelines soon.
>
> I'm not sure this was a good choice. GitHub recently announced stricter
> limits on CPU minutes available though GitHub actions.

I was only aware of GitLab decreasing the limits. According to
https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/about-billing-for-github-actions
we have 200 free minutes per month? That's literally useless if true.

(It's not clear to me whether Azure will stay free forever. We should
probably be looking into setting up our own builders in the long run.)

Mojca

Re: Travis CI timeouts for MacPorts builds

[Clemens Lang]

Hi,

On Tue, Sep 08, 2020 at 06:48:58PM +, Zero King wrote:
> On Tue, Sep 08, 2020 at 06:39:41PM +0200, Mojca Miklavec wrote:
> > After a while Azure came around with 6 hours of timeout which was a
> > lot more useful, and the same solution was ported from Travis to
> > Azure. I forgot why we never switched from Azure to GitHub Actions.
> 
> Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
> remove Travis CI and Azure Pipelines soon.

I'm not sure this was a good choice. GitHub recently announced stricter
limits on CPU minutes available though GitHub actions. IIRC those limits
will be per project per month, which is especially bad for us because
most of our CI usage is in a single project, and we usually need many
minutes per build. Additional minutes will require payment.

I'm not aware of any such limitation with Azure at the moment, except
for the per-build time limit, so I would prefer to stay with them, since
it scales with the number of PRs we get. I do agree on dropping Travis,
though. It just fails too often to be useful and is generally slow.


> GitHub Actions' IPs should be added to the pastebin allowlist. Logs
> are uploaded as artifacts, but they are zipped[1], so uploading to
> paste.macports.org is still preferred.

Is there a documented IP block used by GitHub Actions?

-- 
Clemens

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Tue, 8 Sep 2020 at 20:49, Zero King wrote:
>
> > I forgot why we never switched from Azure to GitHub Actions.
>
> Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
> remove Travis CI and Azure Pipelines soon.

Awesome, thank you very much!

I'm just confused about one thing. The docs says that only 10.15 is supported:

https://docs.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners
while the github actions lists 10.14. Is the truth somewhere inbetween?

Mojca

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Tue, 8 Sep 2020 at 19:11, Ralph Seichter wrote:
>
> * Ryan Schmidt:
>
> >> Tries being the operative word here. ;-)
> >
> > What are you trying to say?
>
> Did I step on somebody's toes?

I'm not aware why you would step on anyone's toes, but I also didn't
understand what you tried to say in that sentence.

> I am saying that the build that concerns
> me timed out setting up the dependencies. That, from my understanding,
> means the dependencies were either unavailable in binary form, or that
> the attempt to access said binary form failed. Hence my comment above,
> in an attempt to make light of the situation, as the smiley indicates.
>
> > We would need to see logs of the installation of the dependencies to
> > know why they're taking so long, but I don't see any such logs, and
> > don't know where or how that's happening or not happening.
>
> That makes tackling the underlying issue difficult, of course.

You can in fact look at the equivalent builds from Azure, for example here:
https://dev.azure.com/macports/macports-ports/_build/results?buildId=9200=logs=dde54236-b8c5-5879-af99-818e7b59912f=204c7e84-6a60-5987-1292-f9e129aab6e1

At the bottom you have a pastebin link with the build log of the port itself.
However the logs from building dependencies are apparently missing. I
thought those were available as well.
I'm not 100% sure, but maybe we deliberately removed those at some
point because it was consuming significant resources and it was deemed
nearly useless. (I'm not saying that's true in your case.)

> > We don't "ignore Travis results".
>
> Mojca Miklavec wrote (quote) "Generally we mostly ignore Travis results,
> so there's no real reason to worry in your case.", which led me to
> believe that you generally mostly ignore Travis results. How silly of
> me. :-)

Just to clarify my oversimplification.

We don't literally ignore the results.

If the Travis builds succeed ... we usually ignore the details :) :) :)

If the Travis build fails, we would usually look into the reason for
the failure, however:

(1) In nearly all the cases (not all though!!!) the reason for the
failure is either timeout or some network or timing issue. If that
happens, we would "ignore" the fact that Travis build failed. What
this mostly means is that users should not worry that their PR won't
be accepted just because Travis decided to put a red cross to the CI
job. In some cases the port would be manually checked by someone else
to ensure that it builds. We could have payed a bit more attention to
the individual builds to see if there is some flaw and we could have
forced the dependencies to build faster.

(2) In some other cases there would be a real build failure, but again
this most often means that it would fail everywhere, so we would see a
failure on Azure as well.

(3) There's another large class of cases where the software is only
supported on, say, 10.14 and newer. In those cases the Azure build
would succeed (just because it only uses the newer OS versions) and
the Travis build would fail (as it tries to build on older builders).
But again, if someone is porting software that upstream doesn't bother
supporting on older OSes, we won't generally block the PR just because
the older macOS versions are no longer compatible.

There certainly are cases where the results are useful. It's just not
as often as we wish it would be.

Mojca

Re: Travis CI timeouts for MacPorts builds

[Zero King]

Hello,

On Tue, Sep 08, 2020 at 06:39:41PM +0200, Mojca Miklavec wrote:

On Mon, 7 Sep 2020 at 20:25, Ralph Seichter wrote:

If you ignore Travis results, why run Travis CI in the first place?


Travis was the first CI that we supported for pull requests.
@l2dy wrote the initial support as a GSOC project which included
writing tons of workarounds for Travis limitations.
He redirected output to pastebin in order to be able to even show it
(most of our output exceeds the Travis limit for how long the log can
be), he had to inject bogus writes to prevent Travis from thinking
that the job was stuck, he implemented a lot of tricks to ensure that
the bootstrapping phase was as fast as possible, ... but we could not
change the time limit itself.

After a while Azure came around with 6 hours of timeout which was a
lot more useful, and the same solution was ported from Travis to
Azure. I forgot why we never switched from Azure to GitHub Actions.


Done in commit db7b40d8691e1fcd8b6be5e3c2a2a00a7ce0bdf4. We should
remove Travis CI and Azure Pipelines soon.

GitHub Actions' IPs should be added to the pastebin allowlist. Logs are
uploaded as artifacts, but they are zipped[1], so uploading to
paste.macports.org is still preferred.

[1]: 
https://github.com/actions/upload-artifact/blob/v2/README.md#zipped-artifact-downloads


It's not that Travis is completely ignored. There are some useful
cases, like this one for example:
   https://travis-ci.org/github/macports/macports-ports/builds/724968783
(I checked a random failed build from the recent PRs.)

You can see that the build passed on 10.14, but failed on 10.11 up to
10.13. It failed quickly with
   "Xcode 9.0 or greater is needed to build carthage; only found
version 7.3.1."
which is something that the portfile should have declared and it
should have at least failed in pre-fetch phase already (ideally we
would need a declarative syntax for this).

This case is useful. Now, we do have Azure build for 10.13, so this
one is overlapping (and we should probably remove that one), but
there's no builder for 10.11 and 10.12 on Azure.

In reality most of the Travis build failures fall into two categories:
- timeout (most of the time)
- network or timing issues (sources could not be fetched with "server
not found", or the clocks didn't match, and MacPorts refuses to run if
file appear to be "from the future")

As far as I know there were attempts to address both of these. Timeout
will always be the case when a single PR touches many ports, or where
ports are simply time-consuming to build (no chance for Qt, ...) There
might be room for improvement as far as build time of dependencies is
concerned though.

On Tue, 8 Sep 2020 at 18:28, Ryan Schmidt wrote:

On Sep 7, 2020, at 11:24, Ruben Di Battista wrote:

> Would à Linux machine with some virtualization method (libvirt?) be 
acceptable as CI runner?

I can't see how, since we're trying to verify builds of ports, and there is no 
general expectation that port builds would work on non-Mac operating systems.


We are not talking about running the builds on Linux, but running the
virtual machines on Linux, and if wasn't for the legal reasons, this
would be a perfectly viable solution (provided you get all the VMs
working, to start with). But if you get it working on linux for
testing purposes, a very similar recipe might work on macOS.

Mojca

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Mojca Miklavec:

>> If I can assist, let me know, but I don't think it likely at this
>> point.
>
> Because you lack time or for some other reason?

I would make the time for MacPorts. My assumption is just that I still
know to little about how the MacPorts builds work to be of assistance,
and that "you guys" have matters in hand. If you think that I can
actually help, and if there is documentation or source code available,
please let me know. I'm ready for some reading. ;-)

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Ryan Schmidt:

>> Tries being the operative word here. ;-)
>
> What are you trying to say?

Did I step on somebody's toes? I am saying that the build that concerns
me timed out setting up the dependencies. That, from my understanding,
means the dependencies were either unavailable in binary form, or that
the attempt to access said binary form failed. Hence my comment above,
in an attempt to make light of the situation, as the smiley indicates.

> We would need to see logs of the installation of the dependencies to
> know why they're taking so long, but I don't see any such logs, and
> don't know where or how that's happening or not happening.

That makes tackling the underlying issue difficult, of course.

> We don't "ignore Travis results".

Mojca Miklavec wrote (quote) "Generally we mostly ignore Travis results,
so there's no real reason to worry in your case.", which led me to
believe that you generally mostly ignore Travis results. How silly of
me. :-)

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ryan Schmidt]

On Sep 8, 2020, at 11:39, Mojca Miklavec wrote:

> On Tue, 8 Sep 2020 at 18:28, Ryan Schmidt wrote:
>> On Sep 7, 2020, at 11:24, Ruben Di Battista wrote:
>> 
>>> Would à Linux machine with some virtualization method (libvirt?) be 
>>> acceptable as CI runner?
>> 
>> I can't see how, since we're trying to verify builds of ports, and there is 
>> no general expectation that port builds would work on non-Mac operating 
>> systems.
> 
> We are not talking about running the builds on Linux, but running the
> virtual machines on Linux, and if wasn't for the legal reasons, this
> would be a perfectly viable solution (provided you get all the VMs
> working, to start with). But if you get it working on linux for
> testing purposes, a very similar recipe might work on macOS.

Ah. Sure, you could develop a method on Linux if you really wanted to, but to 
actually run macOS VMs, it'll have to run on a Mac.

Re: Travis CI timeouts for MacPorts builds

[Ryan Schmidt]

On Sep 7, 2020, at 13:25, Ralph Seichter wrote:

> * Mojca Miklavec:
> 
>> It tries to install dependencies as binaries, provided the binaries
>> are available.
> 
> Tries being the operative word here. ;-)

What are you trying to say? It uses binary archives if available, and they 
should be available. Private archives are available to the CI machines provided 
their IP address has been whitelisted in our CDN config. I just verified that 
the Travis IP addresses haven't changed.

It is possible that for some reason it is unable to access the private 
archives. Or it's possible that downloading the private archives is taking too 
long. (They're hosted on a server with limited bandwidth. We have a CDN in 
front, but if the file is not on the CDN yet, it needs to come from the slow 
server.) We would need to see logs to know for sure what's happening.


>> We should figure out:
>> - which dependencies time out
>> - why they are not installed from (the private) binary package repository
> 
> I'm guessing that "we" means the devs who maintain the build config. If
> I can assist, let me know, but I don't think it likely at this point.

I deal with the buildbot infrastructure but am less familiar with how we've set 
things up at Travis and Azure. We would need to see logs of the installation of 
the dependencies to know why they're taking so long, but I don't see any such 
logs, and don't know where or how that's happening or not happening.


> At a glance, I'd expect a glib2 build from source to be quite a drag.
> The various xorg-* ports might also take some time. I don't see why the
> latter would be required,

Both glib2 and most of the xorg ports, as far as I recall, are distributable, 
meaning binary archives should be publicly available.

> given that the build apparently did not use
> the +emacs variant?

The build infrastructure, both buildbot and Travis and Azure, builds for 
default variants only.


>> Generally we mostly ignore Travis results, so there's no real reason
>> to worry in your case.
> 
> If you ignore Travis results, why run Travis CI in the first place? I
> suggest to either fix the underlying issue or remove Travis builds. That
> would at least free up some server resources, for the benefit of other
> projects which actually rely on their Travis builds.

We don't "ignore Travis results". However if a pull request comes in that shows 
a failure from Travis due to the 50 minute timeout, and successful builds from 
Azure, we'll accept the PR. I already said that we don't need two different 
systems testing the same OS/Xcode combination and if we're currently doing that 
then we should get rid of the brittle Travis builds for those OS/Xcode 
combinations and use just the more reliable Azure ones.

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Mon, 7 Sep 2020 at 20:25, Ralph Seichter wrote:
> * Mojca Miklavec:
>
> > We should figure out:
> > - which dependencies time out
> > - why they are not installed from (the private) binary package repository
>
> I'm guessing that "we" means the devs who maintain the build config.

Oh, no, that's the wrong interpretation.
"We" includes anyone interested in helping out.
There are certain tasks (like enabling some external service on the
github, connecting with some new hardware, ...) that can only be done
by a small group of people with administration access, but everything
else happens via pull requests (or sometimes direct commits) or new
code / repositories.

The community depends exclusively on volunteers. Many of the
contributors who devoted more time in the past and made some major
contributions have a lot less time to spend with the project nowadays,
so anyone willing to help out is mostly welcome.

> If
> I can assist, let me know, but I don't think it likely at this point.

Because you lack time or for some other reason?
If you think you could help, we would be most grateful.

> If you ignore Travis results, why run Travis CI in the first place?

Travis was the first CI that we supported for pull requests.
@l2dy wrote the initial support as a GSOC project which included
writing tons of workarounds for Travis limitations.
He redirected output to pastebin in order to be able to even show it
(most of our output exceeds the Travis limit for how long the log can
be), he had to inject bogus writes to prevent Travis from thinking
that the job was stuck, he implemented a lot of tricks to ensure that
the bootstrapping phase was as fast as possible, ... but we could not
change the time limit itself.

After a while Azure came around with 6 hours of timeout which was a
lot more useful, and the same solution was ported from Travis to
Azure. I forgot why we never switched from Azure to GitHub Actions.

It's not that Travis is completely ignored. There are some useful
cases, like this one for example:
https://travis-ci.org/github/macports/macports-ports/builds/724968783
(I checked a random failed build from the recent PRs.)

You can see that the build passed on 10.14, but failed on 10.11 up to
10.13. It failed quickly with
"Xcode 9.0 or greater is needed to build carthage; only found
version 7.3.1."
which is something that the portfile should have declared and it
should have at least failed in pre-fetch phase already (ideally we
would need a declarative syntax for this).

This case is useful. Now, we do have Azure build for 10.13, so this
one is overlapping (and we should probably remove that one), but
there's no builder for 10.11 and 10.12 on Azure.

In reality most of the Travis build failures fall into two categories:
- timeout (most of the time)
- network or timing issues (sources could not be fetched with "server
not found", or the clocks didn't match, and MacPorts refuses to run if
file appear to be "from the future")

As far as I know there were attempts to address both of these. Timeout
will always be the case when a single PR touches many ports, or where
ports are simply time-consuming to build (no chance for Qt, ...) There
might be room for improvement as far as build time of dependencies is
concerned though.

On Tue, 8 Sep 2020 at 18:28, Ryan Schmidt wrote:
> On Sep 7, 2020, at 11:24, Ruben Di Battista wrote:
>
> > Would à Linux machine with some virtualization method (libvirt?) be 
> > acceptable as CI runner?
>
> I can't see how, since we're trying to verify builds of ports, and there is 
> no general expectation that port builds would work on non-Mac operating 
> systems.

We are not talking about running the builds on Linux, but running the
virtual machines on Linux, and if wasn't for the legal reasons, this
would be a perfectly viable solution (provided you get all the VMs
working, to start with). But if you get it working on linux for
testing purposes, a very similar recipe might work on macOS.

Mojca

Re: Travis CI timeouts for MacPorts builds

[Ryan Schmidt]

On Sep 7, 2020, at 11:24, Ruben Di Battista wrote:

> Would à Linux machine with some virtualization method (libvirt?) be 
> acceptable as CI runner? 

I can't see how, since we're trying to verify builds of ports, and there is no 
general expectation that port builds would work on non-Mac operating systems.

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Mojca Miklavec:

> What we need is:
> - a list of recipes to set up images for a bunch of different macOS
> versions (as far back into history as possible)
> - a recipe for how to fire up a VM, do something basic (port install
> foo) and trash the result

Looking at the Travis CI config and the raw Travis logs, I don't see
much. Are more detailed log files available? The magic seems to happen
in the code hosted in mpbot-github.

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Mojca Miklavec:

> It tries to install dependencies as binaries, provided the binaries
> are available.

Tries being the operative word here. ;-)

> We should figure out:
> - which dependencies time out
> - why they are not installed from (the private) binary package repository

I'm guessing that "we" means the devs who maintain the build config. If
I can assist, let me know, but I don't think it likely at this point.

At a glance, I'd expect a glib2 build from source to be quite a drag.
The various xorg-* ports might also take some time. I don't see why the
latter would be required, given that the build apparently did not use
the +emacs variant?

> Generally we mostly ignore Travis results, so there's no real reason
> to worry in your case.

If you ignore Travis results, why run Travis CI in the first place? I
suggest to either fix the underlying issue or remove Travis builds. That
would at least free up some server resources, for the benefit of other
projects which actually rely on their Travis builds.

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Mon, 7 Sep 2020 at 18:25, Ruben Di Battista wrote:
>
> I was wondering... Would à Linux machine with some virtualization method 
> (libvirt?) be acceptable as CI runner?

This is something we've been looking at, among others.
Now, a Linux machine with libvirt is probably not entirely legal, but
a mac with libvirt should be, and I guess it would work in exactly the
same way.

We tried to follow these instructions:
https://github.com/kholia/OSX-KVM
and we got it mostly working, it's just annoyingly slow because the
image gets downloaded as part of the process (I don't know if you can
easily cache it if the installation fails).

The other thing is that I'm not sure if one could get this working
with older OS versions, or how to handle ARM.

Running the image on macOS should be acceptable.

> Any of you has experience in terms of performance?

I wouldn't worry too much. It's not like we are getting thousand PRs
per day. And if that becomes the bottleneck, we could always set up
another worker for the same macOS version.

We are already running the existing builders inside virtual machines.
It may take two weeks when the new macOS comes out (and days after
1000+ perl submodules get new subports), but the machines keep up with
the load. The only bottleneck was PowerMac.

> Also, Gitlab CI allows to attach personal runners to project very easily 
> (just a package to install from the os package manager) . How does it work 
> for Github?

GitHub doesn't have the CI as "tightly integrated" as the gitlab runners.
Basically you can "attach" almost any given CI.
If you look at commit history, you'll see green checks and red crosses.
If you click on those, you'll see the list of builds with links to our
buildbot builds.

Those are the "official builds" of the packages. We could have
something similar for the CI.

On Mon, 7 Sep 2020 at 18:52, Ralph Seichter wrote:
>
> * Mojca Miklavec:
>
> > If you volunteer to do some research / work in this area ... that
> > would likely be the most significant step in "increasing your chances"
>
> I actually have some experience in this field. I use GitLab (Omnibus
> edition), Docker, and GitLab Runners inside Docker for CI/CD. That's
> basically the setup which gitlab.com uses (albeit sans their S3-based
> caching), and it can handle serious builds.
>
> Doing research re MacPorts would require knowledge about the MP build
> infrastructure, which I don't yet have. I'd be willing to look into it,
> though, if that infrastructure is properly documented somewhere.

The part that requires research hasn't been implemented yet, so the
good news is that you don't need any special knowledge about the
existing infrastructure :)

What we need is:
- a list of recipes to set up images for a bunch of different macOS
versions (as far back into history as possible)
- a recipe for how to fire up a VM, do something basic (port install
foo) and trash the result

The next step would be to perhaps integrate this into buildbot, but
there's no need to use exactly buildbot for this. Anything else that
does the job would also serve just fine.

Mojca

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Mon, 7 Sep 2020 at 18:44, Ralph Seichter wrote:
> * Joshua Root:
>
> >> [1] https://travis-ci.org/github/macports/macports-ports/jobs/724689780
> >
> > It's just a matter of how long your port takes to build (including
> > installing all its dependencies).
>
> Notmuch, which is what was built in the job [1], is small and builds in
> less than a minute on my MacBook once the prerequisites are installed.
> Perhaps you'd like to try it for yourself.
>
> Is the Travis-based build just installing the dependencies (as in: the
> binary packages), or does it also build the dependencies from source?

It tries to install dependencies as binaries, provided the binaries
are available.

The job [1] clearly times out during installation of dependencies.
We should figure out:
- which dependencies time out
- why they are not installed from (the private) binary package repository

> I suspect the latter. I actually tried that here, and it took ages,
> even on my octa-core machine.

There are some packages that are not distributable for licence
conflicts. On your own machine they would always be installed from
source unless we figure out that the licence conflict resolution is
too greedy and fix it by either removing some dependencies or by
explicitly declaring a dependency as a non-conflicting one.

But on Travis & Azure those dependencies should be fetched from the
private archive. If we find out why they are not, we might easily find
the cure for your particular case (or maybe for many other cases as
well). We could start by investigating the logs revealing which
packages built from source on Azure.

Generally we mostly ignore Travis results, so there's no real reason
to worry in your case. It's just that we might be able to solve a
larger cluster of builds if we diagnose & fix the problem.

(We should probably write the list of dependencies as they are being
installed on stdout inside the CI script, without the debug output of
course.)

Mojca

Re: Travis CI timeouts for MacPorts builds

[Joshua Root]

On 2020-9-8 02:44 , Ralph Seichter wrote:
> * Joshua Root:
> 
>>> [1] https://travis-ci.org/github/macports/macports-ports/jobs/724689780
>>
>> It's just a matter of how long your port takes to build (including
>> installing all its dependencies).
> 
> Notmuch, which is what was built in the job [1], is small and builds in
> less than a minute on my MacBook once the prerequisites are installed.
> Perhaps you'd like to try it for yourself.
> 
> Is the Travis-based build just installing the dependencies (as in: the
> binary packages), or does it also build the dependencies from source?
> I suspect the latter. I actually tried that here, and it took ages,
> even on my octa-core machine.

Yes, if the VM's clock is to be believed it took about 12 seconds to
install notmuch itself on Azure once the 181 dependencies were
installed. But the build as a whole took 1 hour and 5 minutes.



The ports modified by the PR are always built from source, but for
dependencies it should be installing from archives if possible.

- Josh

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Ryan Schmidt:

> I feel that the Azure Pipelines are already giving us good results on
> current systems, and we could probably turn off the Travis builds for
> the systems that Azure also covers.

That seems like a reasonable approach to me.

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Mojca Miklavec:

> If you volunteer to do some research / work in this area ... that
> would likely be the most significant step in "increasing your chances"

I actually have some experience in this field. I use GitLab (Omnibus
edition), Docker, and GitLab Runners inside Docker for CI/CD. That's
basically the setup which gitlab.com uses (albeit sans their S3-based
caching), and it can handle serious builds.

Doing research re MacPorts would require knowledge about the MP build
infrastructure, which I don't yet have. I'd be willing to look into it,
though, if that infrastructure is properly documented somewhere.

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ralph Seichter]

* Joshua Root:

>> [1] https://travis-ci.org/github/macports/macports-ports/jobs/724689780
>
> It's just a matter of how long your port takes to build (including
> installing all its dependencies).

Notmuch, which is what was built in the job [1], is small and builds in
less than a minute on my MacBook once the prerequisites are installed.
Perhaps you'd like to try it for yourself.

Is the Travis-based build just installing the dependencies (as in: the
binary packages), or does it also build the dependencies from source?
I suspect the latter. I actually tried that here, and it took ages,
even on my octa-core machine.

-Ralph

Re: Travis CI timeouts for MacPorts builds

[Ruben Di Battista]

I was wondering... Would à Linux machine with some virtualization method
(libvirt?) be acceptable as CI runner? Any of you has experience in terms
of performance?

Also, Gitlab CI allows to attach personal runners to project very easily
(just a package to install from the os package manager) . How does it work
for Github?

On Mon, 7 Sep 2020, 15:50 Mojca Miklavec,  wrote:

> On Mon, 7 Sep 2020 at 10:46, Ralph Seichter wrote:
> >
> > I fail to remember the last time one of my builds successfully passed
> > Travis CI. All I see are timeouts [1]. Other peoples' jobs apparently
> > make it through Travis OK, so I wander what I can do to increase my
> > chances?
>
> By far the best thing would be to implement the functionality that
> would allow firing up a clean VM, do the build, close the VM & destroy
> it, and run this on our own buildbot workers.
>
> This requires both manpower to implement the functionality as well as
> additional hardware resources to run the builds.
> If we solve the first part, I guess we should be able to solve the
> second one as well.
> If you volunteer to do some research / work in this area ... that
> would likely be the most significant step in "increasing your chances"
> ... probably up to 100% for the ports that have no chance in building
> in < 50 minutes on a super slow hardware owned by Travis :)
>
> Mojca
>

Re: Travis CI timeouts for MacPorts builds

[Ryan Schmidt]

> On Sep 7, 2020, at 08:50, Mojca Miklavec wrote:
> 
> On Mon, 7 Sep 2020 at 10:46, Ralph Seichter wrote:
>> 
>> I fail to remember the last time one of my builds successfully passed
>> Travis CI. All I see are timeouts [1]. Other peoples' jobs apparently
>> make it through Travis OK, so I wander what I can do to increase my
>> chances?
> 
> By far the best thing would be to implement the functionality that
> would allow firing up a clean VM, do the build, close the VM & destroy
> it, and run this on our own buildbot workers.
> 
> This requires both manpower to implement the functionality as well as
> additional hardware resources to run the builds.
> If we solve the first part, I guess we should be able to solve the
> second one as well.
> If you volunteer to do some research / work in this area ... that
> would likely be the most significant step in "increasing your chances"
> ... probably up to 100% for the ports that have no chance in building
> in < 50 minutes on a super slow hardware owned by Travis :)

I feel that the Azure Pipelines are already giving us good results on current 
systems, and we could probably turn off the Travis builds for the systems that 
Azure also covers.

Re: Travis CI timeouts for MacPorts builds

[Mojca Miklavec]

On Mon, 7 Sep 2020 at 10:46, Ralph Seichter wrote:
>
> I fail to remember the last time one of my builds successfully passed
> Travis CI. All I see are timeouts [1]. Other peoples' jobs apparently
> make it through Travis OK, so I wander what I can do to increase my
> chances?

By far the best thing would be to implement the functionality that
would allow firing up a clean VM, do the build, close the VM & destroy
it, and run this on our own buildbot workers.

This requires both manpower to implement the functionality as well as
additional hardware resources to run the builds.
If we solve the first part, I guess we should be able to solve the
second one as well.
If you volunteer to do some research / work in this area ... that
would likely be the most significant step in "increasing your chances"
... probably up to 100% for the ports that have no chance in building
in < 50 minutes on a super slow hardware owned by Travis :)

Mojca

Re: Travis CI timeouts for MacPorts builds

[Joshua Root]

On 2020-9-7 18:45 , Ralph Seichter wrote:
> I fail to remember the last time one of my builds successfully passed
> Travis CI. All I see are timeouts [1]. Other peoples' jobs apparently
> make it through Travis OK, so I wander what I can do to increase my
> chances?
> 
> [1] https://travis-ci.org/github/macports/macports-ports/jobs/724689780

It's just a matter of how long your port takes to build (including
installing all its dependencies). Travis has a fixed time limit of 50
minutes. Azure has a longer limit but it's still too short for some ports.

- Josh