Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)

[Enrico Maria Crisostomo]

Hi,

Thanks Ryan.

My answer is very similar to Ben’s:

  *   I’d be happy to provide you exclusive access to the resources (encrypted 
VMs, your own users, network and machine are UPS-protected, firewalled, etc.)
  *   I completely agree with you about the safety concerns: those should not 
be relaxed.
  *   I volunteered because I thought they were needed: I love MacPorts, and I 
want it to thrive.

Bye,
Enrico


From: Ben Greenfield 
Date: Friday, 21 May 2021 at 13:26
To: Ryan Schmidt 
Cc: Andrew Janke , Enrico Maria Crisostomo 
, MacPorts Developers 

Subject: Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)
Hey All,

Thanks for the direction Ryan.

> On May 21, 2021, at 12:46 AM, Ryan Schmidt  wrote:
>
> On May 19, 2021, at 12:38, Andrew Janke wrote:
>
>> I have a small stack of Mac Minis I got to use as a buildbot farm for 
>> Octave.app; I might be able to have them pull double duty for MacPorts 
>> depending on your change volume.
>
>
> On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote:
>
>> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite 
>> often idle.
>> I'm not privy with how our build system work, but if we could get to a point 
>> where agents can be added, stopped, throttled, trusted members of our 
>> community could volunteer the computational power they have at their 
>> disposal without fully dedicating a machine.
>> In my specific case: I'm happy to offer VMs on that machine to volunteer 
>> computational resources.
>
>
> On May 20, 2021, at 08:20, Ben Greenfield wrote:
>
>> I can definitely donate the facilities if not the talent.
>>
>> I have a symmetrical fiber connection and a static ip. I also have battery 
>> backup.
>> I’m in the final weeks of making the building legal and I haven’t configured 
>> the final network set-up for the building. I was going to set-up a vlan on 
>> my hp procurve switch.
>> I’m still shopping for a router to run OPNsense I think.
>>
>> I have been a mac sysadmin long time.
>
>
> There seem to be a lot of people suddenly volunteering hardware for our build 
> system. First, thank you; I didn't know we had people interested in that.
>
> Our build system has never been designed to accommodate external hardware. It 
> has always been designed as a centralized system controlled by one 
> administrator. When it was first set up in 2011-12 it was under the control 
> of our Apple administrator at macOS forge. I became the macOS forge 
> administrator temporarily in late 2015, and MacPorts left macOS forge in late 
> 2016 as that service shut down, and I recreated the buildbot system on my own 
> hardware and have run it since then.
>
> We now have one external Apple Silicon build machine hosted at another data 
> center, but it's still under my exclusive control so that I can keep 
> everything working together.
>

I would be happy to provide the same service. I don’t need a log-in and I can 
probably provide out of band power reset. The system could be on it’s own vlan.


> There are currently many situations where the build system gets into a state 
> that requires manual intervention. Because I control all the machines, I'm 
> able to make those fixes and get things back up and running quickly.
>
> We currently have all the builders we need: one for each OS version / arch 
> combination. The system was never designed to have more than that. If for 
> example we added a second macOS 11 / x86_64 builder, there could be confusion 
> and problems if the two machines have different OS / Xcode / command line 
> tools / java versions installed.
>
> There are security issues to consider. The binaries produced by our buildbot 
> workers are signed on the master with our private key. This is our "seal of 
> approval" that says we believe these binaries to be good and safe. Users 
> trust that. If we start allowing other people to run build machines, then we 
> have the problem that we do not know for certain whether those other build 
> machines are free of malware or other problems. We would be signing binaries 
> for distribution to users without being certain of their safety or 
> correctness. I'm not very comfortable with that.

Yes, that safety should be maintained.

>
> Why is this discussion happening? Why do people think we need more hardware? 
> If we need more or faster CPUs or more memory, I can make those changes to 
> the hardware I already manage.

I volunteered because it sounded like resources might be needed:).

Let me know if the free-hosting is needed.

Ben

>

Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)

[Ben Greenfield via macports-dev]

Hey All,

Thanks for the direction Ryan.

> On May 21, 2021, at 12:46 AM, Ryan Schmidt  wrote:
> 
> On May 19, 2021, at 12:38, Andrew Janke wrote:
> 
>> I have a small stack of Mac Minis I got to use as a buildbot farm for 
>> Octave.app; I might be able to have them pull double duty for MacPorts 
>> depending on your change volume.
> 
> 
> On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote:
> 
>> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite 
>> often idle.
>> I'm not privy with how our build system work, but if we could get to a point 
>> where agents can be added, stopped, throttled, trusted members of our 
>> community could volunteer the computational power they have at their 
>> disposal without fully dedicating a machine.
>> In my specific case: I'm happy to offer VMs on that machine to volunteer 
>> computational resources.
> 
> 
> On May 20, 2021, at 08:20, Ben Greenfield wrote:
> 
>> I can definitely donate the facilities if not the talent.
>> 
>> I have a symmetrical fiber connection and a static ip. I also have battery 
>> backup.
>> I’m in the final weeks of making the building legal and I haven’t configured 
>> the final network set-up for the building. I was going to set-up a vlan on 
>> my hp procurve switch.
>> I’m still shopping for a router to run OPNsense I think.
>> 
>> I have been a mac sysadmin long time.
> 
> 
> There seem to be a lot of people suddenly volunteering hardware for our build 
> system. First, thank you; I didn't know we had people interested in that.
> 
> Our build system has never been designed to accommodate external hardware. It 
> has always been designed as a centralized system controlled by one 
> administrator. When it was first set up in 2011-12 it was under the control 
> of our Apple administrator at macOS forge. I became the macOS forge 
> administrator temporarily in late 2015, and MacPorts left macOS forge in late 
> 2016 as that service shut down, and I recreated the buildbot system on my own 
> hardware and have run it since then.
> 
> We now have one external Apple Silicon build machine hosted at another data 
> center, but it's still under my exclusive control so that I can keep 
> everything working together.
> 

I would be happy to provide the same service. I don’t need a log-in and I can 
probably provide out of band power reset. The system could be on it’s own vlan.


> There are currently many situations where the build system gets into a state 
> that requires manual intervention. Because I control all the machines, I'm 
> able to make those fixes and get things back up and running quickly.
> 
> We currently have all the builders we need: one for each OS version / arch 
> combination. The system was never designed to have more than that. If for 
> example we added a second macOS 11 / x86_64 builder, there could be confusion 
> and problems if the two machines have different OS / Xcode / command line 
> tools / java versions installed.
> 
> There are security issues to consider. The binaries produced by our buildbot 
> workers are signed on the master with our private key. This is our "seal of 
> approval" that says we believe these binaries to be good and safe. Users 
> trust that. If we start allowing other people to run build machines, then we 
> have the problem that we do not know for certain whether those other build 
> machines are free of malware or other problems. We would be signing binaries 
> for distribution to users without being certain of their safety or 
> correctness. I'm not very comfortable with that.

Yes, that safety should be maintained.

> 
> Why is this discussion happening? Why do people think we need more hardware? 
> If we need more or faster CPUs or more memory, I can make those changes to 
> the hardware I already manage.

I volunteered because it sounded like resources might be needed:).

Let me know if the free-hosting is needed.

Ben

> 

Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)

[Ryan Schmidt]

On May 19, 2021, at 12:38, Andrew Janke wrote:

> I have a small stack of Mac Minis I got to use as a buildbot farm for 
> Octave.app; I might be able to have them pull double duty for MacPorts 
> depending on your change volume.


On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote:

> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite 
> often idle.
> I'm not privy with how our build system work, but if we could get to a point 
> where agents can be added, stopped, throttled, trusted members of our 
> community could volunteer the computational power they have at their disposal 
> without fully dedicating a machine.
> In my specific case: I'm happy to offer VMs on that machine to volunteer 
> computational resources.


On May 20, 2021, at 08:20, Ben Greenfield wrote:

> I can definitely donate the facilities if not the talent.
> 
> I have a symmetrical fiber connection and a static ip. I also have battery 
> backup.
> I’m in the final weeks of making the building legal and I haven’t configured 
> the final network set-up for the building. I was going to set-up a vlan on my 
> hp procurve switch.
> I’m still shopping for a router to run OPNsense I think.
> 
>  I have been a mac sysadmin long time.


There seem to be a lot of people suddenly volunteering hardware for our build 
system. First, thank you; I didn't know we had people interested in that.

Our build system has never been designed to accommodate external hardware. It 
has always been designed as a centralized system controlled by one 
administrator. When it was first set up in 2011-12 it was under the control of 
our Apple administrator at macOS forge. I became the macOS forge administrator 
temporarily in late 2015, and MacPorts left macOS forge in late 2016 as that 
service shut down, and I recreated the buildbot system on my own hardware and 
have run it since then.

We now have one external Apple Silicon build machine hosted at another data 
center, but it's still under my exclusive control so that I can keep everything 
working together.

There are currently many situations where the build system gets into a state 
that requires manual intervention. Because I control all the machines, I'm able 
to make those fixes and get things back up and running quickly.

We currently have all the builders we need: one for each OS version / arch 
combination. The system was never designed to have more than that. If for 
example we added a second macOS 11 / x86_64 builder, there could be confusion 
and problems if the two machines have different OS / Xcode / command line tools 
/ java versions installed.

There are security issues to consider. The binaries produced by our buildbot 
workers are signed on the master with our private key. This is our "seal of 
approval" that says we believe these binaries to be good and safe. Users trust 
that. If we start allowing other people to run build machines, then we have the 
problem that we do not know for certain whether those other build machines are 
free of malware or other problems. We would be signing binaries for 
distribution to users without being certain of their safety or correctness. I'm 
not very comfortable with that.

Why is this discussion happening? Why do people think we need more hardware? If 
we need more or faster CPUs or more memory, I can make those changes to the 
hardware I already manage.

Re: Framing the MacPorts discussion

[Ryan Schmidt]

On May 19, 2021, at 10:01, David Gilman wrote:

> but maybe the right mindset here is to do the best with what we've got.

We do, and have, for over ten years.

Re: Framing the MacPorts discussion

[Ryan Schmidt]

On May 19, 2021, at 07:56, Bjarne D Mathiesen wrote:

> Ryan Schmidt wrote:
>> I personally am not comfortable adding other build machines to our buildbot 
>> system that I do not control. When I control the machines, I know what is 
>> installed on them and that they are set up correctly. Having build machines 
>> located outside of my local network also poses additional challenges, as 
>> I've learned by having our Apple Silicon build machine outside of my 
>> network, challenges which I would prefer to minimize, not increase.
>> 
>> We currently use one build machine per OS version / arch, and have the 
>> hardware needed to do that. Adding more hardware such that we have more than 
>> one build machine per OS version / arch is not something our buildbot system 
>> was ever designed to accommodate, and would introduce problems.
> 
> So we are more-or-less single-point-of-failure regarding the buildbots
> !?

The buildbot system has always been a single point of failure, as has almost 
every other aspect of MacPorts infrastructure, with the exception of our 
mirrors. We have one mailing list server. One main repository. One main web 
server.

Buildbot is not critical infrastructure, in that it is nice to have but the 
world does not end if it is down for a short period of time. People can still 
get any packages that were built before from our distributed network of mirror 
servers, and if a binary is not available for whatever the user wants to 
install, then it can be built from source on their system.


> Personally, I'ld deem it advisable to have at least 3 complete
> systems geographically dispersed.

That seems completely infeasible to me.

As far as the buildmaster goes, Buildbot is designed to have just one. There is 
a more complicated optional multiple-master mode available in the latest 
Buildbot, but AFAIK it's intended for each master to handle separate tasks. For 
example, one master handles the web interface while a second master handles 
scheduling. I intend to use this mode in the new buildbot setup.

As for the workers, our system is designed with the assumption that there is 
one worker for each OS version / arch. We already have this with our existing 
hardware. Adding more hardware running additional copies of the workers we 
already have will introduce complications. If we want to do that, we would have 
to modify mpbb and the buildbot configuration to accommodate those 
complications. We would have to make the system much more robust against 
failures than it currently is. Right now, it works because one person (me) is 
able to intervene to fix problems that arise.

There are also security considerations in allowing additional individuals to 
run the build machines that produce our officially signed binaries.


>> 
>> Using Linux and commodity hardware is not applicable because it the macOS 
>> EULA only permits running macOS on Apple hardware, as we currently do.
> 
> So it's not the proposed software stack that's the problem, but the
> hardware stack. Then how about an old 2010/12 MacPro with all the bells
> and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
> Ubuntu without any problems.

We already have four 2009 Xserves. Their CPU and memory could be upgraded if 
needed.

One of these is offline at the moment due to hardware failure. It can be 
repaired. If necessary, I'll repair it.

Re: Framing the MacPorts discussion

[Ben Greenfield via macports-dev]

I can definitely donate the facilities if not the talent.

I have a symmetrical fiber connection and a static ip. I also have battery 
backup.
I’m in the final weeks of making the building legal and I haven’t configured 
the final network set-up for the building. I was going to set-up a vlan on my 
hp procurve switch.
I’m still shopping for a router to run OPNsense I think.

 I have been a mac sysadmin long time.

happy to accommodate.


Ben


> On May 19, 2021, at 11:01 AM, David Gilman  wrote:
> 
> Three distributed machines is a great goal, I agree with you, but maybe the 
> right mindset here is to do the best with what we've got. If we have three or 
> more trusted volunteers and the CI can support it let's do it, if we have 
> fewer, we'll live with it.
> 
> On Wed, May 19, 2021, 8:57 AM Bjarne D Mathiesen  > wrote:
> 
> 
> Ryan Schmidt wrote:
> > I personally am not comfortable adding other build machines to our buildbot 
> > system that I do not control. When I control the machines, I know what is 
> > installed on them and that they are set up correctly. Having build machines 
> > located outside of my local network also poses additional challenges, as 
> > I've learned by having our Apple Silicon build machine outside of my 
> > network, challenges which I would prefer to minimize, not increase.
> > 
> > We currently use one build machine per OS version / arch, and have the 
> > hardware needed to do that. Adding more hardware such that we have more 
> > than one build machine per OS version / arch is not something our buildbot 
> > system was ever designed to accommodate, and would introduce problems.
> 
> So we are more-or-less single-point-of-failure regarding the buildbots
> !? Personally, I'ld deem it advisable to have at least 3 complete
> systems geographically dispersed.
> 
> > 
> > Using Linux and commodity hardware is not applicable because it the macOS 
> > EULA only permits running macOS on Apple hardware, as we currently do.
> 
> So it's not the proposed software stack that's the problem, but the
> hardware stack. Then how about an old 2010/12 MacPro with all the bells
> and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
> Ubuntu without any problems.
> 
> -- 
> Bjarne D Mathiesen
> Korsør ; Danmark ; Europa
> ---
> denne besked er skrevet i et totalt M$-frit miljø
> OpenCore + macOS 10.15.7 Catalina
> MacPro 2010 ; 2 x 3,46 GHz 6-Core Intel Xeon ; 256 GB 1333 MHz DDR3 ECC
> ATI Radeon RX 590 8 GB

Re: Framing the MacPorts discussion

[Enrico Maria Crisostomo]

Hi all,

I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is
quite often idle.
I'm not privy with how our build system work, but if we could get to a
point where agents can be added, stopped, throttled, trusted members of our
community could volunteer the computational power they have at their
disposal without fully dedicating a machine.
In my specific case: I'm happy to offer VMs on that machine to volunteer
computational resources.

On Wed, May 19, 2021 at 7:38 PM Andrew Janke  wrote:

> I have a small stack of Mac Minis I got to use as a buildbot farm for
> Octave.app; I might be able to have them pull double duty for MacPorts
> depending on your change volume.
>
> Cheers,
> Andrew
>
> On 5/19/21 1:16 PM, Mark Anderson wrote:
>
> Yeah - we are certainly short staffed everywhere - I try to add more and
> more of my time to the project but aside from my ports, I'm still in
> learning mode digging through all the asciidoc and tcl and everything. I'm
> trying to build some tools to help me, but again, more time.
>
> Once we move to the latest build bot, we might want to see if we can get
> other volunteers to host some hardware, but the problem is, we're going to
> have to hit up ebay or something and the infrastructure will be tougher.
>
> I'm honestly not sure how we can manage to staff up more at all - I mean
> this is a FOSS problem all over for non-company sponsored projects.
>
> —Mark
> ___
> Mark E. Anderson 
> MacPorts Trac WikiPage 
> GitHub Profile 
>
>
>
> On Tue, May 18, 2021 at 2:35 AM Ryan Schmidt 
> wrote:
>
>> MacPorts is short-staffed in all areas, not just infrastructure.
>>
>> Our Buildbot system works. It produces all the binaries we are able to.
>>
>> Our buildbot system was already substantially redesigned when we took it
>> over from Apple in 2016 and will be substantially redesigned again as we
>> upgrade to the latest version of the buildbot software.
>>
>> We already have a small infrastructure team who are interested in working
>> on improving the buildbot system and our other infrastructure, and do so.
>>
>> We already use GitHub Actions for CI. We cannot use it to replace
>> buildbot because it only offers recent OS versions and because it does not
>> offer persistent machines.
>>
>> I personally am not comfortable adding other build machines to our
>> buildbot system that I do not control. When I control the machines, I know
>> what is installed on them and that they are set up correctly. Having build
>> machines located outside of my local network also poses additional
>> challenges, as I've learned by having our Apple Silicon build machine
>> outside of my network, challenges which I would prefer to minimize, not
>> increase.
>>
>> We currently use one build machine per OS version / arch, and have the
>> hardware needed to do that. Adding more hardware such that we have more
>> than one build machine per OS version / arch is not something our buildbot
>> system was ever designed to accommodate, and would introduce problems.
>>
>> Using Linux and commodity hardware is not applicable because it the macOS
>> EULA only permits running macOS on Apple hardware, as we currently do.
>
>
>

Re: Framing the MacPorts discussion

[Andrew Janke]

I have a small stack of Mac Minis I got to use as a buildbot farm for 
Octave.app; I might be able to have them pull double duty for MacPorts 
depending on your change volume.


Cheers,
Andrew

On 5/19/21 1:16 PM, Mark Anderson wrote:
Yeah - we are certainly short staffed everywhere - I try to add more 
and more of my time to the project but aside from my ports, I'm still 
in learning mode digging through all the asciidoc and tcl and 
everything. I'm trying to build some tools to help me, but again, more 
time.


Once we move to the latest build bot, we might want to see if we can 
get other volunteers to host some hardware, but the problem is, we're 
going to have to hit up ebay or something and the infrastructure will 
be tougher.


I'm honestly not sure how we can manage to staff up more at all - I 
mean this is a FOSS problem all over for non-company sponsored projects.


—Mark
___
Mark E. Anderson mailto:m...@macports.org>>
MacPorts Trac WikiPage 
GitHub Profile 



On Tue, May 18, 2021 at 2:35 AM Ryan Schmidt > wrote:


MacPorts is short-staffed in all areas, not just infrastructure.

Our Buildbot system works. It produces all the binaries we are
able to.

Our buildbot system was already substantially redesigned when we
took it over from Apple in 2016 and will be substantially
redesigned again as we upgrade to the latest version of the
buildbot software.

We already have a small infrastructure team who are interested in
working on improving the buildbot system and our other
infrastructure, and do so.

We already use GitHub Actions for CI. We cannot use it to replace
buildbot because it only offers recent OS versions and because it
does not offer persistent machines.

I personally am not comfortable adding other build machines to our
buildbot system that I do not control. When I control the
machines, I know what is installed on them and that they are set
up correctly. Having build machines located outside of my local
network also poses additional challenges, as I've learned by
having our Apple Silicon build machine outside of my network,
challenges which I would prefer to minimize, not increase.

We currently use one build machine per OS version / arch, and have
the hardware needed to do that. Adding more hardware such that we
have more than one build machine per OS version / arch is not
something our buildbot system was ever designed to accommodate,
and would introduce problems.

Using Linux and commodity hardware is not applicable because it
the macOS EULA only permits running macOS on Apple hardware, as we
currently do.

Re: Framing the MacPorts discussion

[Mark Anderson]

Yeah - we are certainly short staffed everywhere - I try to add more and
more of my time to the project but aside from my ports, I'm still in
learning mode digging through all the asciidoc and tcl and everything. I'm
trying to build some tools to help me, but again, more time.

Once we move to the latest build bot, we might want to see if we can get
other volunteers to host some hardware, but the problem is, we're going to
have to hit up ebay or something and the infrastructure will be tougher.

I'm honestly not sure how we can manage to staff up more at all - I mean
this is a FOSS problem all over for non-company sponsored projects.

—Mark
___
Mark E. Anderson 
MacPorts Trac WikiPage 
GitHub Profile 



On Tue, May 18, 2021 at 2:35 AM Ryan Schmidt 
wrote:

> MacPorts is short-staffed in all areas, not just infrastructure.
>
> Our Buildbot system works. It produces all the binaries we are able to.
>
> Our buildbot system was already substantially redesigned when we took it
> over from Apple in 2016 and will be substantially redesigned again as we
> upgrade to the latest version of the buildbot software.
>
> We already have a small infrastructure team who are interested in working
> on improving the buildbot system and our other infrastructure, and do so.
>
> We already use GitHub Actions for CI. We cannot use it to replace buildbot
> because it only offers recent OS versions and because it does not offer
> persistent machines.
>
> I personally am not comfortable adding other build machines to our
> buildbot system that I do not control. When I control the machines, I know
> what is installed on them and that they are set up correctly. Having build
> machines located outside of my local network also poses additional
> challenges, as I've learned by having our Apple Silicon build machine
> outside of my network, challenges which I would prefer to minimize, not
> increase.
>
> We currently use one build machine per OS version / arch, and have the
> hardware needed to do that. Adding more hardware such that we have more
> than one build machine per OS version / arch is not something our buildbot
> system was ever designed to accommodate, and would introduce problems.
>
> Using Linux and commodity hardware is not applicable because it the macOS
> EULA only permits running macOS on Apple hardware, as we currently do.

Re: Framing the MacPorts discussion

[David Gilman]

Three distributed machines is a great goal, I agree with you, but maybe the
right mindset here is to do the best with what we've got. If we have three
or more trusted volunteers and the CI can support it let's do it, if we
have fewer, we'll live with it.

On Wed, May 19, 2021, 8:57 AM Bjarne D Mathiesen 
wrote:

>
>
> Ryan Schmidt wrote:
> > I personally am not comfortable adding other build machines to our
> buildbot system that I do not control. When I control the machines, I know
> what is installed on them and that they are set up correctly. Having build
> machines located outside of my local network also poses additional
> challenges, as I've learned by having our Apple Silicon build machine
> outside of my network, challenges which I would prefer to minimize, not
> increase.
> >
> > We currently use one build machine per OS version / arch, and have the
> hardware needed to do that. Adding more hardware such that we have more
> than one build machine per OS version / arch is not something our buildbot
> system was ever designed to accommodate, and would introduce problems.
>
> So we are more-or-less single-point-of-failure regarding the buildbots
> !? Personally, I'ld deem it advisable to have at least 3 complete
> systems geographically dispersed.
>
> >
> > Using Linux and commodity hardware is not applicable because it the
> macOS EULA only permits running macOS on Apple hardware, as we currently do.
>
> So it's not the proposed software stack that's the problem, but the
> hardware stack. Then how about an old 2010/12 MacPro with all the bells
> and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
> Ubuntu without any problems.
>
> --
> Bjarne D Mathiesen
> Korsør ; Danmark ; Europa
> ---
> denne besked er skrevet i et totalt M$-frit miljø
> OpenCore + macOS 10.15.7 Catalina
> MacPro 2010 ; 2 x 3,46 GHz 6-Core Intel Xeon ; 256 GB 1333 MHz DDR3 ECC
> ATI Radeon RX 590 8 GB
>

Re: Framing the MacPorts discussion

[Bjarne D Mathiesen]

Ryan Schmidt wrote:
> I personally am not comfortable adding other build machines to our buildbot 
> system that I do not control. When I control the machines, I know what is 
> installed on them and that they are set up correctly. Having build machines 
> located outside of my local network also poses additional challenges, as I've 
> learned by having our Apple Silicon build machine outside of my network, 
> challenges which I would prefer to minimize, not increase.
> 
> We currently use one build machine per OS version / arch, and have the 
> hardware needed to do that. Adding more hardware such that we have more than 
> one build machine per OS version / arch is not something our buildbot system 
> was ever designed to accommodate, and would introduce problems.

So we are more-or-less single-point-of-failure regarding the buildbots
!? Personally, I'ld deem it advisable to have at least 3 complete
systems geographically dispersed.

> 
> Using Linux and commodity hardware is not applicable because it the macOS 
> EULA only permits running macOS on Apple hardware, as we currently do.

So it's not the proposed software stack that's the problem, but the
hardware stack. Then how about an old 2010/12 MacPro with all the bells
and whistles ? I've been able to boot a 2010 MacPro w/ 256GB RAM under
Ubuntu without any problems.

-- 
Bjarne D Mathiesen
Korsør ; Danmark ; Europa
---
denne besked er skrevet i et totalt M$-frit miljø
OpenCore + macOS 10.15.7 Catalina
MacPro 2010 ; 2 x 3,46 GHz 6-Core Intel Xeon ; 256 GB 1333 MHz DDR3 ECC
ATI Radeon RX 590 8 GB

Re: Framing the MacPorts discussion

[Ryan Schmidt]

MacPorts is short-staffed in all areas, not just infrastructure.

Our Buildbot system works. It produces all the binaries we are able to.

Our buildbot system was already substantially redesigned when we took it over 
from Apple in 2016 and will be substantially redesigned again as we upgrade to 
the latest version of the buildbot software.

We already have a small infrastructure team who are interested in working on 
improving the buildbot system and our other infrastructure, and do so.

We already use GitHub Actions for CI. We cannot use it to replace buildbot 
because it only offers recent OS versions and because it does not offer 
persistent machines.

I personally am not comfortable adding other build machines to our buildbot 
system that I do not control. When I control the machines, I know what is 
installed on them and that they are set up correctly. Having build machines 
located outside of my local network also poses additional challenges, as I've 
learned by having our Apple Silicon build machine outside of my network, 
challenges which I would prefer to minimize, not increase.

We currently use one build machine per OS version / arch, and have the hardware 
needed to do that. Adding more hardware such that we have more than one build 
machine per OS version / arch is not something our buildbot system was ever 
designed to accommodate, and would introduce problems.

Using Linux and commodity hardware is not applicable because it the macOS EULA 
only permits running macOS on Apple hardware, as we currently do.

Framing the MacPorts discussion

[David Gilman]

Over the past month or so there has been a few threads on the
direction of the project, what can be improved and what the vision
should be. I wanted to summarize my two cents because I haven't been
otherwise participating and hopefully this can adjust the community
mindset towards what the problems are and frame their solutions.

TL; DR
- We're all package maintainers so MacPorts is structurally going to
be short staffed on CI (builder) work because CI is a different set of
skills and interests. Implement a structural solution like
positions/delegation to volunteers to explicitly work on CI
- Cloud hardware too expensive to consider
- In lieu of cloud, design CI that can be run on trusted volunteer's
home/office donated machines
- Consider Linux/QEMU for builder virtualization

As a user of MacPorts I greatly value having binaries available. It
would be incredible if the only packages I build are the ones that I'm
trying to maintain. There have been a few threads where users thought
binary builds weren't available or were rarely available so it seems
like it's in high demand. It is worth it to prioritize CI changes that
allow for more binaries to be available.

MacPorts has always been short on labor for the CI system. We're still
using the roots of a buildbot setup that was originally built on
Apple's dime, if my history is correct. I don't think this should
surprise any of us, though. We're all here, involved with MacPorts and
port maintenance, because we're interested in the packaging side of
things. We're not people with the time or inclination to work on CI
tasks. That's not a bad thing, but it's something that the
organizational structure should account for in order to keep stuff
maintained. Because this is a structural problem I think the mindset
of the MacPorts community is that a fix to the issue is going to
involve structural changes.

I think the way out of this is to recruit volunteers explicitly to
work on CI tasks. Trying to get package maintainers to cover CI work
is, and has been, getting blood from a stone. On the MacPorts
administration side there needs to be a bit of work put into figuring
out what the formal relationship and expectations are between CI
volunteers and the MacPorts leadership. There are lots of ways forward
there and lots of good ways to structure it. There is going to have to
be some delegation and the relationship needs to handle that.

For the CI work that does initial builds of PRs, I unfortunately think
the only technical solution here is GitHub Actions and third party
services like it. There's a plague of cryptocurrency miners looking to
exploit these systems and nothing a MacPorts volunteer team can build
will win in that game of cat and mouse. So for the future of PR CI I
think the conversation needs to be around using and maintaining those
third party solutions.

There's a thread on here about raising money to pay for CI (for
builders). I'd like to remind everyone that cloud bills are damn
expensive and I don't think MacPorts has the dedicated user base to
donate to keep that CI alive. For the vast majority of humans on the
planet a monthly donation of $20-$40 USD is a massive expenditure, and
I would be surprised if we could get a reliable $500 a month out of
everyone for MacPorts. That $500 a month can get you some cloud
computing but not a ton, and likely not what we need for reliable CI.
It's more likely than not we'd have trouble getting donations to keep
the lights on, it would be a constant struggle to fund raise, and the
whole issue around accepting donations and incorporation and that
stuff. I encourage everyone involved to write off the idea of doing
cloud CI for MacPorts builders.

Not being able to use cloud technology also hurts the ability to
recruit CI volunteers, so this tradeoff needs to be made explicit up
front. But hopefully some people are interested in a challenge.

So if a recurring cash donation is a struggle for individual donors I
think the way to get hardware for CI is a bit of a psychological hack:
ask people to buy/donate build machines and run them out of their own
offices or houses. My gut is that individuals are willing to spend
more (or at all) if it's a one time thing and it's tied to a concrete,
physical item. There's also probably enough used and aged hardware
amongst the MacPorts community that could be donated to make up a good
enough build farm. The operators of the machines are also likely
comfortable donating the electricity cost as it'll just be lumped in
with the rest of their bill. And for new machines an achievable
Kickstarter of $1000-$2000 could probably get enough donors and would
result in a fine build server.

This has a few implementation challenges. The CI system will have to
be built to work with a build farm of machines of varying reliability
and speed. Only trusted members of the community would be allowed to
host. The CI design would also need to have network isolation between
the host's personal/corporate