Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)
Hi, Thanks Ryan. My answer is very similar to Ben’s: * I’d be happy to provide you exclusive access to the resources (encrypted VMs, your own users, network and machine are UPS-protected, firewalled, etc.) * I completely agree with you about the safety concerns: those should not be relaxed. * I volunteered because I thought they were needed: I love MacPorts, and I want it to thrive. Bye, Enrico From: Ben Greenfield Date: Friday, 21 May 2021 at 13:26 To: Ryan Schmidt Cc: Andrew Janke , Enrico Maria Crisostomo , MacPorts Developers Subject: Re: Buildbot hardware (was: Re: Framing the MacPorts discussion) Hey All, Thanks for the direction Ryan. > On May 21, 2021, at 12:46 AM, Ryan Schmidt wrote: > > On May 19, 2021, at 12:38, Andrew Janke wrote: > >> I have a small stack of Mac Minis I got to use as a buildbot farm for >> Octave.app; I might be able to have them pull double duty for MacPorts >> depending on your change volume. > > > On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote: > >> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite >> often idle. >> I'm not privy with how our build system work, but if we could get to a point >> where agents can be added, stopped, throttled, trusted members of our >> community could volunteer the computational power they have at their >> disposal without fully dedicating a machine. >> In my specific case: I'm happy to offer VMs on that machine to volunteer >> computational resources. > > > On May 20, 2021, at 08:20, Ben Greenfield wrote: > >> I can definitely donate the facilities if not the talent. >> >> I have a symmetrical fiber connection and a static ip. I also have battery >> backup. >> I’m in the final weeks of making the building legal and I haven’t configured >> the final network set-up for the building. I was going to set-up a vlan on >> my hp procurve switch. >> I’m still shopping for a router to run OPNsense I think. >> >> I have been a mac sysadmin long time. > > > There seem to be a lot of people suddenly volunteering hardware for our build > system. First, thank you; I didn't know we had people interested in that. > > Our build system has never been designed to accommodate external hardware. It > has always been designed as a centralized system controlled by one > administrator. When it was first set up in 2011-12 it was under the control > of our Apple administrator at macOS forge. I became the macOS forge > administrator temporarily in late 2015, and MacPorts left macOS forge in late > 2016 as that service shut down, and I recreated the buildbot system on my own > hardware and have run it since then. > > We now have one external Apple Silicon build machine hosted at another data > center, but it's still under my exclusive control so that I can keep > everything working together. > I would be happy to provide the same service. I don’t need a log-in and I can probably provide out of band power reset. The system could be on it’s own vlan. > There are currently many situations where the build system gets into a state > that requires manual intervention. Because I control all the machines, I'm > able to make those fixes and get things back up and running quickly. > > We currently have all the builders we need: one for each OS version / arch > combination. The system was never designed to have more than that. If for > example we added a second macOS 11 / x86_64 builder, there could be confusion > and problems if the two machines have different OS / Xcode / command line > tools / java versions installed. > > There are security issues to consider. The binaries produced by our buildbot > workers are signed on the master with our private key. This is our "seal of > approval" that says we believe these binaries to be good and safe. Users > trust that. If we start allowing other people to run build machines, then we > have the problem that we do not know for certain whether those other build > machines are free of malware or other problems. We would be signing binaries > for distribution to users without being certain of their safety or > correctness. I'm not very comfortable with that. Yes, that safety should be maintained. > > Why is this discussion happening? Why do people think we need more hardware? > If we need more or faster CPUs or more memory, I can make those changes to > the hardware I already manage. I volunteered because it sounded like resources might be needed:). Let me know if the free-hosting is needed. Ben >
Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)
Hey All, Thanks for the direction Ryan. > On May 21, 2021, at 12:46 AM, Ryan Schmidt wrote: > > On May 19, 2021, at 12:38, Andrew Janke wrote: > >> I have a small stack of Mac Minis I got to use as a buildbot farm for >> Octave.app; I might be able to have them pull double duty for MacPorts >> depending on your change volume. > > > On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote: > >> I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite >> often idle. >> I'm not privy with how our build system work, but if we could get to a point >> where agents can be added, stopped, throttled, trusted members of our >> community could volunteer the computational power they have at their >> disposal without fully dedicating a machine. >> In my specific case: I'm happy to offer VMs on that machine to volunteer >> computational resources. > > > On May 20, 2021, at 08:20, Ben Greenfield wrote: > >> I can definitely donate the facilities if not the talent. >> >> I have a symmetrical fiber connection and a static ip. I also have battery >> backup. >> I’m in the final weeks of making the building legal and I haven’t configured >> the final network set-up for the building. I was going to set-up a vlan on >> my hp procurve switch. >> I’m still shopping for a router to run OPNsense I think. >> >> I have been a mac sysadmin long time. > > > There seem to be a lot of people suddenly volunteering hardware for our build > system. First, thank you; I didn't know we had people interested in that. > > Our build system has never been designed to accommodate external hardware. It > has always been designed as a centralized system controlled by one > administrator. When it was first set up in 2011-12 it was under the control > of our Apple administrator at macOS forge. I became the macOS forge > administrator temporarily in late 2015, and MacPorts left macOS forge in late > 2016 as that service shut down, and I recreated the buildbot system on my own > hardware and have run it since then. > > We now have one external Apple Silicon build machine hosted at another data > center, but it's still under my exclusive control so that I can keep > everything working together. > I would be happy to provide the same service. I don’t need a log-in and I can probably provide out of band power reset. The system could be on it’s own vlan. > There are currently many situations where the build system gets into a state > that requires manual intervention. Because I control all the machines, I'm > able to make those fixes and get things back up and running quickly. > > We currently have all the builders we need: one for each OS version / arch > combination. The system was never designed to have more than that. If for > example we added a second macOS 11 / x86_64 builder, there could be confusion > and problems if the two machines have different OS / Xcode / command line > tools / java versions installed. > > There are security issues to consider. The binaries produced by our buildbot > workers are signed on the master with our private key. This is our "seal of > approval" that says we believe these binaries to be good and safe. Users > trust that. If we start allowing other people to run build machines, then we > have the problem that we do not know for certain whether those other build > machines are free of malware or other problems. We would be signing binaries > for distribution to users without being certain of their safety or > correctness. I'm not very comfortable with that. Yes, that safety should be maintained. > > Why is this discussion happening? Why do people think we need more hardware? > If we need more or faster CPUs or more memory, I can make those changes to > the hardware I already manage. I volunteered because it sounded like resources might be needed:). Let me know if the free-hosting is needed. Ben >
Re: Buildbot hardware (was: Re: Framing the MacPorts discussion)
On May 19, 2021, at 12:38, Andrew Janke wrote: > I have a small stack of Mac Minis I got to use as a buildbot farm for > Octave.app; I might be able to have them pull double duty for MacPorts > depending on your change volume. On May 20, 2021, at 08:10, Enrico Maria Crisostomo wrote: > I've got an iMac Pro in my LAN with 16 vCores and 64GB or RAM which is quite > often idle. > I'm not privy with how our build system work, but if we could get to a point > where agents can be added, stopped, throttled, trusted members of our > community could volunteer the computational power they have at their disposal > without fully dedicating a machine. > In my specific case: I'm happy to offer VMs on that machine to volunteer > computational resources. On May 20, 2021, at 08:20, Ben Greenfield wrote: > I can definitely donate the facilities if not the talent. > > I have a symmetrical fiber connection and a static ip. I also have battery > backup. > I’m in the final weeks of making the building legal and I haven’t configured > the final network set-up for the building. I was going to set-up a vlan on my > hp procurve switch. > I’m still shopping for a router to run OPNsense I think. > > I have been a mac sysadmin long time. There seem to be a lot of people suddenly volunteering hardware for our build system. First, thank you; I didn't know we had people interested in that. Our build system has never been designed to accommodate external hardware. It has always been designed as a centralized system controlled by one administrator. When it was first set up in 2011-12 it was under the control of our Apple administrator at macOS forge. I became the macOS forge administrator temporarily in late 2015, and MacPorts left macOS forge in late 2016 as that service shut down, and I recreated the buildbot system on my own hardware and have run it since then. We now have one external Apple Silicon build machine hosted at another data center, but it's still under my exclusive control so that I can keep everything working together. There are currently many situations where the build system gets into a state that requires manual intervention. Because I control all the machines, I'm able to make those fixes and get things back up and running quickly. We currently have all the builders we need: one for each OS version / arch combination. The system was never designed to have more than that. If for example we added a second macOS 11 / x86_64 builder, there could be confusion and problems if the two machines have different OS / Xcode / command line tools / java versions installed. There are security issues to consider. The binaries produced by our buildbot workers are signed on the master with our private key. This is our "seal of approval" that says we believe these binaries to be good and safe. Users trust that. If we start allowing other people to run build machines, then we have the problem that we do not know for certain whether those other build machines are free of malware or other problems. We would be signing binaries for distribution to users without being certain of their safety or correctness. I'm not very comfortable with that. Why is this discussion happening? Why do people think we need more hardware? If we need more or faster CPUs or more memory, I can make those changes to the hardware I already manage.
