Over the past month or so there has been a few threads on the direction of the project, what can be improved and what the vision should be. I wanted to summarize my two cents because I haven't been otherwise participating and hopefully this can adjust the community mindset towards what the problems are and frame their solutions.
TL; DR - We're all package maintainers so MacPorts is structurally going to be short staffed on CI (builder) work because CI is a different set of skills and interests. Implement a structural solution like positions/delegation to volunteers to explicitly work on CI - Cloud hardware too expensive to consider - In lieu of cloud, design CI that can be run on trusted volunteer's home/office donated machines - Consider Linux/QEMU for builder virtualization As a user of MacPorts I greatly value having binaries available. It would be incredible if the only packages I build are the ones that I'm trying to maintain. There have been a few threads where users thought binary builds weren't available or were rarely available so it seems like it's in high demand. It is worth it to prioritize CI changes that allow for more binaries to be available. MacPorts has always been short on labor for the CI system. We're still using the roots of a buildbot setup that was originally built on Apple's dime, if my history is correct. I don't think this should surprise any of us, though. We're all here, involved with MacPorts and port maintenance, because we're interested in the packaging side of things. We're not people with the time or inclination to work on CI tasks. That's not a bad thing, but it's something that the organizational structure should account for in order to keep stuff maintained. Because this is a structural problem I think the mindset of the MacPorts community is that a fix to the issue is going to involve structural changes. I think the way out of this is to recruit volunteers explicitly to work on CI tasks. Trying to get package maintainers to cover CI work is, and has been, getting blood from a stone. On the MacPorts administration side there needs to be a bit of work put into figuring out what the formal relationship and expectations are between CI volunteers and the MacPorts leadership. There are lots of ways forward there and lots of good ways to structure it. There is going to have to be some delegation and the relationship needs to handle that. For the CI work that does initial builds of PRs, I unfortunately think the only technical solution here is GitHub Actions and third party services like it. There's a plague of cryptocurrency miners looking to exploit these systems and nothing a MacPorts volunteer team can build will win in that game of cat and mouse. So for the future of PR CI I think the conversation needs to be around using and maintaining those third party solutions. There's a thread on here about raising money to pay for CI (for builders). I'd like to remind everyone that cloud bills are damn expensive and I don't think MacPorts has the dedicated user base to donate to keep that CI alive. For the vast majority of humans on the planet a monthly donation of $20-$40 USD is a massive expenditure, and I would be surprised if we could get a reliable $500 a month out of everyone for MacPorts. That $500 a month can get you some cloud computing but not a ton, and likely not what we need for reliable CI. It's more likely than not we'd have trouble getting donations to keep the lights on, it would be a constant struggle to fund raise, and the whole issue around accepting donations and incorporation and that stuff. I encourage everyone involved to write off the idea of doing cloud CI for MacPorts builders. Not being able to use cloud technology also hurts the ability to recruit CI volunteers, so this tradeoff needs to be made explicit up front. But hopefully some people are interested in a challenge. So if a recurring cash donation is a struggle for individual donors I think the way to get hardware for CI is a bit of a psychological hack: ask people to buy/donate build machines and run them out of their own offices or houses. My gut is that individuals are willing to spend more (or at all) if it's a one time thing and it's tied to a concrete, physical item. There's also probably enough used and aged hardware amongst the MacPorts community that could be donated to make up a good enough build farm. The operators of the machines are also likely comfortable donating the electricity cost as it'll just be lumped in with the rest of their bill. And for new machines an achievable Kickstarter of $1000-$2000 could probably get enough donors and would result in a fine build server. This has a few implementation challenges. The CI system will have to be built to work with a build farm of machines of varying reliability and speed. Only trusted members of the community would be allowed to host. The CI design would also need to have network isolation between the host's personal/corporate network and the builder, and they'd have to be OK with giving SSH credentials to the CI team. All of this CI design has Linux in mind. If it is acceptable to the MacPorts team, I strongly encourage running builders under QEMU virtualization. Linux will let us use commodity hardware and QEMU can reliably run MacOS X versions from a variety of releases, including PPC systems, and with hardware that is way faster than any PPC machine Apple ever shipped. If QEMU can reliably pause, snapshot and resume running builder VMs, and the machines have a SSD from the past 5 years or so, we're looking at a pretty speedy build setup. I'm aware that there are certain things about running MacOS X on QEMU that may make it an unacceptable choice for MacPorts. Personally, my risk tolerance there is fine with it, I think Apple's poem is a nod and a wink to use cases like ours so you can get a solution that works in a reasonable budget and with reasonable hardware performance. But I don't know the history of this discussion and maybe it is a non-starter for MacPorts. -- David Gilman :DG<
