In order to catch up a bit since I wasn't subscribed to the mailing list with this email at the time I found this thread. If anything sounds odd, read through to the end. I'm trying to top reply so I'm leaving my 'backstory' till the end.
> Rich Freeman <rich0 <at> gentoo.org> writes: > > > James <wireless <at> tampabay.rr.com> writes: > > > > I bet our friends at RackSpace will provide all the virtual HorsePower > > you need, should google not provide the hundreds/thousands or cores for > > you to run on. > > > My guess is that the hardware to run all this on is the simplest part > of the problem. If somebody writes something good and we build the > processes to support it, then I'm sure we can find donors to provide > the CPU cycles. ChromeOS could probably steal whatever we put > together so there is an obvious synergy there, and I'm sure > Amazon/Rackspace/etc are always looking for use cases to show off. > I agree about 80% with that, well put. The disagreeing 20% is pretty much all about physical hardware, this is where, to quote a Red Hat employee I know 'the work gets interesting'. For the bulk of work, we can easily use virtual machines, scavenge bargain basement EC2 spot instance hours, and have lots of other options, your right that will be easy, the x86/amd64 arch testing wont be hard to find a home for. Its all the other arch work that wont be easy. I'm currently in the process of obtaining AMD Opteron A1100 dev kit boards, and lets just say I'm not expecting our software to 'boot first time'. Red Hat kindly keep the Beaker project (https://beaker-project.org) moving forward which will be how I deal with these AMD dev kits. It only really helps when you have hardware you can put aside for being part of a test pool. But it is one of the few tools available to easily boot hardware, splat an OS onto it, connect and perform automated tests on it, get as much info out as possible even if there are kernel issues and it doesn't boot properly. Once things progress I'd be amenable to letting my dev boards do ebuild test runs when I'm not using them to port our software stack. So while I dont have the 'idle hardware budget' of AWS, Google or Rackspace, I am however building a cloud platform, with a customised version of CoreOS, which is based on ChromeOS, which is based on Gentoo. And the further the company progresses, the more drift I see between our 'OS' and 'CoreOS'. I could not imagine tackling this if the entire thing wasnt built on top of the foundation of a Gentoo based OS. So consider me an ardent supporter of actually getting Gentoo automatic testing. > Rich Freeman <rich0 <at> gentoo.org> writes: > > From past feedback from Diego and such the biggest issue with any kind > of tinderbox is dealing with the output. As has been pointed out > there are folks running Repoman regularly already, and there have been > past tinderbox efforts. The real issue is to improve the signal/noise > ratio. You'll only get so far with that using code - there has to be > process change to support it.> > > If we were to do something like this long-term I'd envision that this > would run on every commit or something like that, and the commit > doesn't go into the rsync tree until it passes. If the tree goes red > then people stop doing commits until it is fixed, and somebody gets > their wrist slapped. That is my sense of how most mature > organizations do CI. The tinderbox is really just doing verification > - stuff should already be tested BEFORE it goes in the tree. There > also shouldn't be any false positives. There would need to be a > mechanism to flag ebuilds with known issues so that the tinderbox can > ignore them, and of course we can monitor that to ensure it isn't > abused. > > Basically doing this sort of thing right requires a big change in > mindset. You can't just throw stuff at the tree and wait for the bug > reports to come in. You can't just make dealing with the tinderbox > the problem of the poor guy running the tinderbox. The tinderbox > basically has to become the boss and everybody has to make part of > their job keeping the boss happy. 1st - Mindset change, definitely required. Can't agree more. 2nd - CI on this kind of thing is a multi-headed hydra of a thing. The processes we wind up with will be quite similar philosophically but not necessarily similar in implementation, staging or any other area. For starters most CI pipelines aren't testing an entire distro as complex as Gentoo ;-) 3rd - Looking at this linearly is less than idea. If an update breaks 5 downstream packages, the entire tree shouldn't go red and the only person who should stop is the maintainer and/or commiter who submitted the broken package. It should be more like automated QA 'gates' than a pass fail build pipeline. 4th - Signal to noise is crucial! I'm going to be actively doing something here because I have a build process that is building ebuilds using portage, and when a build can take 30 minutes then an hour to test the final machine image, I need to fail the build early, just flat out avoid any broken but installable package versions ever reaching the machine image testing stage of things. 5th - Ideally I think this will need multiple tools working together. However as far as being able to build something that can start doing the job, I feel like buildbot (http://buildbot.net) is probably the best place to start. Its more of a CI 'framework' than a CI 'tool', so it wont place too many 'not designed to do that' roadblocks at us. Its just my first recommendation, I'm sure the matter will need much more discussion. > James <wireless <at> tampabay.rr.com> writes: > > Or some other kind of hook? Fishing was very difficult till the hook > was refined. WE need a hook, imho. Easy Gentoo and a project to assist > in testing code; you've got a killer idea there rich, and I'm glad to > be on your team! > In my mind. Gentoo has the kind of hook you speak of. We just havent refined it or done a good job showing it off to the world. The ebuild format is one of the most powerful packaging standards. It isn't perfect but its an extremely good start. It lets us go anywhere... About a year ago I surprised a long time OSX user / Linux software developer twice in the same conversation after they told me how they used to use Macports before switching to Homebrew. First by mentioning that netbsd's pkg-src works on OSX, then the second time when I told them about the Gentoo Prefix project. They wanted 'freshness' in the software, and felt only Homebrew had it, but that was only because neither pkg-src or the Gentoo Prefix guys go out of their way to promote themselves as a tool for that job the way Homebrew and Macports do. An automatic test battery for Gentoo could possibly catapult Prefix and similar sub projects forward to much greater adoption, and help Gentoo reap secondary benefits as a result. Now all the replying is done, here's the big question, where to organise the bigger picture things? This mailing list? IRC? A new mailing list? I want to get involved in this because I'm going to be building some parts of this already. Why wouldn't I want to give back and help make Gentoo better.
