On 07/02/14 at 18:16 +0100, Peter Palfrader wrote: > On Sat, 08 Feb 2014, Thomas Goirand wrote: > > > It'd be super nice to have > > the archive rebuild jobs running on the Debian infrastructure rather > > than on AWS for example. > > I agree, and it has been proposed several times over the last few years. > To say there was no interest whatsoever would overstate the amount of > excitement those suggestions have received.
The archive rebuilds are currently running over a very short amount of time (< 8 hours). That's an important feature, because it allows one to rebuild a snapshot of the archive (or a quasi-snapshot, since that could span two dinstalls) and make sure to find all build failures in that snapshot. Unfortunately, to do that, one requires quite a lot of computing power. As a wild guess, I would say 20 to 30 recent servers (I've been using more than 100 EC2 VM for some rebuilds, especially when doing two rebuilds at the same time to compare the rebuilds, e.g. gcc 4.n vs gcc 4.(n+1)). We currently get this computing power for free from Amazon. We used to get it from my employer through a research project (Grid'5000), and we could return to that if needed: the move to Amazon was an attempt at switching from 'only Lucas can work on that' to 'other people can work on that', which was a success given at least David Suarez (general purpose rebuilds + bug filing) and Sylvestre Ledru (clang rebuilds) have been actively running rebuilds over the last year, and I haven't. Also, we made sure of not using anything Amazon-specific in the rebuild infrastructure. Doing those rebuilds on Debian infrastructure would require either: 1) investing in a set of powerful machines to do the rebuilds ($50k as a guess, $2k * 25 -- and that's probably the bare minimum we would need), finding hosting for them, managing them. We get all of that for free from Amazon, so I don't think that this would be good use of Debian's funds. 2) degrading the service: loosen the requirement to rebuild quasi-snapshots of the archive, or use snapshot.debian.org to build a possibly older snapshot of the archive (which would result in filing bugs that were already fixed). None of those options look particularly exciting. That's why I never explored further the idea of running those rebuilds on Debian infrastructure. But maybe you have a nice solution, that you haven't explained yet? One thing we could do, though, is move all the "static" part of the rebuild infrastructure to Debian infrastructure. That's the virtual machine used to schedule builds on all temporary nodes, and the virtual machine used to store logs. Those logs used to be stored in http://people.d.o/~lucas/, which did not work scale well to several people doing the rebuilds. I approached DSA asking if they would be willing to provide similar archival space in a place accessible by a team including non-DDs. (#debian-admin, 2013-06-03) All of the suggested solutions (mail all build logs as attachments to the bug reports; ask someone else or a cron job to rsync from external host to qa.d.o; push to buildd.d.o) had drawbacks or required additional work that I didn't have time for at the time, so we have just been using an EC2 VM (aws-logs.d.n) to store the logs since then. Probably that should be revisited, but I'm no longer the good point of contact for that (David Suarez is). Lucas -- To UNSUBSCRIBE, email to [email protected] with a subject of "unsubscribe". Trouble? Contact [email protected] Archive: http://lists.debian.org/[email protected]
