Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 2018-03-27 14:29, Wookey wrote: On 2018-03-22 22:31 +0100, Aurelien Jarno wrote: On 2018-03-07 12:08, Hector Oron wrote: > 2018-03-07 12:01 GMT+01:00 Philipp Kern: To do more tests I agree that we should deploy it on more buildds. For *this phase*, I believe enabling lingering and deploying it by hand is the best. Is it something acceptable? Anything is OK for _testing_, but I think it's always been a problem that we don't use packaged code to do our build infrastrucutre, because this makes it very hard to reproduce. And thre are lots of reasons why people do want to have their own build infra of various sorts. So if we are having a re-arrange, I think we should try very hard to use our own mechanisms and package this just like we package everything else, not because it's easier for us, but because it's easier for everyone else. I relate with that. However this particular setup is not really intended to be reproducible by others at this point. I needed to cut the knot somewhere first, but wanna-build really is a pain to keep running. Once we are rewriting that, it might make more sense. As it stands it's pretty specific to our needs, unfortunately. But as it turns out it shows rather nicely how the reality of providing a service and using raw packages from Debian don't really work. It works for the most basic case (e.g. "I just need a DNS/DHCP server"). But as soon as you scale that out or add some kind of high availability, you run into the need to orchestrate using Puppet or so, or even need to run your own custom-built solution on top of everything else Debian provides. If you have a setup where DSA provides the VMs and does not want to give out root access and you have no access to the orchestration tools either, it's hard to keep a service running unless you run to DSA for each and every request. It's not surprising that they don't want that either if it can be avoided. Plus making changes hard by requiring that every change goes to testing and then backports isn't really feasible either. Kind regards Philipp Kern
Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 2018-03-22 22:31 +0100, Aurelien Jarno wrote: > On 2018-03-07 12:08, Hector Oron wrote: > > 2018-03-07 12:01 GMT+01:00 Philipp Kern: > To do more tests I agree that we should deploy it on more buildds. For > *this phase*, I believe enabling lingering and deploying it by hand is > the best. Is it something acceptable? Anything is OK for _testing_, but I think it's always been a problem that we don't use packaged code to do our build infrastrucutre, because this makes it very hard to reproduce. And thre are lots of reasons why people do want to have their own build infra of various sorts. So if we are having a re-arrange, I think we should try very hard to use our own mechanisms and package this just like we package everything else, not because it's easier for us, but because it's easier for everyone else. Wookey -- Principal hats: Linaro, Debian, Wookware, ARM http://wookware.org/ signature.asc Description: PGP signature
Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 27.03.2018 09:22, Julien Cristau wrote: > On 03/22/2018 10:31 PM, Aurelien Jarno wrote: >> To do more tests I agree that we should deploy it on more buildds. For >> *this phase*, I believe enabling lingering and deploying it by hand is >> the best. Is it something acceptable? >> > Sounds ok to me, though I'd prefer if the buildd user (as in the user > the service runs as) didn't have write access to the deployed code or > configuration. Maybe that's a later step though. Does that really matter? If we were to use user sessions it's a little hard to prevent that. Unless we tighten up the permissions to make all service directories owned by someone else, which then makes sudo'ing to that user a little awkward. Plus chown'ing files around. Because otherwise someone with write access to ~buildd's home can still juggle around things. We could run the build as a different user in theory, but that's somewhat hard with sbuild right now unless we "sudo -u" over when executing sbuild. And then we need to manage files as a different user as well. If the threat model is that the ongoing build tampers with the files, I'm open to constrain the build environment even further. Unfortunately while it seems that a newer schroot version (in a branch that did not receive any updates for years anymore) would support those, that version has just been removed from experimental. That all said: I'm happy to brainstorm and to implement proposed solutions. Right now that seems to require more infrastructural changes though and I'd rather focus on getting rid of buildd itself first. Kind regards and thanks Philipp Kern
Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 03/27/2018 09:35 AM, Philipp Kern wrote: > That all said: I'm happy to brainstorm and to implement proposed > solutions. Right now that seems to require more infrastructural changes > though and I'd rather focus on getting rid of buildd itself first. > That's fair. Cheers, Julien
Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 03/22/2018 10:31 PM, Aurelien Jarno wrote: > To do more tests I agree that we should deploy it on more buildds. For > *this phase*, I believe enabling lingering and deploying it by hand is > the best. Is it something acceptable? > Sounds ok to me, though I'd prefer if the buildd user (as in the user the service runs as) didn't have write access to the deployed code or configuration. Maybe that's a later step though. Cheers, Julien
Re: Python buildd (was Re: let me help you with progressing bikesheds)
Hi, On 2018-03-07 12:08, Hector Oron wrote: > Hello, > > 2018-03-07 12:01 GMT+01:00 Philipp Kern: > > Has it been tested? Well, lightly. Unfortunately I'm not really allowed to > > touch the buildds as root so it's kinda hard to replace the existing ones > > given that stuff is enforced by Puppet. So I did a few builds on zemlinsky. > > Should it be tested on one or two? Yes. In theory it should work. :) > > > >>> There were concerns about how DSA wanted this to be packaged (essentially > >>> not at all). > >> > >> > >> Sorry I missed those concerns, could you expand on them or link to > >> discussion? > > > > > > I mostly worked on this with Aurelien and the concern seems to have been > > that DSA would prefer user-based services with linger rather than packages. > > There are two ways to deploy it: > 1. Get the code in buildd via DSA puppet. > 2. Create Debian package, upload it, make a backport. Then it can be > deployed via backport. Can we move forward on that issue? Discussions on IRC suggested that using apt.buildd.debian.org, like it's done for the current packages is not the right choice. The suggested way to run things would be to just deploy things in ~buildd. As pybuildd uses systemd services, it means we need to enable lingering on buildds. Is it fine? The alternative as suggested by Hector is to use the package from backports. This means less flexibility if we need to update the code, as it will take at minimum 5 days following the policy. To do more tests I agree that we should deploy it on more buildds. For *this phase*, I believe enabling lingering and deploying it by hand is the best. Is it something acceptable? Thanks Aurelien -- Aurelien Jarno GPG: 4096R/1DDD8C9B aurel...@aurel32.net http://www.aurel32.net signature.asc Description: PGP signature
Re: Python buildd (was Re: let me help you with progressing bikesheds)
Hello, 2018-03-07 12:01 GMT+01:00 Philipp Kern: > Has it been tested? Well, lightly. Unfortunately I'm not really allowed to > touch the buildds as root so it's kinda hard to replace the existing ones > given that stuff is enforced by Puppet. So I did a few builds on zemlinsky. > Should it be tested on one or two? Yes. In theory it should work. :) > >>> There were concerns about how DSA wanted this to be packaged (essentially >>> not at all). >> >> >> Sorry I missed those concerns, could you expand on them or link to >> discussion? > > > I mostly worked on this with Aurelien and the concern seems to have been > that DSA would prefer user-based services with linger rather than packages. There are two ways to deploy it: 1. Get the code in buildd via DSA puppet. 2. Create Debian package, upload it, make a backport. Then it can be deployed via backport. Regards, -- Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-. -- Would you like to make a donation towards the upcoming Debian conference? Brochure: https://media.debconf.org/dc18/fundraising/debconf18_sponsorship_brochure_en.pdf ** https://debconf18.debconf.org/sponsors/become-a-sponsor/ **
Re: Python buildd (was Re: let me help you with progressing bikesheds)
On 2018-03-07 10:39, Hector Oron wrote: 2018-03-07 7:01 GMT+01:00 Philipp Kern: FWIW, I have rewritten the buildd side in Python (https://salsa.debian.org/pkern/pybuildd) and am currently waiting for that to be deployed. There were concerns about how DSA wanted this to be packaged (essentially not at all). Replacing the buildd side with something relatively simple to understand is somewhat a prerequisite to change the wanna-build side... Thanks for taking the time to rewrite that part. Is it ready for production? Has it been tested? Has it been tested? Well, lightly. Unfortunately I'm not really allowed to touch the buildds as root so it's kinda hard to replace the existing ones given that stuff is enforced by Puppet. So I did a few builds on zemlinsky. Should it be tested on one or two? Yes. In theory it should work. :) There were concerns about how DSA wanted this to be packaged (essentially not at all). Sorry I missed those concerns, could you expand on them or link to discussion? I mostly worked on this with Aurelien and the concern seems to have been that DSA would prefer user-based services with linger rather than packages. Kind regards Philipp Kern
Python buildd (was Re: let me help you with progressing bikesheds)
Hello Philipp, 2018-03-07 7:01 GMT+01:00 Philipp Kern: > FWIW, I have rewritten the buildd side in Python > (https://salsa.debian.org/pkern/pybuildd) and am currently waiting for that > to be deployed. There were concerns about how DSA wanted this to be packaged > (essentially not at all). Replacing the buildd side with something > relatively simple to understand is somewhat a prerequisite to change the > wanna-build side... Thanks for taking the time to rewrite that part. Is it ready for production? Has it been tested? > There were concerns about how DSA wanted this to be packaged (essentially not > at all). Sorry I missed those concerns, could you expand on them or link to discussion? Regards -- Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-. -- Would you like to make a donation towards the upcoming Debian conference? Brochure: https://media.debconf.org/dc18/fundraising/debconf18_sponsorship_brochure_en.pdf ** https://debconf18.debconf.org/sponsors/become-a-sponsor/ **