Hi Ben, On Fri, 2020-10-09 at 14:24 +0000, Benjamin Schubert wrote: > TLDR: I would like a way of allowing network access during builds on a > per-element basis, to ease transition into BuildStream. > > I realize this has a potential to be highly controversial, but I think that > the benefits in terms of ease of adoption makes it worth it.
While it's obviously not ideal, I think it makes sense to support it if there are real cases where this will significantly ease transition. This would obviously need a big and clear disclaimer in the documentation. > [...] > > How would that look like? > -------------------------- > > I would envision a new variable option, similar to 'max-jobs', that would be > transparent to the plugin, but used by BuildStream to open network access. `max-jobs` is a configuration option, not an element variable. We could follow the approach of the `notparallel` element variable. However, I think the already existing `sandbox` section would be a better fit. > How much work would we need to do for this? > ------------------------------------------ > > We already have mechanisms to open the sandbox when running a shell. > We would need to make this possible for regular builds. A potential issue is `resolv.conf`. Projects can bind-mount it in their `shell` configuration. However, I don't know whether bind-mounting is a good idea for build sandboxes. It would also leave the question of where to configure it. And bind-mounting is not supported on non- Linux/bubblewrap systems, although a fallback to copying the file may be acceptable. Have you already thought about this? > This would also mean we need to be able to be able to set it on platform > properties for remote execution. buildbox-run-bubblewrap already supports this with the platform property `network` (used by `bst shell` for local execution), however, this isn't a standard platform property. > Mitigations > ----------- > > Some potential mitigations around this being 'bad' could be: > > - Optionally disable by default the pushing of such artifacts to remote > caches, > like we currently handle workspaces, with the ability to enable it either at > the project level or the element level. I don't think we need this and I'd rather keep it simple. The new network setting must affect the element cache key but then I don't see any real benefit in not pushing such artifacts. It may make sense to print a warning message for every build job that enables network access (which could be configured to be fatal, similar to overlaps). Another improvement would be to make the new configuration option a list of host names to allow access to, instead of blanket network access. A tiny DNS proxy could cover this at least to a certain degree (without packet filtering or full IP proxy we can't prevent access by IP address or alternative DNS server), or maybe even a generated `hosts` file with entries for permitted hosts. This may also help solve the `resolv.conf` issue. Cheers, Jürg
