Hi,
This is a bit of a two in one proposal, which sets out to address the
following two problems:
Mirroring of sources from subprojects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We don't have any user configuration to override the mirror definitions
for subprojects.
All we have to achieve this is a way to override the default mirror
which will be tried in a given subproject.
Fetching from mirrors only
~~~~~~~~~~~~~~~~~~~~~~~~~~
We don't have any way to prevent BuildStream from fetching sources from
default URIs, or defined mirrors which we don't want to trust.
This means it is impossible to be sure that all of the data in your
build is coming from a location under your control.
One reason this is important is repeatability, you cannot be sure that
you can repeat a build in 10 years if you are not sure that you have
stored all of the input data.
Proposal
========
Here are two actions I would propose to solve the afore mentioned
problems.
Use configuration to override mirrors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In addition to allowing the user configuration to specify the default
mirror[0] on a per project basis, I would suggest that we also allow
the user configuration to redefine the mirrors dictionary[1] which
specifies the mirror names and URI values.
This would allow the user to completely define the mirrors to use for
each project in the session using that configuration file.
Taking the sample configuration from the docs, it might look like this:
projects:
#
# Configure the mirrors for project `project-name`
#
project-name:
mirrors:
- name: middle-earth
aliases:
foo:
- http://www.middle-earth.com/foo/1
- http://www.middle-earth.com/foo/2
bar:
- http://www.middle-earth.com/bar/1
- http://www.middle-earth.com/bar/2
- name: oz
aliases:
foo:
- http://www.oz.com/foo
bar:
- http://www.oz.com/bar
Restrict operation to use only mirrors
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In order to restrict operation to use mirrors only, I would suggest
only using a single user configuration rather than a per project
configuration, which doesn't seem to be useful.
It would be interesting for this to be an enumeration, allowing the
following behavior:
* Unrestricted fetch/tracking URIs
* Restrict fetching and tracking to use only URIs defined by mirrors
* Restrict fetching and tracking to use only URIs defined by mirrors
that are defined in the user configuration, ignoring any mirrors
which were declared by project configuration.
The last option is especially important in order to ensure that you are
in fact mirroring all the required assets, as otherwise you might end
up falling back on mirrors defined by third party projects you depend
on instead of mirroring all of the data yourself.
Let's suggest that the name of this configuration be the "fetch
source", and the value is allowed to be "all", "mirror" or "user", and
that we add a new "fetch" section after the "build"[2] section of the
docs in order to create an appropriate category for the option.
It could look like this:
#
# Fetch controls
#
fetch:
#
# Control the source of where data can be fetched and tracked from
#
# all - Any URI, including default alias values and mirrors
# mirror - Only allow fetching from mirrors
# user - Only allow fetching from mirrors declared in user
# configuration
#
source: all
In addition, I would like to add a similar CLI option `--fetch-source`
to set these values on a per-invocation basis.
It is interesting to control this on the CLI so that you can
alternatively default to the upstream URIs when running `bst source
track` - this can help organizations to discover if their mirroring
solutions are not working, if new refs are discovered at `bst source
track` time but cannot be found later at `bst source fetch` time.
Thoughts ?
Cheers,
-Tristan
[0]: https://docs.buildstream.build/master/using_config.html#default-mirror
[1]: https://docs.buildstream.build/master/format_project.html#mirrors
[2]: https://docs.buildstream.build/master/using_config.html#build-controls
