> Right, and while I'm happy to entertain the possibility to support something > like this, it seems a bit like one of those gargantuan projects we might end > up > with. So, I guess I'd also like to understand the motivations, and perhaps > whether other alternatives are not fulfilling? :)
We're going to be building inside of Docker containers. This allows us to better verify dependencies as well as to be able to work on multiple different releases concurrently. The workflow which has inspired this is a standard developer workflow where a source code repo is checked out using eg. git clone. Then a repeated cycle of edit/build/test occurs. Once done, a commit/submit/PR is generated and the workspace is presumably deleted. As a part of this, we want to avoid forking build paths for different cases. For example, if the developer build uses a different build process from the build process which produces our "official" package builds, it's quite likely that one or the other will break without someone promptly noticing. Going through the dpkg tools every time will minimize the forking paths. Also, we want to ensure that incremental builds are fast. For trivial repositories, building from scratch every time is fine. For larger repositories the overhead of a clean build can add substantial overhead on this cycle. So we need to preserve intermediate build state. The next issue we're considering is changes made to repository files by the build process itself. For example, the autoreconf process can change the contents of the configure script. If that's done in the developer's sandbox, the changes may easily be picked up in a commit/submit/PR. Likewise for any patches applied by the dpkg tools. Though "don't do that" is entirely reasonable on a small scale, especially for issues which may be reverted, part of the practice of engineering is to prevent the possibility of mistakes occurring in the first place through engineering controls. In this case, the approach seems fairly obvious: as a part of running the build in a Docker container, make sure that the developer's source code repository is bind-mounted read-only. This way the build process can't inadvertently alter the developer's sandbox. I have a work-around to this ticket which allows this process to work. I'm using rsync to copy the developer's sandbox (preserving timestamps) to a RW build directory which is preserved across build invocations. This keeps the build objects for fast rebuilds while ensuring that the developer's sandbox isn't inadvertently modified. The down-sides to this include extra I/O and thus time required by rsync to determine if each file needs to be copied, (trivially) extra disk space, and additional tools to be installed in our build Docker image. Part of this design is based off of some experimental work done with Bazel which explicitly uses hermetic builds. See: https://bazel.build/basics/hermeticity Though not attempting to achieve hermeticity, out-of-tree builds manage to provide a large portion of the benefits while requiring only a small amount of the work. The rsync solution is certainly acceptable in the short term, but something cleaner would be preferred. > I've not checked very deep on what might be blocking this, or whether there > might be other difficulties, so do not take the following as extensive. > > > For this to be possible, it would require two "independent" features > > in the Debian build stack: > > > > C1) Provide a writeable directory for the output artefacts (the > > produced .debs, the .changes file etc.) > > - This would enable to the parent directory be read-only. > > This would currently be blocked by at least something like #657401, I think > > > C2) Provide a writeable directory for the build process itself. > > - This would enable the source directory itself to be read-only. > > > > => There will be packages that *cannot* support C2 as they only > > support "in-source" built (which is probably why we have the > > current setup in the first place). > > I think this is partially blocked on something like > <https://lists.debian.org/debian-devel/2020/03/msg00085.html> > , and because several of the currently written files > are part of the expected interface. :/ My suggestion would be to keep that tree as is, but relative to the build directory rather then in the source directory. This would effectively be namespacing the 2 different directory paths, but that's not unusual for build systems. Bazel does Magic(tm) to build in isolated temporary directories. Automake supports parallel build trees: https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html Cmake not only supports this: https://cgold.readthedocs.io/en/latest/tutorials/out-of-source.html but goes further to state: 'CMake does support the contrary “in-source build” layout, but such an approach has no real benefit and is not recommended.' This is clearly seen as a popular feature of build systems, even if every wrapped build system can't support it easily. > The main problem I see there, besides the current pathnames as interfaces, > is that I don't think this can be universally supported by any package, and as > such something like dpkg-buildpackage would need to way to know whether > the package supports this mode, for every and each package (say a field such > as the R³ marking), and if it that does not get tested per package, then it > will > tend to break. :/ > > So I think I'm also currently in the "does it outweigh the cost?" > pondering phase too. Yeah. Being a maintainer of critical infrastructure requires a lot of thoughtful analysis. > > > Many of the required capabilities are already supported. For > > > example, the --builddirectory flag allows the building step to be > > > placed in a different directory. > > > > While there are many options that package can use to tweak locations, > > they are not aimed at the thing you are doing. Notably the dh_* -P > > option is really painful to use at scale (if you have multiple binary > > packages in the same source package). > > Also I'd expect many source packaging (say overrides in debian/rules or even > debian/rules not using dh or debhelper), ancillary packaging tools (say dh > extensions) might expect pathnames to be in specific locations. I agree - this is looking like it's non-trivial. It would require addressing the assumptions in the underlying tools as well. > Thanks, I've not yet checked what might be going on, but I'll do and try to > fix > it! Great! > In any case, sorry, I think the outlook might not be very bright, but as I've > said > at the beginning, I'm happy to still entertain this. Thank you, - Garrett
