Hello, I'd like to invite you to look at the new reproducibility test script we developed for freedesktop-sdk, I believe that reproducibility testing is a important feature on any build tool and strangely, it's something that is missing in all of them. even modern build toolkits like Meson, CMake and Bazel have no direct way to trigger a build to check reproducibility for the same architecture.
Buildstream also misses that, and the only way to implement is to bypass buildstream and hack around the tool. Currently it's not trivial to implement this, at all. You also need to force a build to ignore the remote artifact cache, but not the sources - as downloading from the remote cache twice and comparing the results is not actually testing reproducibility. We have to deal with two things on the connection point of view: we need to be able to block connection to the remote artifact cache, but at the same time we do want to download the sources, so that we can download from the remote cache once, copy the local cache to folder A, build locally folder B with the downloaded sources, and compare both. We need a way to disable internet completely for certain actions, but that's not trivial. The initial script used unshare, but we found out that this approach created more issues, then we went to a LD_PRELOAD approach with a library that forbid internet execution.There should be a better approach for this. I would like a round of review on the script, as the current idea is to move that to buildstream as a separate tool. I might missed something (well, I probably missed something, this is the first time I touch something so deep in buildstream code). Currently it will just work on freedesktop-sdk as it requires the LD_PRELOAD hack, but the idea is to move this to bst in the future, when bst supports everything that's needed for this. The script lives currently here: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/blob/master/utils/test_reproducibility.py
