Thanks! A really useful one. It's definitely not about bit-by-bit reproducibility. That also came in our discussion about images with the Apache Security team. I am much more advocate of what we currently have in our Production images - they are designed in the way that you should be able to have the "buildable" at any time and you should get the results that you want but they might be rather different depending on when you are running them (but they should be pretty much guaranteed to work whether you run them now or in 4 months). But if you build it 4 months from now, the result should include all (or rather all possible) security fixes that were released between now and then.
Our image building process works exactly this way. It took quite some time and effort (especially because of our dependency management "we are both application and library"). But those properties below always were - my goal from the beginning of designing the process. And quite recently we finally got where I wanted it to be. For example, the current image when you build it using 1.10.12 tag and Breeze - following the instructions from IMAGES.rst), it will (automatically): * take as a base latest released debian-buster-pyhon image (LTS support - will be supported till 2022 or even 2024). * install latest versions of all apt dependencies that are available for that release * install latest versions of PIP packages that we tested (automatically) in 1.10.10 branch (we automatically bump up constraints in our repo when all the tests pass after pushing to v1-10-stable branch) All this is automatically tested - so latest base image + latest apt deps + latest PIP packages are automatically tested and are only "blessed" once all tests pass for them. Thanks to that we know, that when user tries to build the same image at any time (following the --constraint mechanism) - it will work. And we give instructions to the user how to do it. This is what I mean by "rebuildability". J. On Wed, Sep 9, 2020 at 12:40 PM Ash Berlin-Taylor <[email protected]> wrote: > > Jarek, you might want to check out https://reproducible-builds.org/ -- > it's come out of Debian with the aim to try and make binary builds of > .deb/software installed via .deb binary reproducible. > > I think you have less strict goals in mind than bit-for-bit identical > rebuilds of docker images? If so probably worth highlighting that as a > "non-goal" of your propsal (assuming it is a non-goal?) > > -a > > On Sep 9 2020, at 11:35 am, Jarek Potiuk <[email protected]> wrote: > > > Hello Niclas, > > > > Thanks for that. > > > > I feel that this guidance already answers most of my questions. > > > > I volunteered to lead proposal discussion and preparation for the ASF Board > > on this subject (and I am sure other PMCs from Airflow will also be engaged > > a lot, so I hope we can work out some reasonable policies on that. I hope > > to have the first draft proposal for discussion this week. I also invited > > Apache Security team members who are already commenting on that > > thread, as > > I think those policies should at least provide guidance on all those > > topics: licensing, security, stability, and "rebuildability" (for the lack > > of a better term). Those are IMHO super important if we want to > > address the > > needs of corporate users especially (looking at the requirements of the > > corporates we are working with). > > > > J > > > > > > On Wed, Sep 9, 2020 at 8:38 AM Niclas Hedhman <[email protected]> wrote: > > > >> Hi everyone, > >> > >> The report submitted to the September Board meeting is requesting guidance > >> on binary releases, such as Docker and Helm. I act as the board's shepherd > >> of Airflow, and here to help if needed. > >> > >> First of all, Apache Software Foundation releases Open SOURCE > >> software, and > >> the source release is always the primary one. There are many reasons for > >> this, such as security (one can know for sure what it contains), > >> jurisprudence (trace origin,++) and usability on platforms that the > >> community may not provide binaries for. > >> > >> Many communities provides additional binary releases, that has been called > >> "Convenience Binaries", but the term is under review/reconsideration as > >> they are for some communities (say, OpenOffice) the primary > >> artifacts for > >> the majority of users (OpenOffice users are typically not > >> developers). The > >> exact policies around this are being reviewed and worked on at the moment. > >> Things like credentials to DockerHub or npm are for instance of > >> concern, as > >> well as the long-term stability of some of these distribution systems. > >> > >> That said; in general, as long as the binaries are buildable (with > >> instructions) and the product can be built and used without reliance on > >> such external systems, then it is mostly OK and it is up to each community > >> to decide if binaries are provided and how. If there are specific questions > >> on release policy or special requests, then contact the > >> Infrastructure team > >> and ask if it is Ok with them. If there are more general > >> thoughts/feedback/discussion items in this space, ComDev is the place to > >> approach. > >> > >> I will also try to do my best to answer questions here... > >> > >> Niclas Hedhman > >> > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea | Principal Software Engineer M: +48 660 796 129
