tl;dr - xnox wants to remove 1 344 (35%) source packages from main Google Doc for suggestions & comments: https://docs.google.com/document/d/1dJBtLLCppH2yt664S8G2jB_tK-iWi_D7wqaN6S4ddwI/edit
Sample old/new germinate output is at: http://people.canonical.com/~xnox/germinate-output/ --- = Archive Reorg Episode VII: Follow Build-Depends = == Introduction == Loosely this builds up on the existing portions of the Archive Reorg evolution. To recap, we have 4 components in the archive, which are defined by their licensing on one axis, and based on seeds for the other axis. Thus e.g. sufficiently free packages, that are seeded in specific products are published in main component. Over the years, the structure, processes and permissions of the archive have evolved. And today we have a spectrum of upload rights, at per-package, per-seed, per-packageset, per-component and all-in-one core-dev. We have also been expanding our confidence in using universe to test and validate main. For example, autopkgtest are executed archive-wide without main/universe segregation and thus packages in universe can catch and prevent release of broken packages in main. And vice versa. Similarly we have been using components in a flexible manner, e.g. source packages in main can build binary packages that are published in universe. Currently, main is a closed set across three dpkg relationships: - Depends - Recommends (* with follow-depends feature, default for ubuntu-platform) - Pre-Depends - Built-Using (* not currently implemented but should have been) - Build-Depends[ -Arch | -Indep ] With rigorous processes around these - e.g. seeds, germinate, components-mismatches, MIR, etc. This has caused a creep of technical debt, and high ongoing maintenance. For example, to keep unsupported languages/runtimes out of main, a permanent fork had to be established from Debian to split the sources, and disable build dependencies. E.g. we have two identical boost packages (one in main, one in universe) to keep openmpi out of main. Similarly we have disabled pypy extensions of many packages in main, to keep pypy in universe. Or even trivial things like disabling building documentation (desired), due to required tooling being unsuitable for seeding in main (undesired). In effect this cripples Ubuntu in many ways, and generates extra and potentially unnecessary work for Ubuntu developers. == Proposal == I would like to propose to relax the requirement for build-depends of packages in main to come from main only, and thus stop following build-dependencies to be part of a closed set in main component. This would mean that the universe component will always be available to get build-dependencies. Main will remain a closed set of binary packages dependencies, sources and built-using. This will enable building, in a single pass, packages with testsuite or optional/additional bindings that require universe build-dependencies or will be published into universe anyway. Some examples: - boost can become one src package in main, with mpi portions built, yet packaged/published into universe - testsuite only dependencies can be used to run the full testsuite at buildtime (e.g. automake has a plethora of things it wants to validate in universe) - documentation tooling can be pulled from universe to build documentation, and ship the resultant static files in main (e.g. zsh) - we can build pypy bindings for packages in main, and e.g. python2 may (in the future) drop to universe This does not abolish the MIR process. If a hard binary dependency is gained (e.g. shared linking), the binary and source package still must go through MIR process and be published in main. This scheme however creates new caveats, and potential for “static” linking to leak from universe into main binaries: - a C++ template only library, becomes available to be used as a build-dependency, without going through main, or generating a closed-set relationship of “depends”, or “recommends”. - a statically linked library from universe, can end up in main. - macros and other code in C header files from universe can end up in main - bootstrap & complete build-dependencies chain may end up exponentially growing, thus e.g. it may become harder to remove obsolete universe packages For these cases built-using should be used. And built-using should be included in the main closed set. == Technical changes and feasibility == To accomplish above the following changes would be required: germinate - add an option to [no]-follow-build-depends - make sure “built-using” sources are followed components-mismatches - make sure “built-using” are included launchpad - enable “universe” component by default, per distroseries == Feedback and Retrospective == Some may recall Archive Reorg Episode VI plans, which were drafted years ago. This proposal is much smaller in scope, aiming to resolve the build-dependencies issue as a stand alone quirk. This is a technically small change that can be done in 16.04 LTS, in launchpad / server side, without affecting client side installations, and/or Ubuntu Mirror network. Specifically, original plan had much larger proposals to reform all components, and expose “sets” client side which would require more engineering time than there is for 16.04 LTS. Whilst simple in its smallest wording “enable universe by default on buildds”, it is quite a large change which will affect all of Ubuntu development thus careful consideration, planning, and feedback from the Ubuntu Developer community is required. -- Regards, Dimitri. -- ubuntu-devel mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
