This is a writeup on a problem we’re facing with modularity, and some ideas on how to resolve it.
# The "Problem" Imagine I have an **httpd module**. To simplify things, let’s say that this module has only one stream: **2.4**. Today, in the modulemd for this module, I declare build and runtime dependencies like this: dependencies: buildrequires: platform: f26 requires: platform: f26 This works. Now, imagine that Fedora 27 comes along. I want the httpd module to be installable on f27, so I update those streams to point to f27. But now I can’t ship updates to f26 anymore! Uh oh. ## Just use more streams? We could just ask packagers to create more streams to deal with this. httpd wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a f27-2.4 stream. But, this gets us exactly back into the place we were **before** we had arbitrary branching! You need to have as many branches for *every component* as you have releases of the distro (or, as you have "products"). Noooo! ## Solution: "Input" Modulemd Syntax Changes We’re going to extend the modulemd syntax to allow specifying multiple dependencies in an "input" modulemd (the one that packagers modify). When submitted to the build system, the module-build-service (MBS) will *expand* these values under the hood, and submit **multiple** module builds to itself based on the expansion. Here are some examples of modulemd files (maintained by humans) that might be submitted to the MBS. Build on **all** active streams of the platform module. dependencies: buildrequires: platform:  Build on **only** the f27 and f26 platform streams. dependencies: buildrequires: platform: [ f27, f26 ] Build on all active streams of platform **except** for the f26 stream. dependencies: buildrequires: platform: [ -f26 ] The following syntax, which builds on **only** the f26 stream, will still be supported: dependencies: buildrequires: platform: f26 -- Take the following example (described above): dependencies: buildrequires: platform: [ f27, f26 ] Submitting this modulemd to the MBS would in turn generate **two** module builds: one of our httpd module against the f27 stream and another against the f26 stream. Each module build would be associated with its own unique "flattened" *output* modulemd file that specifies exactly which platform stream it was built against. # New Problems Having **multiple** module builds for a single dist-git commit of a modulemd file poses new implementation problems. ## NSV uniqueness Today, we uniquely identify a module build in *a variety of systems* with **<name>-<stream>-<version>** (NSV) where version is derived from the dist-git commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an http-2.4 module built on f27: two different module builds with the same name, stream, and version. How can we tell them apart? We will introduce a new value called the **context** of a built module and include it in the modulemd metadata that gets carried with the built module. * For user facing cases (dnf installation) it will generally be hidden. The old NSV value will still appear. If a user ever needs to surface the value, client tooling can find it in the modulemd metadata. The additional identifier will give users access to explicit pointers to the content that is installed: * For cloning a machine. * For comparing two installed hosts. * For reporting bugs. * Some build and compose time systems will have to be modified to use the context as part of a new unique identifier. **NSVC** will be the **<name>-<stream>-<version>-<context>**. The systems in question are PDC, koji/brew, pungi, and bodhi. The context value will be a **hash**, generating as the first step in the build process (but after expansion). Consider what metadata needs to be hashed: we think that hashing the whole modulemd is problematic, because the modulemd can and will be modified after build time. Therefore, the *context_hash* value will need to be derived from only the stuff that uniquely identifies the build and runtime context of the module -- name, stream, version and crucially, its dependencies ## Buildtime/Runtime Dep Correlation Another problem. We now have input modulemd files for a single stream that can expand buildtime dependencies to ‘f27’ and ‘f26’, but what about the runtime dependencies? Consider the following yaml file: dependencies: buildrequires: platform: [f28, f27, f26] requires: platform: [f28, f27, f26] With our thinking caps on, this should obviously generate three module builds (one that buildrequires f28 and run requires f28, a second that buildrequires f27 and run requires f27, and a third for f26). The naive cross-product of all streams is not valuable; the MBS needs some logic to **correlate** the build time requirements with the run-time requirements. That logic will contain assumptions about how this will be used, and it needs to be well-defined. Let’s try to define that. Consider a more complicated yaml file that depends on both multiple streams of platform and on multiple hypothetical streams of a "shared-userspace" module: dependencies: buildrequires: platform: [f28, f27, f26] shared-userspace: [fancy, nonfancy] requires: platform: [f28, f27, f26] shared-userspace: [fancy, nonfancy] Here are some rules the MBS should obey: * If a build-time dep contains an expansion, that dep **does not** have to also appear as a run-time dep. * If a build-time dep contains an expansion, and if that dep *also appears* as a runtime dep, the runtime expansion **must** match the buildtime expansion exactly. Now that the streams from build-time and runtime *can be mapped one-to-one*, the cross-product of the deps is taken to produce a set of modules builds. In our example here, we get: 1. One build with platform f28 and shared-userspace fancy. 2. One build with platform f27 and shared-userspace fancy. 3. One build with platform f26 and shared-userspace fancy. 4. One build with platform f28 and shared-userspace nonfancy. 5. One build with platform f27 and shared-userspace nonfancy. 6. One build with platform f26 and shared-userspace nonfancy. We like this. -- But, a new** corner case** emerges! What if the *nonfancy* branch of shared-userspace is only compatible with f27 and f28, but not with f26? Build #6 will always fail - or, if it doesn’t fail it will be unusable. We thought through a number of ways that corner cases like these could be handled automatically, and are still working through them. At the moment, we are *leaning towards* suggesting that a failing corner **requires** a new stream of the module in question to handle the exclusion. I.e., no convenient syntax to say "all combinations except for this one". In this case, you would have two streams of the httpd-2.4 module: httpd-2.4 and httpd-2.4_f26. The first modulemd file in the 2.4 stream would be nice, like this: dependencies: buildrequires: platform: [f28, f27] shared-userspace: [fancy, nonfancy] requires: platform: [f28, f27] shared-userspace: [fancy, nonfancy] The second in the 2.4_f26 stream would exist only to support the desired corner case: dependencies: buildrequires: platform: [f26] shared-userspace: [fancy] requires: platform: [f26] shared-userspace: [fancy] -- We’re still working through more details, especially on this last part (and how it may affect upgrades), but figured its best to share now. Any help appreciated! Yours- -Ralph
Description: PGP signature
_______________________________________________ devel mailing list -- email@example.com To unsubscribe send an email to devel-le...@lists.fedoraproject.org