On Tue, Nov 26, 2019 at 6:35 PM Zbigniew Jędrzejewski-Szmek <zbys...@in.waw.pl> wrote: > > As a meta-goal, we should break up "Modularity" into a number of > separate components, some build-side, some client-side. Modularity > suffers greatly from trying to encode everything into one document. > This greatly raises the complexity of the task and makes it hard to > consider alternative proposals for various parts. > > Let's consider them separately: > > 1. what to build against what branches > > You said: > > * each has package.cfg or some alternative which specifies what is the > > EOL and/or for which releases to build > > Right now we use dist-git branches for this. Having a package.cfg > might sound easier , but the caveat is that building against different > branches requires tweaks sometimes. So we immediately hit a choice: do > we want a single branch and the %ifdef mess, or multiple branches and > the cherry picking troubles.
Yeah, but if there is clear map between branches & builds stored somewhere, it could be quite convenient for both. If you say "I want to have separate branches", then you still have f31/f32 while you should be able to say "I have 2.x branch and it is going to be built in all releases". That should be easily discoverable. > Right now, big disadvantage of multiple branches is that %changelogs > and Release bumps introduce trivial conflicts. I think we should > explore the proposals (incl. your and Neal's) to make this better. > Fixing this would be good also for normal packaging, because it'd remove > one or two more steps that need to be done every time. Yeah, at the current job I'm exploring something similar so when I get some time, I'll try to write proposal. > We should explore both directions, i.e. package.conf and easier branches. +1 > We should experiment with better/easier control of what to build > where. This would benefit normal packaging too. Right now our tooling > is just hard to use, with multiple commands in different tools needed > to start builds, schedule updates, control koji status, etc. We need > changes for "normal" packages as much as for "streams". +1 > 2. how to deliver rpms to users and how to let them control installed versions > > > How I would imagine having multiple streams: > > * rpms/nodejs has multiple branches - 10, 11, 12 > > * in all of them, nodejs.spec exists as-is without "mangled" name > > (just with different versions and such) > > * when builds are submitted, Koji automatically adds suffix to a main > > package and subpackages containing branch name > > * during the rpmbuild, extra provides are added (for the unversioned > > names and indication of a stream) and conflicts (possibly depending on > > some macro which user can disable) automatically > > Yes. The crucial part is that the rpms that are delivered are > just normal rpms. All the complications are kept on the build-side > and the client side doesn't need to care if this magic happened, the > same result could have been achieved with manually mangled spec file. > > > * If software needs to specify that it wants 9 ≤ nodejs ≤ 11 it can > > describe this in standard way (Requires: (nodejs >= 9 with nodejs <= > > 11)) and one will be picked up automatically > > ... > > * All standard Conflicts/Requires/Obsoletes/… will simply work > > Yes. More of "just normal packages". > > > * If user wants to switch from 9 to 10, he can run `dnf swap nodejs9 > > nodejs10` > > * If that requires some conflicting dependency to be switched, it will > > be switched automatically > > * Packages produced from nodejs.src have Provides: stream(nodejs) > > stream(nodejs:9) and Conflicts: stream(nodejs) so that it is > > explicitly not possible to install 9 and 10 at the same time > > * If desired, packages can depend on a specific stream via Requires: > > stream(nodejs:10) without actually depending on nodejs (this requires > > some small piece of code in libsolv) > > * In the very similar way, user can lock themselves to a specific > > stream by writing some conf file which DNF would read (if point above > > is implemented, about just tens of lines of the code in libdnf, or > > even libsolv can be teached about that as well) > > * DNF can be teached about these special things so that you can > > automatically swap conflicting dependencies, but lock yourself to some > > streams of a package > > ... and here I'm not sure. I'm not convinced we need to try to manage > all this like that. We already have packages which Conflict, or specify > conflicting Requires, and the user gives a dnf command, and a certain > transaction is prepared for them. The transaction will occasionally > specify a list of rpms to remove. If the user doesn't like this, they can > cancel and give a different command. This would imply that we have --allowerasing set by default, but we don't... And probably we should not since it is dangerous (a bit?). > Specifically, some streams might conflict, other might not, etc. > We could have a macro that helps to declare this in one line, but > let the maintainers of the rpms declare this as needed. > > > 3. [your 2.] Do we want to support buildtime-only packages? > > > I would rather generalize this category as "less-supported packages". > > I maintain 800+ Rust packages and very often I need to update them to > > an incompatible version. In Rawhide I just do it, update all dependent > > packages to use new version, and if I can't do that for some reason, > > create "compat" package. Obviously, all patches are sent to the > > upstream. > >[...] > > * After branching, there will be a script which untags all packages > > which should not be supported in the new release > > > > Upstreams are removing features, I need to deal with Obsoletes but I > > simply can't continuously add new Obsoletes into the > > fedora-obsolete-packages. And what for if they are used only during > > build of other, more important, packages? Why do I have to spend time > > with upgradepaths? I definitely want some mechanism which will tell to > > user that "THIS PACKAGE IS NOT FULLY SUPPORTED." > > Obviously, for packages which are used in runtime need a proper > > support as we do today for all packages to share work (that's the > > place where I agree with Kevin Kofler.) > > A different proposal: we add a new rpm flag (maybe as a special > Requires: line) that specifies that a package is only supported as a > build-time dependency. DNF could then gain a trivial patch that > it would ask the user "Do you want to install those unsupported packages? > If you do, please do not file bugs, because ...". > The advantages: > a) packages are available for local builds and downstream rebuilds > b) no problem with mock and koji being different > c) we avoid any perception of "welding down the engine hood". > > This would just acknowledge the truth that there are many packages > in Fedora that are not supported beyond an occasional version bump > and rebuild, giving better transparency for the users in general. +1 > 4. > > 3. Can we have different lifecycles for the software in Fedora? > a.k.a. Should we allow more in-release upgrades? > > > Right now we have to keep all versions of software which was there at > > GA point. And from the updates repo. We never remove packages > > entirely. That said, if package was there at GA time, it will have to > > be "supported" until the end of that Fedora release. > > I don't think we can (should?) do much in this regard. If we make > > packages to build from "stream" branch, we can put an information to > > that branch that this package should be built for all distributions > > (Fedora, EPEL) until this date. Or even store this info somewhere else > > like PDC (yes, I know we want to kill it). fedpkg build will take this > > into account and submit proper builds. And we can design some API in > > infra which would tell until when some particular stream of a package > > is supported. Much like it is done with Fedora releases & GNOME > > Software. > > We can (and do) encode this information directly in dist-git. > No need to provide a meta-mechanism. Even if package.conf becomes a > thing, this would contain this information. > > I think we should allow "rolling" versions more widely, like we do with > firefox, the kernel, spam blockers, and probably some other things. > As long some piece software is most backwards compatible, I think we'd > be better of abandoning the strict guidelines we have now. > This depends on each specific package though. Yeah, the problem I hit with Rust packages that they are not backwards-compatible. I, for sure, can create compat package but I would like to have way of telling user "if you have installed this package, don't expect working upgradepath or anything like that". > If this happens, it should be as a FPC guideline change, discussed > and specified independently of the tooling changes. If we detach > it from the technical side, answer the questions when and if at all > we allow this will be clearer. > > > 5. > > 4. Do we want to have some kind of "stream expansion" where software > > builds against all Pythons, Perls and whatnot? > > > > I think this should be conscious choice of a distribution, and in > > specific cases, maintainer. Using some examples from previous threads, > > why does bugzilla have to be built against 2 different versions of > > perl and users could choose? I think maintainer should choose one > > version of perl and let bugzilla use it. Being able to build > > combinations of software is definitely nice, but I don't think this > > should be standard practice. openSUSE does that with ruby and python > > (they build all modules automatically for all versions) and I like > > this. But having all packages built against all is just combinatoric > > explosion. Given how many updates have feedback in Bodhi, I'm pretty > > sure 99% of combination won't be tested (or even installed?) ever. > > Right now this would be prohibitively expensive. But maybe this could > become much easier if points 1-2 and 4 were better answered. > So I think we should shelve this question for now. > > > 6. How to deal with packages which need outdated BuildRequires > > This is another part of Modularity, and you proposal glosses over this. > But you said: > > * With side tags on demand it is now much easier > > ... and I fully agree here. There was > https://fedoraproject.org/wiki/Changes/OnDemandSideTags which plays > with this idea. I think side-tags could/should be the basis of a > solution. > > > > 5. Are we still trying to be a Linux distribution or we are just > > letting people to do whatever they want in our infrastructure? > > This question bothers me from time to time and I don't have answer. > > For example, Modularity is very flexible and I very often find people > > saying that you can expand this and that, but in Fedora we limit its > > usefulness. Do we actually need to develop something like this > > (knowing in advance that probably nobody outside of Fedora/RHEL will > > be using this software / technology)? Are we trying to create > > technologies which would be very extensible and used in other > > distributions instead of solving some specific problems? If former, > > why don't we talk to others about things in advance and not getting > > other people to work on these cool things? > > I don't have an answer for this. > > > * fedpkg will be teached about stream branches and will show which > > branches build where, so that maintainers can easily maintain them > > * Packaging guidelines should be adopted to accept conflicting > > packages and tooling should be improved to show the conflicts and how > > to resolve them > > * fedora-release will have Suggests: stream(nodejs:10) so that when > > user types dnf install nodejs, for example nodejs10 would be picked up > > (instead of nodejs12) > > * Some macros can be added to generate virtual package with "default > > stream", so that there will be actual nodejs package depending on > > nodejs12 if that would be default in given Fedora release > > * Going further, I would extend Koji with fedora plugin which would > > deal with the build submission instead of having this logic on clients > > so that all people get same result > > > > === > > > > Now let me quote Matthew Miller with the list of requirements and > > answer how we achieve them. > > > > 1. Users should have alternate streams of software available. > > I think this is achieved with what I have said above. We will ship > > nodejs9, nodejs10 at the same time. > > > > 2. Those alternate streams should be able to have different lifecycles. > > nodejs 9 and 10 can automatically build for all distros until EOL > > (which would be stored somewhere). EOL-date aware tooling (fedpkg) > > would not build for excluded versions of distros and build for distros > > where EOL of distro is further than EOL of component. > > > > 3. Packaging an individual stream for multiple outputs should be > > easier than before. > > As I wrote above, we will have one nodejs rpm repo with multiple > > branches (aka streams) and no name mangling needed. fedpkg build will > > spawn builds for all releases matching criteria (EOL date, excluded > > distributions). > > I think all this sounds reasonable. The main part is that it does not > require massive changes in the koji or dnf, and that it integrates in > a backwards-compatible way with what we have now. > > Let's split the problem into chunks and discuss solutions. +1, sadly we can't submit hackfest for devconf.cz, but let's probably meet there with some ideas prepared and discuss them? > > This is not the exact approach we have tried back in the days, but it > > is very similar to that. Unfortunately I did not save any documents > > from those times but Adam or Brian probably still can find them (since > > they are RH internal). > > Zbyszek _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
