Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 04:04, Richard W.M. Jones wrote: > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. > As again... Very strong no. git changelog and spec changelog have different purposes. For some cases it even makes sense to have a separate changelog for each section in the SPEC file, %build, %install etc. Best, Orcan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 12:36:23PM +0100, Pierre-Yves Chibon wrote: > > There are a few tricky things about this, but overall I think it's doable and > some of the tricky things may just be things we just have to accept as being > different from the current situation. > > We are playing a bit of a proof of concept of what a RPM changelog generated > from git logs could look like. The outcome of this is at: > https://pagure.io/Fedora-Infra/generate_changelog/tree/master > > Feel free to poke at it and see how it behaves. > > The script only considers: > - the last two years of commits > - commits touching either the spec file or patches (ending with .patch) > > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current > commit that should be ignored), as well as a way to say: [Replace XXX] to be > able to replace an existing commit message, but that's not there yet. > > One of the thing that may change is that we may end up with changelog entry > that > do not have a corresponding nvr at the end. > The python script above shows that, it'll only add the v-r in the changelog > when > either one of them changed, if they are the same, it doesn't specify them. > > Another aspect that is getting trickier is: > - update to 1.1-1 > > - fix missing BR > > - spec clean up > > 3 commits, can all be either on 3 different days but could also be on the same > day. Could be from 3 different people, but could also be from the same person. > So we could have 3 commits, on 1 day from 1 person, two of which would make > sense to group together (update to 1.1-1 and fix missing BR) and one that > shouldn't since the build succeeded before that and thus the -1 that goes out > to > the mirror will not have this clean up entry. > The current approach we took is: we have 3 entries in the changelog (and only > one has the v-r specified). It's not ideal, but we don't quite see how to > solve > this question yet. > If the "spec clean up" contains "[ignore]", that question is solved, but if we > replace "spec clean up" with "Drop sub-package foo" then we do not have a good > idea on how to consolidate the commit messages into a single changelog entry. > Suggestions welcome :) How about: * maintainer commits whatever in N commits. Commit messages can be just whatever maintainer/pr submittor likes. * Any PR's or commits that are 'notable', the commit should include a 'changenotes-hash' file. * When ready to do a build/update, all the changenotes-hash files are assembled into the changelog if they are part of a commit since the last build. Or I guess they could be consumed/removed after this and just any that exist are considered. > > > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > That was part of the proposal we debated last fall and there seemed to be much > more people against this than I thought there would be. > Maybe we could start with having an opt-in approach. I was for it before that last thread, then against it now. :) I think maintainers need to be able to do commits that they do not want to build against. We could look at tags for this info... ie, 'NOBUILD: true' or whatever. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
* Remi Collet [29/01/2020 15:18] : > > And I really prefer to see stabilization of our current tools and > infrastructure before breaking it again. Same here, to be honest. I'm far more held up in my work by the infrastructure breaking up from under me than the absence of any of the suggestions from this thread. Emmanuel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 2:11 PM Ken Dreyer wrote: > > On Wed, Jan 29, 2020 at 7:18 AM Remi Collet wrote: > > > There are different: > > > > * Changelog is for end user > > * Git log is for package maintainer > > I completely agree with this distinction. We're creating more "noise" > for end users if we end up adding all the "whoops" commits into the > %changelog. And asking maintainers to add "[skip]" strings or whatever > into the Git commit logs adds yet another magic/manual thing that > packagers must learn. > > This is why rdopkg writes the dist-git commit messages from the RPM > %changelog instead of the other way around. For example I can write or > modify my message in %changelog, run "rdopkg amend", and it > automatically updates my dist-git message to match what I wrote in > %changelog. I have not had to copy-and-paste information from > %changelog into the dist-git message in years. > fedpkg will do the same thing if you use "fedpkg ci -c" (or if you love long forms: "fedpkg commit --with-changelog"). > > > * committing to git should build the package > > > > > > Is there a reason why this wouldn't be the case? > > > > Yes, because I often commit various changse "before" the build > > (some being cherry-pick on other branch, some not) > > This one I happen to disagree with. I've set up a Jenkins job that > automatically builds on another (non-Fedora) Koji instance for a large > number of packages, and it is fantastic to never worry about > remembering to run "rpkg build" by hand any more. I trigger the builds > on the "push" bus messages though, instead of building every single > commit. If I'm making many small changes, then I'll commit them all to > Git locally, then push them all at once. When other developers don't > follow that practice of batching up their commit pushes, it does mean > that Koji ends up building a lot more frequently, and we trigger > composes more frequently throughout each day... but frankly that leads > to some other advantages, because it makes things more "continuous" > overall. > Exactly. And the vast majority of packagers do largely atomic changes, so it's really not going to be a huge deal if we get some rejected build requests because the NVR hasn't changed yet. > > IMHO, remember KISS, and don't try to add more magic to our tooling > > > > And I really prefer to see stabilization of our current tools and > > infrastructure before breaking it again. > > I completely agree with this Remi. We're still suffering from the loss > of pkgdb's features. I would prefer to fix what's broken before trying > to mess with changelogs and end up with something half-baked. > *sigh* -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 04:45:22PM +0200, Damian Ivanov wrote: > >That is looking for a boogeyman under the bed to blame something that > >has a long long history of not happening. Ever since OBS has been out, > >there has been a yearly 'why isn't Fedora moving to OBS' thread > > It has always been a bad management decision to not change. That's an odd view of management, and not one I recognize. I can assure you that Red Hat management don't care a fig about whether Fedora uses Koji or OBS or anything else. IBM acquiring Red Hat will not change that one iota. I doubt anyone in IBM's management even knows what Koji or OBS are, or would care if they did. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
Pierre-Yves Chibon writes: > On Tue, Jan 28, 2020 at 11:51:29PM +0100, Dan Čermák wrote: >> "Richard W.M. Jones" writes: >> >> > I always think that Fedora works fine if you maintain 1-5 packages. >> > It's possible to maintain 20 with a lot of work. And if you want to >> > maintain 100+ (things like the ocaml-* set that I help to maintain) >> > then you have to write your own automation. Could we do things >> > better? No one asked for them, but here are my ideas ... >> > >> > --- >> > >> > * kill the %changelog >> > >> > Please, let's kill it, and generate it from the git changelog. >> > I'm glad to see there's a proposal to do this. >> > >> > A general principle I'm following here is a packager should never >> > be asked to enter the same information twice. >> > >> > * committing to git should build the package >> > >> > Is there a reason why this wouldn't be the case? >> >> Not really no. I've been involved quite a bit in building packages on >> openSUSE's Buildservice and every commit there results in a rebuild. So it >> is doable, *but* OBS has the advantage that it discards all builds >> except for the most recent successful one, or it would have run out of >> storage ages ago. >> >> Although someone (Randy Barlow maybe?) had the idea to generate the >> changelog and to trigger builds from git tags instead of commits, which >> has a certain charm to it as well. If there was a choice whether to >> trigger builds from commits or tags and how to generate the %changelog, >> then I think that most use cases should be covered? > > The original idea of using git tag is Jeremy Cline's. > If we support: automatically building from commit or not, then I don't think > we > need to support building from git tag, using the current approach would work > just as well when you don't want to build from commit. Building from a git tag would have the advantage, that each build would show up on pagure under the "Releases" tab, albeit that is probably just a cosmetic advantage. > >> > * commit groups of packages together >> > >> > One reason why automatic commit -> build might not be desirable is if >> > you're trying to build a group of packages in a side tag. In my >> > opinion this means we should allow groups of packages to be committed >> > together. (Unfortunately because we chose to put every package in its >> > own git repo, the obvious way to do this can't work, but we could >> > still have a "ChangeId:" header in the commit message that ties >> > packages together). >> > >> > The packages should be built in build dependency order into a side >> > tag, and the side tag turned into an update, with no further >> > involvement from the packager unless something fails to build. >> > >> > This change on its own would solve the main problem that >> > affects people maintaining large sets of packages. >> >> How about something like `fedpkg add-to-side-tag` (yes the name needs >> work) that would work like this: >> >> $ fedpkg request-side-tag >> $ fedpkg add-to-side-tag $tag_name $pkgA $pkgB $pkgC $pkgGlob >> >> Now all git commits/tags pushed to $pkgA $pkgB $pkgC $pkgGlob result in >> them being built in the side tag $tag_name by default. Once the side tag >> is merged, you go back to building the "ordinary way". > > Isn't that just fedpkg chain-build --target=$tag_name ... ? Could be, I've never used it. > This is already existing and working, however, it requires the different > packages are up to date in git (ie: you've made and push the commit updating > the > spec file to the next release/version) which goes against the point above > about > automatically build on commit. Well, my idea was that you tell the infra: "hey, from this point onward (until the side tag is merged or I turn it off), I want every commit to be built in the side tag". signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 7:18 AM Remi Collet wrote: > There are different: > > * Changelog is for end user > * Git log is for package maintainer I completely agree with this distinction. We're creating more "noise" for end users if we end up adding all the "whoops" commits into the %changelog. And asking maintainers to add "[skip]" strings or whatever into the Git commit logs adds yet another magic/manual thing that packagers must learn. This is why rdopkg writes the dist-git commit messages from the RPM %changelog instead of the other way around. For example I can write or modify my message in %changelog, run "rdopkg amend", and it automatically updates my dist-git message to match what I wrote in %changelog. I have not had to copy-and-paste information from %changelog into the dist-git message in years. > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > Yes, because I often commit various changse "before" the build > (some being cherry-pick on other branch, some not) This one I happen to disagree with. I've set up a Jenkins job that automatically builds on another (non-Fedora) Koji instance for a large number of packages, and it is fantastic to never worry about remembering to run "rpkg build" by hand any more. I trigger the builds on the "push" bus messages though, instead of building every single commit. If I'm making many small changes, then I'll commit them all to Git locally, then push them all at once. When other developers don't follow that practice of batching up their commit pushes, it does mean that Koji ends up building a lot more frequently, and we trigger composes more frequently throughout each day... but frankly that leads to some other advantages, because it makes things more "continuous" overall. > IMHO, remember KISS, and don't try to add more magic to our tooling > > And I really prefer to see stabilization of our current tools and > infrastructure before breaking it again. I completely agree with this Remi. We're still suffering from the loss of pkgdb's features. I would prefer to fix what's broken before trying to mess with changelogs and end up with something half-baked. - Ken ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, 29 Jan 2020 at 09:46, Damian Ivanov wrote: > > >That is looking for a boogeyman under the bed to blame something that > >has a long long history of not happening. Ever since OBS has been out, > >there has been a yearly 'why isn't Fedora moving to OBS' thread > > It has always been a bad management decision to not change. > Ever since OBS has been out there has been a regular thread about this > (usually not started by me), because it's the correct thing to do in > a long term perspective for RH and the Linux community/ecosystem. > I think you are conflating management and leadership and assuming that they have ever been 1:1 in Fedora or Red Hat. For the most part managers are people who push paperwork, attend meetings and do whatever it is to allow the developers do what they feel is needed to get an OS out the door. They also are used as a convenient fall-guy when a developer doesn't really want to do something by saying management decided something. [There have been many a time the manager finds out about the decision at the same as the people being told a decision was made.] In the end, stuff gets done by leader's who commit to doing the lion's share of the work. Sometimes it is a manager, but mostly it is by various developers or maintainers who go ahead and make something happen either by skunk-works or putting in enough that it is done. The fact that OBS has not been chosen is more about no one wanting to do all the hardwork to make it happen versus management saying 'its from SuSE so we can't use it.' The way things have worked instead is that someone will say 'I want to show that this can be done with X' and they will go and do a lot of the work to make that happen. Mandrake(Mandriva? Mageia?) was started as someone wanting Red Hat Linux rebuilt with various Pentium+ flags versus i386. They also wanted to show other things that the Red Hat Linux developers were not comfortable with. Eventually that hard work was what got Red Hat Linux to start adding in Pentium+ items and other things. The same with various desktops.. in no case was it management saying 'dont do that' it was developers saying 'I have enough work already and this isn't going to make it less.' until it was less. OBS is fighting the 'Devil I know versus the devil I don't know' with every Fedora developer who knows fedpkg and the current tools. They may hate those tools, but they know they work AND they know that maintenance and keeping them working is somebody else's problem. If people want to set up an OBS build system, they need to become the people who that it is 'somebody else's problem' . They need to set up a buildsystem, build the OS, document how it works and be pretty honest about what doesn't. Then they will either get the old packagers to try it out or a bunch of new ones to take up the burden. -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 10:07:55AM -0500, Randy Barlow wrote: > On Wed, 2020-01-29 at 09:43 +, Richard W.M. Jones wrote: > > Also AIUI fedpkg chain-build doesn't work except in > > Rawhide, although I'm not sure why that is? > > It doesn't work in stable because you need to create buildroot > overrides for each dependency before you can proceed with building the > next package, and fedpkg chain-build doesn't create the BROs. > > However, I think the multi-package gating can do this now that it > exists, because AFAIK you should be able to chain-build in a side tag. > I have not tried this yet. It's less the gating itself than the work done to support mulit-builds updates gating, where we introduced on-demand side-tag which are available in rawhide as well as stable releases. Thus you can now use fedpkg chain-build and point it to your side-tag in any Fedora release. The only thing to be aware of with stable releases is that the side-tag will be removed once an update is created for it. So if you need to add a build to the update, you'll have to go back to the buildroot overrides. Pierre pgpLqlSaKv6Wm.pgp Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, 2020-01-29 at 09:43 +, Richard W.M. Jones wrote: > Also AIUI fedpkg chain-build doesn't work except in > Rawhide, although I'm not sure why that is? It doesn't work in stable because you need to create buildroot overrides for each dependency before you can proceed with building the next package, and fedpkg chain-build doesn't create the BROs. However, I think the multi-package gating can do this now that it exists, because AFAIK you should be able to chain-build in a side tag. I have not tried this yet. signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
>That is looking for a boogeyman under the bed to blame something that >has a long long history of not happening. Ever since OBS has been out, >there has been a yearly 'why isn't Fedora moving to OBS' thread It has always been a bad management decision to not change. Ever since OBS has been out there has been a regular thread about this (usually not started by me), because it's the correct thing to do in a long term perspective for RH and the Linux community/ecosystem. On Wed, Jan 29, 2020 at 4:26 PM Stephen John Smoogen wrote: > > On Wed, 29 Jan 2020 at 06:10, Damian Ivanov wrote: > > > > Maybe now that RH is part of IBM they have changed their short sighted view > > of not collaborating on a better build system like OBS. > > That is looking for a boogeyman under the bed to blame something that > has a long long history of not happening. Ever since OBS has been out, > there has been a yearly 'why isn't Fedora moving to OBS' thread > somewhere and there have been long lists of 'why we aren't' over and > over again. It has nothing to do with being part of IBM and a lot > about > > 1. Moving to OBS would have required rewriting a lot of spec files to > match what OBS looked for (this has gotten better over time but some > rewrite would be needed). > 2. Different needs of the build system. The koji system has an > accounting logic which is tied into giving certain assurances about > when, where, who, and how software was compiled. OBS has different > accounting logic. You either have to spend a long time trying to > figure out how to convert the two, or dump one and start over with > OBS. Then you are pretty much starting a new operating system. > 3. Changes occur by the people who put the work in. The people who are > spending 60-80 hours a week trying to keep Fedora compiling and > running know Koji. If another group wants to set up and run OBS to > make their own OS, they need to show up with the hardware, storage, > and time to do so because there is no slack with current people, > hardware, storage to stop running Koji. > 4. Creative differences which are a fine line from Not Invented Here > but still exist. > > We tend to focus on 4 a lot in these conversations because it makes > for the most 'fun' drama. We tend to ignore 2 and 3 because they are > 'someone else's problem' that could become our own if we let it. The > problem is that 4 is a molehill and 1,2,3 are mountains of different > heights. > > > > As I recall back than it was already able to bootstrap on centos and fedora > > and build packages and the only argument against it was legacy support with > > mock / koji which provide a fraction of functions that OBS does (mostly > > things that make it easy to get new contributers like a web interface to > > edit spec files and webhooks not to mention that these Mass rebuilt emails > > could be a thing of the past). > > > > On Wed, 29 Jan 2020, 12:17 Christophe de Dinechin, > > wrote: > >> > >> > >> > >> > On 29 Jan 2020, at 00:26, Robert-André Mauchin wrote: > >> > > >> > On Tuesday, 28 January 2020 10:03:09 CET Richard W.M. Jones wrote: > >> >> * committing to git should build the package > >> >> > >> >> Is there a reason why this wouldn't be the case? > >> > > >> > Please no. Sometimes you just fix a typo or add a comment and there's no > >> > need > >> > to rebuild until a next release. > >> > >> This “sometimes" is the rare case. So having a “push —nobuild” would > >> address that need. > >> > >> > > >> > ___ > >> > devel mailing list -- devel@lists.fedoraproject.org > >> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > >> > Fedora Code of Conduct: > >> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> > List Archives: > >> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > >> ___ > >> devel mailing list -- devel@lists.fedoraproject.org > >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org > >> Fedora Code of Conduct: > >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > >> List Archives: > >> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > > > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > > > > -- > Stephen J Smoogen. > ___ > devel mailing list --
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 4:05 PM Pierre-Yves Chibon wrote: > And ... you lost me right there... > Pierre That's too bad. Even If it's sounds harsh it's the reality. It has been discussed before and there was no technical reason not to. Just someone going for a short term solution. Maybe it is time for someone who works at redhat to stand up going to the manager dude or dudeline (which is a support role just like HR btw.) and explain why the current solution is/has/will(be-en) inferior,and is hurtful to the Linux ecosystem as a whole and a very bad long term solution. The only thing that is at the same time a plus and a minus for a rh manager would be that other distributions will benefit as well (and maybe giving credit to someone else as it was not developed in house). Anyone who spends objectively more than 30 minutes with the two systems will understand and be able to confirm. OBS is an example of KISS. The web interface attracting contributors. Different repo's for different versions and build targets like Qt could use currently. Compare COPR with OBS. Some people may not like the sound of that but I believe that every word of that is true. Br, Damian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, 29 Jan 2020 at 06:10, Damian Ivanov wrote: > > Maybe now that RH is part of IBM they have changed their short sighted view > of not collaborating on a better build system like OBS. That is looking for a boogeyman under the bed to blame something that has a long long history of not happening. Ever since OBS has been out, there has been a yearly 'why isn't Fedora moving to OBS' thread somewhere and there have been long lists of 'why we aren't' over and over again. It has nothing to do with being part of IBM and a lot about 1. Moving to OBS would have required rewriting a lot of spec files to match what OBS looked for (this has gotten better over time but some rewrite would be needed). 2. Different needs of the build system. The koji system has an accounting logic which is tied into giving certain assurances about when, where, who, and how software was compiled. OBS has different accounting logic. You either have to spend a long time trying to figure out how to convert the two, or dump one and start over with OBS. Then you are pretty much starting a new operating system. 3. Changes occur by the people who put the work in. The people who are spending 60-80 hours a week trying to keep Fedora compiling and running know Koji. If another group wants to set up and run OBS to make their own OS, they need to show up with the hardware, storage, and time to do so because there is no slack with current people, hardware, storage to stop running Koji. 4. Creative differences which are a fine line from Not Invented Here but still exist. We tend to focus on 4 a lot in these conversations because it makes for the most 'fun' drama. We tend to ignore 2 and 3 because they are 'someone else's problem' that could become our own if we let it. The problem is that 4 is a molehill and 1,2,3 are mountains of different heights. > As I recall back than it was already able to bootstrap on centos and fedora > and build packages and the only argument against it was legacy support with > mock / koji which provide a fraction of functions that OBS does (mostly > things that make it easy to get new contributers like a web interface to edit > spec files and webhooks not to mention that these Mass rebuilt emails could > be a thing of the past). > > On Wed, 29 Jan 2020, 12:17 Christophe de Dinechin, > wrote: >> >> >> >> > On 29 Jan 2020, at 00:26, Robert-André Mauchin wrote: >> > >> > On Tuesday, 28 January 2020 10:03:09 CET Richard W.M. Jones wrote: >> >> * committing to git should build the package >> >> >> >> Is there a reason why this wouldn't be the case? >> > >> > Please no. Sometimes you just fix a typo or add a comment and there's no >> > need >> > to rebuild until a next release. >> >> This “sometimes" is the rare case. So having a “push —nobuild” would address >> that need. >> >> > >> > ___ >> > devel mailing list -- devel@lists.fedoraproject.org >> > To unsubscribe send an email to devel-le...@lists.fedoraproject.org >> > Fedora Code of Conduct: >> > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> > List Archives: >> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org >> ___ >> devel mailing list -- devel@lists.fedoraproject.org >> To unsubscribe send an email to devel-le...@lists.fedoraproject.org >> Fedora Code of Conduct: >> https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: >> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
Le 28/01/2020 à 10:03, Richard W.M. Jones a écrit : > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... I'm probably one of the 100+ packages maintainers... > --- > > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. Please no. Yes, I probably the only one to not sync all branches to have different changelog, and I also don't want the mass-rebuild stuff there. Simple proposal, we have rpmdev-bumpspec, improve it to retrieve information from git log is you want to use it. > A general principle I'm following here is a packager should never > be asked to enter the same information twice. There are different: * Changelog is for end user * Git log is for package maintainer > * committing to git should build the package > > Is there a reason why this wouldn't be the case? Yes, because I often commit various changse "before" the build (some being cherry-pick on other branch, some not) IMHO, remember KISS, and don't try to add more magic to our tooling And I really prefer to see stabilization of our current tools and infrastructure before breaking it again. Remi ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 01:09:04PM +0200, Damian Ivanov wrote: >Maybe now that RH is part of IBM they have changed their short sighted >view of not collaborating on a better build system like OBS. As I recall And ... you lost me right there... Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
Maybe now that RH is part of IBM they have changed their short sighted view of not collaborating on a better build system like OBS. As I recall back than it was already able to bootstrap on centos and fedora and build packages and the only argument against it was legacy support with mock / koji which provide a fraction of functions that OBS does (mostly things that make it easy to get new contributers like a web interface to edit spec files and webhooks not to mention that these Mass rebuilt emails could be a thing of the past). On Wed, 29 Jan 2020, 12:17 Christophe de Dinechin, wrote: > > > > On 29 Jan 2020, at 00:26, Robert-André Mauchin > wrote: > > > > On Tuesday, 28 January 2020 10:03:09 CET Richard W.M. Jones wrote: > >> * committing to git should build the package > >> > >> Is there a reason why this wouldn't be the case? > > > > Please no. Sometimes you just fix a typo or add a comment and there's no > need > > to rebuild until a next release. > > This “sometimes" is the rare case. So having a “push —nobuild” would > address that need. > > > > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
> On 29 Jan 2020, at 00:26, Robert-André Mauchin wrote: > > On Tuesday, 28 January 2020 10:03:09 CET Richard W.M. Jones wrote: >> * committing to git should build the package >> >> Is there a reason why this wouldn't be the case? > > Please no. Sometimes you just fix a typo or add a comment and there's no need > to rebuild until a next release. This “sometimes" is the rare case. So having a “push —nobuild” would address that need. > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
> On 28 Jan 2020, at 10:03, Richard W.M. Jones wrote: > > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... > > --- > > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. > > A general principle I'm following here is a packager should never > be asked to enter the same information twice. Right now, you can do it the other way round, i.e. update the change log and fedpkg commit -c. But it is not very logical, and I agree it would be more logical to deduce the change log from the commit messages. > > * committing to git should build the package > > Is there a reason why this wouldn't be the case? > > * commit groups of packages together > > One reason why automatic commit -> build might not be desirable is if > you're trying to build a group of packages in a side tag. In my > opinion this means we should allow groups of packages to be committed > together. (Unfortunately because we chose to put every package in its > own git repo, the obvious way to do this can't work, but we could > still have a "ChangeId:" header in the commit message that ties > packages together). > > The packages should be built in build dependency order into a side > tag, and the side tag turned into an update, with no further > involvement from the packager unless something fails to build. > > This change on its own would solve the main problem that > affects people maintaining large sets of packages. > > * “Fixes:” etc headers in git > > RHEL already does this. It should be possible to add special headers > to the commit message to automatically close bugs, ie: > > $ git commit -m "ocaml: Update to version 4.10.0 > Fixes: RHBZ#12345" > > Note the build, update and bug closing would happen completely > automatically, assuming there was no build or test failure. > > * CVE bugs should autoclose when a package is rebased > > Fabiano built the mingw-openssl package recently, but there are still > a load of open CVE bugs against this package referring to the older > version. These should be closed automatically. I think this will > require collecting the version of the package that fixes a CVE and > recording that in Bugzilla (or in the package itself in some standard > way). > > * create streams of packages automatically depending on quality scores > > We know a lot about packages such as: > > - How many bugs are opened against them? > - What's the average time between a bug being filed and fixed? > - Do they get regularly updated? > - Do they pass or fail CI tests? > - How many rpmlint / fedora-packager problems do they have? > > Why don't we use that data to build streams of high quality and lesser > quality packages? Allow the end users to choose whether they want the > best curated packages only, or they're prepared to accept the risk of > taking a package with lots of bugs or CVEs (this is assuming the > previous point is fixed, so these are real CVEs, not irrelevant bugs). > > If you want to go even further with this idea, then it could even be > possible we allow packages into Fedora without any review. They would > start in the outermost stream in a "there be dragons" repository that > only the foolhardy would enable, but as their quality improved they > would *automatically* migrate into the mainstream. > > --- > > Rich. > > -- > Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones > Read my programming and virtualization blog: http://rwmj.wordpress.com > Fedora Windows cross-compiler. Compile Windows programs, test, and > build Windows installers. Over 100 libraries supported. > http://fedoraproject.org/wiki/MinGW > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
> On 28 Jan 2020, at 11:32, Guido Aulisi wrote: > > Il giorno mar 28 gen 2020 alle ore 10:04 Richard W.M. Jones > ha scritto: >> >> I always think that Fedora works fine if you maintain 1-5 packages. >> It's possible to maintain 20 with a lot of work. And if you want to >> maintain 100+ (things like the ocaml-* set that I help to maintain) >> then you have to write your own automation. Could we do things >> better? No one asked for them, but here are my ideas ... >> >> --- >> >> * kill the %changelog >> >> Please, let's kill it, and generate it from the git changelog. >> I'm glad to see there's a proposal to do this. >> >> A general principle I'm following here is a packager should never >> be asked to enter the same information twice. >> >> * committing to git should build the package >> >> Is there a reason why this wouldn't be the case? > > Sometimes you only add comments to the spec file and a rebuild is not needed. You don’t know that. What if you accidentally insert a newline in your comment? > > Ciao > Guido > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 10:04:32AM +0100, Pierre-Yves Chibon wrote: > On Tue, Jan 28, 2020 at 11:51:29PM +0100, Dan Čermák wrote: > > "Richard W.M. Jones" writes: > > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > > It's possible to maintain 20 with a lot of work. And if you want to > > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > > then you have to write your own automation. Could we do things > > > better? No one asked for them, but here are my ideas ... > > > > > > --- > > > > > > * kill the %changelog > > > > > > Please, let's kill it, and generate it from the git changelog. > > > I'm glad to see there's a proposal to do this. > > > > > > A general principle I'm following here is a packager should never > > > be asked to enter the same information twice. > > > > > > * committing to git should build the package > > > > > > Is there a reason why this wouldn't be the case? > > > > Not really no. I've been involved quite a bit in building packages on > > openSUSE's Buildservice and every commit there results in a rebuild. So it > > is doable, *but* OBS has the advantage that it discards all builds > > except for the most recent successful one, or it would have run out of > > storage ages ago. > > > > Although someone (Randy Barlow maybe?) had the idea to generate the > > changelog and to trigger builds from git tags instead of commits, which > > has a certain charm to it as well. If there was a choice whether to > > trigger builds from commits or tags and how to generate the %changelog, > > then I think that most use cases should be covered? > > The original idea of using git tag is Jeremy Cline's. > If we support: automatically building from commit or not, then I don't think > we > need to support building from git tag, using the current approach would work > just as well when you don't want to build from commit. > > > > * commit groups of packages together > > > > > > One reason why automatic commit -> build might not be desirable is if > > > you're trying to build a group of packages in a side tag. In my > > > opinion this means we should allow groups of packages to be committed > > > together. (Unfortunately because we chose to put every package in its > > > own git repo, the obvious way to do this can't work, but we could > > > still have a "ChangeId:" header in the commit message that ties > > > packages together). > > > > > > The packages should be built in build dependency order into a side > > > tag, and the side tag turned into an update, with no further > > > involvement from the packager unless something fails to build. > > > > > > This change on its own would solve the main problem that > > > affects people maintaining large sets of packages. > > > > How about something like `fedpkg add-to-side-tag` (yes the name needs > > work) that would work like this: > > > > $ fedpkg request-side-tag > > $ fedpkg add-to-side-tag $tag_name $pkgA $pkgB $pkgC $pkgGlob > > > > Now all git commits/tags pushed to $pkgA $pkgB $pkgC $pkgGlob result in > > them being built in the side tag $tag_name by default. Once the side tag > > is merged, you go back to building the "ordinary way". > > Isn't that just fedpkg chain-build --target=$tag_name ... ? > This is already existing and working, however, it requires the different > packages are up to date in git (ie: you've made and push the commit updating > the > spec file to the next release/version) which goes against the point above > about > automatically build on commit. Needs to be something which builds in BuildRequire order to solve the actual problem. Also AIUI fedpkg chain-build doesn't work except in Rawhide, although I'm not sure why that is? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Wed, Jan 29, 2020 at 09:26:43AM +, Richard W.M. Jones wrote: > On Tue, Jan 28, 2020 at 02:06:40PM -0500, Stephen John Smoogen wrote: > > My main concern is that we have been coming up with 'standard' > > proposals for 20 years and we can't seem to get more than any 4 > > maintainers to agree to what that means... even if they do the same > > work in Debian/SuSE/Arch etc. Too many idiosyncratic techniques which > > they use to keep themselves sane or working in whatever environments > > they have. > > > > At this point, I will take whatever we can standardize on even if it > > is clay tablets mailed to Babylon (ok maybe something a little less > > archaic) > > Likely we can get the GNU project involved here. > > A simple "METADATA" file in the top level of each project > containing the information we need in a format to be decided. > > If you think about it, we cannot even get the *name or version* of a > project mechanically at the moment, which is incredible really. AppStream metadata is probably the best upstream adopted metadata system that exists right now: https://www.freedesktop.org/software/appstream/docs/ It has wide adoption amongst desktop applications, since it is used to populate the software install tools used by various desktop environments, but it isn't restricted to just desktop apps. One caveat is that often the metadata file shipped in the tarball is not complete & is expanded by the build system. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 11:51:29PM +0100, Dan Čermák wrote: > "Richard W.M. Jones" writes: > > * CVE bugs should autoclose when a package is rebased > > I don't think this is a good idea as you should actually check that this > update fixes the CVE. If we collect the data that version X fixes CVE Y, then the bug can be closed automatically when version >= X is built, and it's entirely as safe as today. We don't tell packagers they must try to actively exploit their new build to ensure the exploit has been fixed (at least I hope we don't ...) Collecting that data should be possible. I have suggested that we work with GNU and other Linux distros to start encouraging upstreams to provide this data mechanically. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 02:06:40PM -0500, Stephen John Smoogen wrote: > My main concern is that we have been coming up with 'standard' > proposals for 20 years and we can't seem to get more than any 4 > maintainers to agree to what that means... even if they do the same > work in Debian/SuSE/Arch etc. Too many idiosyncratic techniques which > they use to keep themselves sane or working in whatever environments > they have. > > At this point, I will take whatever we can standardize on even if it > is clay tablets mailed to Babylon (ok maybe something a little less > archaic) Likely we can get the GNU project involved here. A simple "METADATA" file in the top level of each project containing the information we need in a format to be decided. If you think about it, we cannot even get the *name or version* of a project mechanically at the moment, which is incredible really. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 04:10:17PM -0500, Robbie Harwood wrote: > Stephen John Smoogen writes: > > > On Tue, 28 Jan 2020 at 13:01, Robbie Harwood wrote: > >> > >> "Richard W.M. Jones" writes: > >> > >> > I always think that Fedora works fine if you maintain 1-5 packages. > >> > It's possible to maintain 20 with a lot of work. And if you want to > >> > maintain 100+ (things like the ocaml-* set that I help to maintain) > >> > then you have to write your own automation. Could we do things > >> > better? No one asked for them, but here are my ideas ... > >> > > >> > --- > >> > > >> > * CVE bugs should autoclose when a package is rebased > >> > > >> > Fabiano built the mingw-openssl package recently, but there are still > >> > a load of open CVE bugs against this package referring to the older > >> > version. These should be closed automatically. I think this will > >> > require collecting the version of the package that fixes a CVE and > >> > recording that in Bugzilla (or in the package itself in some standard > >> > way). > >> > >> This is an interesting idea, and I appreciate you're considering ways to > >> resolve this problem. However, I'm concerned that this will lead to > >> maintainers not actually checking whether a version fixes an issue - > >> since we don't have automatic verification (or even usually manual > >> verification) of security fixes, that wouldn't get caught. > >> > > > > You are assuming that maintainers actually check to see if a version > > fixes an issue already. If a packager has 100's or 1000's of > > packages.. there is no way they will have done so except on a 1 in a > > million case set. I think if are going to aim that a packager can > > 'maintain' hundreds or thousands of packages that we also assume that > > this security is not going to be checked by the maintainer. If it > > needs to be checked it will need to be 'outsourced' to some group who > > can do so. > > Per Package Maintainer Responsibilities [1], maintainers are expected to > "deal with reported bugs in a timely manner" and reach out if they > cannot handle the load. I think it's reasonable to expect maintainers > to be close to compliance with the policy they agreed to when becoming > maintainers :) > > Personally, I think if you have enough packages that you cannot actually > triage your own bugtracker, you have too many packages. I don't think > it's at all reasonable for one person to be responsible for "100's or > 1000s of packages", and I think not knowing whether security issues are > or are not fixed in a given version of them is a perfect illustration of > why that doesn't work. > > What I would like us to do is move away from needing to do that. > Whether that involves more SIG-like maintainer groups, or a different > format, I don't know; but one thing we do need is more monitoring of the > security issues than we have right now. I fundamentally disagree. These things are entirely possible if there is more automation, and I've shown how I think it could be done. AND we need to do this too, increasing packager productivity is the number 1 issue with Fedora packaging and has knock-on benefits everywhere. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 11:51:29PM +0100, Dan Čermák wrote: > "Richard W.M. Jones" writes: > > > I always think that Fedora works fine if you maintain 1-5 packages. > > It's possible to maintain 20 with a lot of work. And if you want to > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > then you have to write your own automation. Could we do things > > better? No one asked for them, but here are my ideas ... > > > > --- > > > > * kill the %changelog > > > > Please, let's kill it, and generate it from the git changelog. > > I'm glad to see there's a proposal to do this. > > > > A general principle I'm following here is a packager should never > > be asked to enter the same information twice. > > > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > Not really no. I've been involved quite a bit in building packages on > openSUSE's Buildservice and every commit there results in a rebuild. So it > is doable, *but* OBS has the advantage that it discards all builds > except for the most recent successful one, or it would have run out of > storage ages ago. > > Although someone (Randy Barlow maybe?) had the idea to generate the > changelog and to trigger builds from git tags instead of commits, which > has a certain charm to it as well. If there was a choice whether to > trigger builds from commits or tags and how to generate the %changelog, > then I think that most use cases should be covered? The original idea of using git tag is Jeremy Cline's. If we support: automatically building from commit or not, then I don't think we need to support building from git tag, using the current approach would work just as well when you don't want to build from commit. > > * commit groups of packages together > > > > One reason why automatic commit -> build might not be desirable is if > > you're trying to build a group of packages in a side tag. In my > > opinion this means we should allow groups of packages to be committed > > together. (Unfortunately because we chose to put every package in its > > own git repo, the obvious way to do this can't work, but we could > > still have a "ChangeId:" header in the commit message that ties > > packages together). > > > > The packages should be built in build dependency order into a side > > tag, and the side tag turned into an update, with no further > > involvement from the packager unless something fails to build. > > > > This change on its own would solve the main problem that > > affects people maintaining large sets of packages. > > How about something like `fedpkg add-to-side-tag` (yes the name needs > work) that would work like this: > > $ fedpkg request-side-tag > $ fedpkg add-to-side-tag $tag_name $pkgA $pkgB $pkgC $pkgGlob > > Now all git commits/tags pushed to $pkgA $pkgB $pkgC $pkgGlob result in > them being built in the side tag $tag_name by default. Once the side tag > is merged, you go back to building the "ordinary way". Isn't that just fedpkg chain-build --target=$tag_name ... ? This is already existing and working, however, it requires the different packages are up to date in git (ie: you've made and push the commit updating the spec file to the next release/version) which goes against the point above about automatically build on commit. Pierre signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tuesday, 28 January 2020 10:03:09 CET Richard W.M. Jones wrote: > * committing to git should build the package > > Is there a reason why this wouldn't be the case? Please no. Sometimes you just fix a typo or add a comment and there's no need to rebuild until a next release. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
"Richard W.M. Jones" writes: > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... > > --- > > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. > > A general principle I'm following here is a packager should never > be asked to enter the same information twice. > > * committing to git should build the package > > Is there a reason why this wouldn't be the case? Not really no. I've been involved quite a bit in building packages on openSUSE's Buildservice and every commit there results in a rebuild. So it is doable, *but* OBS has the advantage that it discards all builds except for the most recent successful one, or it would have run out of storage ages ago. Although someone (Randy Barlow maybe?) had the idea to generate the changelog and to trigger builds from git tags instead of commits, which has a certain charm to it as well. If there was a choice whether to trigger builds from commits or tags and how to generate the %changelog, then I think that most use cases should be covered? > > * commit groups of packages together > > One reason why automatic commit -> build might not be desirable is if > you're trying to build a group of packages in a side tag. In my > opinion this means we should allow groups of packages to be committed > together. (Unfortunately because we chose to put every package in its > own git repo, the obvious way to do this can't work, but we could > still have a "ChangeId:" header in the commit message that ties > packages together). > > The packages should be built in build dependency order into a side > tag, and the side tag turned into an update, with no further > involvement from the packager unless something fails to build. > > This change on its own would solve the main problem that > affects people maintaining large sets of packages. How about something like `fedpkg add-to-side-tag` (yes the name needs work) that would work like this: $ fedpkg request-side-tag $ fedpkg add-to-side-tag $tag_name $pkgA $pkgB $pkgC $pkgGlob Now all git commits/tags pushed to $pkgA $pkgB $pkgC $pkgGlob result in them being built in the side tag $tag_name by default. Once the side tag is merged, you go back to building the "ordinary way". > > * “Fixes:” etc headers in git > > RHEL already does this. It should be possible to add special headers > to the commit message to automatically close bugs, ie: > > $ git commit -m "ocaml: Update to version 4.10.0 > Fixes: RHBZ#12345" > > Note the build, update and bug closing would happen completely > automatically, assuming there was no build or test failure. +1 on this one. > > * CVE bugs should autoclose when a package is rebased I don't think this is a good idea as you should actually check that this update fixes the CVE. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
Stephen John Smoogen writes: > On Tue, 28 Jan 2020 at 13:01, Robbie Harwood wrote: >> >> "Richard W.M. Jones" writes: >> >> > I always think that Fedora works fine if you maintain 1-5 packages. >> > It's possible to maintain 20 with a lot of work. And if you want to >> > maintain 100+ (things like the ocaml-* set that I help to maintain) >> > then you have to write your own automation. Could we do things >> > better? No one asked for them, but here are my ideas ... >> > >> > --- >> > >> > * CVE bugs should autoclose when a package is rebased >> > >> > Fabiano built the mingw-openssl package recently, but there are still >> > a load of open CVE bugs against this package referring to the older >> > version. These should be closed automatically. I think this will >> > require collecting the version of the package that fixes a CVE and >> > recording that in Bugzilla (or in the package itself in some standard >> > way). >> >> This is an interesting idea, and I appreciate you're considering ways to >> resolve this problem. However, I'm concerned that this will lead to >> maintainers not actually checking whether a version fixes an issue - >> since we don't have automatic verification (or even usually manual >> verification) of security fixes, that wouldn't get caught. >> > > You are assuming that maintainers actually check to see if a version > fixes an issue already. If a packager has 100's or 1000's of > packages.. there is no way they will have done so except on a 1 in a > million case set. I think if are going to aim that a packager can > 'maintain' hundreds or thousands of packages that we also assume that > this security is not going to be checked by the maintainer. If it > needs to be checked it will need to be 'outsourced' to some group who > can do so. Per Package Maintainer Responsibilities [1], maintainers are expected to "deal with reported bugs in a timely manner" and reach out if they cannot handle the load. I think it's reasonable to expect maintainers to be close to compliance with the policy they agreed to when becoming maintainers :) Personally, I think if you have enough packages that you cannot actually triage your own bugtracker, you have too many packages. I don't think it's at all reasonable for one person to be responsible for "100's or 1000s of packages", and I think not knowing whether security issues are or are not fixed in a given version of them is a perfect illustration of why that doesn't work. What I would like us to do is move away from needing to do that. Whether that involves more SIG-like maintainer groups, or a different format, I don't know; but one thing we do need is more monitoring of the security issues than we have right now. Thanks, --Robbie 1: https://docs.fedoraproject.org/en-US/fesco/Package_maintainer_responsibilities/ signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 2:39 PM Randy Barlow wrote: > > On Tue, 2020-01-28 at 09:03 +, Richard W.M. Jones wrote: > > If you want to go even further with this idea, then it could even be > > possible we allow packages into Fedora without any review. They > > would > > start in the outermost stream in a "there be dragons" repository that > > only the foolhardy would enable, but as their quality improved they > > would *automatically* migrate into the mainstream. > > We would need to at least have license review. Though automation can > help with licensing, there are weird things sometimes that only a human > could detect, like this[0]: > > https://github.com/szymach/c-pchart/issues/35 > > I do think we could automate a lot of the other elements of review > though, and I agree that it would be helpful. > > Having a bot at least check for the obvious licence problems would > still be helpful, but a bot that approves a package license still needs > to be double checked by a human, in my opinion. The bot would be > helpful in catching negatives (no license, or unacceptable license, > etc.) > This is something that I'm working on trying to bring over from the openSUSE community to Fedora. They've written a web app[1] that actually does this and they wired it into the contribution processes built into their build system so that they can get these done reasonably well and have a good understanding of the license makeup of their distribution of packages. There's a few things left and I'm hoping to try to spin up a test instance running on Fedora to see how it works. [1]: https://github.com/openSUSE/cavil -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 2020-01-28 at 09:03 +, Richard W.M. Jones wrote: > If you want to go even further with this idea, then it could even be > possible we allow packages into Fedora without any review. They > would > start in the outermost stream in a "there be dragons" repository that > only the foolhardy would enable, but as their quality improved they > would *automatically* migrate into the mainstream. We would need to at least have license review. Though automation can help with licensing, there are weird things sometimes that only a human could detect, like this[0]: https://github.com/szymach/c-pchart/issues/35 I do think we could automate a lot of the other elements of review though, and I agree that it would be helpful. Having a bot at least check for the obvious licence problems would still be helpful, but a bot that approves a package license still needs to be double checked by a human, in my opinion. The bot would be helpful in catching negatives (no license, or unacceptable license, etc.) [0] Thanks to Remi for catching it in https://bugzilla.redhat.com/show_bug.cgi?id=1425275 - I hadn't even noticed it myself! signature.asc Description: This is a digitally signed message part ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 14:01, Emmanuel Seyman wrote: > > * Stephen John Smoogen [28/01/2020 13:08] : > > > > You are assuming that maintainers actually check to see if a version > > fixes an issue already. If a packager has 100's or 1000's of > > packages.. there is no way they will have done so except on a 1 in a > > million case set. I think if are going to aim that a packager can > > 'maintain' hundreds or thousands of packages that we also assume that > > this security is not going to be checked by the maintainer. > > I really don't think I'm *that* special a case so I'ld prefer you check > that this is actually true rather than assume something wrong. > I realized after I sent it that someone would assume I was personally talking about their work. It is not what I meant and I apologize for my imprecise language. The issue I was aiming at that we currently have a high probability that we are missing CVE's just from the mass of packages and the mass of CVE's out there. My assumption that most packagers don't have the time to set up test cases for each CVE to confirm it is fixed comes from listening to 15 years of #fedora-devel, bugzilla and mailing lists where the fix is 'moved to latest upstream to fix CVE-123456' and then followups of 'updated to newer version to get right fix.. etc etc'. The nature of the work is that this is happening and will continue to happen whether or not we automate parts to help handle more packages per developer. Even if the packager is checking, mistakes will happen.. you may not replicate the CVE environment correctly.. it may be found that a cornercase still occurs... etc etc. The time to commit and build is short also for a lot of people and so the probability of it actually happening all the time is remote. That said, giving it an actual number (1 in a million) was wrong. -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 13:49, Richard W.M. Jones wrote: > > On Tue, Jan 28, 2020 at 01:08:11PM -0500, Stephen John Smoogen wrote: > > On Tue, 28 Jan 2020 at 13:01, Robbie Harwood wrote: > > > > > > "Richard W.M. Jones" writes: > > > > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > > > It's possible to maintain 20 with a lot of work. And if you want to > > > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > > > then you have to write your own automation. Could we do things > > > > better? No one asked for them, but here are my ideas ... > > > > > > > > --- > > > > > > > > * CVE bugs should autoclose when a package is rebased > > > > > > > > Fabiano built the mingw-openssl package recently, but there are still > > > > a load of open CVE bugs against this package referring to the older > > > > version. These should be closed automatically. I think this will > > > > require collecting the version of the package that fixes a CVE and > > > > recording that in Bugzilla (or in the package itself in some standard > > > > way). > > > > > > This is an interesting idea, and I appreciate you're considering ways to > > > resolve this problem. However, I'm concerned that this will lead to > > > maintainers not actually checking whether a version fixes an issue - > > > since we don't have automatic verification (or even usually manual > > > verification) of security fixes, that wouldn't get caught. > > > > > > > You are assuming that maintainers actually check to see if a version > > fixes an issue already. If a packager has 100's or 1000's of > > packages.. there is no way they will have done so except on a 1 in a > > million case set. I think if are going to aim that a packager can > > 'maintain' hundreds or thousands of packages that we also assume that > > this security is not going to be checked by the maintainer. If it > > needs to be checked it will need to be 'outsourced' to some group who > > can do so. > > Also the bit where I said > "(or in the package itself in some standard way)." > > Of course that standard way doesn't exist (or does it?) but we could > surely encourage upstreams to provide data that we need in a standard > format, especially if we coorperate with Debian, SUSE, Arch and others. > > For example, for my upstream projects I write release notes (!= the > changelog) but they're text files and not really in any standard > location or format. Also we have security bug pages but again not in > a standard way. > > If it was standardized we could suck that data straight into Fedora > and no packager, whether maintaining 1 or 100 packages, would have to > be troubled by this. > My main concern is that we have been coming up with 'standard' proposals for 20 years and we can't seem to get more than any 4 maintainers to agree to what that means... even if they do the same work in Debian/SuSE/Arch etc. Too many idiosyncratic techniques which they use to keep themselves sane or working in whatever environments they have. At this point, I will take whatever we can standardize on even if it is clay tablets mailed to Babylon (ok maybe something a little less archaic) -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
* Stephen John Smoogen [28/01/2020 13:08] : > > You are assuming that maintainers actually check to see if a version > fixes an issue already. If a packager has 100's or 1000's of > packages.. there is no way they will have done so except on a 1 in a > million case set. I think if are going to aim that a packager can > 'maintain' hundreds or thousands of packages that we also assume that > this security is not going to be checked by the maintainer. I really don't think I'm *that* special a case so I'ld prefer you check that this is actually true rather than assume something wrong. Emmanuel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 01:08:11PM -0500, Stephen John Smoogen wrote: > On Tue, 28 Jan 2020 at 13:01, Robbie Harwood wrote: > > > > "Richard W.M. Jones" writes: > > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > > It's possible to maintain 20 with a lot of work. And if you want to > > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > > then you have to write your own automation. Could we do things > > > better? No one asked for them, but here are my ideas ... > > > > > > --- > > > > > > * CVE bugs should autoclose when a package is rebased > > > > > > Fabiano built the mingw-openssl package recently, but there are still > > > a load of open CVE bugs against this package referring to the older > > > version. These should be closed automatically. I think this will > > > require collecting the version of the package that fixes a CVE and > > > recording that in Bugzilla (or in the package itself in some standard > > > way). > > > > This is an interesting idea, and I appreciate you're considering ways to > > resolve this problem. However, I'm concerned that this will lead to > > maintainers not actually checking whether a version fixes an issue - > > since we don't have automatic verification (or even usually manual > > verification) of security fixes, that wouldn't get caught. > > > > You are assuming that maintainers actually check to see if a version > fixes an issue already. If a packager has 100's or 1000's of > packages.. there is no way they will have done so except on a 1 in a > million case set. I think if are going to aim that a packager can > 'maintain' hundreds or thousands of packages that we also assume that > this security is not going to be checked by the maintainer. If it > needs to be checked it will need to be 'outsourced' to some group who > can do so. Also the bit where I said "(or in the package itself in some standard way)." Of course that standard way doesn't exist (or does it?) but we could surely encourage upstreams to provide data that we need in a standard format, especially if we coorperate with Debian, SUSE, Arch and others. For example, for my upstream projects I write release notes (!= the changelog) but they're text files and not really in any standard location or format. Also we have security bug pages but again not in a standard way. If it was standardized we could suck that data straight into Fedora and no packager, whether maintaining 1 or 100 packages, would have to be troubled by this. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 13:01, Robbie Harwood wrote: > > "Richard W.M. Jones" writes: > > > I always think that Fedora works fine if you maintain 1-5 packages. > > It's possible to maintain 20 with a lot of work. And if you want to > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > then you have to write your own automation. Could we do things > > better? No one asked for them, but here are my ideas ... > > > > --- > > > > * CVE bugs should autoclose when a package is rebased > > > > Fabiano built the mingw-openssl package recently, but there are still > > a load of open CVE bugs against this package referring to the older > > version. These should be closed automatically. I think this will > > require collecting the version of the package that fixes a CVE and > > recording that in Bugzilla (or in the package itself in some standard > > way). > > This is an interesting idea, and I appreciate you're considering ways to > resolve this problem. However, I'm concerned that this will lead to > maintainers not actually checking whether a version fixes an issue - > since we don't have automatic verification (or even usually manual > verification) of security fixes, that wouldn't get caught. > You are assuming that maintainers actually check to see if a version fixes an issue already. If a packager has 100's or 1000's of packages.. there is no way they will have done so except on a 1 in a million case set. I think if are going to aim that a packager can 'maintain' hundreds or thousands of packages that we also assume that this security is not going to be checked by the maintainer. If it needs to be checked it will need to be 'outsourced' to some group who can do so. > I feel like bodhi updates help with this for non-rawhide versions (at > least, in the web interface) by proposing possible "fixed in this > version" bugs, but I'm not sure how to get that for rawhide without > requiring bodhi there (which I don't want to do). > > Thanks, > --Robbie > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
"Richard W.M. Jones" writes: > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... > > --- > > * CVE bugs should autoclose when a package is rebased > > Fabiano built the mingw-openssl package recently, but there are still > a load of open CVE bugs against this package referring to the older > version. These should be closed automatically. I think this will > require collecting the version of the package that fixes a CVE and > recording that in Bugzilla (or in the package itself in some standard > way). This is an interesting idea, and I appreciate you're considering ways to resolve this problem. However, I'm concerned that this will lead to maintainers not actually checking whether a version fixes an issue - since we don't have automatic verification (or even usually manual verification) of security fixes, that wouldn't get caught. I feel like bodhi updates help with this for non-rawhide versions (at least, in the web interface) by proposing possible "fixed in this version" bugs, but I'm not sure how to get that for rawhide without requiring bodhi there (which I don't want to do). Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
It would be nice if you could look into existing code instead of writing new one: https://github.com/ignatenkobrain/git-rpm-changelog On Tue, Jan 28, 2020, 12:43 Pierre-Yves Chibon wrote: > On Tue, Jan 28, 2020 at 09:03:09AM +, Richard W.M. Jones wrote: > > I always think that Fedora works fine if you maintain 1-5 packages. > > It's possible to maintain 20 with a lot of work. And if you want to > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > then you have to write your own automation. Could we do things > > better? No one asked for them, but here are my ideas ... > > > > --- > > > > * kill the %changelog > > > > Please, let's kill it, and generate it from the git changelog. > > I'm glad to see there's a proposal to do this. > > > > A general principle I'm following here is a packager should never > > be asked to enter the same information twice. > > There are a few tricky things about this, but overall I think it's doable > and > some of the tricky things may just be things we just have to accept as > being > different from the current situation. > > We are playing a bit of a proof of concept of what a RPM changelog > generated > from git logs could look like. The outcome of this is at: > https://pagure.io/Fedora-Infra/generate_changelog/tree/master > > Feel free to poke at it and see how it behaves. > > The script only considers: > - the last two years of commits > - commits touching either the spec file or patches (ending with .patch) > > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current > commit that should be ignored), as well as a way to say: [Replace XXX] to > be > able to replace an existing commit message, but that's not there yet. > > One of the thing that may change is that we may end up with changelog > entry that > do not have a corresponding nvr at the end. > The python script above shows that, it'll only add the v-r in the > changelog when > either one of them changed, if they are the same, it doesn't specify them. > > Another aspect that is getting trickier is: > - update to 1.1-1 > > - fix missing BR > > - spec clean up > > 3 commits, can all be either on 3 different days but could also be on the > same > day. Could be from 3 different people, but could also be from the same > person. > So we could have 3 commits, on 1 day from 1 person, two of which would make > sense to group together (update to 1.1-1 and fix missing BR) and one that > shouldn't since the build succeeded before that and thus the -1 that goes > out to > the mirror will not have this clean up entry. > The current approach we took is: we have 3 entries in the changelog (and > only > one has the v-r specified). It's not ideal, but we don't quite see how to > solve > this question yet. > If the "spec clean up" contains "[ignore]", that question is solved, but > if we > replace "spec clean up" with "Drop sub-package foo" then we do not have a > good > idea on how to consolidate the commit messages into a single changelog > entry. > Suggestions welcome :) > > > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > That was part of the proposal we debated last fall and there seemed to be > much > more people against this than I thought there would be. > Maybe we could start with having an opt-in approach. > > > * commit groups of packages together > > > > One reason why automatic commit -> build might not be desirable is if > > you're trying to build a group of packages in a side tag. In my > > opinion this means we should allow groups of packages to be committed > > together. (Unfortunately because we chose to put every package in its > > own git repo, the obvious way to do this can't work, but we could > > still have a "ChangeId:" header in the commit message that ties > > packages together). > > > > The packages should be built in build dependency order into a side > > tag, and the side tag turned into an update, with no further > > involvement from the packager unless something fails to build. > > > > This change on its own would solve the main problem that > > affects people maintaining large sets of packages. > > If we use a magic word to support opt-into automating commit -> build, we > could > get partly there. > BuildIn: Fedora, EPEL > BuildIn: > BuildIn: rawhide > ... > > > * “Fixes:” etc headers in git > > > > RHEL already does this. It should be possible to add special headers > > to the commit message to automatically close bugs, ie: > > > > $ git commit -m "ocaml: Update to version 4.10.0 > > Fixes: RHBZ#12345" > > > > Note the build, update and bug closing would happen completely > > automatically, assuming there was no build or test failure. > > We clearly don't have a good mapping from commit messages to bugzilla. > dist-git has the info, either in the rpm changelog or in the git commit > messages > koji has only the info of the git commit it was triggered from > bodhi works at the
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 8:59 AM Pierre-Yves Chibon wrote: > > On Tue, Jan 28, 2020 at 02:51:28PM +0100, Florian Weimer wrote: > > * Pierre-Yves Chibon: > > > > >> Sorry for being unclear. The spec file in dist-git would still show > > >> some (older) version of the %changelog entries under this model. > > >> > > >> The corrections would update the %changelog with all the historic > > >> entries. Since auto-generation stops at this commit, the new generated > > >> %changelog will include the fixed entries. > > >> > > >> This would also avoid the need for a new knob to adjust how far back the > > >> %changelog generation goes. > > > > > > Let me rephrase to see if I understand correctly. > > > The changelog would be something like: > > > `` > > > %changelog > > > %generate_changelog > > > > > > > > > `` > > > > > > This way we don't need to generate the old entries, we can just focus on > > > the > > > commits once the packager has opted in and not look at the old commits in > > > the > > > git history. > > > > > > Is that what you were thinking? > > > > Yes, but the generation would not stop when %generate_changelog was > > added, but when the last edit below %changelog was made. > > > > Does this make sense? Sorry if I'm not explaining this properly. > > I think I understand, by going up until the time the changelog was last > touched > (be that the macro or something else), we give an easy way to edit it. > > We would also need to have fepdkg generate-changelog to help with this. > This is pretty similar to how it works in Mageia, actually. In the Mageia workflow, changelog entries are managed manually until it is imported into our Dist-SVN. The changelog section is stripped and split into a separate file in a separate folder in the SVN tree (which is the equivalent of an orphan branch) and appended after the VCS-based changelog on package builds. This can be viewed locally as well, like so (example from python-distro in Mageia): $ mgarepo rpmlog -o python-distro * Thu Sep 20 2018 Sysadmin Bot 1.0.2-7.mga7 + Revision: 1288676 - Mageia 7 Mass Rebuild * Sat Aug 05 2017 Pascal Terjan 1.0.2-6.mga7 + Revision: 1135369 - Rebuild for python 3.6 * Thu Mar 16 2017 Neal Gompa 1.0.2-5.mga6 + Revision: 1093149 - Port to Mageia - imported package python-distro * Sat Feb 11 2017 Fedora Release Engineering - 1.0.2-4 - Rebuilt for https://fedoraproject.org/wiki/Fedora_26_Mass_Rebuild * Tue Jan 24 2017 Miroslav Suchý 1.0.2-3 - typo in license macro * Tue Jan 24 2017 Miroslav Suchý 1.0.2-2 - add license macro for el6 You can see from the above example where the generated / manual changelog split occurs. mgarepo is packaged in Fedora and you can play with this to see how it works. :) -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 02:51:28PM +0100, Florian Weimer wrote: > * Pierre-Yves Chibon: > > >> Sorry for being unclear. The spec file in dist-git would still show > >> some (older) version of the %changelog entries under this model. > >> > >> The corrections would update the %changelog with all the historic > >> entries. Since auto-generation stops at this commit, the new generated > >> %changelog will include the fixed entries. > >> > >> This would also avoid the need for a new knob to adjust how far back the > >> %changelog generation goes. > > > > Let me rephrase to see if I understand correctly. > > The changelog would be something like: > > `` > > %changelog > > %generate_changelog > > > > > > `` > > > > This way we don't need to generate the old entries, we can just focus on the > > commits once the packager has opted in and not look at the old commits in > > the > > git history. > > > > Is that what you were thinking? > > Yes, but the generation would not stop when %generate_changelog was > added, but when the last edit below %changelog was made. > > Does this make sense? Sorry if I'm not explaining this properly. I think I understand, by going up until the time the changelog was last touched (be that the macro or something else), we give an easy way to edit it. We would also need to have fepdkg generate-changelog to help with this. Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
* Pierre-Yves Chibon: >> Sorry for being unclear. The spec file in dist-git would still show >> some (older) version of the %changelog entries under this model. >> >> The corrections would update the %changelog with all the historic >> entries. Since auto-generation stops at this commit, the new generated >> %changelog will include the fixed entries. >> >> This would also avoid the need for a new knob to adjust how far back the >> %changelog generation goes. > > Let me rephrase to see if I understand correctly. > The changelog would be something like: > `` > %changelog > %generate_changelog > > > `` > > This way we don't need to generate the old entries, we can just focus on the > commits once the packager has opted in and not look at the old commits in the > git history. > > Is that what you were thinking? Yes, but the generation would not stop when %generate_changelog was added, but when the last edit below %changelog was made. Does this make sense? Sorry if I'm not explaining this properly. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 02:25:33PM +0100, Florian Weimer wrote: > * Pierre-Yves Chibon: > > > On Tue, Jan 28, 2020 at 01:53:32PM +0100, Florian Weimer wrote: > >> * Pierre-Yves Chibon: > >> > >> > Feel free to poke at it and see how it behaves. > >> > > >> > The script only considers: > >> > - the last two years of commits > >> > - commits touching either the spec file or patches (ending with .patch) > >> > > >> > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the > >> > current > >> > commit that should be ignored), as well as a way to say: [Replace XXX] > >> > to be > >> > able to replace an existing commit message, but that's not there yet. > >> > >> Have you considered only auto-generating %changelog entries since the > >> last commit that changed the %changelog section? > > > > We thought about that, pull the last changelog from the last build, get the > > commit hash of that last build and only generate the new changelog from > > HEAD... > > The issue was: how to fix existing entries (typo, wrong content...) and our > > thought on this was that it would be easier to re-generate the entire > > changelog > > for this reason. > > > >> To fix entries, you would run a tool to materialize the current > >> auto-generated %changelog (and Release:), and then make the necessary > >> corrections and additions. > > > > Where would this corrections/additions live though? > > We were thinking to include the corrections in other/new commits but if we > > don't > > reach the old commits, we can't correct them. > > > > What did you have in mind? > > Sorry for being unclear. The spec file in dist-git would still show > some (older) version of the %changelog entries under this model. > > The corrections would update the %changelog with all the historic > entries. Since auto-generation stops at this commit, the new generated > %changelog will include the fixed entries. > > This would also avoid the need for a new knob to adjust how far back the > %changelog generation goes. Let me rephrase to see if I understand correctly. The changelog would be something like: `` %changelog %generate_changelog `` This way we don't need to generate the old entries, we can just focus on the commits once the packager has opted in and not look at the old commits in the git history. Is that what you were thinking? If so I think I like it, if not I still like this but then I didn't catch what you meant :) Thanks, Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
* Pierre-Yves Chibon: > On Tue, Jan 28, 2020 at 01:53:32PM +0100, Florian Weimer wrote: >> * Pierre-Yves Chibon: >> >> > Feel free to poke at it and see how it behaves. >> > >> > The script only considers: >> > - the last two years of commits >> > - commits touching either the spec file or patches (ending with .patch) >> > >> > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current >> > commit that should be ignored), as well as a way to say: [Replace XXX] to >> > be >> > able to replace an existing commit message, but that's not there yet. >> >> Have you considered only auto-generating %changelog entries since the >> last commit that changed the %changelog section? > > We thought about that, pull the last changelog from the last build, get the > commit hash of that last build and only generate the new changelog from > HEAD... > The issue was: how to fix existing entries (typo, wrong content...) and our > thought on this was that it would be easier to re-generate the entire > changelog > for this reason. > >> To fix entries, you would run a tool to materialize the current >> auto-generated %changelog (and Release:), and then make the necessary >> corrections and additions. > > Where would this corrections/additions live though? > We were thinking to include the corrections in other/new commits but if we > don't > reach the old commits, we can't correct them. > > What did you have in mind? Sorry for being unclear. The spec file in dist-git would still show some (older) version of the %changelog entries under this model. The corrections would update the %changelog with all the historic entries. Since auto-generation stops at this commit, the new generated %changelog will include the fixed entries. This would also avoid the need for a new knob to adjust how far back the %changelog generation goes. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 01:53:32PM +0100, Florian Weimer wrote: > * Pierre-Yves Chibon: > > > Feel free to poke at it and see how it behaves. > > > > The script only considers: > > - the last two years of commits > > - commits touching either the spec file or patches (ending with .patch) > > > > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current > > commit that should be ignored), as well as a way to say: [Replace XXX] to be > > able to replace an existing commit message, but that's not there yet. > > Have you considered only auto-generating %changelog entries since the > last commit that changed the %changelog section? We thought about that, pull the last changelog from the last build, get the commit hash of that last build and only generate the new changelog from HEAD... The issue was: how to fix existing entries (typo, wrong content...) and our thought on this was that it would be easier to re-generate the entire changelog for this reason. > To fix entries, you would run a tool to materialize the current > auto-generated %changelog (and Release:), and then make the necessary > corrections and additions. Where would this corrections/additions live though? We were thinking to include the corrections in other/new commits but if we don't reach the old commits, we can't correct them. What did you have in mind? Thanks, Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
* Pierre-Yves Chibon: > Feel free to poke at it and see how it behaves. > > The script only considers: > - the last two years of commits > - commits touching either the spec file or patches (ending with .patch) > > We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current > commit that should be ignored), as well as a way to say: [Replace XXX] to be > able to replace an existing commit message, but that's not there yet. Have you considered only auto-generating %changelog entries since the last commit that changed the %changelog section? To fix entries, you would run a tool to materialize the current auto-generated %changelog (and Release:), and then make the necessary corrections and additions. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 05:41, Daniel P. Berrangé wrote: > > On Tue, Jan 28, 2020 at 11:32:46AM +0100, Guido Aulisi wrote: > > Il giorno mar 28 gen 2020 alle ore 10:04 Richard W.M. Jones > > ha scritto: > > > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > > It's possible to maintain 20 with a lot of work. And if you want to > > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > > then you have to write your own automation. Could we do things > > > better? No one asked for them, but here are my ideas ... > > > > > > --- > > > > > > * kill the %changelog > > > > > > Please, let's kill it, and generate it from the git changelog. > > > I'm glad to see there's a proposal to do this. > > > > > > A general principle I'm following here is a packager should never > > > be asked to enter the same information twice. > > > > > > * committing to git should build the package > > > > > > Is there a reason why this wouldn't be the case? > > > > Sometimes you only add comments to the spec file and a rebuild is not > > needed. > > What % of commits to dist-git are this scenario ? I've done this myself > but a few times but for myself it is a 2-3% of my commits don't involve > a build. There is no real harm in doing a build in these cases IMHO. It > would have negligible extra burden on Fedora build infrastructure, while > potentially simplifying the common case for maintainers. > I want to start off by saying I think that Richard's ideas on the whole would be good.. however I do not think any of them would be a negligible extra burden. The problem is we only have so much disk space, so many builders, and we have way too many ways for builds and composes to fail which takes up time that would be used to fix other problems. And finally we have a limited budget of people who work on the build system nearly full-time. That means we usually have to graft on some new set of tools onto the existing build system which ends up adding even more ways to cause failures. -- Stephen J Smoogen. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 11:32:46AM +0100, Guido Aulisi wrote: > Il giorno mar 28 gen 2020 alle ore 10:04 Richard W.M. Jones > ha scritto: > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > It's possible to maintain 20 with a lot of work. And if you want to > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > then you have to write your own automation. Could we do things > > better? No one asked for them, but here are my ideas ... > > > > --- > > > > * kill the %changelog > > > > Please, let's kill it, and generate it from the git changelog. > > I'm glad to see there's a proposal to do this. > > > > A general principle I'm following here is a packager should never > > be asked to enter the same information twice. > > > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > Sometimes you only add comments to the spec file and a rebuild is not needed. In the current situation, the build would fail in koji because you did not touch either version nor release. So koji will very quickly fail and move on. In the future, if you opt-in for auto-generated changelog and release fields, this may result in an actual build, which will make koji busy for a little while long and then it'll stay there, on its own, until it's garbage collected. The only release for which this would end up going to the user would be rawhide. Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 09:03:09AM +, Richard W.M. Jones wrote: > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... > > --- > > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. > > A general principle I'm following here is a packager should never > be asked to enter the same information twice. There are a few tricky things about this, but overall I think it's doable and some of the tricky things may just be things we just have to accept as being different from the current situation. We are playing a bit of a proof of concept of what a RPM changelog generated from git logs could look like. The outcome of this is at: https://pagure.io/Fedora-Infra/generate_changelog/tree/master Feel free to poke at it and see how it behaves. The script only considers: - the last two years of commits - commits touching either the spec file or patches (ending with .patch) We want a way to say: [Ignore XXX] (or simply [Ignore] if it's the current commit that should be ignored), as well as a way to say: [Replace XXX] to be able to replace an existing commit message, but that's not there yet. One of the thing that may change is that we may end up with changelog entry that do not have a corresponding nvr at the end. The python script above shows that, it'll only add the v-r in the changelog when either one of them changed, if they are the same, it doesn't specify them. Another aspect that is getting trickier is: - update to 1.1-1 - fix missing BR - spec clean up 3 commits, can all be either on 3 different days but could also be on the same day. Could be from 3 different people, but could also be from the same person. So we could have 3 commits, on 1 day from 1 person, two of which would make sense to group together (update to 1.1-1 and fix missing BR) and one that shouldn't since the build succeeded before that and thus the -1 that goes out to the mirror will not have this clean up entry. The current approach we took is: we have 3 entries in the changelog (and only one has the v-r specified). It's not ideal, but we don't quite see how to solve this question yet. If the "spec clean up" contains "[ignore]", that question is solved, but if we replace "spec clean up" with "Drop sub-package foo" then we do not have a good idea on how to consolidate the commit messages into a single changelog entry. Suggestions welcome :) > * committing to git should build the package > > Is there a reason why this wouldn't be the case? That was part of the proposal we debated last fall and there seemed to be much more people against this than I thought there would be. Maybe we could start with having an opt-in approach. > * commit groups of packages together > > One reason why automatic commit -> build might not be desirable is if > you're trying to build a group of packages in a side tag. In my > opinion this means we should allow groups of packages to be committed > together. (Unfortunately because we chose to put every package in its > own git repo, the obvious way to do this can't work, but we could > still have a "ChangeId:" header in the commit message that ties > packages together). > > The packages should be built in build dependency order into a side > tag, and the side tag turned into an update, with no further > involvement from the packager unless something fails to build. > > This change on its own would solve the main problem that > affects people maintaining large sets of packages. If we use a magic word to support opt-into automating commit -> build, we could get partly there. BuildIn: Fedora, EPEL BuildIn: BuildIn: rawhide ... > * “Fixes:” etc headers in git > > RHEL already does this. It should be possible to add special headers > to the commit message to automatically close bugs, ie: > > $ git commit -m "ocaml: Update to version 4.10.0 > Fixes: RHBZ#12345" > > Note the build, update and bug closing would happen completely > automatically, assuming there was no build or test failure. We clearly don't have a good mapping from commit messages to bugzilla. dist-git has the info, either in the rpm changelog or in the git commit messages koji has only the info of the git commit it was triggered from bodhi works at the nevr level. A lot of food for thoughts here :) Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 11:32:46AM +0100, Guido Aulisi wrote: > Il giorno mar 28 gen 2020 alle ore 10:04 Richard W.M. Jones > ha scritto: > > > > I always think that Fedora works fine if you maintain 1-5 packages. > > It's possible to maintain 20 with a lot of work. And if you want to > > maintain 100+ (things like the ocaml-* set that I help to maintain) > > then you have to write your own automation. Could we do things > > better? No one asked for them, but here are my ideas ... > > > > --- > > > > * kill the %changelog > > > > Please, let's kill it, and generate it from the git changelog. > > I'm glad to see there's a proposal to do this. > > > > A general principle I'm following here is a packager should never > > be asked to enter the same information twice. > > > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > Sometimes you only add comments to the spec file and a rebuild is not needed. What % of commits to dist-git are this scenario ? I've done this myself but a few times but for myself it is a 2-3% of my commits don't involve a build. There is no real harm in doing a build in these cases IMHO. It would have negligible extra burden on Fedora build infrastructure, while potentially simplifying the common case for maintainers. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 28 Jan 2020 at 10:53, Richard W.M. Jones wrote: > > On Tue, Jan 28, 2020 at 10:21:41AM +0100, Milan Crha wrote: > > On Tue, 2020-01-28 at 10:03 +0100, Richard W.M. Jones wrote: > > > * committing to git should build the package > > > > > > Is there a reason why this wouldn't be the case? > > > > Hi, > > the answer for the above is just your following point: > > > > > * commit groups of packages together > > > > aka the dependencies. Sometimes you want a special side tag, sometimes > > it's not needed. The way I do it right now (it's only about 4 packages > > depending on each other, not hundreds), is that I commit to master, > > then to stable, then the second package to master, to stable, then > > third and finally to the fourth and then I ran a chain-build as this: > > "a : b : c " in package 'd', (which builds 'c' and 'd' in parallel, > > once 'a' and 'b' are built in serial). Then I just refresh the koji > > build page from time to time and verify that the build still runs > > and/.or it finished successfully. I can run chain-build in stable too, > > it only needs a bit more intervention, to define overrides for 'a' and > > 'b' in bodhi, to be able to build them. > > > > I'm afraid fully automating such things might be a challenge. In other > > words, properly solving dependencies is problematic. Having yet another > > syntax to describe it, or the groups you suggest, scares me a bit. And > > we are not talking about inter-package dependencies, with packages you > > are not maintaining. > > This is not a problem - it has been solved several times > independently. I most recently proposed this, but it's certainly > isn't the first time it has been done: > > https://rwmj.wordpress.com/2020/01/14/goals-an-experimental-new-tool-which-generalizes-make/ > http://git.annexia.org/?p=fedora-ocaml-rebuild.git;a=summary > > What we need is Fedora to recognize that maintaining 100s of packages > mostly automatically should be a goal. If you look at the ecosystems > around language packaging (cpan, nodejs, crates, opam, etc) this ought > to be self-evident. When there's a unified well-organized language-specific ecosystem (and rpm-friendly; see the Java case...) it's definitely easier to do this, and yet... See [1] for example (and follow the "Homepage" button to see the tooling and setup) for an attempt to maintain thousands of packages for a particular ecosystem that is quite strict and homogeneous. And yet, as I said, many aspects of those specs wouldn't pass a package review. That said, I completely agree that "maintaining 100s of packages mostly automatically" should be a goal. Iñaki [1] https://copr.fedorainfracloud.org/coprs/iucar/cran/ ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
Il giorno mar 28 gen 2020 alle ore 10:04 Richard W.M. Jones ha scritto: > > I always think that Fedora works fine if you maintain 1-5 packages. > It's possible to maintain 20 with a lot of work. And if you want to > maintain 100+ (things like the ocaml-* set that I help to maintain) > then you have to write your own automation. Could we do things > better? No one asked for them, but here are my ideas ... > > --- > > * kill the %changelog > > Please, let's kill it, and generate it from the git changelog. > I'm glad to see there's a proposal to do this. > > A general principle I'm following here is a packager should never > be asked to enter the same information twice. > > * committing to git should build the package > > Is there a reason why this wouldn't be the case? Sometimes you only add comments to the spec file and a rebuild is not needed. Ciao Guido ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, Jan 28, 2020 at 10:21:41AM +0100, Milan Crha wrote: > On Tue, 2020-01-28 at 10:03 +0100, Richard W.M. Jones wrote: > > * committing to git should build the package > > > > Is there a reason why this wouldn't be the case? > > Hi, > the answer for the above is just your following point: > > > * commit groups of packages together > > aka the dependencies. Sometimes you want a special side tag, sometimes > it's not needed. The way I do it right now (it's only about 4 packages > depending on each other, not hundreds), is that I commit to master, > then to stable, then the second package to master, to stable, then > third and finally to the fourth and then I ran a chain-build as this: > "a : b : c " in package 'd', (which builds 'c' and 'd' in parallel, > once 'a' and 'b' are built in serial). Then I just refresh the koji > build page from time to time and verify that the build still runs > and/.or it finished successfully. I can run chain-build in stable too, > it only needs a bit more intervention, to define overrides for 'a' and > 'b' in bodhi, to be able to build them. > > I'm afraid fully automating such things might be a challenge. In other > words, properly solving dependencies is problematic. Having yet another > syntax to describe it, or the groups you suggest, scares me a bit. And > we are not talking about inter-package dependencies, with packages you > are not maintaining. This is not a problem - it has been solved several times independently. I most recently proposed this, but it's certainly isn't the first time it has been done: https://rwmj.wordpress.com/2020/01/14/goals-an-experimental-new-tool-which-generalizes-make/ http://git.annexia.org/?p=fedora-ocaml-rebuild.git;a=summary What we need is Fedora to recognize that maintaining 100s of packages mostly automatically should be a goal. If you look at the ecosystems around language packaging (cpan, nodejs, crates, opam, etc) this ought to be self-evident. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Ideas for better development processes when maintaining hundreds of packages
On Tue, 2020-01-28 at 10:03 +0100, Richard W.M. Jones wrote: > * committing to git should build the package > > Is there a reason why this wouldn't be the case? Hi, the answer for the above is just your following point: > * commit groups of packages together aka the dependencies. Sometimes you want a special side tag, sometimes it's not needed. The way I do it right now (it's only about 4 packages depending on each other, not hundreds), is that I commit to master, then to stable, then the second package to master, to stable, then third and finally to the fourth and then I ran a chain-build as this: "a : b : c " in package 'd', (which builds 'c' and 'd' in parallel, once 'a' and 'b' are built in serial). Then I just refresh the koji build page from time to time and verify that the build still runs and/.or it finished successfully. I can run chain-build in stable too, it only needs a bit more intervention, to define overrides for 'a' and 'b' in bodhi, to be able to build them. I'm afraid fully automating such things might be a challenge. In other words, properly solving dependencies is problematic. Having yet another syntax to describe it, or the groups you suggest, scares me a bit. And we are not talking about inter-package dependencies, with packages you are not maintaining. Bye, Milan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Ideas for better development processes when maintaining hundreds of packages
I always think that Fedora works fine if you maintain 1-5 packages. It's possible to maintain 20 with a lot of work. And if you want to maintain 100+ (things like the ocaml-* set that I help to maintain) then you have to write your own automation. Could we do things better? No one asked for them, but here are my ideas ... --- * kill the %changelog Please, let's kill it, and generate it from the git changelog. I'm glad to see there's a proposal to do this. A general principle I'm following here is a packager should never be asked to enter the same information twice. * committing to git should build the package Is there a reason why this wouldn't be the case? * commit groups of packages together One reason why automatic commit -> build might not be desirable is if you're trying to build a group of packages in a side tag. In my opinion this means we should allow groups of packages to be committed together. (Unfortunately because we chose to put every package in its own git repo, the obvious way to do this can't work, but we could still have a "ChangeId:" header in the commit message that ties packages together). The packages should be built in build dependency order into a side tag, and the side tag turned into an update, with no further involvement from the packager unless something fails to build. This change on its own would solve the main problem that affects people maintaining large sets of packages. * “Fixes:” etc headers in git RHEL already does this. It should be possible to add special headers to the commit message to automatically close bugs, ie: $ git commit -m "ocaml: Update to version 4.10.0 Fixes: RHBZ#12345" Note the build, update and bug closing would happen completely automatically, assuming there was no build or test failure. * CVE bugs should autoclose when a package is rebased Fabiano built the mingw-openssl package recently, but there are still a load of open CVE bugs against this package referring to the older version. These should be closed automatically. I think this will require collecting the version of the package that fixes a CVE and recording that in Bugzilla (or in the package itself in some standard way). * create streams of packages automatically depending on quality scores We know a lot about packages such as: - How many bugs are opened against them? - What's the average time between a bug being filed and fixed? - Do they get regularly updated? - Do they pass or fail CI tests? - How many rpmlint / fedora-packager problems do they have? Why don't we use that data to build streams of high quality and lesser quality packages? Allow the end users to choose whether they want the best curated packages only, or they're prepared to accept the risk of taking a package with lots of bugs or CVEs (this is assuming the previous point is fixed, so these are real CVEs, not irrelevant bugs). If you want to go even further with this idea, then it could even be possible we allow packages into Fedora without any review. They would start in the outermost stream in a "there be dragons" repository that only the foolhardy would enable, but as their quality improved they would *automatically* migrate into the mainstream. --- Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
