Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
On Fri, Jul 03, 2020 at 11:11:30AM +0200, Nicolas Mailhot via devel wrote: > Le vendredi 03 juillet 2020 à 09:48 +0200, Pierre-Yves Chibon a écrit : > > On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote: > > > Nicolas Mailhot wrote: > > > > The same process that commits a new state of the changelog file > > > > in > > > > sources, > > > > commits the date that was written in the changelog in a separate > > > > key = > > > > value > > > > file (with the components of the build evr, the last packager id, > > > > etc). > > > > > > Do you mean that the key/value file will be committed to Git from > > > inside > > > Koji? Do the Koji builders have write access to Git? > > > > This is the part that worries me a little about this approach. > > Builders currently do not have commit access to git and I'm not sure > > if we want them to considering they have git installed (so they can > > clone) as well as access to all the packages in dist-git from a > > networking point of view (again so they can clone). > > So if we were to give the builders commit access to dist-git, an > > attacker could easily commit to any other packages, potentially from > > something as easy as a scratch-build. > > From a purely architecture POW I’m convinced the proposed approach is > the correct approach. Anything else proposed so far involves: > – tying a low-level event like "build occurred at date XXX" to high- > level Fedora infra (making our workflow non portable and > incompatible with downstreams and third parties) I don't think this point is true in the case of rpmautospec. > – taking bets in git that a build will occur and succeed (before it > actually occurs and succeeds, in real life builds fail for various > reasons), and I don't think this point is true, in the case of rpmautospec where the tagging is done by the build system, once that build has succeeded. > – attempting to munge spec file behind the packager back (unlikely to > work fine the more automated and dynamic we made those). I'd contest the "behind the packager back" in the case of rpmautospec considering this is well documented, but I fear that qualification may just have been a little trolling :) > However, because it’s the correct architecture solution, it also forces > to make hard architectural choices, instead of mixing unrelated things > in git and pretending that makes the result fine. Mixing unrelated > things in a pile of container or git poo and pretending the result is > fine is exactly what I hate in contenerish build workflows and why I > work on rpm packaging. > > From a pure high-level view, the thing in our infra that gates builds > and decides whether they are official or scratched is bodhi. > > So if you want to push Fedora release logic to its ultimate conclusion, > the thing that should be in charge of committing the new > release/changelog build state to package history in git is bodhi, not > koji. And you can put security related checks there, since deciding to > push things to users requires security related checks anyway (that > probably also involves branching while a bodhi update is in flight and > not approved yet). > > However, that’s if you want to push the model to its ultimate > conclusion and have something nice solid, automated, and future-proof. > > If you don’t want to touch bodhi, and it you do not want koji to commit > to git (which, is not the best of things for the reasons you stated, > and for the reasons I stated), you can just: > – make the koji client return the URL that will contain the SRPM at the > end of the build process if it succeeds. > – have the person of script that called the koji client (and has, > presumably, write access to the corresponding packages) consult the > build results later > – and have this person or script decide if he or it wants to commit the > build result to history or not So if I understand correctly the proposed workflow is: - git commit your changes && git push - fedpkg build the commit - if build succeeded: (the build system if we manage to, or the packager:) - $editor release_file - adjust the release value in that file - git add release_file && git commit && git push The suggestion I'm proposing would be: - git commit your changes && git push - fedpkg build the commit - if build succeeded: - the build system tags the commit built with the release value And we could trivially re-use the koji plugin that was written for rpmautospec to do this, in whichever format your macros need. This lowers the amount of manual interventions and reduces the risk of conflict as the release_file would change for every build (conflicts which would also appear when doing git merge and potentially in PRs, especially if that file remains manually updated). Basically, we are introducing another "sources" file that will conflict for every release change rather than version change. Also, if you have long running builds (say chromium
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Le 2020-07-05 23:55, Dan Čermák a écrit : Hi Dan So essentially you store the changelog in a separate file The changelog is already detached in the F33 change https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs This F34 change adds bumping to the detached file https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping and use that to calculate the next release field? The changelog file is not used as source of data, except as a reference state that will be added to. Using the changelog file as source of data would require quite a lot of complex and unreliable changelog format parsing, so the bumping data is taken from another key = value file (that uses the same persistence mechanisms). Also the packager may decide to trim quite a lot of intermediary changelog events, so the EVR and date in the last changelog entry are not necessarily the EVR and date of the last build. Given the other replies to this thread (that this is all local only, unless koji does git commits), does that mean that it's still: dist-git commit = rebuild. To be part of official Fedora history the result of a build needs to be committed yes. The change does not force you to build every commit, nor to commit every build out there. The "only" difference to the current state of affairs being, that I don't have to specify the Release field myself? Once you've modified a spec, and set a starting evr point in this spec, further rebuilds do not involve touching the spec. Spec changes are real spec changes, not spec changes to bump a release or bump a changelog. The packager does not have to request the modification in his spec, it’s done as part of the various %auto_foo calls the change introduced Could you provide an example how this would look in practice? If you want a demonstration of the auto framework and of changelog detaching, you can take any of the non macro builds in https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-changelog-fonts/builds/ If you want to see a demonstration of autobumping, you need to rpmbuild -ba manually right now, because of the two small limitations mock side. So you need to take the redhat-rpm-config and fonts macro packages in: https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/ and rebuild one of the other packages in there. The only difference between the two coprs is the redhat-rpm-config package, there is no change in the fonts macro package or in the automated packages themselves. Autobumping can be implemented without any spec file change once the auto framework is used. (The mock limitations are first, the fact that mock currently collects the SRPM at the start, not end of the build process, and second, the fact you need to pass packager ID that will end up in the changelog bump to the build process and there is no way in the copr/copr UI do do that.) What I am currently missing from this proposal though is: - How is this actually even implemented? - How will this look in practice? See above ↑ - Given that additional files would be put into dist-git, how do we roll this back in case things go wrong? (Having thousands of "remove %autorelease" commits by releng could be an option here, albeit not a pretty one). Since bumping is a feature over the auto framework, and does not require any additional spec change, it is enabled by registering bumping processing in this framework, and disabled by removing this registration. There is no need to change spec files or history. In fact I use the same spec files to QA the auto framework and bumping, and depending if the redhat-rpm-config version I test includes the bumping or not, they will bump or not. When bumping code is not present the additional key=value file bumping uses is not auto-added to sources, so the next srpm import will clear it from sources the same way patches disappear from sources once no longer used (and can linger forever if a packager does not import srpms and does not git rm those files explicitly). Removing the auto framework is something else altogether. Because its aim is to massively simplify spec files (in opt-in, not mandatory mode), you can not go back without undoing the spec simplifications. However, because great care was taken to define a clean and generic spec syntax when using the auto framework, you could replace it will multiple reimplementations without changing spec files. The %auto framework spec API is basically %prep %auto_prep ← automated processing It’s hard to go more generic than that. (You might want to remove the %auto calls altogether and have %prep, for example, call %prep by default, but that would remove the packager choice to use or not the %auto calls, and to insert custom processing before those calls). The only "irregularity" is that the %auto macro
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Hi Nicolas, Nicolas Mailhot writes: >> How do I let rpm generate the changelog automatically? > > This feature is not changelog generation, just changelog bumping on > build events. You still need some other method to put non-build events > in the changelog. > > The detached changelog is just one more file in SRPM sources, which is > modified by rpmbuild at `%build` time with other files rpmbuild > modifies. The tricky part is to modify the source file as a source file > so rpmbuild adds the result to the produced SRPM (and, that does not > work in mock right now, because mock serves the SRPM that existed at > the start, not at the end of the build. Though it’s probably just a > matter of getting mock to call again its SRPM creation method at the > end of the build). So essentially you store the changelog in a separate file (like it is done in openSUSE) and use that to calculate the next release field? Given the other replies to this thread (that this is all local only, unless koji does git commits), does that mean that it's still: dist-git commit = rebuild. The "only" difference to the current state of affairs being, that I don't have to specify the Release field myself? > > The packager does not have to request the modification in his spec, > it’s done as part of the various %auto_foo calls the change introduced Could you provide an example how this would look in practice? > >> And is this related to Piere/Pingou's work on the same topic that was >> deployed to koji staging? > > It’s a different implementation, at the rpm level, that does not tie > bumping to Fedora infra (koji included). Though, it is probably > complementary to what pingou did on the changelog alimentation front. > > IMHO the design mistake so far was to conflate bumping and non-build > event changelog filling. You need to do both of course but build event > should be a build event driven by the lowest common denominator > (rpmbuild) with koji/infra scrapping rpmbuild results as usual and > exposing them to users. This is a good point imho: not every rebuild warrants a changelog entry and having both separated appears sensible to me. What I am currently missing from this proposal though is: - How is this actually even implemented? - How will this look in practice? - Given that additional files would be put into dist-git, how do we roll this back in case things go wrong? (Having thousands of "remove %autorelease" commits by releng could be an option here, albeit not a pretty one). Cheers, Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Le vendredi 03 juillet 2020 à 09:48 +0200, Pierre-Yves Chibon a écrit : > On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote: > > Nicolas Mailhot wrote: > > > The same process that commits a new state of the changelog file > > > in > > > sources, > > > commits the date that was written in the changelog in a separate > > > key = > > > value > > > file (with the components of the build evr, the last packager id, > > > etc). > > > > Do you mean that the key/value file will be committed to Git from > > inside > > Koji? Do the Koji builders have write access to Git? > > This is the part that worries me a little about this approach. > Builders currently do not have commit access to git and I'm not sure > if we want them to considering they have git installed (so they can > clone) as well as access to all the packages in dist-git from a > networking point of view (again so they can clone). > So if we were to give the builders commit access to dist-git, an > attacker could easily commit to any other packages, potentially from > something as easy as a scratch-build. From a purely architecture POW I’m convinced the proposed approach is the correct approach. Anything else proposed so far involves: – tying a low-level event like "build occurred at date XXX" to high- level Fedora infra (making our workflow non portable and incompatible with downstreams and third parties) – taking bets in git that a build will occur and succeed (before it actually occurs and succeeds, in real life builds fail for various reasons), and – attempting to munge spec file behind the packager back (unlikely to work fine the more automated and dynamic we made those). However, because it’s the correct architecture solution, it also forces to make hard architectural choices, instead of mixing unrelated things in git and pretending that makes the result fine. Mixing unrelated things in a pile of container or git poo and pretending the result is fine is exactly what I hate in contenerish build workflows and why I work on rpm packaging. From a pure high-level view, the thing in our infra that gates builds and decides whether they are official or scratched is bodhi. So if you want to push Fedora release logic to its ultimate conclusion, the thing that should be in charge of committing the new release/changelog build state to package history in git is bodhi, not koji. And you can put security related checks there, since deciding to push things to users requires security related checks anyway (that probably also involves branching while a bodhi update is in flight and not approved yet). However, that’s if you want to push the model to its ultimate conclusion and have something nice solid, automated, and future-proof. If you don’t want to touch bodhi, and it you do not want koji to commit to git (which, is not the best of things for the reasons you stated, and for the reasons I stated), you can just: – make the koji client return the URL that will contain the SRPM at the end of the build process if it succeeds. – have the person of script that called the koji client (and has, presumably, write access to the corresponding packages) consult the build results later – and have this person or script decide if he or it wants to commit the build result to history or not That’s the REST way of doing things. It’s a co-out because you push hard commit decisions to the client, but it’s a prefectly valid approach. The commit decision exists with or without my change, it’s just people have (successfully) convinced themselves git is magic and git makes release decisions go away. You could also try to filter source files to limit the back commit to specific files. But really, if you don’t trust your build process to modify files in a secure way, you should not distribute the produced RPMs in the first place. > rpmautospec relies on git tags to store the build info, could it be > considered here? As explained above, that does not solve any of the hard problems, that handwaves them away by pretending that because someone filled some metadata in git, it corresponds to the actual build state. Regards, -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
On Friday, July 3, 2020 9:48:06 AM CEST Pierre-Yves Chibon wrote: > So if we were to give the builders commit access to dist-git, an attacker > could easily commit to any other packages, potentially from something as easy > as a scratch-build. Absolutely! Koji authenticates build submitters (I'm not sure it authorizes them). So technically, _something_ on backend could be allowed to commit to dist-git (in the name of build submitter). Before the SRPM build task, Koji could request "GetReleaseBumpPatch" task, the builder could then just read-only clone the git, bump the release, return the patch back for backend -- and let Koji apply it. But yeah, that's off topic a bit. This is not what the current proposal is about. Pavel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote: > Nicolas Mailhot wrote: > > The same process that commits a new state of the changelog file in > > sources, > > commits the date that was written in the changelog in a separate key = > > value > > file (with the components of the build evr, the last packager id, etc). > > Do you mean that the key/value file will be committed to Git from inside > Koji? Do the Koji builders have write access to Git? This is the part that worries me a little about this approach. Builders currently do not have commit access to git and I'm not sure if we want them to considering they have git installed (so they can clone) as well as access to all the packages in dist-git from a networking point of view (again so they can clone). So if we were to give the builders commit access to dist-git, an attacker could easily commit to any other packages, potentially from something as easy as a scratch-build. rpmautospec relies on git tags to store the build info, could it be considered here? It may make things a little safer as we could then restrict the access of that user/ssh key to only git tags (or do like rpmautospec and query pagure's API to have it create the git tag, thus dropping the need for ssh key). Pierre pgpTnAYdhtGD_.pgp Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Nicolas Mailhot wrote: > The same process that commits a new state of the changelog file in > sources, > commits the date that was written in the changelog in a separate key = > value > file (with the components of the build evr, the last packager id, etc). Do you mean that the key/value file will be committed to Git from inside Koji? Do the Koji builders have write access to Git? > commit the new build event timestamp in > the detached changelog file at %build time %build is executed once per arch, on different builders, so which builder's timestamp gets committed to Git? Björn Persson pgpGqB7XyDrRt.pgp Description: OpenPGP digital signatur ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Le 2020-07-02 11:21, Igor Raits a écrit : -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Thu, 2020-07-02 at 11:17 +0200, Nicolas Mailhot wrote: Le 2020-07-02 09:52, Florian Weimer a écrit : > * Nicolas Mailhot via devel: > > > > How do I let rpm generate the changelog automatically? > > > > This feature is not changelog generation, just changelog bumping > > on > > build events. You still need some other method to put non-build > > events > > in the changelog. > > What is “changelog bumping”? Why is it needed? What about release > bumping? Changelog bumping is the act of putting the actual release bump and build time in the changelog. With the change, the spec is able to self-compute its next release if the spec file evr is older or equal to the last build event. How does it know that "last build event"? The same process that commits a new state of the changelog file in sources, commits the date that was written in the changelog in a separate key = value file (with the components of the build evr, the last packager id, etc). That means, you can trim the detached changelog file (if you find the list of build events uninteresting), the SRPM will still remember to bump the next EVR to something above the last build (even if it does not appear in the changelog file). (That also means I could dispense with writing a parser for the custom timestamp format rpm changelogs use, and save the date in easy to parse RFC 3339/ ISO 8601 format) Regards, -- Nicolas Mailhot ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On Thu, 2020-07-02 at 11:17 +0200, Nicolas Mailhot wrote: > Le 2020-07-02 09:52, Florian Weimer a écrit : > > * Nicolas Mailhot via devel: > > > > > > How do I let rpm generate the changelog automatically? > > > > > > This feature is not changelog generation, just changelog bumping > > > on > > > build events. You still need some other method to put non-build > > > events > > > in the changelog. > > > > What is “changelog bumping”? Why is it needed? What about release > > bumping? > > Changelog bumping is the act of putting the actual release bump and > build time in the changelog. > > With the change, the spec is able to self-compute its next release if > the spec file evr is older or equal to the last build event. How does it know that "last build event"? > On build, it will both bump the release, without touching the spec > file > (that is release bumping) and commit the new build event timestamp in > the detached changelog file at %build time (that is changelog > bumping). > > > > The detached changelog is just one more file in SRPM sources, > > > which is > > > modified by rpmbuild at `%build` time with other files rpmbuild > > > modifies. The tricky part is to modify the source file as a > > > source > > > file > > > so rpmbuild adds the result to the produced SRPM (and, that does > > > not > > > work in mock right now, because mock serves the SRPM that existed > > > at > > > the start, not at the end of the build. Though it’s probably just > > > a > > > matter of getting mock to call again its SRPM creation method at > > > the > > > end of the build). > > > > > > The packager does not have to request the modification in his > > > spec, > > > it’s done as part of the various %auto_foo calls the change > > > introduced > > > > Can you list the relevant %auto macros explicitly somewhere? Is > > %autosetup included in the set of macros that trigger this > > behavior? > > %autosetup is not part of the new framework, all the new %auto entry > points have %auto_something name/ > > Auto release bumping and auto changelog bumping involve registering > some > processing in the preamble (to compute the next evr), in %sourcelist > (to > deal with the source files involved in saving state) in %build (to > commit the new data to disk once the build is ongoing) and in > %changelog > (to get rpmbuild to record the new changelog state in package > metadata) > > ie it registers processing in %auto_pkg, %auto_sources, %auto_build > and > %auto_changelog > > The bumping is done by the buildsys subsystem ie practically by > %new_package (called by %auto_pkg, directly or via %buildsys_pkg), by > %buildsys_sources (called by %auto_sources), %buildsys_build (called > by > %auto_build) and %buildsys_changelog (called by %auto_changelog). > > It’s done by the buildsys subsystem because the %buildsys subsystem > is > tasked with writing the SRPM header in the new %auto_call framework, > so > only it knows which of the various (sub)package epochs and versions > are > the ones that apply to the SRPM. > > This may seem a bit complex and convoluted, but that’s because > autobumping > > https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs > > is a small addition over the big %auto_macros change. > > https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs > > And it is small because the big change provides all the low-level > infra > to code such high level features easily. > > The big change was not done for autobumping. It’s only once I coded > it > for other packaging needs that I realized it made implementing > autobumping trivial (trivial to me after all the other changes, maybe > not so trivial for the average macro reviewer). > > Regards, > > -- > Nicolas Mailhot > ___ > packaging mailing list -- packag...@lists.fedoraproject.org > To unsubscribe send an email to > packaging-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: > https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org - -- Igor Raits -BEGIN PGP SIGNATURE- iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl79pwsACgkQEV1auJxc Hh5oixAAlyinJfmNKCVQcx/Kh11xb+MwW9UzGynhW1cTUAe1D8vslH0jtEJjJRRm nXMtIyNoj0ny5Uo+ddABdQ3V86qqh/U46K5XK2FbGPq9a0hmI254KgJDLt4hqtaT Dqw7+LK2jwbb63WBacsxJG6dGhvS9cOGxoxo+jMQ3uocLN1RrbTI/Du64i8d3Enk Jmu3v0YKm3V+VyRtal2O+BGphzANS0D0rodHMH/8zmcT50Mt41QMFl+16PPDBcsn qgyy/3tmruPmxUDCO5xFzJlA50qT5AMSWy8pOKqFdr+5hUaYW6rPkvXoC7uun08V FnW/XGVHHv7iwz2CUCqoLwzb6wlyKzOyjxh3RGTIt+FXz1AfQ2tZWSbSvlElBce9 eRMb+v5yoURHMYK4Iazy9HMZa2mp7lcXFWE7qkTxwVwClkQ1YuaCEeSIVpblFuFw l8w47QzcyGPwIi6GMqJyp5dpqLD15JetVIXQfF8/V5OCWMHNgqbbaef5Q9JNnc9K PQ31
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Le mercredi 01 juillet 2020 à 12:27 +0200, Dominik 'Rathann' Mierzejewski a écrit : Hi, > That's not detailed at all. You should provide at least one example > here (or a direct link to one somewhere else on the wiki). Thank you for your attention and kind review. yes the wiki page was completely insufficient, I did it at the last minute to honor deadlines. Please check if it is ok for you now. Anyway here are some answers I added to the wiki > > What's "autobumping" here The change will make packages that use the %auto_ redhat-rpm- config macros auto-bump and auto-changelog at the rpm level, in an infrastructure-independent way. The %auto_ framework is proposed in https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs In that context, auto-bumping means that a SRPM, produced in any compatible build system (that is, any build system that does not inhibit low-level rpmbuild behaviour), will rebuild by default to a release higher, than the last build release, in the next build system it is imported into, without any manual change to the SRPM source files. Auto-changelog means that the build event will also be traced in the rpm changelog (again, without any manual change). > and how is it better than rpmdev-bumpsec? Unlike rpmdev-bumpsec, the feature is automatic. It does not require explicit human action. Releases get bumped even if the human forgot a particular release has already been built. It does not rely on an external tool, nor requires this external tool to be able to parse a spec file (which can be difficult for heavily automated spec files like the ones that take advantage of %auto macros). A rebuild does not touch the spec file at all. That means, the spec files changes tracked by your favorite scm, will show only spec logic changes, without drowning those in no-logic-change build events. > [...] > > == How To Test == > > > > A redhat-rpm-config packages with the changes and some example > > packages are available in > > > > https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/ > > A diff with changes The current code state is visible in https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/commits/forge-with-patches It’s one small commit on top of the huge change queued in: https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/95 That PR can still evolve based on feedback and testing. Therefore, I can’t promise that the auto-bumping logic will always apply directly, just that it will look more or less this way after rebasing. I do not rebase it on every change to the other PR. This is very young code, there are probably lots of easy things to tidy up in there. However it works. > > Why is a separate "rpm-changelog.txt" file with manually maintained > changelog better than current manually maintained changelog inside > .spec? This change does not separates the changelog. The separation is already done in https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs that this change builds upon Without separation, we would lose the benefit of auto-bumping at the SCM level, since no-logic-change rebuilds would still result in a spec file change. Separation makes automation a lot easier since adding to the changelog is just pre-pending some lines, and does not require any knowledge of rpm syntax. Auto-bumping will add a "* date name evr" line on the next rebuild, so changelog additions can limit themselves to plain text descriptions of new changes at the top of the existing file. Separation is a requirement for auto-changelog bumping at the rpm level. Once rpmbuilt is lauched, it can not modify the processed spec file. Therefore making the changelog modifiable by the build process requires splitting it out of the spec file first. > How about using git commit log for changelog instead? This is a low level change that does not depend on any specific infrastructure, git included. It works directly at the rpm level. An infrastructure that uses git, can feed git commit events to the detached changelog file, using dumb or elaborate git commit hooks, and any other method it wants to implement. The auto-bump logic does not care, it will use the detached changelog file in the state it exists at the start of the build process. Because the logic catches all rebuilds, regular manual trimming of the lines that add no value is recommended. > [...] > > To get beautiful changelogs, you also need to add > > > > > > %buildsys_name Your name > > %buildsys_email Your email > > > > > > in ~/.rpmmacros > > What about having one macro called %buildsys_packager instead of two? > You're always using them together, anyway. It'd be similar to the > existing %packager macro, too. This is certainly doable and
Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal
Le mardi 30 juin 2020 à 21:33 +0200, Igor Raits a écrit :
> On Tue, 2020-06-30 at 15:19 -0400, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> >
> > The change will make those packages auto-bump and auto-changelog at
> > the rpm level, in an infrastructure-independent way.
>
> So how exactly is this supposed to work? From where will it get old
> changelog, how packagers will migrate to this, how does it affect
> reproducibility?
The changelog is just one file in SRPM sources, and bumping is done
from last build state which is just one key = value file in sources
(storing the evr components, the last built time, and last build
packager id).
In reproducible mode last evr and packager id and build time are
applied without bumping. You need to set
%reproducible_build = true (or anything except false) in your
~/.rpmmacros (or config_opts['macros'], or as rpmbuild --define flags).
The auto framework (and %new_package) split Release in separate
%{source_release} %{dist} and %{post_release} components, which makes
the implementation way easier than multiplexing things in a single
Release tag and trying to untangle the mess later.
A production implementation would probably split %{dist} in %{distcore}
and %{distprefix} (the .gitdatehash things we stuff in Releases and in
rpm changelogs as opposed to the fcX part we want to appear in Release
but not in changelogs). I know where the offending code is in fedora-
release and the split up is trivial to implement, but there’s no point
in worrying about this level of detail before the core of the feature
is approved (or not).
The implementation is really simple and easy, it took me two days to
write and test it because it reuses all the building blocks I had
already done for my other change (without those jungling all the bits
involved at various points of the spec file would have been challenging
to say the least).
>
> So are you asking mock and koji people to implement something? Did
> you
> talk to them before submitting this proposal?
>
> > * Mock issue:
> > https://github.com/rpm-software-management/mock/issues/599
I filled the mock issue to inform them. Again, this is a feature that
took me two days to code (it did not exist, even in thought, last
saturday). I was actually surprised at how easy it was to implement,
given the months if was discussed on this list.
At the mock level, there are two issues.
The main one (and only critical one) is that bumping MUST occur when
%build is executed, because a spectool or rpmbuild -bs is not a build
event, only a full build is. That means the SRPM produced by
rpmbuild -bs
is not the bumped SRPM, only the SRPM produced by
rpmbuild -ba
is bumped. My (imperfect) understanding of the mock issue is that mock
serves the first, not the second one at the end of the build process.
The second issue is that bumping changelog requires filling a builder
name and mail in the changelog line, and mock provides not easy way to
pass those to the build process.
If those two problems are lifted I see no special problem copr side
(except using the new mock plumbing to pass builder iname & mail to
mock).
For koji/fedpkg things are a bit more challenging because you will want
to back-commit the bumping to git once a build succeeds. Which is the
main point clime and me disagreed earlier on this year. Though, it is
not a show-stopper initially, a packager can back-commit manually if he
wants the bump recorded till tooling catches up.
While that adds constrains on the koji/git interface, that gives you a
bumping mecanism totally generic and independant of the build infra,
that does not rely on external python/ruby/whatever scripts to bum, and
does not require messing with someone else’s spec just to trace and
bump a rebuild.
Just importing an srpm from one system to another will preserve the
bumping state without any data loss.
> >
> > == Contingency Plan ==
> >
> > There is no contingency plan because the change will happen or not
> > at all.
>
> This is not true. If it will happen but then something will be
> entirely broken we need to revert it.
Thank you for your vote of confidence. I hope you realise that by that
yardstick, nothing would be accepted, including your own changes,
because something may always happen someday causing someone to revisit
something. And, the last time a problem occurred, it was traced to an
undocumented and unannounced rpm change that no one knew how to fix
rpm-side, and that you spent more energy proving it need not be fixed
than on constructive solution-finding.
I freely admit that my code sucks and is way worse than the perfect
code no one has written yet nor intends to write any day soon.
--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/e
