Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Tuesday, January 19, 2021 8:50:38 PM CET Dan Čermák wrote: > Dominik 'Rathann' Mierzejewski writes: > > > On Monday, 18 January 2021 at 23:29, Dan Čermák wrote: > >> clime writes: > > [...] > >> > But when you said "workaround", I was thinking that you actually saw > >> > the correct solution because "workaround" is imho used usually when > >> > someone can't or don't want to solve things the right way so he/she > >> > takes a shortcut. So I was curious what you think is "the right way" > >> > here. > >> > >> Imho the "right way" would be to integrate this into rpmbuild itself > >> instead of adding another layer on top of it. > > > > +1. Maybe it's time to introduce RPM spec file format versioning > > and say .spec files with e.g.: > > > > SPEC-Version: 2 > > > > should be pre-processed by rpmbuild first. > > When we go down that route, we might even think about throwing out m4 > altogether and using a different templating language. I think that m4 isn't used actually. :-) I though that it would be awesome if we could actually finish the m4-as-a-library concept [1] - and maybe teach RPM to use m4, one day. At least it sounds like a good experiment WRT macros (I wished to have something like that when I reached the "max-macro-buffer-size" in RPM, in m4 such limit shouldn't exist). [1] m4 v2.0 sources http://git.savannah.gnu.org/gitweb/?p=m4.git;a=shortlog;h=refs/heads/master Pavel > But that a very OT > discussion and would rather belong to the rpm development mailinglist. > > > Cheers, > > Dan > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Tue, Jan 19, 2021 at 2:51 PM Dan Čermák wrote: > > Dominik 'Rathann' Mierzejewski writes: > > > On Monday, 18 January 2021 at 23:29, Dan Čermák wrote: > >> clime writes: > > [...] > >> > But when you said "workaround", I was thinking that you actually saw > >> > the correct solution because "workaround" is imho used usually when > >> > someone can't or don't want to solve things the right way so he/she > >> > takes a shortcut. So I was curious what you think is "the right way" > >> > here. > >> > >> Imho the "right way" would be to integrate this into rpmbuild itself > >> instead of adding another layer on top of it. > > > > +1. Maybe it's time to introduce RPM spec file format versioning > > and say .spec files with e.g.: > > > > SPEC-Version: 2 > > > > should be pre-processed by rpmbuild first. > > When we go down that route, we might even think about throwing out m4 > altogether and using a different templating language. But that a very OT > discussion and would rather belong to the rpm development mailinglist. > > > Cheers, > > Dan The yak says "b-a-a-a-a-a-a-h-h-h-hack-pthui". For those of us familiar with the concept, "yak shaving" is when you pursue every exciting feature possible to avoid doing the actual work. It's a temptation when dealing with old, stable, familiar software. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
Dominik 'Rathann' Mierzejewski writes: > On Monday, 18 January 2021 at 23:29, Dan Čermák wrote: >> clime writes: > [...] >> > But when you said "workaround", I was thinking that you actually saw >> > the correct solution because "workaround" is imho used usually when >> > someone can't or don't want to solve things the right way so he/she >> > takes a shortcut. So I was curious what you think is "the right way" >> > here. >> >> Imho the "right way" would be to integrate this into rpmbuild itself >> instead of adding another layer on top of it. > > +1. Maybe it's time to introduce RPM spec file format versioning > and say .spec files with e.g.: > > SPEC-Version: 2 > > should be pre-processed by rpmbuild first. When we go down that route, we might even think about throwing out m4 altogether and using a different templating language. But that a very OT discussion and would rather belong to the rpm development mailinglist. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Monday, 18 January 2021 at 23:29, Dan Čermák wrote: > clime writes: [...] > > But when you said "workaround", I was thinking that you actually saw > > the correct solution because "workaround" is imho used usually when > > someone can't or don't want to solve things the right way so he/she > > takes a shortcut. So I was curious what you think is "the right way" > > here. > > Imho the "right way" would be to integrate this into rpmbuild itself > instead of adding another layer on top of it. +1. Maybe it's time to introduce RPM spec file format versioning and say .spec files with e.g.: SPEC-Version: 2 should be pre-processed by rpmbuild first. Regards, Dominik -- Fedora https://getfedora.org | RPM Fusion http://rpmfusion.org There should be a science of discontent. People need hard times and oppression to develop psychic muscles. -- from "Collected Sayings of Muad'Dib" by the Princess Irulan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
clime writes: > On Mon, 4 Jan 2021 at 20:43, Dan Čermák > wrote: >> >> clime writes: >> >> > ...snip... >> >> > >> >> > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, >> >> > pkg.spec.rpkg is a spec template >> >> >> >> This would be a viable workaround, but a workaround nevertheless. Since >> >> I am not frequently rebuilding Fedora rpms outside of mock, koji & copr, >> >> I cannot tell how much of a show-stopper that is though. At least I >> >> think that the benefits of the change need to be pretty big to outweigh >> >> this potential downside. >> > >> > If I may ask, why do you think using preproc-rpmspec is a workaround? >> >> I consider it a workaround, because it adds another step to the >> packaging process outside of koji & copr: it still works, but you have >> to go through extra steps. > > Yes, there is an extra step that would be wrapped in higher-level tooling. > > But when you said "workaround", I was thinking that you actually saw > the correct solution because "workaround" is imho used usually when > someone can't or don't want to solve things the right way so he/she > takes a shortcut. So I was curious what you think is "the right way" > here. Imho the "right way" would be to integrate this into rpmbuild itself instead of adding another layer on top of it. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
...snip... Btw. I posted a long comment here: https://pagure.io/fesco/issue/2532 basically trying to explain the proposal more and mention the use-cases where it would be useful. So, please, read if you are interested. I guess, if there is a further discussion it should be probably carried out here so that it is available for everyone. One more thing... it's interesting to note that 12 years ago, we had a very similar discussion to what we have today: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/I35MFMMIDO5FFY3ZFCXGE26PREPPWPCA/#DEJUEWXMAZND626D7ZOSOIFHGWSYGPAG - the problem of generating release by using build system vs. using just cvs/git was discussed there (Colin Walters vs. Jesse Keating). It would be interesting to know what people discussing the topic in the thread think about it today. clime ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Mon, 4 Jan 2021 at 20:43, Dan Čermák wrote: > > clime writes: > > > ...snip... > >> > > >> > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, > >> > pkg.spec.rpkg is a spec template > >> > >> This would be a viable workaround, but a workaround nevertheless. Since > >> I am not frequently rebuilding Fedora rpms outside of mock, koji & copr, > >> I cannot tell how much of a show-stopper that is though. At least I > >> think that the benefits of the change need to be pretty big to outweigh > >> this potential downside. > > > > If I may ask, why do you think using preproc-rpmspec is a workaround? > > I consider it a workaround, because it adds another step to the > packaging process outside of koji & copr: it still works, but you have > to go through extra steps. Yes, there is an extra step that would be wrapped in higher-level tooling. But when you said "workaround", I was thinking that you actually saw the correct solution because "workaround" is imho used usually when someone can't or don't want to solve things the right way so he/she takes a shortcut. So I was curious what you think is "the right way" here. > > > Cheers, > > Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
clime writes: > ...snip... >> > >> > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, >> > pkg.spec.rpkg is a spec template >> >> This would be a viable workaround, but a workaround nevertheless. Since >> I am not frequently rebuilding Fedora rpms outside of mock, koji & copr, >> I cannot tell how much of a show-stopper that is though. At least I >> think that the benefits of the change need to be pretty big to outweigh >> this potential downside. > > If I may ask, why do you think using preproc-rpmspec is a workaround? I consider it a workaround, because it adds another step to the packaging process outside of koji & copr: it still works, but you have to go through extra steps. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, Dec 17, 2020 at 09:14:40PM +0100, Fabio Valentini wrote: > On Thu, Dec 17, 2020 at 8:06 PM Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > As far as I can tell, this is the third implementation of generated > changelogs ... did the autorelease / autochangelog work that was even > already deployed in staging not go anywhere? If you're thinking about rpmautospec, I have in mind to submit it as a change proposal in the coming month. Fall has been quite busy after the data center move and I also did not want to have everyone check it and FESCo spend time on it in a timeframe where we couldn't put up with the work afterward. So my idea is to submit it in Q1 (Jan-March) to start on it (if approved) in Q2 (May-June). Pierre ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
...snip... > > > > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, > > pkg.spec.rpkg is a spec template > > This would be a viable workaround, but a workaround nevertheless. Since > I am not frequently rebuilding Fedora rpms outside of mock, koji & copr, > I cannot tell how much of a show-stopper that is though. At least I > think that the benefits of the change need to be pretty big to outweigh > this potential downside. If I may ask, why do you think using preproc-rpmspec is a workaround? > > > Cheers, > > Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
clime writes: > On Sat, 19 Dec 2020 at 20:07, Dan Čermák > wrote: >> >> clime writes: >> >> > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: >> >> >> >> On Thu, 17 Dec 2020 14:05:40 -0500 >> >> Ben Cotton wrote: >> >> >> >> 1. How does this affect users who download, maybe modify, and rebuild >> >> the SRPM? Can they continue to use rpmbuid and mock as they have >> >> been? Does the SRPM contain the pre-processed or post-processed spec >> >> file? >> > >> > They can use mock if the preprocessing will be enabled for the >> > respective chroots where it is enabled in Koji/Fedora. >> > They can't directly use rpmbuild for those packages that contain the >> > macros. But they can use rpkg/fedpkg to do the work. >> > Or preprocess spec first and then use rpmbuild. I am aware this is a >> > negative point of this change. >> >> This is a pretty big downside imho, as that means that building Fedora >> packages that use these new kinds of macros in other build systems will >> become impossible or at the very least, very, very difficult. There is >> quite some development going on in OBS (afaik e.g. Igor exported all >> Fedora Rust rpms to OBS for automated rebuilds) and enabling this >> preprocessing will make these packages FTBFS in OBS. >> > > It depends on how the srpms are being built and if Fedora DistGit is > used directly as the source (as Adam has said also). If there is such > a possible breakage, we can look at fixing it in advance. > > The tooling that implements preprocessing has minimal requirements > (git or git-core, bash, python, libgit2-devel, rpm-devel) so there > should be a very low barrier for entry for any environment that would > need it. > >> Don't get me wrong, I am not opposed to this proposal per-se. But as far >> as I recall, many people were pretty upset about modular packages being >> effectively only buildable in Fedora's infra and nowhere else. And I'd >> very much like not to repeat this. >> >> > While having an option to use rpmbuild directly to build srpm/rpm from >> > a dist-git repo is nice, I would say that fedpkg or mock are the main >> > interfaces to do this. >> > I know this answer won't satisfy everyone. >> >> Indeed. I think there *should* be at least a way how to produce a srpm >> that can be rebuild *without* having access to Koji, mock and fedpkg >> (ideally by our own infrastructure). > > Well, that excludes lots of options already :). One can also use > preproc-rpmspec tool to get a rendered spec file (this is what mock > uses). > > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, > pkg.spec.rpkg is a spec template This would be a viable workaround, but a workaround nevertheless. Since I am not frequently rebuilding Fedora rpms outside of mock, koji & copr, I cannot tell how much of a show-stopper that is though. At least I think that the benefits of the change need to be pretty big to outweigh this potential downside. Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
Adam Williamson writes: > On Sat, 2020-12-19 at 20:07 +0100, Dan Čermák wrote: >> clime writes: >> >> > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: >> > > >> > > On Thu, 17 Dec 2020 14:05:40 -0500 >> > > Ben Cotton wrote: >> > > >> > > 1. How does this affect users who download, maybe modify, and rebuild >> > > the SRPM? Can they continue to use rpmbuid and mock as they have >> > > been? Does the SRPM contain the pre-processed or post-processed spec >> > > file? >> > >> > They can use mock if the preprocessing will be enabled for the >> > respective chroots where it is enabled in Koji/Fedora. >> > They can't directly use rpmbuild for those packages that contain the >> > macros. But they can use rpkg/fedpkg to do the work. >> > Or preprocess spec first and then use rpmbuild. I am aware this is a >> > negative point of this change. >> >> This is a pretty big downside imho, as that means that building Fedora >> packages that use these new kinds of macros in other build systems will >> become impossible or at the very least, very, very difficult. There is >> quite some development going on in OBS (afaik e.g. Igor exported all >> Fedora Rust rpms to OBS for automated rebuilds) and enabling this >> preprocessing will make these packages FTBFS in OBS. > > I mean, only if you're sourcing from dist-git? Yes, but that's how one would use the sources when developing/testing on OBS (or at least I would). Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 17:03, clime wrote: > > On Fri, 18 Dec 2020 at 16:23, James Szinger wrote: > > > > On Fri, 18 Dec 2020 00:51:49 +0100 > > clime wrote: > > > > > Well, the users here are still packagers here no? I thought the "User" > > > in the title means "end user" who shouldn't be affected by it. Maybe > > > Ben can clarify this. > > > > I am making a distinction between Fedora packagers who use the Fedora > > infrastructure to build RPMs for the Fedora repositories, and second > > and third party packagers who use their own infrastructure. External > > packagers count as ‘users’ for the purposes of change proposals, > > especially infrastructure changes such as this. Many change proposals > > have no effect for external packagers, or have the same effect as for > > the general user base. This proposal, however, seems potentially > > disruptive for external packagers, and I want to see those issues > > specifically addressed in the change proposal. > > > > I will advocate that external packagers are of strategic importance to > > the Fedora Project. The software they provide encourages the adoption > > of Fedora. They form a pool a potential Fedora packagers. They are > > the technical experts that support the local users. They are > > fundamental to the Fedora Mission: > > > > Fedora creates an innovative platform for hardware, clouds, and > > containers that enables software developers and community members > > to build tailored solutions for their users. > > > > Supporting external packagers drives the Four Foundations: Freedom, > > Friends, Features, and First. > > I agree the change proposal should address those issues. I have updated the change description (https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing) with the feedback I've been given in this thread. I have also expanded the "User experience" section so that it addresses the mentioned issues of having spec file templates (i.e. something which is not understood by rpmbuild) in dist-git instead of just plain spec files. Also some links to interesting past discussions about the dynamic changelog and release generation were added into the description (the problem actually has a very long history and was discussed many times). Hopefully, I didn't forget to mention something (will add it later if I did). Best regards! clime > > > > > Jim > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 21. 12. 20 2:38, clime wrote: On Sun, 20 Dec 2020 at 11:39, Miro Hrončok wrote: On 12/20/20 1:38 AM, clime wrote: I view this proposal as a risk that the spec files will look a bit more weird, and the spec files maintenance will start diverging too much. Everything happening for an overestimated triviality as IMO the release/changelog is [1]. Well, even if this change is accepted, it doesn't mean people will use the feature so if the feature is overkill or it is generally bad, it will just die on its own. No. In fact, even if one maintainer keep using this (e.g. you), as a provenpackager I still need to be able to deal with that. So, while this is "opt-in only" for the individual packagers, it impacts all our provenackagers (and some of our downstreams as well). In some cases, the effect of this change on the work of ProvenPackagers will be positive. Manual release bumping (because of soname-bump or a hotfix change) is easier because you can just call `fedpkg tag` and write a message instead of manually messing with a spec file. And in case a script is used, then it's exactly the same amount of work (calling the script). It is not easier, because you need to special case it in all scripts *and* manual work. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sun, 20 Dec 2020 at 11:39, Miro Hrončok wrote: > > On 12/20/20 1:38 AM, clime wrote: > >> I view this proposal as a risk that the spec files will look a bit more > >> weird, and the spec files maintenance will start diverging too much. > >> Everything happening for an overestimated triviality as IMO > >> the release/changelog is [1]. > > Well, even if this change is accepted, it doesn't mean people will use > > the feature so if the feature is overkill or it is generally bad, it > > will just die on its own. > > No. In fact, even if one maintainer keep using this (e.g. you), as a > provenpackager I still need to be able to deal with that. So, while this is > "opt-in only" for the individual packagers, it impacts all our provenackagers > (and some of our downstreams as well). In some cases, the effect of this change on the work of ProvenPackagers will be positive. Manual release bumping (because of soname-bump or a hotfix change) is easier because you can just call `fedpkg tag` and write a message instead of manually messing with a spec file. And in case a script is used, then it's exactly the same amount of work (calling the script). > > As such, I think this is not worth it. > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/20/20 1:38 AM, clime wrote: I view this proposal as a risk that the spec files will look a bit more weird, and the spec files maintenance will start diverging too much. Everything happening for an overestimated triviality as IMO the release/changelog is [1]. Well, even if this change is accepted, it doesn't mean people will use the feature so if the feature is overkill or it is generally bad, it will just die on its own. No. In fact, even if one maintainer keep using this (e.g. you), as a provenpackager I still need to be able to deal with that. So, while this is "opt-in only" for the individual packagers, it impacts all our provenackagers (and some of our downstreams as well). As such, I think this is not worth it. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sun, 20 Dec 2020 at 00:23, Pavel Raiskup wrote:
>
> On Thursday, December 17, 2020 8:05:40 PM CET Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing
> >
> > == Summary ==
> > This change should enable an opt-in spec file preprocessor in Fedora
> > infrastructure for the benefit of packagers. The preprocessor allows
> > some very neat tricks that were impossible before, for example
> > generate changelog and release automatically from git metadata or pack
> > the entire dist-git repository into an rpm-source tarball (effectively
> > allowing unpacked repos to live in DistGit).
>
> It would be nice to see this in some concrete example. E.g. in some
> 'private-rpkg-preview' branch in some of the existing Fedora packages, so
> we can make a clearer idea about what this causes with real spec file
> readability (diff) and the initial newcomer barrier. So we could e.g.
> checkout that branch, and try to build the package. And see the real
> pros/cons.
Is this example good?:
https://pagure.io/hello_rpkg/blob/master/f/hello_rpkg.spec.rpkg
It can be built in Copr by SCM rpkg method.
There is also:
https://pagure.io/hello_rpkg_release/blob/master/f/hello_rpkg.spec.rpkg
with a dynamic release, the macro used is actually v3 macro, not yet
supported by Copr.
v3 macros are being introduced by this change. One can try to build
that spec either by rpkg-util (from Copr repo referenced in the
change) or by mock with rpkg_preprocessor enabled
I think pros/cons can be also revealed by discussion which is perhaps
better because it is shared...
>
> > == Benefit to Fedora ==
> >
> > This change offers solution to some long-standing issues in Fedora
> > around packaging (i.e. automatic release and changelog generation)
>
> I personally wouldn't overestimate these issues, at least according to the
> questionnaire I tried to do some time ago [1, 2] not many maintainers were
> interested in the problem to even vote (and I was not surprised).
Well but there were also quite many mailing lists threads indicating
that people are interested in the topic. I cannot say if people are
interested or not...
>
> These problems have trivial work-arounds/solutions, discussed in [1].
Idk If I want to get into the whole theory behind this problem. If
there is a trivial solution to the problem (i.e. automatic release and
changelog in this case), then perhaps it should be stated here and it
can be taken into account when deciding about this change. I would
like to note that this change tries to solve the problem but also in a
way that allows for more applications in future (e.g. unpacked repos),
which might be considered a potential advantage.
>
> > while also offering some interesting future options (for example
> > unpacked dist-git repos).
>
> I think that it would be good to consider tito as alternative, when we
> speak about binding spec with git-archive feature. Do you think it would be
> possible to allow tito in future?
Maybe...If people want to use tito and ProvenPackagers and relengs are
okay with that...it's not really something I can decide.
> There's also some Packit-team feature named source-git, is this related?
It is related with regards to the "unpacked repos" that I described earlier.
> -
> Honestly, in general, I don't like tito, and I don't like rpkg much more.
> Both are probably better for upstream development and release processes
> (tito is more standard and convenient IMO) than some custom scripting.
> But people need to know deeply the use-cases (and even implementation) to
> compare.
>
> Also note that we used to have problems [IIRC 3, 4] in Copr builds from
> Fedora DistGit -- as the '{{{' syntax collided with some of the existing
> packages.
Yes, there were two of them (last time I checked):
python-dns-lexicon.spec - uses {{{ }}} in comments
python-suds.spec - uses {{{ }}} in changelog at one place
But the feature is opt-in so a maintainer can tweak his/her spec file if needed.
>
> I view this proposal as a risk that the spec files will look a bit more
> weird, and the spec files maintenance will start diverging too much.
> Everything happening for an overestimated triviality as IMO
> the release/changelog is [1].
Well, even if this change is accepted, it doesn't mean people will use
the feature so if the feature is overkill or it is generally bad, it
will just die on its own.
>
> [1]
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/2G6OSOSM76O2V6K4COIE2QHNXIBSXPFR/
> [2]
> https://docs.google.com/forms/d/183dSFIN-i9rauEZ0_gtDia7dzkeX-hzfX0ncpqFMYxw/edit#responses
> [3] https://pagure.io/copr/copr/issue/798
> [4] https://pagure.io/copr/copr/issue/1219
>
> Pavel
>
>
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
>
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thursday, December 17, 2020 8:05:40 PM CET Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing
>
> == Summary ==
> This change should enable an opt-in spec file preprocessor in Fedora
> infrastructure for the benefit of packagers. The preprocessor allows
> some very neat tricks that were impossible before, for example
> generate changelog and release automatically from git metadata or pack
> the entire dist-git repository into an rpm-source tarball (effectively
> allowing unpacked repos to live in DistGit).
It would be nice to see this in some concrete example. E.g. in some
'private-rpkg-preview' branch in some of the existing Fedora packages, so
we can make a clearer idea about what this causes with real spec file
readability (diff) and the initial newcomer barrier. So we could e.g.
checkout that branch, and try to build the package. And see the real pros/cons.
> == Benefit to Fedora ==
>
> This change offers solution to some long-standing issues in Fedora
> around packaging (i.e. automatic release and changelog generation)
I personally wouldn't overestimate these issues, at least according to the
questionnaire I tried to do some time ago [1, 2] not many maintainers were
interested in the problem to even vote (and I was not surprised).
These problems have trivial work-arounds/solutions, discussed in [1].
> while also offering some interesting future options (for example
> unpacked dist-git repos).
I think that it would be good to consider tito as alternative, when we
speak about binding spec with git-archive feature. Do you think it would be
possible to allow tito in future? There's also some Packit-team feature
named source-git, is this related?
-
Honestly, in general, I don't like tito, and I don't like rpkg much more.
Both are probably better for upstream development and release processes
(tito is more standard and convenient IMO) than some custom scripting.
But people need to know deeply the use-cases (and even implementation) to
compare.
Also note that we used to have problems [IIRC 3, 4] in Copr builds from
Fedora DistGit -- as the '{{{' syntax collided with some of the existing
packages.
I view this proposal as a risk that the spec files will look a bit more
weird, and the spec files maintenance will start diverging too much.
Everything happening for an overestimated triviality as IMO
the release/changelog is [1].
[1]
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/2G6OSOSM76O2V6K4COIE2QHNXIBSXPFR/
[2]
https://docs.google.com/forms/d/183dSFIN-i9rauEZ0_gtDia7dzkeX-hzfX0ncpqFMYxw/edit#responses
[3] https://pagure.io/copr/copr/issue/798
[4] https://pagure.io/copr/copr/issue/1219
Pavel
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sat, 19 Dec 2020 at 20:57, clime wrote: > > On Sat, 19 Dec 2020 at 20:07, Dan Čermák > wrote: > > > > clime writes: > > > > > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: > > >> > > >> On Thu, 17 Dec 2020 14:05:40 -0500 > > >> Ben Cotton wrote: > > >> > > >> 1. How does this affect users who download, maybe modify, and rebuild > > >> the SRPM? Can they continue to use rpmbuid and mock as they have > > >> been? Does the SRPM contain the pre-processed or post-processed spec > > >> file? > > > > > > They can use mock if the preprocessing will be enabled for the > > > respective chroots where it is enabled in Koji/Fedora. > > > They can't directly use rpmbuild for those packages that contain the > > > macros. But they can use rpkg/fedpkg to do the work. > > > Or preprocess spec first and then use rpmbuild. I am aware this is a > > > negative point of this change. > > > > This is a pretty big downside imho, as that means that building Fedora > > packages that use these new kinds of macros in other build systems will > > become impossible or at the very least, very, very difficult. There is > > quite some development going on in OBS (afaik e.g. Igor exported all > > Fedora Rust rpms to OBS for automated rebuilds) and enabling this > > preprocessing will make these packages FTBFS in OBS. > > > > It depends on how the srpms are being built and if Fedora DistGit is > used directly as the source (as Adam has said also). If there is such > a possible breakage, we can look at fixing it in advance. > > The tooling that implements preprocessing has minimal requirements > (git or git-core, bash, python, libgit2-devel, rpm-devel) so there > should be a very low barrier for entry for any environment that would > need it. > > > Don't get me wrong, I am not opposed to this proposal per-se. But as far > > as I recall, many people were pretty upset about modular packages being > > effectively only buildable in Fedora's infra and nowhere else. And I'd > > very much like not to repeat this. > > > > > While having an option to use rpmbuild directly to build srpm/rpm from > > > a dist-git repo is nice, I would say that fedpkg or mock are the main > > > interfaces to do this. > > > I know this answer won't satisfy everyone. > > > > Indeed. I think there *should* be at least a way how to produce a srpm > > that can be rebuild *without* having access to Koji, mock and fedpkg > > (ideally by our own infrastructure). > Well, also once you produce an srpm, it will be just a generic srpm that can be rebuilt in any rpm-compatible system. > Well, that excludes lots of options already :). One can also use > preproc-rpmspec tool to get a rendered spec file (this is what mock > uses). > > $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, > pkg.spec.rpkg is a spec template > > > > > > > Cheers, > > > > Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sat, 19 Dec 2020 at 20:07, Dan Čermák wrote: > > clime writes: > > > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: > >> > >> On Thu, 17 Dec 2020 14:05:40 -0500 > >> Ben Cotton wrote: > >> > >> 1. How does this affect users who download, maybe modify, and rebuild > >> the SRPM? Can they continue to use rpmbuid and mock as they have > >> been? Does the SRPM contain the pre-processed or post-processed spec > >> file? > > > > They can use mock if the preprocessing will be enabled for the > > respective chroots where it is enabled in Koji/Fedora. > > They can't directly use rpmbuild for those packages that contain the > > macros. But they can use rpkg/fedpkg to do the work. > > Or preprocess spec first and then use rpmbuild. I am aware this is a > > negative point of this change. > > This is a pretty big downside imho, as that means that building Fedora > packages that use these new kinds of macros in other build systems will > become impossible or at the very least, very, very difficult. There is > quite some development going on in OBS (afaik e.g. Igor exported all > Fedora Rust rpms to OBS for automated rebuilds) and enabling this > preprocessing will make these packages FTBFS in OBS. > It depends on how the srpms are being built and if Fedora DistGit is used directly as the source (as Adam has said also). If there is such a possible breakage, we can look at fixing it in advance. The tooling that implements preprocessing has minimal requirements (git or git-core, bash, python, libgit2-devel, rpm-devel) so there should be a very low barrier for entry for any environment that would need it. > Don't get me wrong, I am not opposed to this proposal per-se. But as far > as I recall, many people were pretty upset about modular packages being > effectively only buildable in Fedora's infra and nowhere else. And I'd > very much like not to repeat this. > > > While having an option to use rpmbuild directly to build srpm/rpm from > > a dist-git repo is nice, I would say that fedpkg or mock are the main > > interfaces to do this. > > I know this answer won't satisfy everyone. > > Indeed. I think there *should* be at least a way how to produce a srpm > that can be rebuild *without* having access to Koji, mock and fedpkg > (ideally by our own infrastructure). Well, that excludes lots of options already :). One can also use preproc-rpmspec tool to get a rendered spec file (this is what mock uses). $ preproc-rpmspec pkg.spec.rpkg # prints rendered spec to stdout, pkg.spec.rpkg is a spec template > > > Cheers, > > Dan ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sat, Dec 19, 2020 at 2:21 PM Adam Williamson wrote: > > On Sat, 2020-12-19 at 20:07 +0100, Dan Čermák wrote: > > clime writes: > > > > > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: > > > > > > > > On Thu, 17 Dec 2020 14:05:40 -0500 > > > > Ben Cotton wrote: > > > > > > > > 1. How does this affect users who download, maybe modify, and rebuild > > > > the SRPM? Can they continue to use rpmbuid and mock as they have > > > > been? Does the SRPM contain the pre-processed or post-processed spec > > > > file? > > > > > > They can use mock if the preprocessing will be enabled for the > > > respective chroots where it is enabled in Koji/Fedora. > > > They can't directly use rpmbuild for those packages that contain the > > > macros. But they can use rpkg/fedpkg to do the work. > > > Or preprocess spec first and then use rpmbuild. I am aware this is a > > > negative point of this change. > > > > This is a pretty big downside imho, as that means that building Fedora > > packages that use these new kinds of macros in other build systems will > > become impossible or at the very least, very, very difficult. There is > > quite some development going on in OBS (afaik e.g. Igor exported all > > Fedora Rust rpms to OBS for automated rebuilds) and enabling this > > preprocessing will make these packages FTBFS in OBS. > > I mean, only if you're sourcing from dist-git? > > The .src.rpm will have the post-processed spec file and should rebuild > anywhere without issues, presumably. They're pushed from Dist-Git directly using a script I wrote: https://pagure.io/obs-packaging-scripts/blob/master/f/distgit-obsimport.py -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sat, 2020-12-19 at 20:07 +0100, Dan Čermák wrote: > clime writes: > > > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: > > > > > > On Thu, 17 Dec 2020 14:05:40 -0500 > > > Ben Cotton wrote: > > > > > > 1. How does this affect users who download, maybe modify, and rebuild > > > the SRPM? Can they continue to use rpmbuid and mock as they have > > > been? Does the SRPM contain the pre-processed or post-processed spec > > > file? > > > > They can use mock if the preprocessing will be enabled for the > > respective chroots where it is enabled in Koji/Fedora. > > They can't directly use rpmbuild for those packages that contain the > > macros. But they can use rpkg/fedpkg to do the work. > > Or preprocess spec first and then use rpmbuild. I am aware this is a > > negative point of this change. > > This is a pretty big downside imho, as that means that building Fedora > packages that use these new kinds of macros in other build systems will > become impossible or at the very least, very, very difficult. There is > quite some development going on in OBS (afaik e.g. Igor exported all > Fedora Rust rpms to OBS for automated rebuilds) and enabling this > preprocessing will make these packages FTBFS in OBS. I mean, only if you're sourcing from dist-git? The .src.rpm will have the post-processed spec file and should rebuild anywhere without issues, presumably. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
clime writes: > On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: >> >> On Thu, 17 Dec 2020 14:05:40 -0500 >> Ben Cotton wrote: >> >> 1. How does this affect users who download, maybe modify, and rebuild >> the SRPM? Can they continue to use rpmbuid and mock as they have >> been? Does the SRPM contain the pre-processed or post-processed spec >> file? > > They can use mock if the preprocessing will be enabled for the > respective chroots where it is enabled in Koji/Fedora. > They can't directly use rpmbuild for those packages that contain the > macros. But they can use rpkg/fedpkg to do the work. > Or preprocess spec first and then use rpmbuild. I am aware this is a > negative point of this change. This is a pretty big downside imho, as that means that building Fedora packages that use these new kinds of macros in other build systems will become impossible or at the very least, very, very difficult. There is quite some development going on in OBS (afaik e.g. Igor exported all Fedora Rust rpms to OBS for automated rebuilds) and enabling this preprocessing will make these packages FTBFS in OBS. Don't get me wrong, I am not opposed to this proposal per-se. But as far as I recall, many people were pretty upset about modular packages being effectively only buildable in Fedora's infra and nowhere else. And I'd very much like not to repeat this. > While having an option to use rpmbuild directly to build srpm/rpm from > a dist-git repo is nice, I would say that fedpkg or mock are the main > interfaces to do this. > I know this answer won't satisfy everyone. Indeed. I think there *should* be at least a way how to produce a srpm that can be rebuild *without* having access to Koji, mock and fedpkg (ideally by our own infrastructure). Cheers, Dan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Sat, 19 Dec 2020 at 03:02, Luya Tshimbalanga wrote: > > > On 2020-12-18 11:13 a.m., Robbie Harwood wrote: > > clime writes: > > > >> On Fri, 18 Dec 2020 at 18:20, Robbie Harwood wrote: > >>> Robert-André Mauchin writes: > >>> > On 12/18/20 3:52 PM, James Szinger wrote: > > No. One can also download the sources from upstream using spectool or > > similar, even wget or curl. My local work flow is typically get or > > create spec file and patches, spectool -g, rpmbuild -bs, mock. > > > Unrelated to the topic at hand, but why do people still use rpmbuild -bs > instead of using a fedpkg mockbuild? You get a clean environment to > build and you don't have to install tons of devel packages on your > system. > >>> For me it's speed. Yes, mock gives a clean environment, but I'd rather > >>> not use it if I don't have to: the tradeoff is I don't have to *wait* > >>> for the mock to go get the tons of devel packages (and generally for > >>> repo/dnf slowness) - they're already installed on my system. > >> But you have fedpkg installed, right? I think fedpkg srpm should do a > >> good job as well. > > Probably, but that wasn't the question - the question was about > > mockbuild. > > > > Thanks, > > --Robbie > Mockbuild stores packages in /var/lib/mock and will skip those already > downloaded. You can configure dnf (slowness is a illusion because the > package manager does many task as verify the security and possible > update packages) to only use cache when needed at the cost of getting > outdated packages. Make sure to select the fastest mirror for effectiveness Well, I also feel dnf is very slow, especially on my current connection now, it takes about 10 minutes to spawn a new container and install a package. It should be much faster once it doesn't download filelists.xml unless needed. > > -- > > Luya Tshimbalanga > Fedora Design Team > Fedora Design Suite maintainer > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 2020-12-18 11:13 a.m., Robbie Harwood wrote: clime writes: On Fri, 18 Dec 2020 at 18:20, Robbie Harwood wrote: Robert-André Mauchin writes: On 12/18/20 3:52 PM, James Szinger wrote: No. One can also download the sources from upstream using spectool or similar, even wget or curl. My local work flow is typically get or create spec file and patches, spectool -g, rpmbuild -bs, mock. Unrelated to the topic at hand, but why do people still use rpmbuild -bs instead of using a fedpkg mockbuild? You get a clean environment to build and you don't have to install tons of devel packages on your system. For me it's speed. Yes, mock gives a clean environment, but I'd rather not use it if I don't have to: the tradeoff is I don't have to *wait* for the mock to go get the tons of devel packages (and generally for repo/dnf slowness) - they're already installed on my system. But you have fedpkg installed, right? I think fedpkg srpm should do a good job as well. Probably, but that wasn't the question - the question was about mockbuild. Thanks, --Robbie Mockbuild stores packages in /var/lib/mock and will skip those already downloaded. You can configure dnf (slowness is a illusion because the package manager does many task as verify the security and possible update packages) to only use cache when needed at the cost of getting outdated packages. Make sure to select the fastest mirror for effectiveness -- Luya Tshimbalanga Fedora Design Team Fedora Design Suite maintainer ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
clime writes: > On Fri, 18 Dec 2020 at 18:20, Robbie Harwood wrote: >> >> Robert-André Mauchin writes: >> >> > On 12/18/20 3:52 PM, James Szinger wrote: >> >> >> >> No. One can also download the sources from upstream using spectool or >> >> similar, even wget or curl. My local work flow is typically get or >> >> create spec file and patches, spectool -g, rpmbuild -bs, mock. >> >> >> > >> > Unrelated to the topic at hand, but why do people still use rpmbuild -bs >> > instead of using a fedpkg mockbuild? You get a clean environment to >> > build and you don't have to install tons of devel packages on your system. >> >> For me it's speed. Yes, mock gives a clean environment, but I'd rather >> not use it if I don't have to: the tradeoff is I don't have to *wait* >> for the mock to go get the tons of devel packages (and generally for >> repo/dnf slowness) - they're already installed on my system. > > But you have fedpkg installed, right? I think fedpkg srpm should do a > good job as well. Probably, but that wasn't the question - the question was about mockbuild. Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 18:20, Robbie Harwood wrote: > > Robert-André Mauchin writes: > > > On 12/18/20 3:52 PM, James Szinger wrote: > >> > >> No. One can also download the sources from upstream using spectool or > >> similar, even wget or curl. My local work flow is typically get or > >> create spec file and patches, spectool -g, rpmbuild -bs, mock. > >> > > > > Unrelated to the topic at hand, but why do people still use rpmbuild -bs > > instead of using a fedpkg mockbuild? You get a clean environment to > > build and you don't have to install tons of devel packages on your system. > > For me it's speed. Yes, mock gives a clean environment, but I'd rather > not use it if I don't have to: the tradeoff is I don't have to *wait* > for the mock to go get the tons of devel packages (and generally for > repo/dnf slowness) - they're already installed on my system. But you have fedpkg installed, right? I think fedpkg srpm should do a good job as well. > > Thanks, > --Robbie > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
Robert-André Mauchin writes: > On 12/18/20 3:52 PM, James Szinger wrote: >> >> No. One can also download the sources from upstream using spectool or >> similar, even wget or curl. My local work flow is typically get or >> create spec file and patches, spectool -g, rpmbuild -bs, mock. >> > > Unrelated to the topic at hand, but why do people still use rpmbuild -bs > instead of using a fedpkg mockbuild? You get a clean environment to > build and you don't have to install tons of devel packages on your system. For me it's speed. Yes, mock gives a clean environment, but I'd rather not use it if I don't have to: the tradeoff is I don't have to *wait* for the mock to go get the tons of devel packages (and generally for repo/dnf slowness) - they're already installed on my system. Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
Hello, On Fri, Dec 18, 2020 at 5:53 PM Robert-André Mauchin wrote: > > On 12/18/20 3:52 PM, James Szinger wrote: > > > > No. One can also download the sources from upstream using spectool or > > similar, even wget or curl. My local work flow is typically get or > > create spec file and patches, spectool -g, rpmbuild -bs, mock. > > > > Unrelated to the topic at hand, but why do people still use rpmbuild -bs > instead of using a fedpkg mockbuild? You get a clean environment to > build and you don't have to install tons of devel packages on your system. I think 'rpmbuild -bs' is the quickest way to a source rpm package and it doesn't require any dependencies to be installed on your system. My workflow is almost identical to that of James and that first srpm might get reused multiple times before I get the one from mock. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/18/20 3:52 PM, James Szinger wrote: No. One can also download the sources from upstream using spectool or similar, even wget or curl. My local work flow is typically get or create spec file and patches, spectool -g, rpmbuild -bs, mock. Unrelated to the topic at hand, but why do people still use rpmbuild -bs instead of using a fedpkg mockbuild? You get a clean environment to build and you don't have to install tons of devel packages on your system. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 15:53, James Szinger wrote: > > On Fri, 18 Dec 2020 03:04:01 +0100 > clime wrote: > > > I wouldn't call it "deprecating rpmbuild". That's certainly not at all > > my intention. > > > > As a side-point, I think the cases where bare rpmbuild is used to > > build an rpm/srpm from a dist-git repo are rather limited because you > > probably need to first download sources from lookaside cache so you > > probably need fedpkg/rpkg/centpkg/rfpkg or a similar dedicated tool. > > These tools then offer the `srpm` and `local` commands so It would > > make sense to rather use these commands or mock for subsequent > > srpm/rpm building. > > No. One can also download the sources from upstream using spectool or > similar, even wget or curl. My local work flow is typically get or > create spec file and patches, spectool -g, rpmbuild -bs, mock. > > I could not find simple instructions for the *pkg tools the last time > I looked. It should be just `fedpkg srpm` for a Fedora package. Or `rpkg srpm` with rpkg-util tool. spectool could contain native support for preprocessing as well, I will look at opening a PR for it. I think with rpmbuild something like: `rpmbuild --define "%_sourcedir $PWD" -bs file.spec` is needed, right? There is also yum/dnf builddep command that I wanted to look at. > > Jim > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 16:23, James Szinger wrote: > > On Fri, 18 Dec 2020 00:51:49 +0100 > clime wrote: > > > Well, the users here are still packagers here no? I thought the "User" > > in the title means "end user" who shouldn't be affected by it. Maybe > > Ben can clarify this. > > I am making a distinction between Fedora packagers who use the Fedora > infrastructure to build RPMs for the Fedora repositories, and second > and third party packagers who use their own infrastructure. External > packagers count as ‘users’ for the purposes of change proposals, > especially infrastructure changes such as this. Many change proposals > have no effect for external packagers, or have the same effect as for > the general user base. This proposal, however, seems potentially > disruptive for external packagers, and I want to see those issues > specifically addressed in the change proposal. > > I will advocate that external packagers are of strategic importance to > the Fedora Project. The software they provide encourages the adoption > of Fedora. They form a pool a potential Fedora packagers. They are > the technical experts that support the local users. They are > fundamental to the Fedora Mission: > > Fedora creates an innovative platform for hardware, clouds, and > containers that enables software developers and community members > to build tailored solutions for their users. > > Supporting external packagers drives the Four Foundations: Freedom, > Friends, Features, and First. I agree the change proposal should address those issues. > > Jim > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 00:51:49 +0100 clime wrote: > Well, the users here are still packagers here no? I thought the "User" > in the title means "end user" who shouldn't be affected by it. Maybe > Ben can clarify this. I am making a distinction between Fedora packagers who use the Fedora infrastructure to build RPMs for the Fedora repositories, and second and third party packagers who use their own infrastructure. External packagers count as ‘users’ for the purposes of change proposals, especially infrastructure changes such as this. Many change proposals have no effect for external packagers, or have the same effect as for the general user base. This proposal, however, seems potentially disruptive for external packagers, and I want to see those issues specifically addressed in the change proposal. I will advocate that external packagers are of strategic importance to the Fedora Project. The software they provide encourages the adoption of Fedora. They form a pool a potential Fedora packagers. They are the technical experts that support the local users. They are fundamental to the Fedora Mission: Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users. Supporting external packagers drives the Four Foundations: Freedom, Friends, Features, and First. Jim ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 03:04:01 +0100 clime wrote: > I wouldn't call it "deprecating rpmbuild". That's certainly not at all > my intention. > > As a side-point, I think the cases where bare rpmbuild is used to > build an rpm/srpm from a dist-git repo are rather limited because you > probably need to first download sources from lookaside cache so you > probably need fedpkg/rpkg/centpkg/rfpkg or a similar dedicated tool. > These tools then offer the `srpm` and `local` commands so It would > make sense to rather use these commands or mock for subsequent > srpm/rpm building. No. One can also download the sources from upstream using spectool or similar, even wget or curl. My local work flow is typically get or create spec file and patches, spectool -g, rpmbuild -bs, mock. I could not find simple instructions for the *pkg tools the last time I looked. Jim ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
Dne pá 18. 12. 2020 10:52 dop. uživatel Miro Hrončok napsal: > On 12/18/20 1:19 AM, clime wrote: > >> I'd very much like to understand the impact of this on the following: > >> > >> > >> 1) Provenpackagers doing mass spec changes/updates. > > If the mass spec change/update doesn't involve an rpkg macro, then > > there is no difference. > > I don't understand how there is no difference. The spec "bare" spec file > is no > longer there. How do I parse it? How do I grep it? How do I sed it? > But the spec template is there. And the template can be modified. You can grep it and sed it, you cannot directly parse it with rpm without preprocessing it first. > >> 2) Provenpackagers and/or RelEngs doing (targeted) mass rebuilds. > > There should be no impact here. If the source git repo stays the same, > > then the same srpm as for a previous build will be produced. > > I don't understand you answer. What does this have to do with "source git > repo > stays the same" and "same srpm"? The release must be bumped, they are not > the same. > Yes, you are right. I was thinking along the lines of a feature that doesn't exist, which is inserting buildID into built rpms (not srpms) and making that an effective part of the full resulting rpm name. But there is nothing like that. Sorry for that. Release needs to be bumped even for soname-bump rebuilds (at least nowadays) so there is an impact if the target package uses git_dir_release preprocessing macro. In that case, bumping is done either by creating a new commit or a new annotated tag (the new tag will also add a new changelog entry and will make the release look nicer). Basically, if releng/ProvenPackager does a rebuild of a package manually, the recommended action currently for such a package is to add a new annotated tag (`fedpkg tag`) and write the reason for the rebuild. If a script is used, that script can be modified to automatically recognize if the target package uses the rpkg macro or not and do the required action for bumping accordingly. This is kind of a logic that should land in rpmdev-bumpspec. If this seems inconvenient, we can further discuss the options. We could e.g. make it so that just creating a new empty commit would be enough instead of tagging. > Thanks for the other answers. > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/18/20 12:57 AM, Matthew Miller wrote:
On Fri, Dec 18, 2020 at 12:24:10AM +0100, clime wrote:
It would be possible to specify the spec template as an rpm Source so
it would get included into the resulting srpm as well.
Yeah I was thinking the spec file templating system could automatically add
the original spec as Source where N is one higher than the highest
existing source file.
If that's automatically happening, it could produce undesired results with the
%{sources} macro.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/18/20 1:19 AM, clime wrote: I'd very much like to understand the impact of this on the following: 1) Provenpackagers doing mass spec changes/updates. If the mass spec change/update doesn't involve an rpkg macro, then there is no difference. I don't understand how there is no difference. The spec "bare" spec file is no longer there. How do I parse it? How do I grep it? How do I sed it? 2) Provenpackagers and/or RelEngs doing (targeted) mass rebuilds. There should be no impact here. If the source git repo stays the same, then the same srpm as for a previous build will be produced. I don't understand you answer. What does this have to do with "source git repo stays the same" and "same srpm"? The release must be bumped, they are not the same. Thanks for the other answers. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Friday, 18 December 2020, Tom Stellard wrote: > On 12/17/20 11:05 AM, Ben Cotton wrote: > >> https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing >> >> >> == Summary == >> This change should enable an opt-in spec file preprocessor in Fedora >> infrastructure for the benefit of packagers. The preprocessor allows >> some very neat tricks that were impossible before, for example >> generate changelog and release automatically from git metadata or pack >> the entire dist-git repository into an rpm-source tarball (effectively >> allowing unpacked repos to live in DistGit). >> >> == Owner == >> * Name: [[User:clime| Michal Novotný]] >> * Email: cl...@fedoraproject.org >> >> >> == Detailed Description == >> >> There is a recently added feature into mock: >> [https://github.com/rpm-software-management/mock/wiki/Plugin >> -rpkg-preprocessor >> the rpkg preprocessor] which, if enabled, introduces an intermediate >> step just before srpm building. This step consists of running the spec >> file through a text preprocessing engine that includes an already >> present library of macros designed specifically for rpm spec file >> generation from git metadata. This library is called >> [https://docs.pagure.org/rpkg-util/v3/macro_reference.html >> rpkg-macros]. The macros there allow packagers to have their >> `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields >> automatically generated from dist-git repository data and metadata. >> The library can be easily extended in future to support more packager >> use-cases or even a completely new library can be developed that >> doesn't look at git metadata at all and instead, for example, analyses >> already present tarball content to render spec file based on upstream >> information. This doesn't mean it will happen but the framework is >> generic enough to support that. There is also support for user-defined >> macros that are loaded on-demand from a file placed alongside the >> package sources, maintained by packager. This feature wouldn't be >> enabled by this change from start but it's an example of freedom that >> the preprocessing framework is able to provide. Enabling this change >> should be very easy, basically adding: >> >> >> config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True >> >> >> into mock configuration of Koji builders and using at least mock 2.7. >> Some very minor change may be also needed in Koji regarding the spec >> file lookup. >> >> Even if the change is enabled on the infrastructure level like this, >> the packager will still need to opt-in to use the preprocessor. The >> opting-in is done by placing `rpkg.conf` file into the package >> top-level directory with the following content: >> >> >> [rpkg] >> preprocess_spec = True >> >> >> When this is done by a packager, the preprocessor will be finally >> enabled for the given package. >> >> Alongside, there is an ongoing work to add the preprocessor support >> into the `rpkg` python library so that a packager can easily work with >> the spec files containing the preprocessor (rpkg) macros: >> https://pagure.io/rpkg/pull-request/530 >> >> > Is this pull request needed so that the preprocessor will run if I do > `fedpkg mockbuild` or does fedpkg already do this if the preprocessor is > enabled? The pull request is needed for it. fedpkg doesn't currently have support for preprocessing. Cheers clime > > -Tom > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://docs.fedoraproject.org > /en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.or > g/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/17/20 11:05 AM, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing == Summary == This change should enable an opt-in spec file preprocessor in Fedora infrastructure for the benefit of packagers. The preprocessor allows some very neat tricks that were impossible before, for example generate changelog and release automatically from git metadata or pack the entire dist-git repository into an rpm-source tarball (effectively allowing unpacked repos to live in DistGit). == Owner == * Name: [[User:clime| Michal Novotný]] * Email: cl...@fedoraproject.org == Detailed Description == There is a recently added feature into mock: [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor the rpkg preprocessor] which, if enabled, introduces an intermediate step just before srpm building. This step consists of running the spec file through a text preprocessing engine that includes an already present library of macros designed specifically for rpm spec file generation from git metadata. This library is called [https://docs.pagure.org/rpkg-util/v3/macro_reference.html rpkg-macros]. The macros there allow packagers to have their `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields automatically generated from dist-git repository data and metadata. The library can be easily extended in future to support more packager use-cases or even a completely new library can be developed that doesn't look at git metadata at all and instead, for example, analyses already present tarball content to render spec file based on upstream information. This doesn't mean it will happen but the framework is generic enough to support that. There is also support for user-defined macros that are loaded on-demand from a file placed alongside the package sources, maintained by packager. This feature wouldn't be enabled by this change from start but it's an example of freedom that the preprocessing framework is able to provide. Enabling this change should be very easy, basically adding: config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True into mock configuration of Koji builders and using at least mock 2.7. Some very minor change may be also needed in Koji regarding the spec file lookup. Even if the change is enabled on the infrastructure level like this, the packager will still need to opt-in to use the preprocessor. The opting-in is done by placing `rpkg.conf` file into the package top-level directory with the following content: [rpkg] preprocess_spec = True When this is done by a packager, the preprocessor will be finally enabled for the given package. Alongside, there is an ongoing work to add the preprocessor support into the `rpkg` python library so that a packager can easily work with the spec files containing the preprocessor (rpkg) macros: https://pagure.io/rpkg/pull-request/530 Is this pull request needed so that the preprocessor will run if I do `fedpkg mockbuild` or does fedpkg already do this if the preprocessor is enabled? -Tom ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 02:27, Japheth Cleaver wrote: > > On 12/17/2020 3:59 PM, Matthew Miller wrote: > > On Fri, Dec 18, 2020 at 12:51:49AM +0100, clime wrote: > > This change proposal does affect users. The User Experience section > needs to answer the following: > > Well, the users here are still packagers here no? I thought the "User" > in the title means "end user" who shouldn't be affected by it. Maybe > Ben can clarify this. > > It _is_ meant to refer to end users, but we have a lot of highly technical > end users and sysadmins who might want to download and build source RPMs. So > the answers to these questions seem like reasonable things to add. > > > They can use mock if the preprocessing will be enabled for the > respective chroots where it is enabled in Koji/Fedora. > They can't directly use rpmbuild for those packages that contain the > macros. But they can use rpkg/fedpkg to do the work. > Or preprocess spec first and then use rpmbuild. I am aware this is a > negative point of this change. > While having an option to use rpmbuild directly to build srpm/rpm from > a dist-git repo is nice, I would say that fedpkg or mock are the main > interfaces to do this. > I know this answer won't satisfy everyone. > > > Not to mention the many folks who use Fedora .src.rpms as a starting point > for EL-derivatives, or other RPM-based distros. Every time a Rawhide (or > Fedora) SRPM fails to compile because of a non-backwards-compatible change, > another frustrated sysadmin sheds a single tear. > > Deprecating rpmbuild is a major change. I wouldn't call it "deprecating rpmbuild". That's certainly not at all my intention. As a side-point, I think the cases where bare rpmbuild is used to build an rpm/srpm from a dist-git repo are rather limited because you probably need to first download sources from lookaside cache so you probably need fedpkg/rpkg/centpkg/rfpkg or a similar dedicated tool. These tools then offer the `srpm` and `local` commands so It would make sense to rather use these commands or mock for subsequent srpm/rpm building. > > -jc > > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/17/2020 3:59 PM, Matthew Miller wrote: On Fri, Dec 18, 2020 at 12:51:49AM +0100, clime wrote: This change proposal does affect users. The User Experience section needs to answer the following: Well, the users here are still packagers here no? I thought the "User" in the title means "end user" who shouldn't be affected by it. Maybe Ben can clarify this. It _is_ meant to refer to end users, but we have a lot of highly technical end users and sysadmins who might want to download and build source RPMs. So the answers to these questions seem like reasonable things to add. They can use mock if the preprocessing will be enabled for the respective chroots where it is enabled in Koji/Fedora. They can't directly use rpmbuild for those packages that contain the macros. But they can use rpkg/fedpkg to do the work. Or preprocess spec first and then use rpmbuild. I am aware this is a negative point of this change. While having an option to use rpmbuild directly to build srpm/rpm from a dist-git repo is nice, I would say that fedpkg or mock are the main interfaces to do this. I know this answer won't satisfy everyone. Not to mention the many folks who use Fedora .src.rpms as a starting point for EL-derivatives, or other RPM-based distros. Every time a Rawhide (or Fedora) SRPM fails to compile because of a non-backwards-compatible change, another frustrated sysadmin sheds a single tear. Deprecating rpmbuild is a major change. -jc ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, 18 Dec 2020 at 00:58, Matthew Miller wrote: > > On Fri, Dec 18, 2020 at 12:24:10AM +0100, clime wrote: > > It would be possible to specify the spec template as an rpm Source so > > it would get included into the resulting srpm as well. > > Yeah I was thinking the spec file templating system could automatically add > the original spec as Source where N is one higher than the highest > existing source file. I mean something like that could be done but I need to avoid parsing spec file from within the rpkg macros because when the rpkg macros are being evaluated, there is not-yet a valid rpm spec file to be parsed. That means I cannot get the highest used Source number in the context where I would like to get it...but I think this is just one line that doesn't really need to be dynamically generated because it shouldn't change. > > > -- > Matthew Miller > > Fedora Project Leader > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, 17 Dec 2020 at 21:23, Miro Hrončok wrote: > > On 12/17/20 8:05 PM, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > ... > > I'd very much like to understand the impact of this on the following: > > > 1) Provenpackagers doing mass spec changes/updates. If the mass spec change/update doesn't involve an rpkg macro, then there is no difference. If it does involve an rpkg macro, then we should directly change the macro so that the spec file gets rendered correctly according to the latest state-of-art. That probably means notifying affected packagers first but for a proven packager it is less work. There is also an option that a packager would specify the macros version with which to evaluate the spec file in `rpkg.conf` file. In that case, he would need to bump that version first so that the updated macro gets used. Not sure if something like this would be needed but this would prevent any changes in spec file unless the packager wants them on his/her own. > > 2) Provenpackagers and/or RelEngs doing (targeted) mass rebuilds. There should be no impact here. If the source git repo stays the same, then the same srpm as for a previous build will be produced. > > 3) Packagers doing `fedpkg local` builds. This PR makes sure `fedpkg local` will work: https://pagure.io/rpkg/pull-request/530 if preprocessing is enabled. There is a bit of additional work to open a PR for fedpkg to parse rpkg.conf file if it is present in a dist-git repo and enable preprocessing if it is enabled there in the file. It's just a few lines that I plan to write when I get feedback on the `rpkg` pull request. > > 5) Our downstreams rebuilding from dist-git. If they use mock or fedpkg, there should be no impact. If they use bare rpmbuild, it will no longer work and some changes will be needed. > > 4) Packages needed to be installed in buildSRPMFromSCM mock and/or Koji host. I am not sure if I understand correctly here so maybe this will need some more explanation. But the preprocessing needs some additional tooling to get installed that I tried to minimize. Basically: preproc, rpkg-macros, rpm-git-tag-sort, libgit2, git-core are the main packages needed to run the preprocessing. These packages get installed into the target chroot where the srpm is also built afterwards. They are installed only if preprocessing is enabled. > > > I'd also like to know, where exactly is the spec file pre-processed. Is it in > the buildSRPMFromSCM mock, or on the Koji host? It is preprocessed in the target chroot, i.e. in the same environment where rpmbuild -bs is called afterwards. > > > It feels like this will open a can of worms and I don't think the benefits are > worth it. IMHO we should strive to make RPM specs more flexible instead of > adding another layer on top of it. But I admit that I don't have all the > information yet. I think something like this is needed whether it is in rpm or in rpkg/mock. I think having implemented it on an upper layer than rpm is not such a huge deal. > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, Dec 18, 2020 at 12:51:49AM +0100, clime wrote: > > This change proposal does affect users. The User Experience section > > needs to answer the following: > Well, the users here are still packagers here no? I thought the "User" > in the title means "end user" who shouldn't be affected by it. Maybe > Ben can clarify this. It _is_ meant to refer to end users, but we have a lot of highly technical end users and sysadmins who might want to download and build source RPMs. So the answers to these questions seem like reasonable things to add. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Fri, Dec 18, 2020 at 12:24:10AM +0100, clime wrote: > It would be possible to specify the spec template as an rpm Source so > it would get included into the resulting srpm as well. Yeah I was thinking the spec file templating system could automatically add the original spec as Source where N is one higher than the highest existing source file. -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, 17 Dec 2020 at 22:04, James Szinger wrote: > > On Thu, 17 Dec 2020 14:05:40 -0500 > Ben Cotton wrote: > > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > > > == User Experience == > > This change is intended for packagers. It should help to make a bit of > > their work easier and offer them some new interesting options. > > This change proposal does affect users. The User Experience section > needs to answer the following: Well, the users here are still packagers here no? I thought the "User" in the title means "end user" who shouldn't be affected by it. Maybe Ben can clarify this. > > 1. How does this affect users who download, maybe modify, and rebuild > the SRPM? Can they continue to use rpmbuid and mock as they have > been? Does the SRPM contain the pre-processed or post-processed spec > file? They can use mock if the preprocessing will be enabled for the respective chroots where it is enabled in Koji/Fedora. They can't directly use rpmbuild for those packages that contain the macros. But they can use rpkg/fedpkg to do the work. Or preprocess spec first and then use rpmbuild. I am aware this is a negative point of this change. While having an option to use rpmbuild directly to build srpm/rpm from a dist-git repo is nice, I would say that fedpkg or mock are the main interfaces to do this. I know this answer won't satisfy everyone. > > 2. How does this affect users who download the spec file from > src.fedoraproject.org? Do they have the tools to build the RPM? How > much harder is it? The tools will be available. It should be no-work if a person uses fedpkg or mock. Otherwise, if they really use bare rpmbuild, they will need to modify their scripts to use fedpkg/mock. Or they can use preproc first to render the template and then pass it to rpmbuild. I also planned to do a simple wrapper called '"preproc-rpmbuild" which would do that while otherwise providing the same command-line interface as rpmbuild does. > > Please remember that this is Free Software and the spec files are > useful for a broader audience than just the Fedora packagers. Right, we also use the (CentOS) spec files in a company where I currently work at. > > Jim > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, 17 Dec 2020 at 21:34, James Cassell wrote: > > > On Thu, Dec 17, 2020, at 2:05 PM, Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > > > I'm skeptical. If it does pass, I'd insist on having the non-processed spec > and any required supporting files in the SRPM. It would be possible to specify the spec template as an rpm Source so it would get included into the resulting srpm as well. > > Does this relate in any way to the magic done by rdopkg dist-git <-> > source-git translation? Their approach seems very good to me, but might not > exactly overlap here. There is some overlap in the goal (allow people to work with upstream sources when convenient) but the method to achieve it is slightly different. The approach suggested here is more declarative while the rdopkg's approach is more imperative. Basically, with rdopkg, you do the upstream<->downstream conversion on the client and then just push the results to the server. With the preprocessing engine, you define the conversion in a spec file and then let the infrastructure (i.e. builders) do the work just before srpm is built. We can go into more details. There probably is some inaccuracy in what I have just said (but tried my best to explain the difference as I see it). > > V/r, > James Cassell > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
...snip... > > I'm generally not excited about this, as it adds a huge layer of > indirection and a ton of extra magic that makes it harder to decipher > what is happening. > I don't think there is any magic in it. Everything is clearly documented and every expansion clearly defined and also intuitive. If something changes spec files "behind my back" - that I would call magic but If a packager explicitly states: "Here, in this place in the spec file, I want this particular dynamic snippet to be expanded", then I think that is very transparent. Especially, if some other person looks at the spec file, he/she will know something will happen with the spec file and also what it will be (perhaps after quickly checking the docs). > > -- > 真実はいつも一つ！/ Always, there's only one truth! > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, 17 Dec 2020 at 21:15, Fabio Valentini wrote: > > On Thu, Dec 17, 2020 at 8:06 PM Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > As far as I can tell, this is the third implementation of generated > changelogs ... did the autorelease / autochangelog work that was even > already deployed in staging not go anywhere? I know about it but I don't have much information about its current state. > > > == Owner == > > * Name: [[User:clime| Michal Novotný]] > > * Email: cl...@fedoraproject.org > > > > > > == Detailed Description == > > > > There is a recently added feature into mock: > > [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor > > the rpkg preprocessor] which, if enabled, introduces an intermediate > > step just before srpm building. This step consists of running the spec > > file through a text preprocessing engine that includes an already > > present library of macros designed specifically for rpm spec file > > generation from git metadata. This library is called > > [https://docs.pagure.org/rpkg-util/v3/macro_reference.html > > rpkg-macros]. The macros there allow packagers to have their > > `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields > > automatically generated from dist-git repository data and metadata. > > The library can be easily extended in future to support more packager > > use-cases or even a completely new library can be developed that > > doesn't look at git metadata at all and instead, for example, analyses > > already present tarball content to render spec file based on upstream > > information. This doesn't mean it will happen but the framework is > > generic enough to support that. There is also support for user-defined > > macros that are loaded on-demand from a file placed alongside the > > package sources, maintained by packager. This feature wouldn't be > > enabled by this change from start but it's an example of freedom that > > the preprocessing framework is able to provide. Enabling this change > > should be very easy, basically adding: > > > > > > config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True > > > > > > into mock configuration of Koji builders and using at least mock 2.7. > > Some very minor change may be also needed in Koji regarding the spec > > file lookup. > > > > Even if the change is enabled on the infrastructure level like this, > > the packager will still need to opt-in to use the preprocessor. The > > opting-in is done by placing `rpkg.conf` file into the package > > top-level directory with the following content: > > > > > > [rpkg] > > preprocess_spec = True > > > > > > When this is done by a packager, the preprocessor will be finally > > enabled for the given package. > > > > Alongside, there is an ongoing work to add the preprocessor support > > into the `rpkg` python library so that a packager can easily work with > > the spec files containing the preprocessor (rpkg) macros: > > https://pagure.io/rpkg/pull-request/530 > > > > The macros are supported since epel6 until the most latest Fedora > > (preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The > > spec preprocessing step in mock happens in a target chroot just before > > the srpm build. > > > > > > == Benefit to Fedora == > > > > This change offers solution to some long-standing issues in Fedora > > around packaging (i.e. automatic release and changelog generation) > > while also offering some interesting future options (for example > > unpacked dist-git repos). The big advantage of this approach is that > > it is explicit. Spec file stays the source of truth and by looking > > inside one, you will be able to determine how the text will expand for > > a certain git repository state. > > What does "unpacked dist-git repos" mean? Is this a euphemism for source-git? In this context, it means a dist-git repository with the original upstream sources placed directly in it (i.e. the sources are not packed in a tarball but they are directly present in a dist-git repo). To remind us of our collective knowledge, there can be multiple variants of this. One of them is that the spec file is placed next to the unpacked upstream sources (the simplest one). Then there are approaches that try to keep the upstream (unpacked) sources and the spec file separate - i.e. they can be in different branches, different repos, different subdirectories - each of these use-cases can be supported by the preprocessing engine. But I would say this is an
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/17/20 9:21 PM, Miro Hrončok wrote: On 12/17/20 8:05 PM, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing == Summary == This change should enable an opt-in spec file preprocessor in Fedora infrastructure for the benefit of packagers. The preprocessor allows some very neat tricks that were impossible before, for example generate changelog and release automatically from git metadata or pack the entire dist-git repository into an rpm-source tarball (effectively allowing unpacked repos to live in DistGit). ... 3) Packagers doing `fedpkg local` builds. Also, generally, fedpkg parses the NEVR from the spec when performing various operations. How would that work here? -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, 17 Dec 2020 14:05:40 -0500 Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > == Summary == > This change should enable an opt-in spec file preprocessor in Fedora > infrastructure for the benefit of packagers. The preprocessor allows > some very neat tricks that were impossible before, for example > generate changelog and release automatically from git metadata or pack > the entire dist-git repository into an rpm-source tarball (effectively > allowing unpacked repos to live in DistGit). > > == User Experience == > This change is intended for packagers. It should help to make a bit of > their work easier and offer them some new interesting options. This change proposal does affect users. The User Experience section needs to answer the following: 1. How does this affect users who download, maybe modify, and rebuild the SRPM? Can they continue to use rpmbuid and mock as they have been? Does the SRPM contain the pre-processed or post-processed spec file? 2. How does this affect users who download the spec file from src.fedoraproject.org? Do they have the tools to build the RPM? How much harder is it? Please remember that this is Free Software and the spec files are useful for a broader audience than just the Fedora packagers. Jim ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, Dec 17, 2020, at 2:05 PM, Ben Cotton wrote: > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > == Summary == > This change should enable an opt-in spec file preprocessor in Fedora > infrastructure for the benefit of packagers. The preprocessor allows > some very neat tricks that were impossible before, for example > generate changelog and release automatically from git metadata or pack > the entire dist-git repository into an rpm-source tarball (effectively > allowing unpacked repos to live in DistGit). > I'm skeptical. If it does pass, I'd insist on having the non-processed spec and any required supporting files in the SRPM. Does this relate in any way to the magic done by rdopkg dist-git <-> source-git translation? Their approach seems very good to me, but might not exactly overlap here. V/r, James Cassell ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On 12/17/20 8:05 PM, Ben Cotton wrote: https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing == Summary == This change should enable an opt-in spec file preprocessor in Fedora infrastructure for the benefit of packagers. The preprocessor allows some very neat tricks that were impossible before, for example generate changelog and release automatically from git metadata or pack the entire dist-git repository into an rpm-source tarball (effectively allowing unpacked repos to live in DistGit). ... I'd very much like to understand the impact of this on the following: 1) Provenpackagers doing mass spec changes/updates. 2) Provenpackagers and/or RelEngs doing (targeted) mass rebuilds. 3) Packagers doing `fedpkg local` builds. 5) Our downstreams rebuilding from dist-git. 4) Packages needed to be installed in buildSRPMFromSCM mock and/or Koji host. I'd also like to know, where exactly is the spec file pre-processed. Is it in the buildSRPMFromSCM mock, or on the Koji host? It feels like this will open a can of worms and I don't think the benefits are worth it. IMHO we should strive to make RPM specs more flexible instead of adding another layer on top of it. But I admit that I don't have all the information yet. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, Dec 17, 2020 at 3:15 PM Fabio Valentini wrote: > > On Thu, Dec 17, 2020 at 8:06 PM Ben Cotton wrote: > > > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > As far as I can tell, this is the third implementation of generated > changelogs ... did the autorelease / autochangelog work that was even > already deployed in staging not go anywhere? > That effort, as far as I'm aware, was waiting for the staging Koji to be restored to finish the work, which it was recently. I believe those developers are interested in getting this in place in production Koji. > > == Owner == > > * Name: [[User:clime| Michal Novotný]] > > * Email: cl...@fedoraproject.org > > > > > > == Detailed Description == > > > > There is a recently added feature into mock: > > [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor > > the rpkg preprocessor] which, if enabled, introduces an intermediate > > step just before srpm building. This step consists of running the spec > > file through a text preprocessing engine that includes an already > > present library of macros designed specifically for rpm spec file > > generation from git metadata. This library is called > > [https://docs.pagure.org/rpkg-util/v3/macro_reference.html > > rpkg-macros]. The macros there allow packagers to have their > > `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields > > automatically generated from dist-git repository data and metadata. > > The library can be easily extended in future to support more packager > > use-cases or even a completely new library can be developed that > > doesn't look at git metadata at all and instead, for example, analyses > > already present tarball content to render spec file based on upstream > > information. This doesn't mean it will happen but the framework is > > generic enough to support that. There is also support for user-defined > > macros that are loaded on-demand from a file placed alongside the > > package sources, maintained by packager. This feature wouldn't be > > enabled by this change from start but it's an example of freedom that > > the preprocessing framework is able to provide. Enabling this change > > should be very easy, basically adding: > > > > > > config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True > > > > > > into mock configuration of Koji builders and using at least mock 2.7. > > Some very minor change may be also needed in Koji regarding the spec > > file lookup. > > > > Even if the change is enabled on the infrastructure level like this, > > the packager will still need to opt-in to use the preprocessor. The > > opting-in is done by placing `rpkg.conf` file into the package > > top-level directory with the following content: > > > > > > [rpkg] > > preprocess_spec = True > > > > > > When this is done by a packager, the preprocessor will be finally > > enabled for the given package. > > > > Alongside, there is an ongoing work to add the preprocessor support > > into the `rpkg` python library so that a packager can easily work with > > the spec files containing the preprocessor (rpkg) macros: > > https://pagure.io/rpkg/pull-request/530 > > > > The macros are supported since epel6 until the most latest Fedora > > (preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The > > spec preprocessing step in mock happens in a target chroot just before > > the srpm build. > > > > > > == Benefit to Fedora == > > > > This change offers solution to some long-standing issues in Fedora > > around packaging (i.e. automatic release and changelog generation) > > while also offering some interesting future options (for example > > unpacked dist-git repos). The big advantage of this approach is that > > it is explicit. Spec file stays the source of truth and by looking > > inside one, you will be able to determine how the text will expand for > > a certain git repository state. > > What does "unpacked dist-git repos" mean? Is this a euphemism for source-git? > > > == Scope == > > * Proposal owners: > > For the very basic support, probably a small patch in Koji is needed > > to be able to lookup not only `.spec` files but also `.spec.rpkg` > > files (the `.spec.rpkg` extension explicitly states that the spec file > > is a template). Also the `rpmdevtools/rpmdev-bumpspec` script should > > be tweaked to be compatible with spec files using the macros. > > Is the Change owner going to submit patches for fedpkg and rpmdevtools? > > > * Other developers: > > Some optional help with
Re: Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
On Thu, Dec 17, 2020 at 8:06 PM Ben Cotton wrote: > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > == Summary == > This change should enable an opt-in spec file preprocessor in Fedora > infrastructure for the benefit of packagers. The preprocessor allows > some very neat tricks that were impossible before, for example > generate changelog and release automatically from git metadata or pack > the entire dist-git repository into an rpm-source tarball (effectively > allowing unpacked repos to live in DistGit). As far as I can tell, this is the third implementation of generated changelogs ... did the autorelease / autochangelog work that was even already deployed in staging not go anywhere? > == Owner == > * Name: [[User:clime| Michal Novotný]] > * Email: cl...@fedoraproject.org > > > == Detailed Description == > > There is a recently added feature into mock: > [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor > the rpkg preprocessor] which, if enabled, introduces an intermediate > step just before srpm building. This step consists of running the spec > file through a text preprocessing engine that includes an already > present library of macros designed specifically for rpm spec file > generation from git metadata. This library is called > [https://docs.pagure.org/rpkg-util/v3/macro_reference.html > rpkg-macros]. The macros there allow packagers to have their > `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields > automatically generated from dist-git repository data and metadata. > The library can be easily extended in future to support more packager > use-cases or even a completely new library can be developed that > doesn't look at git metadata at all and instead, for example, analyses > already present tarball content to render spec file based on upstream > information. This doesn't mean it will happen but the framework is > generic enough to support that. There is also support for user-defined > macros that are loaded on-demand from a file placed alongside the > package sources, maintained by packager. This feature wouldn't be > enabled by this change from start but it's an example of freedom that > the preprocessing framework is able to provide. Enabling this change > should be very easy, basically adding: > > > config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True > > > into mock configuration of Koji builders and using at least mock 2.7. > Some very minor change may be also needed in Koji regarding the spec > file lookup. > > Even if the change is enabled on the infrastructure level like this, > the packager will still need to opt-in to use the preprocessor. The > opting-in is done by placing `rpkg.conf` file into the package > top-level directory with the following content: > > > [rpkg] > preprocess_spec = True > > > When this is done by a packager, the preprocessor will be finally > enabled for the given package. > > Alongside, there is an ongoing work to add the preprocessor support > into the `rpkg` python library so that a packager can easily work with > the spec files containing the preprocessor (rpkg) macros: > https://pagure.io/rpkg/pull-request/530 > > The macros are supported since epel6 until the most latest Fedora > (preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The > spec preprocessing step in mock happens in a target chroot just before > the srpm build. > > > == Benefit to Fedora == > > This change offers solution to some long-standing issues in Fedora > around packaging (i.e. automatic release and changelog generation) > while also offering some interesting future options (for example > unpacked dist-git repos). The big advantage of this approach is that > it is explicit. Spec file stays the source of truth and by looking > inside one, you will be able to determine how the text will expand for > a certain git repository state. What does "unpacked dist-git repos" mean? Is this a euphemism for source-git? > == Scope == > * Proposal owners: > For the very basic support, probably a small patch in Koji is needed > to be able to lookup not only `.spec` files but also `.spec.rpkg` > files (the `.spec.rpkg` extension explicitly states that the spec file > is a template). Also the `rpmdevtools/rpmdev-bumpspec` script should > be tweaked to be compatible with spec files using the macros. Is the Change owner going to submit patches for fedpkg and rpmdevtools? > * Other developers: > Some optional help with `rpmdevtools/rpmdev-bumpspec` changes would be > welcome. rpmdev-bumpspec is a tangled mess of spaghetti code and I'd rather not touch it or make it even more complicated. I also think this can't be optional. releng uses rpmdev-bumpspec as part of scripted mass rebuilds, so it must work with all packages. > * Release engineering: [https://pagure.io/releng/issue/9910 #9910] (a > check of an impact with Release Engineering is needed) > Enabling the rpkg_preprocessor plugin in
Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing
== Summary ==
This change should enable an opt-in spec file preprocessor in Fedora
infrastructure for the benefit of packagers. The preprocessor allows
some very neat tricks that were impossible before, for example
generate changelog and release automatically from git metadata or pack
the entire dist-git repository into an rpm-source tarball (effectively
allowing unpacked repos to live in DistGit).
== Owner ==
* Name: [[User:clime| Michal Novotný]]
* Email: cl...@fedoraproject.org
== Detailed Description ==
There is a recently added feature into mock:
[https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor
the rpkg preprocessor] which, if enabled, introduces an intermediate
step just before srpm building. This step consists of running the spec
file through a text preprocessing engine that includes an already
present library of macros designed specifically for rpm spec file
generation from git metadata. This library is called
[https://docs.pagure.org/rpkg-util/v3/macro_reference.html
rpkg-macros]. The macros there allow packagers to have their
`%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields
automatically generated from dist-git repository data and metadata.
The library can be easily extended in future to support more packager
use-cases or even a completely new library can be developed that
doesn't look at git metadata at all and instead, for example, analyses
already present tarball content to render spec file based on upstream
information. This doesn't mean it will happen but the framework is
generic enough to support that. There is also support for user-defined
macros that are loaded on-demand from a file placed alongside the
package sources, maintained by packager. This feature wouldn't be
enabled by this change from start but it's an example of freedom that
the preprocessing framework is able to provide. Enabling this change
should be very easy, basically adding:
config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True
into mock configuration of Koji builders and using at least mock 2.7.
Some very minor change may be also needed in Koji regarding the spec
file lookup.
Even if the change is enabled on the infrastructure level like this,
the packager will still need to opt-in to use the preprocessor. The
opting-in is done by placing `rpkg.conf` file into the package
top-level directory with the following content:
[rpkg]
preprocess_spec = True
When this is done by a packager, the preprocessor will be finally
enabled for the given package.
Alongside, there is an ongoing work to add the preprocessor support
into the `rpkg` python library so that a packager can easily work with
the spec files containing the preprocessor (rpkg) macros:
https://pagure.io/rpkg/pull-request/530
The macros are supported since epel6 until the most latest Fedora
(preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The
spec preprocessing step in mock happens in a target chroot just before
the srpm build.
== Benefit to Fedora ==
This change offers solution to some long-standing issues in Fedora
around packaging (i.e. automatic release and changelog generation)
while also offering some interesting future options (for example
unpacked dist-git repos). The big advantage of this approach is that
it is explicit. Spec file stays the source of truth and by looking
inside one, you will be able to determine how the text will expand for
a certain git repository state.
== Scope ==
* Proposal owners:
For the very basic support, probably a small patch in Koji is needed
to be able to lookup not only `.spec` files but also `.spec.rpkg`
files (the `.spec.rpkg` extension explicitly states that the spec file
is a template). Also the `rpmdevtools/rpmdev-bumpspec` script should
be tweaked to be compatible with spec files using the macros.
* Other developers:
Some optional help with `rpmdevtools/rpmdev-bumpspec` changes would be welcome.
* Release engineering: [https://pagure.io/releng/issue/9910 #9910] (a
check of an impact with Release Engineering is needed)
Enabling the rpkg_preprocessor plugin in mock config for Koji builders.
* Policies and guidelines:
The new macro support should be mentioned or even described in the
packaging guidelines. We should decide if the full power of the
rpkg-macros library should be allowed from start (i.e. even unpacked
repos).
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
Because of the opt-in nature on packager side, there should be no
compatibility issues.
== How To Test ==
Once the feature is enabled, one can test it by providing the
`rpkg.conf` file with the required content in a package repository and
use some rpkg macro in the spec file: e.g.
Name: {{{ git_dir_name }}}
to generate the name of the package from the repository name (this
should actually produce the original text
Fedora 34 Change: Enable spec file preprocessing (System-Wide Change proposal)
https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing
== Summary ==
This change should enable an opt-in spec file preprocessor in Fedora
infrastructure for the benefit of packagers. The preprocessor allows
some very neat tricks that were impossible before, for example
generate changelog and release automatically from git metadata or pack
the entire dist-git repository into an rpm-source tarball (effectively
allowing unpacked repos to live in DistGit).
== Owner ==
* Name: [[User:clime| Michal Novotný]]
* Email: cl...@fedoraproject.org
== Detailed Description ==
There is a recently added feature into mock:
[https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor
the rpkg preprocessor] which, if enabled, introduces an intermediate
step just before srpm building. This step consists of running the spec
file through a text preprocessing engine that includes an already
present library of macros designed specifically for rpm spec file
generation from git metadata. This library is called
[https://docs.pagure.org/rpkg-util/v3/macro_reference.html
rpkg-macros]. The macros there allow packagers to have their
`%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields
automatically generated from dist-git repository data and metadata.
The library can be easily extended in future to support more packager
use-cases or even a completely new library can be developed that
doesn't look at git metadata at all and instead, for example, analyses
already present tarball content to render spec file based on upstream
information. This doesn't mean it will happen but the framework is
generic enough to support that. There is also support for user-defined
macros that are loaded on-demand from a file placed alongside the
package sources, maintained by packager. This feature wouldn't be
enabled by this change from start but it's an example of freedom that
the preprocessing framework is able to provide. Enabling this change
should be very easy, basically adding:
config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True
into mock configuration of Koji builders and using at least mock 2.7.
Some very minor change may be also needed in Koji regarding the spec
file lookup.
Even if the change is enabled on the infrastructure level like this,
the packager will still need to opt-in to use the preprocessor. The
opting-in is done by placing `rpkg.conf` file into the package
top-level directory with the following content:
[rpkg]
preprocess_spec = True
When this is done by a packager, the preprocessor will be finally
enabled for the given package.
Alongside, there is an ongoing work to add the preprocessor support
into the `rpkg` python library so that a packager can easily work with
the spec files containing the preprocessor (rpkg) macros:
https://pagure.io/rpkg/pull-request/530
The macros are supported since epel6 until the most latest Fedora
(preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The
spec preprocessing step in mock happens in a target chroot just before
the srpm build.
== Benefit to Fedora ==
This change offers solution to some long-standing issues in Fedora
around packaging (i.e. automatic release and changelog generation)
while also offering some interesting future options (for example
unpacked dist-git repos). The big advantage of this approach is that
it is explicit. Spec file stays the source of truth and by looking
inside one, you will be able to determine how the text will expand for
a certain git repository state.
== Scope ==
* Proposal owners:
For the very basic support, probably a small patch in Koji is needed
to be able to lookup not only `.spec` files but also `.spec.rpkg`
files (the `.spec.rpkg` extension explicitly states that the spec file
is a template). Also the `rpmdevtools/rpmdev-bumpspec` script should
be tweaked to be compatible with spec files using the macros.
* Other developers:
Some optional help with `rpmdevtools/rpmdev-bumpspec` changes would be welcome.
* Release engineering: [https://pagure.io/releng/issue/9910 #9910] (a
check of an impact with Release Engineering is needed)
Enabling the rpkg_preprocessor plugin in mock config for Koji builders.
* Policies and guidelines:
The new macro support should be mentioned or even described in the
packaging guidelines. We should decide if the full power of the
rpkg-macros library should be allowed from start (i.e. even unpacked
repos).
* Trademark approval: N/A (not needed for this Change)
* Alignment with Objectives: N/A
== Upgrade/compatibility impact ==
Because of the opt-in nature on packager side, there should be no
compatibility issues.
== How To Test ==
Once the feature is enabled, one can test it by providing the
`rpkg.conf` file with the required content in a package repository and
use some rpkg macro in the spec file: e.g.
Name: {{{ git_dir_name }}}
to generate the name of the package from the repository name (this
should actually produce the original text
