Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-20 Thread Nicolas Mailhot via devel
Le dimanche 19 juillet 2020 à 10:39 -0700, Kevin Fenzi a écrit :
> On Mon, Jul 13, 2020 at 10:17:11AM +0200, Nicolas Mailhot via devel
> wrote:
> > Tough it is a litteral key = value file with no fancy formatting
> > nor
> > even ini-like sections, and a handful of variables. Very boring
> > KISS
> > stuff.
> 
> Sure, but a file with 16 spaces and a newline is pretty confusing.

I will add a line of comment instead. The confusion is due to the fact
rpmbuild wants source files to exist before the build starts, and wants
them to be more than 13 bytes. I thought filling with spaces would
convey the empty file idea, I was wrong, adding a comment is trivial
and the parsing code is robust WRT comment lines, so no biggie to
change and improve.

Thank you for the enhancement suggestion!

> But then you know that exact thing was built. 
> With this setup you look at the git hash in koji and... it's the next
> commit after that thats exactly this build? Thats really messy.

The system separates build commits from modification commits.

It’s not like right now when most commits are modification *and* build
commits, except when they are not (because the change was done in
several commits, because the build failed but git still pretends it
occured, because a branch was EOLed before the modification process
finished, because the commit tracked a build that did not pass bodhi
gating, etc etc.)

The proposed system is a lot more clear and strict, for a build to be
tracked in git it needs to have actually succeeded and be commited
back, anything else is packager WIP. Making git history something you
can rely upon.

> Right, I was suggesting _changing_ your macros for this workflow. 
> 
> maintainer makes changes, runs scratch builds, tests, etc.
> They decide everything is good for an official build. 
> They generate the files and the macros just use those generated
> files,
> they don't bump anything or require a post build checkin.

You do not need to change the macros for this workflow, they already
support reproducible replay. Just prepare your build out of band and
then ask to build it in reproducible mode.

That only requires buildsys support to set the variable that activates
reproducible mode, and buildsys or packager support to make sure the
buildroot content matches the test builds (macros can do a lot of
things, they can not manage buildroot state in stead of the build
system).

However what this amounts to is the following:

1. build packages in scratch mode
2. check
3. commit
4. rebuild scratch builds in reproducible mode

That is more complex than the target workflow in the macros

1. build packages
2. check
3. decide to keep the packages, and commit back

The second workflow is more reliable than the first one, because the
second preserves the packages you tested, instead of hoping nothing
changed in the buildroot between the two build steps.

Of course doing it in 3 steps requires pairing back commit with bodhi
gating, absent bodhi changes it will probably be more involved at
first.

But the fact is it *can* evolve to a 3-step process with infra help,
and the evolution is consistent with what we’ve been trying to achieve
in the past years, and the current 4-step process is also supported at
macro level, as long as fedpkg/koji learns to set one variable (*not*
rocket science). And setting one variable is no different from the
generic bcond stuff ELN people want to happen.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-19 Thread Kevin Fenzi
On Mon, Jul 13, 2020 at 10:17:11AM +0200, Nicolas Mailhot via devel wrote:
> Le dimanche 12 juillet 2020 à 13:07 -0700, Kevin Fenzi a écrit :
> > On Sun, Jul 05, 2020 at 02:15:23PM +0200, Nicolas Mailhot via devel
> > wrote:
> > > 
> > > This is now done in the latest code refresh and in the test copr
> > > https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/c/bc4e6a79355f3a2b73abeea7e5c0e1f53b09579e?branch=forge-with-patches-auto-call-auto-changelog-auto-srpm-bump
> > > https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> > > 
> > > I fleshed out the change page a little too
> > > https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> > 
> > So I finally carved out a few minutes to look at this, but... 
> 
> Thank you for taking a look at it
> 
> >  
> > https://copr-dist-git.fedorainfracloud.org/cgit/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/sil-charis-fonts.git
> > seems to not exist? Is that some copr issue?
> 
> I honestly do not know. The copr was created with a mass import of
> srpms themselves created by rpmbuild -bs. I mainly use copr to check
> the srpm I build locally in mock work in another system before feeding
> them to koji, so I never used the git part of copr much.

ok. No worries.

> > Looking in the src.rpms I see some of the files I want to look at are
> > oddly empty? 
> > 
> > ➜  ~ od -c rpmbuild/SOURCES/buildsys.conf
> > 000  \n
> > 016
> > 
> > Is that because it's the 'before' src.rpm without the version
> > information injected yet?
> 
> Yes, almost all of the srpm have not been bumped before they were fed
> to copr, so they only include the minimal whitespace necessary for
> rpmbuild to register them as sources (the evil "your source is less
> than 13 bytes" rpmbuild assertion).
> 
> IIRC, I scratched sources for most of them to check the new spectool
> had not problem with the specs, so they won’t have any bump history in
> them.
> 
> You have an example of bumped srpm in adf-accanthis-fonts (with an
> almost full buildsys.conf though this package do not use the postbuild
> counter). 
> 
> >  Might be nice for these files to have a small
> > doc block at the top?
> 
> The detached changelog – certainly not, that would complexify its
> maintenance quite a lot. The conf file, why not, that’s fairly trivial
> to add.
> 
> Tough it is a litteral key = value file with no fancy formatting nor
> even ini-like sections, and a handful of variables. Very boring KISS
> stuff.

Sure, but a file with 16 spaces and a newline is pretty confusing. 

> > Or at least the wiki page could explain each file,
> > whats in it and whats the format for it. 
> > 
> > I really dislike having a second commit to say 'yes, this build was
> > the last one'. Could you generate that info in advance and just
> > commit it before the build? 
> 
> Well I really dislike people who commit that a build happened when it
> has not yet, and who rewrite git history multiple times to "fix" when
> they guess wrong:) Or worse, who do *not* rewrite git, and have a
> changelog that clearly does not correspond to their build history.

But then you know that exact thing was built. 
With this setup you look at the git hash in koji and... it's the next
commit after that thats exactly this build? Thats really messy.
> 
> The release part of autobumping will only happen if the spec needs it.
> ie if the release the packager stored in the spec already makes the evr
> newer than the last recorded evr no bump will happen. Changelog, not
> sure, I suppose I could tie it up to release bumping so if one does not
> happen the other does not happen either (that would make the code more
> complex, but not too hard to add).
> 
> However, is not possible and it won’t be possible to pre-fill
> buildsys.conf because the decision to bump or not is made by comparing
> the current evr to the one stored in there. So if you pre-bump the
> file, the logic will decide you have already done the corresponding
> build (which, if I follow you correctly will be *true*) and add another
> release bump to make sure two consecutive builds in a branch have
> different evrs.

Right, I was suggesting _changing_ your macros for this workflow. 

maintainer makes changes, runs scratch builds, tests, etc.
They decide everything is good for an official build. 
They generate the files and the macros just use those generated files,
they don't bump anything or require a post build checkin.
> 
> Basically, what you want is to reproduce a build you already did some
> other way. Then just set the variable that triggers reproducible mode
> and you’ll be done (that would require koji/copr cooperation however).
> 
> If anything changed in the buildroots you build against your build may
> fail the same way it fails today, and your git and changelog will be a
> lie the same way they are a lie today, but 

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-15 Thread jkonecny
On Thu, 2020-07-02 at 16:31 +, Nicolas Mailhot via devel wrote:
> 
> Le July 2, 2020 2:47:49 PM UTC, Vitaly Zaitsev via devel <
> devel@lists.fedoraproject.org> a écrit :
> > On 02.07.2020 11:27, Nicolas Mailhot wrote:
> > > Why? Koji schedules a build. The build registers its own build
> > > date
> > in
> > > the produced packages. Koji decides to keep and commit the
> > > result, or
> > > drop it (scratch build, failed side tag, whatever). Koji is still
> > > in
> > > charge, the bumping is just integrated in the build process with
> > > the
> > > rest of the package creation.
> > 
> > Koji was just an example. %changelog section should be auto-
> > generated
> > from commits messages. I don't want to maintain a separate file
> > with
> > the changelog.
> 
> The feature is completely compatible with this workflow. 
> 
> It registers build events in a detached file (the only part of the
> rpm changelog that requires knowledge of the rpm format)
> 
> You can prime this file via git hooks or any other system you like,
> the feature will take it as is, add the timestamp the build occurred
> at, and feed the result to the correct parts of the rpm build
> process.

+1 I was looking for a similar feature for a long time. To basically
completely automate changelog generation in a meaningful way.

> -- 
> Nicolas Mailhot
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-15 Thread Pierre-Yves Chibon
On Fri, Jul 03, 2020 at 11:11:30AM +0200, Nicolas Mailhot via devel wrote:
> Le vendredi 03 juillet 2020 à 09:48 +0200, Pierre-Yves Chibon a écrit :
> > On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote:
> > > Nicolas Mailhot wrote:
> > > > The same process that commits a new state of the changelog file
> > > > in 
> > > > sources,
> > > > commits the date that was written in the changelog in a separate
> > > > key = 
> > > > value
> > > > file (with the components of the build evr, the last packager id,
> > > > etc).
> > > 
> > > Do you mean that the key/value file will be committed to Git from
> > > inside
> > > Koji? Do the Koji builders have write access to Git?
> > 
> > This is the part that worries me a little about this approach.
> > Builders currently do not have commit access to git and I'm not sure
> > if we want them to considering they have git installed (so they can
> > clone) as well as access to all the packages in dist-git from a
> > networking point of view (again so they can clone).
> > So if we were to give the builders commit access to dist-git, an
> > attacker could easily commit to any other packages, potentially from
> > something as easy as a scratch-build.
> 
> From a purely architecture POW I’m convinced the proposed approach is
> the correct approach. Anything else proposed so far involves:
>   – tying a low-level event like "build occurred at date XXX" to high-
> level Fedora infra (making our workflow non portable and 
> incompatible with downstreams and third parties)

I don't think this point is true in the case of rpmautospec.

>   – taking bets in git that a build will occur and succeed (before it 
> actually occurs and succeeds, in real life builds fail for various
> reasons), and

I don't think this point is true, in the case of rpmautospec where the tagging
is done by the build system, once that build has succeeded.

>   – attempting to munge spec file behind the packager back (unlikely to
> work fine the more automated and dynamic we made those).

I'd contest the "behind the packager back" in the case of rpmautospec
considering this is well documented, but I fear that qualification may just have
been a little trolling :)

> However, because it’s the correct architecture solution, it also forces
> to make hard architectural choices, instead of mixing unrelated things
> in git and pretending that makes the result fine. Mixing unrelated
> things in a pile of container or git poo and pretending the result is
> fine is exactly what I hate in contenerish build workflows and why I
> work on rpm packaging.
> 
> From a pure high-level view, the thing in our infra that gates builds
> and decides whether they are official or scratched is bodhi.
> 
> So if you want to push Fedora release logic to its ultimate conclusion,
> the thing that should be in charge of committing the new
> release/changelog build state to package history in git is bodhi, not
> koji. And you can put security related checks there, since deciding to
> push things to users requires security related checks anyway (that
> probably also involves branching while a bodhi update is in flight and
> not approved yet).
> 
> However, that’s if you want to push the model to its ultimate
> conclusion and have something nice solid, automated, and future-proof.
> 
> If you don’t want to touch bodhi, and it you do not want koji to commit
> to git (which, is not the best of things for the reasons you stated,
> and for the reasons I stated), you can just:
> – make the koji client return the URL that will contain the SRPM at the
>   end of the build process if it succeeds.
> – have the person of script that called the koji client (and has,
>   presumably, write access to the corresponding packages) consult the
>   build results later
> – and have this person or script decide if he or it wants to commit the
>   build result to history or not

So if I understand correctly the proposed workflow is:
- git commit your changes && git push
- fedpkg build the commit
- if build succeeded:
  (the build system if we manage to, or the packager:)
  - $editor release_file
  - adjust the release value in that file
  - git add release_file && git commit && git push

The suggestion I'm proposing would be:
- git commit your changes && git push
- fedpkg build the commit
- if build succeeded:
  -  the build system tags the commit built with the release value
And we could trivially re-use the koji plugin that was written for rpmautospec
to do this, in whichever format your macros need.


This lowers the amount of manual interventions and reduces the risk of conflict
as the release_file would change for every build (conflicts which would also
appear when doing git merge and potentially in PRs, especially if that file
remains manually updated). Basically, we are introducing another "sources" file
that will conflict for every release change rather than version change.

Also, if you have long running builds (say chromium 

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-13 Thread Nicolas Mailhot via devel
Le dimanche 12 juillet 2020 à 13:07 -0700, Kevin Fenzi a écrit :
> On Sun, Jul 05, 2020 at 02:15:23PM +0200, Nicolas Mailhot via devel
> wrote:
> > 
> > This is now done in the latest code refresh and in the test copr
> > https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/c/bc4e6a79355f3a2b73abeea7e5c0e1f53b09579e?branch=forge-with-patches-auto-call-auto-changelog-auto-srpm-bump
> > https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> > 
> > I fleshed out the change page a little too
> > https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> 
> So I finally carved out a few minutes to look at this, but... 

Thank you for taking a look at it

>  
> https://copr-dist-git.fedorainfracloud.org/cgit/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/sil-charis-fonts.git
> seems to not exist? Is that some copr issue?

I honestly do not know. The copr was created with a mass import of
srpms themselves created by rpmbuild -bs. I mainly use copr to check
the srpm I build locally in mock work in another system before feeding
them to koji, so I never used the git part of copr much.


> Looking in the src.rpms I see some of the files I want to look at are
> oddly empty? 
> 
> ➜  ~ od -c rpmbuild/SOURCES/buildsys.conf
> 000  \n
> 016
> 
> Is that because it's the 'before' src.rpm without the version
> information injected yet?

Yes, almost all of the srpm have not been bumped before they were fed
to copr, so they only include the minimal whitespace necessary for
rpmbuild to register them as sources (the evil "your source is less
than 13 bytes" rpmbuild assertion).

IIRC, I scratched sources for most of them to check the new spectool
had not problem with the specs, so they won’t have any bump history in
them.

You have an example of bumped srpm in adf-accanthis-fonts (with an
almost full buildsys.conf though this package do not use the postbuild
counter). 

>  Might be nice for these files to have a small
> doc block at the top?

The detached changelog – certainly not, that would complexify its
maintenance quite a lot. The conf file, why not, that’s fairly trivial
to add.

Tough it is a litteral key = value file with no fancy formatting nor
even ini-like sections, and a handful of variables. Very boring KISS
stuff.

> Or at least the wiki page could explain each file,
> whats in it and whats the format for it. 
> 
> I really dislike having a second commit to say 'yes, this build was
> the last one'. Could you generate that info in advance and just
> commit it before the build? 

Well I really dislike people who commit that a build happened when it
has not yet, and who rewrite git history multiple times to "fix" when
they guess wrong:) Or worse, who do *not* rewrite git, and have a
changelog that clearly does not correspond to their build history.

The release part of autobumping will only happen if the spec needs it.
ie if the release the packager stored in the spec already makes the evr
newer than the last recorded evr no bump will happen. Changelog, not
sure, I suppose I could tie it up to release bumping so if one does not
happen the other does not happen either (that would make the code more
complex, but not too hard to add).

However, is not possible and it won’t be possible to pre-fill
buildsys.conf because the decision to bump or not is made by comparing
the current evr to the one stored in there. So if you pre-bump the
file, the logic will decide you have already done the corresponding
build (which, if I follow you correctly will be *true*) and add another
release bump to make sure two consecutive builds in a branch have
different evrs.

Basically, what you want is to reproduce a build you already did some
other way. Then just set the variable that triggers reproducible mode
and you’ll be done (that would require koji/copr cooperation however).

If anything changed in the buildroots you build against your build may
fail the same way it fails today, and your git and changelog will be a
lie the same way they are a lie today, but I guess than is that mode
that’ll be what you want.

There is no miracle, pre-filling means build roulette and bumping
mistakes. The proposed system is reliable because nothing except
rpmbuild itself records that a build actually happened in sources.

The change is quite simple and robust code wise but it turns people
assumptions on their head. It is *real* autobumping and not the
approximations they are used to.

In a real autobumper setup you do not pre-bump manually, anymore than a
real automated ansible setup you do not mess manually with the target
systems before you run your playbook. You can pre-mess manually and
pre-automation people are used to pre-mess manually but that’s a real
bad idea to continue once you switch to automated mode.

Regards,

-- 
Nicolas Mailhot
___

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-12 Thread James Cassell

On Sun, Jul 12, 2020, at 4:07 PM, Kevin Fenzi wrote:
> On Sun, Jul 05, 2020 at 02:15:23PM +0200, Nicolas Mailhot via devel wrote:
> > Le mercredi 01 juillet 2020 à 12:27 +0200, Dominik 'Rathann'
> > Mierzejewski a écrit :
> > > 
> > > > To get beautiful changelogs, you also need to add
> > > > 
> > > > 
> > > > %buildsys_name Your name
> > > > %buildsys_email Your email
> > > > 
> > > > 
> > > > in ~/.rpmmacros
> > > 
> > > What about having one macro called %buildsys_packager instead of two?
> > > You're always using them together, anyway. It'd be similar to the
> > > existing %packager macro, too.
> > 
> > This is now done in the latest code refresh and in the test copr
> > https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/c/bc4e6a79355f3a2b73abeea7e5c0e1f53b09579e?branch=forge-with-patches-auto-call-auto-changelog-auto-srpm-bump
> > https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> > 
> > I fleshed out the change page a little too
> > https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> 
> So I finally carved out a few minutes to look at this, but... 
> 
> https://copr-dist-git.fedorainfracloud.org/cgit/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/sil-charis-fonts.git
> seems to not exist? Is that some copr issue?
> 

I've been unable to clone COPR dist-git for quite some time. Seems it's broken 
from outside COPR itself...

V/r,
James Cassell
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-12 Thread Kevin Fenzi
On Sun, Jul 05, 2020 at 02:15:23PM +0200, Nicolas Mailhot via devel wrote:
> Le mercredi 01 juillet 2020 à 12:27 +0200, Dominik 'Rathann'
> Mierzejewski a écrit :
> > 
> > > To get beautiful changelogs, you also need to add
> > > 
> > > 
> > > %buildsys_name  Your name
> > > %buildsys_email Your email
> > > 
> > > 
> > > in ~/.rpmmacros
> > 
> > What about having one macro called %buildsys_packager instead of two?
> > You're always using them together, anyway. It'd be similar to the
> > existing %packager macro, too.
> 
> This is now done in the latest code refresh and in the test copr
> https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/c/bc4e6a79355f3a2b73abeea7e5c0e1f53b09579e?branch=forge-with-patches-auto-call-auto-changelog-auto-srpm-bump
> https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> 
> I fleshed out the change page a little too
> https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

So I finally carved out a few minutes to look at this, but... 

https://copr-dist-git.fedorainfracloud.org/cgit/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/sil-charis-fonts.git
seems to not exist? Is that some copr issue?

Looking in the src.rpms I see some of the files I want to look at are
oddly empty? 

➜  ~ od -c rpmbuild/SOURCES/buildsys.conf
000  \n
016

Is that because it's the 'before' src.rpm without the version
information injected yet? Might be nice for these files to have a small
doc block at the top? Or at least the wiki page could explain each file,
whats in it and whats the format for it. 

I really dislike having a second commit to say 'yes, this build was the
last one'. Could you generate that info in advance and just commit it
before the build? ie, do your update/changes, test it, then run
something 'fedpkg build'? that just fills the things and lets you commit
them than requests the build?

I'm sure there's a ton more here... but thats what I had time for today. 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-06 Thread Nicolas Mailhot via devel

Le 2020-07-05 23:55, Dan Čermák a écrit :

Hi Dan


So essentially you store the changelog in a separate file


The changelog is already detached in the F33 change
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

This F34 change adds bumping to the detached file
https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping


and use that to calculate the next release field?


The changelog file is not used as source of data, except as a reference 
state that will be added to. Using the changelog file as source of data 
would require quite a lot of complex and unreliable changelog format 
parsing, so the bumping data is taken from another key = value file 
(that uses the same persistence mechanisms).


Also the packager may decide to trim quite a lot of intermediary 
changelog events, so the EVR and date in the last changelog entry are 
not necessarily the EVR and date of the last build.



Given the other replies to this thread (that this is all local only,
unless koji does git commits), does that mean that it's still: dist-git
commit = rebuild.


To be part of official Fedora history the result of a build needs to be 
committed yes. The change does not force you to build every commit, nor 
to commit every build out there.



The "only" difference to the current state of affairs
being, that I don't have to specify the Release field myself?


Once you've modified a spec, and set a starting evr point in this spec, 
further rebuilds do not involve touching the spec. Spec changes are real 
spec changes, not spec changes to bump a release or bump a changelog.



The packager does not have to request the modification in his spec,
it’s done as part of the various %auto_foo calls the change introduced


Could you provide an example how this would look in practice?


If you want a demonstration of the auto framework and of changelog 
detaching, you can take any of the non macro builds in

https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-changelog-fonts/builds/

If you want to see a demonstration of autobumping, you need to rpmbuild 
-ba manually right now, because of the two small limitations mock side. 
So you need to take the redhat-rpm-config and fonts macro packages in:

https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/

and rebuild one of the other packages in there.

The only difference between the two coprs is the redhat-rpm-config 
package, there is no change in the fonts macro package or in the 
automated packages themselves. Autobumping can be implemented without 
any spec file change once the auto framework is used.


(The mock limitations are first, the fact that mock currently collects 
the SRPM at the start, not end of the build process, and second, the 
fact you need to pass packager ID that will end up in the changelog bump 
to the build process and there is no way in the copr/copr UI do do 
that.)



What I am currently missing from this proposal though is:
- How is this actually even implemented?
- How will this look in practice?


See above ↑

- Given that additional files would be put into dist-git, how do we 
roll

  this back in case things go wrong? (Having thousands of "remove
  %autorelease" commits by releng could be an option here, albeit not a
  pretty one).


Since bumping is a feature over the auto framework, and does not require 
any additional spec change, it is enabled by registering bumping 
processing in this framework, and disabled by removing this 
registration. There is no need to change spec files or history. In fact 
I use the same spec files to QA the auto framework and bumping, and 
depending if the redhat-rpm-config version I test includes the bumping 
or not, they will bump or not.


When bumping code is not present the additional key=value file bumping 
uses is not auto-added to sources, so the next srpm import will clear it 
from sources the same way patches disappear from sources once no longer 
used (and can linger forever if a packager does not import srpms and 
does not git rm those files explicitly).


Removing the auto framework is something else altogether. Because its 
aim is to massively simplify spec files (in opt-in, not mandatory mode), 
you can not go back without undoing the spec simplifications.


However, because great care was taken to define a clean and generic spec 
syntax when using the auto framework, you could replace it will multiple 
reimplementations without changing spec files. The %auto framework spec 
API is basically


%prep
%auto_prep ← automated processing

It’s hard to go more generic than that. (You might want to remove the 
%auto calls altogether and have %prep, for example, call %prep by 
default, but that would remove the packager choice to use or not the 
%auto calls, and to insert custom processing before those calls).


The only "irregularity" is that the %auto 

Re: Can we do away with release and changelog bumping? (was: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal)

2020-07-06 Thread Nicolas Mailhot via devel
Le dimanche 05 juillet 2020 à 23:36 +0200, Dan Čermák a écrit :
> Nicolas Mailhot via devel  writes:
> 
> > Le dimanche 05 juillet 2020 à 17:46 +0200, Björn Persson a écrit :
> > > Nicolas Mailhot via devel wrote:
> > > > So if you want to push Fedora release logic to its ultimate
> > > > conclusion,
> > > > the thing that should be in charge of committing the new
> > > > release/changelog build state to package history in git is
> > > > bodhi,
> > > > not
> > > > koji.
> > > 
> > > Why do build events need to be recorded in the Git history in the
> > > first place? 
> > 
> > The changelog is built-in the rpm format. Therefore, it needs to
> > exists
> > at rpmbuild stage. Therefore, you need to record past changelog
> > state
> > so new builds are consistent with previous builds.
> 
> The changelog should be consistent, but it needn't record every
> single
> build event. Otherwise OBS would not work at all: the build system
> bumps
> the release field automatically on each rebuild, but it does not
> touch the changelog at all.

Well just bumping the changelog by default, and letting the packager
remove unecessary lines when he has the time to look at the changelog,
is a much friendlier workflow than asking the packager to think about
it on every single build, redoing builds when he forgot about it
because redoing builds is the only way to get a new changelog included
in the binary payload.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-05 Thread Dan Čermák
Hi Nicolas,

Nicolas Mailhot  writes:

>> How do I let rpm generate the changelog automatically?
>
> This feature is not changelog generation, just changelog bumping on
> build events. You still need some other method to put non-build events
> in the changelog.
>
> The detached changelog is just one more file in SRPM sources, which is
> modified by rpmbuild at `%build` time with other files rpmbuild
> modifies. The tricky part is to modify the source file as a source file
> so rpmbuild adds the result to the produced SRPM (and, that does not
> work in mock right now, because mock serves the SRPM that existed at
> the start, not at the end of the build. Though it’s probably just a
> matter of getting mock to call again its SRPM creation method at the
> end of the build).

So essentially you store the changelog in a separate file (like it is
done in openSUSE) and use that to calculate the next release field?

Given the other replies to this thread (that this is all local only,
unless koji does git commits), does that mean that it's still: dist-git
commit = rebuild. The "only" difference to the current state of affairs
being, that I don't have to specify the Release field myself?

>
> The packager does not have to request the modification in his spec,
> it’s done as part of the various %auto_foo calls the change introduced

Could you provide an example how this would look in practice?

>
>> And is this related to Piere/Pingou's work on the same topic that was
>> deployed to koji staging?
>
> It’s a different implementation, at the rpm level, that does not tie
> bumping to Fedora infra (koji included). Though, it is probably
> complementary to what pingou did on the changelog alimentation front.
>
> IMHO the design mistake so far was to conflate bumping and non-build
> event changelog filling. You need to do both of course but build event
> should be a build event driven by the lowest common denominator
> (rpmbuild) with koji/infra scrapping rpmbuild results as usual and
> exposing them to users.

This is a good point imho: not every rebuild warrants a changelog entry
and having both separated appears sensible to me.


What I am currently missing from this proposal though is:
- How is this actually even implemented?
- How will this look in practice?
- Given that additional files would be put into dist-git, how do we roll
  this back in case things go wrong? (Having thousands of "remove
  %autorelease" commits by releng could be an option here, albeit not a
  pretty one).


Cheers,

Dan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: Can we do away with release and changelog bumping? (was: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal)

2020-07-05 Thread Dan Čermák
Nicolas Mailhot via devel  writes:

> Le dimanche 05 juillet 2020 à 17:46 +0200, Björn Persson a écrit :
>> Nicolas Mailhot via devel wrote:
>> > So if you want to push Fedora release logic to its ultimate
>> > conclusion,
>> > the thing that should be in charge of committing the new
>> > release/changelog build state to package history in git is bodhi,
>> > not
>> > koji.
>> 
>> Why do build events need to be recorded in the Git history in the
>> first place? 
>
> The changelog is built-in the rpm format. Therefore, it needs to exists
> at rpmbuild stage. Therefore, you need to record past changelog state
> so new builds are consistent with previous builds.

The changelog should be consistent, but it needn't record every single
build event. Otherwise OBS would not work at all: the build system bumps
the release field automatically on each rebuild, but it does not touch
the changelog at all.


Cheers,

Dan


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: Can we do away with release and changelog bumping? (was: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal)

2020-07-05 Thread Nicolas Mailhot via devel
Le dimanche 05 juillet 2020 à 18:41 +0200, Nicolas Mailhot a écrit :
> 
> While timestamping would remove the need to pass the last build info
> to the next one it would also break all the workflows where several
> rebuilds are done in parallel for separate needs, and the latest
> rebuild is not necessarily the one you want to keep.

(This is why git is not relying on timestamps to construct commit
timelines, for example. A reliable timeline system knows what change
occured before the next one.)

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: Can we do away with release and changelog bumping? (was: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal)

2020-07-05 Thread Nicolas Mailhot via devel
Le dimanche 05 juillet 2020 à 17:46 +0200, Björn Persson a écrit :
> Nicolas Mailhot via devel wrote:
> > So if you want to push Fedora release logic to its ultimate
> > conclusion,
> > the thing that should be in charge of committing the new
> > release/changelog build state to package history in git is bodhi,
> > not
> > koji.
> 
> Why do build events need to be recorded in the Git history in the
> first place? 

The changelog is built-in the rpm format. Therefore, it needs to exists
at rpmbuild stage. Therefore, you need to record past changelog state
so new builds are consistent with previous builds.

You may argue that the existence of scms like git makes built-in
changelogs irrelevant. People that have to debug problems in systems
that mix packages sources will disagree – they have no wish to comb
multiple external scms to find what was changed in a package set that
breaks (hard to do when the thing that broke is networking for
example).

And, even if you removed completely the changelog from the rpm format,
you’d still need to manage package releases.

> So why is NEVR required to change when nothing in the package source
> has changed? 

The NEVR is required to change because you need to publish a new
package ID to clients, so clients know they have an update to deploy.
That has nothing to do with koji limitations, it’s a requirement or the
rpm update system, or pretty much any change management system for that
matter.

> It seems that several problems would just disappear if a rebuild
> would generate a unique package ID without a Git commit.

That’s exacly what the change does.

> Here's an idea: We could mandate that Release must expand a macro
> called buildtag. This macro would have a new value every time,
> monotonically increasing. A timestamp seems easiest, but that should
> bean implementation detail that could be changed without breaking
> things.

Because things are slightly more complex than you think they are, the
counter is not just a timestamp, and there are two not one counter, but
yes, again, that’s exacly what the change does.

> (The build time is already stored in packages, and could
> theoretically be used to tell builds apart, but that would require
> changes to lots of tools I guess.)

The build time by itself is not sufficient because you have branches,
and scratch builds (which are basically abandonware branches) so
parallel package history exists and a single monotonic timeline can not
represent that.

Nevertheless the last build time is useful (for changelog bumping, if
nothing else) and is one of the things the change stores as past state
(you could try to deduce it from other things, but why bother, a
timestamp variable is cheap and easy to manipulate).

> Mass rebuilds would no longer make any Git commits. They would just
> take the latest version and submit it for building. 

Again, that’s basically what the change does. A build stores counters
and date as they existed as the time of the build in one of the SRPM
source files. The next build reads this file and computes a higher
release. No external bump script involved. No spec file change
required. The spec file can be left unchanged forever, the release info
in there is just the last release someone made a change to the spec
files, and everything autobumps from there.

All you need is to pass the "last build" info from one build to
another, which is done via importing the results of last build at the
start of the new build. Since the import relies on SRPM content and
nothing else you can even move the build chain from one buildsystem to
another it will still work. 

While timestamping would remove the need to pass the last build info to
the next one it would also break all the workflows where several
rebuilds are done in parallel for separate needs, and the latest
rebuild is not necessarily the one you want to keep.

Regards,


-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Can we do away with release and changelog bumping? (was: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal)

2020-07-05 Thread Björn Persson
Nicolas Mailhot via devel wrote:
> So if you want to push Fedora release logic to its ultimate conclusion,
> the thing that should be in charge of committing the new
> release/changelog build state to package history in git is bodhi, not
> koji.

Why do build events need to be recorded in the Git history in the first
place? A version control system is supposed to track changes to the
source code. A rebuild that doesn't change any sources, patches,
scriptlets or metadata shouldn't need to change anything in Git.

As far as I can tell, this happens only because Koji refuses to build a
NEVR that has been built before, so a rebuild requires a new release
number, which requires an RPM changelog entry, and both of those are
defined in the spec, which is stored in Git.

So why is NEVR required to change when nothing in the package source
has changed? Apparently because *other* packages are likely to have
changed: new versions of libraries, compiler et cetera causing
differences in the generated code. That's the usual reason for rebuilds
after all. But if one takes a source package and rebuilds it locally,
then one won't have the same version of every other package, so one
gets another binary package with the same NEVR but probably different
binary code.

It seems that several problems would just disappear if a rebuild would
generate a unique package ID without a Git commit. Instead of a lot of
complex tooling to automate release and changelog bumping, I would like
to see the need for release and changelog bumping go away.

Here's an idea: We could mandate that Release must expand a macro
called buildtag. This macro would have a new value every time,
monotonically increasing. A timestamp seems easiest, but that should be
an implementation detail that could be changed without breaking things.

The macro could be defined like this for example:

  %buildtag .%(date +%%s)

It would be used in each spec like this:

  Release: 1%{?dist}%{?buildtag}

Putting the buildtag after the disttag makes it possible to change how
the buildtag is generated in a future Fedora release without breaking
upgrade paths.

(The build time is already stored in packages, and could theoretically
be used to tell builds apart, but that would require changes to lots of
tools I guess.)

Mass rebuilds would no longer make any Git commits. They would just
take the latest version and submit it for building. The release number
would remain as a downstream version number. Packagers would update the
release number when they make actual changes to the package. When one
wants a specific version of a package, one would look at the version
and release numbers, ignoring the buildtag. The buildtag would
distinguish between different builds of the same version-release.

What flaws can you all find in this idea?

Björn Persson


pgpIGk2i3K6iH.pgp
Description: OpenPGP digital signatur
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-05 Thread Nicolas Mailhot via devel
Le mercredi 01 juillet 2020 à 12:27 +0200, Dominik 'Rathann'
Mierzejewski a écrit :
> 
> > To get beautiful changelogs, you also need to add
> > 
> > 
> > %buildsys_name  Your name
> > %buildsys_email Your email
> > 
> > 
> > in ~/.rpmmacros
> 
> What about having one macro called %buildsys_packager instead of two?
> You're always using them together, anyway. It'd be similar to the
> existing %packager macro, too.

This is now done in the latest code refresh and in the test copr
https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/c/bc4e6a79355f3a2b73abeea7e5c0e1f53b09579e?branch=forge-with-patches-auto-call-auto-changelog-auto-srpm-bump
https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/

I fleshed out the change page a little too
https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Nicolas Mailhot via devel
Le vendredi 03 juillet 2020 à 09:48 +0200, Pierre-Yves Chibon a écrit :
> On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote:
> > Nicolas Mailhot wrote:
> > > The same process that commits a new state of the changelog file
> > > in 
> > > sources,
> > > commits the date that was written in the changelog in a separate
> > > key = 
> > > value
> > > file (with the components of the build evr, the last packager id,
> > > etc).
> > 
> > Do you mean that the key/value file will be committed to Git from
> > inside
> > Koji? Do the Koji builders have write access to Git?
> 
> This is the part that worries me a little about this approach.
> Builders currently do not have commit access to git and I'm not sure
> if we want them to considering they have git installed (so they can
> clone) as well as access to all the packages in dist-git from a
> networking point of view (again so they can clone).
> So if we were to give the builders commit access to dist-git, an
> attacker could easily commit to any other packages, potentially from
> something as easy as a scratch-build.

From a purely architecture POW I’m convinced the proposed approach is
the correct approach. Anything else proposed so far involves:
  – tying a low-level event like "build occurred at date XXX" to high-
level Fedora infra (making our workflow non portable and 
incompatible with downstreams and third parties)
  – taking bets in git that a build will occur and succeed (before it 
actually occurs and succeeds, in real life builds fail for various
reasons), and
  – attempting to munge spec file behind the packager back (unlikely to
work fine the more automated and dynamic we made those).

However, because it’s the correct architecture solution, it also forces
to make hard architectural choices, instead of mixing unrelated things
in git and pretending that makes the result fine. Mixing unrelated
things in a pile of container or git poo and pretending the result is
fine is exactly what I hate in contenerish build workflows and why I
work on rpm packaging.

From a pure high-level view, the thing in our infra that gates builds
and decides whether they are official or scratched is bodhi.

So if you want to push Fedora release logic to its ultimate conclusion,
the thing that should be in charge of committing the new
release/changelog build state to package history in git is bodhi, not
koji. And you can put security related checks there, since deciding to
push things to users requires security related checks anyway (that
probably also involves branching while a bodhi update is in flight and
not approved yet).

However, that’s if you want to push the model to its ultimate
conclusion and have something nice solid, automated, and future-proof.

If you don’t want to touch bodhi, and it you do not want koji to commit
to git (which, is not the best of things for the reasons you stated,
and for the reasons I stated), you can just:
– make the koji client return the URL that will contain the SRPM at the
  end of the build process if it succeeds.
– have the person of script that called the koji client (and has,
  presumably, write access to the corresponding packages) consult the
  build results later
– and have this person or script decide if he or it wants to commit the
  build result to history or not

That’s the REST way of doing things. It’s a co-out because you push
hard commit decisions to the client, but it’s a prefectly valid
approach. The commit decision exists with or without my change, it’s
just people have (successfully) convinced themselves git is magic and
git makes release decisions go away.

You could also try to filter source files to limit the back commit to
specific files. But really, if you don’t trust your build process to
modify files in a secure way, you should not distribute the produced
RPMs in the first place.

> rpmautospec relies on git tags to store the build info, could it be
> considered here?

As explained above, that does not solve any of the hard problems, that
handwaves them away by pretending that because someone filled some
metadata in git, it corresponds to the actual build state.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Pavel Raiskup
On Friday, July 3, 2020 9:48:06 AM CEST Pierre-Yves Chibon wrote:
> So if we were to give the builders commit access to dist-git, an attacker
> could easily commit to any other packages, potentially from something as easy
> as a scratch-build.

Absolutely!

Koji authenticates build submitters (I'm not sure it authorizes them).  So
technically, _something_ on backend could be allowed to commit to dist-git
(in the name of build submitter).

Before the SRPM build task, Koji could request "GetReleaseBumpPatch" task, the
builder could then just read-only clone the git, bump the release, return the
patch back for backend -- and let Koji apply it.

But yeah, that's off topic a bit.  This is not what the current proposal is
about.

Pavel


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Pierre-Yves Chibon
On Thu, Jul 02, 2020 at 03:44:51PM +0200, Nicolas Mailhot via devel wrote:
> Le 2020-07-02 15:11, Kamil Dudka a écrit :
> >On Thursday, July 2, 2020 1:02:05 PM CEST Nicolas Mailhot via
> >devel wrote:
> >>If there is buy-in, it will be implemented by goodwill people.
> >>If there
> >>is no buy-in, it won’t, normal community development process. Put
> >>yourself in the category you want to be in, your choice, not mine.
> >
> >I believe that Change submission guidance is pretty clear on this:
> >
> >"If you have improvement in mind, work to get implementers
> >committed
> >to the effort _before_ filing a Change proposal, rather than
> >expecting
> >them to show up for work once the Change is accepted."
> 
> This is a F34 Change (not that it could not be done for F33 if
> people were willing). 

This is apparently something that was missed, just by seeing the subject of this
thread.


Pierre
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Nicolas Mailhot via devel

Le 2020-07-02 15:11, Kamil Dudka a écrit :
On Thursday, July 2, 2020 1:02:05 PM CEST Nicolas Mailhot via devel 
wrote:
If there is buy-in, it will be implemented by goodwill people. If 
there

is no buy-in, it won’t, normal community development process. Put
yourself in the category you want to be in, your choice, not mine.


I believe that Change submission guidance is pretty clear on this:

"If you have improvement in mind, work to get implementers 
committed
to the effort _before_ filing a Change proposal, rather than 
expecting

them to show up for work once the Change is accepted."


This is a F34 Change (not that it could not be done for F33 if people 
were willing). It was filled at the same time as the F33 change because 
it’s the logical continuation of the F33 change and people were sure to 
ask about it in the other change discussion (as they did).


I believe a full cycle is largely enough to get people used to the idea 
and committed or not. It was split from the F33 change to give this 
cycle for things to mature.


The F33 change is continuation of work I began in 2017, and has served 
Fedora well since, and the same objections were raised against the 2017 
change, and the same empty promises were made by naysayers that they 
would do better someday, and 2+ years later no one has seen any part of 
their alternative implementation in real life (laughably, some of the 
naysayers complained it was not documented well enough, and blocked the 
documentation merge later when it was provided). Also the naysayers are 
not the ones that would do the implementation since no one sane would 
count on them to do anything anyway.


The work is done, anyone can get the SRPMS in the copr and check by 
himself that they do autobump and auto changelog, there are minor 
blockers because the feature changes slightly the point at which the 
SRPM should be collected in the build process, but they are *minor*, not 
anything that requires a huge re-architecture of current tooling.


Which is pretty cool because current tooling was not designed around 
this feature, and yet it works fine, with minor adjustments.


Regards,
--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Nicolas Mailhot via devel


Le July 2, 2020 2:47:49 PM UTC, Vitaly Zaitsev via devel 
 a écrit :
>On 02.07.2020 11:27, Nicolas Mailhot wrote:
>> Why? Koji schedules a build. The build registers its own build date
>in
>> the produced packages. Koji decides to keep and commit the result, or
>> drop it (scratch build, failed side tag, whatever). Koji is still in
>> charge, the bumping is just integrated in the build process with the
>> rest of the package creation.
>
>Koji was just an example. %changelog section should be auto-generated
>from commits messages. I don't want to maintain a separate file with
>the changelog.

The feature is completely compatible with this workflow. 

It registers build events in a detached file (the only part of the rpm 
changelog that requires knowledge of the rpm format)

You can prime this file via git hooks or any other system you like, the feature 
will take it as is, add the timestamp the build occurred at, and feed the 
result to the correct parts of the rpm build process.
-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-03 Thread Pierre-Yves Chibon
On Thu, Jul 02, 2020 at 12:10:58PM +0200, Björn Persson wrote:
> Nicolas Mailhot wrote:
> > The same process that commits a new state of the changelog file in 
> > sources,
> > commits the date that was written in the changelog in a separate key = 
> > value
> > file (with the components of the build evr, the last packager id, etc).
> 
> Do you mean that the key/value file will be committed to Git from inside
> Koji? Do the Koji builders have write access to Git?

This is the part that worries me a little about this approach.
Builders currently do not have commit access to git and I'm not sure if we want
them to considering they have git installed (so they can clone) as well as
access to all the packages in dist-git from a networking point of view (again so
they can clone).
So if we were to give the builders commit access to dist-git, an attacker could
easily commit to any other packages, potentially from something as easy as a
scratch-build.

rpmautospec relies on git tags to store the build info, could it be considered
here? It may make things a little safer as we could then restrict the access of
that user/ssh key to only git tags (or do like rpmautospec and query pagure's
API to have it create the git tag, thus dropping the need for ssh key).


Pierre



pgpTnAYdhtGD_.pgp
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Vitaly Zaitsev via devel
On 02.07.2020 11:27, Nicolas Mailhot wrote:
> Why? Koji schedules a build. The build registers its own build date in
> the produced packages. Koji decides to keep and commit the result, or
> drop it (scratch build, failed side tag, whatever). Koji is still in
> charge, the bumping is just integrated in the build process with the
> rest of the package creation.

Koji was just an example. %changelog section should be auto-generated
from commits messages. I don't want to maintain a separate file with the
changelog.

-- 
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Kamil Dudka
On Thursday, July 2, 2020 1:02:05 PM CEST Nicolas Mailhot via devel wrote:
> If there is buy-in, it will be implemented by goodwill people. If there 
> is no buy-in, it won’t, normal community development process. Put 
> yourself in the category you want to be in, your choice, not mine.

I believe that Change submission guidance is pretty clear on this:

"If you have improvement in mind, work to get implementers committed
to the effort _before_ filing a Change proposal, rather than expecting
them to show up for work once the Change is accepted."

See https://docs.fedoraproject.org/en-US/program_management/changes_guide/

Kamil

> Implementation is moving the call to SRPM creation at the end of the 
> build process instead of relying on the SRPM as it existed at the start 
> of the build process. So, while it is work, it is not complex work (I’m 
> sure there is more than that because tuning a production process is more 
> than the "it works" POC stage, but that’s tuning, not reconception).
> 
> Regards,
> 
> -- 
> Nicolas Mailhot

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 12:42, Igor Raits a écrit :


So who is going to implement necessary changes in mock and koji for
this proposal to be complete?


If there is buy-in, it will be implemented by goodwill people. If there 
is no buy-in, it won’t, normal community development process. Put 
yourself in the category you want to be in, your choice, not mine.


Implementation is moving the call to SRPM creation at the end of the 
build process instead of relying on the SRPM as it existed at the start 
of the build process. So, while it is work, it is not complex work (I’m 
sure there is more than that because tuning a production process is more 
than the "it works" POC stage, but that’s tuning, not reconception).


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Igor Raits
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Thu, 2020-07-02 at 12:20 +0200, Nicolas Mailhot via devel wrote:
> Le 2020-07-02 11:59, Florian Weimer a écrit :
> > * Nicolas Mailhot via devel:
> > 
> > > Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :
> > > > On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
> > > > > The detached changelog is just one more file in SRPM sources,
> > > > > which 
> > > > > is
> > > > > modified by rpmbuild at `%build` time with other files
> > > > > rpmbuild
> > > > > modifies.
> > > > 
> > > > I don't like that. %changelog should be generated automatically
> > > > on 
> > > > Koji
> > > > side.
> > > 
> > > Why? Koji schedules a build.
> > 
> > No, Koji also builds the SRPM, via fedpkg-simple or a similar 
> > mechanism.
> 
> Sure, by build I intended both the deployment packages and the SRPM.
> 
> The main difference between the current workflow (the reason it fails
> in 
> mock right now, but that should not be too hard to fix) is that the
> SRPM 
> that includes the build info is itself a result of the rest of the 
> build.
> 
> That seems the main point people misunderstand (thank you for making
> me 
> clarify it), the proposal does not involve preparing a special SRPM
> out 
> of band, that is then fed to koji, the SRPM containing the bumped 
> changelog and last build info is the result of the build process 
> alongside the binary packages.
> 
> mock (and koji) just have to pick this SRPM at the end of the build
> and 
> not use the SRPM as it existed before the build occurred.

So who is going to implement necessary changes in mock and koji for
this proposal to be complete?

> And why is it that way? You do not consider a rpmbuild -bs a build
> event 
> do you? We do it all the time to import packages from one system to 
> another. The only thing which is a real build that produces a bump
> and 
> is stored in changelog history is a full rpmbuild -ba build.
> 
> Regards,
> 
> -- 
> Nicolas Mailhot
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

- -- 
Igor Raits 
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl79uiIACgkQEV1auJxc
Hh6DWA//QXrE2wOef89QaKs/2KW9boWCZu3uapwHC+479iB2b1E3J7iw5c1JW03s
0BQ9sk+blA7WntICR2ebkq+wjv5A8upLDj7OGABxIag04MlMHXKD3HZtT67QoMUi
vNEdxHr98oim7S/wI5ZizAAjxFApVoZmpsokaATc0r32wcyczVsWvBxEYmjw7Uo6
QqN38o/Z8Y9xaTDHf/S4d+FTUTbKmetpLmb5tLg25m62+0GaFMpfYo0lCuFID2on
Ionk7+b0LH8DHaacxBwoV+6HiOno+qYLqDBLOKh5kJGFGr+o33uqqTfc3/7KouZh
ui9F7TxeRpd+bbVhfQbAdQZEIWQCVaFSIza8YN8mXe6jcyyqqWjUOhWgGt6ihaSs
h255wdI8VcZJXTSYt+qkXj5Z6fGvvDXjD+AoEtFyhP9sqm5qmFyEIm7xE/NGwFMj
BO6xIumEFavP6lEylFtYH9j6q9LFvYBB5ITw33jI4Wq/lYbD4cB96ksuzz1FvDzp
R76Y10H0naFmqN0IdLC7L8LV45uXFg/jZ0NCLQ/6d0kUm/FbggmhpePzJUZI/b3T
phokAdf0RQ32NXXRj/unK8yMwaZfFd8ec+UK4VFb1+pECjgyvGxojp+vo7pVzEu8
QCTy+Cb5T96mwfO9ejFQMGV2VZBBoVPYsGc0hTmg4j7eA7fPiE8=
=FeYg
-END PGP SIGNATURE-
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 11:59, Florian Weimer a écrit :

* Nicolas Mailhot via devel:


Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :

On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
The detached changelog is just one more file in SRPM sources, which 
is

modified by rpmbuild at `%build` time with other files rpmbuild
modifies.


I don't like that. %changelog should be generated automatically on 
Koji

side.


Why? Koji schedules a build.


No, Koji also builds the SRPM, via fedpkg-simple or a similar 
mechanism.


Sure, by build I intended both the deployment packages and the SRPM.

The main difference between the current workflow (the reason it fails in 
mock right now, but that should not be too hard to fix) is that the SRPM 
that includes the build info is itself a result of the rest of the 
build.


That seems the main point people misunderstand (thank you for making me 
clarify it), the proposal does not involve preparing a special SRPM out 
of band, that is then fed to koji, the SRPM containing the bumped 
changelog and last build info is the result of the build process 
alongside the binary packages.


mock (and koji) just have to pick this SRPM at the end of the build and 
not use the SRPM as it existed before the build occurred.


And why is it that way? You do not consider a rpmbuild -bs a build event 
do you? We do it all the time to import packages from one system to 
another. The only thing which is a real build that produces a bump and 
is stored in changelog history is a full rpmbuild -ba build.


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Björn Persson
Nicolas Mailhot wrote:
> The same process that commits a new state of the changelog file in 
> sources,
> commits the date that was written in the changelog in a separate key = 
> value
> file (with the components of the build evr, the last packager id, etc).

Do you mean that the key/value file will be committed to Git from inside
Koji? Do the Koji builders have write access to Git?

> commit the new build event timestamp in 
> the detached changelog file at %build time

%build is executed once per arch, on different builders, so which
builder's timestamp gets committed to Git?

Björn Persson


pgpGqB7XyDrRt.pgp
Description: OpenPGP digital signatur
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Florian Weimer
* Nicolas Mailhot via devel:

> Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :
>> On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
>>> The detached changelog is just one more file in SRPM sources, which is
>>> modified by rpmbuild at `%build` time with other files rpmbuild
>>> modifies.
>>
>> I don't like that. %changelog should be generated automatically on Koji
>> side.
>
> Why? Koji schedules a build.

No, Koji also builds the SRPM, via fedpkg-simple or a similar mechanism.
That seems to be natural place where the SPEC file is completed for
inclusion in the SRPM, in my opinion.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 11:38, Igor Raits a écrit :

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Thu, 2020-07-02 at 11:27 +0200, Nicolas Mailhot via devel wrote:

Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :
> On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
> > The detached changelog is just one more file in SRPM sources,
> > which is
> > modified by rpmbuild at `%build` time with other files rpmbuild
> > modifies.
>
> I don't like that. %changelog should be generated automatically on
> Koji
> side.

Why? Koji schedules a build. The build registers its own build date
in
the produced packages. Koji decides to keep and commit the result, or
drop it (scratch build, failed side tag, whatever). Koji is still in
charge, the bumping is just integrated in the build process with the
rest of the package creation.


I think Change Page does not mention that Koji will be committing
anything to the dist-git.


It does not need koji to commit anything to dist-git. While that would 
be nice to have, the back-commit can be done by the human who scheduled 
the build, or by fedpkg, or whatever. That also means that till this 
back commit is done whoever scheduled the build can decide to scratch it 
all as a dead evolutionary branch. And you can do "I feel lucky" tests 
and forget about them if they turn out bad.


But yes, as was discussed on this list in the beginning of the year, in 
a model where bumping is infrastructure-independent and done at the rpm 
level, various infrastructures still needs to pick up the results of rpm 
builds and do whatever they want to do with them.


rpmbuild can create rpms, it can not record things in 
organization-specific systems.


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Igor Raits
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Thu, 2020-07-02 at 11:27 +0200, Nicolas Mailhot via devel wrote:
> Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :
> > On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
> > > The detached changelog is just one more file in SRPM sources,
> > > which is
> > > modified by rpmbuild at `%build` time with other files rpmbuild
> > > modifies.
> > 
> > I don't like that. %changelog should be generated automatically on
> > Koji
> > side.
> 
> Why? Koji schedules a build. The build registers its own build date
> in 
> the produced packages. Koji decides to keep and commit the result, or
> drop it (scratch build, failed side tag, whatever). Koji is still in 
> charge, the bumping is just integrated in the build process with the 
> rest of the package creation.

I think Change Page does not mention that Koji will be committing
anything to the dist-git.

> And, unlike something done specifically by koji, the bumping will
> import 
> and export across all build systems, ie all the bumps that occurred
> in 
> rpmbuild, or mock, or copr, or obs, or whatever are imported in
> fedpkg 
> with the rest of the srpm, and an srpm produced by koji can import in
> rpmbuild, or mock, or copr, or obs, or whatever without loss of bump 
> information.
> 
> And you no longer waste hours wondering why a package you just fixed
> is 
> still failing on your test systems before realizing it is masked by 
> another build that did not share the same bump history.
> 
> Regards,
> 
> -- 
> Nicolas Mailhot
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

- -- 
Igor Raits 
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl79qyoACgkQEV1auJxc
Hh7/uQ/+PU+KAwd+lCd4YMAEd41Caii7BBMSIOn7WQ5EJEJu9V4g7QvNFGi147Pw
95DH9jZNf6KIFbVNh32gi44fZ60Q2MCj4/H5Mq3PoGe+wa3ks4twbLox8rG2M1BG
EWQVWXi8HWYZ5gR7QBUmVW0rpEYuyGNUMSFsavdAAv6hfq2LQMAA9lbSerWmOikK
w+2SUuoo1D5l1ps8Bi7L91ZFH6Z7K9sdXxG8rhuK9EXuf63DwfXbraYCQdwOeCHP
wXvyRkQ8D5w9MiFriN4e6zqp8SI8QvPnqhLWO9RoqYDr9FMc1tqKeaZN/yuIdu+Z
zIQU1M1N4cfOm91TvNH/gAev9e9X9dvJMKPsVZF2K4yAE0MSy4EgjY0ixvv0KZWq
RRTXqs/fYF/KLVfuwrI5ZVJTcthvuUi4YL+LDoC/TL7SfLJVTVv5t1ub8l0geTBc
isFdIntggJEUxtovM/FG2/affYNJnxIW9diGJkQYYtqo7TpuHtW2yoYF2QJ3eKim
yW0UQiyNZ81wpfHns0dzGm3EyUrqrWlazRCcgaxfH++MvVj2TSYnyiAmEF4YZg0r
jFcjyMZ7ZXyekUvhjBBNMBow5dZ1F2178cEOeHhC8S4NFrX+UkuUkQE+UjMarr3p
X6W2z2Gz/L/m3c1P+kLKYbMcpQCmzh2U6Or6F9zlUdwFLJTt4qw=
=Y5uf
-END PGP SIGNATURE-
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 11:21, Igor Raits a écrit :

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Thu, 2020-07-02 at 11:17 +0200, Nicolas Mailhot wrote:

Le 2020-07-02 09:52, Florian Weimer a écrit :
> * Nicolas Mailhot via devel:
>
> > > How do I let rpm generate the changelog automatically?
> >
> > This feature is not changelog generation, just changelog bumping
> > on
> > build events. You still need some other method to put non-build
> > events
> > in the changelog.
>
> What is “changelog bumping”?  Why is it needed?  What about release
> bumping?

Changelog bumping is the act of putting the actual release bump and
build time in the changelog.

With the change, the spec is able to self-compute its next release if
the spec file evr is older or equal to the last build event.


How does it know that "last build event"?


The same process that commits a new state of the changelog file in 
sources,
commits the date that was written in the changelog in a separate key = 
value

file (with the components of the build evr, the last packager id, etc).

That means, you can trim the detached changelog file (if you find the 
list of build events uninteresting), the SRPM will still remember to 
bump the next EVR to something above the last build (even if it does not 
appear in the changelog file).


(That also means I could dispense with writing a parser for the custom 
timestamp format rpm changelogs use, and save the date in easy to parse 
RFC 3339/ ISO 8601 format)


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 11:17, Nicolas Mailhot via devel a écrit :

This may seem a bit complex and convoluted, but that’s because 
autobumping


https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping


is a small addition over the big %auto_macros change.

https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs



--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 09:59, Vitaly Zaitsev via devel a écrit :

On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:

The detached changelog is just one more file in SRPM sources, which is
modified by rpmbuild at `%build` time with other files rpmbuild
modifies.


I don't like that. %changelog should be generated automatically on Koji
side.


Why? Koji schedules a build. The build registers its own build date in 
the produced packages. Koji decides to keep and commit the result, or 
drop it (scratch build, failed side tag, whatever). Koji is still in 
charge, the bumping is just integrated in the build process with the 
rest of the package creation.


And, unlike something done specifically by koji, the bumping will import 
and export across all build systems, ie all the bumps that occurred in 
rpmbuild, or mock, or copr, or obs, or whatever are imported in fedpkg 
with the rest of the srpm, and an srpm produced by koji can import in 
rpmbuild, or mock, or copr, or obs, or whatever without loss of bump 
information.


And you no longer waste hours wondering why a package you just fixed is 
still failing on your test systems before realizing it is masked by 
another build that did not share the same bump history.


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Florian Weimer
* Nicolas Mailhot:

> Le 2020-07-02 09:52, Florian Weimer a écrit :
>> * Nicolas Mailhot via devel:
>>
 How do I let rpm generate the changelog automatically?
>>>
>>> This feature is not changelog generation, just changelog bumping on
>>> build events. You still need some other method to put non-build events
>>> in the changelog.
>>
>> What is “changelog bumping”?  Why is it needed?  What about release
>> bumping?
>
> Changelog bumping is the act of putting the actual release bump and
> build time in the changelog.
>
> With the change, the spec is able to self-compute its next release if
> the spec file evr is older or equal to the last build event.
>
> On build, it will both bump the release, without touching the spec
> file (that is release bumping) and commit the new build event
> timestamp in the detached changelog file at %build time (that is
> changelog bumping).

Isn't this totally over-engineered?

A much simpler alternative would just put the Koji task ID into the
%dist tag.  Since the %dist tag is not included in the version-release
fields on the %changelog section, no changelog update is needed.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Igor Raits
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Thu, 2020-07-02 at 11:17 +0200, Nicolas Mailhot wrote:
> Le 2020-07-02 09:52, Florian Weimer a écrit :
> > * Nicolas Mailhot via devel:
> > 
> > > > How do I let rpm generate the changelog automatically?
> > > 
> > > This feature is not changelog generation, just changelog bumping
> > > on
> > > build events. You still need some other method to put non-build
> > > events
> > > in the changelog.
> > 
> > What is “changelog bumping”?  Why is it needed?  What about release
> > bumping?
> 
> Changelog bumping is the act of putting the actual release bump and 
> build time in the changelog.
> 
> With the change, the spec is able to self-compute its next release if
> the spec file evr is older or equal to the last build event.

How does it know that "last build event"?

> On build, it will both bump the release, without touching the spec
> file 
> (that is release bumping) and commit the new build event timestamp in
> the detached changelog file at %build time (that is changelog
> bumping).
> 
> > > The detached changelog is just one more file in SRPM sources,
> > > which is
> > > modified by rpmbuild at `%build` time with other files rpmbuild
> > > modifies. The tricky part is to modify the source file as a
> > > source 
> > > file
> > > so rpmbuild adds the result to the produced SRPM (and, that does
> > > not
> > > work in mock right now, because mock serves the SRPM that existed
> > > at
> > > the start, not at the end of the build. Though it’s probably just
> > > a
> > > matter of getting mock to call again its SRPM creation method at
> > > the
> > > end of the build).
> > > 
> > > The packager does not have to request the modification in his
> > > spec,
> > > it’s done as part of the various %auto_foo calls the change
> > > introduced
> > 
> > Can you list the relevant %auto macros explicitly somewhere?  Is
> > %autosetup included in the set of macros that trigger this
> > behavior?
> 
> %autosetup is not part of the new framework, all the new %auto entry 
> points have %auto_something name/
> 
> Auto release bumping and auto changelog bumping involve registering
> some 
> processing in the preamble (to compute the next evr), in %sourcelist
> (to 
> deal with the source files involved in saving state) in %build (to 
> commit the new data to disk once the build is ongoing) and in
> %changelog 
> (to get rpmbuild to record the new changelog state in package
> metadata)
> 
> ie it registers processing in %auto_pkg, %auto_sources, %auto_build
> and 
> %auto_changelog
> 
> The bumping is done by the buildsys subsystem ie practically by
> %new_package (called by %auto_pkg, directly or via %buildsys_pkg), by
> %buildsys_sources (called by %auto_sources), %buildsys_build (called
> by 
> %auto_build) and %buildsys_changelog (called by %auto_changelog).
> 
> It’s done by the buildsys subsystem because the %buildsys subsystem
> is 
> tasked with writing the SRPM header in the new %auto_call framework,
> so 
> only it knows which of the various (sub)package epochs and versions
> are 
> the ones that apply to the SRPM.
> 
> This may seem a bit complex and convoluted, but that’s because 
> autobumping
> 
> https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs
> 
> is a small addition over the big %auto_macros change.
> 
> https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs
> 
> And it is small because the big change provides all the low-level
> infra 
> to code such high level features easily.
> 
> The big change was not done for autobumping. It’s only once I coded
> it 
> for other packaging needs that I realized it made implementing 
> autobumping trivial (trivial to me after all the other changes, maybe
> not so trivial for the average macro reviewer).
> 
> Regards,
> 
> -- 
> Nicolas Mailhot
> ___
> packaging mailing list -- packag...@lists.fedoraproject.org
> To unsubscribe send an email to 
> packaging-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: 
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org

- -- 
Igor Raits 
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl79pwsACgkQEV1auJxc
Hh5oixAAlyinJfmNKCVQcx/Kh11xb+MwW9UzGynhW1cTUAe1D8vslH0jtEJjJRRm
nXMtIyNoj0ny5Uo+ddABdQ3V86qqh/U46K5XK2FbGPq9a0hmI254KgJDLt4hqtaT
Dqw7+LK2jwbb63WBacsxJG6dGhvS9cOGxoxo+jMQ3uocLN1RrbTI/Du64i8d3Enk
Jmu3v0YKm3V+VyRtal2O+BGphzANS0D0rodHMH/8zmcT50Mt41QMFl+16PPDBcsn
qgyy/3tmruPmxUDCO5xFzJlA50qT5AMSWy8pOKqFdr+5hUaYW6rPkvXoC7uun08V
FnW/XGVHHv7iwz2CUCqoLwzb6wlyKzOyjxh3RGTIt+FXz1AfQ2tZWSbSvlElBce9
eRMb+v5yoURHMYK4Iazy9HMZa2mp7lcXFWE7qkTxwVwClkQ1YuaCEeSIVpblFuFw
l8w47QzcyGPwIi6GMqJyp5dpqLD15JetVIXQfF8/V5OCWMHNgqbbaef5Q9JNnc9K

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Nicolas Mailhot via devel

Le 2020-07-02 09:52, Florian Weimer a écrit :

* Nicolas Mailhot via devel:


How do I let rpm generate the changelog automatically?


This feature is not changelog generation, just changelog bumping on
build events. You still need some other method to put non-build events
in the changelog.


What is “changelog bumping”?  Why is it needed?  What about release
bumping?


Changelog bumping is the act of putting the actual release bump and 
build time in the changelog.


With the change, the spec is able to self-compute its next release if 
the spec file evr is older or equal to the last build event.


On build, it will both bump the release, without touching the spec file 
(that is release bumping) and commit the new build event timestamp in 
the detached changelog file at %build time (that is changelog bumping).



The detached changelog is just one more file in SRPM sources, which is
modified by rpmbuild at `%build` time with other files rpmbuild
modifies. The tricky part is to modify the source file as a source 
file

so rpmbuild adds the result to the produced SRPM (and, that does not
work in mock right now, because mock serves the SRPM that existed at
the start, not at the end of the build. Though it’s probably just a
matter of getting mock to call again its SRPM creation method at the
end of the build).

The packager does not have to request the modification in his spec,
it’s done as part of the various %auto_foo calls the change introduced


Can you list the relevant %auto macros explicitly somewhere?  Is
%autosetup included in the set of macros that trigger this behavior?


%autosetup is not part of the new framework, all the new %auto entry 
points have %auto_something name/


Auto release bumping and auto changelog bumping involve registering some 
processing in the preamble (to compute the next evr), in %sourcelist (to 
deal with the source files involved in saving state) in %build (to 
commit the new data to disk once the build is ongoing) and in %changelog 
(to get rpmbuild to record the new changelog state in package metadata)


ie it registers processing in %auto_pkg, %auto_sources, %auto_build and 
%auto_changelog


The bumping is done by the buildsys subsystem ie practically by
%new_package (called by %auto_pkg, directly or via %buildsys_pkg), by 
%buildsys_sources (called by %auto_sources), %buildsys_build (called by 
%auto_build) and %buildsys_changelog (called by %auto_changelog).


It’s done by the buildsys subsystem because the %buildsys subsystem is 
tasked with writing the SRPM header in the new %auto_call framework, so 
only it knows which of the various (sub)package epochs and versions are 
the ones that apply to the SRPM.


This may seem a bit complex and convoluted, but that’s because 
autobumping


https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

is a small addition over the big %auto_macros change.

https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

And it is small because the big change provides all the low-level infra 
to code such high level features easily.


The big change was not done for autobumping. It’s only once I coded it 
for other packaging needs that I realized it made implementing 
autobumping trivial (trivial to me after all the other changes, maybe 
not so trivial for the average macro reviewer).


Regards,

--
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Vitaly Zaitsev via devel
On 02.07.2020 07:35, Nicolas Mailhot via devel wrote:
> The detached changelog is just one more file in SRPM sources, which is
> modified by rpmbuild at `%build` time with other files rpmbuild
> modifies.

I don't like that. %changelog should be generated automatically on Koji
side.

-- 
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-02 Thread Florian Weimer
* Nicolas Mailhot via devel:

>> How do I let rpm generate the changelog automatically?
>
> This feature is not changelog generation, just changelog bumping on
> build events. You still need some other method to put non-build events
> in the changelog.

What is “changelog bumping”?  Why is it needed?  What about release
bumping?

> The detached changelog is just one more file in SRPM sources, which is
> modified by rpmbuild at `%build` time with other files rpmbuild
> modifies. The tricky part is to modify the source file as a source file
> so rpmbuild adds the result to the produced SRPM (and, that does not
> work in mock right now, because mock serves the SRPM that existed at
> the start, not at the end of the build. Though it’s probably just a
> matter of getting mock to call again its SRPM creation method at the
> end of the build).
>
> The packager does not have to request the modification in his spec,
> it’s done as part of the various %auto_foo calls the change introduced

Can you list the relevant %auto macros explicitly somewhere?  Is
%autosetup included in the set of macros that trigger this behavior?

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-01 Thread Nicolas Mailhot via devel
Le mercredi 01 juillet 2020 à 23:48 +0200, Dan Čermák a écrit :
> Hi Nicolas,

Hi Dan

> This is a system-wide change because all packages build with
> > redhat-rpm-config, but it only concerns packages that opted to use
> > this part of redhat-rpm-config (auto framework).
> > 
> > The change will make those packages auto-bump and auto-changelog at
> > the rpm level, in an infrastructure-independent way.
> 
> Please forgive the silly questions (it's getting late here…), but how
> does this look in practice?

There are no silly questions, do ask if you have more.

> How do I let rpm generate the changelog automatically?

This feature is not changelog generation, just changelog bumping on
build events. You still need some other method to put non-build events
in the changelog.

The detached changelog is just one more file in SRPM sources, which is
modified by rpmbuild at `%build` time with other files rpmbuild
modifies. The tricky part is to modify the source file as a source file
so rpmbuild adds the result to the produced SRPM (and, that does not
work in mock right now, because mock serves the SRPM that existed at
the start, not at the end of the build. Though it’s probably just a
matter of getting mock to call again its SRPM creation method at the
end of the build).

The packager does not have to request the modification in his spec,
it’s done as part of the various %auto_foo calls the change introduced

> Is the old changelog discarded?

The old changelog file is replaced in srpm sources with a new file
containing new build event lines before the old lines. It is not done
before the build is effective and %build has been started (ie a
rpmbuild -bs or rpmspec -P will show the future bump, but nothing will
be stored before the build is effective)

> And is this related to Piere/Pingou's work on the same topic that was
> deployed to koji staging?

It’s a different implementation, at the rpm level, that does not tie
bumping to Fedora infra (koji included). Though, it is probably
complementary to what pingou did on the changelog alimentation front.

IMHO the design mistake so far was to conflate bumping and non-build
event changelog filling. You need to do both of course but build event
should be a build event driven by the lowest common denominator
(rpmbuild) with koji/infra scrapping rpmbuild results as usual and
exposing them to users.

Regards,

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-01 Thread Dan Čermák
Hi Nicolas,

Ben Cotton  writes:

> https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
>
> == Summary ==
>
> redhat-rpm-config will be updated so users of the auto framework get
> automated release and changelog bumping.
>
> == Owner ==
>
> * Name: [[User:nim| Nicolas Mailhot]]
> * Email: 
>
> == Detailed Description ==
>
> This is a system-wide change because all packages build with
> redhat-rpm-config, but it only concerns packages that opted to use
> this part of redhat-rpm-config (auto framework).
>
> The change will make those packages auto-bump and auto-changelog at
> the rpm level, in an infrastructure-independent way.

Please forgive the silly questions (it's getting late here…), but how
does this look in practice?
How do I let rpm generate the changelog automatically?
Is the old changelog discarded?
And is this related to Piere/Pingou's work on the same topic that was
deployed to koji staging?


Cheers,

Dan


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-01 Thread Nicolas Mailhot via devel
Le mercredi 01 juillet 2020 à 12:27 +0200, Dominik 'Rathann'
Mierzejewski a écrit :

Hi,

> That's not detailed at all. You should provide at least one example
> here (or a direct link to one somewhere else on the wiki).

Thank you for your attention and kind review. yes the wiki page was
completely insufficient, I did it at the last minute to honor
deadlines. Please check if it is ok for you now.

Anyway here are some answers I added to the wiki


> 
> What's "autobumping" here 

The change will make packages that use the %auto_ redhat-rpm-
config macros auto-bump and auto-changelog at the rpm level, in an
infrastructure-independent way. The %auto_ framework is proposed
in
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

In that context, auto-bumping means that a SRPM, produced in any
compatible build system (that is, any build system that does not
inhibit low-level rpmbuild behaviour), will rebuild by default to a
release higher, than the last build release, in the next build system
it is imported into, without any manual change to the SRPM source
files.

Auto-changelog means that the build event will also be traced in the
rpm changelog (again, without any manual change). 

> and how is it better than rpmdev-bumpsec?

Unlike rpmdev-bumpsec, the feature is automatic.

It does not require explicit human action. Releases get bumped even if
the human forgot a particular release has already been built.

It does not rely on an external tool, nor requires this external tool
to be able to parse a spec file (which can be difficult for heavily
automated spec files like the ones that take advantage of %auto
macros).

A rebuild does not touch the spec file at all. That means, the spec
files changes tracked by your favorite scm, will show only spec logic
changes, without drowning those in no-logic-change build events. 


> [...]
> > == How To Test ==
> > 
> > A redhat-rpm-config packages with the changes and some example
> > packages are available in
> >   
> > https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> 
> A diff with changes

The current code state is visible in

https://src.fedoraproject.org/fork/nim/rpms/redhat-rpm-config/commits/forge-with-patches

It’s one small commit on top of the huge change queued in:
https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs
https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/95

That PR can still evolve based on feedback and testing. Therefore, I
can’t promise that the auto-bumping logic will always apply directly,
just that it will look more or less this way after rebasing. I do not
rebase it on every change to the other PR.

This is very young code, there are probably lots of easy things to tidy
up in there. However it works. 


> 
> Why is a separate "rpm-changelog.txt" file with manually maintained
> changelog better than current manually maintained changelog inside
> .spec?

This change does not separates the changelog. The separation is already
done in

https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs
that this change builds upon

Without separation, we would lose the benefit of auto-bumping at the
SCM level, since no-logic-change rebuilds would still result in a spec
file change.

Separation makes automation a lot easier since adding to the changelog
is just pre-pending some lines, and does not require any knowledge of
rpm syntax. Auto-bumping will add a "* date name evr" line on the next
rebuild, so changelog additions can limit themselves to plain text
descriptions of new changes at the top of the existing file.

Separation is a requirement for auto-changelog bumping at the rpm
level. Once rpmbuilt is lauched, it can not modify the processed spec
file. Therefore making the changelog modifiable by the build process
requires splitting it out of the spec file first. 

> How about using git commit log for changelog instead?

This is a low level change that does not depend on any specific
infrastructure, git included. It works directly at the rpm level. 

An infrastructure that uses git, can feed git commit events to the
detached changelog file, using dumb or elaborate git commit hooks, and
any other method it wants to implement. The auto-bump logic does not
care, it will use the detached changelog file in the state it exists at
the start of the build process.

Because the logic catches all rebuilds, regular manual trimming of the
lines that add no value is recommended. 

> [...]
> > To get beautiful changelogs, you also need to add
> > 
> > 
> > %buildsys_name  Your name
> > %buildsys_email Your email
> > 
> > 
> > in ~/.rpmmacros
> 
> What about having one macro called %buildsys_packager instead of two?
> You're always using them together, anyway. It'd be similar to the
> existing %packager macro, too.

This is certainly doable and 

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-07-01 Thread Dominik 'Rathann' Mierzejewski
On Tuesday, 30 June 2020 at 21:19, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> 
> == Summary ==
> 
> redhat-rpm-config will be updated so users of the auto framework get
> automated release and changelog bumping.
> 
> == Owner ==
> 
> * Name: [[User:nim| Nicolas Mailhot]]
> * Email: 
> 
> == Detailed Description ==
> 
> This is a system-wide change because all packages build with
> redhat-rpm-config, but it only concerns packages that opted to use
> this part of redhat-rpm-config (auto framework).
> 
> The change will make those packages auto-bump and auto-changelog at
> the rpm level, in an infrastructure-independent way.

That's not detailed at all. You should provide at least one example
here (or a direct link to one somewhere else on the wiki).

> == Benefit to Fedora ==
> 
> Autobumping removes a huge packager shore and makes timestamping in
  ^
Did you mean "chore"?

> changelogs more reliable.

What's "autobumping" here and how is it better than rpmdev-bumpsec?

[...]
> == How To Test ==
> 
> A redhat-rpm-config packages with the changes and some example
> packages are available in
>   
> https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/

A diff with changes and some explanation would be more accessible
than having to dig inside a copr repo.

Why is a separate "rpm-changelog.txt" file with manually maintained
changelog better than current manually maintained changelog inside .spec?

How about using git commit log for changelog instead?

[...]
> To get beautiful changelogs, you also need to add
> 
> 
> %buildsys_name  Your name
> %buildsys_email Your email
> 
> 
> in ~/.rpmmacros

What about having one macro called %buildsys_packager instead of two?
You're always using them together, anyway. It'd be similar to the
existing %packager macro, too.

Regards,
Dominik
-- 
Fedora   https://getfedora.org  |  RPM Fusion  http://rpmfusion.org
There should be a science of discontent. People need hard times and
oppression to develop psychic muscles.
-- from "Collected Sayings of Muad'Dib" by the Princess Irulan
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org


Re: [Fedora-packaging] Re: RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-06-30 Thread Nicolas Mailhot via devel
Le mardi 30 juin 2020 à 21:33 +0200, Igor Raits a écrit :
> On Tue, 2020-06-30 at 15:19 -0400, Ben Cotton wrote:
> > https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> > 
> > The change will make those packages auto-bump and auto-changelog at
> > the rpm level, in an infrastructure-independent way.
> 
> So how exactly is this supposed to work? From where will it get old
> changelog, how packagers will migrate to this, how does it affect
> reproducibility?

The changelog is just one file in SRPM sources, and bumping is done
from last build state which is just one key = value file in sources
(storing the evr components, the last built time, and last build
packager id).

In reproducible mode last evr and packager id and build time are
applied without bumping. You need to set
%reproducible_build = true (or anything except false) in your
~/.rpmmacros (or config_opts['macros'], or as rpmbuild --define flags).

The auto framework  (and %new_package) split Release in separate
%{source_release} %{dist} and %{post_release} components, which makes
the implementation way easier than multiplexing things in a single
Release tag and trying to untangle the mess later.

A production implementation would probably split %{dist} in %{distcore}
and %{distprefix} (the .gitdatehash things we stuff in Releases and in
rpm changelogs as opposed to the fcX part we want to appear in Release
but not in changelogs). I know where the offending code is in fedora-
release and the split up is trivial to implement, but there’s no point
in worrying about this level of detail before the core of the feature
is approved (or not).

The implementation is really simple and easy, it took me two days to
write and test it because it reuses all the building blocks I had
already done for my other change (without those jungling all the bits
involved at various points of the spec file would have been challenging
to say the least).

> 
> So are you asking mock and koji people to implement something? Did
> you
> talk to them before submitting this proposal?
> 
> > * Mock issue:   
> > https://github.com/rpm-software-management/mock/issues/599

I filled the mock issue to inform them. Again, this is a feature that
took me two days to code (it did not exist, even in thought, last
saturday). I was actually surprised at how easy it was to implement,
given the months if was discussed on this list.

At the mock level, there are two issues.

The main one (and only critical one) is that bumping MUST occur when
%build is executed, because a spectool or rpmbuild -bs is not a build
event, only a full build is. That means the SRPM produced by
rpmbuild -bs
is not the bumped SRPM, only the SRPM produced by
rpmbuild -ba

is bumped. My (imperfect) understanding of the mock issue is that mock
serves the first, not the second one at the end of the build process.

The second issue is that bumping changelog requires filling a builder
name and mail in the changelog line, and mock provides not easy way to
pass those to the build process.

If those two problems are lifted I see no special problem copr side
(except using the new mock plumbing to pass builder iname & mail to
mock).

For koji/fedpkg things are a bit more challenging because you will want
to back-commit the bumping to git once a build succeeds. Which is the
main point clime and me disagreed earlier on this year. Though, it is
not a show-stopper initially, a packager can back-commit manually if he
wants the bump recorded till tooling catches up.

While that adds constrains on the koji/git interface, that gives you a
bumping mecanism totally generic and independant of the build infra,
that does not rely on external python/ruby/whatever scripts to bum, and
does not require messing with someone else’s spec just to trace and
bump a rebuild.

Just importing an srpm from one system to another will preserve the
bumping state without any data loss.

> > 
> > == Contingency Plan ==
> > 
> > There is no contingency plan because the change will happen or not
> > at all.
> 
> This is not true. If it will happen but then something will be
> entirely broken we need to revert it.

Thank you for your vote of confidence. I hope you realise that by that
yardstick, nothing would be accepted, including your own changes,
because something may always happen someday causing someone to revisit
something. And, the last time a problem occurred, it was traced to an
undocumented and unannounced rpm change that no one knew how to fix
rpm-side, and that you spent more energy proving it need not be fixed
than on constructive solution-finding.

I freely admit that my code sucks and is way worse than the perfect
code no one has written yet nor intends to write any day soon.

-- 
Nicolas Mailhot
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 

Re: [Fedora-packaging] RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-06-30 Thread Igor Raits
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Tue, 2020-06-30 at 15:19 -0400, Ben Cotton wrote:
> https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping
> 
> == Summary ==
> 
> redhat-rpm-config will be updated so users of the auto framework get
> automated release and changelog bumping.
> 
> == Owner ==
> 
> * Name: [[User:nim| Nicolas Mailhot]]
> * Email: 
> 
> == Detailed Description ==
> 
> This is a system-wide change because all packages build with
> redhat-rpm-config, but it only concerns packages that opted to use
> this part of redhat-rpm-config (auto framework).
> 
> The change will make those packages auto-bump and auto-changelog at
> the rpm level, in an infrastructure-independent way.

So how exactly is this supposed to work? From where will it get old
changelog, how packagers will migrate to this, how does it affect
reproducibility?

> == Benefit to Fedora ==
> 
> Autobumping removes a huge packager shore and makes timestamping in
> changelogs more reliable.
> 
> == Scope ==
> * Proposal owners: The feature is coded and works at the rpm level.
> Unfortunately, mock filters away the srpms containing the bump state,
> so it does not work in upper layers.
> 
> * Other developers: The feature requires buy-in by mock developers
> (and probably koji developers) to lift the restrictions that block it
> above the rpm level. Also, it requires a mechanism to pass the user
> name and email that will be used in bumped changelogs (defining two
> variables in ~/.rpmmacros is sufficient at rpm level)

So are you asking mock and koji people to implement something? Did you
talk to them before submitting this proposal?

> * Mock issue:   
> https://github.com/rpm-software-management/mock/issues/599
> 
> * Release engineering: https://pagure.io/releng/issue/9567
> * Policies and guidelines: maybe eventually if things work out on the
> technical level
> * FPC issue: https://pagure.io/packaging-committee/issue/998
> * Trademark approval: N/A (not needed for this Change)
> 
> == Upgrade/compatibility impact ==
> 
> This is a pure build tooling update, it changes how things are built
> not what is built.
> 
> == How To Test ==
> 
> A redhat-rpm-config packages with the changes and some example
> packages are available in
>     
> https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/
> 
> Since the mock/copr layer is currently blocking the feature, you need
> to install the redhat-rpm-config and forge macro packages available
> in
> this repo locally. Afterwards you can take any of the example
> packages
> in the repo and rebuild them with rpmbuild -ba to your heart content,
> and see the releases bump and the changelogs being updated
> accordingly.
> 
> To get beautiful changelogs, you also need to add
> 
> 
> %buildsys_name  Your name
> %buildsys_email Your email
> 
> 
> in ~/.rpmmacros
> 
> == User Experience ==
> 
> N/A Packager experience change only
> 
> == Dependencies ==
> 
> The change is a spin-off of
> 
>   
> https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs
> 
> Therefore, it depends on the success of that other change and will
> probably need rebasing if the code in this other change evolves
> during
> the redhat-rpm-config merge.
> 
> It also depends on mock / copr/ koji buy-in and changes, that may add
> their own requirements.
> 
> == Contingency Plan ==
> 
> There is no contingency plan because the change will happen or not at
> all.

This is not true. If it will happen but then something will be entirely
broken we need to revert it.. And we need to know when, how and who
will do that.

> == Documentation ==
> 
> There is as much documentation as the average redhat-rpm-config
> change
> (ie comments in the macro files themselves)
> 
> == Release Notes ==
> 
> N/A Packager productivity change only
> 
> -- 
> Ben Cotton
> He / Him / His
> Senior Program Manager, Fedora & CentOS Stream
> Red Hat
> TZ=America/Indiana/Indianapolis
> ___
> packaging mailing list -- packag...@lists.fedoraproject.org
> To unsubscribe send an email to   
> packaging-le...@lists.fedoraproject.org
> Fedora Code of Conduct:   
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:   
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:   
> https://lists.fedoraproject.org/archives/list/packag...@lists.fedoraproject.org
- -- 
Igor Raits 
-BEGIN PGP SIGNATURE-

iQIzBAEBCgAdFiEEcwgJ58gsbV5f5dMcEV1auJxcHh4FAl77k5MACgkQEV1auJxc
Hh5U4xAAq12qjd3UImlWtshF6wil2f7DnC92TcoKifzWFNm2/FJGZw5RfvFJwy0n
6xFcyF+E8mtGLDG0l5xVi4lxfxkyErAiIzmBxZWc17h3G3gL6PeSBCgJRURJSh/e
1bXtOJb847bRfjbcMjoPKIc6Gcl+kA6C+kLr4QpTV7GDeHdEbeYX/xjGVTr6RbeN
ZxBBM7h0Cvwj8LIN6uXJsJKVz92lalElyY4GYHAUuNyCNztdxARB93sNT4MdbbJ4
gsVHAFiep6g3A7HflaBgTsRelvbpmmdIj1XC3SSwrw3eD4nZ5c+ET0F/M2nv2hVx

RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-06-30 Thread Ben Cotton
https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

== Summary ==

redhat-rpm-config will be updated so users of the auto framework get
automated release and changelog bumping.

== Owner ==

* Name: [[User:nim| Nicolas Mailhot]]
* Email: 

== Detailed Description ==

This is a system-wide change because all packages build with
redhat-rpm-config, but it only concerns packages that opted to use
this part of redhat-rpm-config (auto framework).

The change will make those packages auto-bump and auto-changelog at
the rpm level, in an infrastructure-independent way.

== Benefit to Fedora ==

Autobumping removes a huge packager shore and makes timestamping in
changelogs more reliable.

== Scope ==
* Proposal owners: The feature is coded and works at the rpm level.
Unfortunately, mock filters away the srpms containing the bump state,
so it does not work in upper layers.

* Other developers: The feature requires buy-in by mock developers
(and probably koji developers) to lift the restrictions that block it
above the rpm level. Also, it requires a mechanism to pass the user
name and email that will be used in bumped changelogs (defining two
variables in ~/.rpmmacros is sufficient at rpm level)

* Mock issue: https://github.com/rpm-software-management/mock/issues/599

* Release engineering: https://pagure.io/releng/issue/9567
* Policies and guidelines: maybe eventually if things work out on the
technical level
* FPC issue: https://pagure.io/packaging-committee/issue/998
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==

This is a pure build tooling update, it changes how things are built
not what is built.

== How To Test ==

A redhat-rpm-config packages with the changes and some example
packages are available in
  
https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/

Since the mock/copr layer is currently blocking the feature, you need
to install the redhat-rpm-config and forge macro packages available in
this repo locally. Afterwards you can take any of the example packages
in the repo and rebuild them with rpmbuild -ba to your heart content,
and see the releases bump and the changelogs being updated
accordingly.

To get beautiful changelogs, you also need to add


%buildsys_name  Your name
%buildsys_email Your email


in ~/.rpmmacros

== User Experience ==

N/A Packager experience change only

== Dependencies ==

The change is a spin-off of

https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

Therefore, it depends on the success of that other change and will
probably need rebasing if the code in this other change evolves during
the redhat-rpm-config merge.

It also depends on mock / copr/ koji buy-in and changes, that may add
their own requirements.

== Contingency Plan ==

There is no contingency plan because the change will happen or not at all.

== Documentation ==

There is as much documentation as the average redhat-rpm-config change
(ie comments in the macro files themselves)

== Release Notes ==

N/A Packager productivity change only

-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
___
devel-announce mailing list -- devel-announce@lists.fedoraproject.org
To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org


RPM-level auto release and changelog bumping - Fedora 33 System-Wide Change proposal

2020-06-30 Thread Ben Cotton
https://fedoraproject.org/wiki/Changes/rpm_level_auto_release_and_changelog_bumping

== Summary ==

redhat-rpm-config will be updated so users of the auto framework get
automated release and changelog bumping.

== Owner ==

* Name: [[User:nim| Nicolas Mailhot]]
* Email: 

== Detailed Description ==

This is a system-wide change because all packages build with
redhat-rpm-config, but it only concerns packages that opted to use
this part of redhat-rpm-config (auto framework).

The change will make those packages auto-bump and auto-changelog at
the rpm level, in an infrastructure-independent way.

== Benefit to Fedora ==

Autobumping removes a huge packager shore and makes timestamping in
changelogs more reliable.

== Scope ==
* Proposal owners: The feature is coded and works at the rpm level.
Unfortunately, mock filters away the srpms containing the bump state,
so it does not work in upper layers.

* Other developers: The feature requires buy-in by mock developers
(and probably koji developers) to lift the restrictions that block it
above the rpm level. Also, it requires a mechanism to pass the user
name and email that will be used in bumped changelogs (defining two
variables in ~/.rpmmacros is sufficient at rpm level)

* Mock issue: https://github.com/rpm-software-management/mock/issues/599

* Release engineering: https://pagure.io/releng/issue/9567
* Policies and guidelines: maybe eventually if things work out on the
technical level
* FPC issue: https://pagure.io/packaging-committee/issue/998
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==

This is a pure build tooling update, it changes how things are built
not what is built.

== How To Test ==

A redhat-rpm-config packages with the changes and some example
packages are available in
  
https://copr.fedorainfracloud.org/coprs/nim/refactoring-forge-patches-auto-call-bump-changelog-fonts/builds/

Since the mock/copr layer is currently blocking the feature, you need
to install the redhat-rpm-config and forge macro packages available in
this repo locally. Afterwards you can take any of the example packages
in the repo and rebuild them with rpmbuild -ba to your heart content,
and see the releases bump and the changelogs being updated
accordingly.

To get beautiful changelogs, you also need to add


%buildsys_name  Your name
%buildsys_email Your email


in ~/.rpmmacros

== User Experience ==

N/A Packager experience change only

== Dependencies ==

The change is a spin-off of

https://fedoraproject.org/wiki/Changes/Patches_in_Forge_macros_-_Auto_macros_-_Detached_rpm_changelogs

Therefore, it depends on the success of that other change and will
probably need rebasing if the code in this other change evolves during
the redhat-rpm-config merge.

It also depends on mock / copr/ koji buy-in and changes, that may add
their own requirements.

== Contingency Plan ==

There is no contingency plan because the change will happen or not at all.

== Documentation ==

There is as much documentation as the average redhat-rpm-config change
(ie comments in the macro files themselves)

== Release Notes ==

N/A Packager productivity change only

-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org