Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Kent Fredric]

On Sat, 3 Dec 2016 15:06:36 +
Markos Chandras  wrote:

> That's reasonable but I also think that bumping and fixing an ebuild at
> the same time can be considered an atomic change since it's effectively
> a _new_ ebuild

One problem is that can seriously confuse git about what's happening,
given how similar our ebuilds are, and given gits only mechanism
for determining "rename" is 

+ filename/a 0xSHA1
- filename/b 0xSHA1

If "filename/a" is sufficiently different from "filename/b" and close enough
to some other non-target "filename/c", it could retroactively be confused into
thinking this happened.

copy filename/c -> filename/a (98%)
remove filename/b 

And then you go and rebase that somewhere were filename/c was actually modified,
and filename/b was modified, and you end up with merge conflicts
where git propagates changes from filename/c to filename/a 
and then makes you battle with "modified filename/b vs removed filename/b"

This is why I prefer to take the approach of:

1. Fix everything first.
2. -r1 bump in a dedicated (or commits) at the end.

Then git tends to be smarter about things, and realises the change in the ebuild
was applied to a moved file, and attempts to modify the moved file instead,
and the only headaches are the -r bumps themselves, and git itself is *usually*
more disposed to getting these right.

But uh. 

Don't look at what github likes to do: 

https://github.com/gentoo/gentoo/pull/2956/commits/52da7b3a525ff1c4f4e1ad73fe9e0ab91cf6e799

..rl-JSON-PP/perl-JSON-PP-2.273.0-r3.ebuild → 
...ive-Tar/perl-Archive-Tar-2.40.0-r4.ebuild 
..ocale-Maketext-Simple-0.210.100_rc.ebuild → 
...ar/perl-Archive-Tar-2.40.100_rc-r1.ebuild 
..ive-Tar/perl-Archive-Tar-2.40.0-r3.ebuild → 
...rl-JSON-PP/perl-JSON-PP-2.273.0-r4.ebuild 
..e-Tar/perl-Archive-Tar-2.40.100_rc.ebuild → 
...le-Maketext-Simple-0.210.100_rc-r1.ebuild 

Its totally understandable why it gets confused:

md5sum perl-Archive-Tar/perl-Archive-Tar-2.40.0-r4.ebuild \
   perl-Archive-Tar/perl-Archive-Tar-2.40.100_rc-r1.ebuild \
   perl-JSON-PP/perl-JSON-PP-2.273.0-r4.ebuild \
   
perl-Locale-Maketext-Simple/perl-Locale-Maketext-Simple-0.210.100_rc-r1.ebuild 

1fe782df38a2c2cc01615012b9915da9  
perl-Archive-Tar/perl-Archive-Tar-2.40.0-r4.ebuild
5102f6ac13ac331465c64258d8f413c5  
perl-Archive-Tar/perl-Archive-Tar-2.40.100_rc-r1.ebuild
1fe782df38a2c2cc01615012b9915da9  perl-JSON-PP/perl-JSON-PP-2.273.0-r4.ebuild
5102f6ac13ac331465c64258d8f413c5  
perl-Locale-Maketext-Simple/perl-Locale-Maketext-Simple-0.210.100_rc-r1.ebuild


Rationale: most users are working from rsync. So for them, the -r step is 
mostly a "propagate this change" mechansim.
These users are unlikely to see the intermediate stage where you modified the 
file before the -r bump, because the -r bump
was pushed atomically with the rest.

People who clone from git are also not typically "winding back" the head, and 
so they'll get their -r bumps atomically
with the changes when they pull.

People who *are* winding back the head are likely winding back to a specific 
moment in the history:

1. If they're winding back to before your commit series, then it doesn't matter
2. If they're winding back to after your commit series, then they have the 
needed -r bump 
3. If they're winding back to some point within the commit series, things may 
be a little hairy. ( and maybe this is where
the "-r bump first" strategy is more useful? )

But then it depends what they're rewinding *For*

(prebumping) If they were trying to avoid the change you made after you -r 
bumped, rewinding to before the change won't trip portage updates.
(postbumping) If they were meaning to include the change you made before you -r 
bumped, rewinding to after the change won't trip portage updates.

It seems in either case, post/pre bumping to within the change set will have an 
edge case somewhere
where a user has to be assumed to explicitly tell portage to reinstall the 
target package instead of relying
on -r bumps to propagate it.



pgpDOM2vPVkCC.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Markos Chandras]

On 12/02/2016 03:14 PM, Andrew Savchenko wrote:
> 
> What about the following forkflow:
> - version bump first with minimal changes required, but without
> pushing commit to the tree;
> - make each logical change as a separate commit without revision
> bumps and without pushing stuff to the tree (of course repoman
> scan/full is required as usual for each commit);
> - well test package after the last commit (that it builds with
> various USE flag combinations, old and new functionality works fine
> and so on);
> - fix any problems found and only afterwards push changes to the
> tree.
> 
> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> while git will still retain each logical change as a separate
> commit, which will make future maintenance and debugging much
> easier.

That's reasonable but I also think that bumping and fixing an ebuild at
the same time can be considered an atomic change since it's effectively
a _new_ ebuild

-- 
Regards,
Markos Chandras

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Daniel Campbell]

On 12/02/2016 07:14 AM, Andrew Savchenko wrote:
> Hi all!
> 
> Right now we have two somewhat conflicting policies (at least up to
> my understanding of them):
> 
> 1) git atomic commits [1]:
> each logical change should be a separate commit.
> 
> 2) revision bump policy [2]:
> each change sufficiently affecting application run-time or
> installed files should have a revision bump.
> 
> Let's consider the following quite common scenario: package foo-1.0
> should be updated to foo-1.1, but aside from version bump there is
> a set of accumulated issues which maintainer is willing to handle,
> e.g.:
> - bump to EAPI 6;
> - fix several runtime bugs (still present in the new version);
> - install missing documentation;
> - add previously omitted USE flags for some tools of controllable
> functionalities;
> - etc...
> 
> If both policies are to be followed, users will see something like:
> foo-1.0 -> foo-1.1-r8 (assuming each sufficient change was made as
> a separate commit with a revision bump).
> 
> While such versioning change is technically correct, it is
> confusing for our users and makes future maintainance harder,
> because of multiple file renames (yeah, I know about git diff
> --find-renames, but this kludge is not perfect).
> 
> What about the following forkflow:
> - version bump first with minimal changes required, but without
> pushing commit to the tree;
> - make each logical change as a separate commit without revision
> bumps and without pushing stuff to the tree (of course repoman
> scan/full is required as usual for each commit);
> - well test package after the last commit (that it builds with
> various USE flag combinations, old and new functionality works fine
> and so on);
> - fix any problems found and only afterwards push changes to the
> tree.
> 
> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> while git will still retain each logical change as a separate
> commit, which will make future maintenance and debugging much
> easier.
> 
> Of course a separate git branch may be used as well, but using
> branches for each half-a-dozen set of commits looks like an
> overkill to me.
> 
> Thoughts, comments?
> 
> [1] https://devmanual.gentoo.org/ebuild-maintenance/index.html
> [2] https://devmanual.gentoo.org/general-concepts/ebuild-revisions/index.html
> 
> Best regards,
> Andrew Savchenko
> 
I've always worked as if each commit needed to be working for Gentoo
users. So if I need to version bump, that's a separate commit. However,
let's say I found a bug in the ebuild itself for foo-1.0. The way I see
it is I should bump to foo-1.0-r1 to fix the bug in that ebuild, _then_
version bump so that foo-1.1 already has the fixes that foo-1.0-r1 has.
If I version bump first, then I have to revbump both and that just
increases my odds of forgetting to put the fixes into all the correct
ebuilds.

It results in the appropriate fixes in the older package, and the new
version comes with the old one's fixes (plus any changes the new ebuild
might need due to upstream changes).

Does that make any sense?

-- 
Daniel Campbell - Gentoo Developer
OpenPGP Key: 0x1EA055D6 @ hkp://keys.gnupg.net
fpr: AE03 9064 AE00 053C 270C  1DE4 6F7A 9091 1EA0 55D6



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Mart Raudsepp]

Ühel kenal päeval, R, 02.12.2016 kell 11:26, kirjutas Matt Turner:
> On Fri, Dec 2, 2016 at 7:14 AM, Andrew Savchenko 
> wrote:
> > 
> > Hi all!
> > 
> > Right now we have two somewhat conflicting policies (at least up to
> > my understanding of them):
> > 
> > 1) git atomic commits [1]:
> > each logical change should be a separate commit.
> > 
> > 2) revision bump policy [2]:
> > each change sufficiently affecting application run-time or
> > installed files should have a revision bump.
> > 
> > Let's consider the following quite common scenario: package foo-1.0
> > should be updated to foo-1.1, but aside from version bump there is
> > a set of accumulated issues which maintainer is willing to handle,
> > e.g.:
> > - bump to EAPI 6;
> > - fix several runtime bugs (still present in the new version);
> > - install missing documentation;
> > - add previously omitted USE flags for some tools of controllable
> > functionalities;
> > - etc...
> > 
> > If both policies are to be followed, users will see something like:
> > foo-1.0 -> foo-1.1-r8 (assuming each sufficient change was made as
> > a separate commit with a revision bump).
> > 
> > While such versioning change is technically correct, it is
> > confusing for our users and makes future maintainance harder,
> > because of multiple file renames (yeah, I know about git diff
> > --find-renames, but this kludge is not perfect).
> > 
> > What about the following forkflow:
> > - version bump first with minimal changes required, but without
> > pushing commit to the tree;
> > - make each logical change as a separate commit without revision
> > bumps and without pushing stuff to the tree (of course repoman
> > scan/full is required as usual for each commit);
> > - well test package after the last commit (that it builds with
> > various USE flag combinations, old and new functionality works fine
> > and so on);
> > - fix any problems found and only afterwards push changes to the
> > tree.
> > 
> > This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> > while git will still retain each logical change as a separate
> > commit, which will make future maintenance and debugging much
> > easier.
> > 
> > Of course a separate git branch may be used as well, but using
> > branches for each half-a-dozen set of commits looks like an
> > overkill to me.
> > 
> > Thoughts, comments?
> 
> Thanks for starting the discussion. I completely agree.
> 
> Though my case might have been a bit more clear-cut since I was
> working on an ebuild that initially didn't have any KEYWORDS, I think
> what I did for freeradius is the best way of handling the situation
> you describe.
> 
> See 97704b400b7^..e84dc52a816
> 
> An initial commit that copied the 3.0.12 ebuild to 3.0.12-r1 without
> making any other changes, followed by three self-contained
> fixes/commits, and finally a patch to add KEYWORDS to 3.0.12-r1.
> 

That makes me think that it might be an idea to have no KEYWORDS on the
intermediate commits. So removing them at first for the -r1 copy in tis
example, even if -r0 had them, and then after all is done add back
~arch ones.
Just an idea, not even sure I think it's a good idea myself.

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Matt Turner]

On Fri, Dec 2, 2016 at 7:14 AM, Andrew Savchenko  wrote:
> Hi all!
>
> Right now we have two somewhat conflicting policies (at least up to
> my understanding of them):
>
> 1) git atomic commits [1]:
> each logical change should be a separate commit.
>
> 2) revision bump policy [2]:
> each change sufficiently affecting application run-time or
> installed files should have a revision bump.
>
> Let's consider the following quite common scenario: package foo-1.0
> should be updated to foo-1.1, but aside from version bump there is
> a set of accumulated issues which maintainer is willing to handle,
> e.g.:
> - bump to EAPI 6;
> - fix several runtime bugs (still present in the new version);
> - install missing documentation;
> - add previously omitted USE flags for some tools of controllable
> functionalities;
> - etc...
>
> If both policies are to be followed, users will see something like:
> foo-1.0 -> foo-1.1-r8 (assuming each sufficient change was made as
> a separate commit with a revision bump).
>
> While such versioning change is technically correct, it is
> confusing for our users and makes future maintainance harder,
> because of multiple file renames (yeah, I know about git diff
> --find-renames, but this kludge is not perfect).
>
> What about the following forkflow:
> - version bump first with minimal changes required, but without
> pushing commit to the tree;
> - make each logical change as a separate commit without revision
> bumps and without pushing stuff to the tree (of course repoman
> scan/full is required as usual for each commit);
> - well test package after the last commit (that it builds with
> various USE flag combinations, old and new functionality works fine
> and so on);
> - fix any problems found and only afterwards push changes to the
> tree.
>
> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> while git will still retain each logical change as a separate
> commit, which will make future maintenance and debugging much
> easier.
>
> Of course a separate git branch may be used as well, but using
> branches for each half-a-dozen set of commits looks like an
> overkill to me.
>
> Thoughts, comments?

Thanks for starting the discussion. I completely agree.

Though my case might have been a bit more clear-cut since I was
working on an ebuild that initially didn't have any KEYWORDS, I think
what I did for freeradius is the best way of handling the situation
you describe.

See 97704b400b7^..e84dc52a816

An initial commit that copied the 3.0.12 ebuild to 3.0.12-r1 without
making any other changes, followed by three self-contained
fixes/commits, and finally a patch to add KEYWORDS to 3.0.12-r1.

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Michał Górny]

On Fri, 2 Dec 2016 18:14:05 +0300
Andrew Savchenko  wrote:

> Hi all!
> 
> Right now we have two somewhat conflicting policies (at least up to
> my understanding of them):
> 
> 1) git atomic commits [1]:
> each logical change should be a separate commit.
> 
> 2) revision bump policy [2]:
> each change sufficiently affecting application run-time or
> installed files should have a revision bump.
> 
> Let's consider the following quite common scenario: package foo-1.0
> should be updated to foo-1.1, but aside from version bump there is
> a set of accumulated issues which maintainer is willing to handle,
> e.g.:
> - bump to EAPI 6;
> - fix several runtime bugs (still present in the new version);
> - install missing documentation;
> - add previously omitted USE flags for some tools of controllable
> functionalities;
> - etc...
> 
> If both policies are to be followed, users will see something like:
> foo-1.0 -> foo-1.1-r8 (assuming each sufficient change was made as
> a separate commit with a revision bump).
> 
> While such versioning change is technically correct, it is
> confusing for our users and makes future maintainance harder,
> because of multiple file renames (yeah, I know about git diff
> --find-renames, but this kludge is not perfect).

To be honest, I don't really see how multiple successive renames are
really any worse than a single rename in git. Not that I oppose your
idea, just nitpicking.

> What about the following forkflow:
> - version bump first with minimal changes required, but without
> pushing commit to the tree;
> - make each logical change as a separate commit without revision
> bumps and without pushing stuff to the tree (of course repoman
> scan/full is required as usual for each commit);
> - well test package after the last commit (that it builds with
> various USE flag combinations, old and new functionality works fine
> and so on);
> - fix any problems found and only afterwards push changes to the
> tree.
> 
> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> while git will still retain each logical change as a separate
> commit, which will make future maintenance and debugging much
> easier.

Well, this is pretty much what we're doing and what fits common sense,
and I'm not sure if we really need to word this into a formal policy.

The definition of 'one logical change' is pretty inconclusive. One
could claim the whole bump to be one logical change, and I don't think
that we should really force people to split every change into a single
commit because it is too cumbersome.

What really makes sense is using the common sense to determine which
changes are best made together, and which really deserve a separate
commit. And of course, once a revbump or version bump is necessary, you
have to do it on the first commit involved (to avoid altering the old
file, in case you are keeping it) and you don't have to bump further
until the thing hits users.

What we need to keep in mind is the spirit of doing split commits --
it's meant to:

a) ease reviewing, through providing better context for each change,

b) ease reverting, especially in case one of the changes is more likely
to cause mayhem than others.

-- 
Best regards,
Michał Górny



pgpUt4Jm3Vurw.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Michael Orlitzky]

On 12/02/2016 10:14 AM, Andrew Savchenko wrote:
> 
> If both policies are to be followed, users will see something like:
> foo-1.0 -> foo-1.1-r8 (assuming each sufficient change was made as
> a separate commit with a revision bump).
> 
> While such versioning change is technically correct, it is
> confusing for our users and makes future maintainance harder,
> because of multiple file renames (yeah, I know about git diff
> --find-renames, but this kludge is not perfect).

Do you have a situation in mind where going from -r0 to -r8 in one `git
push` is a problem? What maintenance becomes harder? I can see how it
could be confusing, but only in the sense that we've been doing it wrong
for so long that now it's surprising to see it done right. Now it makes
sense: the -r8 version contains 8 more fixes than -r0.

I should be able to `git checkout` any commit in the repo and have the
tree not be broken. Your proposal eliminates that guarantee, and the bar
for doing so should be high, I think.

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Ian Stakenvicius]

On 02/12/16 10:32 AM, Dirkjan Ochtman wrote:
> On Fri, Dec 2, 2016 at 4:14 PM, Andrew Savchenko  wrote:
>> What about the following forkflow:
>> - version bump first with minimal changes required, but without
>> pushing commit to the tree;
>> - make each logical change as a separate commit without revision
>> bumps and without pushing stuff to the tree (of course repoman
>> scan/full is required as usual for each commit);
>> - well test package after the last commit (that it builds with
>> various USE flag combinations, old and new functionality works fine
>> and so on);
>> - fix any problems found and only afterwards push changes to the
>> tree.
>>
>> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
>> while git will still retain each logical change as a separate
>> commit, which will make future maintenance and debugging much
>> easier.
>>
>> Of course a separate git branch may be used as well, but using
>> branches for each half-a-dozen set of commits looks like an
>> overkill to me.
>>
>> Thoughts, comments?
> 
> Sounds sensible to me, possibly to the point of not having to spell it
> out? (As in, I don't see the mentioned policies as necessarily
> conflicting.)
> 
> Cheers,
> 
> Dirkjan
> 

If it matters, it may make sense to adhere to the order of operations
presented above more precisely.  For instance, I tend to do all of the
minor changes first and then revbump the ebuild (usually because I'm
adjusting these changes over a few days and then evaluate if it needs
a revbump once it's done).




signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Revision bumps vs git commits atomicity

[Dirkjan Ochtman]

On Fri, Dec 2, 2016 at 4:14 PM, Andrew Savchenko  wrote:
> What about the following forkflow:
> - version bump first with minimal changes required, but without
> pushing commit to the tree;
> - make each logical change as a separate commit without revision
> bumps and without pushing stuff to the tree (of course repoman
> scan/full is required as usual for each commit);
> - well test package after the last commit (that it builds with
> various USE flag combinations, old and new functionality works fine
> and so on);
> - fix any problems found and only afterwards push changes to the
> tree.
>
> This way users will see only foo-1.0 -> foo-1.1 change in the tree,
> while git will still retain each logical change as a separate
> commit, which will make future maintenance and debugging much
> easier.
>
> Of course a separate git branch may be used as well, but using
> branches for each half-a-dozen set of commits looks like an
> overkill to me.
>
> Thoughts, comments?

Sounds sensible to me, possibly to the point of not having to spell it
out? (As in, I don't see the mentioned policies as necessarily
conflicting.)

Cheers,

Dirkjan