Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Daniel Campbell]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 09/16/2014 05:18 AM, hasufell wrote:
> Ulrich Mueller:
>> 
>> ChangeLogs are aimed at users
> 
> Did any1 ask them if they care?
> 

If the tree switches to git and there's an option within
Portage/emerge to fetch via git instead of rsync, then I'd rather rely
on `git log` than a bunch of scattered files.
-BEGIN PGP SIGNATURE-
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJUGRVnAAoJEJUrb08JgYgH1pQH/RiBmtnUvewDY+dm1cdbAdWb
A8YXcHDHhYnVtll3x7hB+YphKLNYBN+baLLiKXHAR4LaWIfc+Z0NHMpfN3pNQTwZ
o3XjzShWMhZ9Z5mTafPuFgR1f+sAuqSG0lOhMm3tHwKmBEHt3fh2bnAZVkGtnJRE
L/xDCU5sniGPJCLhXBaPfU3om99xeEQtahXWR+rVHj64h93t9Cb1hHIlWRvjPzDT
M5kC9Rz/BS1wO4mwPqi/jW5mbQnLUhcy7y4OSszQeAMyroCIhkxwwKLeWES62XQr
bo6AKqv1SKMFVYIgYVRei0iTXbQ2/pWzlpatM11G6djqMtTvDlMR7f3wPbAiw2U=
=EIKj
-END PGP SIGNATURE-

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Daniel Campbell]

On 09/16/2014 01:56 PM, hasufell wrote:
> Luca Barbato:
>> On 15/09/14 01:21, Patrick Lauer wrote:
>>> On Sunday 14 September 2014 15:42:15 hasufell wrote:
 Patrick Lauer:
>> Are we going to disallow merge commits and ask devs to rebase local
>> changes in order to keep the history "clean"?
>
> Is that going to be sane with our commit frequency?

 You have to merge or rebase anyway in case of a push conflict, so the
 only difference is the method and the effect on the history.

 Currently... CVS allows you to run repoman on an outdated tree and push
 broken ebuilds with repoman being happy. Git will not allow this.
>>>
>>> iow, git doesn't allow people to work on more than one item at a time?
>>
>> It does.
>>
> 
> I think we really have to write up a step-by-step guide (not just
> workflow policies) for people who have never seriously worked with git.
> 
> On the other hand... there are thousands of tutorials on the net already.
> 
> For the workflow model, I already have created a draft which is in no
> way finished or even correct and there are still some controversially
> discussed issues.
> https://wiki.gentoo.org/wiki/Gentoo_git_workflow
> 
As a prospective Gentoo developer, having a guide around meant
specifically for Gentoo's practices would be incredibly helpful. I use
git in my own hobby development and learned from Pro Git, et al, but
it'd still be really nice to have, and give developers a place to
point to if a new developer is having troubles.

Just my 2¢.



signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Add bc back to the stage3

[Luca Barbato]

The bc utility is part of the posix tools and it might be used to build
linux among the other stuff.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Michał Górny]

Dnia 2014-09-16, o godz. 19:05:18
Luca Barbato  napisał(a):

> On 14/09/14 16:46, Michał Górny wrote:
> > Of course, if we can't spare the resources to do intermediate updates,
> > we may as well switch to cron-based update method.
> 
> The mirror have a sync time, so basically regenerating the cache and
> pushing the tree with further toward the user can happen the same way
> w/out impacting anybody.
> 
> We could just snapshot the tree when the regen starts and push that
> commit to the user git and rsync.
> 
> People is still supposed to play nice and sync not every minute.

People don't have to sync. They will pull, and pulling often doesn't
really hurt servers like rsync does.

Of course, I'm considering the users switching to git there. However,
I don't think limitations of rsync should impact them.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[viv...@gmail.com]

Il 16/09/2014 20:02, Duncan ha scritto:
> Rich Freeman posted on Tue, 16 Sep 2014 09:55:31 -0400 as excerpted:
>
>> Or they could just clone the git tree, and they can look at per-file
>> logs anytime they want to.
> Give me ro access to a current git repo and I'll *VERY* happily leave 
> changelogs to history along with 8-track tapes and 5.25-inch floppies! 
> =:^)
>
> I was strongly in favor of keeping changelogs (and mandating proper add/
> change/deletion entries) the last time the topic came up, but that was in 
> the context of (web)?rsync being the only viable user sync method and 
> thus changelogs being the only user-local-accessible record.  With user-
> git-repo access, I'll /very/ (very, very, very...) happily leave rsync 
> behind for git, and changelogs along with it! =:^)
>
yes, this probably is the same for everyone, and if it's not it should
be anyway.

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Luca Barbato:
> On 15/09/14 01:21, Patrick Lauer wrote:
>> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>>> Patrick Lauer:
> Are we going to disallow merge commits and ask devs to rebase local
> changes in order to keep the history "clean"?

 Is that going to be sane with our commit frequency?
>>>
>>> You have to merge or rebase anyway in case of a push conflict, so the
>>> only difference is the method and the effect on the history.
>>>
>>> Currently... CVS allows you to run repoman on an outdated tree and push
>>> broken ebuilds with repoman being happy. Git will not allow this.
>>
>> iow, git doesn't allow people to work on more than one item at a time?
> 
> It does.
> 

I think we really have to write up a step-by-step guide (not just
workflow policies) for people who have never seriously worked with git.

On the other hand... there are thousands of tutorials on the net already.

For the workflow model, I already have created a draft which is in no
way finished or even correct and there are still some controversially
discussed issues.
https://wiki.gentoo.org/wiki/Gentoo_git_workflow

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Brian Dolbec]

On Tue, 16 Sep 2014 10:52:13 -0700
"W. Trevor King"  wrote:

> On Tue, Sep 16, 2014 at 05:35:08PM +, Duncan wrote:
> > W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as
> > excerpted:
> > > On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
> > >> I don't see any benefit to using rsync vs. a shallow clone as the
> > >> transmission protocol.
> > > 
> > > Other than the fact that before you dropped it you'd need to push
> > > a ‘emerge sync’ that could handle either rsync or Git, stabilize
> > > that Portage, and then wait for folks to adopt it.
> > 
> > Portage already handles it. =:^)
> 
> Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
> support for multiple repositories in `emerge --sync`, 2013-07-23).
> There are older Portages in the tree though (back to 2.1.6.7_p1), so
> you'd still want to wait until those were gone before dropping rsync.
> 
> Also, I don't see a way to say “use Git to sync, but keep a shallow
> repository”.  Ideally, we'd want:
> 
>   $ git clone --depth=1 git://git.gentoo.org/gentoo-portage.git
> 
> for the initial clone (modulo whatever URI), and:
> 
>   $ git pull --depth=1
> 
> for subsequent syncs.  pym/_emerge/actions.py currently hardcodes ‘git
> pull’ for the latter, and doesn't seem to have any code for the
> former.  On the other hand, it wouldn't be too terrible to force users
> to shallow their history manually whenever they felt like it.
> 
> Cheers,
> Trevor
> 

The depth option will be added to the new portage plugin-sync system in
final stages of development now.  

There will be 2 new repos.conf options 

1 for new repo install
eg. git clone: --depth=1

another for sync options:
eg. git pull: --rebase origin master
this will allow changes to a repo in a different branch be
updated from the master branch.

they will be repo specific options, not global.


The new system will allow
emerge-webrync type repos to be synced via emerge --sync instead of the
emerge-webrsync command.  Plus it will add an svn type and the ability
for a layman type which layman already has code for.  Layman is just
waiting for the new sync system to land in portage's master branch
before enabling it to be installed.  It will also allow for third party
sync modules to be created and easily installed.  So a squashfs sync
type could be created and installed for those repos where that a
squashfs tree is offered.

-- 
Brian Dolbec 



signature.asc
Description: PGP signature

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

W. Trevor King posted on Tue, 16 Sep 2014 10:52:13 -0700 as excerpted:

> Also, I don't see a way to say “use Git to sync, but keep a shallow
> repository”.

Presumably at some point we'd get the PORTAGE_GIT* equivalent of the 
PORTAGE_RSYNC* settings from make.conf, but settable in both make.conf 
for global git-type settings and repos.conf for individual repo settings.

The big step, basic git support, is already there, but it doesn't have 
the broad usage of rsync and thus people haven't yet run into all the 
corner-cases that triggered the development of all those rich rsync 
option settings we have access to these days.

Tho I don't think it should be a blocker to the git migration, as I guess 
it'll be pretty mandatory that we keep an rsync backport for at least the 
gentoo-standard year of support.  Immediately dropping stuff like 
changelogs from that rsync backport should make that simpler.  If people 
want them, they can arrange to use the user-git repo.

> On the other hand, it wouldn't be too terrible to force users to shallow
> their history manually whenever they felt like it.

True.

Which brings up another point.  A user-focused git migration guide 
document would be very nice and very gentoo. =:^)  Once we have the git 
thing up and running for the early-adopters who can more or less find 
their own way, I'd call that a blocker to the possibility of eventually 
turning off rsync entirely, for our stable users.  Again, *NOT* a blocker 
to turning on git for those who can find their own way, but to eventually 
turning off the rsync backward compatibility machinery for stable.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

On Tue, Sep 16, 2014 at 10:52:13AM -0700, W. Trevor King wrote:
> Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
> support for multiple repositories in `emerge --sync`, 2013-07-23).

Actually, ‘git pull’ support in one form or another dates back to
ba797c11 (Add --sync support for `git pull`, 2008-12-11), which landed
in v2.2_rc18.

> There are older Portages in the tree though (back to 2.1.6.7_p1), so
> you'd still want to wait until those were gone before dropping rsync.

The ‘git pull’ support was also backported to the 2.1.6.7_p1 series
with d3c42937 (Add --sync support for `git pull`, 2008-12-11), which
landed in v2.1.6.1, so I doubt any Portage users lack pull support.
I'm not sure about folks using other package managers though.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: RFC: kde5 and kde5-functions eclass

[Michael Palimaka]

On 09/16/2014 02:19 AM, Davide Pesavento wrote:
>>  if [[ -a "CMakeLists.txt" ]]; then
> 
> Unnecessary quoting. Also, -e is more common than -a
I guess both the eclasses (and a lot of Gentoo stuff in general) has
quoting that's not strictly necessary.

Thanks for the review, everything else has been actioned.

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

Rich Freeman posted on Tue, 16 Sep 2014 09:55:31 -0400 as excerpted:

> Or they could just clone the git tree, and they can look at per-file
> logs anytime they want to.

Give me ro access to a current git repo and I'll *VERY* happily leave 
changelogs to history along with 8-track tapes and 5.25-inch floppies! 
=:^)

I was strongly in favor of keeping changelogs (and mandating proper add/
change/deletion entries) the last time the topic came up, but that was in 
the context of (web)?rsync being the only viable user sync method and 
thus changelogs being the only user-local-accessible record.  With user-
git-repo access, I'll /very/ (very, very, very...) happily leave rsync 
behind for git, and changelogs along with it! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[W. Trevor King]

On Tue, Sep 16, 2014 at 05:35:08PM +, Duncan wrote:
> W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as excerpted:
> > On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
> >> I don't see any benefit to using rsync vs. a shallow clone as the
> >> transmission protocol.
> > 
> > Other than the fact that before you dropped it you'd need to push a
> > ‘emerge sync’ that could handle either rsync or Git, stabilize that
> > Portage, and then wait for folks to adopt it.
> 
> Portage already handles it. =:^)

Oh, lovely :).  Looks like that landed in 2.2.0 with 47e8d22d (Add
support for multiple repositories in `emerge --sync`, 2013-07-23).
There are older Portages in the tree though (back to 2.1.6.7_p1), so
you'd still want to wait until those were gone before dropping rsync.

Also, I don't see a way to say “use Git to sync, but keep a shallow
repository”.  Ideally, we'd want:

  $ git clone --depth=1 git://git.gentoo.org/gentoo-portage.git

for the initial clone (modulo whatever URI), and:

  $ git pull --depth=1

for subsequent syncs.  pym/_emerge/actions.py currently hardcodes ‘git
pull’ for the latter, and doesn't seem to have any code for the
former.  On the other hand, it wouldn't be too terrible to force users
to shallow their history manually whenever they felt like it.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy


signature.asc
Description: OpenPGP digital signature

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

W. Trevor King posted on Mon, 15 Sep 2014 13:33:46 -0700 as excerpted:

> On Mon, Sep 15, 2014 at 01:29:44PM -0700, W. Trevor King wrote:
>> I don't see any benefit to using rsync vs. a shallow clone as the
>> transmission protocol.
> 
> Other than the fact that before you dropped it you'd need to push a
> ‘emerge sync’ that could handle either rsync or Git, stabilize that
> Portage, and then wait for folks to adopt it.

Portage already handles it. =:^)

Quoting from the emerge (1) manpage (listed as v2.2.12, dated Mar 2014):

--sync
  Updates  repositories, for which sync-type and sync-uri
  attributes are set in repos.conf. See portage(5) for more
  information.


Then in the portage (5) manpage:

repos.conf
  Specifies site-specific repository configuration information.
  Configuration specified in repos.conf can be overriden by
  PORTAGE_REPOSITORIES environmental variable, which has the
  same format as repos.conf.
 
  [...]
   
  Attributes supported in sections of repositories:

[...]

sync-type
  Specifies type of synchronization performed by `emerge
  --sync`. Valid non-empty values: cvs, git, rsync
  This  attribute  can be set to empty value to disable
  synchronization of given repository. Empty value is
  default.
   
sync-uri
  Specifies URI of repository used for synchronization
  performed by `emerge --sync`.  This attribute can be
  set to empty value to disable synchronization of given
  repository. Empty value is default.
   
  Syntax:
cvs: [cvs://]:access_method:[username@]hostname[:port]:/path
git: (git|git+ssh|http|https)://[username@]hostname[:port]/path
rsync: (rsync|ssh)://[username@]hostname[:port]/(module|path)
  

So portage already handles it. =:^)

Tho it's possible there are still bugs as surely that doesn't have the 
extremely broad long-term testing that rsync does, and I'm ~arch of 
course and don't know if that's stable, yet.  But it's there.


-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 1:07 PM, Luca Barbato  wrote:
> On 14/09/14 17:30, Patrick Lauer wrote:
>>> Are we going to disallow merge commits and ask devs to rebase local
>>> changes in order to keep the history "clean"?
>>
>> Is that going to be sane with our commit frequency?
>>
>
> Which is our commit frequency? Worst case we can aggregate changes and
> push them in bulk.
>

I don't think commit frequency will be a problem other than maybe
causing repoman issues if you're doing tree-wide changes (since
repoman takes a while to run).

See:
https://github.com/rich0/gentoo-gitmig-2014-09-15/graphs/commit-activity

Our tree gets 50-150 commits/day on average it seems.  I have no idea
how far back the punchcard view goes, but that should give a relative
sense of distribution.

--
Rich

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 15/09/14 01:21, Patrick Lauer wrote:
> On Sunday 14 September 2014 15:42:15 hasufell wrote:
>> Patrick Lauer:
 Are we going to disallow merge commits and ask devs to rebase local
 changes in order to keep the history "clean"?
>>>
>>> Is that going to be sane with our commit frequency?
>>
>> You have to merge or rebase anyway in case of a push conflict, so the
>> only difference is the method and the effect on the history.
>>
>> Currently... CVS allows you to run repoman on an outdated tree and push
>> broken ebuilds with repoman being happy. Git will not allow this.
> 
> iow, git doesn't allow people to work on more than one item at a time?

It does.

> That'd mean I need half a dozen checkouts just to emulate cvs, which somehow 
> doesn't make much sense to me ...

Your statement sounds strange to me.

commands you need to know:

git rebase -i

git add (-p)

git commit (-p)

git branch/checkout

Examples

edit cat/pkg/foo.ebuild

edit cat2/pkg/bar.ebuild

edit profile

git add -p# to select by line what you want in the commit

git commit# and you write down the commit message

git commit -p # to do both at the same time.

git commit -p # again to lump other changes line by line

OR

edit cat/pkg/foo.ebuild

git commit -a # everything (that's tracked) you edited gets in a commit

edit cat/pkg/bar.ebuild

git commit -a # everything (that's tracked) you edited gets in again

...

git rebase -i # sort out what you want commit merge, edit, drop etc

git push.




Git let you do whatever you do in cvs, but in a _much_ saner and faster way.

[gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Duncan]

Anthony G. Basile posted on Mon, 15 Sep 2014 17:43:09 -0400 as excerpted:

> We could just push out the word that ChangeLogs are going away and they
> have to read the git repo.  That might be the easiest solution.  I do
> have users that quote my ChangeLogs though.

As such a user... 

Given the proposed three-level system, dev-git, power-user-git, general-
user-rsync, dumping changelogs in rsync is my preferred solution as 
well.  Given power-user-git access I expect anyone who actually reads 
changelogs will be switching to the power-user-git level in a heartbeat, 
and I can't see the folks remaining on rsync actually caring about 
changelogs, either.  In the one-off case they find themselves needing a 
changelog, they can read the git commit log online.

When the discussion came up previously I was strongly in favor of keeping 
changelogs, because users /don't/ currently have a reasonable 
alternative.  With the proposed three-level system, the power-user git 
repo will be that alternative, and the changelogs can simply go away in 
favor of people actually having access to a full git repo and git log.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 17:30, Patrick Lauer wrote:
>> Are we going to disallow merge commits and ask devs to rebase local
>> changes in order to keep the history "clean"?
> 
> Is that going to be sane with our commit frequency?
> 

Which is our commit frequency? Worst case we can aggregate changes and
push them in bulk.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 16:46, Michał Górny wrote:
> Of course, if we can't spare the resources to do intermediate updates,
> we may as well switch to cron-based update method.

The mirror have a sync time, so basically regenerating the cache and
pushing the tree with further toward the user can happen the same way
w/out impacting anybody.

We could just snapshot the tree when the regen starts and push that
commit to the user git and rsync.

People is still supposed to play nice and sync not every minute.

lu

Re: [gentoo-dev] My masterplan for git migration (+ looking for infra to test it)

[Luca Barbato]

On 14/09/14 17:15, Kent Fredric wrote:
> On 15 September 2014 02:40, Michał Górny  wrote:
> 
>> However, I'm wondering if it would be possible to restrict people from
>> accidentally committing straight into github (e.g. merging pull
>> requests there instead of to our main server).
>>
> 
> 
> Easy.
> 
> Put the Gentoo repo in its own group.
> Don't give anyone any kinds of permissions on it.
> Have only one approved account for the purpose of pushing commits.
> Have a post-push hook that replicates to github as that approved account
> 
> => Github is just a read only mirror, any pull reqs submitted there will be
> fielded and pushed to gentoo directly.
> 
> Only downside there is the way github pull reqs work is if the final SHA1's
> that hit tree don't match, the pull req doesn't close.
> 
> Solutions:
> 
> - A) Have somebody tasked with reaping old pull reqs with permissions
> granted. ( Uck )
> - B) Always use a merge of some kind to mark the pull req as dead ( for
> instance, an "ours" merge to mark the branch as deprecated )

C) Ask nicely Github to have an application key and have a pull-request
bridge to avoid the problem completely.

I'd complete the migration first and discuss this kind of details later.

lu

Re: [gentoo-dev] git security (SHA-1)

[Peter Stuge]

Rich Freeman wrote:
> If you want to satisfy yourself I believe you can get git to dump
> the contents of any object without formatting/etc.

git ls-tree HEAD .
git show $blobhash
git show --pretty=raw HEAD


//Peter

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Pacho Ramos]

El mar, 16-09-2014 a las 09:55 -0400, Rich Freeman escribió:
> On Tue, Sep 16, 2014 at 9:44 AM, Pacho Ramos  wrote:
> >
> > Maybe one option would be to kill Changelogs and provide a script to let
> > people get git messages and reformat them in a way similar as current
> > ChangeLog files, that way people will still be able to save this
> > information for the future (if they won't have internet conection later
> > for example) and read it simply with "less" for example. With this
> > option, we won't need to provide Changelogs and distribute them but
> > people wanting to have them will still be able to generate them if
> > wanted (for example, just after updating portage tree)
> 
> Or they could just clone the git tree, and they can look at per-file
> logs anytime they want to.
> 
> I mean, sure, we COULD do this stuff.  But, why?
> 
> It isn't like kernel.org has some tool that lets kernel users generate
> per-file changelog histories just in case they don't want to use git.
> 
> If somebody wants to build a tool like this by all means go ahead and
> do it. I just don't see it as something that should be a migration
> pre-requisite.  That's just my opinion though.
> 
> --
> Rich
> 

I don't consider it a pre-requisite either, was only trying to give an
option to still tell people how to get a ChangeLog similar to current
ones easily (as looks like they are used a lot per the past
discussion :/) I remember something similar was done in the past when
gnome stuff moved to git:
https://wiki.gnome.org/Git/ChangeLog

But I guess once we get habituated to simply review something equivalent
to https://git.gnome.org/browse/ not many people will miss the old
Changelogs ;)

Re: [gentoo-dev] git security (SHA-1)

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 16/09/14 10:33 AM, Kent Fredric wrote:
> 
> On 17 September 2014 01:44, Ian Stakenvicius  > wrote:
> 
>> bottom of the comment a clearsign on the contents of the commit?
> 
> [...] the very best I think you could do is sign the commit *diff*,
> 
Yes, that's what I meant by the contents of the commit.  Apologies for
not using proper terminology.

-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlQYTIMACgkQ2ugaI38ACPBQzQD8Cc/7i/Q7v0NVv0duVNkpDAER
1PDOT+TLIDbv9DG0YW4BAIGSSLb0dahPp6+bdbe9AUR63m9QIrw51TqL3lGpGsm8
=jEVx
-END PGP SIGNATURE-

Re: [gentoo-dev] git security (SHA-1)

[hasufell]

Michael Orlitzky:
> On 09/16/2014 10:03 AM, Rich Freeman wrote:
>>
>> The gpg signature is on the entire contents of the "commit."  However,
>> the contents of the commit do not include the files that are being
>> committed - it includes hashes of the parent commit, the commit
>> message, other headers, and the hash of the tree being committed,
>> which is sha1.  That last hash is the only thing that ties the commit
>> to the files being committed, so you can modify the files all you like
>> as long as the sha1 is the same.
>>
> 
> To put things in perspective, all I had to do was ask for commit access
> and somebody eventually gave it to me. We should worry about this when
> breaking SHA1 becomes less expensive than the ebuild quizzes.
> 
> 

Yep, that's what I'd try to do actually if I was working for NSA
(uh-oh). Try to get "collaborators" into every possible opensource project.

There are so many thing you can do... e.g. "fix" a security bug, but
reference a self-packaged tarball from your dev space (which still
contains the exploit) in the ebuild. No one will know.
And that's a pretty low hanging fruit.

Re: [gentoo-dev] git security (SHA-1)

[Kent Fredric]

On 17 September 2014 01:44, Ian Stakenvicius  wrote:

> bottom of the comment a clearsign on the contents of the commit?
>


I don't see that being useful as written, because that's presently exactly
what git does.

the very best I think you could do is sign the commit *diff*, ie:
recursively compare TREE and PARENT->TREE and feed only *new* objects
through GPG. That's the most meaningful thing you'll get. ( And it really
is up-streams responsibility to implement this feature if they care,
bolting it on after the fact is horrible )

-- 
Kent

*KENTNL* - https://metacpan.org/author/KENTNL

Re: [gentoo-dev] git security (SHA-1)

[Michael Orlitzky]

On 09/16/2014 10:03 AM, Rich Freeman wrote:
> 
> The gpg signature is on the entire contents of the "commit."  However,
> the contents of the commit do not include the files that are being
> committed - it includes hashes of the parent commit, the commit
> message, other headers, and the hash of the tree being committed,
> which is sha1.  That last hash is the only thing that ties the commit
> to the files being committed, so you can modify the files all you like
> as long as the sha1 is the same.
> 

To put things in perspective, all I had to do was ask for commit access
and somebody eventually gave it to me. We should worry about this when
breaking SHA1 becomes less expensive than the ebuild quizzes.

Re: [gentoo-dev] git security (SHA-1)

[Rich Freeman]

On Tue, Sep 16, 2014 at 9:44 AM, Ian Stakenvicius  wrote:
>
> If the issue preventing protection is that the gpg signature only
> signs the hash, couldn't we just make repoman automatically add to the
> bottom of the comment a clearsign on the contents of the commit?
>

The gpg signature is on the entire contents of the "commit."  However,
the contents of the commit do not include the files that are being
committed - it includes hashes of the parent commit, the commit
message, other headers, and the hash of the tree being committed,
which is sha1.  That last hash is the only thing that ties the commit
to the files being committed, so you can modify the files all you like
as long as the sha1 is the same.

I don't think we should try to fix git.  It makes far more sense to
have upstream fix it.  I don't think we should hold up the migration
over it - NOBODY is holding off on adopting git over this stuff and
I'm not even aware of any projects that gpg sign their git commits.

Remember, the data model is:
commit --> tree --> [tree...] --> blob
The signature is against the commit, and sha1 hashes are what tie
everything else to it.

If you want to satisfy yourself I believe you can get git to dump the
contents of any object without formatting/etc.  You'll see that the
gpg signature matches the content of the commit (minus the gpg
signature header, of course).  If you directly access objects from the
filesystem I think git prepends a hash or something to the start of
every file.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 9:44 AM, Pacho Ramos  wrote:
>
> Maybe one option would be to kill Changelogs and provide a script to let
> people get git messages and reformat them in a way similar as current
> ChangeLog files, that way people will still be able to save this
> information for the future (if they won't have internet conection later
> for example) and read it simply with "less" for example. With this
> option, we won't need to provide Changelogs and distribute them but
> people wanting to have them will still be able to generate them if
> wanted (for example, just after updating portage tree)

Or they could just clone the git tree, and they can look at per-file
logs anytime they want to.

I mean, sure, we COULD do this stuff.  But, why?

It isn't like kernel.org has some tool that lets kernel users generate
per-file changelog histories just in case they don't want to use git.

If somebody wants to build a tool like this by all means go ahead and
do it. I just don't see it as something that should be a migration
pre-requisite.  That's just my opinion though.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Pacho Ramos]

El mar, 16-09-2014 a las 07:26 -0400, Rich Freeman escribió:
> On Tue, Sep 16, 2014 at 6:18 AM, hasufell  wrote:
> > Ulrich Mueller:
> >>
> >> ChangeLogs are aimed at users
> >
> > Did any1 ask them if they care?
> >
> 
> I'm sure somebody will reply and say that they care.
> 
> It still seems like a lot of overhead to me for a very one-off
> workflow.  Maybe if portage automatically output the relevant
> changelog entries in pretend mode we could pretend that they're news
> or something like that.  Most likely, if you stick something important
> in the changelog it will be read by maybe 0.1% of our users before
> emerging the package.  Maybe if you're lucky 20% of people running
> into some kind of breakage will read the changelog after the fact.  I
> imagine that 19.5% of those 20% would check the git log if the
> changelog didn't exist.
> 
> If we actually move to a model where many users actually sync their
> trees from git, then I'd expect the changelogs to be even less useful.
> After all, git will actually tell you what changed since your last
> sync.
> 
> --
> Rich
> 

Maybe one option would be to kill Changelogs and provide a script to let
people get git messages and reformat them in a way similar as current
ChangeLog files, that way people will still be able to save this
information for the future (if they won't have internet conection later
for example) and read it simply with "less" for example. With this
option, we won't need to provide Changelogs and distribute them but
people wanting to have them will still be able to generate them if
wanted (for example, just after updating portage tree)

Re: [gentoo-dev] git security (SHA-1)

[Ian Stakenvicius]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

On 15/09/14 07:59 PM, Rich Freeman wrote:
> On Mon, Sep 15, 2014 at 6:11 PM, Gordon Pettey
>  wrote:
>> 
>> Even if you wanted to burn the money to find that magical
>> collision that actually contains working code, you've still got
>> to somehow propagate that to other repositories, since they'll
>> just ignore it for having the same hash as an already-existing
>> object.
>> 
> 
> Well, if you're willing to trust that nobody is able to tamper
> with repositories, then you don't need gpg signatures in the first
> place.
> 
> I think that gpg signatures protected by an SHA1 hash provide
> fairly little security - a chain is as strong as its weakest link
> and sha1 has been considered fairly weak for years now.
> 
> However, I think it does make sense to at least get gpg into the 
> workflow in the hopes that some day git will move to a stronger
> hash, and since it isn't a huge hardship to do so.
> 
> I wouldn't make too light of the use of SHA1 though.  As you point
> out simply exploiting it isn't enough, but the whole reason for
> having signatures is to make an attack on a central repository
> useless. Having gpg on top of ssh keys and all that is obviously
> redundant, but that is the whole point of it.
> 
> -- Rich
> 

If the issue preventing protection is that the gpg signature only
signs the hash, couldn't we just make repoman automatically add to the
bottom of the comment a clearsign on the contents of the commit?


-BEGIN PGP SIGNATURE-
Version: GnuPG v2

iF4EAREIAAYFAlQYPskACgkQ2ugaI38ACPDjowEAmfMQePUgmLSDrmKyXxdUfbil
g6KVaPkL1yfDwrLP7J8BAK+g5MMCMDgH9wDzEHIYerDi9ZIm39AfwazQF3mz3dPR
=slAr
-END PGP SIGNATURE-

[gentoo-dev] Re: [gentoo-scm] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Mon, Sep 15, 2014 at 11:50 PM, Brian Harring  wrote:
>
> On Sun, Sep 14, 2014 at 10:33 AM, Rich Freeman  wrote:
>>
>> On Sun, Sep 14, 2014 at 8:03 AM, Michał Górny  wrote:
>> >
>> > Of course, that assumes infra is
>> > going to cooperate quickly or someone else is willing to provide the
>> > infra for it.
>>
>> The infra components to a git infrastructure are one of the main
>> blockers at this point.  I don't really see cooperation as the issue -
>> just lack of manpower or interest.
>
>
> I can't speak for current gentoo infra, but it'd help to think through
> exactly who will have RO access to the git repo.
>
> Trying to have the entire gentoo userbase accessing the repo via git is
> going to be resource intensive- if you're intending that, well, start
> donating frankly.
>
> If however you're angling for just devs having git access, and utilizing the
> existing rsync mirror, that should be a helluva lot more doable (even with
> shitty hardware).

I believe that is the proposal here.  There would be a dev-only main
repository which is where all commits would go.  Then there would be a
power-user sync repo derived from that, and then an rsync tree derived
from that.  This isn't all that different from what we do with cvs.

Git is pretty easy to mirror, since it is after all a distributed
system at its heart.  It would probably make sense to do one push/pull
from the main repository to the power-user repo, do whatever magic
needs to be done with it, and then push out from there to a mirror
network, github, wherever.  We'd probably want mirrors of both the
original dev tree and the one with metadata and such.


>> People love to point out linux and its insane commit rate.  The thing
>> is, the mainline git repo with all those commits has exactly one
>> committer - Linus himself.  They don't have one big repo with one
>> master branch that everybody pushes to.  At least, that is my
>> understanding (and there are certainly others here who are more
>> involved with kernel development).
>
>
> To be frank, I think you're seriously overthinking it and worrying about a
> nonissue.  Worrying about this while ignoring security enhancements like
> requiring signed commits on push is a bit weird to me.

So far the discussion in the thread has included the commit signing issue.

I tend to agree that commit rate probably won't be a problem after
some discussion, as long as we don't require a repoman check 100% of
the time in-between pull and push.  For a single package that won't be
a problem, but doing it at the category/tree level is just going to
make collisions inevitable.

> One additional point no one has mentioned; when cutting over to git,
> afterwards someone will need to go through and rip out all of the cvs $Id
> style tags that exist in the tree- depending on how folks are doing the
> conversion, it might be worth just jamming that in right off the bat rather
> than trying to clean the tree afterwards.

I'd suggest that we not go too much further with editing history -
consolidating Manifest/package commits did make sense, and the other
fixes do as well.  I'd prefer not to go trying to remove keywords from
the tree during conversion.  We've already had all kinds of headaches
from keywords as it is (especially with patches/etc).  I suggest that
we initially convert the tree and leave the keywords in, and we can
always clean them up later, either by script or manually.  I'd suggest
keeping scripting to a minimum guaranteed-safe level (such as just our
ebuild headers), and leave it up to maintainers to get any stray ones.

In-band signaling is lousy design in this day and age.  Oh, did
somebody bring up Changelogs?  :)

> ~ferringb, who was enjoying his retirement very much thank you

I appreciate your email - there is a lot of history we can stand to
benefit from.  I think I have a container set up now that can
efficiently migrate cvs->git using your scripts, and I'm just doing
some more testing and work to make it fully transportable.  From there
we just need to figure out where to run it when the time comes, which
should be the easy part.

I do believe that mgorny was offering to step up to help out with the
scripting though.  If we have the scripts/hooks needed to manipulate
gentoo-x86 to produce the various trees, then I think that
transplanting them to infra will not be a difficult next step.  Really
that part is the part which is most lacking right now, so if a few are
willing to step up on this I think we can get somewhere.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Rich Freeman:
> On Tue, Sep 16, 2014 at 6:18 AM, hasufell  wrote:
>> Ulrich Mueller:
>>>
>>> ChangeLogs are aimed at users
>>
>> Did any1 ask them if they care?
>>
> 
> I'm sure somebody will reply and say that they care.
> 
> It still seems like a lot of overhead to me for a very one-off
> workflow.  Maybe if portage automatically output the relevant
> changelog entries in pretend mode we could pretend that they're news
> or something like that.  Most likely, if you stick something important
> in the changelog it will be read by maybe 0.1% of our users before
> emerging the package.  Maybe if you're lucky 20% of people running
> into some kind of breakage will read the changelog after the fact.  I
> imagine that 19.5% of those 20% would check the git log if the
> changelog didn't exist.
> 
> If we actually move to a model where many users actually sync their
> trees from git, then I'd expect the changelogs to be even less useful.
> After all, git will actually tell you what changed since your last
> sync.
> 

And git allows you to _properly_ check for changes, because all changes
are in one history, so you don't have to grep 3+ ChangeLogs (e.g. in
eclasses, profiles and licenses) in order to know what happened.
Even easier... related changes might just go in one commit and when you
look for it you'll also see the other files that have been modified as
part of a version bump (e.g. a useflag mask or whatever).

The only place I actually look for changes is the gentoo-commits ML
which is kind of the poor version of a git history.

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Rich Freeman]

On Tue, Sep 16, 2014 at 6:18 AM, hasufell  wrote:
> Ulrich Mueller:
>>
>> ChangeLogs are aimed at users
>
> Did any1 ask them if they care?
>

I'm sure somebody will reply and say that they care.

It still seems like a lot of overhead to me for a very one-off
workflow.  Maybe if portage automatically output the relevant
changelog entries in pretend mode we could pretend that they're news
or something like that.  Most likely, if you stick something important
in the changelog it will be read by maybe 0.1% of our users before
emerging the package.  Maybe if you're lucky 20% of people running
into some kind of breakage will read the changelog after the fact.  I
imagine that 19.5% of those 20% would check the git log if the
changelog didn't exist.

If we actually move to a model where many users actually sync their
trees from git, then I'd expect the changelogs to be even less useful.
After all, git will actually tell you what changed since your last
sync.

--
Rich

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[Alan McKinnon]

On 16/09/2014 12:18, hasufell wrote:
> Ulrich Mueller:
>>
>> ChangeLogs are aimed at users
> 
> Did any1 ask them if they care?
> 
> 
> 



I'm a user and I don't care.

I use diff. I only go to the Changelog when I can't determine the
maintainers intent from diff and the ebuild content. That happens maybe
once a year.

-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-dev] Re: My masterplan for git migration (+ looking for infra to test it)

[hasufell]

Ulrich Mueller:
> 
> ChangeLogs are aimed at users

Did any1 ask them if they care?