Re: [gentoo-dev] Rotating oversized ChangeLog files

[Andreas K. Huettel]

On Donnerstag 03 November 2011 11:59:55 Ulrich Mueller wrote:
> > On Thu, 3 Nov 2011, Andreas K Huettel wrote:
> > On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote:
> >> As we do have the "$delay before breaking old" period, usually with
> >> $delay="1 year": Should we also apply this $delay to the output of
> >> above command?
> > 
> > Makes all perfect sense... however I suggest to agree on a general
> > scheme and go ahead manually first if implementation and/or
> > discussion of its details or planned features is lingering...
> 
> How about this:
> - Possible split points are only at the end of years.
> - Start at the end of the file and go backwards.
> - Split it whenever the piece after the split point is larger than $size.
> - Stop if the next split point is less than $delay ago.
> 
> Reasonable values are 50 or 100 KiB for $size and 1 year for $delay,
> IMHO.

Sounds good. So, we have a spec... and the portage team has two months to get 
it into "emerge --changelog". :)

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] ChangeLog in eclass dir (was: Old changelogs / eclass dir)

[Andreas K. Huettel]

> > 1) Why is there no ChangeLog in the eclass directory?
> > In my personal opinion this is missing there, if only for historical
> > reasons... Should we still start one?
> 
> as there was only positive feedback to this suggestion, I'll create a
> ChangeLog file in the eclass directory during the next days. (Last chance
> to protest!)

And done. :)

huettel@pinacolada ~/Gentoo/gentoo-x86/eclass $ cvs commit
/var/cvsroot/gentoo-x86/eclass/ChangeLog,v  <--  ChangeLog
initial revision: 1.1
huettel@pinacolada ~/Gentoo/gentoo-x86/eclass $ 


-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Manifest signing

[Robin H. Johnson]

On Thu, Nov 03, 2011 at 10:55:52PM +0100, enno+gen...@groeper-berlin.de wrote:
> >> If it is (also) for the users, why is there no code for it in portage
> >> anymore [3]?
> > Hmm, I hadn't see that removal, but it makes sense unless the entire
> > tree is developer-signed, which isn't likely to happen soon.
> I don't agree here. Of course the implementation shouldn't stop the user
> from installing an unsigned package at the moment. But it could give a
> warning instead and ask the user what to do.
> In this way developers are encouraged to sign their packages (to make
> the warning go away) and users get the ability to check the signatures,
> that already exist.
> Key problem here is the Gentoo keyring (how to ensure it didn't get
> manipulated).
Distributing the keyring itself signed is how Debian does it IIRC.

> > There's a chicken & egg problem with most signing. You need to
> > communicate the valid keys out of band from the actual repo.
> > Maybe the layman data is a good place for that, but until such a
> > location is figured out, you have zero security gain (if the 'correct'
> > keys are only listed in a file in the repo, any attacker just replaces
> > that when he puts his other content in).
> Of course. But security is always worth thinking about it.
> First step: What are the possibilities the check the signatures? FAIL.
> In my case some (most?) of the users of my overlay should know my GPG
> key already. The web of trust works here. The drawback for possible
> other users would be a false sense of security.
That's why I say the gpg key should be in the layman data.
Overlays team, do you think this is reasonable?

> > There was a prototype keyserver at one point as well, and I can generate
> > new keyrings if needed based on the LDAP data.
> This could be okay for a first creation. Later I would prefer something
> like Debian does:
> http://keyring.debian.org/replacing_keys.html
> That way you would decouple the LDAP and the keyring and trust only the
> data, that is already in the keyring (somebody whose key is already in
> the keyring signing the request for a new key).
> See also: http://keyring.debian.org/
> Perhaps the prototype keyserver already did something like that.
The Debian model was discussed, and the main problem was finding enough
people to sign the keys near all of the devs, esp. if you require
meeting in person.

You need two factors to be able to change your GPG key on file anyway.

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-dev] Manifest signing

[enno+gentoo]

Hi,

Am 02.11.2011 17:11, schrieb Robin H. Johnson:
> On Wed, Nov 02, 2011 at 01:03:21PM +0100, enno+gen...@groeper-berlin.de wrote:
>> I followed the threads about manifest signing with interest and even had
>> a look at the manifest signing guide [4]. Sounds nice at first view.
>> But, please correct me, if I'm wrong. I didn't find a place where these
>> signatures are verified.
>> Is manifest signing for the infrastructure team, enabling them to verify
>> the author of a commit (see GLEP57 [1])? Wouldn't this be obsoleted by
>> commit signing if the move to git is done ([2])?
> Developer signing is radically altered in the face of git yes, that's
> one of the reasons not much has happened there. But the other larger
> reason is that developer signing pales in importance to the signature
> chain between infra->user.
If developer signing works, it could act as a trust chain between
(developer->)infra->user. And it could work with good old default
"emerge --sync", not only with emerge-webrsync and snapshots.
If its senseless to do anything in this area as long as the move to git
isn't done, there is no need to wine about unsigned manifests.
At least if there isn't anyone checking developer signatures at the moment.

>> If it is (also) for the users, why is there no code for it in portage
>> anymore [3]?
> Hmm, I hadn't see that removal, but it makes sense unless the entire
> tree is developer-signed, which isn't likely to happen soon.
I don't agree here. Of course the implementation shouldn't stop the user
from installing an unsigned package at the moment. But it could give a
warning instead and ask the user what to do.
In this way developers are encouraged to sign their packages (to make
the warning go away) and users get the ability to check the signatures,
that already exist.
Key problem here is the Gentoo keyring (how to ensure it didn't get
manipulated).

>> Okay "why" is clear. Obviously nobody was maintaining it...
> The code worked when I used it...
I didn't check it. All I have are the commit messages and the
feature-removal of the portage team.

>> I thought about signing the manifests of my overlay. But this is
>> senseless, if there is no automatic check. I can't think of any user
>> verifying manifest signatures by hand.
> There's a chicken & egg problem with most signing. You need to
> communicate the valid keys out of band from the actual repo.
> Maybe the layman data is a good place for that, but until such a
> location is figured out, you have zero security gain (if the 'correct'
> keys are only listed in a file in the repo, any attacker just replaces
> that when he puts his other content in).
Of course. But security is always worth thinking about it.
First step: What are the possibilities the check the signatures? FAIL.
In my case some (most?) of the users of my overlay should know my GPG
key already. The web of trust works here. The drawback for possible
other users would be a false sense of security.

>> How does infrastructure team check, if a GPG key belongs to a developer?
>> The Manifest signing guide [4] simply says "Upload the key to a
>> keyserver". Everbody can upload a key to the public keyservers. An
>> attacker, able to modify a signed Manifest, could simply create a new
>> key on the developers name and use it to sign the modified manifest.
>> Therefore it must be clear which key really belongs to a dev.
> Developers specify in their LDAP data what keys are theirs, and this
> gets exported here, amongst other places:
> http://www.gentoo.org/proj/en/devrel/roll-call/userinfo.xml
Thanks for the enlightenment. But I doubt, if this should be the way to
go (see below).

> There was a prototype keyserver at one point as well, and I can generate
> new keyrings if needed based on the LDAP data.
This could be okay for a first creation. Later I would prefer something
like Debian does:
http://keyring.debian.org/replacing_keys.html
That way you would decouple the LDAP and the keyring and trust only the
data, that is already in the keyring (somebody whose key is already in
the keyring signing the request for a new key).
See also: http://keyring.debian.org/
Perhaps the prototype keyserver already did something like that.

> 
>> Furthermore the Tree-Signing-GLEPs [5] seem to be incomplete.
>> This looks like the right place to continue work on Tree Signing.
> Those were the draft copies, which were finalized into GLEP 57..61.
> You are correct that there are two unfinished GLEPs in that directory:
> 02-developer-process-security
> 03-gnupg-policies-and-handling
> 
> Of those, 03 can probably be written at this point.
> 02 is going to change radically when git comes into play.
I had those 2 in mind, yes.

Regards,
Enno



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] enew{user,group}: killing off [extra] argument

[Mike Frysinger]

http://sources.gentoo.org/eclass/user.eclass?r1=1.8&r2=1.9
-mike


signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] recovering from corrupted vdb

[Zac Medico]

On 11/03/2011 04:15 AM, "Paweł Hajdan, Jr." wrote:
> Shouldn't portage offer some means to recover from a corrupted vdb?
> 
> I just stumbled upon
> 
> and it seems really bad.
> 
> It would suck if the only solution to this is reinstall (I remember
> package database becoming corrupted in some RPM-based distro I had years
> ago and I hated it).

Your best protection is to have a redundant backup on a separate disk.
When I do updates, I always clone my root partition to another partition
on a separate disk and chroot into that for the updates, and I keep that
spare partition as a backup in case one of my disks fails. For cloning,
a command like `rsync -axH --delete / /mnt/backup_rootfs/` works well
for me. I've also used btrfs subvolume snapshots for quick and efficient
cloning of my root filesystem, but that doesn't give the kind of
redundancy that a separate filesystem on a separate disk gives.

> If the recovery is already possible, we should get a doc explaining what
> to do. Otherwise it'd be really great to implement some recovery logic.
> 
> I think we can't salvage much from a corrupted db (anything can happen,
> and the reporter mentions some code being present in the files), but at
> least "emerge -e world" or equivalent should be possible.

Due to circular dependencies, you need a seed vdb to start with. If you
don't have a redundant backup, another option would be to simply remove
the corrupt /var/db/pkg and replace it with a copy from a stage3 tarball.
-- 
Thanks,
Zac

Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)

[James Broadhead]

On Nov 3, 2011 10:25 a.m., "Andreas K. Huettel" 
wrote:
>
> On Donnerstag 03 November 2011 09:09:19 Michał Górny wrote:
> >
> > Maybe we should keep old changelogs in a separate directory to decrease
> > ebuilddir pollution?
>
> Not sure about that.

Thank you for this infusion of practicality.

How about in ${FILESDIR} ?

Re: [gentoo-dev] recovering from corrupted vdb

[Rich Freeman]

On Thu, Nov 3, 2011 at 7:15 AM, "Paweł Hajdan, Jr."
 wrote:
> I think we can't salvage much from a corrupted db (anything can happen,
> and the reporter mentions some code being present in the files), but at
> least "emerge -e world" or equivalent should be possible.

I'm not sure how portage handles not having ANYTHING in the vdb, but
wouldn't it at least be possible to just wipe out the entire directory
tree and then do an emerge -e world?  As long as the packages
themselves are working I would think that this should work.

The only thing I'm not sure about is that if portage thinks that
nothing is installed it might run into circular dependency issues.
Maybe we need an option to include dependencies in the list of
packages to install but not bail out on circular dependency issues
since the reality is that the packages are there.  Or, we need to give
the user a script to follow (maybe try to follow whatever the logic is
in catalyst since obviously that works).

If the packages themselves are corrupted then installing from binary
packages for @system would make sense.

That thread really sounds like some kind of filesystem corruption
issue, even if fsck doesn't report any problems.  Something like that
happened to me ages ago with some kind of mdadm+lvm+ext3 bug (an fsck
on one lvm partition destroyed data on a different partition).

A more intelligent solution would be to actually check the system for
consistency (file hashes, etc), and then just re-emerge the stuff that
is broken.

Rich

[gentoo-dev] recovering from corrupted vdb

[Paweł Hajdan, Jr.]

Shouldn't portage offer some means to recover from a corrupted vdb?

I just stumbled upon

and it seems really bad.

It would suck if the only solution to this is reinstall (I remember
package database becoming corrupted in some RPM-based distro I had years
ago and I hated it).

If the recovery is already possible, we should get a doc explaining what
to do. Otherwise it'd be really great to implement some recovery logic.

I think we can't salvage much from a corrupted db (anything can happen,
and the reporter mentions some code being present in the files), but at
least "emerge -e world" or equivalent should be possible.



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Rotating oversized ChangeLog files

[Ulrich Mueller]

> On Thu, 3 Nov 2011, Andreas K Huettel wrote:

> On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote:

>> As we do have the "$delay before breaking old" period, usually with
>> $delay="1 year": Should we also apply this $delay to the output of
>> above command?

> Makes all perfect sense... however I suggest to agree on a general
> scheme and go ahead manually first if implementation and/or
> discussion of its details or planned features is lingering...

How about this:
- Possible split points are only at the end of years.
- Start at the end of the file and go backwards.
- Split it whenever the piece after the split point is larger than $size.
- Stop if the next split point is less than $delay ago.

Reasonable values are 50 or 100 KiB for $size and 1 year for $delay,
IMHO.

ulrich

Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)

[Andreas K. Huettel]

On Donnerstag 03 November 2011 09:09:19 Michał Górny wrote:
> 
> Maybe we should keep old changelogs in a separate directory to decrease
> ebuilddir pollution?

Not sure about that.

> 
> > The new ChangeLog file will be identical to the current ChangeLog
> > file except for being truncated at 1/1/2011.
> 
> Maybe it'd be a good idea to add some kind of footer saying 'for
> further entries, please inquiry ChangeLog-2010 file'.

Yes, makes sense and is easily done.

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Rotating oversized ChangeLog files

[Andreas K. Huettel]

On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote:
>
> Again for 'emerge --changelog':
> 
> As we do have the "$delay before breaking old" period, usually with
> $delay="1 year": Should we also apply this $delay to the output of above
> command?
> 
> If yes, what I can think of ATM is:
> * Do that first splitting in January 2012 (in less than 2 months).
> * Have PM search in the old ChangeLogs if necessary.
> 
> The latter would also allow for completely emptying current ChangeLog each
> January by moving that to ChangeLog-$lastyear.
> 

Makes all perfect sense... however I suggest to agree on a general scheme and 
go ahead manually first if implementation and/or discussion of its details or 
planned features is lingering...

As opposed to EAPI features, this here is one of the cases where "availability 
of an implementation" means "I'm here and can do it quickly".

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Rotating oversized ChangeLog files

[Andreas K. Huettel]

On Donnerstag 03 November 2011 09:24:07 Ulrich Mueller wrote:
> > On Thu, 3 Nov 2011, Andreas K Huettel wrote:
> > The "old entries" file ChangeLog-2010 will be identical to the
> > current ChangeLog file except for skipping at the start all entries
> > added later than 31/12/2010.
> 
> Just to make sure that I understand it: Does this imply that the old
> entries file will have a proper header?

Yes.

> > 774821  profiles/ChangeLog
> 
> Maybe you could split this one on a yearly basis, to keep the chunks
> close to 100 kB?

Makes sense, yes. (With the earliest splitting when the file exceeds 100k for 
the first time.)

-- 

Andreas K. Huettel
Gentoo Linux developer 
dilfri...@gentoo.org
http://www.akhuettel.de/



signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-dev] Rotating oversized ChangeLog files

[Michael Haubenwallner]

On 11/03/2011 01:33 AM, Andreas K. Huettel wrote:
> In a week's time I personally, manually, will "rotate" all ChangeLog files 
> larger than 100k in the tree, by splitting them at 31/12/2010-1/1/2011.

> Opinions, flames, ...?



Again for 'emerge --changelog':

As we do have the "$delay before breaking old" period, usually with $delay="1 
year":
Should we also apply this $delay to the output of above command?

If yes, what I can think of ATM is:
* Do that first splitting in January 2012 (in less than 2 months).
* Have PM search in the old ChangeLogs if necessary.

The latter would also allow for completely emptying current ChangeLog each 
January
by moving that to ChangeLog-$lastyear.



/haubi/

Re: [gentoo-dev] Rotating oversized ChangeLog files

[Ulrich Mueller]

> On Thu, 3 Nov 2011, Andreas K Huettel wrote:

> The "old entries" file ChangeLog-2010 will be identical to the
> current ChangeLog file except for skipping at the start all entries
> added later than 31/12/2010.

Just to make sure that I understand it: Does this imply that the old
entries file will have a proper header?

> 774821  profiles/ChangeLog

Maybe you could split this one on a yearly basis, to keep the chunks
close to 100 kB?

Ulrich

Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)

[Michał Górny]

On Thu, 3 Nov 2011 01:33:38 +0100
"Andreas K. Huettel"  wrote:

> Dear all,
> 
> > 2) I'd like to suggest that for changelogs that grow beyond a
> > certain size (e.g. profiles/ChangeLog) the file is "rotated"
> > similar to /var/log logfiles. I.e. the current file is renamed with
> > a date extension and a new file is started. This has the benefit
> > that the archived file is static and will never be retransmitted by
> > rsync.
> 
> to prevent that this becomes a victim of general ChangeLog
> bikeshedding (we must rotate at a logical point, how could it be
> automatized even if it is relevant for only a few files, then how do
> we prevent epmty files...) I suggest the following procedure:
> 
> In a week's time I personally, manually, will "rotate" all ChangeLog
> files larger than 100k in the tree, by splitting them at
> 31/12/2010-1/1/2011. The old entries file will in each case be named
> ChangeLog-2010 in the same directory. (PMS: "A package directory may
> contain other files or directories, whose purpose is not covered by
> this specification.")

Maybe we should keep old changelogs in a separate directory to decrease
ebuilddir pollution?

> The new ChangeLog file will be identical to the current ChangeLog
> file except for being truncated at 1/1/2011.

Maybe it'd be a good idea to add some kind of footer saying 'for
further entries, please inquiry ChangeLog-2010 file'.

-- 
Best regards,
Michał Górny


signature.asc
Description: PGP signature