Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)
On Thu, 3 Nov 2011 01:33:38 +0100 Andreas K. Huettel dilfri...@gentoo.org wrote: Dear all, 2) I'd like to suggest that for changelogs that grow beyond a certain size (e.g. profiles/ChangeLog) the file is rotated similar to /var/log logfiles. I.e. the current file is renamed with a date extension and a new file is started. This has the benefit that the archived file is static and will never be retransmitted by rsync. to prevent that this becomes a victim of general ChangeLog bikeshedding (we must rotate at a logical point, how could it be automatized even if it is relevant for only a few files, then how do we prevent epmty files...) I suggest the following procedure: In a week's time I personally, manually, will rotate all ChangeLog files larger than 100k in the tree, by splitting them at 31/12/2010-1/1/2011. The old entries file will in each case be named ChangeLog-2010 in the same directory. (PMS: A package directory may contain other files or directories, whose purpose is not covered by this specification.) Maybe we should keep old changelogs in a separate directory to decrease ebuilddir pollution? The new ChangeLog file will be identical to the current ChangeLog file except for being truncated at 1/1/2011. Maybe it'd be a good idea to add some kind of footer saying 'for further entries, please inquiry ChangeLog-2010 file'. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Rotating oversized ChangeLog files
On Thu, 3 Nov 2011, Andreas K Huettel wrote: The old entries file ChangeLog-2010 will be identical to the current ChangeLog file except for skipping at the start all entries added later than 31/12/2010. Just to make sure that I understand it: Does this imply that the old entries file will have a proper header? 774821 profiles/ChangeLog Maybe you could split this one on a yearly basis, to keep the chunks close to 100 kB? Ulrich
Re: [gentoo-dev] Rotating oversized ChangeLog files
On 11/03/2011 01:33 AM, Andreas K. Huettel wrote: In a week's time I personally, manually, will rotate all ChangeLog files larger than 100k in the tree, by splitting them at 31/12/2010-1/1/2011. Opinions, flames, ...? opinion Again for 'emerge --changelog': As we do have the $delay before breaking old period, usually with $delay=1 year: Should we also apply this $delay to the output of above command? If yes, what I can think of ATM is: * Do that first splitting in January 2012 (in less than 2 months). * Have PM search in the old ChangeLogs if necessary. The latter would also allow for completely emptying current ChangeLog each January by moving that to ChangeLog-$lastyear. /opinion /haubi/
Re: [gentoo-dev] Rotating oversized ChangeLog files
On Donnerstag 03 November 2011 09:24:07 Ulrich Mueller wrote: On Thu, 3 Nov 2011, Andreas K Huettel wrote: The old entries file ChangeLog-2010 will be identical to the current ChangeLog file except for skipping at the start all entries added later than 31/12/2010. Just to make sure that I understand it: Does this imply that the old entries file will have a proper header? Yes. 774821 profiles/ChangeLog Maybe you could split this one on a yearly basis, to keep the chunks close to 100 kB? Makes sense, yes. (With the earliest splitting when the file exceeds 100k for the first time.) -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Rotating oversized ChangeLog files
On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote: Again for 'emerge --changelog': As we do have the $delay before breaking old period, usually with $delay=1 year: Should we also apply this $delay to the output of above command? If yes, what I can think of ATM is: * Do that first splitting in January 2012 (in less than 2 months). * Have PM search in the old ChangeLogs if necessary. The latter would also allow for completely emptying current ChangeLog each January by moving that to ChangeLog-$lastyear. Makes all perfect sense... however I suggest to agree on a general scheme and go ahead manually first if implementation and/or discussion of its details or planned features is lingering... As opposed to EAPI features, this here is one of the cases where availability of an implementation means I'm here and can do it quickly. -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)
On Donnerstag 03 November 2011 09:09:19 Michał Górny wrote: Maybe we should keep old changelogs in a separate directory to decrease ebuilddir pollution? Not sure about that. The new ChangeLog file will be identical to the current ChangeLog file except for being truncated at 1/1/2011. Maybe it'd be a good idea to add some kind of footer saying 'for further entries, please inquiry ChangeLog-2010 file'. Yes, makes sense and is easily done. -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Rotating oversized ChangeLog files
On Thu, 3 Nov 2011, Andreas K Huettel wrote: On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote: As we do have the $delay before breaking old period, usually with $delay=1 year: Should we also apply this $delay to the output of above command? Makes all perfect sense... however I suggest to agree on a general scheme and go ahead manually first if implementation and/or discussion of its details or planned features is lingering... How about this: - Possible split points are only at the end of years. - Start at the end of the file and go backwards. - Split it whenever the piece after the split point is larger than $size. - Stop if the next split point is less than $delay ago. Reasonable values are 50 or 100 KiB for $size and 1 year for $delay, IMHO. ulrich
[gentoo-dev] recovering from corrupted vdb
Shouldn't portage offer some means to recover from a corrupted vdb? I just stumbled upon http://forums.gentoo.org/viewtopic-t-900382-start-0-postdays-0-postorder-asc-highlight-.html and it seems really bad. It would suck if the only solution to this is reinstall (I remember package database becoming corrupted in some RPM-based distro I had years ago and I hated it). If the recovery is already possible, we should get a doc explaining what to do. Otherwise it'd be really great to implement some recovery logic. I think we can't salvage much from a corrupted db (anything can happen, and the reporter mentions some code being present in the files), but at least emerge -e world or equivalent should be possible. signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] recovering from corrupted vdb
On Thu, Nov 3, 2011 at 7:15 AM, Paweł Hajdan, Jr. phajdan...@gentoo.org wrote: I think we can't salvage much from a corrupted db (anything can happen, and the reporter mentions some code being present in the files), but at least emerge -e world or equivalent should be possible. I'm not sure how portage handles not having ANYTHING in the vdb, but wouldn't it at least be possible to just wipe out the entire directory tree and then do an emerge -e world? As long as the packages themselves are working I would think that this should work. The only thing I'm not sure about is that if portage thinks that nothing is installed it might run into circular dependency issues. Maybe we need an option to include dependencies in the list of packages to install but not bail out on circular dependency issues since the reality is that the packages are there. Or, we need to give the user a script to follow (maybe try to follow whatever the logic is in catalyst since obviously that works). If the packages themselves are corrupted then installing from binary packages for @system would make sense. That thread really sounds like some kind of filesystem corruption issue, even if fsck doesn't report any problems. Something like that happened to me ages ago with some kind of mdadm+lvm+ext3 bug (an fsck on one lvm partition destroyed data on a different partition). A more intelligent solution would be to actually check the system for consistency (file hashes, etc), and then just re-emerge the stuff that is broken. Rich
Re: [gentoo-dev] Rotating oversized ChangeLog files (was: Old changelogs / eclass dir)
On Nov 3, 2011 10:25 a.m., Andreas K. Huettel dilfri...@gentoo.org
wrote:
On Donnerstag 03 November 2011 09:09:19 Michał Górny wrote:
Maybe we should keep old changelogs in a separate directory to decrease
ebuilddir pollution?
Not sure about that.
Thank you for this infusion of practicality.
How about in ${FILESDIR} ?
Re: [gentoo-dev] recovering from corrupted vdb
On 11/03/2011 04:15 AM, Paweł Hajdan, Jr. wrote: Shouldn't portage offer some means to recover from a corrupted vdb? I just stumbled upon http://forums.gentoo.org/viewtopic-t-900382-start-0-postdays-0-postorder-asc-highlight-.html and it seems really bad. It would suck if the only solution to this is reinstall (I remember package database becoming corrupted in some RPM-based distro I had years ago and I hated it). Your best protection is to have a redundant backup on a separate disk. When I do updates, I always clone my root partition to another partition on a separate disk and chroot into that for the updates, and I keep that spare partition as a backup in case one of my disks fails. For cloning, a command like `rsync -axH --delete / /mnt/backup_rootfs/` works well for me. I've also used btrfs subvolume snapshots for quick and efficient cloning of my root filesystem, but that doesn't give the kind of redundancy that a separate filesystem on a separate disk gives. If the recovery is already possible, we should get a doc explaining what to do. Otherwise it'd be really great to implement some recovery logic. I think we can't salvage much from a corrupted db (anything can happen, and the reporter mentions some code being present in the files), but at least emerge -e world or equivalent should be possible. Due to circular dependencies, you need a seed vdb to start with. If you don't have a redundant backup, another option would be to simply remove the corrupt /var/db/pkg and replace it with a copy from a stage3 tarball. -- Thanks, Zac
Re: [gentoo-dev] enew{user,group}: killing off [extra] argument
http://sources.gentoo.org/eclass/user.eclass?r1=1.8r2=1.9 -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Manifest signing
Hi, Am 02.11.2011 17:11, schrieb Robin H. Johnson: On Wed, Nov 02, 2011 at 01:03:21PM +0100, enno+gen...@groeper-berlin.de wrote: I followed the threads about manifest signing with interest and even had a look at the manifest signing guide [4]. Sounds nice at first view. But, please correct me, if I'm wrong. I didn't find a place where these signatures are verified. Is manifest signing for the infrastructure team, enabling them to verify the author of a commit (see GLEP57 [1])? Wouldn't this be obsoleted by commit signing if the move to git is done ([2])? Developer signing is radically altered in the face of git yes, that's one of the reasons not much has happened there. But the other larger reason is that developer signing pales in importance to the signature chain between infra-user. If developer signing works, it could act as a trust chain between (developer-)infra-user. And it could work with good old default emerge --sync, not only with emerge-webrsync and snapshots. If its senseless to do anything in this area as long as the move to git isn't done, there is no need to wine about unsigned manifests. At least if there isn't anyone checking developer signatures at the moment. If it is (also) for the users, why is there no code for it in portage anymore [3]? Hmm, I hadn't see that removal, but it makes sense unless the entire tree is developer-signed, which isn't likely to happen soon. I don't agree here. Of course the implementation shouldn't stop the user from installing an unsigned package at the moment. But it could give a warning instead and ask the user what to do. In this way developers are encouraged to sign their packages (to make the warning go away) and users get the ability to check the signatures, that already exist. Key problem here is the Gentoo keyring (how to ensure it didn't get manipulated). Okay why is clear. Obviously nobody was maintaining it... The code worked when I used it... I didn't check it. All I have are the commit messages and the feature-removal of the portage team. I thought about signing the manifests of my overlay. But this is senseless, if there is no automatic check. I can't think of any user verifying manifest signatures by hand. There's a chicken egg problem with most signing. You need to communicate the valid keys out of band from the actual repo. Maybe the layman data is a good place for that, but until such a location is figured out, you have zero security gain (if the 'correct' keys are only listed in a file in the repo, any attacker just replaces that when he puts his other content in). Of course. But security is always worth thinking about it. First step: What are the possibilities the check the signatures? FAIL. In my case some (most?) of the users of my overlay should know my GPG key already. The web of trust works here. The drawback for possible other users would be a false sense of security. How does infrastructure team check, if a GPG key belongs to a developer? The Manifest signing guide [4] simply says Upload the key to a keyserver. Everbody can upload a key to the public keyservers. An attacker, able to modify a signed Manifest, could simply create a new key on the developers name and use it to sign the modified manifest. Therefore it must be clear which key really belongs to a dev. Developers specify in their LDAP data what keys are theirs, and this gets exported here, amongst other places: http://www.gentoo.org/proj/en/devrel/roll-call/userinfo.xml Thanks for the enlightenment. But I doubt, if this should be the way to go (see below). There was a prototype keyserver at one point as well, and I can generate new keyrings if needed based on the LDAP data. This could be okay for a first creation. Later I would prefer something like Debian does: http://keyring.debian.org/replacing_keys.html That way you would decouple the LDAP and the keyring and trust only the data, that is already in the keyring (somebody whose key is already in the keyring signing the request for a new key). See also: http://keyring.debian.org/ Perhaps the prototype keyserver already did something like that. Furthermore the Tree-Signing-GLEPs [5] seem to be incomplete. This looks like the right place to continue work on Tree Signing. Those were the draft copies, which were finalized into GLEP 57..61. You are correct that there are two unfinished GLEPs in that directory: 02-developer-process-security 03-gnupg-policies-and-handling Of those, 03 can probably be written at this point. 02 is going to change radically when git comes into play. I had those 2 in mind, yes. Regards, Enno signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] Manifest signing
On Thu, Nov 03, 2011 at 10:55:52PM +0100, enno+gen...@groeper-berlin.de wrote: If it is (also) for the users, why is there no code for it in portage anymore [3]? Hmm, I hadn't see that removal, but it makes sense unless the entire tree is developer-signed, which isn't likely to happen soon. I don't agree here. Of course the implementation shouldn't stop the user from installing an unsigned package at the moment. But it could give a warning instead and ask the user what to do. In this way developers are encouraged to sign their packages (to make the warning go away) and users get the ability to check the signatures, that already exist. Key problem here is the Gentoo keyring (how to ensure it didn't get manipulated). Distributing the keyring itself signed is how Debian does it IIRC. There's a chicken egg problem with most signing. You need to communicate the valid keys out of band from the actual repo. Maybe the layman data is a good place for that, but until such a location is figured out, you have zero security gain (if the 'correct' keys are only listed in a file in the repo, any attacker just replaces that when he puts his other content in). Of course. But security is always worth thinking about it. First step: What are the possibilities the check the signatures? FAIL. In my case some (most?) of the users of my overlay should know my GPG key already. The web of trust works here. The drawback for possible other users would be a false sense of security. That's why I say the gpg key should be in the layman data. Overlays team, do you think this is reasonable? There was a prototype keyserver at one point as well, and I can generate new keyrings if needed based on the LDAP data. This could be okay for a first creation. Later I would prefer something like Debian does: http://keyring.debian.org/replacing_keys.html That way you would decouple the LDAP and the keyring and trust only the data, that is already in the keyring (somebody whose key is already in the keyring signing the request for a new key). See also: http://keyring.debian.org/ Perhaps the prototype keyserver already did something like that. The Debian model was discussed, and the main problem was finding enough people to sign the keys near all of the devs, esp. if you require meeting in person. You need two factors to be able to change your GPG key on file anyway. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Re: [gentoo-dev] ChangeLog in eclass dir (was: Old changelogs / eclass dir)
1) Why is there no ChangeLog in the eclass directory? In my personal opinion this is missing there, if only for historical reasons... Should we still start one? as there was only positive feedback to this suggestion, I'll create a ChangeLog file in the eclass directory during the next days. (Last chance to protest!) And done. :) huettel@pinacolada ~/Gentoo/gentoo-x86/eclass $ cvs commit /var/cvsroot/gentoo-x86/eclass/ChangeLog,v -- ChangeLog initial revision: 1.1 huettel@pinacolada ~/Gentoo/gentoo-x86/eclass $ -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
Re: [gentoo-dev] Rotating oversized ChangeLog files
On Donnerstag 03 November 2011 11:59:55 Ulrich Mueller wrote: On Thu, 3 Nov 2011, Andreas K Huettel wrote: On Donnerstag 03 November 2011 10:16:53 Michael Haubenwallner wrote: As we do have the $delay before breaking old period, usually with $delay=1 year: Should we also apply this $delay to the output of above command? Makes all perfect sense... however I suggest to agree on a general scheme and go ahead manually first if implementation and/or discussion of its details or planned features is lingering... How about this: - Possible split points are only at the end of years. - Start at the end of the file and go backwards. - Split it whenever the piece after the split point is larger than $size. - Stop if the next split point is less than $delay ago. Reasonable values are 50 or 100 KiB for $size and 1 year for $delay, IMHO. Sounds good. So, we have a spec... and the portage team has two months to get it into emerge --changelog. :) -- Andreas K. Huettel Gentoo Linux developer dilfri...@gentoo.org http://www.akhuettel.de/ signature.asc Description: This is a digitally signed message part.
