Re: [gentoo-portage-dev] Normaliser function for distfiles

2022-05-16 Thread Robin H. Johnson
On Mon, May 16, 2022 at 07:37:40PM +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve

Re: [gentoo-portage-dev] [PATCH] ecompress: optimize docompress -x precompressed comparison

2020-06-29 Thread Robin H. Johnson
On Sun, Jun 28, 2020 at 12:54:56PM -0700, Zac Medico wrote: > Use sort and comm with temporary files in order to compare lists > of docompress -x and precompressed files, since the file lists > can be extremely large. Also strip ${D%/} from paths in order to > reduce length. +1 looks much better.

Re: [gentoo-portage-dev] [PATCH] ecompress: fix "Argument list too long" for sed (bug 727522)

2020-06-24 Thread Robin H. Johnson
On Tue, Jun 23, 2020 at 05:36:14PM -0700, Zac Medico wrote: > From: Patrick McLean > > Use sed -f to feed commands to sed via stdin, in order to avoid > the "Argument list too long" error reported in bug 727522. Will this need to move to a tempfile in the near future, for the size of sed_args?

[gentoo-portage-dev] [r...@gentoo.org: Cron /usr/local/bin/pidlock -s sync-distfiles /usr/bin/timeout -k 2h 1h /usr/local/bin/mastermirror/sync-distfiles.sh]

2019-11-05 Thread Robin H. Johnson
Can Portage handle this error more gracefully please? The symlink test file is there to verify mirror behavior, so we don't want to delete it either. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E

Re: [gentoo-portage-dev] [PATCH v2] misc: Distribute a repo.postsync.d hook to run gemato verification

2018-01-16 Thread Robin H. Johnson
On Tue, Jan 16, 2018 at 11:32:28AM -0800, Zac Medico wrote: > > But app-crypt/gentoo-keys doesn't include that executable, and it has > > no dependency on app-crypt/gkeys. I'd rather not introduce an artificial > > dependency here. > > I suppose we could using a separate ebuild to install this

Re: [gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment

2017-11-06 Thread Robin H. Johnson
On Mon, Nov 06, 2017 at 09:14:56AM +0100, Michał Górny wrote: > -# Future events: > -# > -# After WHIRLPOOL is supported in stable portage for at least 1 year: > -# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL. > -# - Remove SHA256 from MANIFEST2_HASH_*. > -# - Set manifest-hashes in

Re: [gentoo-portage-dev] [PATCH] ebuild.sh: Completely ban external commands in global scope

2017-09-08 Thread Robin H. Johnson
On Thu, Aug 31, 2017 at 10:45:42PM +0200, Michał Górny wrote: > + export PATH=/dev/null Minor nitpick: The Single UNIX spec says that PATH is a set of prefixes, and that they're treated as directories. http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html I think it might be good to

Re: [gentoo-portage-dev] [PATCH] repos.conf: support strict-misc-digests attribute (bug 600128)

2016-11-26 Thread Robin H. Johnson
On Fri, Nov 25, 2016 at 12:32:28PM -0800, Zac Medico wrote: > I question the usefulness of producing warnings that people probably > want to ignore anyway, but I suppose we could make the > Manifest.checkTypeHashes method implement this behavior internally. Ok, then I'd like to flip your default

Re: [gentoo-portage-dev] [PATCH] repos.conf: support strict-misc-digests attribute (bug 600128)

2016-11-24 Thread Robin H. Johnson
On Wed, Nov 23, 2016 at 11:04:54PM -0800, Zac Medico wrote: > The current GLEP 60 draft specifies that non-strict handling of MISC > digests should be supported. In my followup post about how it should work, I noted that in non-strict mode, a non-fatal warning should be issued for the mismatch of

Re: [gentoo-portage-dev] Enforced OpenPGP signatures

2016-06-14 Thread Robin H. Johnson
On Tue, Jun 14, 2016 at 10:41:38AM +0200, Alexander Berntsen wrote: > Friends, > > I saw Brian asking Michał to OpenPGP-sign his commits in IRC, to which > Michał quipped that we would have if it were enforced. So perhaps we > should just enforce it. Most of us do it -- but I see Zac not doing it

Re: [gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.

2015-11-13 Thread Robin H. Johnson
On Fri, Nov 13, 2015 at 01:26:24PM +0100, Alexander Berntsen wrote: > On 12/11/15 22:21, Robin H. Johnson wrote: > > Thanks, merged. > Sorry, what? You're not in the Portage team. The last time you > committed directly was in 2007; I can't speak for that period of time, > b

Re: [gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.

2015-11-12 Thread Robin H. Johnson
On Wed, Nov 11, 2015 at 10:09:42PM -0800, Zac Medico wrote: > On 11/11/2015 02:30 PM, robb...@gentoo.org wrote: > > From: "Robin H. Johnson" <robb...@gentoo.org> > > > > If GIT_DIR is used, and .git is outside the root of the checkout, then > > --work-tr

Re: [gentoo-portage-dev] [PATCH] egencache: parallelize --update-changelogs (bug 565540)

2015-11-11 Thread Robin H. Johnson
Confirmed to work; please merge (also my prior patch that fixes relative GIT_DIR implications, or ACK and I will merge it myself). Speedup is less than expected however. Running with: --jobs 10 --load-average 6 yields a ~3.3x speedup on the previous non-parallel version. With the system load

[gentoo-portage-dev] Portage Feature Request: making thirdpartymirrors easier to manage

2014-01-06 Thread Robin H. Johnson
This is a small feature request, but it will require a modification to PMS, so I describe it here. The present thirdpartymirrors file is unwieldy, and difficult to manage due to it's format with very long lines. It also doesn't permit easy comments. Presently commits to it look very ugly, because

Re: [gentoo-portage-dev] [PATCH] make.conf.5: Document PYTHON_TARGETS, bug #493180

2013-12-03 Thread Robin H. Johnson
On Tue, Dec 03, 2013 at 11:05:51AM -0500, Mike Frysinger wrote: as for the patch, i'm of the opinion that make.conf is not for documenting random USE_EXPAND-ed variables. ... there is the matter of visibility ... we could add a generic pointer to the make.conf man page discussing that there

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-02 Thread Robin H. Johnson
On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote: On 10/02/2011 05:46 AM, Robin H. Johnson wrote: On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote: If we control these hashes via metadata/layout.conf, then we can toggle it atomically for all commiters. Otherwise, we'll

[gentoo-portage-dev] [GLEP59v2 4/5] Manifest2 hash backend provider: mhash

2011-10-01 Thread Robin H. Johnson
From: Robin H. Johnson robb...@gentoo.org Offer mhash as a provider for Manifest2 hash generation and validation. This is important as either of pycrypto or fchksum offer an accelerated Whirlpool implementation, and hashlib might not offer it. Additionally, the mhash implementation is accelerated

[gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

2011-10-01 Thread Robin H. Johnson
From: Robin H. Johnson robb...@gentoo.org Provide public-domain implementation of the Whirlpool hash algorithm to be used as new Manifest2 hash. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/checksum.py |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff

[gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

2011-10-01 Thread Robin H. Johnson
Respun now with the help of ferringb. Cleans up the implementation and catches a few bug and improvements: - mhash priority moved lower than pycrypto/hashlib because mhash holds GIL while the other implementations don't. - hashlib does offer whirlpool if it was built against openssl 1.0. 1/5:

[gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-01 Thread Robin H. Johnson
From: Robin H. Johnson robb...@gentoo.org Change Manifest2 hashes to a more secure set as approved in GLEP59. SHA512 and WHIRLPOOL are added, SHA1 and RMD160 are dropped. SHA256 is now the lowest security hash, and must remain in Manifest files for at least 1 year, otherwise older Portage

[gentoo-portage-dev] [GLEP59v2 1/5] Refactor RMD160 hashlib code for less-hardcoding

2011-10-01 Thread Robin H. Johnson
From: Robin H. Johnson robb...@gentoo.org To be used shortly for WHIRLPOOL as well as RMD160. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/checksum.py | 21 - 1 files changed, 12 insertions(+), 9 deletions(-) diff --git a/pym/portage/checksum.py b

Re: [gentoo-portage-dev] [PATCH 3/4] Manifest2 hash backend provider: mhash

2011-09-30 Thread Robin H. Johnson
On Fri, Sep 30, 2011 at 01:27:41AM +, Robin H. Johnson wrote: Offer mhash as a provider for Manifest2 hash generation and validation. This is important as none of pycrypto/hashlib/fchksum offer an accelerated Whirlpool implementaiton. Additionally, the mhash implementation is accelerated

[gentoo-portage-dev] [PATCH 3/4] Manifest2 hash backend provider: mhash

2011-09-29 Thread Robin H. Johnson
Offer mhash as a provider for Manifest2 hash generation and validation. This is important as none of pycrypto/hashlib/fchksum offer an accelerated Whirlpool implementaiton. Additionally, the mhash implementation is accelerated and ships with a rigorious testsuite. Signed-off-by: Robin H. Johnson

[gentoo-portage-dev] [PATCH 2/4] Manifest2 hash: SHA512

2011-09-29 Thread Robin H. Johnson
Provide SHA512 hash algorithm to be used as new Manifest2 hash. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/checksum.py |1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py index 3d674c8..b2c9333 100644

[gentoo-portage-dev] [PATCH 4/4] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-09-29 Thread Robin H. Johnson
any of the hashes in the Manifest files. Future events: After 2012/10/01: - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL. - Remove SHA256 from MANIFEST2_HASH_FUNCTIONS. After SHA-3 is approved: - Add new hashes to MANIFEST2_HASH_FUNCTIONS. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym

Re: [gentoo-portage-dev] VCS used for development of portage

2010-03-05 Thread Robin H. Johnson
On Fri, Mar 05, 2010 at 04:33:14PM +0100, Sebastian Pipping wrote: I don't feel like proposing anything on that matter at the moment. With that said: what do you and Robin think? Here's a related question. Did the previous CVS - SVN question generate the svn:ignore files from .cvsignore, or

Re: [gentoo-portage-dev] About boosting sync

2008-12-02 Thread Robin H. Johnson
On Tue, Dec 02, 2008 at 07:46:13PM +0200, Tambet wrote: Has anyone ever noticed that portage tree contains a lot of md5 hashes, which are not at all important for using it? I think that it does not make reliability or functionality smaller any bit if those would all stay in sync servers -

[gentoo-portage-dev] Spam Redux

2008-10-03 Thread Robin H. Johnson
Somebody subscribed a bad list manager to the list, and caused a mail loop. I removed the offending list address now, but I don't know who did it in the first place. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E

Re: [gentoo-portage-dev] [2/4] proto-GLEPS for Tree-signing

2008-07-29 Thread Robin H. Johnson
On Tue, Jul 29, 2008 at 08:51:45PM +0100, Mike Auty wrote: In this Glep (xx+1), in the section discussing the procedure for creating a MetaManifest file, in step 3.3, does that include verification of the manifest's signature if it has one? It would seem odd to ignore the signature if it's

[gentoo-portage-dev] proto-GLEPS for Tree-signing, take 2

2008-07-12 Thread Robin H. Johnson
So I'm not going to directly attach the GLEPs again this time, however I am just going to link to them, and summarize the changes: xx+1: - Add mention of how to defeat the mirror replay attacks from [EMAIL PROTECTED] - Clarify wording of the UNCOVERED=ALL-COVERED set math, and why it's

[gentoo-portage-dev] [0/4] proto-GLEPS for Tree-signing

2008-07-01 Thread Robin H. Johnson
Hi folks, it's that time again, time for the proto-gleps on tree-signing. Barring two minor TODO items, I have completed all of the series dealing with distribution issues and Manifest2. The developer issues and gnupg management issues remain, but they don't block the Manifest2 and distribution

[gentoo-portage-dev] [1/4] proto-GLEPS for Tree-signing

2008-07-01 Thread Robin H. Johnson
, thread includes the first GnuPG signing prototype code, by Robin H. Johnson (robbat2). Andrew Cowie (rac) also produces a proof-of-concept around this time. 2004-03-23, gentoo-dev mailing list, 2004.1 will not include a secure portage - Kurt Lieber (klieber). Signing is nowhere near ready for 2004.1

[gentoo-portage-dev] [2/4] proto-GLEPS for Tree-signing

2008-07-01 Thread Robin H. Johnson
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+1 Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest Version: $Revision:

[gentoo-portage-dev] [3/4] proto-GLEPS for Tree-signing

2008-07-01 Thread Robin H. Johnson
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+4 Title: Manifest2 hash policies and security implications Version: $Revision: 1.10 $ Last-Modified: $Date: 2008/07/01

[gentoo-portage-dev] [4/4] proto-GLEPS for Tree-signing

2008-07-01 Thread Robin H. Johnson
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+5 Title: Manifest2 filetypes Version: $Revision: 1.15 $ Last-Modified: $Date: 2008/07/01 08:52:34 $ Author: Robin Hugh Johnson

Re: [gentoo-portage-dev] state of GPG-signing in portage

2008-01-11 Thread Robin H. Johnson
On Sat, Jan 12, 2008 at 05:49:10AM +0100, Marcel Meyer wrote: I'm wondering if the GPG-signing feature within portage is already useable (if I recall correctly it was startet 2004 or 2005?). If yes, how can I use it correctly and where to get the gpg-key securely? The URLs I found by

Re: [gentoo-portage-dev] Improvement suggestion for emerge: Not using a new connection for every file

2007-02-24 Thread Robin H. Johnson
On Sat, Feb 24, 2007 at 10:00:29PM +0100, Beginner wrote: I recommend not to use wget and not to reconnect to the server for every single packet, but to hold the connection therefore spare traffic and download more fast. If you are doing lots of downloads, use 'emerge -pvf FOO' and feed each

Re: [gentoo-portage-dev] Masked by corruption

2007-01-11 Thread Robin H. Johnson
On Fri, Jan 12, 2007 at 12:18:34AM +0200, Philipp Riegger wrote: On 02.01.2007, at 06:56, Zac Medico wrote: In =portage-2.1.2_rc4-r2 t does that now for installed package (see bug #158931). For /var/cache/edb/dep the sqlite module is available (requires pysqlite or python-2.5 with sqlite