Re: [gentoo-portage-dev] Normaliser function for distfiles
On Mon, May 16, 2022 at 07:37:40PM +0200, Markus Walter wrote: > Hello all, > > is it possible to do the following: after fetching a distfile portage runs > an external normaliser program specified in an ebuild before checking the > hash? > > My use case is the following: I would like to improve the gs-elpa program > and provide a precomputed overlay for melpa. However the melpa distfiles are > rebuilt everyday and cause checksum failures. However the only thing > changing are the timestamps. Hence if a normaliser program could simply set > all timestamps to some predefined value (say 1.1.1970) then this problem > should vanish. I don't know what 'gs-elpa' & 'Melpa' are, but maybe talking to upstream would be good here, and improving that behavior. If the file contents or non-timestamp metadata change, absolutely the timestamps should change. But otherwise, the timestamp should NOT change. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] ecompress: optimize docompress -x precompressed comparison
On Sun, Jun 28, 2020 at 12:54:56PM -0700, Zac Medico wrote:
> Use sort and comm with temporary files in order to compare lists
> of docompress -x and precompressed files, since the file lists
> can be extremely large. Also strip ${D%/} from paths in order to
> reduce length.
+1 looks much better.
--
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Treasurer
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
signature.asc
Description: PGP signature
Re: [gentoo-portage-dev] [PATCH] ecompress: fix "Argument list too long" for sed (bug 727522)
On Tue, Jun 23, 2020 at 05:36:14PM -0700, Zac Medico wrote: > From: Patrick McLean > > Use sed -f to feed commands to sed via stdin, in order to avoid > the "Argument list too long" error reported in bug 727522. Will this need to move to a tempfile in the near future, for the size of sed_args? Maybe do that work now? -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: PGP signature
[gentoo-portage-dev] [r...@gentoo.org: Cron /usr/local/bin/pidlock -s sync-distfiles /usr/bin/timeout -k 2h 1h /usr/local/bin/mastermirror/sync-distfiles.sh]
Can Portage handle this error more gracefully please? The symlink test file is there to verify mirror behavior, so we don't want to delete it either. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 --- Begin Message --- [ERROR] rename symlink-test from distfiles to recycle failed: [Errno 2] No such file or directory Traceback (most recent call last): File "/usr/lib64/python2.7/site-packages/portage/util/_async/ForkProcess.py", line 55, in _spawn rval = self._run() File "/usr/lib64/python2.7/site-packages/portage/util/_async/FileCopier.py", line 16, in _run shutil.copy(self.src_path, self.dest_path) File "/usr/lib64/python2.7/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) File "/usr/lib64/python2.7/shutil.py", line 139, in copy copyfile(src, dst) File "/usr/lib64/python2.7/shutil.py", line 96, in copyfile with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/data/gmirror-distfiles/distfiles/ee/symlink-test' Traceback (most recent call last): File "/usr/bin/emirrordist", line 86, in exec(data, new_globals) File "", line 23, in File "/usr/lib64/python2.7/site-packages/portage/_emirrordist/main.py", line 450, in emirrordist_main signum = run_main_scheduler(MirrorDistTask(config)) File "/usr/lib64/python2.7/site-packages/portage/util/_async/run_main_scheduler.py", line 27, in run_main_scheduler scheduler.wait() File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 83, in wait self.scheduler.run_until_complete(self.async_wait()) File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 831, in run_until_complete self.iteration() File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 285, in iteration return self._iteration(*args) File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 306, in _iteration if self._run_timeouts(): File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 601, in _run_timeouts if self._run_idle_callbacks(): File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 560, in _run_idle_callbacks if x._callback(*x._args): File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 101, in __call__ self._callback(*self._args) File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 84, in wait self._wait_hook() File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 195, in _wait_hook self._exit_listener_stack.pop()(self) File "/usr/lib64/python2.7/site-packages/portage/_emirrordist/DeletionTask.py", line 92, in _recycle_copier_exit "to recycle failed: %s") % (self.distfile, e)) UnboundLocalError: local variable 'e' referenced before assignment --- End Message --- signature.asc Description: PGP signature
Re: [gentoo-portage-dev] [PATCH v2] misc: Distribute a repo.postsync.d hook to run gemato verification
On Tue, Jan 16, 2018 at 11:32:28AM -0800, Zac Medico wrote: > > But app-crypt/gentoo-keys doesn't include that executable, and it has > > no dependency on app-crypt/gkeys. I'd rather not introduce an artificial > > dependency here. > > I suppose we could using a separate ebuild to install this hook, so that > we can update it separately from portage if necessary. The hook can > still live in the portage repository (like emerge-delta-webrsync which > is also installed by a separate ebuild). How about shipping this hook with Gemato, since it's of no use without Gemato installed. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: [gentoo-portage-dev] [PATCH 1/3] portage.const: Remove obsolete manifest-hashes comment
On Mon, Nov 06, 2017 at 09:14:56AM +0100, Michał Górny wrote: > -# Future events: > -# > -# After WHIRLPOOL is supported in stable portage for at least 1 year: > -# - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL. > -# - Remove SHA256 from MANIFEST2_HASH_*. > -# - Set manifest-hashes in gentoo-x86/metadata/layout.conf as follows: > -# manifest-hashes = SHA512 WHIRLPOOL Can we please drop SHA256 as a required hash already? It was scheduled for future removal a long time ago. I don't want that to get lost with the removal of this comment block. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] ebuild.sh: Completely ban external commands in global scope
On Thu, Aug 31, 2017 at 10:45:42PM +0200, Michał Górny wrote: > + export PATH=/dev/null Minor nitpick: The Single UNIX spec says that PATH is a set of prefixes, and that they're treated as directories. http://pubs.opengroup.org/onlinepubs/7908799/xbd/envvar.html I think it might be good to use either a non-existing path, or a known empty directory (/var/empty), rather than /dev/null which DOES exist. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136 signature.asc Description: Digital signature
Re: [gentoo-portage-dev] [PATCH] repos.conf: support strict-misc-digests attribute (bug 600128)
On Fri, Nov 25, 2016 at 12:32:28PM -0800, Zac Medico wrote: > I question the usefulness of producing warnings that people probably > want to ignore anyway, but I suppose we could make the > Manifest.checkTypeHashes method implement this behavior internally. Ok, then I'd like to flip your default to off (not checking MISC by default). Goal being that unless specifically requested, having ChangeLog with a mismatched Manifest is not going to stop them emerging the package. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: [gentoo-portage-dev] [PATCH] repos.conf: support strict-misc-digests attribute (bug 600128)
On Wed, Nov 23, 2016 at 11:04:54PM -0800, Zac Medico wrote: > The current GLEP 60 draft specifies that non-strict handling of MISC > digests should be supported. In my followup post about how it should work, I noted that in non-strict mode, a non-fatal warning should be issued for the mismatch of Manifest and existing MISC file. You take the approach here of just not checking the MISC files, with a default of checking them anyway. The non-fatal warning variant enables keeping the check enabled without breaking the tree. Do I need to codify that into a GLEP update to get traction? -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: [gentoo-portage-dev] Enforced OpenPGP signatures
On Tue, Jun 14, 2016 at 10:41:38AM +0200, Alexander Berntsen wrote: > Friends, > > I saw Brian asking Michał to OpenPGP-sign his commits in IRC, to which > Michał quipped that we would have if it were enforced. So perhaps we > should just enforce it. Most of us do it -- but I see Zac not doing it > sometimes, seemingly at random. In any event, I don't think there's a > good reason *not* to sign things. > > What do you think? And what's the procedure/who do we talk to, to get > a pre-push hook set up to enforce it? A pre-push hook would only do it locally for you, it wouldn't enforce it on the server side. Please file a bug to have infra turn it on for the repos you want (specify them in the bug). Here's the actual hook that's used: https://github.com/gentoo/git-gx86-tools/blob/master/hooks/dev-git/update-02-gpg Note that it only verifies on the master branch, and for merges, only the merge-commit onto master is verified. -- Robin Hugh Johnson Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85 GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
Re: [gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.
On Fri, Nov 13, 2015 at 01:26:24PM +0100, Alexander Berntsen wrote: > On 12/11/15 22:21, Robin H. Johnson wrote: > > Thanks, merged. > Sorry, what? You're not in the Portage team. The last time you > committed directly was in 2007; I can't speak for that period of time, > but today only team members commit to Portage (which is how it works > for essentially any other software development project, ever). > > I'm baffled you even have commit access. Does everyone in Gentoo just > have commit access to Portage or something? Infra can pretty much commit to anywhere; but refrain from doing so without somebody else reviewing our changes and stating that they are good. zmedico did review it, and since it wasn't committed yet after that, I committed it, on the assumption that he wasn't going to at the time. Separately however, from the old days of Portage (SVN and CVS), I was in the old cvsportage and svnportage groups, and permitted to make changes. I know that the Git ACLs came from those, and did explicitly include me at one point, but got cleaned up since infra's permissions are mostly implicit and I didn't need extra explicit bits on my user. Please ping me on IRC so we can clean up the remaining historical Git ACLs; I count 5 people not on the wiki page with explicit bits to commit to Portage.git. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-portage-dev] [PATCH] egencache: fix results when GIT_DIR is used in the environment.
On Wed, Nov 11, 2015 at 10:09:42PM -0800, Zac Medico wrote: > On 11/11/2015 02:30 PM, robb...@gentoo.org wrote: > > From: "Robin H. Johnson" <robb...@gentoo.org> > > > > If GIT_DIR is used, and .git is outside the root of the checkout, then > > --work-tree=... needs to be specified, otherwise any Git command that > > relies on relative directories to the root will be wrong. > > > > Signed-off-by: Robin H. Johnson <robb...@gentoo.org> > > --- > > bin/egencache | 31 +++ > > 1 file changed, 23 insertions(+), 8 deletions(-) > Looks good. Thanks, merged. > > + '--relative=%s' % (cp, ), > This addition could be mentioned in the commit message. Added to the commit message. > > def run(self): > > - repo_path = self._portdb.porttrees[0] > > - os.chdir(repo_path) > > + os.chdir(self._repo_path) > The chdir calls make me nervous, since we should be careful to ensure > that they are timed correctly when we introduce parallelization. I'd > love to add a cwd argument for the grab method to pass into Popen, in > order to eliminate all of the chdir calls. Agreed; I like your later patch for it. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-portage-dev] [PATCH] egencache: parallelize --update-changelogs (bug 565540)
Confirmed to work; please merge (also my prior patch that fixes relative GIT_DIR implications, or ACK and I will merge it myself). Speedup is less than expected however. Running with: --jobs 10 --load-average 6 yields a ~3.3x speedup on the previous non-parallel version. With the system load average stabilizing around 2.9. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead, Foundation Trustee E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
[gentoo-portage-dev] Portage Feature Request: making thirdpartymirrors easier to manage
This is a small feature request, but it will require a modification to PMS, so I describe it here. The present thirdpartymirrors file is unwieldy, and difficult to manage due to it's format with very long lines. It also doesn't permit easy comments. Presently commits to it look very ugly, because diffs are line-based, and we pack a lot into each line. I would like to make it a directory instead of a single file, and extend the internal syntax. 1. New location: $PROFILEDIR/thirdpartymirrors/$MIRRORNAME 1.1. The name of the mirror is now the name of the file. 1.2. We can have a file extension of .mirrors if somebody would like that. 2. New format (for directory-mode): 2.1. Comments permitted, shell-style. 2.2. Blank lines ignored 2.3. One URL per line, optionally prefixed with - or + 2.4. For stack repos/overlays: 2.4.1. No prefix: replace all prior mirrors from masters with new URLS in this file. 2.4.2. - prefix: remove this URL from the list from masters. 2.4.2. + prefix: append this URL to the list from masters. 3. New format (for file-mode): 3.1. This is for cases where thirdpartymirrors is still a file. 3.2. The first token on a line remains the name of the mirror. 3.3. Each subsequent token may be prefixed with + or -, and impacts prior lines/masters. -- Robin Hugh Johnson Gentoo Linux: Developer, Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-portage-dev] [PATCH] make.conf.5: Document PYTHON_TARGETS, bug #493180
On Tue, Dec 03, 2013 at 11:05:51AM -0500, Mike Frysinger wrote:
as for the patch, i'm of the opinion that make.conf is not for documenting
random USE_EXPAND-ed variables.
...
there is the matter of visibility ... we could add a generic pointer to the
make.conf man page discussing that there are many more variables that might
impact the build and to look at the man page for the eclasses.
Can I make a possible additional suggestion?
We do already have profiles/${VAR}.desc, describing what the settings
themselves do. Can we augment those files to describe the variable
itself?
Then put a generic note in make.conf about reading those (or something
to build docs from them like the eclass docs).
--
Robin Hugh Johnson
Gentoo Linux: Developer, Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL
On Sun, Oct 02, 2011 at 01:39:41PM -0700, Zac Medico wrote: On 10/02/2011 05:46 AM, Robin H. Johnson wrote: On Sat, Oct 01, 2011 at 09:40:13PM -0700, Zac Medico wrote: If we control these hashes via metadata/layout.conf, then we can toggle it atomically for all commiters. Otherwise, we'll have an annoying period of time where different committers are committing different sets of hashes, depending on their portage version. How do you suggest doing it via layout.conf? I've kept SHA256 in both sets for now, but if you could enforce new signatures including both WHIRLPOOL and SHA256, that would be great. How about if we put something like this in gentoo-x86/metadata/layout.conf now: Did you mean profiles/layout.conf? I just want to make sure no scripts that pull from CVS and expect that dir to not exist don't break. manifest2-sha1 = true manifest2-whirlpool = false Bikeshedding slightly, but can we figure something like a list or dict instead? (Also gives us a chance to make the required hashes a list). manifest2-hashes = ['SHA1', 'SHA256', 'RMD160'] Then we'll patch portage so that by default it will disable SHA1 and enable WHIRLPOOL, and the above settings will override the defaults. After the patched portage is marked stable in a month or so, we'll send an announcement to gentoo-announce, and remove the above settings from layout.conf. Sounds good to me. Hopefully I'll have more of the MetaManifest prototype code in the next few days to go live around the same time. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
[gentoo-portage-dev] [GLEP59v2 4/5] Manifest2 hash backend provider: mhash
From: Robin H. Johnson robb...@gentoo.org Offer mhash as a provider for Manifest2 hash generation and validation. This is important as either of pycrypto or fchksum offer an accelerated Whirlpool implementation, and hashlib might not offer it. Additionally, the mhash implementation is accelerated and ships with a rigorious testsuite. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/checksum.py | 19 +++ 1 files changed, 19 insertions(+), 0 deletions(-) diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py index 40ae836..c0c7c04 100644 --- a/pym/portage/checksum.py +++ b/pym/portage/checksum.py @@ -75,6 +75,25 @@ sha1hash = _generate_hash_function(SHA1, _new_sha1, origin=internal) from portage.util.whirlpool import new as _new_whirlpool whirlpoolhash = _generate_hash_function(WHIRLPOOL, _new_whirlpool, origin=bundled) +# Try to use mhash if available +# mhash causes GIL presently, so it gets less priority than hashlib and +# pycrypto. However, it might be the only accelerated implementation of +# WHIRLPOOL available. +try: + import mhash, functools + md5hash = _generate_hash_function(MD5, functools.partial(mhash.MHASH, mhash.MHASH_MD5), origin=mhash) + sha1hash = _generate_hash_function(SHA1, functools.partial(mhash.MHASH, mhash.MHASH_SHA1), origin=mhash) + sha256hash = _generate_hash_function(SHA256, functools.partial(mhash.MHASH, mhash.MHASH_SHA256), origin=mhash) + sha512hash = _generate_hash_function(SHA512, functools.partial(mhash.MHASH, mhash.MHASH_SHA512), origin=mhash) + for local_name, hash_name in ((rmd160, ripemd160), (whirlpool, whirlpool)): + if hasattr(mhash, 'MHASH_%s' % local_name.upper()): + globals()['%shash' % local_name] = \ + _generate_hash_function(local_name.upper(), \ + functools.partial(mhash.MHASH, getattr(mhash, 'MHASH_%s' % s.upper())), \ + origin='mhash') +except ImportError as e: + pass + # Use pycrypto when available, prefer it over the internal fallbacks try: from Crypto.Hash import SHA256, RIPEMD -- 1.7.7
[gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool
From: Robin H. Johnson robb...@gentoo.org Provide public-domain implementation of the Whirlpool hash algorithm to be used as new Manifest2 hash. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/checksum.py |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py index e5455fa..3593686 100644 --- a/pym/portage/checksum.py +++ b/pym/portage/checksum.py @@ -71,6 +71,10 @@ except ImportError: sha1hash = _generate_hash_function(SHA1, _new_sha1, origin=internal) +# Bundled WHIRLPOOL implementation +from portage.util.whirlpool import new as _new_whirlpool +whirlpoolhash = _generate_hash_function(WHIRLPOOL, _new_whirlpool, origin=bundled) + # Use pycrypto when available, prefer it over the internal fallbacks try: from Crypto.Hash import SHA256, RIPEMD @@ -80,14 +84,14 @@ except ImportError as e: pass # Use hashlib from python-2.5 if available and prefer it over pycrypto and internal fallbacks. -# Need special handling for RMD160 as it may not always be provided by hashlib. +# Need special handling for RMD160/WHIRLPOOL as they may not always be provided by hashlib. try: import hashlib, functools md5hash = _generate_hash_function(MD5, hashlib.md5, origin=hashlib) sha1hash = _generate_hash_function(SHA1, hashlib.sha1, origin=hashlib) sha256hash = _generate_hash_function(SHA256, hashlib.sha256, origin=hashlib) - for local_name, hash_name in ((rmd160, ripemd160), ): + for local_name, hash_name in ((rmd160, ripemd160), (whirlpool, whirlpool)): try: hashlib.new(hash_name) except ValueError: -- 1.7.7
[gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types
Respun now with the help of ferringb. Cleans up the implementation and catches a few bug and improvements: - mhash priority moved lower than pycrypto/hashlib because mhash holds GIL while the other implementations don't. - hashlib does offer whirlpool if it was built against openssl 1.0. 1/5: Refactor RMD160 hashlib code for less-hardcoding 2/5: Manifest2 hash: Whirlpool 3/5: Manifest2 hash: SHA512 4/5: Manifest2 hash backend provider: mhash 5/5: GLEP59: Change live Manifest2 hashes to SHA256,
[gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL
From: Robin H. Johnson robb...@gentoo.org Change Manifest2 hashes to a more secure set as approved in GLEP59. SHA512 and WHIRLPOOL are added, SHA1 and RMD160 are dropped. SHA256 is now the lowest security hash, and must remain in Manifest files for at least 1 year, otherwise older Portage installs will complain that they do not support any of the hashes in the Manifest files. Future events: After 2012/10/01: - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL. - Remove SHA256 from MANIFEST2_HASH_FUNCTIONS. After SHA-3 is approved: - Add new hashes to MANIFEST2_HASH_FUNCTIONS. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/const.py |6 -- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/pym/portage/const.py b/pym/portage/const.py index 8b5f4ac..a42ebe8 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -109,10 +109,12 @@ EAPI = 4 HASHING_BLOCKSIZE= 32768 MANIFEST1_HASH_FUNCTIONS = (MD5, SHA256, RMD160) -MANIFEST2_HASH_FUNCTIONS = (SHA1, SHA256, RMD160) +MANIFEST2_HASH_FUNCTIONS = (SHA256, SHA512, WHIRLPOOL) +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01 MANIFEST1_REQUIRED_HASH = MD5 -MANIFEST2_REQUIRED_HASH = SHA1 +MANIFEST2_REQUIRED_HASH = SHA256 +# FUTURE: Change to WHIRLPOOL after 2012/10/01 MANIFEST2_IDENTIFIERS= (AUX, MISC, DIST, EBUILD) # === -- 1.7.7
[gentoo-portage-dev] [GLEP59v2 1/5] Refactor RMD160 hashlib code for less-hardcoding
From: Robin H. Johnson robb...@gentoo.org
To be used shortly for WHIRLPOOL as well as RMD160.
Signed-off-by: Robin H. Johnson robb...@gentoo.org
---
pym/portage/checksum.py | 21 -
1 files changed, 12 insertions(+), 9 deletions(-)
diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 9e7e455..e5455fa 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -82,19 +82,22 @@ except ImportError as e:
# Use hashlib from python-2.5 if available and prefer it over pycrypto and
internal fallbacks.
# Need special handling for RMD160 as it may not always be provided by hashlib.
try:
- import hashlib
+ import hashlib, functools
md5hash = _generate_hash_function(MD5, hashlib.md5, origin=hashlib)
sha1hash = _generate_hash_function(SHA1, hashlib.sha1,
origin=hashlib)
sha256hash = _generate_hash_function(SHA256, hashlib.sha256,
origin=hashlib)
- try:
- hashlib.new('ripemd160')
- except ValueError:
- pass
- else:
- def rmd160():
- return hashlib.new('ripemd160')
- rmd160hash = _generate_hash_function(RMD160, rmd160,
origin=hashlib)
+ for local_name, hash_name in ((rmd160, ripemd160), ):
+ try:
+ hashlib.new(hash_name)
+ except ValueError:
+ pass
+ else:
+ globals()['%shash' % local_name] = \
+ _generate_hash_function(local_name.upper(), \
+ functools.partial(hashlib.new, hash_name), \
+ origin='hashlib')
+
except ImportError as e:
pass
--
1.7.7
Re: [gentoo-portage-dev] [PATCH 3/4] Manifest2 hash backend provider: mhash
On Fri, Sep 30, 2011 at 01:27:41AM +, Robin H. Johnson wrote: Offer mhash as a provider for Manifest2 hash generation and validation. This is important as none of pycrypto/hashlib/fchksum offer an accelerated Whirlpool implementaiton. Additionally, the mhash implementation is accelerated and ships with a rigorious testsuite. ferringb has pointed out that mhash holds the GIL lock, and that hashlib does support whirlpool (despite the docs not mentioning it). This portion might end up dropped then. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
[gentoo-portage-dev] [PATCH 3/4] Manifest2 hash backend provider: mhash
Offer mhash as a provider for Manifest2 hash generation and validation.
This is important as none of pycrypto/hashlib/fchksum offer an
accelerated Whirlpool implementaiton. Additionally, the mhash
implementation is accelerated and ships with a rigorious testsuite.
Signed-off-by: Robin H. Johnson robb...@gentoo.org
---
pym/portage/checksum.py | 19 +--
1 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index b2c9333..c3df36b 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -16,7 +16,7 @@ import tempfile
hashfunc_map = {}
hashorigin_map = {}
-def _generate_hash_function(hashtype, hashobject, origin=unknown):
+def _generate_hash_function(hashtype, hashobject, origin=unknown,
hashobjectargs=None):
def pyhash(filename):
Run a checksum against a file.
@@ -41,7 +41,11 @@ def _generate_hash_function(hashtype, hashobject,
origin=unknown):
blocksize = HASHING_BLOCKSIZE
data = f.read(blocksize)
size = 0
- checksum = hashobject()
+ if hashobjectargs is None:
+ checksum = hashobject()
+ else:
+ checksum = hashobject(hashobjectargs)
+
while data:
checksum.update(data)
size = size + len(data)
@@ -103,6 +107,17 @@ try:
except ImportError as e:
pass
+# Try to use mhash if available
+try:
+ import mhash
+ md5hash = _generate_hash_function(MD5, mhash.MHASH, origin=mhash,
hashobjectargs=mhash.MHASH_MD5)
+ sha1hash = _generate_hash_function(SHA1, mhash.MHASH, origin=mhash,
hashobjectargs=mhash.MHASH_SHA1)
+ sha256hash = _generate_hash_function(SHA256, mhash.MHASH,
origin=mhash, hashobjectargs=mhash.MHASH_SHA256)
+ sha512hash = _generate_hash_function(SHA512, mhash.MHASH,
origin=mhash, hashobjectargs=mhash.MHASH_SHA512)
+ rmd160hash = _generate_hash_function(RMD160, mhash.MHASH,
origin=mhash, hashobjectargs=mhash.MHASH_RIPEMD160)
+ whirlpoolhash = _generate_hash_function(WHIRLPOOL, mhash.MHASH,
origin=mhash, hashobjectargs=mhash.MHASH_WHIRLPOOL)
+except ImportError as e:
+ pass
# Use python-fchksum if available, prefer it over all other MD5 implementations
try:
--
1.7.6
[gentoo-portage-dev] [PATCH 2/4] Manifest2 hash: SHA512
Provide SHA512 hash algorithm to be used as new Manifest2 hash.
Signed-off-by: Robin H. Johnson robb...@gentoo.org
---
pym/portage/checksum.py |1 +
1 files changed, 1 insertions(+), 0 deletions(-)
diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 3d674c8..b2c9333 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -91,6 +91,7 @@ try:
md5hash = _generate_hash_function(MD5, hashlib.md5, origin=hashlib)
sha1hash = _generate_hash_function(SHA1, hashlib.sha1,
origin=hashlib)
sha256hash = _generate_hash_function(SHA256, hashlib.sha256,
origin=hashlib)
+ sha512hash = _generate_hash_function(SHA512, hashlib.sha512,
origin=hashlib)
try:
hashlib.new('ripemd160')
except ValueError:
--
1.7.6
[gentoo-portage-dev] [PATCH 4/4] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL
Change Manifest2 hashes to a more secure set as approved in GLEP59. SHA512 and WHIRLPOOL are added, SHA1 and RMD160 are dropped. SHA256 is now the lowest security hash, and must remain in Manifest files for at least 1 year, otherwise older Portage installs will complain that they do not support any of the hashes in the Manifest files. Future events: After 2012/10/01: - Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL. - Remove SHA256 from MANIFEST2_HASH_FUNCTIONS. After SHA-3 is approved: - Add new hashes to MANIFEST2_HASH_FUNCTIONS. Signed-off-by: Robin H. Johnson robb...@gentoo.org --- pym/portage/const.py |6 -- 1 files changed, 4 insertions(+), 2 deletions(-) diff --git a/pym/portage/const.py b/pym/portage/const.py index 8b5f4ac..a42ebe8 100644 --- a/pym/portage/const.py +++ b/pym/portage/const.py @@ -109,10 +109,12 @@ EAPI = 4 HASHING_BLOCKSIZE= 32768 MANIFEST1_HASH_FUNCTIONS = (MD5, SHA256, RMD160) -MANIFEST2_HASH_FUNCTIONS = (SHA1, SHA256, RMD160) +MANIFEST2_HASH_FUNCTIONS = (SHA256, SHA512, WHIRLPOOL) +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01 MANIFEST1_REQUIRED_HASH = MD5 -MANIFEST2_REQUIRED_HASH = SHA1 +MANIFEST2_REQUIRED_HASH = SHA256 +# FUTURE: Change to WHIRLPOOL after 2012/10/01 MANIFEST2_IDENTIFIERS= (AUX, MISC, DIST, EBUILD) # === -- 1.7.6
Re: [gentoo-portage-dev] VCS used for development of portage
On Fri, Mar 05, 2010 at 04:33:14PM +0100, Sebastian Pipping wrote: I don't feel like proposing anything on that matter at the moment. With that said: what do you and Robin think? Here's a related question. Did the previous CVS - SVN question generate the svn:ignore files from .cvsignore, or simply discard them? In either case, I'm starting to wonder if the change is just trivial enough to get done in svn2git or git-svn directly. I think other properties are already there. -- Robin Hugh Johnson Gentoo Linux: Developer, Trustee Infrastructure Lead E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Re: [gentoo-portage-dev] About boosting sync
On Tue, Dec 02, 2008 at 07:46:13PM +0200, Tambet wrote: Has anyone ever noticed that portage tree contains a lot of md5 hashes, which are not at all important for using it? I think that it does not make reliability or functionality smaller any bit if those would all stay in sync servers - anyway, syncing would go much faster and this tree smaller. What about removing all those md5 hashes and downloading them only when they're needed? Umm, what are you on? There are no more MD5s in Manifest2. It should be only RMD160, SHA1, SHA256. If you DO find a Manifest with an MD5, I'd REALLY like to know about it. As for the important of Manifests and the hashes, I'd like to offer the following as suggested reading: http://www.cs.arizona.edu/people/justin/packagemanagersecurity/ Specifically, see the papers page, and find the paper from CCS 2008 [1]. He DID solicit input from me on how Gentoo deals with the issue, and gave it fair coverage in my opinion. It's CRITICALLY important that the checksums go with the content, and that the checksums are later verified themselves against a known up to date source. If you're interested in the Gentoo side of it, specifically how it ties into tree-signing, read my gleps: http://www.gentoo.org/proj/en/glep/glep-0057.html http://www.gentoo.org/proj/en/glep/glep-0058.html http://www.gentoo.org/proj/en/glep/glep-0059.html http://www.gentoo.org/proj/en/glep/glep-0060.html http://www.gentoo.org/proj/en/glep/glep-0061.html [1] Cappos, J. et al. A Look In the Mirror: Attacks on Package Managers. (2008). Published in the proceedings of ACM CCS 2008. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgprZuHMj1Wb3.pgp Description: PGP signature
[gentoo-portage-dev] Spam Redux
Somebody subscribed a bad list manager to the list, and caused a mail loop. I removed the offending list address now, but I don't know who did it in the first place. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85
Re: [gentoo-portage-dev] [2/4] proto-GLEPS for Tree-signing
On Tue, Jul 29, 2008 at 08:51:45PM +0100, Mike Auty wrote: In this Glep (xx+1), in the section discussing the procedure for creating a MetaManifest file, in step 3.3, does that include verification of the manifest's signature if it has one? It would seem odd to ignore the signature if it's wrong (I'm not sure about the case if a signature isn't present). I also don't know how this would then be handled (a complete abort, or ignoring the latest changeset to that ebuild?). It doesn't care whatsoever about signatures inside Manifests. That's because there's no difference between a Manifest that isn't signed by a developer, and a Manifest that is developer-signed but any master signature on the developer has been revoked. It's also totally impossible to just block a changeset at the moment like that, even if we had Git. If the signature check happened here, it could also allow for enforcable revocation of developer certificates (once they're revoked, any signed manifests will have the ebuild changes ignored). That may be a lot of work and may take too long, but if not (and depending on our users' trust needs), it might allow them just to check the MetaManifest's signature, and not that of the individual packages. Does that seems sensible? They don't need to check the signatures of the individual packages unless they are really paranoid anyway. You've missed one of the key points of MetaManifest: It defends ONLY the path from the Gentoo infrastructure to the users. P.S, you don't need to CC me. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpoqlSnuVmb5.pgp Description: PGP signature
[gentoo-portage-dev] proto-GLEPS for Tree-signing, take 2
So I'm not going to directly attach the GLEPs again this time, however I am just going to link to them, and summarize the changes: xx+1: - Add mention of how to defeat the mirror replay attacks from [EMAIL PROTECTED] - Clarify wording of the UNCOVERED=ALL-COVERED set math, and why it's important (genone) - Add a timestamp to the metamanifest. - Mention that it can be implemented without the new Manifest2 filetypes. xx+5: - Update the exclusion lists. - Exclusion list behavior during strict validation. - Fix typos. prototype/generate-metamanifest.py: - Prototype of the MetaManifest generation. - Doesn't sign yet, but does include the timestamp. - Uses existing Manifest2 types. - See header for existing runtime info - it's quite fast. http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/ I'd like to ask for any comments to be in to me by July 14th 23:59UTC. After that I'd like to post the GLEPs to the gentoo-dev mailing list. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpzYedI9xbfX.pgp Description: PGP signature
[gentoo-portage-dev] [0/4] proto-GLEPS for Tree-signing
Hi folks, it's that time again, time for the proto-gleps on tree-signing. Barring two minor TODO items, I have completed all of the series dealing with distribution issues and Manifest2. The developer issues and gnupg management issues remain, but they don't block the Manifest2 and distribution parts, so I'm not including them here at this time. I have included the changes that genone made in message-id [EMAIL PROTECTED] to all of the dev-portage@ alias. I'm going to put each GLEP into a separate email under this thread. XX - this email 00-proposal-overview 01-distribution-process-security 02-developer-process-security - NOT INCLUDED 03-gnupg-policies-and-handling - NOT INCLUDED 04-manifest2-hashes 05-manifest2-clarifications If you want to view the files in CVS: http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/ -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpzl7zstchX2.pgp Description: PGP signature
[gentoo-portage-dev] [1/4] proto-GLEPS for Tree-signing
an unusual key-trust model: [ http://marc.theaimsgroup.com/?l=gentoo-securitym=105073449619892w=2 ] 2003-04, gentoo-core mailing list, New Digests and Signing -- Attempted Explanation 2003-06, gentoo-core mailing list, A quick guide to GPG and key signing. - This overview was one of the first to help developers see how to use their devs, and was mainly intended for keysigning meetups. 2003-08-09, gentoo-core mailing list, Ebuild signing - status query, with an not very positive response, delayed by Nick Jones (carpaski) getting rooted and a safe cleanup taking a long time to affect. 2003-12-02, gentoo-core mailing list, Report: rsync1.it.gentoo.org compromised 2003-12-03, gentoo-core mailing list, Signing of ebuilds 2003-12-07, gentoo-core mailing list, gpg signing of Manifests, thread includes the first GnuPG signing prototype code, by Robin H. Johnson (robbat2). Andrew Cowie (rac) also produces a proof-of-concept around this time. 2004-03-23, gentoo-dev mailing list, 2004.1 will not include a secure portage - Kurt Lieber (klieber). Signing is nowhere near ready for 2004.1 release, and it is realized that it there is insufficient traction and the problem is very large. Many arguments about the checking and verification side. First warning signs that MD5 might be broken in the near future. [ http://thread.gmane.org/gmane.linux.gentoo.devel/16876 ] 2004-03-25, gentoo-dev mailing list, Redux: 2004.1 will not include a secure portage - Robin H. Johnson (robbat2). Yet another proposal, summarizing the points of the previous thread and this time trying to track the various weaknesses. http://marc.theaimsgroup.com/?l=gentoo-devm=108017986400698w=2 2004-05-31, Gentoo managers meeting, portage team reports that FEATURES=sign is now available, but large questions still exist over verification policies and procedures, as well as handing of keys. [ http://www.gentoo.org/proj/en/devrel/manager-meetings/logs/2004/20040531.txt ] 2005-01-17, gentoo-core mailing list, Global objective for 2005 : portage signing. Thierry Carrez (koon) suggests that more go into tree-signing work. Problems at the time later in the thread show that the upstream gpg-agent is not ready, amongst other minor implementation issues. 2005-02-20, gentoo-dev mailing list, post-LWE 2005 - Brian Harring (ferringb). A discussion on the ongoing lack of signing, and that eclasses and profiles need to be signed as well, but this seems to be hanging on GLEP33 in the meantime. [ http://thread.gmane.org/gmane.linux.gentoo.devel/25556/focus=25596 ] 2005-03-08, gentoo-core mailing list, gpg manifest signing stats. Informal statistics show that 26% of packages in the tree include a signed Manifest. Questions are raised regarding key types, and key policies. 2005-11-16, gentoo-core mailing list, Gentoo key signing practices and official Gentoo keyring. A discussion of key handling and other outstanding issues, also mentioning partial Manifests, as well as a comparision between the signing procedures used in Slackware, Debian and RPM-based distros. 2005-11-19, gentoo-portage-dev mailing list, Manifest signing - Robin H. Johnson (robbat2) follows up the previous -core posting, discussion implementation issues. [ http://thread.gmane.org/gmane.linux.gentoo.portage.devel/1401 ] 2006-05-18, gentoo-dev mailing list, Signing everything, for fun and for profit - Patrick Lauer (bonsaikitten). Later brings up that Manifest2 is needed for getting everything right. [ http://thread.gmane.org/gmane.linux.gentoo.devel/38363 ] 2006-05-19, gentoo-dev mailing list, Re: Signing everything, for fun and for profit - Robin H. Johnson (robbat2). An introduction into some of the OpenPGP standard, with a focus on how it affects file signing, key signing, management of keys, and revocation. [ http://thread.gmane.org/gmane.linux.gentoo.devel/38363/focus=38371 ] Thanks == I'd like to thank Patrick Lauer (bonsaikitten) for prodding me to keep working on the tree-signing project, as well helping with spelling, grammar, research (esp. tracking down every possible vulnerability that has been mentioned in past discussions, and integrating them in this overview). Copyright = Copyright (c) 2006 by Robin Hugh Johnson. This material may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0. vim: tw=72 ts=2 expandtab: pgpb6zOOH22Fv.pgp Description: PGP signature
[gentoo-portage-dev] [2/4] proto-GLEPS for Tree-signing
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+1 Title: Security of distribution of Gentoo software - Infrastructure to User distribution - MetaManifest Version: $Revision: 1.13 $ Last-Modified: $Date: 2008/07/01 07:09:56 $ Author: Robin Hugh Johnson [EMAIL PROTECTED], Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: GLEP44, GLEPxx+5 Created: October 2006 Updated: November 2007, June 2008 Post-History: ... Abstract MetaManifest provides a means of verifiable distribution from Gentoo Infrastructure to a user system, while data is conveyed over completely untrusted networks and system, by extending the Manifest2 specification, and adding a top-level Manifest file, with support for other nested Manifests. == Motivation == As part of a comprehensive security plan, we need a way to prove that something originating from Gentoo as an organization (read Gentoo-owned hardware, run by infrastructure), has not been tampered with. This allows the usage of third-party rsync mirrors, without worrying that they have modified something critical (e.g. eclasses, which are still unsigned). Securing the untrusted distribution is one of the easier tasks in the security plan - in short, all that is required is having a hash of every item in the tree, and signing that hash to prove it came from Gentoo. Ironically we have a hashed and signed distribution (it's just not used by most users, due to it's drawbacks): Our tree snapshot tarballs have hashes and signatures. So now we want to add the same verification to our material that is distributed by rsync. We already provide hashes of subsets of the tree - our Manifests protect individual packages. However metadata, eclasses and profiles are not protected at this time. The directories of packages and distfiles are NOT covered by this, as they are not distributed by rsync. This portion of the tree-signing work provides only the following guarantee: A user can prove that the tree from the Gentoo infrastructure has not been tampered with since leaving the Gentoo infrastructure. No other guarantees, either implicit or explicit are made. = Specification = For lack of a better name, the following solution should be known as the MetaManifest. Those responsible for the name have already been sacked. MetaManifest basically contains hashes of every file in the tree, either directly or indirectly. The direct case applies to ANY file that does not appear in an existing Manifest file (e.g. eclasses, Manifest files themselves). The indirect case is covered by the CONTENTS of existing Manifest files. If the Manifest itself is correct, we know that by tracking the hash of the Manifest, we can be assured that the contents are protected. In the following, the MetaManifest file is a file named 'Manifest', located at the root of a repository. - Procedure for creating the MetaManifest file: - 1. Start at the root of the Gentoo Portage tree (gentoo-x86, although this procedure applies to overlays as well). 2. Initialize two unordered sets: COVERED, ALL. 2.1. 'ALL' will contain every file in the tree. 2.2. 'COVERED' will contain every file that is mentioned in an existing Manifest2. 3. Traverse the tree, depth-first. 3.1. At the top level only, ignore the distfiles and packages directories. 3.2. If the directory contains a Manifest file add it to the ALL set and don't descend any further, otherwise add all files to the ALL set 3.3. If a directory contains a Manifest file, extract all relevant local files from it (presently: AUX, MISC, EBUILD; but should follow the evolution of Manifest2 entry types per [GLEPxx+5]), and place them into the COVERED set. 4. Produce a new set, UNCOVERED, as the set-difference (ALL)-(COVERED). This is every item that is not covered by another Manifest. 5. If an existing MetaManifest file is present, remove it. 6. For each file in UNCOVERED, assign a Manifest2 type, produce the hashes, and add with the filetype to the MetaManifest file. 7. The MetaManifest must ultimately be GnuPG-signed. 7.1. For the initial implementation, the same key as used for snapshot tarball signing is sufficient. 7.2. For the future, the key used for fully automated signing by infra should not be on the same keyring as developer keys. See [GLEPxx+3 for further notes]. The above does not conflict the proposal contained in GLEP33, which restructure eclasses to include subdirectories and Manifest files, as the Manifest rules above still provide indirect verification for all files after the GLEP33 restructuring if it comes to pass. If other Manifests are added (such as per-category, or protecting versioned eclases), the size of the MetaManifest
[gentoo-portage-dev] [3/4] proto-GLEPS for Tree-signing
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+4 Title: Manifest2 hash policies and security implications Version: $Revision: 1.10 $ Last-Modified: $Date: 2008/07/01 07:18:43 $ Author: Robin Hugh Johnson [EMAIL PROTECTED], Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: GLEP44 Created: October 2006 Updated: November 2007, June 2008 Post-History: ... Updates: GLEP44 Abstract While Manifest2 format allows multiple hashes, the question of which checksums should be present, why, and the security implications of such have never been resolved. This GLEP covers all of these issues, and makes recommendations as to how to handle checksums both now, and in future. Motivation == This GLEP is being written as part of the work on signing the Portage tree, but is only tangentially related to the actual signing of Manifests. Checksums present one possible weak point in the overall security of the tree - and a comprehensive security plan is needed. Specification = The bad news First of all, I'd like to cover the bad news in checksum security. A much discussed point, as been the simple question: What is the security of multiple independent checksums on the same data? The most common position (and indeed the one previously held by myself), is that multiple checksums would be an increase in security, but we could not provably quantify the amount of security this added. The really bad news, is that this position is completely and utterly wrong. Many of you will be aghast at this. There is extremely little added security in multiple checksums [J04]. For any set of checksums, the actual strength lies in that of the strongest checksum. How fast can MD5 be broken? --- For a general collision, not a pre-image attack, since the original announcement by Wang et al [W04], the time required to break MD5 has been massively reduced. Originally at 1 hour on a near-supercomputer (IBM P690) and estimated at 64 hours with a Pentium-3 1.7Ghz. This has gone down to less than in two years, to 17 seconds [K06a]! 08/2004 - 1 hour, IBM pSeries 690 (32x 1.7Ghz POWER4+) = 54.4 GHz-Hours 03/2005 - 8 hours, Pentium-M 1.6Ghz = 12.8 Ghz-Hours 11/2005 - 5 hours, Pentium-4 1.7Ghz = 8.5 Ghz-Hours 03/2006 - 1 minute, Pentium-4 3.2Ghz = .05 Ghz-Hours 04/2006 - 17 seconds, Pentium-4 3.2Ghz = .01 Ghz-Hours If we accept a factor of 800x as a sample of how much faster a checksum may be broken over the course of 2 years (MD5 using the above data is 2000x), then existing checksums do not stand a significant chance of survival in the future. We should thus accept that whatever checksums we are using today, will be broken in the near future, and plan as best as possible. (A brief review [H04] of the present SHA1 attacks indicates an improvement of ~600x in the same timespan). And for those that claim implementation of these procedures is not yet feasible, see [K06b] for an application that can produce two self-extracting .exe files, with identical MD5s, and whatever payload you want. The good news - Of the checksums presently used by Manifest2, one stands close to being completely broken: SHA1. The SHA2 series has suffered some attacks, but still remains reasonably solid [G07],[K08]. No attacks against RIPEMD160 have been published, however it is constructed in the same manner as MD5, SHA1 and SHA2, so is also vulnerable to the new methods of cryptanalysis [H04]. To reduce the potential for future problems and any single checksum break leading to a rapid decrease in security, we should incorporate the strongest hash available from each family of checksums, and be prepared to retire old checksums actively, unless there is a overriding reason to keep a specific checksum. What should be done --- Portage should always try to verify all supported hashes that are available in a Manifest2, starting with the strongest ones as maintained by a preference list. Over time, the weaker checksums should be removed from Manifest2 files, once all old Portage installations have had sufficient time to upgrade. We should be prepared to add stronger checksums wherever possible, and to remove those that have been defeated. An unsupported hash is not considered to be a failure unless no supported hashes are available. Checksum depreciation ~ For the current Portage, SHA1 should be gradually removed, as presents no advantages over SHA256. Beyond one specific problem (see the next paragraph), we should add SHA512 (SHA2, 512 bit size), the Whirlpool checksum (standardized checksum, with no known weaknesses). In future, as stream-based checksums are developed (in response to the development by NIST [AHS]), they should be considered and used. There is one temporary stumbling block at hand - the existing Portage
[gentoo-portage-dev] [4/4] proto-GLEPS for Tree-signing
Attached. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 GLEP: xx+5 Title: Manifest2 filetypes Version: $Revision: 1.15 $ Last-Modified: $Date: 2008/07/01 08:52:34 $ Author: Robin Hugh Johnson [EMAIL PROTECTED] Status: Draft Type: Standards Track Content-Type: text/x-rst Requires: GLEP44 Created: November 2007 Updated: June 2008 Post-History: ... Updates: GLEP44 Abstract Clarification of the Manifest2 [GLEP44] specification, including new types to help in the tree-signing specification. Motivation == [GLEP44] was not entirely clear on the usage of filetype specifiers. This document serves to provide some of the internal logic used by Portage at the point of writing, as well as adding new types to cover the rest of the tree, for the purposes of tree-signing coverage. Specification = General --- For any given directory with a Manifest file, every file located in that directory, or a sub-directory must be listed in that Manifest file, unless stated otherwise in the following sections. The Manifest file must not contain an entry for itself. Excluded files -- When generating or validating a Manifest, or commiting to a version control system, the package manager should endeavour to ignore files created by a version control system, backup files from text editors. A non-exhaustive list is suggested here: .cvs/, .svn/, .git/, .hg/, .#*, *.rej, *.orig, *.bak, *~. Additionally, for a transitional Manifest1-Manifest2 system, old-style digest files located in a 'files/' directory, may be excluded from Manifest2 generation, or included with a type of MISC. Existing filetypes: --- AUX ~~~ - The AUX type is used for all items under the 'files' subdirectory. - They should be verified relative to $FILESDIR. - The string 'files/' is left out of the Manifest line. - The absence of a file mentioned by AUX must be treated as an error. - The AUX type is intended to denote potentially executable content (either directly or indirectly), that must be treated an error if modified or absent. EBUILD ~~ - The EBUILD type is used solely for files ending in .ebuild, or other suffixes as defined by the EAPI. - The files are located in the same directory as the Manifest file. - The modification or absence of a file mentioned by EBUILD must be treated as an error. DIST - The DIST type is used for distfiles - They may be found directly via the $DISTDIR setting of the package manager. - During simple verification of a Manifest, a missing DIST file should not be consider as a validation error (it is however a failure to fetch or unpack). MISC - The MISC type covers all remaining files in a directory. - MISC is intended to mark all content that was not used in some way that directly affected execution of the package manager. - This includes metadata.xml and ChangeLog entries, and any other purely informational content. - MISC entries where the file is missing may optionally be ignored as by non-strict package managers. - It should be possible to install a package while all MISC entries have been deleted from the tree. New filetypes: -- _INFO (new, abstract) ~ - This is the functionality of the old AUX, but does not include the implicit 'files/' prefix in the path, and is verified relative to the working directory instead of $FILESDIR. - The modification or absence of a file listed as a _INFO-derived type is not an error unless the package manager is attempting to be strict. _CRIT (new, abstract) ~ - _CRIT is based off the _INFO type. - The modification or absence of a file listed as a _CRIT-derived type must be treated as an error. EBUILD ~~ - Now derived from _CRIT. - Otherwise unchanged. DIST - Now derived from _CRIT. - Otherwise unchanged. MISC - Now derived from _INFO. - Otherwise unchanged. MANIFEST (new) ~~ - The MANIFEST type is explicitly to cover all nested Manifest files. - During validation, this serves as an indicator that the package manager may need to check subtree Manifest file. - A missing MANIFEST file may be treated as a minor (eg excluding an entire category) or critical validation failure. - The failure should be considered as critical only if files that would be directly covered by this Manifest are missing. Deletion of a category-level Manifest while preserving the packages is forbidden. Deletion of an entire category is not. ECLASS (new) - uses _CRIT. - This type shall be used for all eclasses only. - TODO: What about patches etc under eclasses/? Probably EXEC? DATA (new) ~~ - uses _CRIT. - The DATA type shall be used for all files that directly affect the package manager, such as metadata/cache/* and profiles/. EXEC (new) ~ - uses _CRIT. - If the file gets
Re: [gentoo-portage-dev] state of GPG-signing in portage
On Sat, Jan 12, 2008 at 05:49:10AM +0100, Marcel Meyer wrote: I'm wondering if the GPG-signing feature within portage is already useable (if I recall correctly it was startet 2004 or 2005?). If yes, how can I use it correctly and where to get the gpg-key securely? The URLs I found by googling do no longer exist :-( You can read the conceptual stuff in progress here: http://sources.gentoo.org/viewcvs.py/gentoo/users/robbat2/tree-signing-gleps/ I need to find time to integrate the last round of comments into it, and then several of them will be getting posted as proposal GLEPs for the mandatory period. -- Robin Hugh Johnson Gentoo Linux Developer Infra Guy E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgpxnPUXVpz3F.pgp Description: PGP signature
Re: [gentoo-portage-dev] Improvement suggestion for emerge: Not using a new connection for every file
On Sat, Feb 24, 2007 at 10:00:29PM +0100, Beginner wrote: I recommend not to use wget and not to reconnect to the server for every single packet, but to hold the connection therefore spare traffic and download more fast. If you are doing lots of downloads, use 'emerge -pvf FOO' and feed each line of that output to whatever you want to do your fetching. On that, I haven't kept up with the code in recent years, is there a way that portage itself can hand off those entire lines to a fetching application, instead of putting them in one by one? (Telling the app about the expected size and checksums would be handy too). -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgptiKi5nS5d9.pgp Description: PGP signature
Re: [gentoo-portage-dev] Masked by corruption
On Fri, Jan 12, 2007 at 12:18:34AM +0200, Philipp Riegger wrote: On 02.01.2007, at 06:56, Zac Medico wrote: In =portage-2.1.2_rc4-r2 t does that now for installed package (see bug #158931). For /var/cache/edb/dep the sqlite module is available (requires pysqlite or python-2.5 with sqlite support enabled). Where can i find documentation about this? I use metadata-transfer at the moment, but all info i got was from the examples-section in man emerge and from the forum. Is there some official complete list of theese modules with some description? The portage can probably answer better than I can, but here's a quick hack. grep 'class database' -rl /usr/lib/portage/pym/cache/ -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : [EMAIL PROTECTED] GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 pgplwrZfNf3z7.pgp Description: PGP signature
