Re: [gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

2011-10-01 Thread Zac Medico 
On 10/01/2011 02:41 AM, Brian Harring wrote:
> On Sat, Oct 01, 2011 at 07:40:52AM +, Robin H. Johnson wrote:
>> From: "Robin H. Johnson" 
>>
>> Provide public-domain implementation of the Whirlpool hash algorithm to
>> be used as new Manifest2 hash.
>>
>> Signed-off-by: Robin H. Johnson 
>> ---
>>  pym/portage/checksum.py |8 ++--
>>  1 files changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
>> index e5455fa..3593686 100644
>> --- a/pym/portage/checksum.py
>> +++ b/pym/portage/checksum.py
>> @@ -71,6 +71,10 @@ except ImportError:
>>  
>>  sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
>>  
>> +# Bundled WHIRLPOOL implementation
>> +from portage.util.whirlpool import new as _new_whirlpool
>> +whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, 
>> origin="bundled")
>> +
> 
> Likely should shift this to a trailing check if no whirlpool 
> implementation was found; via this, we can avoid the import unless 
> it's needed.
> ~brian

Thanks, that's done now:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=06ad8911b5790a2ed963fe1b981751ab0a2be8d5

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-01 Thread Zac Medico 
On 10/01/2011 09:40 PM, Zac Medico wrote:
> On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
>> diff --git a/pym/portage/const.py b/pym/portage/const.py
>> index 8b5f4ac..a42ebe8 100644
>> --- a/pym/portage/const.py
>> +++ b/pym/portage/const.py
>> @@ -109,10 +109,12 @@ EAPI = 4
>>  
>>  HASHING_BLOCKSIZE= 32768
>>  MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
>> -MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
>> +MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
>> +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
>>  
>>  MANIFEST1_REQUIRED_HASH  = "MD5"
>> -MANIFEST2_REQUIRED_HASH  = "SHA1"
>> +MANIFEST2_REQUIRED_HASH  = "SHA256" 
>> +# FUTURE: Change to WHIRLPOOL after 2012/10/01
>>  
>>  MANIFEST2_IDENTIFIERS= ("AUX", "MISC", "DIST", "EBUILD")
>>  # 
>> ===
> 
> If we control these hashes via metadata/layout.conf, then we can toggle
> it atomically for all commiters. Otherwise, we'll have an annoying
> period of time where different committers are committing different sets
> of hashes, depending on their portage version.

I've applied the whole series, except for 5/5:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f27473d04e6dee44983d1e5ac32ea9d4d375b5a2
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=f3b05d6eed63e19cdfa7f645cf0190ee8019dd90
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=8ac29097395f24ad331602d8e87fdf105ebd972b
http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=faf87ba9877e3b5a7866c6649f956f15950e789a

-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-01 Thread Zac Medico 
On 10/01/2011 12:40 AM, Robin H. Johnson wrote:
> diff --git a/pym/portage/const.py b/pym/portage/const.py
> index 8b5f4ac..a42ebe8 100644
> --- a/pym/portage/const.py
> +++ b/pym/portage/const.py
> @@ -109,10 +109,12 @@ EAPI = 4
>  
>  HASHING_BLOCKSIZE= 32768
>  MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
> -MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
> +MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
> +# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
>  
>  MANIFEST1_REQUIRED_HASH  = "MD5"
> -MANIFEST2_REQUIRED_HASH  = "SHA1"
> +MANIFEST2_REQUIRED_HASH  = "SHA256" 
> +# FUTURE: Change to WHIRLPOOL after 2012/10/01
>  
>  MANIFEST2_IDENTIFIERS= ("AUX", "MISC", "DIST", "EBUILD")
>  # ===

If we control these hashes via metadata/layout.conf, then we can toggle
it atomically for all commiters. Otherwise, we'll have an annoying
period of time where different committers are committing different sets
of hashes, depending on their portage version.
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 1/4] Manifest2 hash: Whirlpool

2011-10-01 Thread Robin H. Johnson 
On Sat, Oct 01, 2011 at 02:08:57PM -0400, Mike Frysinger wrote:
> On Thursday, September 29, 2011 21:27:39 Robin H. Johnson wrote:
> > Provide public-domain implementation of the Whirlpool hash algorithm to
> > be used as new Manifest2 hash.
> > 
> > Signed-off-by: Robin H. Johnson 
> > ---
> >  pym/portage/checksum.py   |4 +
> >  pym/portage/util/whirlpool.py |  788
> > + 2 files changed, 792
> > insertions(+), 0 deletions(-)
> >  create mode 100644 pym/portage/util/whirlpool.py
> 
> shouldn't we add pycryptoplus to the tree and depend on that rather than 
> copying & pasting their code into portage ?
It looks like they got it from the same source [1] that I did, and
pycryptoplus was written after I wrote the earliest version of this
patch. The source I got it from does NOT package it up in any way.

1. http://www.bjrn.se/code/whirlpoolpy.txt

-- 
Robin Hugh Johnson
Gentoo Linux: Developer, Trustee & Infrastructure Lead
E-Mail : robb...@gentoo.org
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

Re: [gentoo-portage-dev] [PATCH 1/4] Manifest2 hash: Whirlpool

2011-10-01 Thread Alec Warner 
On Sat, Oct 1, 2011 at 11:08 AM, Mike Frysinger  wrote:
> On Thursday, September 29, 2011 21:27:39 Robin H. Johnson wrote:
>> Provide public-domain implementation of the Whirlpool hash algorithm to
>> be used as new Manifest2 hash.
>>
>> Signed-off-by: Robin H. Johnson 
>> ---
>>  pym/portage/checksum.py       |    4 +
>>  pym/portage/util/whirlpool.py |  788
>> + 2 files changed, 792
>> insertions(+), 0 deletions(-)
>>  create mode 100644 pym/portage/util/whirlpool.py
>
> shouldn't we add pycryptoplus to the tree and depend on that rather than
> copying & pasting their code into portage ?
> -mike

It is ironic with all the gentoo devs complaining about bundled libs.
I think part of the problem is the maintenance issue (upgrades from
EAPI0, extra deps, etc..)

>
>

Re: [gentoo-portage-dev] [PATCH 1/4] Manifest2 hash: Whirlpool

2011-10-01 Thread Mike Frysinger 
On Thursday, September 29, 2011 21:27:39 Robin H. Johnson wrote:
> Provide public-domain implementation of the Whirlpool hash algorithm to
> be used as new Manifest2 hash.
> 
> Signed-off-by: Robin H. Johnson 
> ---
>  pym/portage/checksum.py   |4 +
>  pym/portage/util/whirlpool.py |  788
> + 2 files changed, 792
> insertions(+), 0 deletions(-)
>  create mode 100644 pym/portage/util/whirlpool.py

shouldn't we add pycryptoplus to the tree and depend on that rather than 
copying & pasting their code into portage ?
-mike

Re: [gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

2011-10-01 Thread Brian Harring 
On Sat, Oct 01, 2011 at 07:40:52AM +, Robin H. Johnson wrote:
> From: "Robin H. Johnson" 
> 
> Provide public-domain implementation of the Whirlpool hash algorithm to
> be used as new Manifest2 hash.
> 
> Signed-off-by: Robin H. Johnson 
> ---
>  pym/portage/checksum.py |8 ++--
>  1 files changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> index e5455fa..3593686 100644
> --- a/pym/portage/checksum.py
> +++ b/pym/portage/checksum.py
> @@ -71,6 +71,10 @@ except ImportError:
>  
>  sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
>  
> +# Bundled WHIRLPOOL implementation
> +from portage.util.whirlpool import new as _new_whirlpool
> +whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, 
> origin="bundled")
> +

Likely should shift this to a trailing check if no whirlpool 
implementation was found; via this, we can avoid the import unless 
it's needed.
~brian

>  # Use pycrypto when available, prefer it over the internal fallbacks
>  try:
>   from Crypto.Hash import SHA256, RIPEMD
> @@ -80,14 +84,14 @@ except ImportError as e:
>   pass
>  
>  # Use hashlib from python-2.5 if available and prefer it over pycrypto and 
> internal fallbacks.
> -# Need special handling for RMD160 as it may not always be provided by 
> hashlib.
> +# Need special handling for RMD160/WHIRLPOOL as they may not always be 
> provided by hashlib.
>  try:
>   import hashlib, functools
>   
>   md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
>   sha1hash = _generate_hash_function("SHA1", hashlib.sha1, 
> origin="hashlib")
>   sha256hash = _generate_hash_function("SHA256", hashlib.sha256, 
> origin="hashlib")
> - for local_name, hash_name in (("rmd160", "ripemd160"), ):
> + for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", 
> "whirlpool")):
>   try:
>   hashlib.new(hash_name)
>   except ValueError:
> -- 
> 1.7.7
>

[gentoo-portage-dev] [GLEP59v2 1/5] Refactor RMD160 hashlib code for less-hardcoding

2011-10-01 Thread Robin H. Johnson 
From: "Robin H. Johnson" 

To be used shortly for WHIRLPOOL as well as RMD160.

Signed-off-by: Robin H. Johnson 
---
 pym/portage/checksum.py |   21 -
 1 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 9e7e455..e5455fa 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -82,19 +82,22 @@ except ImportError as e:
 # Use hashlib from python-2.5 if available and prefer it over pycrypto and 
internal fallbacks.
 # Need special handling for RMD160 as it may not always be provided by hashlib.
 try:
-   import hashlib
+   import hashlib, functools

md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
sha1hash = _generate_hash_function("SHA1", hashlib.sha1, 
origin="hashlib")
sha256hash = _generate_hash_function("SHA256", hashlib.sha256, 
origin="hashlib")
-   try:
-   hashlib.new('ripemd160')
-   except ValueError:
-   pass
-   else:
-   def rmd160():
-   return hashlib.new('ripemd160')
-   rmd160hash = _generate_hash_function("RMD160", rmd160, 
origin="hashlib")
+   for local_name, hash_name in (("rmd160", "ripemd160"), ):
+   try:
+   hashlib.new(hash_name)
+   except ValueError:
+   pass
+   else:
+   globals()['%shash' % local_name] = \
+   _generate_hash_function(local_name.upper(), \
+   functools.partial(hashlib.new, hash_name), \
+   origin='hashlib')
+
 except ImportError as e:
pass

-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 5/5] GLEP59: Change live Manifest2 hashes to SHA256, SHA512, WHIRLPOOL

2011-10-01 Thread Robin H. Johnson 
From: "Robin H. Johnson" 

Change Manifest2 hashes to a more secure set as approved in GLEP59.
SHA512 and WHIRLPOOL are added, SHA1 and RMD160 are dropped.

SHA256 is now the lowest security hash, and must remain in Manifest
files for at least 1 year, otherwise older Portage installs will
complain that they do not support any of the hashes in the Manifest
files.

Future events:
After 2012/10/01:
- Change MANIFEST2_REQUIRED_HASH to WHIRLPOOL.
- Remove SHA256 from MANIFEST2_HASH_FUNCTIONS.
After SHA-3 is approved:
- Add new hashes to MANIFEST2_HASH_FUNCTIONS.

Signed-off-by: Robin H. Johnson 
---
 pym/portage/const.py |6 --
 1 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/pym/portage/const.py b/pym/portage/const.py
index 8b5f4ac..a42ebe8 100644
--- a/pym/portage/const.py
+++ b/pym/portage/const.py
@@ -109,10 +109,12 @@ EAPI = 4
 
 HASHING_BLOCKSIZE= 32768
 MANIFEST1_HASH_FUNCTIONS = ("MD5", "SHA256", "RMD160")
-MANIFEST2_HASH_FUNCTIONS = ("SHA1", "SHA256", "RMD160")
+MANIFEST2_HASH_FUNCTIONS = ("SHA256", "SHA512", "WHIRLPOOL") 
+# FUTURE: Add SHA-3 when available; remove SHA256 after 2012/10/01
 
 MANIFEST1_REQUIRED_HASH  = "MD5"
-MANIFEST2_REQUIRED_HASH  = "SHA1"
+MANIFEST2_REQUIRED_HASH  = "SHA256" 
+# FUTURE: Change to WHIRLPOOL after 2012/10/01
 
 MANIFEST2_IDENTIFIERS= ("AUX", "MISC", "DIST", "EBUILD")
 # ===
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 0/5] GLEP59: Manifest2 hash types

2011-10-01 Thread Robin H. Johnson 
Respun now with the help of ferringb. Cleans up the implementation and catches
a few bug and improvements:
- mhash priority moved lower than pycrypto/hashlib because mhash holds GIL
  while the other implementations don't.
- hashlib does offer whirlpool if it was built against openssl 1.0.

1/5: Refactor RMD160 hashlib code for less-hardcoding
2/5: Manifest2 hash: Whirlpool
3/5: Manifest2 hash: SHA512
4/5: Manifest2 hash backend provider: mhash
5/5: GLEP59: Change live Manifest2 hashes to SHA256,

[gentoo-portage-dev] [GLEP59v2 2/5] Manifest2 hash: Whirlpool

2011-10-01 Thread Robin H. Johnson 
From: "Robin H. Johnson" 

Provide public-domain implementation of the Whirlpool hash algorithm to
be used as new Manifest2 hash.

Signed-off-by: Robin H. Johnson 
---
 pym/portage/checksum.py |8 ++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index e5455fa..3593686 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -71,6 +71,10 @@ except ImportError:
 
 sha1hash = _generate_hash_function("SHA1", _new_sha1, origin="internal")
 
+# Bundled WHIRLPOOL implementation
+from portage.util.whirlpool import new as _new_whirlpool
+whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, 
origin="bundled")
+
 # Use pycrypto when available, prefer it over the internal fallbacks
 try:
from Crypto.Hash import SHA256, RIPEMD
@@ -80,14 +84,14 @@ except ImportError as e:
pass
 
 # Use hashlib from python-2.5 if available and prefer it over pycrypto and 
internal fallbacks.
-# Need special handling for RMD160 as it may not always be provided by hashlib.
+# Need special handling for RMD160/WHIRLPOOL as they may not always be 
provided by hashlib.
 try:
import hashlib, functools

md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
sha1hash = _generate_hash_function("SHA1", hashlib.sha1, 
origin="hashlib")
sha256hash = _generate_hash_function("SHA256", hashlib.sha256, 
origin="hashlib")
-   for local_name, hash_name in (("rmd160", "ripemd160"), ):
+   for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", 
"whirlpool")):
try:
hashlib.new(hash_name)
except ValueError:
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 4/5] Manifest2 hash backend provider: mhash

2011-10-01 Thread Robin H. Johnson 
From: "Robin H. Johnson" 

Offer mhash as a provider for Manifest2 hash generation and validation.
This is important as either of pycrypto or fchksum offer an accelerated
Whirlpool implementation, and hashlib might not offer it. Additionally,
the mhash implementation is accelerated and ships with a rigorious
testsuite.

Signed-off-by: Robin H. Johnson 
---
 pym/portage/checksum.py |   19 +++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 40ae836..c0c7c04 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -75,6 +75,25 @@ sha1hash = _generate_hash_function("SHA1", _new_sha1, 
origin="internal")
 from portage.util.whirlpool import new as _new_whirlpool
 whirlpoolhash = _generate_hash_function("WHIRLPOOL", _new_whirlpool, 
origin="bundled")
 
+# Try to use mhash if available
+# mhash causes GIL presently, so it gets less priority than hashlib and
+# pycrypto. However, it might be the only accelerated implementation of
+# WHIRLPOOL available.
+try:
+   import mhash, functools
+   md5hash = _generate_hash_function("MD5", functools.partial(mhash.MHASH, 
mhash.MHASH_MD5), origin="mhash")
+   sha1hash = _generate_hash_function("SHA1", 
functools.partial(mhash.MHASH, mhash.MHASH_SHA1), origin="mhash")
+   sha256hash = _generate_hash_function("SHA256", 
functools.partial(mhash.MHASH, mhash.MHASH_SHA256), origin="mhash")
+   sha512hash = _generate_hash_function("SHA512", 
functools.partial(mhash.MHASH, mhash.MHASH_SHA512), origin="mhash")
+   for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", 
"whirlpool")):
+   if hasattr(mhash, 'MHASH_%s' % local_name.upper()):
+   globals()['%shash' % local_name] = \
+   _generate_hash_function(local_name.upper(), \
+   functools.partial(mhash.MHASH, getattr(mhash, 
'MHASH_%s' % s.upper())), \
+   origin='mhash')
+except ImportError as e:
+   pass
+
 # Use pycrypto when available, prefer it over the internal fallbacks
 try:
from Crypto.Hash import SHA256, RIPEMD
-- 
1.7.7

[gentoo-portage-dev] [GLEP59v2 3/5] Manifest2 hash: SHA512

2011-10-01 Thread Robin H. Johnson 
From: "Robin H. Johnson" 

Provide SHA512 hash algorithm to be used as new Manifest2 hash.

Signed-off-by: Robin H. Johnson 
---
 pym/portage/checksum.py |1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index 3593686..40ae836 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -91,6 +91,7 @@ try:
md5hash = _generate_hash_function("MD5", hashlib.md5, origin="hashlib")
sha1hash = _generate_hash_function("SHA1", hashlib.sha1, 
origin="hashlib")
sha256hash = _generate_hash_function("SHA256", hashlib.sha256, 
origin="hashlib")
+   sha512hash = _generate_hash_function("SHA512", hashlib.sha512, 
origin="hashlib")
for local_name, hash_name in (("rmd160", "ripemd160"), ("whirlpool", 
"whirlpool")):
try:
hashlib.new(hash_name)
-- 
1.7.7