[gentoo-dev] Re: Manifest2 hashes, take n+1-th

[Duncan]

Michał Górny posted on Sat, 21 Oct 2017 01:39:55 +0200 as excerpted:

> W dniu pią, 20.10.2017 o godzinie 18∶42 -0400, użytkownik Anton Molyboha
> napisał:

>> Would it make sense then to support several hashes but let the user
>> optionally turn off the verification of some of them, depending on the
>> user's security vs performance requirements?
>> 
> I won't block anyone from implementing such an option but I won't spend
> my time on it either. However, if you believe verifying two checksums
> could be a problem, then I have serious doubts if you hardware is
> capable of building anything.

When does this verification happen?

If it's during --sync or --pretend/ask, as I believe it is based on when 
I get errors if I edit and forget to manifest/digest, then arguably time 
matters rather more than it does if it's only after the user has OKed the 
merge and it's doing the build.

Because the time before the PM tells me what it's going to do and asks my 
OK before doing it is time I'm generally actually waiting for it (tho I'm 
normally doing something else while waiting, but I /am/ waiting) to 
decide whether I want to go ahead, or perhaps I need to change something 
first, while the actual build time after I've OKed it, doesn't matter so 
much, because I'm not actually waiting on it, I'm doing other things, 
which can actually include turning in for the night or going to work, 
with the intent being that it'll be done when I get back to it.

So the hash verification time really does matter, even if it's minutes 
compared to hours of actual build time, because that's time I'm actively 
waiting for it, vs. letting it do its thing in the background, with much 
less concern about how long (within reason) it might take.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Jason A. Donenfeld]

Blake2 is in coreutils already, provides an excellent security margin, and
is considerably faster than both sha2 and sha3.

On Oct 19, 2017 21:09, "Michał Górny"  wrote:

> Hi, everyone.
>
> The previous discussion on Manifest2 hashes pretty much died away
> pending fixes to Portage. Since Portage was fixed a while ago, and we
> can now safely switch, I'd like to reboot the discussion before
> submitting the item for the next Council meeting.
>
> Considering all arguments made so far, I'd like to propose changing:
>
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
>
> to:
>
>   manifest-hashes = SHA512 SHA3_512
>
> In other words, removing SHA256 and WHIRLPOOL, and adding SHA3_512.
>
>
> Rationale
> -
>
> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.
>
> 2. For the above to hold, the hashes should be diverse. SHA256
> and SHA512 are the same algorithm, so a weakness discovered in either
> would probably apply to both -- keeping both does not make sense at all.
> Furthermore, both SHA2 and WHIRLPOOL use the same construct (MD), so
> a weakness in the construct would apply to both.
>
> 3. Keeping one of the three old hashes is necessary for compatibility
> reasons. Furthermore, the current versions of Portage consider SHA512
> obligatory, so we can't remove it without redesigning Portage first
> (though I think this applies only to developer installs, i.e. those
> creating Manifests).
>
> 4. The new hashes that are stronger and commonly available are
> SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> our current algorithms, so either is a good candidate. The choice of
> Keccak is purely arbitrary (because it's the winner?).
>
> All the above considered, I think it's most reasonable to use two hashes
> with diverse constructs. SHA512 needs to be one of them, for
> compatibility reasons. The other could be either SHA3_512 or BLAKE2B,
> as a strong, future-proof hash. SHA3 is probably a better choice because
> it's going to have more support as the official recommendation.
>
> --
> Best regards,
> Michał Górny
>
>
>

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

On Fri, Oct 20, 2017 at 8:04 AM, Kristian Fiskerstrand  wrote:
> On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
>>
>> I support Hanno's suggestion of doing just SHA512, but would be
>> interested in hearing opinions from others who have apparent
>> security/crypto experience. Maybe the Security project can weigh the
>> suggestions as well?
>>
>
> The whole discussion is moot so long as we don't have OpenPGP signed
> gentoo repository in rsync.
>
> SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
> considerably slower for some architectures. Introducing a non-optimized
> keccak on top of it will have a significant negative performance impact
> for these arches without much security gain.
>
> if we still want two separate hashes, the choice of sha2 and sha3
> compination is a good one given they are based on separate constructs.
>
> But IMHO we should start where things matter and complete an
> implementation for OpenPGP signatures of MetaManifests in Portage.
>

This is why I use webrsync-gpg. Git commits are supposed to be
GPG-signed, so that may be suitable for your purposes.

Cheers,
 R0b0t1.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

Hello,

I missed some messages in the time I wrote my reply. This also touches
on some of the points in Mr. Górny's other message about time.

On Fri, Oct 20, 2017 at 6:38 PM, Michał Górny  wrote:
> W dniu pią, 20.10.2017 o godzinie 00∶20 +0200, użytkownik Francesco
> Riosa napisał:
>> 2017-10-19 23:00 GMT+02:00 Michał Górny :
>>
>> > W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
>> > napisał:
>> > >
>> > > 4. The new hashes that are stronger and commonly available are
>> > > SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
>> > > our current algorithms, so either is a good candidate. The choice of
>> > > Keccak is purely arbitrary (because it's the winner?).
>> > >
>> >
>> > Actually, a small correction here: we support more implementations
>> > of SHA3 than BLAKE2, so the first one is less problematic for us.
>> >
>>
>> Not researched in depth but:
>> B2sum provided by coreutils is quite satisfacting*, it's pretty fast, while
>> sha-3 is deemed to be slower than sha-2, maybe this could be weighted while
>> choosing the algorithm wanted.
>>
>> Both seem to take advantage of modern multicore CPUs but sha-3 does 11.7
>> [cpb]#0 (see #1) while an email seen on the internet say blake2 can reach 1
>> [cpb] (see #2)
>>
>> #0 cpb = cpu cycles per byte
>> #1 https://en.wikipedia.org/wiki/SHA-3#Speed
>> #2 http://www.metzdowd.com/pipermail/cryptography/2016-May/029297.html
>> * (in my limited experience)
>
> I've taken a closer look at BLAKE2 possibilities, and it seems that it's
> going to be our choice after all. I'm adding dev-python/pyblake2
> as a fallback implementation now.
>
> Curious enough, after disabling the 'optimized' SSE2 assembly, the plain
> register implementation is 2.5 times faster than the SSE2 implementation
> that is used by default, and two times faster than the built-in SHA2
> implementation in Python.
>

It is likely that the register implementation makes better use of the
data and instruction cache and processor instruction pipeline. For a
similar reason, functions with larger block sizes tend to run more
slowly over the same amount of data than their counterparts with
smaller block sizes.

If speed truly is crucial then it may be a better idea to use one very
strong hash function and two weaker but faster hash functions. This is
why I recommended RIPEMD160. If BLAKE2B is used, it may be possible to
switch SHA512 for SHA256.

It seems important to me to use three hash functions.

Again, though, I think it needs to be pointed out that on slower
machines the hash time is on the order of tens of seconds. This should
be negligible compared to the build time.

Cheers,
 R0b0t1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 18∶42 -0400, użytkownik Anton Molyboha
napisał:
> On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey  wrote:
> 
> > On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck  wrote:
> > 
> > > On Thu, 19 Oct 2017 21:08:40 +0200
> > > Michał Górny  wrote:
> > > 
> > > >   manifest-hashes = SHA512 SHA3_512
> > > 
> > > Counterproposal: Just use SHA512.
> > > 
> > > There isn't any evidence that any SHA2-based hash algorithm is going to
> > > be broken any time soon. If that changes there will very likely be
> > > decades of warning before a break becomes practical.
> > > 
> > > Having just one hash is simpler and using a well supported one like
> > > SHA512 may make things easier than using something that's still not
> > > very widely supported.
> > 
> > 
> > Yet having more than one lets you match make sure nobody hijacked your
> > manifest file when an attack vector is inevitably discovered for the old
> > new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> > confirm the file is the same one that matched the old checksum in addition
> > to the new one.
> > 
> 
> Would it make sense then to support several hashes but let the user
> optionally turn off the verification of some of them, depending on the
> user's security vs performance requirements?
> 

I won't block anyone from implementing such an option but I won't spend
my time on it either. However, if you believe verifying two checksums
could be a problem, then I have serious doubts if you hardware is
capable of building anything.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 00∶20 +0200, użytkownik Francesco
Riosa napisał:
> 2017-10-19 23:00 GMT+02:00 Michał Górny :
> 
> > W dniu czw, 19.10.2017 o godzinie 21∶08 +0200, użytkownik Michał Górny
> > napisał:
> > > 
> > > 4. The new hashes that are stronger and commonly available are
> > > SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> > > our current algorithms, so either is a good candidate. The choice of
> > > Keccak is purely arbitrary (because it's the winner?).
> > > 
> > 
> > Actually, a small correction here: we support more implementations
> > of SHA3 than BLAKE2, so the first one is less problematic for us.
> > 
> 
> Not researched in depth but:
> B2sum provided by coreutils is quite satisfacting*, it's pretty fast, while
> sha-3 is deemed to be slower than sha-2, maybe this could be weighted while
> choosing the algorithm wanted.
> 
> Both seem to take advantage of modern multicore CPUs but sha-3 does 11.7
> [cpb]#0 (see #1) while an email seen on the internet say blake2 can reach 1
> [cpb] (see #2)
> 
> #0 cpb = cpu cycles per byte
> #1 https://en.wikipedia.org/wiki/SHA-3#Speed
> #2 http://www.metzdowd.com/pipermail/cryptography/2016-May/029297.html
> * (in my limited experience)

I've taken a closer look at BLAKE2 possibilities, and it seems that it's
going to be our choice after all. I'm adding dev-python/pyblake2
as a fallback implementation now.

Curious enough, after disabling the 'optimized' SSE2 assembly, the plain
register implementation is 2.5 times faster than the SSE2 implementation
that is used by default, and two times faster than the built-in SHA2
implementation in Python.

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Gordon Pettey]

On Fri, Oct 20, 2017 at 5:42 PM, Anton Molyboha  wrote:

> On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey 
> wrote:
>
>> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck  wrote:
>>
>>> On Thu, 19 Oct 2017 21:08:40 +0200
>>> Michał Górny  wrote:
>>>
>>> >   manifest-hashes = SHA512 SHA3_512
>>>
>>> Counterproposal: Just use SHA512.
>>>
>>> There isn't any evidence that any SHA2-based hash algorithm is going to
>>> be broken any time soon. If that changes there will very likely be
>>> decades of warning before a break becomes practical.
>>>
>>> Having just one hash is simpler and using a well supported one like
>>> SHA512 may make things easier than using something that's still not
>>> very widely supported.
>>
>>
>> Yet having more than one lets you match make sure nobody hijacked your
>> manifest file when an attack vector is inevitably discovered for the old
>> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
>> confirm the file is the same one that matched the old checksum in addition
>> to the new one.
>>
>
> Would it make sense then to support several hashes but let the user
> optionally turn off the verification of some of them, depending on the
> user's security vs performance requirements?
>
I would strongly question whether anybody is actually running emerge (or
whatever command that would be using the manifests) on systems that don't
have the CPU power to check a few hashes. If the CPU is really that weak,
there are likely much more important issues to deal with than what
combination of hashing algorithms manifests use. Things like "I should be
using pre-built system images because my CPU is orders of magnitude to even
do dependency tree calculation in less than a decade"...

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Anton Molyboha]

On Thu, Oct 19, 2017 at 6:49 PM, Gordon Pettey  wrote:

> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck  wrote:
>
>> On Thu, 19 Oct 2017 21:08:40 +0200
>> Michał Górny  wrote:
>>
>> >   manifest-hashes = SHA512 SHA3_512
>>
>> Counterproposal: Just use SHA512.
>>
>> There isn't any evidence that any SHA2-based hash algorithm is going to
>> be broken any time soon. If that changes there will very likely be
>> decades of warning before a break becomes practical.
>>
>> Having just one hash is simpler and using a well supported one like
>> SHA512 may make things easier than using something that's still not
>> very widely supported.
>
>
> Yet having more than one lets you match make sure nobody hijacked your
> manifest file when an attack vector is inevitably discovered for the old
> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> confirm the file is the same one that matched the old checksum in addition
> to the new one.
>

Would it make sense then to support several hashes but let the user
optionally turn off the verification of some of them, depending on the
user's security vs performance requirements?

-- 
Anton

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[R0b0t1]

Hello,

On Thu, Oct 19, 2017 at 2:08 PM, Michał Górny  wrote:
> Hi, everyone.
>
> The previous discussion on Manifest2 hashes pretty much died away
> pending fixes to Portage. Since Portage was fixed a while ago, and we
> can now safely switch, I'd like to reboot the discussion before
> submitting the item for the next Council meeting.
>
> Considering all arguments made so far, I'd like to propose changing:
>
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
>
> to:
>
>   manifest-hashes = SHA512 SHA3_512
>
> In other words, removing SHA256 and WHIRLPOOL, and adding SHA3_512.
>

In general I do not mind updating the algorithms used, but I do feel
it is important to keep at least three present. Without at least three
(or a larger odd number) it is not possible to break a tie.

That may ultimately be beside the point, as any invalid hashes should
result in the user contacting the developers or doing something else,
but it is hard to know.

>
> Rationale
> -
>
> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.
>
> 2. For the above to hold, the hashes should be diverse. SHA256
> and SHA512 are the same algorithm, so a weakness discovered in either
> would probably apply to both -- keeping both does not make sense at all.
> Furthermore, both SHA2 and WHIRLPOOL use the same construct (MD), so
> a weakness in the construct would apply to both.
>

Algorithms of different block sizes can easily have different
cryptographic properties.

Notable is https://en.wikipedia.org/wiki/Advanced_Encryption_Standard,
see box. More relevant is https://en.wikipedia.org/wiki/SHA-2.

In general only the computational complexity is affected, but in the
case of AES some attacks work for the larger key lengths that do not
work for smaller ones. There is one other case I am having problems
remembering where a longer keylength led to a massively easier to
attack function.

For SHA3_512, I am worried that some of the novel properties of the
function that do not currently lead to attacks may in the future lead
to attacks.

> 3. Keeping one of the three old hashes is necessary for compatibility
> reasons. Furthermore, the current versions of Portage consider SHA512
> obligatory, so we can't remove it without redesigning Portage first
> (though I think this applies only to developer installs, i.e. those
> creating Manifests).
>
> 4. The new hashes that are stronger and commonly available are
> SHA3/Keccak (using sponges) and BLAKE2 (HAIFA). Both are diverse from
> our current algorithms, so either is a good candidate. The choice of
> Keccak is purely arbitrary (because it's the winner?).
>
> All the above considered, I think it's most reasonable to use two hashes
> with diverse constructs. SHA512 needs to be one of them, for
> compatibility reasons. The other could be either SHA3_512 or BLAKE2B,
> as a strong, future-proof hash. SHA3 is probably a better choice because
> it's going to have more support as the official recommendation.
>

I would like to present my suggestions:

SHA512, (RIPEMD160 | WHIRLPOOL | BLAKE2B), (SHA3_512 | BLAKE2B);

or more definitively:

SHA512, RIPEMD160, BLAKE2B.

My understanding is that whirlpool is the least secure of the 512 bit
hashes, but it would still be a serious undertaking to generate a
collision. If there is any cause to replace it RIPEMD160 is likely a
good candidate. It is similar in operation to the SHA1/2 functions but
was produced by the academic community and is quite dissimilar from
the SHA1/2 family. The "definitive" suggestions provide a wide sample
of time, size, and implementation.

SHA3_512 is novel and may be resistant to attack in ways the others
are not, but it is my opinion that BLAKE2B's author is better known
and respected.

I could also see the combination SHA512, BLAKE2B, SHA3_512 being
desirable, as that selection represents 3 very diverse algorithms.

If the suggestion of RIPEMD160 seems odd, realize that some
distribution platforms still use MD5 along with other functions. It is
fast to compute and with another hash function quite credible. If the
time taken to generate the hashes is a legitimate concern then using a
older, simpler, and heavily optimized function may be better for one
of the choices.

Respectfully,
 R0b0t1

[gentoo-dev] last rites: app-crypt/zuluCrypt

[Andreas K. Huettel]

# Andreas K. Hüttel  (20 Oct 2017)
# Doesn't build with glibc-2.25, doesn't build, needs a
# version bump, needs a maintainer. Bugs 604364, 599004,
# 627064. Removal in 30 days.
app-crypt/zuluCrypt


-- 
Andreas K. Hüttel
dilfri...@gentoo.org
Gentoo Linux developer (council, perl, libreoffice)

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-portage-dev] [PATCH] portage.checksum: Support pyblake2 fallback for BLAKE2 hashes

[Zac Medico]

On 10/20/2017 12:26 PM, Michał Górny wrote:
> ---
>  .travis.yml |  2 ++
>  pym/portage/checksum.py | 15 +--
>  2 files changed, 15 insertions(+), 2 deletions(-)
> 
> diff --git a/.travis.yml b/.travis.yml
> index 20078530e..ebcfbeab9 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -12,6 +12,8 @@ install:
>  # python3.6+ has sha3 built-in, for older versions install pysha3
>  # (except for pypy where pysha3 is broken)
>  - "[[ ${TRAVIS_PYTHON_VERSION} == 3.[6789] || ${TRAVIS_PYTHON_VERSION} 
> == pypy ]] || pip install pysha3"
> +# python3.6+ has blake2 built-in, for older versions install pyblake2
> +- "[[ ${TRAVIS_PYTHON_VERSION} == 3.[6789] ]] || pip install pyblake2"
>  # always install pygost for Streebog
>  - pip install pygost
>  
> diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
> index ff132751b..ad090ddb3 100644
> --- a/pym/portage/checksum.py
> +++ b/pym/portage/checksum.py
> @@ -27,8 +27,8 @@ import tempfile
>  # SHA512: hashlib
>  # RMD160: hashlib, pycrypto, mhash
>  # WHIRLPOOL: hashlib, mhash, bundled
> -# BLAKE2B (512): hashlib (3.6+), pycrypto
> -# BLAKE2S (512): hashlib (3.6+), pycrypto
> +# BLAKE2B (512): hashlib (3.6+), pyblake2, pycrypto
> +# BLAKE2S (512): hashlib (3.6+), pyblake2, pycrypto
>  # SHA3_256: hashlib (3.6+), pysha3, pycrypto
>  # SHA3_512: hashlib (3.6+), pysha3, pycrypto
>  
> @@ -124,6 +124,17 @@ for local_name, hash_name in (
>   origin='hashlib')
>  
>  
> +# Support using pyblake2 as fallback for python<3.6
> +if "BLAKE2B" not in hashfunc_map or "BLAKE2S" not in hashfunc_map:
> + try:
> + import pyblake2
> +
> + _generate_hash_function("BLAKE2B", pyblake2.blake2b, 
> origin="pyblake2")
> + _generate_hash_function("BLAKE2S", pyblake2.blake2s, 
> origin="pyblake2")
> + except ImportError:
> + pass
> +
> +
>  # Support using pysha3 as fallback for python<3.6
>  if "SHA3_256" not in hashfunc_map or "SHA3_512" not in hashfunc_map:
>   try:
> 

Looks good, please merge.
-- 
Thanks,
Zac



signature.asc
Description: OpenPGP digital signature

[gentoo-portage-dev] [PATCH] portage.checksum: Support pyblake2 fallback for BLAKE2 hashes

[Michał Górny]

---
 .travis.yml |  2 ++
 pym/portage/checksum.py | 15 +--
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 20078530e..ebcfbeab9 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -12,6 +12,8 @@ install:
 # python3.6+ has sha3 built-in, for older versions install pysha3
 # (except for pypy where pysha3 is broken)
 - "[[ ${TRAVIS_PYTHON_VERSION} == 3.[6789] || ${TRAVIS_PYTHON_VERSION} == 
pypy ]] || pip install pysha3"
+# python3.6+ has blake2 built-in, for older versions install pyblake2
+- "[[ ${TRAVIS_PYTHON_VERSION} == 3.[6789] ]] || pip install pyblake2"
 # always install pygost for Streebog
 - pip install pygost
 
diff --git a/pym/portage/checksum.py b/pym/portage/checksum.py
index ff132751b..ad090ddb3 100644
--- a/pym/portage/checksum.py
+++ b/pym/portage/checksum.py
@@ -27,8 +27,8 @@ import tempfile
 # SHA512: hashlib
 # RMD160: hashlib, pycrypto, mhash
 # WHIRLPOOL: hashlib, mhash, bundled
-# BLAKE2B (512): hashlib (3.6+), pycrypto
-# BLAKE2S (512): hashlib (3.6+), pycrypto
+# BLAKE2B (512): hashlib (3.6+), pyblake2, pycrypto
+# BLAKE2S (512): hashlib (3.6+), pyblake2, pycrypto
 # SHA3_256: hashlib (3.6+), pysha3, pycrypto
 # SHA3_512: hashlib (3.6+), pysha3, pycrypto
 
@@ -124,6 +124,17 @@ for local_name, hash_name in (
origin='hashlib')
 
 
+# Support using pyblake2 as fallback for python<3.6
+if "BLAKE2B" not in hashfunc_map or "BLAKE2S" not in hashfunc_map:
+   try:
+   import pyblake2
+
+   _generate_hash_function("BLAKE2B", pyblake2.blake2b, 
origin="pyblake2")
+   _generate_hash_function("BLAKE2S", pyblake2.blake2s, 
origin="pyblake2")
+   except ImportError:
+   pass
+
+
 # Support using pysha3 as fallback for python<3.6
 if "SHA3_256" not in hashfunc_map or "SHA3_512" not in hashfunc_map:
try:
-- 
2.15.0.rc1

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michał Górny]

W dniu pią, 20.10.2017 o godzinie 17∶42 +0200, użytkownik Paweł Hajdan,
Jr. napisał:
> On 19/10/2017 21:08, Michał Górny wrote:
> > Considering all arguments made so far, I'd like to propose changing:
> >   manifest-hashes = SHA256 SHA512 WHIRLPOOL
> > to:
> >   manifest-hashes = SHA512 SHA3_512
> 
> +1, fine for me
> 
> > 1. The main argument for using multiple hashes is to prevent the (very
> > unlikely) possibility that if a weakness is discovered in one of
> > the hashes, the other would still hold. This is given by using two
> > algorithms; more than two do not increase security significantly, while
> > they do increase performance cost.
> 
> Curious, do we have any measurements/estimates of the performance cost?
> 

Not specific but I think it's pretty simple, assuming we don't get any
multithread-friendly algorithms.

With a single thread serial processing of all hashes, it's just sum of
times involved in every hash, i.e. Th = T1 + T2 + T3 + ... You'd have to
get some numbers to get something smarter out of it.

If we assume we can do N threads, then cost of N algorithms is equal to
the slowest of them all. Which implies that having N algorithms is
fastest on systems capable of at least N threads.

Taking a random comparison [1], it seems that SHA3/512 is 3-5 times
slower than SHA2/512. If we take that as conclusive, the relative times
would be:

a. single hash:

 SHA512 - 1
 SHA3_512 - 3-5

b. both hashes:

 serial - 4-6
 parallel - 3-5

[1]:http://wireilla.com/papers/ijcis/V3N3/3313ijcis01.pdf

-- 
Best regards,
Michał Górny

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Paweł Hajdan , Jr .]

On 19/10/2017 21:08, Michał Górny wrote:
> Considering all arguments made so far, I'd like to propose changing:
>   manifest-hashes = SHA256 SHA512 WHIRLPOOL
> to:
>   manifest-hashes = SHA512 SHA3_512

+1, fine for me

> 1. The main argument for using multiple hashes is to prevent the (very
> unlikely) possibility that if a weakness is discovered in one of
> the hashes, the other would still hold. This is given by using two
> algorithms; more than two do not increase security significantly, while
> they do increase performance cost.

Curious, do we have any measurements/estimates of the performance cost?

Paweł



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Rich Freeman]

On Fri, Oct 20, 2017 at 6:04 AM, Kristian Fiskerstrand  wrote:
> On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
>>
>> I support Hanno's suggestion of doing just SHA512, but would be
>> interested in hearing opinions from others who have apparent
>> security/crypto experience. Maybe the Security project can weigh the
>> suggestions as well?
>>
>
> The whole discussion is moot so long as we don't have OpenPGP signed
> gentoo repository in rsync.
>
> SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
> considerably slower for some architectures. Introducing a non-optimized
> keccak on top of it will have a significant negative performance impact
> for these arches without much security gain.
>
> if we still want two separate hashes, the choice of sha2 and sha3
> compination is a good one given they are based on separate constructs.
>
> But IMHO we should start where things matter and complete an
> implementation for OpenPGP signatures of MetaManifests in Portage.
>

Nobody is stopping you from implementing this.

However, saying that we can't improve any other aspects of the
security of Gentoo until somebody improves this one aspect (that
people have been talking about for a decade) seems like a false
dependency.  I do get that a chain is as strong as its weakest link,
but there are attacks that an unsigned hash protects us against.  If
somebody has the ability to modify the upstream binary but not the
ability to modify our hashes, then the hash protects us even if it is
unsigned.  Sure, it would be better still if it were signed, but that
is no reason to make improvements all the same.

IMO having two hash functions makes sense.

Sure, the last two times a hash has failed have had a decade of
warning.  However, there is no theoretical reason why this has to be
the case.  At best you can make hand-waving arguments.

I wouldn't look at md5/sha1 and take from them the lesson that we can
relax because if a hash function fails we have plenty of warning, or
that we don't have to worry because we use more bits today.

Rather I would look at md5/sha1 and take from them this lesson: They
were never intended to be defeated in less than brute force time, and
yet both were.  Adding more bits makes it take longer to brute-force,
but that provides no assurance against an attack that runs faster than
brute force.  No such attack should exist, but the same was true of
md5/sha1.

Sure, some hash failures are worse than others, and we're lucky in
that a less severe failure was discovered first, but there is no
theoretical reason that failures have to be discovered in some
particular order.

If it were a matter of adding support for multiple hash functions to
portage that would be one thing.  However, all our tools already
support this, so why not continue to take advantage of this in the
most efficient manner possible?  Why give up a layer of security that
costs us little but a bit of CPU.

-- 
Rich

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Kristian Fiskerstrand]

On 10/20/2017 03:05 PM, Michael Orlitzky wrote:
> Every WiFi network on the planet essentially became Starbucks overnight
> on Sunday->Monday, so in my opinion we shouldn't bet against immediate
> and catastrophic failure of anything, no matter how well-tested.

Post Hoc ergo Propter Hoc

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Michael Orlitzky]

On 10/19/2017 06:32 PM, Hanno Böck wrote:
> 
> Counterproposal: Just use SHA512.
> 
> There isn't any evidence that any SHA2-based hash algorithm is going to
> be broken any time soon. If that changes there will very likely be
> decades of warning before a break becomes practical.
> 

Every WiFi network on the planet essentially became Starbucks overnight
on Sunday->Monday, so in my opinion we shouldn't bet against immediate
and catastrophic failure of anything, no matter how well-tested.

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Kristian Fiskerstrand]

On 10/20/2017 11:10 AM, Dirkjan Ochtman wrote:
> 
> I support Hanno's suggestion of doing just SHA512, but would be
> interested in hearing opinions from others who have apparent
> security/crypto experience. Maybe the Security project can weigh the
> suggestions as well?
> 

The whole discussion is moot so long as we don't have OpenPGP signed
gentoo repository in rsync.

SHA2-512 is generally quicker than sha256 on 64 bit architectures, but
considerably slower for some architectures. Introducing a non-optimized
keccak on top of it will have a significant negative performance impact
for these arches without much security gain.

if we still want two separate hashes, the choice of sha2 and sha3
compination is a good one given they are based on separate constructs.

But IMHO we should start where things matter and complete an
implementation for OpenPGP signatures of MetaManifests in Portage.

-- 
Kristian Fiskerstrand
OpenPGP keyblock reachable at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3



signature.asc
Description: OpenPGP digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Hanno Böck]

On Fri, 20 Oct 2017 11:23:06 +0200
Ulrich Mueller  wrote:

> > On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:  
> 
> > As Hanno was saying, we'll have decades of warning before a break
> > becomes practical, so I don't think this is a real concern.  
> 
> How can we be sure of that? I guess the same reasoning was applied
> when MD5 and SHA1 hashes were used.

MD5 warning 1996:
ftp://ftp.iks-jena.de/mitarb/lutz/crypt/hash/dobbertin.ps

MD5 broken 2005:
http://merlot.usc.edu/csac-f06/papers/Wang05a.pdf

SHA1 warning 2005:
https://people.csail.mit.edu/yiqun/SHA1AttackProceedingVersion.pdf

SHA1 broken 2017:
https://shattered.io/


It's reasonable to assume that modern hash functions will have a far
longer warning period. For two reasons:
* their safety margin is much higher to begin with, particularly if
  you choose something like SHA512 (256 bit collission resistance). It
  was more or less always clear that MD5 (64 bit) and SHA1 (80 bit) are
  in risky terrain even without any cryptographic breakthrough.
* hash function research in 2017 is lightyears ahead of hash function
  research in the 90s and early 2000s. One major outcome of the
  research after the big hash breakdown in 2005 was that SHA-2 is much
  safer than people previously thought.


I don' have a very strong opinion on this. Having two hash functions
probably won't harm. Though I tend to prefer the simplest solutions if
it's secure. And all my cryptographic knowledge tells me that "What if
sha512 is broken?" isn't a realistic problem to be concerned about.


I do feel it's a bit ironic that we have these lengthy discussions
about hash functions while at the same time they provide little
security to begin with, because they aren't transmitted over a secure
channel and not signed...

-- 
Hanno Böck
https://hboeck.de/

mail/jabber: ha...@hboeck.de
GPG: FE73757FA60E4E21B937579FA5880072BBB51E42


pgpR5FdZLlUJa.pgp
Description: OpenPGP digital signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Dirkjan Ochtman]

On Fri, Oct 20, 2017 at 11:23 AM, Ulrich Mueller  wrote:

> > On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:
>
> > As Hanno was saying, we'll have decades of warning before a break
> > becomes practical, so I don't think this is a real concern.
>
> How can we be sure of that? I guess the same reasoning was applied
> when MD5 and SHA1 hashes were used.
>

Yeah, and it actually did happen that way. Typically before preimage
attacks (which are what we really care about here, as far as I understand
it) happen there are several other types of attacks that will happen first,
and that will provide advance warning about the level of security provided
by SHA2.

Cheers,

Dirkjan

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Ulrich Mueller]

> On Fri, 20 Oct 2017, Dirkjan Ochtman wrote:

> As Hanno was saying, we'll have decades of warning before a break
> becomes practical, so I don't think this is a real concern.

How can we be sure of that? I guess the same reasoning was applied
when MD5 and SHA1 hashes were used.

> I think the problem of having this discussion on gentoo-dev this way
> is that people with vastly different levels of security/crypto
> expertise are discussing different options without much regard for
> the level of expertise (and maybe even unaware of others' relevant
> expertise).

> I support Hanno's suggestion of doing just SHA512, but would be
> interested in hearing opinions from others who have apparent
> security/crypto experience. Maybe the Security project can weigh the
> suggestions as well?

Don't put all eggs in one basket. Having at least one additional hash
(and from a different family) doesn't cost us much and provides an
upgrade path when it should become necessary.

Ulrich


pgpJYEGAR7kOA.pgp
Description: PGP signature

Re: [gentoo-dev] Manifest2 hashes, take n+1-th

[Dirkjan Ochtman]

On Fri, Oct 20, 2017 at 12:49 AM, Gordon Pettey 
wrote:

> On Thu, Oct 19, 2017 at 5:32 PM, Hanno Böck  wrote:
>
>> On Thu, 19 Oct 2017 21:08:40 +0200
>> Michał Górny  wrote:
>>
>> >   manifest-hashes = SHA512 SHA3_512
>>
>> Counterproposal: Just use SHA512.
>>
>> There isn't any evidence that any SHA2-based hash algorithm is going to
>> be broken any time soon. If that changes there will very likely be
>> decades of warning before a break becomes practical.
>>
>> Having just one hash is simpler and using a well supported one like
>> SHA512 may make things easier than using something that's still not
>> very widely supported.
>
>
> Yet having more than one lets you match make sure nobody hijacked your
> manifest file when an attack vector is inevitably discovered for the old
> new algorithm (whether SHA2, SHA3, or BLAKE2), because you'll be able to
> confirm the file is the same one that matched the old checksum in addition
> to the new one.
>

As Hanno was saying, we'll have decades of warning before a break becomes
practical, so I don't think this is a real concern.

I think the problem of having this discussion on gentoo-dev this way is
that people with vastly different levels of security/crypto expertise are
discussing different options without much regard for the level of expertise
(and maybe even unaware of others' relevant expertise).

I support Hanno's suggestion of doing just SHA512, but would be interested
in hearing opinions from others who have apparent security/crypto
experience. Maybe the Security project can weigh the suggestions as well?

Cheers,

Dirkjan