Re: [openssl-dev] [EXTERNAL] Re: use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Peter Waltenberg 
Yes, but. LHash hashes internal object names not externally presented input.Certainly if it's used on externally presented data it's a worthwhile change, but AFAIK that isn't the case.Peter-"openssl-dev"  wrote: -To: openssl-dev@openssl.orgFrom: Jeremy Farrell Sent by: "openssl-dev" Date: 01/12/2017 11:17AMSubject: Re: [openssl-dev] [EXTERNAL] Re: use SIPhash for OPENSSL_LH_strhash?  

  
  
For something like SipHash, knowing "whichever
  algo the server uses" effectively implies knowing the 128-bit
  random key currently being used for the hash table in question.Regards,    jjfOn 12/01/2017 00:39, Sands, Daniel
  wrote:  With a small number of buckets, it seems to me that no hash algo willmake you safe from a flooding attack.  You can simply generate yourhashes locally using whichever algo the server uses, and only send thosethat fit into your attack scheme.  The data could even be pre-generated.The only way to guard against a flood that makes sense to me is to limitthe number of items that can be accepted before deciding you're beingtrolled.On Wed, 2017-01-11 at 23:29 +, J. J. Farrell wrote:  Are the issues you raise true of SipHash, given that a prime motivatorfor its design was generating hash tables for short inputs while beingsecure against hash flooding attacks? It achieves this with theperformance of a portable C implementation the order of four timesfaster than MD5, and not much slower than other modern hashalgorithms.I'd have thought the main thing to consider is whether or not there isany practical way a hash flooding attack could be used againstOpenSSL's hash tables, and it sounds like there isn't. In that case,the fastest algorithm for the usage patterns would be best.Regards,                          jjfOn 11/01/2017 22:25, Peter Waltenberg wrote:  And the reason I said you certainly don't need a keyed hash ?Behaviour of the hash function will change with key and in somecases performance would degenerate to that of a linked list. (Ouch).And since the obvious thing to do is use a random key, OpenSSL'sperformance would get *very* erratic.Simpler functions than cryptographic hashes will almost certainlyyield better results here. I note someone further up the threadsomeone else has pointed that out. PeterFrom:        "Salz, Rich" To:        "openssl-dev@openssl.org" Date:        11/01/2017 13:14Subject:        Re: [openssl-dev] use SIPhash forOPENSSL_LH_strhash?Sent by:        "openssl-dev" The needs for OpenSSL's LHASH are exactly what SipHash was designedfor: fast on short strings.OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 codeis commented out.Yes, performance tests would greatly inform the decision.-- J. J. FarrellNot speaking for Oracle-- openssl-dev mailing listTo unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev-- J. J. Farrellw: +44 161 493 4838  

-- openssl-dev mailing listTo unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Peter Waltenberg 
 It pretty much has to be true of any keyed hash if you think about it. If it didn't distribute the hashes differently each time it wouldn't be working, if it distributes the hashes differently, performance has to be key dependent. And with a hash size the same as the key, at least one of the possible combinations has to be the pathological case.I can't currently see any possible vector for a flooding attack, well O.K., I certainly can if you use SipHash with random keys :) and even that would be hard to exploit, but otherwise no. If it's significantly faster using it with a pre-tested fixed key is probably fine, but it gives up the security characteristic you were after. My suspicion is also that simply compressing the string with XOR will work at least as well.Peter-"openssl-dev"  wrote: -To: openssl-dev@openssl.orgFrom: "J. J. Farrell" Sent by: "openssl-dev" Date: 01/12/2017 10:05AMSubject: Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?  

  
  
Are the issues you raise true of SipHash, given
  that a prime motivator for its design was generating hash tables
  for short inputs while being secure against hash flooding attacks?
  It achieves this with the performance of a portable C
  implementation the order of four times faster than MD5, and not
  much slower than other modern hash algorithms.I'd have thought the main thing to consider is whether or not
  there is any practical way a hash flooding attack could be used
  against OpenSSL's hash tables, and it sounds like there isn't. In
  that case, the fastest algorithm for the usage patterns would be
  best.Regards,    jjfOn 11/01/2017 22:25, Peter Waltenberg
  wrote:And the reason I said
you certainly don't
need a keyed hash ?Behaviour of the hash function
will
change with key and in some cases performance would degenerate
to that
of a linked list. (Ouch). And since the obvious thing to do is
use a random
key, OpenSSL's performance would get *very* erratic.Simpler functions than
cryptographic
hashes will almost certainly yield better results here. I note
someone
further up the thread someone else has pointed that out. PeterFrom:      
 "Salz, Rich"
  To:      
 "openssl-dev@openssl.org"  Date:      
 11/01/2017 13:14  Subject:    
   Re: [openssl-dev]
use SIPhash for OPENSSL_LH_strhash?  Sent by:    
   "openssl-dev"
The needs for OpenSSL's LHASH are exactly what
  SipHash
  was designed for: fast on short strings.  OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5
  code is commented
  out.  Yes, performance tests would greatly inform the decision.-- J. J. FarrellNot speaking for Oracle  

-- openssl-dev mailing listTo unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] wiki update for enc command

2017-01-11 Thread Salz, Rich 
That whole “Manual:” thing on the wiki should probably go away and just refer 
to the website which gets updated every time we change things.

Can you open an issue for the manpage bug you found?

Thanks!

--
Senior Architect, Akamai Technologies
Member, OpenSSL Dev Team
IM: richs...@jabber.at Twitter: RichSalz

From: Michel [mailto:michel.sa...@free.fr]
Sent: Wednesday, January 11, 2017 9:18 AM
To: openssl-dev@openssl.org
Subject: [openssl-dev] wiki update for enc command

Hi,

Looks like one of my previous mail (see below) was lost in the ‘cloud’ ;-)
Might be helpfull to send it again here ?

Regards,

Michel


De : Michel [mailto:michel.sa...@free.fr]
Envoyé : samedi 19 novembre 2016 14:16
À : 'wiki-supp...@openssl.org'
Objet : wiki update

HI,

FYI, the page at 
https://wiki.openssl.org/index.php/Manual:Enc(1)
 read options –salt and –nosalt twice.

I believe it would be better to remove the first –salt section and the second 
–nosalt section.

Also the statement in the NOTES : “When the salt is being used the first eight 
bytes of the encrypted data are reserved for the salt” is wrong.
It’s a 16 bytes header including the ‘magic’ “Salted__”.
(the same applies to : 
https://www.openssl.org/docs/man1.1.0/apps/enc.html)

[…]

Regards,

Michel.
https://github.com/EasySec

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Salz, Rich 
> Is there really no use of LHASH tables in OpenSSL where an attacker
> attempting a DoS attack can control the contents of the tables?

The only use of LHASH is in SSL_SESSION and X509_NAME, which use their own 
hashing functions, and are only used after the session and/or certs have been 
cryptographically verified.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Richard Levitte 
A note: I have absolutely nothing against the addition of SIPhash in
our collection of hash algos.  My scepticism was only in regards to
using it as a string hasher for our hash tables indexes.

Cheers,
Richard

In message <20170111.153458.1623912899597806811.levi...@openssl.org> on Wed, 11 
Jan 2017 15:34:58 +0100 (CET), Richard Levitte  said:

levitte> In message 
<1e19cdfea8224717b3eee11e2d8ac...@usma1ex-dag1mb1.msg.corp.akamai.com> on Wed, 
11 Jan 2017 03:13:39 +, "Salz, Rich"  said:
levitte> 
levitte> rsalz> The needs for OpenSSL's LHASH are exactly what SipHash was 
designed for: fast on short strings.
levitte> rsalz> OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 
code is commented out.
levitte> rsalz> Yes, performance tests would greatly inform the decision.
levitte> 
levitte> Done, using the reference siphash implementation.
levitte> 
levitte> https://github.com/levitte/openssl/tree/test-string-hashes
levitte> 
levitte> A run on my laptop gave these results:
levitte> 
levitte> : ; ./util/shlib_wrap.sh apps/openssl speed siphash lhash
levitte> Doing lhash for 3s on 16 size blocks: 27635188 lhash's in 3.00s
levitte> Doing lhash for 3s on 64 size blocks: 6934726 lhash's in 3.00s
levitte> Doing lhash for 3s on 256 size blocks: 1698489 lhash's in 3.00s
levitte> Doing lhash for 3s on 1024 size blocks: 431185 lhash's in 3.00s
levitte> Doing lhash for 3s on 8192 size blocks: 53868 lhash's in 3.00s
levitte> Doing lhash for 3s on 16384 size blocks: 27041 lhash's in 3.00s
levitte> Doing siphash for 3s on 16 size blocks: 22488748 siphash's in 3.00s
levitte> Doing siphash for 3s on 64 size blocks: 10485674 siphash's in 3.00s
levitte> Doing siphash for 3s on 256 size blocks: 3320898 siphash's in 3.00s
levitte> Doing siphash for 3s on 1024 size blocks: 894647 siphash's in 3.00s
levitte> Doing siphash for 3s on 8192 size blocks: 114170 siphash's in 3.00s
levitte> Doing siphash for 3s on 16384 size blocks: 57151 siphash's in 3.00s
levitte> OpenSSL 1.1.1-dev  xx XXX 
levitte> built on: reproducible build, date unspecified
levitte> options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) 
blowfish(ptr) 
levitte> compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_THREADS 
-DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 
-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM 
-DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM 
-DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM 
-DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-1.1\"" 
 -DDEBUG_UNUSED -Wswitch -DPEDANTIC -pedantic -Wno-long-long -Wall 
-Wsign-compare -Wmissing-prototypes -Wshadow -Wformat -Wtype-limits -Werror 
-Wa,--noexecstack
levitte> The 'numbers' are in 1000s of bytes per second processed.
levitte> type 16 bytes 64 bytes256 bytes   1024 bytes   
8192 bytes  16384 bytes
levitte> lhash   147387.67k   147940.82k   144937.73k   147177.81k  
 147095.55k   147679.91k
levitte> siphash 119939.99k   223694.38k   283383.30k   305372.84k  
 311760.21k   312120.66k
levitte> 
levitte> So it seems that for short strings, OPENSSL_LH_strhash (*) wins some,
levitte> while siphash wins big for larger strings.
levitte> 
levitte> I have no idea how they compare with regard to distribution (which,
levitte> considering I ask for the same size output from both, should be the
levitte> main factor that affects the sensitivity to hash flooding)...
levitte> 
levitte> Our use of OPENSSL_LH_strhash() is with configuration sections and
levitte> names, ASN.1 object names and the function names in the openssl app.
levitte> All our other uses of lhash use their own hashing functions, and are
levitte> usually very short still (definitely less than 16 bytes).
levitte> 
levitte> My conclusion is that performance-wise, siphash doesn't give us any
levitte> advantage over OpenSSL_LH_strhash for our uses.
levitte> 
levitte> Cheers,
levitte> Richard
levitte> 
levitte> (*) Strictly speaking, it's a modified version that takes a length and
levitte> tolerates all 8-bit bytes, including 0x00.
levitte> 
levitte> -- 
levitte> Richard Levitte levi...@openssl.org
levitte> OpenSSL Project http://www.openssl.org/~levitte/
levitte> -- 
levitte> openssl-dev mailing list
levitte> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev
levitte> 
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] wiki update for enc command

2017-01-11 Thread Michel 
> Can you open an issue for the manpage bug you found?

 

Yes, I will.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Richard Levitte 
Can we look forward to a github PR?

In message <97d0be2d-11c6-4d01-9a5d-faccc5b27...@akamai.com> on Tue, 10 Jan 
2017 22:42:17 +, "Short, Todd"  said:

tshort> I think I might have an init/update/final version of siphash24 lying
tshort> around somewhere that would be compatible with OpenSSL’s EVP_PKEY
tshort> mechanism (similar to Poly1305, in that it needs a key).
tshort> --
tshort> -Todd Short
tshort> // tsh...@akamai.com
tshort> // "One if by land, two if by sea, three if by the Internet."
tshort> 
tshort> On Jan 10, 2017, at 4:55 PM, Richard Levitte 
tshort> wrote:
tshort> 
tshort> 
tshort> 
tshort> 
tshort> Benjamin Kaduk  skrev: (10 januari 2017
tshort> 20:19:21 CET)
tshort> 
tshort> On 01/10/2017 12:31 PM, Richard Levitte wrote:
tshort> 
tshort> 
tshort> Benjamin Kaduk  skrev: (10 januari 2017
tshort> 18:48:32
tshort> 
tshort> 
tshort> CET)
tshort> 
tshort> On 01/09/2017 10:05 PM, Salz, Rich wrote:
tshort> 
tshort> Should we move to using SIPHash for the default string
tshort> hashing
tshort> function in OpenSSL? It’s now in the kernel
tshort> https://lkml.org/lkml/2017/1/9/619
tshort> 
tshort> 
tshort> 
tshort> Heck, yes!
tshort> -Ben
tshort> 
tshort> 
tshort> I fail to see what that would give us. OPENSSL_LH_strhash
tshort> () is used
tshort> 
tshort> 
tshort> to get a reasonable index for LHASH entries. Also SIPhash
tshort> gives at
tshort> least 64 bits results, do we really expect to see large enough
tshort> hash
tshort> tables to warrant that?
tshort> 
tshort> 
tshort> 
tshort> 
tshort> We don't need to use the full output width of a good hash
tshort> function.
tshort> 
tshort> My main point is, "why would we want to ignore the last 20
tshort> years of
tshort> advancement in hash function research?" Section 7 of the
tshort> siphash paper
tshort> (https://131002.net/siphash/siphash.pdf) explicitly talks
tshort> about using
tshort> it
tshort> for hash tables, including using hash table indices H(m) mod
tshort> l.
tshort> 
tshort> 
tshort> I agree with the advice when one can expect huge tables. The
tshort> tables we handle are pretty small (I think, please correct me if
tshort> I'm wrong) and would in all likelihood not benefit very much if at
tshort> all from SIPhash's relative safety.
tshort> 
tshort> Of course, one can ask the question if someone uses LHASH as a
tshort> general purpose hash table implementation rather than just for the
tshort> stuff OpenSSL. Frankly, I would probably look at a dedicated hash
tshort> table library first...
tshort> 
tshort> Cheers
tshort> Richard
tshort> --
tshort> Sent from my Android device with K-9 Mail. Please excuse my
tshort> brevity.
tshort> --
tshort> openssl-dev mailing list
tshort> To unsubscribe:
tshort> https://mta.openssl.org/mailman/listinfo/openssl-dev
tshort> 
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

[openssl-dev] wiki update for enc command

2017-01-11 Thread Michel 
Hi, 

 

Looks like one of my previous mail (see below) was lost in the ‘cloud’ ;-)

Might be helpfull to send it again here ?

 

Regards,

 

Michel

 

 

De : Michel [mailto:michel.sa...@free.fr] 
Envoyé : samedi 19 novembre 2016 14:16
À : 'wiki-supp...@openssl.org'
Objet : wiki update

 

HI, 

 

FYI, the page at https://wiki.openssl.org/index.php/Manual:Enc(1) read
options –salt and –nosalt twice.

 

I believe it would be better to remove the first –salt section and the
second –nosalt section.

 

Also the statement in the NOTES : “When the salt is being used the first
eight bytes of the encrypted data are reserved for the salt” is wrong.

It’s a 16 bytes header including the ‘magic’ “Salted__”.

(the same applies to : https://www.openssl.org/docs/man1.1.0/apps/enc.html)

 

[…]

 

Regards,

 

Michel.

https://github.com/EasySec

 

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Richard Levitte 
In message 
<1e19cdfea8224717b3eee11e2d8ac...@usma1ex-dag1mb1.msg.corp.akamai.com> on Wed, 
11 Jan 2017 03:13:39 +, "Salz, Rich"  said:

rsalz> The needs for OpenSSL's LHASH are exactly what SipHash was designed for: 
fast on short strings.
rsalz> OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code is 
commented out.
rsalz> Yes, performance tests would greatly inform the decision.

Done, using the reference siphash implementation.

https://github.com/levitte/openssl/tree/test-string-hashes

A run on my laptop gave these results:

: ; ./util/shlib_wrap.sh apps/openssl speed siphash lhash
Doing lhash for 3s on 16 size blocks: 27635188 lhash's in 3.00s
Doing lhash for 3s on 64 size blocks: 6934726 lhash's in 3.00s
Doing lhash for 3s on 256 size blocks: 1698489 lhash's in 3.00s
Doing lhash for 3s on 1024 size blocks: 431185 lhash's in 3.00s
Doing lhash for 3s on 8192 size blocks: 53868 lhash's in 3.00s
Doing lhash for 3s on 16384 size blocks: 27041 lhash's in 3.00s
Doing siphash for 3s on 16 size blocks: 22488748 siphash's in 3.00s
Doing siphash for 3s on 64 size blocks: 10485674 siphash's in 3.00s
Doing siphash for 3s on 256 size blocks: 3320898 siphash's in 3.00s
Doing siphash for 3s on 1024 size blocks: 894647 siphash's in 3.00s
Doing siphash for 3s on 8192 size blocks: 114170 siphash's in 3.00s
Doing siphash for 3s on 16384 size blocks: 57151 siphash's in 3.00s
OpenSSL 1.1.1-dev  xx XXX 
built on: reproducible build, date unspecified
options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) 
blowfish(ptr) 
compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_THREADS 
-DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 
-DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM 
-DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM 
-DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM 
-DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib/engines-1.1\"" 
 -DDEBUG_UNUSED -Wswitch -DPEDANTIC -pedantic -Wno-long-long -Wall 
-Wsign-compare -Wmissing-prototypes -Wshadow -Wformat -Wtype-limits -Werror 
-Wa,--noexecstack
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes256 bytes   1024 bytes   8192 
bytes  16384 bytes
lhash   147387.67k   147940.82k   144937.73k   147177.81k   
147095.55k   147679.91k
siphash 119939.99k   223694.38k   283383.30k   305372.84k   
311760.21k   312120.66k

So it seems that for short strings, OPENSSL_LH_strhash (*) wins some,
while siphash wins big for larger strings.

I have no idea how they compare with regard to distribution (which,
considering I ask for the same size output from both, should be the
main factor that affects the sensitivity to hash flooding)...

Our use of OPENSSL_LH_strhash() is with configuration sections and
names, ASN.1 object names and the function names in the openssl app.
All our other uses of lhash use their own hashing functions, and are
usually very short still (definitely less than 16 bytes).

My conclusion is that performance-wise, siphash doesn't give us any
advantage over OpenSSL_LH_strhash for our uses.

Cheers,
Richard

(*) Strictly speaking, it's a modified version that takes a length and
tolerates all 8-bit bytes, including 0x00.

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Short, Todd 
I’d be doing it in a manner similar to Poly1305, since that’s a fresh memory… 
it shouldn’t take long.
--
-Todd Short
// tsh...@akamai.com
// "One if by land, two if by sea, three if by the Internet."

On Jan 11, 2017, at 9:44 AM, Richard Levitte 
> wrote:

Can we look forward to a github PR?

In message 
<97d0be2d-11c6-4d01-9a5d-faccc5b27...@akamai.com>
 on Tue, 10 Jan 2017 22:42:17 +, "Short, Todd" 
> said:

tshort> I think I might have an init/update/final version of siphash24 lying
tshort> around somewhere that would be compatible with OpenSSL’s EVP_PKEY
tshort> mechanism (similar to Poly1305, in that it needs a key).
tshort> --
tshort> -Todd Short
tshort> // tsh...@akamai.com
tshort> // "One if by land, two if by sea, three if by the Internet."
tshort>
tshort> On Jan 10, 2017, at 4:55 PM, Richard Levitte 
>
tshort> wrote:
tshort>
tshort>
tshort>
tshort>
tshort> Benjamin Kaduk > skrev: 
(10 januari 2017
tshort> 20:19:21 CET)
tshort>
tshort> On 01/10/2017 12:31 PM, Richard Levitte wrote:
tshort>
tshort>
tshort> Benjamin Kaduk 
> skrev: (10 januari 2017
tshort> 18:48:32
tshort>
tshort>
tshort> CET)
tshort>
tshort> On 01/09/2017 10:05 PM, Salz, Rich wrote:
tshort>
tshort> Should we move to using SIPHash for the default string
tshort> hashing
tshort> function in OpenSSL? It’s now in the kernel
tshort> https://lkml.org/lkml/2017/1/9/619
tshort>
tshort>
tshort>
tshort> Heck, yes!
tshort> -Ben
tshort>
tshort>
tshort> I fail to see what that would give us. OPENSSL_LH_strhash
tshort> () is used
tshort>
tshort>
tshort> to get a reasonable index for LHASH entries. Also SIPhash
tshort> gives at
tshort> least 64 bits results, do we really expect to see large enough
tshort> hash
tshort> tables to warrant that?
tshort>
tshort>
tshort>
tshort>
tshort> We don't need to use the full output width of a good hash
tshort> function.
tshort>
tshort> My main point is, "why would we want to ignore the last 20
tshort> years of
tshort> advancement in hash function research?" Section 7 of the
tshort> siphash paper
tshort> (https://131002.net/siphash/siphash.pdf) explicitly talks
tshort> about using
tshort> it
tshort> for hash tables, including using hash table indices H(m) mod
tshort> l.
tshort>
tshort>
tshort> I agree with the advice when one can expect huge tables. The
tshort> tables we handle are pretty small (I think, please correct me if
tshort> I'm wrong) and would in all likelihood not benefit very much if at
tshort> all from SIPhash's relative safety.
tshort>
tshort> Of course, one can ask the question if someone uses LHASH as a
tshort> general purpose hash table implementation rather than just for the
tshort> stuff OpenSSL. Frankly, I would probably look at a dedicated hash
tshort> table library first...
tshort>
tshort> Cheers
tshort> Richard
tshort> --
tshort> Sent from my Android device with K-9 Mail. Please excuse my
tshort> brevity.
tshort> --
tshort> openssl-dev mailing list
tshort> To unsubscribe:
tshort> https://mta.openssl.org/mailman/listinfo/openssl-dev
tshort>
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [EXTERNAL] Re: use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Sands, Daniel 
Just a note from my own experience way back when:  I tried hashing using
various algos and measuring bucket use as the main comparison criteria.
I found that the crypto hashes left a fair number of unused buckets.  Of
course, CRCs were far worse.  What gave the most normal distribution was
to simply take the bytes 4 at a time as 32-bit integers and simply add
them.

Not to say that this is really the holy grail, just pointing out that
compute speed shouldn't be the only criteria you use for comparison.

On Wed, 2017-01-11 at 15:43 +0100, Richard Levitte wrote:
> A note: I have absolutely nothing against the addition of SIPhash in
> our collection of hash algos.  My scepticism was only in regards to
> using it as a string hasher for our hash tables indexes.
> 
> Cheers,
> Richard
> 
> In message <20170111.153458.1623912899597806811.levi...@openssl.org> on Wed, 
> 11 Jan 2017 15:34:58 +0100 (CET), Richard Levitte  said:
> 
> levitte> In message 
> <1e19cdfea8224717b3eee11e2d8ac...@usma1ex-dag1mb1.msg.corp.akamai.com> on 
> Wed, 11 Jan 2017 03:13:39 +, "Salz, Rich"  said:
> levitte> 
> levitte> rsalz> The needs for OpenSSL's LHASH are exactly what SipHash was 
> designed for: fast on short strings.
> levitte> rsalz> OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 
> code is commented out.
> levitte> rsalz> Yes, performance tests would greatly inform the decision.
> levitte> 
> levitte> Done, using the reference siphash implementation.
> levitte> 
> levitte> https://github.com/levitte/openssl/tree/test-string-hashes
> levitte> 
> levitte> A run on my laptop gave these results:
> levitte> 
> levitte> : ; ./util/shlib_wrap.sh apps/openssl speed siphash lhash
> levitte> Doing lhash for 3s on 16 size blocks: 27635188 lhash's in 3.00s
> levitte> Doing lhash for 3s on 64 size blocks: 6934726 lhash's in 3.00s
> levitte> Doing lhash for 3s on 256 size blocks: 1698489 lhash's in 3.00s
> levitte> Doing lhash for 3s on 1024 size blocks: 431185 lhash's in 3.00s
> levitte> Doing lhash for 3s on 8192 size blocks: 53868 lhash's in 3.00s
> levitte> Doing lhash for 3s on 16384 size blocks: 27041 lhash's in 3.00s
> levitte> Doing siphash for 3s on 16 size blocks: 22488748 siphash's in 
> 3.00s
> levitte> Doing siphash for 3s on 64 size blocks: 10485674 siphash's in 
> 3.00s
> levitte> Doing siphash for 3s on 256 size blocks: 3320898 siphash's in 
> 3.00s
> levitte> Doing siphash for 3s on 1024 size blocks: 894647 siphash's in 
> 3.00s
> levitte> Doing siphash for 3s on 8192 size blocks: 114170 siphash's in 
> 3.00s
> levitte> Doing siphash for 3s on 16384 size blocks: 57151 siphash's in 
> 3.00s
> levitte> OpenSSL 1.1.1-dev  xx XXX 
> levitte> built on: reproducible build, date unspecified
> levitte> options:bn(64,64) rc4(16x,int) des(int) aes(partial) idea(int) 
> blowfish(ptr) 
> levitte> compiler: gcc -DDSO_DLFCN -DHAVE_DLFCN_H -DOPENSSL_THREADS 
> -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 
> -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM 
> -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM 
> -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM 
> -DOPENSSLDIR="\"/usr/local/ssl\"" 
> -DENGINESDIR="\"/usr/local/lib/engines-1.1\""  -DDEBUG_UNUSED -Wswitch 
> -DPEDANTIC -pedantic -Wno-long-long -Wall -Wsign-compare -Wmissing-prototypes 
> -Wshadow -Wformat -Wtype-limits -Werror -Wa,--noexecstack
> levitte> The 'numbers' are in 1000s of bytes per second processed.
> levitte> type 16 bytes 64 bytes256 bytes   1024 bytes 
>   8192 bytes  16384 bytes
> levitte> lhash   147387.67k   147940.82k   144937.73k   
> 147177.81k   147095.55k   147679.91k
> levitte> siphash 119939.99k   223694.38k   283383.30k   
> 305372.84k   311760.21k   312120.66k
> levitte> 
> levitte> So it seems that for short strings, OPENSSL_LH_strhash (*) wins some,
> levitte> while siphash wins big for larger strings.
> levitte> 
> levitte> I have no idea how they compare with regard to distribution (which,
> levitte> considering I ask for the same size output from both, should be the
> levitte> main factor that affects the sensitivity to hash flooding)...
> levitte> 
> levitte> Our use of OPENSSL_LH_strhash() is with configuration sections and
> levitte> names, ASN.1 object names and the function names in the openssl app.
> levitte> All our other uses of lhash use their own hashing functions, and are
> levitte> usually very short still (definitely less than 16 bytes).
> levitte> 
> levitte> My conclusion is that performance-wise, siphash doesn't give us any
> levitte> advantage over OpenSSL_LH_strhash for our uses.
> levitte> 
> levitte> Cheers,
> levitte> Richard
> levitte> 
> levitte> (*) Strictly speaking, it's a modified version that takes a length 
> and
> levitte> tolerates all 8-bit bytes,

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Richard Levitte 
In message <001901d26bed$d3746ed0$7a5d4c70$@sa...@free.fr> on Wed, 11 Jan 2017 
10:33:53 +0100, "Michel"  said:

michel.sales> And what about using FNV or CityHash ?
michel.sales> 
michel.sales> 
https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function

I'm a little worried about the zero hash value issue mentioned here:

https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function#Non-cryptographic_hash

michel.sales> https://en.wikipedia.org/wiki/CityHash

Google has replaced that with FarmHash according to that page...

-- 
Richard Levitte levi...@openssl.org
OpenSSL Project http://www.openssl.org/~levitte/
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

[openssl-dev] Build fail when configured using no-nextprotoneg

2017-01-11 Thread Michel 
Can we assume it is temporary due to "the process of transitioning from NPN
to ALPN" mentioned in ssl_locl.h ?

Regards,

Michel.

 

cl  /I "." /I "include" -DDSO_WIN32 -DOPENSSL_THREADS -DOPENSSL_NO_DYNAM

IC_ENGINE -DOPENSSL_PIC -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2
-DOPENSS

L_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM
-DRC4_A

SM -DMD5_ASM -DRMD160_ASM -DAES_ASM -DVPAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM
-DEC

P_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSL_API_COMPAT=0x1010L
"-D

ENGINESDIR=\"C:\\OpenSSL_dbg\\lib\\engines-1_1\""
"-DOPENSSLDIR=\"c:\\OpenSSL_db

g\\ssl\"" -W3 -wd4090 -Gs0 -GF -Gy -nologo -DOPENSSL_SYS_WIN32
-DWIN32_LEAN_AND_

MEAN -DL_ENDIAN -D_CRT_SECURE_NO_DEPRECATE -DUNICODE -D_UNICODE  /Od -DDEBUG
-D_

DEBUG /Zi /Fdossl_static /MT /Zl -c /Fossl\ssl_lib.obj "ssl\ssl_lib.c"

ssl_lib.c

ssl\ssl_lib.c(626): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(627): error C2039: 'alpn_len': is not a member of
''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(627): warning C4047: 'function': 'std::size_t' differs in
levels o

f indirection from 'char [14]'

ssl\ssl_lib.c(627): warning C4024: 'CRYPTO_malloc': different types for
formal a

nd actual parameter 1

ssl\ssl_lib.c(627): warning C4047: 'function': 'const char *' differs in
levels

of indirection from 'int'

ssl\ssl_lib.c(627): warning C4024: 'CRYPTO_malloc': different types for
formal a

nd actual parameter 2

ssl\ssl_lib.c(627): error C2198: 'CRYPTO_malloc': too few arguments for call

ssl\ssl_lib.c(630): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(630): error C2039: 'alpn_len': is not a member of
''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(630): error C2198: 'memcpy': too few arguments for call

ssl\ssl_lib.c(631): error C2039: 'alpn_len': is not a member of
''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2304): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2304): warning C4047: 'function': 'const char *' differs in
levels

of indirection from 'int'

ssl\ssl_lib.c(2304): warning C4024: 'CRYPTO_free': different types for
formal an

d actual parameter 2

ssl\ssl_lib.c(2304): error C2198: 'CRYPTO_free': too few arguments for call

ssl\ssl_lib.c(2305): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2306): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2310): error C2039: 'alpn_len': is not a member of
''

 

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2343): error C2039: 'alpn_select_cb': is not a member of
''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2344): error C2039: 'alpn_select_cb_arg': is not a member of
''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2622): error C2039: 'alpn': is not a member of ''

e:\devsrc\openssl.git\ssl\ssl_locl.h(782): note: see declaration of
''

ssl\ssl_lib.c(2622): warning C4047: 'function': 'const char *' differs in
levels

of indirection from 'int'

ssl\ssl_lib.c(2622): warning C4024: 'CRYPTO_free': different types for
formal an

d actual parameter 2

ssl\ssl_lib.c(2622): error C2198: 'CRYPTO_free': too few arguments for call

NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio
14.0

\VC\BIN\cl.EXE"' : return code '0x2'

Stop.

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Tomas Mraz 
On Wed, 2017-01-11 at 03:13 +, Salz, Rich wrote:
> The needs for OpenSSL's LHASH are exactly what SipHash was designed
> for: fast on short strings.
> OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code is
> commented out.
> Yes, performance tests would greatly inform the decision.

+1

Is there really no use of LHASH tables in OpenSSL where an attacker
attempting a DoS attack can control the contents of the tables? If you
are reasonably sure that there is no such occurrence or that the number
of entries attacker can insert into such table is severally limited by
other means then perhaps it really makes no sense to replace the
existing algorithm. But we need to know this first.

-- 
Tomas Mraz
No matter how far down the wrong road you've gone, turn back.
  Turkish proverb
(You'll never know whether the road is wrong though.)

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Michel 
And what about using FNV or CityHash ?

https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function
https://en.wikipedia.org/wiki/CityHash


-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] Build fail when configured using no-nextprotoneg

2017-01-11 Thread Matt Caswell 


On 11/01/17 08:58, Michel wrote:
> Can we assume it is temporary due to “the process of transitioning from
> NPN to ALPN” mentioned in ssl_locl.h ?

Its a bug:

https://github.com/openssl/openssl/pull/2212

Matt


-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Peter Waltenberg 
And the reason I said you certainly don't need a keyed hash ?

Behaviour of the hash function will change with key and in some cases 
performance would degenerate to that of a linked list. (Ouch). And since 
the obvious thing to do is use a random key, OpenSSL's performance would 
get *very* erratic.

Simpler functions than cryptographic hashes will almost certainly yield 
better results here. I note someone further up the thread someone else has 
pointed that out. 

Peter




From:   "Salz, Rich" 
To: "openssl-dev@openssl.org" 
Date:   11/01/2017 13:14
Subject:Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?
Sent by:"openssl-dev" 



The needs for OpenSSL's LHASH are exactly what SipHash was designed for: 
fast on short strings.
OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code is 
commented out.
Yes, performance tests would greatly inform the decision.
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev





-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread J. J. Farrell 
Are the issues you raise true of SipHash, given that a prime motivator 
for its design was generating hash tables for short inputs while being 
secure against hash flooding attacks? It achieves this with the 
performance of a portable C implementation the order of four times 
faster than MD5, and not much slower than other modern hash algorithms.


I'd have thought the main thing to consider is whether or not there is 
any practical way a hash flooding attack could be used against OpenSSL's 
hash tables, and it sounds like there isn't. In that case, the fastest 
algorithm for the usage patterns would be best.


Regards,
  jjf

On 11/01/2017 22:25, Peter Waltenberg wrote:

And the reason I said you certainly don't need a keyed hash ?

Behaviour of the hash function will change with key and in some cases 
performance would degenerate to that of a linked list. (Ouch). And 
since the obvious thing to do is use a random key, OpenSSL's 
performance would get *very* erratic.


Simpler functions than cryptographic hashes will almost certainly 
yield better results here. I note someone further up the thread 
someone else has pointed that out.


Peter

From: "Salz, Rich" 
To: "openssl-dev@openssl.org" 
Date: 11/01/2017 13:14
Subject: Re: [openssl-dev] use SIPhash for OPENSSL_LH_strhash?
Sent by: "openssl-dev" 


The needs for OpenSSL's LHASH are exactly what SipHash was designed 
for: fast on short strings.
OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code is 
commented out.

Yes, performance tests would greatly inform the decision.


--
J. J. Farrell
Not speaking for Oracle

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [EXTERNAL] Re: use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Sands, Daniel 
With a small number of buckets, it seems to me that no hash algo will
make you safe from a flooding attack.  You can simply generate your
hashes locally using whichever algo the server uses, and only send those
that fit into your attack scheme.  The data could even be pre-generated.

The only way to guard against a flood that makes sense to me is to limit
the number of items that can be accepted before deciding you're being
trolled.

On Wed, 2017-01-11 at 23:29 +, J. J. Farrell wrote:
> Are the issues you raise true of SipHash, given that a prime motivator
> for its design was generating hash tables for short inputs while being
> secure against hash flooding attacks? It achieves this with the
> performance of a portable C implementation the order of four times
> faster than MD5, and not much slower than other modern hash
> algorithms.
> 
> I'd have thought the main thing to consider is whether or not there is
> any practical way a hash flooding attack could be used against
> OpenSSL's hash tables, and it sounds like there isn't. In that case,
> the fastest algorithm for the usage patterns would be best.
> 
> Regards,
>   jjf
> 
> On 11/01/2017 22:25, Peter Waltenberg wrote:
> 
> > And the reason I said you certainly don't need a keyed hash ?
> > 
> > Behaviour of the hash function will change with key and in some
> > cases performance would degenerate to that of a linked list. (Ouch).
> > And since the obvious thing to do is use a random key, OpenSSL's
> > performance would get *very* erratic.
> > 
> > Simpler functions than cryptographic hashes will almost certainly
> > yield better results here. I note someone further up the thread
> > someone else has pointed that out. 
> > 
> > Peter
> > 
> > From:"Salz, Rich" 
> > To:"openssl-dev@openssl.org" 
> > Date:11/01/2017 13:14
> > Subject:Re: [openssl-dev] use SIPhash for
> > OPENSSL_LH_strhash?
> > Sent by:"openssl-dev" 
> > 
> > 
> > 
> > The needs for OpenSSL's LHASH are exactly what SipHash was designed
> > for: fast on short strings.
> > OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code
> > is commented out.
> > Yes, performance tests would greatly inform the decision.
> 
> -- 
> J. J. Farrell
> Not speaking for Oracle
> -- 
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] [EXTERNAL] Re: use SIPhash for OPENSSL_LH_strhash?

2017-01-11 Thread Jeremy Farrell 
For something like SipHash, knowing "whichever algo the server uses" 
effectively implies knowing the 128-bit random key currently being used 
for the hash table in question.


Regards,
  jjf

On 12/01/2017 00:39, Sands, Daniel wrote:

With a small number of buckets, it seems to me that no hash algo will
make you safe from a flooding attack.  You can simply generate your
hashes locally using whichever algo the server uses, and only send those
that fit into your attack scheme.  The data could even be pre-generated.

The only way to guard against a flood that makes sense to me is to limit
the number of items that can be accepted before deciding you're being
trolled.

On Wed, 2017-01-11 at 23:29 +, J. J. Farrell wrote:

Are the issues you raise true of SipHash, given that a prime motivator
for its design was generating hash tables for short inputs while being
secure against hash flooding attacks? It achieves this with the
performance of a portable C implementation the order of four times
faster than MD5, and not much slower than other modern hash
algorithms.

I'd have thought the main thing to consider is whether or not there is
any practical way a hash flooding attack could be used against
OpenSSL's hash tables, and it sounds like there isn't. In that case,
the fastest algorithm for the usage patterns would be best.

Regards,
   jjf

On 11/01/2017 22:25, Peter Waltenberg wrote:


And the reason I said you certainly don't need a keyed hash ?

Behaviour of the hash function will change with key and in some
cases performance would degenerate to that of a linked list. (Ouch).
And since the obvious thing to do is use a random key, OpenSSL's
performance would get *very* erratic.

Simpler functions than cryptographic hashes will almost certainly
yield better results here. I note someone further up the thread
someone else has pointed that out.

Peter

From:"Salz, Rich" 
To:"openssl-dev@openssl.org" 
Date:11/01/2017 13:14
Subject:Re: [openssl-dev] use SIPhash for
OPENSSL_LH_strhash?
Sent by:"openssl-dev" 



The needs for OpenSSL's LHASH are exactly what SipHash was designed
for: fast on short strings.
OpenSSL's hash currently *does not* call MD5 or SHA1; the MD5 code
is commented out.
Yes, performance tests would greatly inform the decision.

--
J. J. Farrell
Not speaking for Oracle
--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev


--
J. J. Farrell
w: +44 161 493 4838

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev