Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-19 Thread Kurt Roeckx
On Wed, Apr 18, 2012 at 08:04:28PM +0200, Andy Polyakov via RT wrote: I've had 2 users report a crash in RC4() on x86_64. The backtrace looks like: #0 RC4 () at rc4-x86_64.s:343 #1 0x012d in ?? () #2 0x00df in ?? () #3 0x020b5660 in ?? () #4

Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-19 Thread Kurt Roeckx
On Thu, Apr 19, 2012 at 09:08:48PM +0200, Kurt Roeckx wrote: On Wed, Apr 18, 2012 at 08:04:28PM +0200, Andy Polyakov via RT wrote: I've had 2 users report a crash in RC4() on x86_64. The backtrace looks like: #0 RC4 () at rc4-x86_64.s:343 #1 0x012d in ?? () #2

Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-19 Thread Andy Polyakov
This looks simular to the AES problem with had, with a length of 0? For reference. What was going on was the RC4_set_key was generating compact key schedule on Intel legacy CPU, while rc4_md5_enc was treating as non-compact with 32-bit elements. As results it was messed up in such way that

Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-19 Thread zhu qun-ying
: Thursday, April 19, 2012 1:52:31 PM Subject: Re: [openssl.org #2792] Crash in rc4 on x86_64 This looks simular to the AES problem with had, with a length of 0? For reference. What was going on was the RC4_set_key was generating compact key schedule on Intel legacy CPU, while rc4_md5_enc

Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-19 Thread zhu qun-ying
To: openssl-dev@openssl.org Cc: Sent: Thursday, April 19, 2012 1:52:31 PM Subject: Re: [openssl.org #2792] Crash in rc4 on x86_64   This looks simular to the AES problem with had, with a length of 0? For reference. What was going on was the RC4_set_key was generating compact

Re: [openssl.org #2792] Crash in rc4 on x86_64

2012-04-18 Thread Andy Polyakov via RT
I've had 2 users report a crash in RC4() on x86_64. The backtrace looks like: #0 RC4 () at rc4-x86_64.s:343 #1 0x012d in ?? () #2 0x00df in ?? () #3 0x020b5660 in ?? () #4 0x7fc075f6a9c9 in rc4_hmac_md5_cipher (ctx=optimized out, out=0x20aae98

[openssl.org #2792] Crash in rc4 on x86_64

2012-04-17 Thread Kurt Roeckx via RT
Hi, I've had 2 users report a crash in RC4() on x86_64. The backtrace looks like: #0 RC4 () at rc4-x86_64.s:343 #1 0x012d in ?? () #2 0x00df in ?? () #3 0x020b5660 in ?? () #4 0x7fc075f6a9c9 in rc4_hmac_md5_cipher (ctx=optimized out, out=0x20aae98