Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Benjamin Kaduk via openssl-dev]

On 05/12/2017 03:05 PM, Blumenthal, Uri - 0553 - MITLL wrote:
>
> I’m not sure. It may well be that it “simply” takes all the memory
> over, so there is no way for anything else to start and do the clean-up…
>

Hmm, I wonder if a top(1) started before the tests would keep running,
in the case that it "simply" takes all the memory over.

>  
>
> From just looking at the code, the only question that comes to mind is
> whether you have a 32- or 64-bit size_t in the build environment in
> question, which is unlikely to cause a eureka moment :(
>
>  
>
> I can tell you that size_t is 64-bit here. It’s certainly not an
> “eureka” moment for me.
>
>  
>
> Some other information to check: do you have MAP_ANON defined by your
> mman.h?
>

mman.h != mmap.h (consult 'man mmap' for authoritative header).

> Todd> Yes, it’s likely this is due to the amount of memory available
> in the machine. I tried to use reasonable values, but apparently not
> reasonable enough
>
>  
>
> Yep. In case it matters, my machine has 16GB of RAM (and runs a ton of
> stuff, besides these tests :).
>
>

Thanks.

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Short, Todd via openssl-dev]

It’s trying to reserve 1<<34 bytes of memory… there goes your 16GB...
--
-Todd Short
// tsh...@akamai.com
// "One if by land, two if by sea, three if by the Internet."

> On May 12, 2017, at 4:05 PM, Blumenthal, Uri - 0553 - MITLL  
> wrote:
> 
> Todd> Yes, it’s likely this is due to the amount of memory available in the 
> machine. I tried to use reasonable values, but apparently not reasonable 
> enough
>  
> Yep. In case it matters, my machine has 16GB of RAM (and runs a ton of stuff, 
> besides these tests :).
> -- 
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev 
> 



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Short, Todd via openssl-dev]

Uri:

Look at https://github.com/openssl/openssl/pull/3455 


I limited the test that hung your machine to Linux.

Rich: this removes the OpenSSL_assert() you see.

--
-Todd Short
// tsh...@akamai.com
// "One if by land, two if by sea, three if by the Internet."

> On May 12, 2017, at 4:46 PM, Short, Todd via openssl-dev 
>  wrote:
> 
> It’s trying to reserve 1<<34 bytes of memory… there goes your 16GB...
> --
> -Todd Short
> // tsh...@akamai.com 
> // "One if by land, two if by sea, three if by the Internet."
> 
>> On May 12, 2017, at 4:05 PM, Blumenthal, Uri - 0553 - MITLL > > wrote:
>> 
>> Todd> Yes, it’s likely this is due to the amount of memory available in the 
>> machine. I tried to use reasonable values, but apparently not reasonable 
>> enough
>>  
>> Yep. In case it matters, my machine has 16GB of RAM (and runs a ton of 
>> stuff, besides these tests :).
>> -- 
>> openssl-dev mailing list
>> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev 
>> 
> 
> -- 
> openssl-dev mailing list
> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Blumenthal, Uri - 0553 - MITLL]

The obvious candidate for closer inspection is a few commits previous,

commit 7031ddac94d0ae616d1b0670263a9265ce672cd2
Author: Todd Short 
Date:   Thu May 11 15:48:10 2017 -0400

which adds test cases intended to trigger the edge cases being fixed.



Point well-taken. ;-)

 

It seems that there should also be a bug report against OS X, as regular 
userspace code running as non-root should not be able to hang a machine like 
that.

I’m not sure. It may well be that it “simply” takes all the memory over, so 
there is no way for anything else to start and do the clean-up…

 

>From just looking at the code, the only question that comes to mind is whether 
>you have a 32- or 64-bit size_t in the build environment in question, which is 
>unlikely to cause a eureka moment :(

 

I can tell you that size_t is 64-bit here. It’s certainly not an “eureka” 
moment for me.

 

Some other information to check: do you have MAP_ANON defined by your mman.h?


/**

 * @addtogroup apr_platform

 * @ingroup APR 

 * @{

 */

 

#define APR_HAVE_SHMEM_MMAP_TMP 1

#define APR_HAVE_SHMEM_MMAP_SHM 1

#define APR_HAVE_SHMEM_MMAP_ZERO0

#define APR_HAVE_SHMEM_SHMGET_ANON  1

#define APR_HAVE_SHMEM_SHMGET   1

#define APR_HAVE_SHMEM_MMAP_ANON1

#define APR_HAVE_SHMEM_BEOS 0

 

#define APR_USE_SHMEM_MMAP_TMP 0

#define APR_USE_SHMEM_MMAP_SHM 0

#define APR_USE_SHMEM_MMAP_ZERO0

#define APR_USE_SHMEM_SHMGET_ANON  0

#define APR_USE_SHMEM_SHMGET   1

#define APR_USE_SHMEM_MMAP_ANON1

#define APR_USE_SHMEM_BEOS 0

 

And this system does not seem to have a straight “mmap.h”. The above is from 
/usr/include/apr-1/apr_mmap.h file.

 

Do you have a /dev/zero that can be opened read/write?

 

Yep:

 

$ ll /dev/zero

crw-rw-rw-  1 root  wheel3,   3 May 12 14:14 /dev/zero

$ 

 

Todd> Yes, it’s likely this is due to the amount of memory available in the 
machine. I tried to use reasonable values, but apparently not reasonable enough

 

Yep. In case it matters, my machine has 16GB of RAM (and runs a ton of stuff, 
besides these tests :).



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Benjamin Kaduk via openssl-dev]

On 05/12/2017 02:05 PM, Blumenthal, Uri - 0553 - MITLL wrote:
> On 5/12/17, 2:49 PM, "openssl-dev on behalf of Benjamin Kaduk via 
> openssl-dev"  openssl-dev@openssl.org> wrote:
>
> ➢ I’m sorry to report that in the current OpenSSL 1.1 master running “make 
> test”
> ➢  freezes up the machine. Mac OS X 10.11.6, Xcode-8.2, current Github 
> master. 
> ➢ Here’s the configuration:
>
>   A commit hash would be more useful than "current github master"
>
> I thought you know what’s in the current master right now. But here’s the 
> last few hashes for your pleasure:

Well, you're right to think that I do, and I do trust you to be accurate
when you make that claim.  But there are many people for which I would
not fully trust the veracity of such a claim, the commit hash is
completely unambiguous, and it is a whole lot easier for those poor
folks reading this thread in the archive two years from now who are
trying to track down a similar-looking issue.  So, on the whole, I
recommend always using commit hashes, with an optional annotation of how
it relates to a branch.

> $ git log
> commit 80a2fc4100daf6f1001eee33ef2f9b9eee05bedf (HEAD -> master, 
> origin/master, origin/HEAD)
> Author: Todd Short 
> Date:   Wed May 10 11:44:55 2017 -0400
>
> Clean up SSL_OP_* a bit
> 
> Reviewed-by: Matt Caswell 
> Reviewed-by: Rich Salz 
> (Merged from https://github.com/openssl/openssl/pull/3439)
>
> commit 33242d9d79e7f06151e905b83dc8f995006fa7cd
> Author: Rich Salz 
> Date:   Thu May 11 20:42:32 2017 -0400
>
> Use scalar, not length; fixes test_evp
> 
> Reviewed-by: Stephen Henson 
> Reviewed-by: Richard Levitte 
> (Merged from https://github.com/openssl/openssl/pull/3452)
>
>
>   I can understand not wanting to have to power-cycle the machine again,
>   but the 'make TESTS=test_secmem V=1 test' output (or some 
> dtruss/similar)
>   would be helpful in tracking things down.

The obvious candidate for closer inspection is a few commits previous,

commit 7031ddac94d0ae616d1b0670263a9265ce672cd2
Author: Todd Short 
Date:   Thu May 11 15:48:10 2017 -0400

Fix infinite loops in secure memory allocation.
   
Issue 1:
   
sh.bittable_size is a size_t but i is and int, which can result in
freelist == -1 if sh.bittable_size exceeds an int.
   
This seems to result in an OPENSSL_assert due to invalid allocation
size, so maybe that is "ok."
   
Worse, if sh.bittable_size is exactly 1<<31, then this becomes an
infinite loop (because 1<<31 is a negative int, so it can be shifted
right forever and sticks at -1).
   
Issue 2:
   
CRYPTO_secure_malloc_init() sets secure_mem_initialized=1 even when
sh_init() returns 0.
   
If sh_init() fails, we end up with secure_mem_initialized=1 but
sh.minsize=0. If you then call secure_malloc(), which then calls,
sh_malloc(), this then enters an infite loop since 0 << anything will
never be larger than size.
   
Issue 3:
   
That same sh_malloc loop will loop forever for a size greater
than size_t/2 because i will proceed (assuming sh.minsize=16):
i=16, 32, 64, ..., size_t/8, size_t/4, size_t/2, 0, 0, 0, 0, 
This sequence will never be larger than "size".
   
Reviewed-by: Rich Salz 
Reviewed-by: Richard Levitte 
(Merged from https://github.com/openssl/openssl/pull/3449)

which adds test cases intended to trigger the edge cases being fixed.

> Sorry.
>
>   How locked up is the machine?  Can you get memory usage stats or is it 
> completely unresponsive?
>
> Completely unresponsive. Totally. No memory usage. The only thing that works 
> at this point is the power button.

It seems that there should also be a bug report against OS X, as regular
userspace code running as non-root should not be able to hang a machine
like that.

>From just looking at the code, the only question that comes to mind is
whether you have a 32- or 64-bit size_t in the build environment in
question, which is unlikely to cause a eureka moment :(

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

[openssl-dev] 90-test_secmem.t hangs the machine for good

[Blumenthal, Uri - 0553 - MITLL]

I’m sorry to report that in the current OpenSSL 1.1 master running “make test” 
freezes up the machine. Mac OS X 10.11.6, Xcode-8.2, current Github master. 
Here’s the configuration:

 

./Configure darwin64-x86_64-cc enable-threads enable-shared enable-zlib 
enable-ec_nistp_64_gcc_128 enable-rfc3779 enable-rc5 enable-tls1_3 
--prefix=/Users/uri/src/openssl-1.1 --openssldir=/Users/uri/src/openssl-1.1/etc

 

Then of course “make depend && make clean && make all && make test”

 

../test/recipes/90-test_ige.t . ok   

../test/recipes/90-test_memleak.t . ok   

../test/recipes/90-test_overhead.t  skipped: Only supported in 
no-shared builds

../test/recipes/90-test_secmem.t .. 

 

At this point the machine has to be power-cycled.

— 

Regards,

Uri

 



smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Benjamin Kaduk via openssl-dev]

On 05/12/2017 01:34 PM, Blumenthal, Uri - 0553 - MITLL wrote:
>
> I’m sorry to report that in the current OpenSSL 1.1 master running
> “make test” freezes up the machine. Mac OS X 10.11.6, Xcode-8.2,
> current Github master. Here’s the configuration:
>

A commit hash would be more useful than "current github master"

>  
>
> ./Configure darwin64-x86_64-cc enable-threads enable-shared
> enable-zlib enable-ec_nistp_64_gcc_128 enable-rfc3779 enable-rc5
> enable-tls1_3 --prefix=/Users/uri/src/openssl-1.1
> --openssldir=/Users/uri/src/openssl-1.1/etc
>
>  
>
> Then of course “make depend && make clean && make all && make test”
>
>  
>
> ../test/recipes/90-test_ige.t . ok   
>
> ../test/recipes/90-test_memleak.t . ok   
>
> ../test/recipes/90-test_overhead.t  skipped: Only
> supported in no-shared builds
>
> ../test/recipes/90-test_secmem.t .. 
>
>  
>
> At this point the machine has to be power-cycled.
>
> —
>


I can understand not wanting to have to power-cycle the machine again,
but the 'make TESTS=test_secmem V=1 test' output (or some
dtruss/similar) would be helpful in tracking things down.

How locked up is the machine?  Can you get memory usage stats or is it
completely unresponsive?

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Salz, Rich via openssl-dev]

Current master 1d0f116 fails on my machine.

../test/recipes/90-test_secmem.t ..
1..1
# Subtest: ./secmemtest
1..1
# INFO:  @ test/secmemtest.c:65
# Possible infinite loop: allocate more than available

# INFO:  @ test/secmemtest.c:71
# Possible infinite loop: small arena

# INFO:  @ test/secmemtest.c:79
# Possible infinite loop: 1<<31 limit

crypto/mem_sec.c:428: OpenSSL internal error: assertion failed: sh.map_result 
!= MAP_FAILED
../util/shlib_wrap.sh ./secmemtest => 134
not ok 1 - running secmemtest

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Blumenthal, Uri - 0553 - MITLL]

On 5/12/17, 2:49 PM, "openssl-dev on behalf of Benjamin Kaduk via openssl-dev" 
 wrote:

➢ I’m sorry to report that in the current OpenSSL 1.1 master running “make test”
➢  freezes up the machine. Mac OS X 10.11.6, Xcode-8.2, current Github master. 
➢ Here’s the configuration:

  A commit hash would be more useful than "current github master"

I thought you know what’s in the current master right now. But here’s the last 
few hashes for your pleasure:

$ git log
commit 80a2fc4100daf6f1001eee33ef2f9b9eee05bedf (HEAD -> master, origin/master, 
origin/HEAD)
Author: Todd Short 
Date:   Wed May 10 11:44:55 2017 -0400

Clean up SSL_OP_* a bit

Reviewed-by: Matt Caswell 
Reviewed-by: Rich Salz 
(Merged from https://github.com/openssl/openssl/pull/3439)

commit 33242d9d79e7f06151e905b83dc8f995006fa7cd
Author: Rich Salz 
Date:   Thu May 11 20:42:32 2017 -0400

Use scalar, not length; fixes test_evp

Reviewed-by: Stephen Henson 
Reviewed-by: Richard Levitte 
(Merged from https://github.com/openssl/openssl/pull/3452)


  I can understand not wanting to have to power-cycle the machine again,
  but the 'make TESTS=test_secmem V=1 test' output (or some dtruss/similar)
  would be helpful in tracking things down.

Sorry.

  How locked up is the machine?  Can you get memory usage stats or is it 
completely unresponsive?

Completely unresponsive. Totally. No memory usage. The only thing that works at 
this point is the power button.


smime.p7s
Description: S/MIME cryptographic signature
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Benjamin Kaduk via openssl-dev]

On 05/12/2017 02:15 PM, Benjamin Kaduk via openssl-dev wrote:
> From just looking at the code, the only question that comes to mind is
> whether you have a 32- or 64-bit size_t in the build environment in
> question, which is unlikely to cause a eureka moment :(

(The test runs happily on my Ubuntu Trusty-ish machine, FWIW.)

Some other information to check: do you have MAP_ANON defined by your
mman.h?
Do you have a /dev/zero that can be opened read/write?

-Ben
-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

Re: [openssl-dev] 90-test_secmem.t hangs the machine for good

[Short, Todd via openssl-dev]

Yes, it’s likely this is due to the amount of memory available in the machine. 
I tried to use reasonable values, but apparently not reasonable enough.

This is certainly a case where we’re trying to stretch the limits of the 
hardware; so it may not be an appropriate test for all hardware.

In the case below, the call to mmap() failed, and there’s an OPENSSL_assert() 
there that’s probably not necessary; the error condition checked by the assert 
is handled safely.

This is different than the problem > 
was seeing, which is likely a condition where memory is being maxed out.

--
-Todd Short
// tsh...@akamai.com
// "One if by land, two if by sea, three if by the Internet."

On May 12, 2017, at 2:51 PM, Salz, Rich via openssl-dev 
> wrote:

Current master 1d0f116 fails on my machine.

../test/recipes/90-test_secmem.t ..
1..1
   # Subtest: ./secmemtest
   1..1
   # INFO:  @ test/secmemtest.c:65
   # Possible infinite loop: allocate more than available

   # INFO:  @ test/secmemtest.c:71
   # Possible infinite loop: small arena

   # INFO:  @ test/secmemtest.c:79
   # Possible infinite loop: 1<<31 limit

crypto/mem_sec.c:428: OpenSSL internal error: assertion failed: sh.map_result 
!= MAP_FAILED
../util/shlib_wrap.sh ./secmemtest => 134
not ok 1 - running secmemtest

--
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev

-- 
openssl-dev mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev