During our memory stress testing we discovered a bug in the BN_generate_prime() function. In the case that the creation of a new BN_CTX fails (line 143), execution drops down into the error path starting on line 204. In the error path, a call is made (line 206) to BN_free() on the local variable 't'. Since 't' isn't initialized (line 151) until after the successful creation of BN_CTX, 't' may contain garbage (it does in our environment, vxWorks) and the call within BN_free() to free memory causes grief.
For the interim, we have moved the call to BN_init() on 't' to come before the attempt to allocate the BN_CTX structure. In this manner, BN_free() does not attempt to free data on garbage pointers.
Cheers,
MS
Mauricio Sanchez
R&D Engineer
Procurve Networking Business
Hewlett-Packard Company
8000 Foothills Blvd. MS5557
Roseville, CA 95747-5557
Email:[EMAIL PROTECTED]
