[openssl-users] Assertion in ssl_free_wbio_buffer() fails after unfinished handshake since OpenSSL 1.1.0

[Marcus . Schafheutle]

Hello,
 
we encountered a strange behaviour of OpenSSL 1.1.0 when our test with "sslscan" provokes an unfinished 
handshake.

 

Our asynchronous communication approach uses memory BIOs in order to read/write data from other 
communication layers into the SSL object. After the read/write operations are done the BIOs are freed.

 

Our example code is basically as follows:
---
err = SSL_accept (ssl); 

 

BIO *rBIO = BIO_new(BIO_s_mem());
BIO *wBIO = BIO_new(BIO_s_mem());
    
SSL_set_bio(ssl, rBIO, wBIO); 

 

// ... operate with read / write BIOs and SSL_read/SSL_write

 

SSL_set_bio(ssl, NULL, NULL); // free BIOs when finished

 

SSL_free(ssl);
---

 

When calling SSL_free() after a complete handshake, everything is fine, but when calling SSL_free() after an 
unfinished handshake, the assertion "assert(s->wbio != NULL);" in ssl_free_wbio_buffer() fails:
---
void ssl_free_wbio_buffer(SSL *s)
{
    /* callers ensure s is never null */
    if (s->bbio == NULL)
        return;
    s->wbio = BIO_pop(s->wbio);
    assert(s->wbio != NULL);            /* <- this assertion fails! */
    BIO_free(s->bbio);
    s->bbio = NULL;
}
---

 

With a complete handshake the new attribute “bbio” is freed by calling the method tls_finish_handshake(), 
which itself calls ssl_free_wbio_buffer(). When the handshake is not finished successfully, the “bbio” is not freed, and 
therefore when calling SSL_free() the assert in ssl_free_wbio_buffer() fails.

 

The problem exists since OpenSSL 1.1.0 - there were no problems with OpenSSL 1.0.2x.

 

Is this a faulty behavior of the OpenSSL 1.1.0? 
 
Thank you for your help!
 
Best regards,
Marcus
 

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users