Hi Grant,
On 05/15/2017 12:14 PM, Emeric Brun wrote:
> On 05/13/2017 01:14 AM, Grant Zhang wrote:
>>
>>> On May 10, 2017, at 04:51, Emeric Brun <[email protected]> wrote:
>>>
>>>> It looks like the main process stalls at DH_free(local_dh_1024) (part of
>>>> __ssl_sock_deinit). Not sure why but I will debug and report back.
>>>>
>>>> Thanks,
>>>
>>> I experienced the same issue (stalled on a futex) if i run haproxy in
>>> foreground and trying to kill it with kill -USR1.
>>>
>>> With this conf (dh param and ssl-async are disabled)
>>> global
>>> # tune.ssl.default-dh-param 2048
>>> ssl-engine qat
>>> # ssl-async
>>> nbproc 1
>>
>> It looks like that the stall on futex issue is related to DH_free() calling
>> ENGINE_finish in openssl 1.1:
>> https://github.com/openssl/openssl/blob/master/crypto/dh/dh_lib.c#L109
>> (gdb) bt
>> #0 __lll_lock_wait () at
>> ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:132
>> #1 0x00007fa1582c5571 in pthread_rwlock_wrlock ()
>> at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_rwlock_wrlock.S:118
>> #2 0x00007fa158a58559 in CRYPTO_THREAD_write_lock () from
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #3 0x00007fa1589d8800 in ENGINE_finish () from
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #4 0x00007fa158975e76 in DH_free () from
>> /tmp/openssl_1.1.0_install/lib/libcrypto.so.1.1
>> #5 0x0000000000417c78 in free_dh () at src/ssl_sock.c:7905
>> #6 0x00007fa1591d58ce in _dl_fini () at dl-fini.c:254
>> #7 0x00007fa158512511 in __run_exit_handlers (status=0,
>> listp=0x7fa15888e688, run_list_atexit=true)
>> at exit.c:78
>> #8 0x00007fa158512595 in __GI_exit (status=<optimized out>) at exit.c:100
>> #9 0x0000000000408814 in main (argc=4, argv=0x7ffe72188548) at
>> src/haproxy.c:2235
>>
>> Openssl 1.1 has changed the way ENGINE_cleanup works:
>> https://www.openssl.org/docs/man1.1.0/crypto/ENGINE_cleanup.html
>> "From OpenSSL 1.1.0 it is no longer necessary to explicitly call
>> ENGINE_cleanup and this function is deprecated. Cleanup automatically takes
>> place at program exit."
>>
>> I suppose by the time the destructor __ssl_sock_deinit is called,
>> engine-related cleanup are already done by openssl and ENGINE_finish (from
>> DH_free) stalls on a non-existing write lock.
>>
>> I have a workaround which moves the DH_free logic out of the destructor
>> __ssl_sock_deinit, and right before process exit. With the workaround I no
>> longer see the stall issue. I am not sure whether it is optimal solution
>> though. Let me know.
>
> What does it look like?
>
> The issue is very similar:
> https://mta.openssl.org/pipermail/openssl-dev/2016-March/005909.html
>
>
>>
>> Thanks,
>>
>> Grant
> R,
> Emeric
>
>
Reading the changelog openssl-1.1.0e changelog:
*) Make various cleanup routines no-ops and mark them as deprecated. Most
global cleanup functions are no longer required because they are handled
via auto-deinit (see OPENSSL_init_crypto and OPENSSL_init_ssl man pages).
Explicitly de-initing can cause problems (e.g. where a library that uses
OpenSSL de-inits, but an application is still using it). The affected
functions are CONF_modules_free(), ENGINE_cleanup(), OBJ_cleanup(),
EVP_cleanup(), BIO_sock_cleanup(), CRYPTO_cleanup_all_ex_data(),
RAND_cleanup(), SSL_COMP_free_compression_methods(), ERR_free_strings() and
COMP_zlib_cleanup().
[Matt Caswell
and reading the OPENSSL_init_ssl/OPENSSL_init_crypto man pages.
We may remove all openssl uninit calls from the ssl_sock.c destructor for
openssl v >= 1.1.
R,
Emeric
