Adar Dembo created KUDU-2700:
--------------------------------
Summary: Memory leak in kudu-tool-test when using OpenSSL 1.1
Key: KUDU-2700
URL: https://issues.apache.org/jira/browse/KUDU-2700
Project: Kudu
Issue Type: Bug
Components: rpc
Affects Versions: 1.8.0
Reporter: Adar Dembo
KUDU-1889 added support for OpenSSL 1.1, but one of the outstanding issues is a
memory leak reported by LSAN when running tests that use the CLI to connect to
a remote server. The memory leak looks like this:
{noformat}
=================================================================
==31426==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 584 byte(s) in 1 object(s) allocated from:
#0 0x620470 in __interceptor_malloc
/home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
b/asan/asan_malloc_linux.cc:88
#1 0x7fa17eba94b8 in CRYPTO_zalloc
(/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
#2 0x7fa17eb8afdd in ERR_get_state
(/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fdd)
#3 0x7fa17eb8b213 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
#4 0x7fa17f44e28d in
kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
const*) ../src/kudu/security/openssl_util.h:201:5
#5 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int,
long*) ../src/kudu/security/tls_socket
.cc:81:3
#6 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&)
../src/kudu/rpc/transfer.cc:218:26
#7 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int)
../src/kudu/rpc/connection.cc:791:31
#8 0x7fa17bbaf698 in ev_invoke_pending
/home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
#9 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*)
../src/kudu/rpc/reactor.cc:176:3
#10 0x7fa17bbb05a0 in ev_run
/home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
#11 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread()
../src/kudu/rpc/reactor.cc:477:9
#12 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
kudu::rpc::ReactorThread>, boost::_bi::lis
t1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()()
../thirdparty/installed/uninstrumented/include/boo
st/bind/bind.hpp:1222:16
#13 0x7fa180050b5b in boost::function0<void>::operator()() const
../thirdparty/installed/uninstrumented/include/
boost/function/function_template.hpp:770:14
#14 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*)
../src/kudu/util/thread.cc:657:3
#15 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long,
__sanitizer::atomic_uintptr_t*) /home/adar/Source
/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
#16 0x7fa1816e06da in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#17 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
Direct leak of 8 byte(s) in 1 object(s) allocated from:
#0 0x620470 in __interceptor_malloc
/home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/li
b/asan/asan_malloc_linux.cc:88
#1 0x7fa17eba94b8 in CRYPTO_zalloc
(/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1574b8)
#2 0x7fa17eba5925 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x153925)
#3 0x7fa17eba60d1 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x1540d1)
#4 0x7fa17eb8afef in ERR_get_state
(/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x138fef)
#5 0x7fa17eb8b213 (/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1+0x139213)
#6 0x7fa17f44e28d in
kudu::security::internal::ScopedCheckNoPendingSSLErrors::ScopedCheckNoPendingSSLErrors(char
const*) ../src/kudu/security/openssl_util.h:201:5
#7 0x7fa17f49fd51 in kudu::security::TlsSocket::Writev(iovec const*, int,
long*) ../src/kudu/security/tls_socket
.cc:81:3
#8 0x7fa18010321a in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&)
../src/kudu/rpc/transfer.cc:218:26
#9 0x7fa17fff8eb6 in kudu::rpc::Connection::WriteHandler(ev::io&, int)
../src/kudu/rpc/connection.cc:791:31
#10 0x7fa17bbaf698 in ev_invoke_pending
/home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3155
#11 0x7fa1800616c9 in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*)
../src/kudu/rpc/reactor.cc:176:3
#12 0x7fa17bbb05a0 in ev_run
/home/adar/Source/kudu/thirdparty/src/libev-4.20/ev.c:3555
#13 0x7fa180061a18 in kudu::rpc::ReactorThread::RunThread()
../src/kudu/rpc/reactor.cc:477:9
#14 0x7fa180078d32 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
kudu::rpc::ReactorThread>,
boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> >
>::operator()()
../thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#15 0x7fa180050b5b in boost::function0<void>::operator()() const
../thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#16 0x7fa17d45057a in kudu::Thread::SuperviseThread(void*)
../src/kudu/util/thread.cc:657:3
#17 0x62e0ae in __asan::AsanThread::ThreadStart(unsigned long,
__sanitizer::atomic_uintptr_t*)
/home/adar/Source/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/asan/asan_thread.cc:259
#18 0x7fa1816e06da in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#19 0x7fa17a57388e in clone sysdeps/unix/sysv/linux/x86_64/clone.S:95
SUMMARY: AddressSanitizer: 592 byte(s) leaked in 2 allocation(s).
../../src/kudu/tools/kudu-tool-test.cc:1540: Failure
Failed
Bad status: Runtime error: /home/adar/Source/kudu/build/asan/bin/kudu: process
exited with non-zero status 1
../../src/kudu/tools/kudu-tool-test.cc:1601: Failure
Expected: RunLoadgen(1, { "--auto_database=foo", "--keep_auto_table=true" })
doesn't generate new fatal failures in the current thread.
Actual: it does.
{noformat}
I can repro this 100% of the time on my Ubuntu 18 machine by running
{{LSAN_OPTIONS=fast_unwind_on_malloc=0 bin/kudu-tool-test
--gtest_filter=*LoadgenDatabaseName*}}.
It's possible this is endemic to the particular version of OpenSSL 1.1 in
Ubuntu 18 (that's OpenSSL 1.1.0g).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
