Quanlong Huang created IMPALA-12915:
---------------------------------------
Summary: double-free memory issue in unifiedbetests if it's built
in dynamically linked mode
Key: IMPALA-12915
URL: https://issues.apache.org/jira/browse/IMPALA-12915
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Quanlong Huang
I have a pretty simple change that just adds one new metric in
common/thrift/metrics.json:
[https://github.com/stiga-huang/impala/commit/3972aed68428b3f307e823f49a2700c1a98aa3b3]
It breaks the clang-tidy job in the step of running "unifiedbetests
--gtest_list_tests".
The job builds Impala using the -so option which means dynamically link
executables (default is static). I'm able to reproduce the issue locally. Using
gdb, I can see the process crashed by a memory issue:
{noformat}
(gdb) bt
#0 0x0000000002ff8d6b in
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned int, int) ()
#1 0x0000000002ff93c5 in
tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned
int) ()
#2 0x00007fffef06db1f in
std::_Function_base::_Base_manager<boost::algorithm::detail::is_any_ofF<char>
>::_M_manager(std::_Any_data&, std::_Any_data const&, std::_Manager_operation)
()
from
/home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#3 0x00007fffef06d7e4 in std::_Function_handler<bool (char),
boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&,
std::_Any_data const&, std::_Manager_operation) ()
from
/home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#4 0x000000000109fc07 in std::_Function_base::~_Function_base() ()
#5 0x00007fffed7a8605 in __cxa_finalize () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007fffeef995a3 in __do_global_dtors_aux () from
/home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
#7 0x00007fffffff03c0 in ?? (){noformat}
[~boroknagyz] found that upgrading gtest to 1.14 resolves the issue. The
current gtest version is 1.6. I tried building with ASAN and also build the
underlying gtest-1.6 with ASAN. The run then shows the memory issue:
{noformat}
==1778==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000098530
at pc 0x00000123a9a9 bp 0x7ffeeaf47d60 sp 0x7ffeeaf47d58
READ of size 8 at 0x603000098530 thread T0
#0 0x123a9a8 in testing::internal::String::~String()
/home/quanlong/workspace/Impala/toolchain/toolchain-packages-gcc10.4.0/gtest-1.6.0/include/gtest/internal/gtest-string.h:218:24
#1 0x530fc2f in void
std::_Destroy_aux<false>::__destroy<testing::internal::String*>(testing::internal::String*,
testing::internal::String*)
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_construct.h:108:6
#2 0x52fdcdd in std::vector<testing::internal::String,
std::allocator<testing::internal::String> >::~vector()
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:677:2
#3 0x7f4ca763a604 in __cxa_finalize
/build/glibc-CVJwZb/glibc-2.27/stdlib/cxa_finalize.c:83
#4 0x7f4ca93a8cb2 in __do_global_dtors_aux
(/home/quanlong/workspace/Impala/be/src/kudu/util/libkudu_test_util.so+0x6ecb2)
0x603000098530 is located 0 bytes inside of 32-byte region
[0x603000098530,0x603000098550)
freed by thread T0 here:
#0 0x1233dcf in operator delete(void*)
(/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x1233dcf)
#1 0x5310da3 in std::_Vector_base<testing::internal::String,
std::allocator<testing::internal::String> >::~_Vector_base()
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:332:2
#2 0x7f4ca763a030 in __run_exit_handlers
/build/glibc-CVJwZb/glibc-2.27/stdlib/exit.c:108
previously allocated by thread T0 here:
#0 0x123305f in operator new(unsigned long)
(/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x123305f)
#1 0x530fdd0 in std::vector<testing::internal::String,
std::allocator<testing::internal::String>
>::_M_realloc_insert(__gnu_cxx::__normal_iterator<testing::internal::String*,
std::vector<testing::internal::String,
std::allocator<testing::internal::String> > >, testing::internal::String
const&)
/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/vector.tcc:440:33
#2 0x52fc053 in void testing::internal::InitGoogleTestImpl<char>(int*,
char**)
/media/quanlong/hdd-backup/native-toolchain/source/gtest/gtest-1.6.0/src/gtest.cc:4868:13
#3 0x1236435 in main
/home/quanlong/workspace/Impala/be/src/service/unified-betest-main.cc:40:3
#4 0x7f4ca7618c86 in __libc_start_main
/build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:310{noformat}
Note that I have to remove the definition of "THttpTransport::CRLF_LEN" in
be/src/transport/THttpTransport.cpp to avoid an ODR violation issue.
libkudu_test_util.so wants to delete a vector that is already deleted by gtest.
https://github.com/google/googletest/blob/release-1.6.0/include/gtest/internal/gtest-port.h#L1057
In the static build, we are fine since there are no libkudu_test_util.so or
libkudu_util.so.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
