Adar Dembo created KUDU-1987:
--------------------------------
Summary: heap-use-after-free in ThreadLocalCache entry
Key: KUDU-1987
URL: https://issues.apache.org/jira/browse/KUDU-1987
Project: Kudu
Issue Type: Bug
Components: util
Affects Versions: 1.4.0
Reporter: Adar Dembo
Assignee: Todd Lipcon
Priority: Critical
Observed in ClientStressTest_LowMemory.TestMemoryThrottling.
{noformat}
=================================================================
==30312==ERROR: AddressSanitizer: heap-use-after-free on address 0x6020003f0f50
at pc 0x7f38bec95ddc bp 0x7f3873225950 sp 0x7f3873225948
READ of size 8 at 0x6020003f0f50 thread T139 (apply [worker]-)
#0 0x7f38bec95ddb in kudu::LongAdder::IncrementBy(long)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:144:48
#1 0x7f38beb05abd in kudu::(anonymous
namespace)::LRUCache::FreeEntry(kudu::(anonymous namespace)::LRUHandle*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/cache.cc:252:26
#2 0x7f38c83f0371 in kudu::cfile::BlockCacheHandle::Release()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.h:178:28
#3 0x7f38c83f0249 in kudu::cfile::BlockCacheHandle::~BlockCacheHandle()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.h:173:7
#4 0x7f38c83efec1 in
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex>::operator()(kudu::cfile::IndexTreeIterator::SeekedIndex*)
const
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:67:2
#5 0x7f38c83efdea in
std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex>
>::~unique_ptr()
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/unique_ptr.h:184:4
#6 0x7f38c83efd14 in void
std::_Destroy_aux<false>::__destroy<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex>
>*>(std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >*,
std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >*)
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_construct.h:103:6
#7 0x7f38c83efb6e in
std::vector<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> >,
std::allocator<std::unique_ptr<kudu::cfile::IndexTreeIterator::SeekedIndex,
std::default_delete<kudu::cfile::IndexTreeIterator::SeekedIndex> > >
>::~vector()
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/bits/stl_vector.h:415:9
#8 0x7f38c19ad25a in
boost::optional_detail::optional_base<kudu::cfile::(anonymous
namespace)::BloomCacheItem>::destroy_impl()
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/optional/optional.hpp:566:49
#9 0x7f38c19ad458 in std::array<std::pair<unsigned long,
boost::optional<kudu::cfile::(anonymous namespace)::BloomCacheItem> >,
4ul>::~array()
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/array:81:12
#10 0x7f38c19acff1 in void
kudu::threadlocal::internal::Destroy<kudu::ThreadLocalCache<unsigned long,
kudu::cfile::(anonymous namespace)::BloomCacheItem> >(void*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.h:136:3
#11 0x7f38becc145c in kudu::threadlocal::internal::InvokeDestructors(void*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.cc:41:5
#12 0x7f38c2373f81 in __nptl_deallocate_tsd
/build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:158
#13 0x7f38c2374196 in start_thread
/build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:325
#14 0x7f38bbd9737c in clone
/build/eglibc-oGUzwX/eglibc-2.19/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:111
0x6020003f0f50 is located 0 bytes inside of 8-byte region
[0x6020003f0f50,0x6020003f0f58)
freed by thread T139 (apply [worker]-) here:
#0 0x50f440 in operator delete(void*)
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:110
#1 0x7f38becc145c in kudu::threadlocal::internal::InvokeDestructors(void*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadlocal.cc:41:5
#2 0x7f38c2373f81 in __nptl_deallocate_tsd
/build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:158
previously allocated by thread T139 (apply [worker]-) here:
#0 0x50ed80 in operator new(unsigned long)
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_new_delete.cc:78
#1 0x7f38bec95ce1 in kudu::LongAdder::IncrementBy(long)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:139:3
#2 0x7f38beb06ad7 in kudu::(anonymous
namespace)::LRUCache::Lookup(kudu::Slice const&, unsigned int, bool)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/cache.cc:286:24
#3 0x7f38c19a4e8d in
kudu::cfile::BlockCache::Lookup(kudu::cfile::BlockCache::CacheKey const&,
kudu::Cache::CacheBehavior, kudu::cfile::BlockCacheHandle*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/block_cache.cc:78:30
#4 0x7f38c19bb7e4 in
kudu::cfile::CFileReader::ReadBlock(kudu::cfile::BlockPointer const&,
kudu::cfile::CFileReader::CacheControl, kudu::cfile::BlockHandle*) const
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/cfile_reader.cc:340:14
#5 0x7f38c19f0d8f in
kudu::cfile::IndexTreeIterator::LoadBlock(kudu::cfile::BlockPointer const&,
int)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:270:3
#6 0x7f38c19f003d in
kudu::cfile::IndexTreeIterator::SeekDownward(kudu::Slice const&,
kudu::cfile::BlockPointer const&, int)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:283:3
#7 0x7f38c19eff4f in
kudu::cfile::IndexTreeIterator::SeekAtOrBefore(kudu::Slice const&)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/index_btree.cc:171:10
#8 0x7f38c19ac3eb in
kudu::cfile::BloomFileReader::CheckKeyPresent(kudu::BloomKeyProbe const&,
bool*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/cfile/bloomfile.cc:283:26
#9 0x7f38c82c78a2 in
kudu::tablet::CFileSet::FindRow(kudu::tablet::RowSetKeyProbe const&, unsigned
int*, kudu::tablet::ProbeStats*) const
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/cfile_set.cc:216:31
#10 0x7f38c82c80b9 in
kudu::tablet::CFileSet::CheckRowPresent(kudu::tablet::RowSetKeyProbe const&,
bool*, unsigned int*, kudu::tablet::ProbeStats*) const
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/cfile_set.cc:246:14
#11 0x7f38c832534a in
kudu::tablet::DiskRowSet::CheckRowPresent(kudu::tablet::RowSetKeyProbe const&,
bool*, kudu::tablet::ProbeStats*) const
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/diskrowset.cc:646:3
#12 0x7f38c81e5911 in
kudu::tablet::Tablet::BulkCheckPresence(kudu::tablet::WriteTransactionState*)::$_3::operator()()
const
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:767:7
#13 0x7f38c81e49b2 in
kudu::tablet::Tablet::BulkCheckPresence(kudu::tablet::WriteTransactionState*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:785:3
#14 0x7f38c81e6005 in
kudu::tablet::Tablet::ApplyRowOperations(kudu::tablet::WriteTransactionState*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/tablet.cc:805:3
#15 0x7f38c82bcfd1 in
kudu::tablet::WriteTransaction::Apply(gscoped_ptr<kudu::consensus::CommitMsg,
kudu::DefaultDeleter<kudu::consensus::CommitMsg> >*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/write_transaction.cc:135:11
#16 0x7f38c82ab9df in kudu::tablet::TransactionDriver::ApplyTask()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:477:5
#17 0x7f38c82b0578 in kudu::internal::InvokeHelper<false, void,
kudu::internal::RunnableAdapter<void (kudu::tablet::TransactionDriver::*)()>,
void
(kudu::tablet::TransactionDriver*)>::MakeItSo(kudu::internal::RunnableAdapter<void
(kudu::tablet::TransactionDriver::*)()>, kudu::tablet::TransactionDriver*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:871:14
#18 0x7f38c82b03d8 in kudu::internal::Invoker<1,
kudu::internal::BindState<kudu::internal::RunnableAdapter<void
(kudu::tablet::TransactionDriver::*)()>, void
(kudu::tablet::TransactionDriver*), void
(kudu::internal::UnretainedWrapper<kudu::tablet::TransactionDriver>)>, void
(kudu::tablet::TransactionDriver*)>::Run(kudu::internal::BindStateBase*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1063:12
#19 0x7f38c76695d7 in boost::_bi::bind_t<void, boost::_mfi::cmf0<void,
kudu::Callback<void ()> >,
boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()> > > >::operator()()
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#20 0x7f38c5e1741e in boost::function0<void>::operator()() const
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#21 0x7f38becc4d5d in kudu::ThreadPool::DispatchThread(bool)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:347:23
#22 0x7f38beccbd67 in boost::_bi::bind_t<void, boost::_mfi::mf1<void,
kudu::ThreadPool, bool>,
boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool>
> >::operator()()
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#23 0x7f38c5e1741e in boost::function0<void>::operator()() const
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#24 0x7f38becb1a83 in kudu::Thread::SuperviseThread(void*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:590:3
#25 0x7f38c2374183 in start_thread
/build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:312
Thread T139 (apply [worker]-) created by T75 (rpc worker-3039) here:
#0 0x4329e1 in __interceptor_pthread_create
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:245
#1 0x7f38becb0fbd in kudu::Thread::StartThread(std::string const&,
std::string const&, boost::function<void ()> const&, unsigned long,
scoped_refptr<kudu::Thread>*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:513:15
#2 0x7f38becc6c74 in kudu::Status kudu::Thread::Create<void
(kudu::ThreadPool::*)(bool), kudu::ThreadPool*, bool>(std::string const&,
std::string const&, void (kudu::ThreadPool::* const&)(bool), kudu::ThreadPool*
const&, bool const&, scoped_refptr<kudu::Thread>*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:164:12
#3 0x7f38becc2fe4 in kudu::ThreadPool::CreateThreadUnlocked()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:392:14
#4 0x7f38becc3bfd in
kudu::ThreadPool::Submit(std::shared_ptr<kudu::Runnable>)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:218:21
#5 0x7f38becc38b3 in kudu::ThreadPool::SubmitFunc(boost::function<void ()>)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:186:10
#6 0x7f38becc36bf in kudu::ThreadPool::SubmitClosure(kudu::Callback<void
()> const&)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/threadpool.cc:182:10
#7 0x7f38c82aaf63 in kudu::tablet::TransactionDriver::ApplyAsync()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:458:23
#8 0x7f38c82a65dd in
kudu::tablet::TransactionDriver::ReplicationFinished(kudu::Status const&)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tablet/transactions/transaction_driver.cc:412:5
#9 0x7f38c829d84c in kudu::internal::InvokeHelper<false, void,
kudu::internal::RunnableAdapter<void
(kudu::tablet::TransactionDriver::*)(kudu::Status const&)>, void
(kudu::tablet::TransactionDriver*, kudu::Status
const&)>::MakeItSo(kudu::internal::RunnableAdapter<void
(kudu::tablet::TransactionDriver::*)(kudu::Status const&)>,
kudu::tablet::TransactionDriver*, kudu::Status const&)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:887:14
#10 0x7f38c829d5a5 in kudu::internal::Invoker<1,
kudu::internal::BindState<kudu::internal::RunnableAdapter<void
(kudu::tablet::TransactionDriver::*)(kudu::Status const&)>, void
(kudu::tablet::TransactionDriver*, kudu::Status const&), void
(kudu::internal::UnretainedWrapper<kudu::tablet::TransactionDriver>)>, void
(kudu::tablet::TransactionDriver*, kudu::Status
const&)>::Run(kudu::internal::BindStateBase*, kudu::Status const&)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/gutil/bind_internal.h:1116:12
#11 0x7f38c76c2be9 in
kudu::consensus::PendingRounds::AdvanceCommittedIndex(long)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus_state.cc:508:12
#12 0x7f38c76968c5 in
kudu::consensus::RaftConsensus::UpdateReplica(kudu::consensus::ConsensusRequestPB
const*, kudu::consensus::ConsensusResponsePB*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus.cc:1295:5
#13 0x7f38c7694a2c in
kudu::consensus::RaftConsensus::Update(kudu::consensus::ConsensusRequestPB
const*, kudu::consensus::ConsensusResponsePB*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/consensus/raft_consensus.cc:752:14
#14 0x7f38ca53651b in
kudu::tserver::ConsensusServiceImpl::UpdateConsensus(kudu::consensus::ConsensusRequestPB
const*, kudu::consensus::ConsensusResponsePB*, kudu::rpc::RpcContext*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_service.cc:843:25
#15 0x7f38c6a2a18a in std::_Function_handler<void
(google::protobuf::Message const*, google::protobuf::Message*,
kudu::rpc::RpcContext*),
kudu::consensus::ConsensusServiceIf::ConsensusServiceIf(scoped_refptr<kudu::MetricEntity>
const&, scoped_refptr<kudu::rpc::ResultTracker>
const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message
const*, google::protobuf::Message*, kudu::rpc::RpcContext*)
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/functional:2071:2
#16 0x7f38c5e90a94 in std::function<void (google::protobuf::Message const*,
google::protobuf::Message*,
kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*,
google::protobuf::Message*, kudu::rpc::RpcContext*) const
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../include/c++/4.8/functional:2471:14
#17 0x7f38c5e8fac6 in
kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_if.cc:134:3
#18 0x7f38c5e92de8 in kudu::rpc::ServicePool::RunThread()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_pool.cc:210:15
#19 0x7f38c5e96f67 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
kudu::rpc::ServicePool>,
boost::_bi::list1<boost::_bi::value<kudu::rpc::ServicePool*> > >::operator()()
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/bind/bind.hpp:1222:16
#20 0x7f38c5e1741e in boost::function0<void>::operator()() const
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:770:14
#21 0x7f38becb1a83 in kudu::Thread::SuperviseThread(void*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:590:3
#22 0x7f38c2374183 in start_thread
/build/eglibc-oGUzwX/eglibc-2.19/nptl/pthread_create.c:312
Thread T75 (rpc worker-3039) created by T0 here:
#0 0x4329e1 in __interceptor_pthread_create
/home/jenkins-slave/workspace/kudu-master/3/thirdparty/src/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_interceptors.cc:245
#1 0x7f38becb0fbd in kudu::Thread::StartThread(std::string const&,
std::string const&, boost::function<void ()> const&, unsigned long,
scoped_refptr<kudu::Thread>*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.cc:513:15
#2 0x7f38c5e95284 in kudu::Status kudu::Thread::Create<void
(kudu::rpc::ServicePool::*)(), kudu::rpc::ServicePool*>(std::string const&,
std::string const&, void (kudu::rpc::ServicePool::* const&)(),
kudu::rpc::ServicePool* const&, scoped_refptr<kudu::Thread>*)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/thread.h:158:12
#3 0x7f38c5e92814 in kudu::rpc::ServicePool::Init(int)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/rpc/service_pool.cc:81:5
#4 0x7f38c915aecb in
kudu::RpcServer::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf,
kudu::DefaultDeleter<kudu::rpc::ServiceIf> >)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/server/rpc_server.cc:122:3
#5 0x7f38c91660e3 in
kudu::server::ServerBase::RegisterService(gscoped_ptr<kudu::rpc::ServiceIf,
kudu::DefaultDeleter<kudu::rpc::ServiceIf> >)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/server/server_base.cc:364:23
#6 0x7f38ca52c173 in kudu::tserver::TabletServer::Start()
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server.cc:117:3
#7 0x5125fa in kudu::tserver::TabletServerMain(int, char**)
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server_main.cc:77:3
#8 0x5120f4 in main
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/tserver/tablet_server_main.cc:91:10
#9 0x7f38bbcbef44 in __libc_start_main
/build/eglibc-oGUzwX/eglibc-2.19/csu/libc-start.c:287
SUMMARY: AddressSanitizer: heap-use-after-free
/home/jenkins-slave/workspace/kudu-master/3/src/kudu/util/striped64.cc:144:48
in kudu::LongAdder::IncrementBy(long)
Shadow bytes around the buggy address:
0x0c0480076190: fa fa fd fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c04800761d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c04800761e0: fa fa fa fa fa fa fa fa fa fa[fd]fa fa fa 00 fa
0x0c04800761f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0480076230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==30312==ABORTING
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
