Andrew Wong created KUDU-3222:
---------------------------------
Summary: std::bad_alloc in full_stack-insert-scan-test.cc
Key: KUDU-3222
URL: https://issues.apache.org/jira/browse/KUDU-3222
Project: Kudu
Issue Type: Bug
Components: test
Reporter: Andrew Wong
Attachments: FullStackScanInsertMRSOnly3.log
Recently we've been starting to see the following in our runs of
full_stack-insert-scan-test:
{code:java}
I1214 13:30:32.995853 39072 full_stack-insert-scan-test.cc:271] Insertion
thread 7 of 50 is 69% done.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
*** Aborted at 1607981433 (unix time) try "date -d @1607981433" if you are
using GNU date ***
PC: @ 0x3f85032625 __GI_raise
*** SIGABRT (@0x115600009802) received by PID 38914 (TID 0x7f81b4a02700) from
PID 38914; stack trace: ***
@ 0xcf8a21 google::(anonymous namespace)::FailureSignalHandler()
@ 0x3f8540f710 (unknown)
@ 0x3f85032625 __GI_raise
@ 0x3f85033e05 __GI_abort
@ 0x3f884bea7d __gnu_cxx::__verbose_terminate_handler()
@ 0x3f884bcbd6 (unknown)
@ 0x3f884bcc03 std::terminate()
@ 0x3f884bcd22 __cxa_throw
@ 0xd14bd5 (anonymous namespace)::handle_oom()
@ 0x2ff3872 tcmalloc::allocate_full_cpp_throw_oom()
@ 0x2cd4c1a
_ZNSt6vectorIN4kudu19DecodedRowOperationESaIS1_EE17_M_realloc_insertIJRKS1_EEEvN9__gnu_cxx17__normal_iteratorIPS1_S3_EEDpOT_
@ 0x2cd535a kudu::RowOperationsPBDecoder::DecodeOperations<>()
@ 0x131c2a6 kudu::tablet::Tablet::DecodeWriteOperations()
I1214 13:30:33.075912 39094 full_stack-insert-scan-test.cc:271] Insertion
thread 29 of 50 is 69% done.
@ 0x135bcb6 kudu::tablet::WriteOp::Prepare()
@ 0x13514ac kudu::tablet::OpDriver::Prepare()
@ 0x13520ed
_ZNSt17_Function_handlerIFvvEZN4kudu6tablet8OpDriver12ExecuteAsyncEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x2e2409e kudu::ThreadPool::DispatchThread()
@ 0x2e1b2c5 kudu::Thread::SuperviseThread()
@ 0x3f854079d1 start_thread
@ 0x3f850e88fd clone {code}
This runs as a part of a suite of several tests in {{scripts/benchmarks.sh}}.
This started happening fairly consistently starting around commit
2943aa701ee092158c2084c614a91f92513ef7c4, when we bumped glog and gperftools,
though I'm not sure they are directly related here.
The attached logs are a run on CentOS 6.6, with around 100GB of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
