[
https://issues.apache.org/jira/browse/KUDU-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249400#comment-17249400
]
Andrew Wong commented on KUDU-3222:
-----------------------------------
FWIW we did have a run succeed on commit
420b07e6490e14f26107088bc1b09866b6d43bba. If this was caused by a code change,
it is likely the gperftools bump 713fee390d0241bf466f490d5b2c678f7ebe5175,
since the glog change was reverted.
> std::bad_alloc in full_stack-insert-scan-test.cc
> ------------------------------------------------
>
> Key: KUDU-3222
> URL: https://issues.apache.org/jira/browse/KUDU-3222
> Project: Kudu
> Issue Type: Bug
> Components: test
> Reporter: Andrew Wong
> Priority: Major
> Attachments: FullStackScanInsertMRSOnly3.log
>
>
> Recently we've been starting to see the following in our runs of
> full_stack-insert-scan-test:
> {code:java}
> I1214 13:30:32.995853 39072 full_stack-insert-scan-test.cc:271] Insertion
> thread 7 of 50 is 69% done.
> terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc
> *** Aborted at 1607981433 (unix time) try "date -d @1607981433" if you are
> using GNU date ***
> PC: @ 0x3f85032625 __GI_raise
> *** SIGABRT (@0x115600009802) received by PID 38914 (TID 0x7f81b4a02700) from
> PID 38914; stack trace: ***
> @ 0xcf8a21 google::(anonymous namespace)::FailureSignalHandler()
> @ 0x3f8540f710 (unknown)
> @ 0x3f85032625 __GI_raise
> @ 0x3f85033e05 __GI_abort
> @ 0x3f884bea7d __gnu_cxx::__verbose_terminate_handler()
> @ 0x3f884bcbd6 (unknown)
> @ 0x3f884bcc03 std::terminate()
> @ 0x3f884bcd22 __cxa_throw
> @ 0xd14bd5 (anonymous namespace)::handle_oom()
> @ 0x2ff3872 tcmalloc::allocate_full_cpp_throw_oom()
> @ 0x2cd4c1a
> _ZNSt6vectorIN4kudu19DecodedRowOperationESaIS1_EE17_M_realloc_insertIJRKS1_EEEvN9__gnu_cxx17__normal_iteratorIPS1_S3_EEDpOT_
> @ 0x2cd535a kudu::RowOperationsPBDecoder::DecodeOperations<>()
> @ 0x131c2a6 kudu::tablet::Tablet::DecodeWriteOperations()
> I1214 13:30:33.075912 39094 full_stack-insert-scan-test.cc:271] Insertion
> thread 29 of 50 is 69% done.
> @ 0x135bcb6 kudu::tablet::WriteOp::Prepare()
> @ 0x13514ac kudu::tablet::OpDriver::Prepare()
> @ 0x13520ed
> _ZNSt17_Function_handlerIFvvEZN4kudu6tablet8OpDriver12ExecuteAsyncEvEUlvE_E9_M_invokeERKSt9_Any_data
> @ 0x2e2409e kudu::ThreadPool::DispatchThread()
> @ 0x2e1b2c5 kudu::Thread::SuperviseThread()
> @ 0x3f854079d1 start_thread
> @ 0x3f850e88fd clone {code}
> This runs as a part of a suite of several tests in {{scripts/benchmarks.sh}}.
> This started happening fairly consistently starting around commit
> 2943aa701ee092158c2084c614a91f92513ef7c4, when we bumped glog and gperftools,
> though I'm not sure they are directly related here.
> The attached logs are a run on CentOS 6.6, with around 100GB of memory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
