Alexey Serbin created KUDU-2030:
-----------------------------------
Summary: Tablet server crashes on using deallocated
Key: KUDU-2030
URL: https://issues.apache.org/jira/browse/KUDU-2030
Project: Kudu
Issue Type: Bug
Components: tserver
Affects Versions: 1.4.0
Reporter: Alexey Serbin
The code in {{RaftConsensus::UpdateReplica()}}
(src/kudu/consensus/raft_consensus.cc) instantiates {{Synchronizer}} on the
stack and then uses the derived StatusCallback in a way that under certain code
path leads to an attempt to use already deallocated {{Mutex}} object
{{CountDownLatch::lock_}}. The instance of {{CountDownLatch}} is aggregated by
the {{Synchronizer}} object itself.
Under certain scenarios, tserver crashes with the following stack trace:
{noformat}
F0605 18:22:23.583866 14144 mutex.cc:76] Check failed: rv == 0 || rv == 16 .
Invalid argument. Owner tid: 23156096; Self tid: 144; To collect the owner
stack trace, enable the flag --debug_mutex_collect_stacktrace
*** Check failure stack trace: ***
@ 0x7fab619a62fd google::LogMessage::Fail() at ??:0
@ 0x7fab619a81bd google::LogMessage::SendToLog() at ??:0
@ 0x7fab619a5e39 google::LogMessage::Flush() at ??:0
@ 0x7fab619a8c5f google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fab627eb453 kudu::Mutex::TryAcquire() at ??:0
@ 0x7fab627eb82c kudu::Mutex::Acquire() at ??:0
@ 0x7fab6aec6b7a kudu::CountDownLatch::CountDown() at ??:0
@ 0x7fab6aec526a kudu::CountDownLatch::CountDown() at ??:0
@ 0x7fab69339633 kudu::Synchronizer::StatusCB() at ??:0
@ 0x7fab69339a21 kudu::internal::RunnableAdapter<>::Run() at ??:0
@ 0x7fab69339964 kudu::internal::InvokeHelper<>::MakeItSo() at ??:0
@ 0x7fab693398f2 kudu::internal::Invoker<>::Run() at ??:0
@ 0x7fab692bce26 kudu::Callback<>::Run() at ??:0
{noformat}
The {{pthread_mutex_trylock()}} in mutex.cc:74 returns {{EINVAL}} since the
underlying pthread mutex handle has already been deallocated.
To reproduce, run the
{{ClientFailoverOnNegotiationTimeoutITest.Kudu1580ConnectToTServer}} from
{{client-negotiation-failover-itest}} built from version
{{5f8442ff67fe87b019c71a09f0556bdcb6868428}} in DEBUG configuration with
--stress-cpu-threads=8 about 1K times. One 1K run usually produces about 3-4
crashes like that.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
