Alexey Serbin created KUDU-3524:
-----------------------------------
Summary: The {{TestScannerKeepAlivePeriodicallyCrossServers}}
scenario fails with SIGABRT
Key: KUDU-3524
URL: https://issues.apache.org/jira/browse/KUDU-3524
Project: Kudu
Issue Type: Bug
Reporter: Alexey Serbin
Running the newly added tests scenario
{{TestScannerKeepAlivePeriodicallyCrossServers}} fails with SIGABRT when run as
the following on macOS (but I guess it's not macOS-specific) in DEBUG build:
{noformat}
./bin/client-test --stress_cpu_threads=32
--gtest_filter='*TestScannerKeepAlivePeriodicallyCrossServers*'
{noformat}
The error message and the stacktrace is below:
{noformat}
F20231113 12:21:13.431455 41195482 thread_restrictions.cc:79] Check failed:
LoadTLS()->wait_allowed Waiting is not allowed to be used on this thread to
prevent server-wide latency aberrations and deadlocks. Thread 41195482 (name:
"rpc reactor", category: "reactor")
*** Check failure stack trace: ***
Process 77090 stopped
* thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
-> 0x7fff205b890e <+10>: jae 0x7fff205b8918 ; <+20>
0x7fff205b8910 <+12>: movq %rax, %rdi
0x7fff205b8913 <+15>: jmp 0x7fff205b2ab9 ; cerror_nocancel
0x7fff205b8918 <+20>: retq
Target 0: (client-test) stopped.
(lldb) bt
* thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
* frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff205e75bd libsystem_pthread.dylib`pthread_kill + 263
frame #2: 0x00007fff2053c406 libsystem_c.dylib`abort + 125
frame #3: 0x000000010f64ebd8
libglog.1.dylib`google::LogMessage::SendToLog() [inlined]
google::LogMessage::Fail() at logging.cc:1946:3 [opt]
frame #4: 0x000000010f64ebd2
libglog.1.dylib`google::LogMessage::SendToLog(this=0x000070001a95e108) at
logging.cc:1920:5 [opt]
frame #5: 0x000000010f64f47a
libglog.1.dylib`google::LogMessage::Flush(this=0x000070001a95e108) at
logging.cc:1777:5 [opt]
frame #6: 0x000000010f65428f
libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=0x000070001a95e108)
at logging.cc:2557:5 [opt]
frame #7: 0x000000010f650349
libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=<unavailable>)
at logging.cc:2556:37 [opt]
frame #8: 0x000000010e545473
libkudu_util.dylib`kudu::ThreadRestrictions::AssertWaitAllowed() at
thread_restrictions.cc:79:3
frame #9: 0x000000010013ebb9
client-test`kudu::CountDownLatch::Wait(this=0x000070001a95e2a0) const at
countdown_latch.h:74:5
frame #10: 0x000000010a1749f5
libkrpc.dylib`kudu::Notification::WaitForNotification(this=0x000070001a95e2a0)
const at notification.h:127:12
frame #11: 0x000000010a1748e9
libkrpc.dylib`kudu::rpc::Proxy::SyncRequest(this=0x000000011317e9b8,
method="ScannerKeepAlive", req=0x000070001a95e428, resp=0x000070001a95e408,
controller=0x000070001a95e458) at proxy.cc:259:8
frame #12: 0x000000010697220f
libtserver_service_proto.dylib`kudu::tserver::TabletServerServiceProxy::ScannerKeepAlive(this=0x000000011317e9b8,
req=0x000070001a95e428, resp=0x000070001a95e408,
controller=0x000070001a95e458) at tserver_service.proxy.cc:98:10
frame #13: 0x000000010525c5b6
libkudu_client.dylib`kudu::client::KuduScanner::Data::KeepAlive(this=0x000000011290c700)
at scanner-internal.cc:664:3
frame #14: 0x0000000105269e76
libkudu_client.dylib`kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(this=0x0000000112899858)::$_0::operator()()
const at scanner-internal.cc:112:16
frame #15: 0x0000000105269e30
libkudu_client.dylib`decltype(__f=0x0000000112899858)::$_0&>(fp)())
std::__1::__invoke<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long,
std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at
type_traits:3694:1
frame #16: 0x0000000105269dd1 libkudu_client.dylib`void
std::__1::__invoke_void_return_wrapper<void,
true>::__call<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(__args=0x0000000112899858)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at
__functional_base:348:9
frame #17: 0x0000000105269d9d
libkudu_client.dylib`std::__1::__function::__alloc_func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0,
std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void
()>::operator(this=0x0000000112899858)() at functional:1558:16
frame #18: 0x0000000105268ac9
libkudu_client.dylib`std::__1::__function::__func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0,
std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned
long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void
()>::operator(this=0x0000000112899850)() at functional:1732:12
frame #19: 0x00000001013ae082
libtserver_test_util.dylib`std::__1::__function::__value_func<void
()>::operator(this=0x0000000112899850)() const at functional:1885:16
frame #20: 0x00000001013adee5
libtserver_test_util.dylib`std::__1::function<void
()>::operator(this=0x0000000112899850)() const at functional:2560:12
frame #21: 0x000000010a16cd62
libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000112899830,
my_callback_generation=1) at periodic.cc:194:5
frame #22: 0x000000010a17159a
libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000111d62528,
s=0x000070001a95e910)::$_0::operator()(kudu::Status const&) const at
periodic.cc:214:14
frame #23: 0x000000010a171512
libkrpc.dylib`decltype(__f=0x0000000111d62528,
__args=0x000070001a95e910)::$_0&>(fp)(std::__1::forward<kudu::Status
const&>(fp0))) std::__1::__invoke<kudu::rpc::PeriodicTimer::Callback(long
long)::$_0&, kudu::Status const&>(kudu::rpc::PeriodicTimer::Callback(long
long)::$_0&, kudu::Status const&) at type_traits:3694:1
frame #24: 0x000000010a1714b2 libkrpc.dylib`void
std::__1::__invoke_void_return_wrapper<void,
true>::__call<kudu::rpc::PeriodicTimer::Callback(__args=0x0000000111d62528,
__args=0x000070001a95e910)::$_0&, kudu::Status
const&>(kudu::rpc::PeriodicTimer::Callback(long long)::$_0&, kudu::Status
const&) at __functional_base:348:9
frame #25: 0x000000010a171462
libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::PeriodicTimer::Callback(long
long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long
long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62528,
__arg=0x000070001a95e910)(kudu::Status const&) at functional:1558:16
frame #26: 0x000000010a16ff11
libkrpc.dylib`std::__1::__function::__func<kudu::rpc::PeriodicTimer::Callback(long
long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long
long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62520,
__arg=0x000070001a95e910)(kudu::Status const&) at functional:1732:12
frame #27: 0x0000000101b256da
libmaster.dylib`std::__1::__function::__value_func<void (kudu::Status
const&)>::operator(this=0x0000000111d62520,
__args=0x000070001a95e910)(kudu::Status const&) const at functional:1885:16
frame #28: 0x0000000101b17cbd libmaster.dylib`std::__1::function<void
(kudu::Status const&)>::operator(this= Lambda in File periodic.cc at Line 208,
__arg=0x000070001a95e910)(kudu::Status const&) const at functional:2560:12
frame #29: 0x000000010a188f89
libkrpc.dylib`kudu::rpc::DelayedTask::TimerHandler(this=0x0000000111d62500,
(null)=0x0000000111d62560, revents=256) at reactor.cc:767:5
frame #30: 0x000000010a1965ce libkrpc.dylib`void ev::base<ev_timer,
ev::timer>::method_thunk<kudu::rpc::DelayedTask, &(loop=0x00000001136f78c0,
w=0x0000000111d62560, revents=256))>(ev_loop*, ev_timer*, int) at ev++.h:479:7
frame #31: 0x000000010f327d41
libev.4.dylib`ev_invoke_pending(loop=0x00000001136f78c0) at ev.c:3155:11
frame #32: 0x000000010a18027e
libkrpc.dylib`kudu::rpc::ReactorThread::InvokePendingCb(loop=0x00000001136f78c0)
at reactor.cc:204:3
frame #33: 0x000000010f3283ce libev.4.dylib`ev_run(loop=0x00000001136f78c0,
flags=0) at ev.c:3555:7
frame #34: 0x000000010a186dbe
libkrpc.dylib`ev::loop_ref::run(this=0x0000000113502a18, flags=0) at
ev++.h:211:7
frame #35: 0x000000010a186ba8
libkrpc.dylib`kudu::rpc::ReactorThread::RunThread(this=0x0000000113502a10) at
reactor.cc:505:9
frame #36: 0x000000010a190fd8
libkrpc.dylib`kudu::rpc::ReactorThread::Init(this=0x00000001133ab378)::$_0::operator()()
const at reactor.cc:196:48
frame #37: 0x000000010a190f9d
libkrpc.dylib`decltype(__f=0x00000001133ab378)::$_0&>(fp)())
std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&)
at type_traits:3694:1
frame #38: 0x000000010a190f4d libkrpc.dylib`void
std::__1::__invoke_void_return_wrapper<void,
true>::__call<kudu::rpc::ReactorThread::Init(__args=0x00000001133ab378)::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&)
at __functional_base:348:9
frame #39: 0x000000010a190f1d
libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0,
std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void
()>::operator(this=0x00000001133ab378)() at functional:1558:16
frame #40: 0x000000010a18fab9
libkrpc.dylib`std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0,
std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void
()>::operator(this=0x00000001133ab370)() at functional:1732:12
frame #41: 0x00000001013ae082
libtserver_test_util.dylib`std::__1::__function::__value_func<void
()>::operator(this=0x00000001133ab370)() const at functional:1885:16
frame #42: 0x00000001013adee5
libtserver_test_util.dylib`std::__1::function<void ()>::operator(this= Lambda
in File reactor.cc at Line 196)() const at functional:2560:12
frame #43: 0x000000010e4f92e7
libkudu_util.dylib`kudu::Thread::SuperviseThread(arg=0x00000001133ab320) at
thread.cc:691:3
frame #44: 0x00007fff205e78fc libsystem_pthread.dylib`_pthread_start + 224
frame #45: 0x00007fff205e3443 libsystem_pthread.dylib`thread_start + 15
{noformat}
The version information is below:
{noformat}
$ ./bin/client-test --version
kudu 1.18.0-SNAPSHOT
revision 8644d88dae6a76c5df3595b8a5aeb13df4d6ab5c
build type DEBUG
built by ... at 13 Nov 2023 11:32:09 PST on ...
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
