[jira] [Updated] (KUDU-3107) TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full
[ https://issues.apache.org/jira/browse/KUDU-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated KUDU-3107: --- Issue Type: Task (was: Bug) > TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service > queue is full > --- > > Key: KUDU-3107 > URL: https://issues.apache.org/jira/browse/KUDU-3107 > Project: Kudu > Issue Type: Task >Reporter: liusheng >Priority: Major > Attachments: rpc-test.txt > > > The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine > due the the "service queue full" error. related error message: > {code:java} > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 318) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 319) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 320) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 321) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 324) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 332) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 334) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 335) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 336) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 337) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 338) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 339) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 340) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 341) > F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed: > controller.status().IsAborted() || controller.status().IsServiceUnavailable() > || controller.status().ok() Remote error: Service unavailable: PushStrings > request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due > to backpressure. The service queue is full; it has 100 items. > *** Check failure stack trace: *** > PC: @0x0 (unknown) > *** SIGABRT (@0x3e86bbf) received by PID 27583 (TID 0x84b1f050) from > PID 27583; stack trace: *** > @ 0x93cf0464 raise at ??:0 > @ 0x93cf18b4 abort at ??:0 > @ 0x942c5fdc google::logging_fail() at ??:0 > @ 0x942c7d40 google::LogMessage::Fail() at ??:0 > @ 0x942c9c78 google::LogMessage::SendToLog() at ??:0 > @ 0x942c7874 google::LogMessage::Flush() at ??:0 > @ 0x942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0 > @ 0xdcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0 > @ 0xdcee4b98 > _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv > at ??:0 > @ 0xdcee76bc > _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ > at ??:0 > @ 0xdcee7484 > _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ > at ??:0 > @ 0xdcee8208 > _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0DTcl8__invokespcl10_S_declvalIXT_ESt12_Index_tupleIJXspT_EEE > at ??:0 > @ 0xdcee8168 > _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv > at ??:0 > @ 0xdcee8110 > _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_E6_M_runEv > at ??:0 > @ 0x93f22e94 (unknown) at ??:0 > @ 0x93e1e088 start_thread at ??:0 > @ 0x93d8e4ec (unknown) at ??:0 > {code} > The attatchment is the full test log -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3107) TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service queue is full
[ https://issues.apache.org/jira/browse/KUDU-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liusheng updated KUDU-3107: --- Parent: KUDU-3007 Issue Type: Sub-task (was: Task) > TestRpc.TestCancellationMultiThreads fail on ARM sometimes due to service > queue is full > --- > > Key: KUDU-3107 > URL: https://issues.apache.org/jira/browse/KUDU-3107 > Project: Kudu > Issue Type: Sub-task >Reporter: liusheng >Priority: Major > Attachments: rpc-test.txt > > > The test TestRpc.TestCancellationMultiThreads fail sometimes on ARM mechine > due the the "service queue full" error. related error message: > {code:java} > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 318) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 319) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 320) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 321) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 324) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 332) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 334) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 335) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 336) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 337) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 338) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 339) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 340) > Call kudu.rpc.GenericCalculatorService.PushStrings from 127.0.0.1:41516 > (request call id 341) > F0416 13:01:38.616358 31937 rpc-test.cc:1471] Check failed: > controller.status().IsAborted() || controller.status().IsServiceUnavailable() > || controller.status().ok() Remote error: Service unavailable: PushStrings > request on kudu.rpc.GenericCalculatorService from 127.0.0.1:41516 dropped due > to backpressure. The service queue is full; it has 100 items. > *** Check failure stack trace: *** > PC: @0x0 (unknown) > *** SIGABRT (@0x3e86bbf) received by PID 27583 (TID 0x84b1f050) from > PID 27583; stack trace: *** > @ 0x93cf0464 raise at ??:0 > @ 0x93cf18b4 abort at ??:0 > @ 0x942c5fdc google::logging_fail() at ??:0 > @ 0x942c7d40 google::LogMessage::Fail() at ??:0 > @ 0x942c9c78 google::LogMessage::SendToLog() at ??:0 > @ 0x942c7874 google::LogMessage::Flush() at ??:0 > @ 0x942ca4fc google::LogMessageFatal::~LogMessageFatal() at ??:0 > @ 0xdcee4940 kudu::rpc::SendAndCancelRpcs() at ??:0 > @ 0xdcee4b98 > _ZZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvENKUlvE_clEv > at ??:0 > @ 0xdcee76bc > _ZSt13__invoke_implIvZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEET_St14__invoke_otherOT0_DpOT1_ > at ??:0 > @ 0xdcee7484 > _ZSt8__invokeIZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS5_DpOS6_ > at ??:0 > @ 0xdcee8208 > _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEE9_M_invokeIJLm0DTcl8__invokespcl10_S_declvalIXT_ESt12_Index_tupleIJXspT_EEE > at ??:0 > @ 0xdcee8168 > _ZNSt6thread8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_EEEclEv > at ??:0 > @ 0xdcee8110 > _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4kudu3rpc41TestRpc_TestCancellationMultiThreads_Test8TestBodyEvEUlvE_E6_M_runEv > at ??:0 > @ 0x93f22e94 (unknown) at ??:0 > @ 0x93e1e088 start_thread at ??:0 > @ 0x93d8e4ec (unknown) at ??:0 > {code} > The attatchment is the full test log -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KUDU-2986) Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables
[ https://issues.apache.org/jira/browse/KUDU-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexey Serbin resolved KUDU-2986. - Fix Version/s: 1.12.0 Resolution: Fixed > Incorrect value for the 'live_row_count' metric with pre-1.11.0 tables > -- > > Key: KUDU-2986 > URL: https://issues.apache.org/jira/browse/KUDU-2986 > Project: Kudu > Issue Type: Bug > Components: CLI, client, master, metrics >Affects Versions: 1.11.0 >Reporter: YifanZhang >Assignee: LiFu He >Priority: Major > Fix For: 1.12.0 > > > When we upgraded the cluster with pre-1.11.0 tables, we got inconsistent > values for the 'live_row_count' metric of these tables: > When visiting masterURL:port/metrics, we got 0 for old tables, and got a > positive integer for a old table with a newly added partition, which is the > count of rows in the newly added partition. > When getting table statistics via `kudu table statistics` CLI tool, we got 0 > for old tables and the old table with a new parition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KUDU-3108) Tablet server crashes when handle scan request
[ https://issues.apache.org/jira/browse/KUDU-3108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] YifanZhang updated KUDU-3108: - Description: When we use KuduBackup Spark job to backup tables in a cluster with 20 tservers, 3 tservers crashed, coredump stacks are the same: {code:java} Program terminated with signal 11, Segmentation fault.Program terminated with signal 11, Segmentation fault. #0 kudu::Schema::Compare (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267 267 /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h: No such file or directory. Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-20.el7_2.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 cyrus-sasl-md5-2.1.26-20.el7_2.x86_64 cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libdb-5.3.21-19.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-6.el7.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7_3.8.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 kudu::Schema::Compare (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267 #1 0x01da51fb in kudu::MergeIterator::RefillHotHeap (this=this@entry=0x78f6ec500) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:720 #2 0x01da622b in kudu::MergeIterator::AdvanceAndReheap (this=this@entry=0x78f6ec500, state=0xd1661a000, num_rows_to_advance=num_rows_to_advance@entry=1) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:690 #3 0x01da7927 in kudu::MergeIterator::MaterializeOneRow (this=this@entry=0x78f6ec500, dst=dst@entry=0x7f0d5cc9ffc0, dst_row_idx=dst_row_idx@entry=0x7f0d5cc9fbb0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:894 #4 0x01da7de3 in kudu::MergeIterator::NextBlock (this=0x78f6ec500, dst=0x7f0d5cc9ffc0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:796 #5 0x00a9ff19 in kudu::tablet::Tablet::Iterator::NextBlock (this=, dst=) at /home/zhangyifan8/work/kudu-xm/src/kudu/tablet/tablet.cc:2499 #6 0x0095475c in kudu::tserver::TabletServiceImpl::HandleContinueScanRequest (this=this@entry=0x53b5a90, req=req@entry=0x7f0d5cca0720, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2565 #7 0x00966564 in kudu::tserver::TabletServiceImpl::HandleNewScanRequest (this=this@entry=0x53b5a90, replica=0xf5c0189c0, req=req@entry=0x2a15c240, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, scanner_id=scanner_id@entry=0x7f0d5cca0940, snap_timestamp=snap_timestamp@entry=0x7f0d5cca0950, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2476 #8 0x00967f4b in kudu::tserver::TabletServiceImpl::Scan (this=0x53b5a90, req=0x2a15c240, resp=0x56f9be6c0, context=0x5e512a460) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:1674 #9 0x01d2e449 in operator() (__args#2=0x5e512a460, __args#1=0x56f9be6c0, __args#0=, this=0x497ecdd8) at /usr/include/c++/4.8.2/functional:2471 #10 kudu::rpc::GeneratedServiceIf::Handle (this=0x53b5a90, call=) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_if.cc:139 #11 0x01d2eb49 in kudu::rpc::ServicePool::RunThread (this=0x2ab69560) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_pool.cc:225 #12 0x01e9e924 in operator() (this=0x90fb52e8) at /home/zhangyifan8/work/kudu-xm/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771 #13 kudu::Thread::SuperviseThread (arg=0x90fb52c0) at /home/zhangyifan8/work/kudu-xm/src/kudu/util/thread.cc:657 #14 0x7f103b20cdc5 in start_thread () from /lib64/libpthread.so.0 #15 0x7f103956673d in clone () from /lib64/libc.so.6 {code} was: When we use KuduBackup{{}} Spark job to backup tables in a cluster with 20 tservers, 3 tservers crashed, coredump stacks are the same: {code:java} [Thread debugging using libthread_db enabled][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".Missing sep
[jira] [Created] (KUDU-3108) Tablet server crashes when handle scan request
YifanZhang created KUDU-3108: Summary: Tablet server crashes when handle scan request Key: KUDU-3108 URL: https://issues.apache.org/jira/browse/KUDU-3108 Project: Kudu Issue Type: Bug Affects Versions: 1.10.1 Reporter: YifanZhang When we use KuduBackup{{}} Spark job to backup tables in a cluster with 20 tservers, 3 tservers crashed, coredump stacks are the same: {code:java} [Thread debugging using libthread_db enabled][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".Missing separate debuginfo for /home/work/app/kudu/zjyprc-hadoop/tablet_server/package/libstdc++.so.6Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/b3/d9128bcf6786292a339a477953167d0ddab5ba.debugCore was generated by `/home/work/app/kudu/zjyprc-hadoop/tablet_server/package/kudu_tablet_server -tse'.Program terminated with signal 11, Segmentation fault.#0 kudu::Schema::Compare (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267267 /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h: No such file or directory.Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-gssapi-2.1.26-20.el7_2.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 cyrus-sasl-md5-2.1.26-20.el7_2.x86_64 cyrus-sasl-plain-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libdb-5.3.21-19.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libselinux-2.5-6.el7.x86_64 ncurses-libs-5.9-13.20130511.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 systemd-libs-219-30.el7_3.8.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64(gdb) bt#0 kudu::Schema::Compare (this=0x25b883680, lhs=..., rhs=...) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/rowblock.h:267#1 0x01da51fb in kudu::MergeIterator::RefillHotHeap (this=this@entry=0x78f6ec500) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:720#2 0x01da622b in kudu::MergeIterator::AdvanceAndReheap (this=this@entry=0x78f6ec500, state=0xd1661a000, num_rows_to_advance=num_rows_to_advance@entry=1) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:690#3 0x01da7927 in kudu::MergeIterator::MaterializeOneRow (this=this@entry=0x78f6ec500, dst=dst@entry=0x7f0d5cc9ffc0, dst_row_idx=dst_row_idx@entry=0x7f0d5cc9fbb0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:894#4 0x01da7de3 in kudu::MergeIterator::NextBlock (this=0x78f6ec500, dst=0x7f0d5cc9ffc0) at /home/zhangyifan8/work/kudu-xm/src/kudu/common/generic_iterators.cc:796#5 0x00a9ff19 in kudu::tablet::Tablet::Iterator::NextBlock (this=, dst=) at /home/zhangyifan8/work/kudu-xm/src/kudu/tablet/tablet.cc:2499#6 0x0095475c in kudu::tserver::TabletServiceImpl::HandleContinueScanRequest (this=this@entry=0x53b5a90, req=req@entry=0x7f0d5cca0720, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2565#7 0x00966564 in kudu::tserver::TabletServiceImpl::HandleNewScanRequest (this=this@entry=0x53b5a90, replica=0xf5c0189c0, req=req@entry=0x2a15c240, rpc_context=rpc_context@entry=0x5e512a460, result_collector=result_collector@entry=0x7f0d5cca0a00, scanner_id=scanner_id@entry=0x7f0d5cca0940, snap_timestamp=snap_timestamp@entry=0x7f0d5cca0950, has_more_results=has_more_results@entry=0x7f0d5cca0886, error_code=error_code@entry=0x7f0d5cca0888) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:2476#8 0x00967f4b in kudu::tserver::TabletServiceImpl::Scan (this=0x53b5a90, req=0x2a15c240, resp=0x56f9be6c0, context=0x5e512a460) at /home/zhangyifan8/work/kudu-xm/src/kudu/tserver/tablet_service.cc:1674#9 0x01d2e449 in operator() (__args#2=0x5e512a460, __args#1=0x56f9be6c0, __args#0=, this=0x497ecdd8) at /usr/include/c++/4.8.2/functional:2471#10 kudu::rpc::GeneratedServiceIf::Handle (this=0x53b5a90, call=) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_if.cc:139#11 0x01d2eb49 in kudu::rpc::ServicePool::RunThread (this=0x2ab69560) at /home/zhangyifan8/work/kudu-xm/src/kudu/rpc/service_pool.cc:225#12 0x01e9e924 in operator() (this=0x90fb52e8) at /home/zhangyifan8/work/kudu-xm/thirdparty/installed/uninstrumented/include/boost/function/function_template.hpp:771#13