daming6 opened a new issue, #3273:
URL: https://github.com/apache/brpc/issues/3273
**Describe the bug**
在两个shell窗口分别执行如下命令：
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset
-c 192-231 ./server -use_rdma 0 -thread_num 40
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset
-c 232-271 ./client -thread_num 40 -queue_depth 40 -attachment_size 37136
-use_rdma 0 -connection_type pooled
加上ASAN_OPTIONS=detect_leaks=0 ，压力中等时仍能偶发出现heap-use-after-free，说明是有效问题
具体的地址消毒函数栈报错如下：
=================================================================
==180482==ERROR: AddressSanitizer: heap-use-after-free on address
0xffffb44092c8 at pc 0x000000425e68 bp 0xffffab7fe9e0 sp 0xffffab7fe9d0
READ of size 4 at 0xffffb44092c8 thread T10 (brpc_wkr:0-2)
#0 0x425e64 in
PerformanceTest::HandleResponse(PerformanceTest::RespClosure*)
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x425e64)
#1 0x4115a0 in
brpc::internal::FunctionClosure1<PerformanceTest::RespClosure*>::Run()
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x4115a0)
#2 0xffffba89d9dc in
brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x6ad9dc)
#3 0xffffba8a1f88 in
brpc::Controller::OnVersionedRPCReturned(brpc::Controller::CompletionInfo
const&, bool, int) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6b1f88)
#4 0xffffbab9b650 in
brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x9ab650)
#5 0xffffba909358 in brpc::ProcessInputMessage(void*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x719358)
#6 0xffffba90ff40 in brpc::InputMessenger::OnNewMessages(brpc::Socket*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x71ff40)
#7 0xffffbaa2db78 in brpc::Socket::ProcessEvent(void*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x83db78)
#8 0xffffba717c98 in bthread::TaskGroup::task_runner(long)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x527c98)
#9 0xffffba68629c in bthread_make_fcontext
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x49629c)
0xffffb44092c8 is located 24 bytes inside of 72-byte region
[0xffffb44092b0,0xffffb44092f8)
freed by thread T6 (brpc_wkr:0-5) here:
#0 0xffffbc443f8c in operator delete(void*, unsigned long)
(/usr/lib64/libasan.so.8.0.0+0xb3f8c)
#1 0x40da44 in DeleteTest(void*)
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40da44)
#2 0xffffba717c98 in bthread::TaskGroup::task_runner(long)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x527c98)
#3 0xffffba68629c in bthread_make_fcontext
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x49629c)
previously allocated by thread T0 here:
#0 0xffffbc442ecc in operator new(unsigned long)
(/usr/lib64/libasan.so.8.0.0+0xb2ecc)
#1 0x40e490 in Test(int, int)
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40e490)
#2 0x40abe8 in main
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40abe8)
#3 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#4 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#5 0x40b82c in _start
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)
Thread T10 (brpc_wkr:0-2) created by T0 here:
#0 0xffffbc3db790 in __interceptor_pthread_create
(/usr/lib64/libasan.so.8.0.0+0x4b790)
#1 0xffffba6cb2ec in bthread::TaskControl::init(int)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x4db2ec)
#2 0xffffba67c570 in bthread::get_or_new_task_control()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x48c570)
#3 0xffffba679ad4 in bthread_start_background
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x489ad4)
#4 0xffffba8e995c in brpc::GlobalInitializeOrDieImpl()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f995c)
#5 0xffffb8b04b68 (/usr/lib64/libc.so.6+0x84b68)
#6 0xffffba8e4d10 in brpc::GlobalInitializeOrDie()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f4d10)
#7 0xffffba9c98e0 in brpc::Server::InitializeOnce()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x7d98e0)
#8 0xffffba9e6658 in brpc::Server::StartInternal(butil::EndPoint const&,
brpc::PortRange const&, brpc::ServerOptions const*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x7f6658)
#9 0xffffba9ec5ec in brpc::Server::Start(butil::EndPoint const&,
brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fc5ec)
#10 0xffffba9ecb34 in brpc::Server::Start(int, brpc::ServerOptions
const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fcb34)
#11 0xffffba9ed114 in brpc::StartDummyServerAt(int,
brpc::ProfilerLinker) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fd114)
#12 0x40a98c in main
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40a98c)
#13 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#14 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#15 0x40b82c in _start
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)
Thread T6 (brpc_wkr:0-5) created by T0 here:
#0 0xffffbc3db790 in __interceptor_pthread_create
(/usr/lib64/libasan.so.8.0.0+0x4b790)
#1 0xffffba6cb2ec in bthread::TaskControl::init(int)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x4db2ec)
#2 0xffffba67c570 in bthread::get_or_new_task_control()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x48c570)
#3 0xffffba679ad4 in bthread_start_background
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x489ad4)
#4 0xffffba8e995c in brpc::GlobalInitializeOrDieImpl()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f995c)
#5 0xffffb8b04b68 (/usr/lib64/libc.so.6+0x84b68)
#6 0xffffba8e4d10 in brpc::GlobalInitializeOrDie()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f4d10)
#7 0xffffba9c98e0 in brpc::Server::InitializeOnce()
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x7d98e0)
#8 0xffffba9e6658 in brpc::Server::StartInternal(butil::EndPoint const&,
brpc::PortRange const&, brpc::ServerOptions const*)
(/home/ci/brpc/pkgs/lib/libbrpc.so+0x7f6658)
#9 0xffffba9ec5ec in brpc::Server::Start(butil::EndPoint const&,
brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fc5ec)
#10 0xffffba9ecb34 in brpc::Server::Start(int, brpc::ServerOptions
const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fcb34)
#11 0xffffba9ed114 in brpc::StartDummyServerAt(int,
brpc::ProfilerLinker) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fd114)
#12 0x40a98c in main
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40a98c)
#13 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#14 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#15 0x40b82c in _start
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)
SUMMARY: AddressSanitizer: heap-use-after-free
(/home/ci/brpc/pkgs/example/rdma_performance/client+0x425e64) in
PerformanceTest::HandleResponse(PerformanceTest::RespClosure*)
Shadow bytes around the buggy address:
0x200ff6881200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881240: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x200ff6881250: fd fa fa fa fa fa fd fd fd[fd]fd fd fd fd fd fa
0x200ff6881260: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x200ff6881270: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x200ff6881280: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
0x200ff6881290: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
0x200ff68812a0: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==180482==ABORTING
**To Reproduce**
在两个shell窗口分别执行如下命令：
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset
-c 192-231 ./server -use_rdma 0 -thread_num 40
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset
-c 232-271 ./client -thread_num 40 -queue_depth 40 -attachment_size 37136
-use_rdma 0 -connection_type pooled
加上ASAN_OPTIONS=detect_leaks=0 ，压力中等时仍能偶发出现heap-use-after-free，说明是有效问题
**Expected behavior**
预期example/rdma_performance -use_rdma 0
默认走TCP协议应该不会出现多线程之间heap-use-after-free问题，可能是example/rdma_performance/server
client为了性能，线程之间的保护有点欠缺
**Versions**
OS: openEuler 24.03 (LTS-SP2)
Compiler: gcc 12.3.1
brpc: 1.16
protobuf: protobuf-25.1-12.oe2403sp2.aarch64
**Additional context/screenshots**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
