malanore-z opened a new issue, #2304: URL: https://github.com/apache/brpc/issues/2304
### 问题描述: 我这边的机器同一个pcie switch下有两个100Gbps网卡: + 如果只使用一个网卡, 使用example/rdma_performance可以跑满带宽. + 如果同时使用两个网卡(启动两个server进程分别绑定一个网卡), 则每个server进程只能跑到30Gbps带宽. + 如果两个网卡用来发送而不是接收(分别发向另外两台机器), 则可以跑满100Gbps带宽 测试中每个server只会有一个client连接. ### 排查记录 1. 工作线程没满(bthread_worker_usage不到2) 2. CPU利用率很低(约200%, 一共192 cores) 3. contention显示锁竞争很少 + 阅读文档: https://github.com/apache/brpc/blob/master/docs/cn/server_debugging.md ### 相关参数: + client: `-thread_num=4 -attachment_size=65536 -queue_depth=4 -bthread_concurrency=10 -rdma_memory_pool_initial_size_mb=10240` 修改参数发现其实只有attachment_size影响测试结果, 其它几个参数基本无影响. + server: `-rdma_memory_pool_initial_size_mb=10240 -rdma_recv_block_type=huge -bthread_concurrency=10 -bthread_min_concurrency=10` 测试发现在不指定`bthread_min_concurrency`参数的情况下, `bthread_concurrency`无效. ### 补充: 用ib_write_bw测试两个网卡同时使用是正常的,可以确定硬件没有问题. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
