[GitHub] [incubator-brpc] lwmonster opened a new issue #887: 大量请求被堵住,进到服务时已经超时
lwmonster opened a new issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887 对我们的服务进行压测发现一个现象: **qps 达到 2000 多的时候,cpu 利用率只有 50%,不会再升高了。此时从客户端看,有大量超时,超时时间是 70ms。** ![image](https://user-images.githubusercontent.com/4264580/62999836-34b88c80-bea2-11e9-9f91-855cb26321ae.png) ![image](https://user-images.githubusercontent.com/4264580/6343-c3c5a480-bea2-11e9-8dd6-c5ce90a70c42.png) 从上图可以看到,有超过一半的请求都超时了。 ## 首先排查系统资源问题,感觉并不存在 io-bound 问题: ![image](https://user-images.githubusercontent.com/4264580/62999714-dbe8f400-bea1-11e9-8b7b-9e4e08b1fcc5.png) ![image](https://user-images.githubusercontent.com/4264580/62999736-eacfa680-bea1-11e9-8cd4-7db62d68e5fd.png) ![image](https://user-images.githubusercontent.com/4264580/62999744-ee632d80-bea1-11e9-950a-279d57d9dbee.png) ![image](https://user-images.githubusercontent.com/4264580/62999748-f15e1e00-bea1-11e9-8195-f20be5ad46d4.png) ## 然后查看 rpcz: ![image](https://user-images.githubusercontent.com/4264580/63000149-0a1b0380-bea3-11e9-9ace-1d90a9666f69.png) ![image](https://user-images.githubusercontent.com/4264580/63000152-0c7d5d80-bea3-11e9-8310-da05e7617e92.png) 从上图可以看出,服务端的实际处理时间很短,感觉请求都堵在了网络上,所以怀疑网络带宽不足,接下来测试网络带宽。 ## 网络带宽测试 ![image](https://user-images.githubusercontent.com/4264580/63000625-31260500-bea4-11e9-9992-548f1ad89f02.png) 可以看到万兆网卡的带宽也没有问题。 ## contention profiling ![image](https://user-images.githubusercontent.com/4264580/63000690-5a469580-bea4-11e9-9413-dd07d55cd641.png) 又做了一下 contention profiling ,也没感觉有啥问题。 思路到这里卡住了,现在看到的现象是:从客户端发起请求,到 brpc Received request 这段时间比较长,各位神牛帮忙看看? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] lwmonster closed issue #763: rpc 请求卡住了,不超时
lwmonster closed issue #763: rpc 请求卡住了,不超时 URL: https://github.com/apache/incubator-brpc/issues/763 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] jamesge commented on issue #887: 大量请求被堵住,进到服务时已经超时
jamesge commented on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521138048 client端是瓶颈么? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时
lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521149389 > client端是瓶颈么? 意思是说,这个时间有可能耗费在 client 端? 这之前倒是没有考虑过。 戈神,有啥办法确定一下吗? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时
lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521150450 > client端是瓶颈么? 对了,在我做压测的时候,我从 client 端 ping server 端的机器,延时稳定在 1.2ms This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] lwmonster edited a comment on issue #887: 大量请求被堵住,进到服务时已经超时
lwmonster edited a comment on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521150450 > client端是瓶颈么? 对了,在我做压测的同时,我从 client 端 ping server 端的机器,延时稳定在 1.2ms This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] jamesge edited a comment on issue #887: 大量请求被堵住,进到服务时已经超时
jamesge edited a comment on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521159396 "从client端ping server的延时"只能说明它们之间的网络是通顺的,并不能说明压测时的client是否成为了瓶颈。 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] jamesge commented on issue #887: 大量请求被堵住,进到服务时已经超时
jamesge commented on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521159396 "从client端ping server的延时"只能说明它们之间的网络是通顺的,并不能说明压测时client的是否成为了瓶颈。 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时
lwmonster commented on issue #887: 大量请求被堵住,进到服务时已经超时 URL: https://github.com/apache/incubator-brpc/issues/887#issuecomment-521164509 > "从client端ping server的延时"只能说明它们之间的网络是通顺的,并不能说明压测时的client是否成为了瓶颈。 明白,我看看怎么 check 一下客户端 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[VOTE] Release Apache brpc(incubating) 0.9.6-rc02
Hi, guys, another round of vote begins I am pleased to be calling this vote for the release of apache brpc(incubating) 0.9.6-rc02. The source code can be found at: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz The release candidate has been tagged in GitHub as 0.9.6-rc01, available here: https://github.com/apache/incubator-brpc/releases/tag/0.9.6-rc02 The SHA-512 checksum is: cd737f4cc9c69cd15d58fbee0d64f14383ef0dc5727b74e2c1bdcec95fcaa1fad4d8e31fa9341931722552231f5b38aa5acfb5558007a03cc6b77f3c4b20c611 which can be found via: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz.sha512 The signature can be found via: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz.asc KEYS file is available here: https://dist.apache.org/repos/dist/dev/incubator/brpc/KEYS [Release Note] * Health (of a connection) can be checked at rpc-level * Fix SSL-related compilation issues on Mac * Support SSL-replacement lib MesaLink * Support consistent hashing with ketama algo. * bvar variables can be exported for prometheus services * String[Multi]Splitter supports '\0' as separator * Support for bilibili discovery service * Improved CircuitBreaker * grpc impl. supports timeout Please vote on releasing this package as: Apache brpc(incubating) 0.9.6-rc02 This vote will be open until “Fri Oct 19 2019 00:00:00 GMT+0800 (CST)" and passes if a majority of at least three +1 Apache brpc IPMC votes are cast. [ ] +1 Release this package [ ] 0 I don't feel strongly about it, but don't object [ ] -1 Do not release this package because... Checklist for reference: [ ]Are release files in correct location? [ ]Do release files have the word incubating in their name? [ ] Are the digital signature and hashes correct? [ ] Does DISCLAIMER file exist? [ ]Do LICENSE and NOTICE files exists? [ ] Is the LICENSE and NOTICE text correct? [ ] Is the NOTICE year correct? [ ] Un-included software dependencies are not mentioned in LICENSE or NOTICE? [ ] License information is not mentioned in NOTICE? Is there any 3rd party code contained inside the release? If so: [ ] Does the software have a compatible license? [ ] Are all software licenses mentioned in LICENSE? [ ] Is the full text of the licenses (or pointers to it) in LICENSE? Is any of this code Apache licensed? Do they have NOTICE files? If so: [ ]Have relevant parts of those NOTICE files been added to this NOTICE file? [ ]Do all source files have ASF headers? [ ] Do the contents of the release match with what's tagged in version control? [ ] Are there any unexpected binary files in the release? [ ] Can you compile from source? Are the instruction clear? Anyone can participate in testing and voting, not just committers, please feel free to try out the release candidate and provide your votes. Thanks
Re: [VOTE] Release Apache brpc(incubating) 0.9.6-rc02
+1 在 2019/8/14 下午9:46,“tan zhongyi” 写入: Hi, guys, another round of vote begins I am pleased to be calling this vote for the release of apache brpc(incubating) 0.9.6-rc02. The source code can be found at: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz The release candidate has been tagged in GitHub as 0.9.6-rc01, available here: https://github.com/apache/incubator-brpc/releases/tag/0.9.6-rc02 The SHA-512 checksum is: cd737f4cc9c69cd15d58fbee0d64f14383ef0dc5727b74e2c1bdcec95fcaa1fad4d8e31fa9341931722552231f5b38aa5acfb5558007a03cc6b77f3c4b20c611 which can be found via: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz.sha512 The signature can be found via: https://dist.apache.org/repos/dist/dev/incubator/brpc/0.9.6-rc02/apache-brpc-0.9.6.rc02-incubating-src.tar.gz.asc KEYS file is available here: https://dist.apache.org/repos/dist/dev/incubator/brpc/KEYS [Release Note] * Health (of a connection) can be checked at rpc-level * Fix SSL-related compilation issues on Mac * Support SSL-replacement lib MesaLink * Support consistent hashing with ketama algo. * bvar variables can be exported for prometheus services * String[Multi]Splitter supports '\0' as separator * Support for bilibili discovery service * Improved CircuitBreaker * grpc impl. supports timeout Please vote on releasing this package as: Apache brpc(incubating) 0.9.6-rc02 This vote will be open until “Fri Oct 19 2019 00:00:00 GMT+0800 (CST)" and passes if a majority of at least three +1 Apache brpc IPMC votes are cast. [ ] +1 Release this package [ ] 0 I don't feel strongly about it, but don't object [ ] -1 Do not release this package because... Checklist for reference: [ ]Are release files in correct location? [ ]Do release files have the word incubating in their name? [ ] Are the digital signature and hashes correct? [ ] Does DISCLAIMER file exist? [ ]Do LICENSE and NOTICE files exists? [ ] Is the LICENSE and NOTICE text correct? [ ] Is the NOTICE year correct? [ ] Un-included software dependencies are not mentioned in LICENSE or NOTICE? [ ] License information is not mentioned in NOTICE? Is there any 3rd party code contained inside the release? If so: [ ] Does the software have a compatible license? [ ] Are all software licenses mentioned in LICENSE? [ ] Is the full text of the licenses (or pointers to it) in LICENSE? Is any of this code Apache licensed? Do they have NOTICE files? If so: [ ]Have relevant parts of those NOTICE files been added to this NOTICE file? [ ]Do all source files have ASF headers? [ ] Do the contents of the release match with what's tagged in version control? [ ] Are there any unexpected binary files in the release? [ ] Can you compile from source? Are the instruction clear? Anyone can participate in testing and voting, not just committers, please feel free to try out the release candidate and provide your votes. Thanks
[GitHub] [incubator-brpc] kevinis commented on issue #878: Use prebuilt MesaLink package in travis CI
kevinis commented on issue #878: Use prebuilt MesaLink package in travis CI URL: https://github.com/apache/incubator-brpc/pull/878#issuecomment-521348656 Any comments on the new commits pushed 5 days ago? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
Showcase your project at ApacheCON at a Podling's Shark Tank
Hi Podlings! in less than a month we're going to have our first ApacheCON this year -- the one in Las Vegas. In about two month there will be one more in Berlin. These are not your regular ApacheCONs -- these are 20th Anniversary of ASF ApacehCONs! In other words, these are not to be missed! And even if your talk didn't get accepted -- you still get an opportunity to highlight your project to, what's likely going to be the biggest audience attending. Here's how: if you (or any community member who's passionate about your project) are going to be at either of those ApacheCONs consider signing up for Podling's Shark Tank events: https://www.apachecon.com/acna19/s/#/scheduledEvent/1038 https://aceu19.apachecon.com/session/podlings-shark-tank Each project presenting will get ~10 min for the pitch and ~5 min of panel grilling them on all sorts of things. Kind of like this ;-) https://www.youtube.com/watch?v=wmenN7NEdBc You've got nothing to lose (in fact, the opposite: you're likely to get a prize!) and you will get a chance to receive feedback that might actually help you grow your community and ultimately graduate to the TLP status. And! Given our awesome panel of judges: * Myrle Krantz * Justin Mclean * Craig Russel * Shane Curcuru We guarantee this to be a fun and useful event for your community! We will be tracking signups over here: https://wiki.apache.org/apachecon/ACNA19PodlingSharkTank https://wiki.apache.org/apachecon/ACEU19PodlingSharkTank but for now: SIMPLY REPLY TO THIS EMAIL if you're interested. It is first come, first serve -- so don't delay -- sign up today! Thanks, Roman. - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] jamesge merged pull request #878: Use prebuilt MesaLink package in travis CI
jamesge merged pull request #878: Use prebuilt MesaLink package in travis CI URL: https://github.com/apache/incubator-brpc/pull/878 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[incubator-brpc] branch master updated (16ef588 -> 024c969)
This is an automated email from the ASF dual-hosted git repository. jamesge pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/incubator-brpc.git. from 16ef588 Merge pull request #881 from zyearn/add_missing_license new cec08d4 Enable MesaLink build in travis CI new 4adb689 Link MesaLink in CMake if enabled new 862be4e Link MesaLink in bazel if enabled new 2f1023b Download prebuilt MesaLink deb package if USE_MESALINK is true new 91e0afe Link libmesalink/libssl selectively in CMake new 024c969 Merge pull request #878 from mesalock-linux/mesalink_1.0 The 1831 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml | 8 +--- BUILD | 20 +--- CMakeLists.txt| 23 ++- build_in_travis_ci.sh | 6 +++--- 4 files changed, 47 insertions(+), 10 deletions(-) - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org
[GitHub] [incubator-brpc] zyearn opened a new pull request #888: add separate PURPOSE for all options in ci
zyearn opened a new pull request #888: add separate PURPOSE for all options in ci URL: https://github.com/apache/incubator-brpc/pull/888 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@brpc.apache.org For additional commands, e-mail: dev-h...@brpc.apache.org