Tien, Or maybe you can use single consumer but set the target busy service's "connections" in configuration to "2" or bigger value.
If you integrated dubbo using XML: http://dubbo.apache.org/en-us/docs/user/references/xml/dubbo-service.html <http://dubbo.apache.org/en-us/docs/user/references/xml/dubbo-service.html> Or if you integrated dubbo using annotations: @Reference(connections = 2) And so on. Just try this for another try. best regards, Jason > On Nov 27, 2019, at 22:55, Tien Dat PHAN <[email protected]> wrote: > > Dear Jason, > > You were right. The fact that when we tried to run two processes of consumer, > we increased the total throughput by almost 2X. This means that the > infrastructure and OS (the laptop and Mac OS in this case) is capable of > providing higher throughput. > > So the suspect comes back to the configuration of the consumer process. > > We have tried the Arthas tool that you suggested. A very useful tool in > general. However, it still does not show any interesting lead that can > explain the problem. > > We are stuck for now :( > > Best > Dat > > On 2019/11/27 10:38:32, Jason Joo <[email protected]> wrote: >> Tien, >> >> Performance issue is quite a complicated problem and the open files count >> you just mentioned in last mail is not the only factor. >> According to the persistent connection in dubbo design I don't think is the >> cause due to your issue and maybe the value you saw in your work machine is >> not the same on your servers in production. And if you hit the limit some >> error logs will occur in your log files. >> >> You mentioned that only one core can reach into a high load work status I >> think you should probably check the IO workers count in the configuration. >> And an efficient debugging tool which could work here is Arthas[1] you can >> dig into the JVM process and check which thread is spinning to 100% cpu time. >> >> Certainly for further performance optimization you should also adjust the >> parameters of JVM after you SOLVING this. >> >> >> [1] https://github.com/alibaba/arthas >> >> best regards, >> >> Jason >> >>> On Nov 27, 2019, at 18:25, Tien Dat PHAN <[email protected]> wrote: >>> >>> Dear all, >>> >>> We just found out that the ulimit on the consumer side is relatively low >>> (ulimit -n returns 256, we are testing it on MAC OS). We suspect this one >>> may be the root cause of the low throughput that we have observed. Do any >>> of you have experience with having performance degradation due to system >>> limit? >>> >>> Best regards >>> Tien Dat >>> >>> On 2019/11/19 10:56:27, Tien Dat PHAN <[email protected]> wrote: >>>> Dear experts, >>>> >>>> Could any of you shed some light on this topic? >>>> >>>> Best regards >>>> Tien Dat PHAN >>>> >>>> On 2019/11/14 15:07:09, Tien Dat PHAN <[email protected]> wrote: >>>>> Dear experts, >>>>> >>>>> We are having a powerful server (48 cores, four disks and 512 GB RAM) >>>>> providing the services. >>>>> From my laptop (2 cores - 4 threads), we start to consume the public >>>>> services. >>>>> In order to bench the throughput of our public service provider, we just >>>>> simply consume a very basic method (an Object receiving method, sending >>>>> object from consumer and provider receives it, no storage or cache is >>>>> invoked after that). The Object size is only 1K. >>>>> >>>>> We use Dubbo protocol for our service provider. >>>>> We noticed that: the consumer does not seem to consume to the second core >>>>> of my laptop (it only consumes one full core). This results in a >>>>> throughput of 4000 requests per second. >>>>> To prove that our laptop can reach higher throughput, we run the second >>>>> process of consuming our public service. With two processes running >>>>> simultaneously, we can reach the throughput of 7500 request per second, >>>>> and almost double the CPU usage, in total. >>>>> Our consumer implementation is multiple threads. So we don't think the >>>>> implementation is the issue. We suspect that some configuration is needed >>>>> to improve the performance and concurrency. For the above results, we >>>>> just use the default configuration of Dubbo. >>>>> >>>>> Do you have any best practice example for improving the performance and >>>>> concurrency of consumer? If yes, could you please enlighten us? >>>>> >>>>> Thank you guys in advance. >>>>> >>>>> Best regards >>>>> Tien Dat PHAN >>>>> >>>>> >>>>> >>>> >> >>
