Tien, Performance issue is quite a complicated problem and the open files count you just mentioned in last mail is not the only factor. According to the persistent connection in dubbo design I don't think is the cause due to your issue and maybe the value you saw in your work machine is not the same on your servers in production. And if you hit the limit some error logs will occur in your log files.
You mentioned that only one core can reach into a high load work status I think you should probably check the IO workers count in the configuration. And an efficient debugging tool which could work here is Arthas[1] you can dig into the JVM process and check which thread is spinning to 100% cpu time. Certainly for further performance optimization you should also adjust the parameters of JVM after you SOLVING this. [1] https://github.com/alibaba/arthas best regards, Jason > On Nov 27, 2019, at 18:25, Tien Dat PHAN <[email protected]> wrote: > > Dear all, > > We just found out that the ulimit on the consumer side is relatively low > (ulimit -n returns 256, we are testing it on MAC OS). We suspect this one may > be the root cause of the low throughput that we have observed. Do any of you > have experience with having performance degradation due to system limit? > > Best regards > Tien Dat > > On 2019/11/19 10:56:27, Tien Dat PHAN <[email protected]> wrote: >> Dear experts, >> >> Could any of you shed some light on this topic? >> >> Best regards >> Tien Dat PHAN >> >> On 2019/11/14 15:07:09, Tien Dat PHAN <[email protected]> wrote: >>> Dear experts, >>> >>> We are having a powerful server (48 cores, four disks and 512 GB RAM) >>> providing the services. >>> From my laptop (2 cores - 4 threads), we start to consume the public >>> services. >>> In order to bench the throughput of our public service provider, we just >>> simply consume a very basic method (an Object receiving method, sending >>> object from consumer and provider receives it, no storage or cache is >>> invoked after that). The Object size is only 1K. >>> >>> We use Dubbo protocol for our service provider. >>> We noticed that: the consumer does not seem to consume to the second core >>> of my laptop (it only consumes one full core). This results in a throughput >>> of 4000 requests per second. >>> To prove that our laptop can reach higher throughput, we run the second >>> process of consuming our public service. With two processes running >>> simultaneously, we can reach the throughput of 7500 request per second, and >>> almost double the CPU usage, in total. >>> Our consumer implementation is multiple threads. So we don't think the >>> implementation is the issue. We suspect that some configuration is needed >>> to improve the performance and concurrency. For the above results, we just >>> use the default configuration of Dubbo. >>> >>> Do you have any best practice example for improving the performance and >>> concurrency of consumer? If yes, could you please enlighten us? >>> >>> Thank you guys in advance. >>> >>> Best regards >>> Tien Dat PHAN >>> >>> >>> >>
