Nice discovery. Pls. keep us posted once you figure out the root cause. Keep digging :)
Thanks, -Ian. On Thu, Dec 20, 2018 at 5:24 PM Jason Joo <[email protected]> wrote: > Dear guys, > > I have submitted an issue below: > > https://github.com/apache/incubator-dubbo/issues/3024 > > Here is the content of it. > > I noticed that someone maybe hit this issue some days ago and no solution > given out. We indeed have worked on it for a month and want to get some > possible directions or solutions. > > Thanks. > Environment > > - Dubbo version: 2.6.3 > - Operating System version: Centos 7 > - Java version: 1.8 > > Steps to reproduce this issue > > For some time we found some performance jitter under some high loads > points. Generally like the chart below (Figure 1) showing the latency on > provider side didn't change much but latency on consumer side had a big > deterioration. > > Figure 1 > > The count of providers in the figure 1 is 3 in three nodes. Each nodes has > 4 cores and 16G. > > We have tried: > > - Check the nat gateway band width and connections monitor > 30MB/s max and less then 3000 connections. They don't hit the limit. > - Try to separate to a new connection for one service to see whether > the problem is in ONE connection model. > The chart is capture after that and issue continues. > - Check the tomcat thread pool usage > Though i think it's irrelevant to the issue but i have integrated > Sentinel to make sure the tomcat thread pool will not be full. > - Doubt if it's a vpc's performance problem (They are deployed in > Aliyun.com <http://aliyun.com/>) but we found similar issue in our IDC > cluster as Figure 2. > 6 Providers deployed. > > Figure 2 > > Now we are continue figuring out what the problem is. Did anyone hit this > issue and kindly gives some advices? > > > best regards, > > Jason > >
