> > I am not familiar with thread-dumping or profiler. Is there any reference > that I can study about how to use those tools to analyze this kind of issue?
You can start collecting jstack [1] frames of the RS process while your test is running. Jstack command dumps state of all threads from the given java process, you can then check which threads are blocked/running most of the time. Below sample script would collect 20 frames with 1 second interval, notice you would need to replace RSPID by the actual RS pid, and it assumes RS process is running as hbase user (jstack needs to be ran as same user as the java process). for i in $(seq 1 20); do sudo -u hbase /usr/java/default/bin/jstack $RSPID > /tmp/rs-jstack-$i.txt; sleep 1s; done > I just observed that all resources CPU, memory,. I/O are still idle, but > the avg response time for a single request (put/get) become slower when the > concurrency increasing. > Interesting, is the CP code using any type of locks? How is the response time for single requests being measured? Is it possible to run this same client with the CP disabled, and see if the performance issue still manifests? [1] https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jstack.html Em qui, 17 de jan de 2019 às 16:14, Ankit Singhal <ankitsingha...@gmail.com> escreveu: > I hope you have configured 200+ handlers in server config and not creating > a new connection in every thread. Sharing your java code, hbase version, > machine configuration and hbase-site.xml may help us comment more. > > Regards, > Ankit Singhal > > On Wed, Jan 16, 2019 at 11:35 PM ming.liu <ming....@esgyn.cn> wrote: > > > Thanks, Stack, > > > > I still cannot figure out the root cause. I am not familiar with > > thread-dumping or profiler. Is there any reference that I can study about > > how to use those tools to analyze this kind of issue? > > Or is there some similar performance test that I can refer to? Maybe my > > test is wrong. > > I just write a simple java program to get a row from a table in a loop. > > And by spawning this program from 1 to 200, and check the response time. > > I just observed that all resources CPU, memory,. I/O are still idle, but > > the avg response time for a single request (put/get) become slower when > the > > concurrency increasing. All the request goes to the same region in my > test. > > So I am trying to understand those parameters. > > > > Thanks, > > Ming > > > > -----Original Message----- > > From: Stack <st...@duboce.net> > > Sent: Thursday, January 17, 2019 3:17 AM > > To: Hbase-User <user@hbase.apache.org> > > Subject: Re: coprocessor and hbase.regionserver.handler.count > > > > On Mon, Jan 14, 2019 at 5:37 PM ming.liu <ming....@esgyn.cn> wrote: > > > > > Hi, all, > > > > > > > > > > > > Our application is using coprocessor to communicate with HBase, not > using > > > the get()/put() client API. When there are large concurrency, the > > > performance is degrading very fast. > > > > > > > > Can you figure why? Thread-dumping, profiler? > > > > > > > I checked there is hbase.regionserver.handler.count which control how > > many > > > thread in RS to handle client request. Will coprocessor use those same > > > threads? > > > > > > > > Yes. CPs decorate the existing read/write paths serviced by handlers. > > > > S > > > > > > > > > > > > > > > thanks, > > > > > > Ming > > > > > > > > > > > > > > > > >
