your CMS is not tuned, please find how tune CMS on java web site. then Xmn is too small, this is not suitable for frequent multi get. please change Xmn=1g
--Send from my Sony mobile. On Apr 12, 2013 10:29 PM, "anand nalya" <[email protected]> wrote: > the block cache size is 0.25 > > Each row holds around 2KB data, so size should not be an issue at least > till the number of records is less than 1000. > > Also, > HBASE_HEAPSIZE=8000 > HBASE_OPTS="-XX:+UseConcMarkSweepGC" > HBASE_REGIONSERVER_OPTS="-Xmx4g -Xms4g -Xmn256m -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" > > On 12 April 2013 19:30, Azuryy Yu <[email protected]> wrote: > > > and whats your block cache size? > > > > there are two possible reasons: > > 1. result is too big > > 2. GC options are not optimized. > > > > can you paste your gc options here? > > > > --Send from my Sony mobile. > > On Apr 12, 2013 9:53 PM, "anand nalya" <[email protected]> wrote: > > > > > Hi Azuryy, > > > > > > I'm using the default cache size of 100 for scanner. For mutigets, I've > > > tried with 1 (13ms), 10(356ms), 100(1135ms), 1000(4330ms), and > > > 10000(17744ms) keys. Normal workload will be around 10000 keys at a > time. > > > > > > Are there any optimization that can be done for multigets. Is HBase a > > good > > > candidate for usecase? > > > > > > Thanks, > > > Anand > > > > > > > > > On 12 April 2013 19:17, anand nalya <[email protected]> wrote: > > > > > > > Hi Ted, > > > > > > > > The region servers are not loaded. It is showing 5% CPU usage. The > > > > datanode is showing around 50% CPU utilization. disk IO is aroung > > 7Mbps. > > > > > > > > There is nothing noticeable in GC log. > > > > > > > > Thanks, > > > > Anand > > > > > > > > > > > > On 12 April 2013 02:56, Ted Yu <[email protected]> wrote: > > > > > > > >> How loaded were the region servers when the query was running ? > > > >> > > > >> Did you check GC log ? > > > >> > > > >> Thanks > > > >> > > > >> On Thu, Apr 11, 2013 at 8:23 AM, anand nalya <[email protected] > > > > > >> wrote: > > > >> > > > >> > Hi, > > > >> > > > > >> > I'm using HBase 0.94.5 with thrift server. I'm trying to get the > > rows > > > >> from > > > >> > HBase using > > > >> > > > > >> > > > > org.apache.hadoop.hbase.thrift.generated.Hbase.Client.getRows(ByteBuffer, > > > >> > List<ByteBuffer>, Map<ByteBuffer, ByteBuffer>) but it is giving > > > results > > > >> > very slowly (around 2 mins for 100 rows). For larger number of > > > records, > > > >> > there is no response. > > > >> > > > > >> > I've two region server and a total of 128 regions. Total data size > > is > > > >> > around 250GB (250 million records) uniformly distributed across > > > regions. > > > >> > > > > >> > Regionserver only show the following in its log: > > > >> > > > > >> > 2013-04-11 19:53:44,535 ERROR > > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: > > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting > > call > > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc > > > >> version=1, > > > >> > client version=29, methodsFingerPrint=-1368823753 from > > > >> > 192.168.145.195:52277after 74994 ms, since caller disconnected > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664) > > > >> > at > > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576) > > > >> > at > > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516) > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > >> > at java.lang.reflect.Method.invoke(Method.java:597) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) > > > >> > at > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > > > >> > 2013-04-11 19:53:46,121 ERROR > > > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: > > > >> > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting > > call > > > >> > multi(org.apache.hadoop.hbase.client.MultiAction@49ac272), rpc > > > >> version=1, > > > >> > client version=29, methodsFingerPrint=-1368823753 from > > > >> > 192.168.145.195:52277after 76580 ms, since caller disconnected > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnected(HBaseServer.java:436) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3723) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3643) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3626) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3664) > > > >> > at > > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4576) > > > >> > at > > > >> org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4549) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2042) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3516) > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > > >> > at java.lang.reflect.Method.invoke(Method.java:597) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) > > > >> > at > > > >> > > > > >> > > > > > > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > > > >> > > > > >> > > > > >> > Any idea what might be wrong here? > > > >> > > > > >> > Thanks, > > > >> > Anand > > > >> > > > > >> > > > > > > > > > > > > > >
