Vimal, In my case, for a single client my queries take less than a second (sub-second performance is on what we were shooting for). But the same queries when run concurrently gives completely degraded performance. That is the reason I wrote that no-op test case which I have attached in the test HBASE-9410 bug. So many queries trying to execute on a region lead to some sort of lock contentions, is my guess which is causing the slow performance. My test case runs fine on 0.95.2 but on 0.94.x even for about 8 concurrent clients it is very high and I see these errors in the log. You should also try running the test case. You will see bad performance even though the coprocessor function does not do anything. Regards, - kiru
Kiru Pakkirisamy | webcloudtech.wordpress.com ________________________________ From: Vimal Jain <[email protected]> To: Kiru Pakkirisamy <[email protected]> Cc: Himanshu Vashishtha <[email protected]>; "[email protected]" <[email protected]>; lars hofhansl <[email protected]> Sent: Wednesday, October 23, 2013 10:50 PM Subject: Re: Warning messages in hbase logs Hi Kiru, Can you please elaborate on this ? On Mon, Oct 21, 2013 at 5:30 PM, Vimal Jain <[email protected]> wrote: Hi Kiru, >Thanks for the reply. >My understanding is not very clear with respect to these warning messages in >region server logs. >When i read documentation , it says its related to slow query logs. >So is it a issue in earlier versions of hbase and is fixed in 0.96 or is it an >issue in my configuration and i have to set some parameters to prevent this? >Please help in understanding the exact issue. > > > > >On Sat, Oct 19, 2013 at 3:31 AM, Kiru Pakkirisamy <[email protected]> >wrote: > >Vimal, >>Maybe you should try running the test case for HBASE-9410 (with some >>modifications) and report back. >>This way Lars/committers/contributors can work on this bug with confirmation. >> >> >> >>The test case makes a coprocessor call (which does nothing) on a table with >>regions in all the region servers. >>Feel free to send me any email in this regards. >> >>Regards, >>- kiru >> >> >> >>Kiru Pakkirisamy | webcloudtech.wordpress.com >> >> >>________________________________ >> From: Vimal Jain <[email protected]> >>To: Himanshu Vashishtha <[email protected]> >>Cc: "[email protected]" <[email protected]>; Kiru Pakkirisamy >><[email protected]>; lars hofhansl <[email protected]> >>Sent: Thursday, October 17, 2013 9:39 PM >> >>Subject: Re: Warning messages in hbase logs >> >> >> >>Hi. >>I am still getting these warnings in region server logs. >>I am still using hbase 0.94.7. >>Is there any impact of these warning messages ? >> >> >> >> >>On Thu, Sep 26, 2013 at 11:19 PM, Himanshu Vashishtha <[email protected]> >>wrote: >> >> >>> >>> >>> >>> >>>On Thu, Sep 26, 2013 at 4:07 AM, Kiru Pakkirisamy >>><[email protected]> wrote: >>> >>>I was on 0.94.11 and still got the same msgs as Vimal. >>>>Ultimately, I moved to 0.95.2. >>>> >>> >>>You know it is a dev release, right? >>> >>> >>> >>> >>>Regards, >>>>- kiru >>>> >>>> >>>>Kiru Pakkirisamy | webcloudtech.wordpress.com >>>> >>>> >>>> >>>>________________________________ >>>> From: lars hofhansl <[email protected]> >>>>To: Vimal Jain <[email protected]>; "[email protected]" >>>><[email protected]> >>>>Cc: Kiru Pakkirisamy <[email protected]> >>>>Sent: Wednesday, September 25, 2013 9:46 PM >>>> >>>>Subject: Re: Warning messages in hbase logs >>>> >>>> >>>>You probably ran into >>>>[HBASE-6870] - HTable#coprocessorExec always scan the whole table (0.94.8) >>>> >>>>Or maybe one of these: >>>> >>>>[HBASE-8806] - Row locks are acquired repeatedly in >>>>HRegion.doMiniBatchMutation for duplicate rows. (0.94.10) >>>>[HBASE-8639] - Poor performance of htable#getscanner in multithreaded >>>>environment due to DNS.getDefaultHost() being called in >>>>ScannerCallable#prepare() (0.94.9) >>>>[HBASE-9087] - Handlers being blocked during reads (0.94.11) >>>> >>>> >>>>-- Lars >>>> >>>> >>>> >>>>________________________________ >>>>From: Vimal Jain <[email protected]> >>>>To: [email protected]; lars hofhansl <[email protected]> >>>>Cc: Kiru Pakkirisamy <[email protected]> >>>>Sent: Wednesday, September 25, 2013 10:01 PM >>>>Subject: Re: Warning messages in hbase logs >>>> >>>> >>>> >>>>I am using Hadoop version - 1.1.2 , Hbase version - 0.94.7 . >>>> >>>> >>>> >>>>On Thu, Sep 26, 2013 at 9:38 AM, lars hofhansl <[email protected]> wrote: >>>> >>>>Eh? That should not be the case, we should figure where 0.95.2 is different >>>>from 0.94. >>>>>With which version of the 0.94 did you see this? >>>>> >>>>>-- Lars >>>>> >>>>> >>>>> >>>>> >>>>>----- Original Message ----- >>>>>From: Kiru Pakkirisamy <[email protected]> >>>>>To: "[email protected]" <[email protected]> >>>>>Cc: >>>>>Sent: Tuesday, September 24, 2013 9:46 PM >>>>>Subject: Re: Warning messages in hbase logs >>>>> >>>>>Are you running many concurrent clients ? I had a similar problem when >>>>>running on 0.94.x and I moved to 0.95.2 for this reason (see HBASE-9410) >>>>> >>>>>Regards, >>>>>- kiru >>>>> >>>>> >>>>>________________________________ >>>>>From: Vimal Jain <[email protected]> >>>>>To: [email protected] >>>>>Sent: Tuesday, September 24, 2013 9:36 PM >>>>>Subject: Warning messages in hbase logs >>>>> >>>>> >>>>>Hi, >>>>>I am seeing some warning message in Hregion log file for quite a few days. >>>>>The message is : >>>>> >>>>>*WARN org.apache.hadoop.ipc.HBaseServer: (responseTooSlow): >>>>>{"processingtimems":12994,"call":"multi(org.apache.hadoop.hbase.client.MultiAction@602cdaf7), >>>>>rpc version=1, client version=29, >>>>>methodsFingerPrint=-1368823753","client":" >>>>>192.168.20.31:42206 >>>>>","starttimems":1380057021850,"queuetimems":0,"class":"HRegionServer","responsesize":0,"method":"multi"} >>>>>* >>>>> >>>>> >>>>>Can someone please help in understanding what this warning message is ? >>>>> >>>>>-- >>>>>Thanks and Regards, >>>>>Vimal Jain >>>>> >>>> >>>> >>>>-- >>>> >>>>Thanks and Regards, >>>>Vimal Jain >>> >> >> >>-- >> >>Thanks and Regards, >>Vimal Jain >> >> > > >-- > >Thanks and Regards, >Vimal Jain -- Thanks and Regards, Vimal Jain
