Backport has been done in HBASE-6724 Cheers
On Wed, Sep 5, 2012 at 5:09 PM, Himanshu Vashishtha <[email protected] > wrote: > It usually happens in a long running setup (at least for me). Can you > throttle your load? > > Replication is evolving; I'd say update if you can (or backport the > jiras?). > > Himanshu > > > On Wed, Sep 5, 2012 at 5:53 PM, Jeff Whiting <[email protected]> wrote: > > hmm. So if we are on 0.92 what suggestion would you have to prevent the > > problem? > > > > ~Jeff > > > > > > On 9/5/2012 5:23 PM, Himanshu Vashishtha wrote: > >> > >> Number of PRI handlers are governed by > >> "hbase.regionserver.metahandler.count"; default is 10. > >> > >> Increasing their number will not solve it, but will delay its > >> occurring (i don't know about your load etc). > >> > >> Another related jira is HBase-6550. > >> > >> Some more context for your use case: > >> > >> > http://search-hadoop.com/m/WHkTxWj0MW/himanshu+vashistha&subj=Re+Long+running+replication+possible+improvements > >> > >> > >> On Wed, Sep 5, 2012 at 5:18 PM, Jeff Whiting <[email protected]> > wrote: > >>> > >>> It looks like that is problem we are having. We are on 0.92 so we > don't > >>> get > >>> the patch. But one solution seems to be increasing the privileged > >>> handlers. > >>> How do we increase the number of privilege handlers? > >>> > >>> > >>> ~Jeff > >>> > >>> On 9/5/2012 4:47 PM, Himanshu Vashishtha wrote: > >>>> > >>>> Your RS priority handlers are blocked on meta lookup, so it becomes > >>>> unresponsive. Looks like you hitting > >>>> https://issues.apache.org/jira/browse/HBASE-6165 > >>>> You running HBase replication? just confirming. > >>>> > >>>> Himanshu > >>>> > >>>> On Wed, Sep 5, 2012 at 4:39 PM, Stack <[email protected]> wrote: > >>>>> > >>>>> On Wed, Sep 5, 2012 at 2:58 PM, Nathaniel Cook > >>>>> <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>> We ran a jstack on the both the RS process and the hbase shell > process > >>>>>> trying to do the scan. > >>>>>> > >>>>>> Jstack log for RS: > >>>>>> http://pastebin.com/9Y9t5ERE > >>>>>> > >>>>> What JVM (I don't know what (20.10-b01 mixed mode) is). > >>>>> > >>>>> I see a bunch of this: > >>>>> > >>>>> "PRI IPC Server handler 5 on 60020" daemon prio=10 > >>>>> tid=0x00002aaac10a1800 nid=0x92f waiting for monitor entry > >>>>> [0x000000004ab0f000] > >>>>> java.lang.Thread.State: BLOCKED (on object monitor) > >>>>> at ..... > >>>>> > >>>>> But when I go to look for other instances of the object monitor, I > >>>>> don't find any. I see this for each instance of BLOCKED (Or at > least, > >>>>> the two or three I checked). > >>>>> > >>>>> Whats your OS? > >>>>> > >>>>> St.Ack > >>> > >>> > >>> -- > >>> Jeff Whiting > >>> Qualtrics Senior Software Engineer > >>> [email protected] > >>> > >>> > >>> > > > > -- > > Jeff Whiting > > Qualtrics Senior Software Engineer > > [email protected] > > > > > > >
