Great. Thank you. Checking out now. I hope I get everything assembled. I keep you posted.
On Sun, Aug 24, 2014 at 9:57 AM, Stack <[email protected]> wrote: > I put up patches in the issue Johannes. Hopefully the reproduced > stackoverflow is same as yours. See HBASE-11813. > St.Ack > > > On Sat, Aug 23, 2014 at 9:25 PM, Johannes Schaback < > [email protected]> wrote: > > > We us all plain gets and puts (sometimes batched). > > > > We have hbase.client.keyvalue.maxsize increased to 536870912 bytes on the > > client. That is the only thing I can see. > > > > I am about to send you a zip file with the respective classes to your > email > > address directly. I probably better dont post the code publicly. > > > > We will also attempt to set hbase.ipc.server.callqueue.handler.factor to > 0 > > now. I keep you posted. > > > > Johannes > > > > > > > > > > On Sun, Aug 24, 2014 at 1:06 AM, Stack <[email protected]> wrote: > > > > > I am having trouble reproducing the stack overflow. Some particular > > > response is triggering it (the code here has been around a while). Any > > > particulars on how your client is accessing hbase? Anything unusual? > > > > > > If you were looking for something to try, set > > > hbase.ipc.server.callqueue.handler.factor > > > to 0. Multiple queues is what is new here. It should not make a > > difference > > > but... > > > > > > St.Ack > > > > > > > > > > > > > > > > > > On Sat, Aug 23, 2014 at 1:23 PM, Johannes Schaback < > > > [email protected]> wrote: > > > > > > > Thank you. > > > > > > > > From the proposed resolution I imagine that the RS would then die in > > case > > > > of a handler error. So the question remains what error originally > > occured > > > > in the handler in the first place. The log of the entire lifecycle of > > the > > > > RS (http://schabby.de/wp-content/uploads/2014/08/filtered.txt) does > > not > > > > reveal much to me unfortunately. Do you find anything in there that > > hints > > > > to something that may cause the handler to end up in the > > soon-to-be-fixed > > > > recursion? > > > > > > > > @Ted, the line "at > > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210)" is > all I > > > can > > > > see unfortunately :( > > > > > > > > > > > > > > > > On Sat, Aug 23, 2014 at 9:43 PM, Andrew Purtell <[email protected] > > > > > > wrote: > > > > > > > > > On Sat, Aug 23, 2014 at 12:11 PM, Johannes Schaback < > > > > > [email protected]> wrote: > > > > > > > > > > > Exception in thread > "defaultRpcServer.handler=5,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > at > > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210) > > > > > > at > > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210) > > > > > > at > > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210) > > > > > > at > > > > org.apache.hadoop.hbase.CellUtil$1.advance(CellUtil.java:210) > > > > > > (and so on...) > > > > > > > > > > > > > > > > > > > > > > > > > > > That is the anonymous CellScanner instance we create from > > > > > CellUtil#createCellScanner. See > > > > > https://issues.apache.org/jira/browse/HBASE-11813 > > > > > > > > > > > > > > > > Filtering the .out file for "Exception" shows that several > > handlers > > > > > > crashed > > > > > > > > > > > > like that: > > > > > > > > > > > > Exception in thread > "defaultRpcServer.handler=5,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=18,queue=0,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=23,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=24,queue=0,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > "defaultRpcServer.handler=2,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=11,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=25,queue=1,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=20,queue=2,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=19,queue=1,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > > "defaultRpcServer.handler=15,queue=0,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > "defaultRpcServer.handler=1,queue=1,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > "defaultRpcServer.handler=7,queue=1,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > Exception in thread > "defaultRpcServer.handler=4,queue=1,port=60020" > > > > > > java.lang.StackOverflowError > > > > > > > > > > > > > > > > > > > > > We should fix this so the RegionServer aborts if it loses a handler > > to > > > an > > > > > Error. > > > > > > > > > > > > > > > -- > > > > > Best regards, > > > > > > > > > > - Andy > > > > > > > > > > Problems worthy of attack prove their worth by hitting back. - Piet > > > Hein > > > > > (via Tom White) > > > > > > > > > > > > > > > > > > > > > -- > > > > LadenZeile.de <http://www.ladenzeile.de/> > > > > powered by Visual Meta GmbH - www.visual-meta.com > > > > > > > > Tel.: +49 30 / 609 84 88 20 > > > > Fax: +49 30 / 609 84 88 21 > > > > E-Mail: [email protected] > > > > > > > > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin > > > > Geschäftsführer: Robert M. Maier, Johannes Schaback > > > > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg > > > > USt-IdNr.: DE263760203 > > > > > > > > > > > > > > > -- > > LadenZeile.de <http://www.ladenzeile.de/> > > powered by Visual Meta GmbH - www.visual-meta.com > > > > Tel.: +49 30 / 609 84 88 20 > > Fax: +49 30 / 609 84 88 21 > > E-Mail: [email protected] > > > > Visual Meta GmbH, Schützenstraße 25, 10117 Berlin > > Geschäftsführer: Robert M. Maier, Johannes Schaback > > Handelsregister HRB 115795 B, Amtsgericht Charlottenburg > > USt-IdNr.: DE263760203 > > > -- LadenZeile.de <http://www.ladenzeile.de/> powered by Visual Meta GmbH - www.visual-meta.com Tel.: +49 30 / 609 84 88 20 Fax: +49 30 / 609 84 88 21 E-Mail: [email protected] Visual Meta GmbH, Schützenstraße 25, 10117 Berlin Geschäftsführer: Robert M. Maier, Johannes Schaback Handelsregister HRB 115795 B, Amtsgericht Charlottenburg USt-IdNr.: DE263760203
