Thanks J-D Yeah..Found out the hard way in prod :) set to zero..since client requests were backing up.. everything stopped working/region server would n't come up..etc..(did not realize hbase client property would be used by server :)
I reverted all retries back to default.. So far everything seems good...(fingers crossed).after making several tunables along the way.. - Using HBase 0.20.6 -Processing about 300 million event puts -85% of requests are under 10 milli.sec..while the mean is about 300 millisecs..Trying to narrow that..if it's during our client GC or Hbase pause..Tuning region server handler count -mapreduce job to process 40 million records takes about an hour..Majority in the reduce phase. Trying to optimize that..by varying buffer size of writes..Going to try the "in_memory" option as well. - Full table scan takes about 30 minutes..Is that reasonable for a table size of 10 mill records? hbase.client.scanner.caching - If set in hbase-site.xml, Scan calls should pick that up correct? thanks venkatesh -----Original Message----- From: Jean-Daniel Cryans <[email protected]> To: [email protected] Sent: Thu, Oct 14, 2010 2:39 pm Subject: Re: hbase.client.retries.number hbase.client.retries.number is used by HConnectionManager, so this means anything that uses the HBase client. I think some parts of the region server code use it, or used it at some point, I'd have to dig in. But definitely never set this to 0, as any region move/split will kill your client, About this RetriesExhaustedException, it seems that either the region is in an "unknown" state or that it just took a lot of time to close and be moved. You need to correlate this with the master log (look for this region's name) since the client cannot possibly know what went on inside the cluster. Also, which version are you using? J-D On Mon, Oct 11, 2010 at 3:06 PM, Venkatesh <[email protected]> wrote: > > BTW..get this exception while trying a new put..& Also, get this exception > on gets on some region servers > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server, retryOnlyOne=true, index=0, islastrow=true, tries=9, numtries=10, i=0, listsize=1, region=user_activity,1286789413060_atanackovics_30306_4a3e0812,1286789581757 for region user_activity,1286789413060_30306_4a3e0812,1286789581757, row '1286823659253_v6_1_df34b22f', but failed after 10 attempts. > Exceptions: > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149) > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230) > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666) > org.apache.hadoop.hbase.client.HTable.close(HTable.java:682) > > com.aol.mail.antispam.Profiler.notifyEmailSendActivity.processGetRequest(notifyEmailSendActivity.java:363) > > com.aol.mail.antispam.Profiler.notifyEmailSendActivity.doGet(notifyEmailSendActivity.java:450) > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > > > -----Original Message----- > From: Venkatesh <[email protected]> > To: [email protected] > Sent: Mon, Oct 11, 2010 2:35 pm > Subject: hbase.client.retries.number > > > > > > HBase was seamless for first couple of weeks..now all kinds of issues in > production :) fun fun.. > Curious ..does this property have to match up on "hbase client side" & region > server side.. > > I've this number set to 0 on region server side & default on client side.. > I can't do any put (new) > > thanks > venkatesh > > > > > >
