Hi Sean: Thx Size of column family is very small < 100 bytes Investigating potential bottleneck spot..Our cluster is small (relatively speaking)..10 node Our hardware is high end (not commodity)
venkatesh -----Original Message----- From: Sean Bigdatafun <[email protected]> To: [email protected] Sent: Fri, Oct 15, 2010 5:28 pm Subject: Re: hbase.client.retries.number On Thu, Oct 14, 2010 at 12:03 PM, Venkatesh <[email protected]> wrote: > > Thanks J-D > > Yeah..Found out the hard way in prod :) set to zero..since client requests > were backing up.. > everything stopped working/region server would n't come up..etc..(did not > realize hbase > client property would be used by server :) > > I reverted all retries back to default.. > > So far everything seems good...(fingers crossed).after making several > tunables along the way.. > > - Using HBase 0.20.6 > > -Processing about 300 million event puts > -85% of requests are under 10 milli.sec..while the mean is about 300 > millisecs..Trying to narrow > that..if it's during our client GC or Hbase pause..Tuning region server > handler count > > This is way slow too. > -mapreduce job to process 40 million records takes about an hour..Majority > in the reduce phase. > Trying to optimize that..by varying buffer size of writes..Going to try > the "in_memory" option as well. > > This is way slow too. > - Full table scan takes about 30 minutes..Is that reasonable for a table > size of 10 mill records? > hbase.client.scanner.caching - If set in hbase-site.xml, Scan calls should > pick that up correct? > This is way slow for a 10 million records table. What size is your column family? > > thanks > venkatesh > > > > > > > > > > > -----Original Message----- > From: Jean-Daniel Cryans <[email protected]> > To: [email protected] > Sent: Thu, Oct 14, 2010 2:39 pm > Subject: Re: hbase.client.retries.number > > > hbase.client.retries.number is used by HConnectionManager, so this > means anything that uses the HBase client. I think some parts of the > region server code use it, or used it at some point, I'd have to dig > in. But definitely never set this to 0, as any region move/split will > kill your client, > > About this RetriesExhaustedException, it seems that either the region > is in an "unknown" state or that it just took a lot of time to close > and be moved. You need to correlate this with the master log (look for > this region's name) since the client cannot possibly know what went on > inside the cluster. > > Also, which version are you using? > > J-D > > On Mon, Oct 11, 2010 at 3:06 PM, Venkatesh <[email protected]> wrote: > > > > BTW..get this exception while trying a new put..& Also, get this > exception on > gets on some region servers > > > > > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact > region server Some server, retryOnlyOne=true, index=0, islastrow=true, > tries=9, > numtries=10, i=0, listsize=1, > region=user_activity,1286789413060_atanackovics_30306_4a3e0812,1286789581757 > for region user_activity,1286789413060_30306_4a3e0812,1286789581757, row > '1286823659253_v6_1_df34b22f', but failed after 10 attempts. > > Exceptions: > > > > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1149) > > > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1230) > > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:666) > > org.apache.hadoop.hbase.client.HTable.close(HTable.java:682) > > > > com.aol.mail.antispam.Profiler.notifyEmailSendActivity.processGetRequest(notifyEmailSendActivity.java:363) > > > > com.aol.mail.antispam.Profiler.notifyEmailSendActivity.doGet(notifyEmailSendActivity.java:450) > > javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > > javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > > > > > > > > > > > > > > > > -----Original Message----- > > From: Venkatesh <[email protected]> > > To: [email protected] > > Sent: Mon, Oct 11, 2010 2:35 pm > > Subject: hbase.client.retries.number > > > > > > > > > > > > HBase was seamless for first couple of weeks..now all kinds of issues in > > production :) fun fun.. > > Curious ..does this property have to match up on "hbase client side" & > region > > server side.. > > > > I've this number set to 0 on region server side & default on client > side.. > > I can't do any put (new) > > > > thanks > > venkatesh > > > > > > > > > > > > > > >
