On Thu, Jan 5, 2017 at 6:09 AM, Rajeshkumar J <rajeshkumarit8...@gmail.com> wrote:
> Adding this > Here Clauster have three machines each have 125 GB as RAM in which 70 GB is > free, One Machine acts as HMaster and also Region Server. Another machine > acts as Secondary Hmaster and also Region Server. Third Machine is a > dedicated Hregion Server. We have about 7000 Regions. > Maybe is this the problem?? Containing more regions > > Hard to diagnose with so little info. Given the above, you have only a few (if beefy) machines with more than the usual allotment of regions. Suggest you do some background reading and study your cluster while under load looking at resource usage, metrics, logs and the usual suspects. St.Ack > On Thu, Jan 5, 2017 at 3:27 AM, Stack <st...@duboce.net> wrote: > > > (Thanks for jumping in Billy) > > > > Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be on: > > https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1) > > > > Otherwise look at other factors. Are the servers loaded? GC on > > client/server or resources being taken up by adjacent processes; etc. > > > > St.Ack > > > > > > > > On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson <williamrwat...@gmail.com> > > wrote: > > > > > Disclaimer: this is what I think I know and I'm not an HBase > > contributor... > > > > > > Anytime we get scanner timeouts like that, we first expand the timeout > > > itself like you're doing to fix the short-term problem. But then we > look > > at > > > our rowkey design and what we're doing. Timeouts like that are usually > > > symptomatic of bad rowkey utilization, too few servers or too many > > regions > > > per server (and therefore too little RAM to serve the regions). > > > > > > Make sure you're using good rowkey design and actually taking advantage > > of > > > your rowkey. HBase wasn't really intended to do random scans of columns > > > without rowkey filters which is why I think that timeout is set so low > by > > > default. > > > > > > This page helps: > > > > > > http://hbase.apache.org/0.94/book/rowkey.design.html > > > > > > William Watson > > > Lead Software Engineer > > > > > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J < > > rajeshkumarit8...@gmail.com > > > > > > > wrote: > > > > > > > Hi Stack, > > > > > > > > Yes it worked. I had a serious doubt regarding lease does not > exist > > > > exception. We have 30 tables which have 45 million records each. We > are > > > > scanning the table initially process is successful but after it gets > > > failed > > > > with lease does not exist exception. By changing hbase.rpc.timeout > and > > > > hbase.client.scanner.timeout.period it got succeeded. It runs for > > > sometime > > > > but again it fails. Then again increasing the value it gets > succeeded. > > > > Initially I have default values for these two property 60000 and then > > > > changed to 70000 and then changed to 80000. > > > > > > > > Can you tell me how to resolve this? > > > > > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack <st...@duboce.net> wrote: > > > > > > > > > Yeah. Try it. > > > > > S > > > > > > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J < > > > > rajeshkumarit8...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > So I need to add property in log4j.properties as below > > > > > > > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > > > > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack <st...@duboce.net> wrote: > > > > > > > > > > > > > See http://hbase.apache.org/book.html#log4j > > > > > > > > > > > > > > Or to set it temporarily via UI, see > > > > > > > http://hbase.apache.org/book.html#trouble.log.levels > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J < > > > > > > rajeshkumarit8...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Do we have to include this property in hbase-site.xml? > > > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack <st...@duboce.net> > > wrote: > > > > > > > > > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > > > > > > > rajeshkumarit8...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Do all rpc calls are logged in hbase region server log? > > Also > > > I > > > > > need > > > > > > > to > > > > > > > > > find > > > > > > > > > > the time taken for each scan calls is this possible > > > > > > > > > > > > > > > > > > > > If you enable trace level logging for the rpcserver > class, > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a > > > bunch > > > > of > > > > > > > > detail > > > > > > > > > on each rpc invocation (including timings, size, etc): > > > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
