(Thanks for jumping in Billy) Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be on: https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1)
Otherwise look at other factors. Are the servers loaded? GC on client/server or resources being taken up by adjacent processes; etc. St.Ack On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson <[email protected]> wrote: > Disclaimer: this is what I think I know and I'm not an HBase contributor... > > Anytime we get scanner timeouts like that, we first expand the timeout > itself like you're doing to fix the short-term problem. But then we look at > our rowkey design and what we're doing. Timeouts like that are usually > symptomatic of bad rowkey utilization, too few servers or too many regions > per server (and therefore too little RAM to serve the regions). > > Make sure you're using good rowkey design and actually taking advantage of > your rowkey. HBase wasn't really intended to do random scans of columns > without rowkey filters which is why I think that timeout is set so low by > default. > > This page helps: > > http://hbase.apache.org/0.94/book/rowkey.design.html > > William Watson > Lead Software Engineer > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J <[email protected] > > > wrote: > > > Hi Stack, > > > > Yes it worked. I had a serious doubt regarding lease does not exist > > exception. We have 30 tables which have 45 million records each. We are > > scanning the table initially process is successful but after it gets > failed > > with lease does not exist exception. By changing hbase.rpc.timeout and > > hbase.client.scanner.timeout.period it got succeeded. It runs for > sometime > > but again it fails. Then again increasing the value it gets succeeded. > > Initially I have default values for these two property 60000 and then > > changed to 70000 and then changed to 80000. > > > > Can you tell me how to resolve this? > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack <[email protected]> wrote: > > > > > Yeah. Try it. > > > S > > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J < > > [email protected] > > > > > > > wrote: > > > > > > > So I need to add property in log4j.properties as below > > > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack <[email protected]> wrote: > > > > > > > > > See http://hbase.apache.org/book.html#log4j > > > > > > > > > > Or to set it temporarily via UI, see > > > > > http://hbase.apache.org/book.html#trouble.log.levels > > > > > > > > > > St.Ack > > > > > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > Do we have to include this property in hbase-site.xml? > > > > > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack <[email protected]> wrote: > > > > > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > Do all rpc calls are logged in hbase region server log? Also > I > > > need > > > > > to > > > > > > > find > > > > > > > > the time taken for each scan calls is this possible > > > > > > > > > > > > > > > > If you enable trace level logging for the rpcserver class, > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can see a > bunch > > of > > > > > > detail > > > > > > > on each rpc invocation (including timings, size, etc): > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
