Hi.. Based on this little information ....Check you rowkey design along with what splitting policy you are using.
Thanks Manjeet On Mon, Jan 9, 2017 at 3:20 PM, Rajeshkumar J <[email protected]> wrote: > we have 7000 regions in which we only have data in 20 regions. Whether this > may be the reason for this? > > On Thu, Jan 5, 2017 at 10:58 PM, Stack <[email protected]> wrote: > > > On Thu, Jan 5, 2017 at 6:09 AM, Rajeshkumar J < > [email protected] > > > > > wrote: > > > > > Adding this > > > Here Clauster have three machines each have 125 GB as RAM in which 70 > GB > > is > > > free, One Machine acts as HMaster and also Region Server. Another > machine > > > acts as Secondary Hmaster and also Region Server. Third Machine is a > > > dedicated Hregion Server. We have about 7000 Regions. > > > Maybe is this the problem?? Containing more regions > > > > > > > > Hard to diagnose with so little info. Given the above, you have only a > few > > (if beefy) machines with more than the usual allotment of regions. > > > > Suggest you do some background reading and study your cluster while under > > load looking at resource usage, metrics, logs and the usual suspects. > > > > St.Ack > > > > > > > > > On Thu, Jan 5, 2017 at 3:27 AM, Stack <[email protected]> wrote: > > > > > > > (Thanks for jumping in Billy) > > > > > > > > Smaller Scans? (This is good on Scans in 1.1.+ which you seem to be > on: > > > > https://blogs.apache.org/hbase/entry/scan_improvements_in_hbase_1) > > > > > > > > Otherwise look at other factors. Are the servers loaded? GC on > > > > client/server or resources being taken up by adjacent processes; etc. > > > > > > > > St.Ack > > > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 11:00 AM, Billy Watson < > > [email protected]> > > > > wrote: > > > > > > > > > Disclaimer: this is what I think I know and I'm not an HBase > > > > contributor... > > > > > > > > > > Anytime we get scanner timeouts like that, we first expand the > > timeout > > > > > itself like you're doing to fix the short-term problem. But then we > > > look > > > > at > > > > > our rowkey design and what we're doing. Timeouts like that are > > usually > > > > > symptomatic of bad rowkey utilization, too few servers or too many > > > > regions > > > > > per server (and therefore too little RAM to serve the regions). > > > > > > > > > > Make sure you're using good rowkey design and actually taking > > advantage > > > > of > > > > > your rowkey. HBase wasn't really intended to do random scans of > > columns > > > > > without rowkey filters which is why I think that timeout is set so > > low > > > by > > > > > default. > > > > > > > > > > This page helps: > > > > > > > > > > http://hbase.apache.org/0.94/book/rowkey.design.html > > > > > > > > > > William Watson > > > > > Lead Software Engineer > > > > > > > > > > On Wed, Jan 4, 2017 at 1:41 AM, Rajeshkumar J < > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Stack, > > > > > > > > > > > > Yes it worked. I had a serious doubt regarding lease does not > > > exist > > > > > > exception. We have 30 tables which have 45 million records each. > We > > > are > > > > > > scanning the table initially process is successful but after it > > gets > > > > > failed > > > > > > with lease does not exist exception. By changing > hbase.rpc.timeout > > > and > > > > > > hbase.client.scanner.timeout.period it got succeeded. It runs > for > > > > > sometime > > > > > > but again it fails. Then again increasing the value it gets > > > succeeded. > > > > > > Initially I have default values for these two property 60000 and > > then > > > > > > changed to 70000 and then changed to 80000. > > > > > > > > > > > > Can you tell me how to resolve this? > > > > > > > > > > > > On Wed, Jan 4, 2017 at 11:42 AM, Stack <[email protected]> wrote: > > > > > > > > > > > > > Yeah. Try it. > > > > > > > S > > > > > > > > > > > > > > On Tue, Jan 3, 2017 at 9:18 PM, Rajeshkumar J < > > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > So I need to add property in log4j.properties as below > > > > > > > > > > > > > > > > log4j.logger.org.apache.hadoop.hbase.ipc.RpcServer = TRACE > > > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 10:23 AM, Stack <[email protected]> > > wrote: > > > > > > > > > > > > > > > > > See http://hbase.apache.org/book.html#log4j > > > > > > > > > > > > > > > > > > Or to set it temporarily via UI, see > > > > > > > > > http://hbase.apache.org/book.html#trouble.log.levels > > > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > On Tue, Jan 3, 2017 at 6:58 PM, Rajeshkumar J < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Do we have to include this property in hbase-site.xml? > > > > > > > > > > > > > > > > > > > > On Wed, Jan 4, 2017 at 2:12 AM, Stack <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > > > > On Mon, Jan 2, 2017 at 10:32 PM, Rajeshkumar J < > > > > > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Do all rpc calls are logged in hbase region server > log? > > > > Also > > > > > I > > > > > > > need > > > > > > > > > to > > > > > > > > > > > find > > > > > > > > > > > > the time taken for each scan calls is this possible > > > > > > > > > > > > > > > > > > > > > > > > If you enable trace level logging for the rpcserver > > > class, > > > > > > > > > > > org.apache.hadoop.hbase.ipc.RpcServer, then you can > see > > a > > > > > bunch > > > > > > of > > > > > > > > > > detail > > > > > > > > > > > on each rpc invocation (including timings, size, etc): > > > > > > > > > > > > > > > > > > > > > > St.Ack > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- luv all
