Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-26 Thread James Moore
It appears the rpc handlers are getting stuck reading the hfiles backing the meta table ( based on the block read errors, hedged reads, and hfilescanner lines in the trace ). I'd check that the backing hdfs cluster is healthy and consider allocating more I/O capacity for the meta table. Meta is

Re: HBASE data been deleted! Please HELP

2017-09-28 Thread James Moore
While this is an exotic situation and may not have a solution, there is some guidance from hortonworks on manually editing the transaction log on the namenode to enable data recovery. https://community.hortonworks.com/articles/26181/how-to-recover-accidentally-deleted-file-in-hdfs.html if you're

Re: What is Dead Region Servers and how to clear them up?

2017-05-26 Thread James Moore
In HBase all data is stored in HDFS rather than inside of the region server. The HBase cluster itself considers any individual region server process a region server and when that process dies it is considered a dead region server, this tracking is particularly important during the crash recovery

Re: What is the cause for RegionTooBusyException?

2017-05-26 Thread James Moore
One mechanism for revealing the error in question is to print one of the individual exceptions which are included in the batch calls response. We use this in a few places to allow inspection of individual Exceptions you can see an example of how to do this over here

Re: Regions in Transition: FAILED_CLOSE status

2017-05-23 Thread James Moore
How many region servers are dead? and we're they colocated with DataNodes? On Tue, May 23, 2017 at 5:20 PM, Vladimir Rodionov wrote: > When Master attempt to assign region to RS and assignment fails, there > should be something in RS log file (check errors), > that

Re: What is the cause for RegionTooBusyException?

2017-05-19 Thread James Moore
That error appears to be coming from a batch call, 12695 out of 69261 operations failed with a RegionTooBusyException Some of the causes can be. 1. A full MemStore such as if you write to the MemStore faster than it can flush or if it's too small to fit incoming writes 2. Too many storefiles for

Re: Pre-emptive Fast Fail not used for scanners?

2017-02-28 Thread James Moore
at 2:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Seems to be a bug. > > Mind trying out this change ? > > http://pastebin.com/qim48Se9 > > Thanks > > On Tue, Feb 28, 2017 at 11:06 AM, James Moore <jcmo...@hubspot.com> wrote: > > > The value is true, and

Re: attachment permissions for HBASE jira?

2017-02-23 Thread James Moore
Thanks! On Tue, Feb 21, 2017 at 1:34 PM, Ted Yu <yuzhih...@gmail.com> wrote: > I have added you as contributor. > > You should be able to attach now. > > On Tue, Feb 21, 2017 at 10:31 AM, James Moore <jcmo...@hubspot.com> wrote: > > > Hi All, >

attachment permissions for HBASE jira?

2017-02-21 Thread James Moore
Hi All, I'm trying to attach a patch file to https://issues.apache.org/jira/browse/HBASE-17501 but it appears that my account doesn't have permissions to attach files to HBASE JIRAs. Is there a place to request those permissions? thanks!

impact of increasing the number of Storefiles per region?

2016-12-14 Thread James Moore
Has anyone collected data on the relationship between the number of Storefiles per region and performance/stability? We currently take HDFS backups for disaster recovery and are interested in allowing more Storefiles to accumulate per region between major compactions to reduce backup size.