Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-30 Thread Stack
On Fri, Jun 28, 2019 at 9:47 AM James Kebinger wrote: > We were thinking that meta replication would make a hot meta table on a > busy cluster less hot rather than more hot. You are running read replicas or replicating the meta to another cluster? S > Is the meta replication feature > for

Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-28 Thread James Kebinger
Hey Stack, thanks for your response We were thinking that meta replication would make a hot meta table on a busy cluster less hot rather than more hot. Is the meta replication feature for uptime rather than performance if it, as you said, will "up the temperature even more"? I was surprised that

Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-27 Thread Stack
If early 2.0 or 2.1 versions, perhaps it is HBASE-21292 ? S On Thu, Jun 27, 2019 at 10:13 PM Stack wrote: > On Wed, Jun 26, 2019 at 6:47 AM James Kebinger > wrote: > >> Hello, >> We're experimenting with meta table replication > > > Replicating the meta table itself? What you thinking? > > >

Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-27 Thread Stack
On Wed, Jun 26, 2019 at 6:47 AM James Kebinger wrote: > Hello, > We're experimenting with meta table replication Replicating the meta table itself? What you thinking? > but have found that the > region servers hosting the replicas will get extremely high priority RPC > handler usage,

Re: High Priority RPC handler usage with meta replication - block read errors?

2019-06-26 Thread James Moore
It appears the rpc handlers are getting stuck reading the hfiles backing the meta table ( based on the block read errors, hedged reads, and hfilescanner lines in the trace ). I'd check that the backing hdfs cluster is healthy and consider allocating more I/O capacity for the meta table. Meta is

High Priority RPC handler usage with meta replication - block read errors?

2019-06-26 Thread James Kebinger
Hello, We're experimenting with meta table replication but have found that the region servers hosting the replicas will get extremely high priority RPC handler usage, sometimes all the way to 100% at which point clients start to experience errors - the priority RPC handler usage is much higher