Hi Jon, Do you see something interesting in the RS logs from KVM15 or the HBase Master? one possibility is that if there are no requests to META coming from the Thrift server or external clients, then it might be possible that one or many region servers for some reason are updating META too frequently (e.g. 1000s of regions in transition or a region that might be corrupt region and is getting resubmitted over and over by the MetaEditor for some kind of failure). If you could enable DEBUG level logging in the RSs and the Master for a short period when you see that large number of requests to META that can help us to find out why is that happening.
cheers, esteban. -- Cloudera, Inc. On Fri, Dec 19, 2014 at 9:09 AM, uamadman <[email protected]> wrote: > > Yes, > I tested the following by restarting the cluster and waiting approximately > 5-10 minutes for its initial ramp up. There are no clients asking for data. > In the following example KVM15 was randomly assigned to serve the META > Table. > > root@KVM15:~# lsof -n | grep :60020- | sed 's/.*->//;s/:.*//' | sort | > uniq > -c | sort -nr > 1 10.40.178.180 > 1 10.40.178.179 > 1 10.40.178.176 > 1 10.40.178.170 > 1 10.40.178.169 > 1 10.40.178.164 > 1 10.40.178.162 > 1 10.40.178.160 > 1 10.40.178.130 > 1 10.40.178.129 > 1 10.40.178.128 > 1 10.40.178.110 > 1 10.40.178.109 > 1 10.40.178.108 > > The following returned nothing > root@KVM15:~# lsof -n | grep :9090- | sed 's/.*->//;s/:.*//' | sort | uniq > -c | sort -nr > > > This picture shows the iftop for KVM15 LeftSide | KVM03 Top Right > | KVM05 > Bottom Right > In this example it was running for about 1-2 minutes before i took the > screen shot and had already transmitted ~2GB of data. > https://www.dropbox.com/s/iy0rkgljyo1872m/KVMThroughputExample.png?dl=0 > > At the 10 minute mark KVM15 had transmitted ~18GB of data. > https://www.dropbox.com/s/cgpo1krebrp2jzh/KVMThroughputExample10.png?dl=0 > > Cheers, > Jon > > PS. I'm currently interested in moving from 5.1.2-1.cdh5.1.2.p0.3 to > 5.2.1-1.cdh5.2.1.p0.12 > and from HBASE 0.98.1 to HBASE 0.98.6. But as im in no rush, I will wait > until you guys feel you know what this issue is. > > > > > -- > View this message in context: > http://apache-hbase.679495.n3.nabble.com/Region-Server-Thread-with-a-Single-High-Idle-CPU-tp4066876p4067014.html > Sent from the HBase User mailing list archive at Nabble.com. >
