Suppose you add new nodes at time T, can you check master log between T and T+45(min) to see how long region movements took ?
Cheers On Fri, Aug 22, 2014 at 11:19 AM, Eric K <[email protected]> wrote: > I'm experimenting with HBase (0.94.4) and Hadoop (1.0.4) runtime balancers > on just a very tiny 4 node cluster and finding that performance is bad for > an hour after adding new nodes, even though all the data is supposedly > offloaded within a few minutes. > > Using YCSB-generated requests, I load the database with about 18 GB worth > of data across 12 million records. The keyspace is initially pre-split to > have ~30 regions per node with single replication of data. Then I hit > HBase with read requests from a set of clients so that there are 2000 > requests outstanding, and new requests are immediately made after replies > are received. While these requests are running, after about 2 minutes, I > double the nodes from 4 to 8, add the new node information to the slaves > and regionservers files, start up the new datanode and regionserver > processes, and call the hdfs balancer with the smallest possible threshold > of 1. The hbase.balancer.period is also set to 10 seconds so as to respond > fast to new nodes. The dfs.bandwidth.bandwidthPerSec is set to 8 MB/s per > node, but I have also tried higher numbers that don't bottleneck the > offload rate and gotten similar results. > > The expected response is that about half the data and regions are offloaded > to the new nodes in the next few minutes, and the logs (and hdfsadmin > reports) confirm that this is indeed happening. However, I'm seeing the > throughput drop from 3000 to 500 requests/sec when the nodes are added, > with latencies jumping from ~1s to 4 or 5 seconds, and this poor > performance persists for almost 45 minutes, when it abruptly gets better to > 4000 requests/sec and 500ms latencies. > > I'd appreciate any ideas as to what could be causing that 45 minute > performance delay, and how I can debug this further. If I add the new > nodes and balance onto them before starting up any read requests, > performance is much better from the start. > > Thanks, > Eric >
