Hi, I've been doing different experiments with a 5-node cluster with YCSB. We have been testing a number of different configurations, so I have been constantly been wiping our cluster up and setting it up again since we configure everything via Chef. At one point, I was able to get the following stats from our cluster which I was pretty happy with: YCSB Client 0.1
Command line: -load -db com.yahoo.ycsb.db.HBaseClient -Pworkloads/workloada -p columnfamily=family -p recordcount=10000000 -s [OVERALL], RunTime(ms), 1057645.0 [OVERALL], Throughput(ops/sec), 9454.96834949345 [INSERT], Operations, 10000000 [INSERT], AverageLatency(ms), 0.0915235 [INSERT], MinLatency(ms), 0 [INSERT], MaxLatency(ms), 6925 [INSERT], 95thPercentileLatency(ms), 0 [INSERT], 99thPercentileLatency(ms), 0 [INSERT], Return=0, 10000000 However, in our most recent server builds, I seem to very quickly deadlock something in HBase. I've backed through all our old revisions and reverted a number of different configuration settings, and I can't seem to figure out now why the cluster is so slow. Our terasort M/R tests are returning the same values as before, so I do not believe that there is anything wrong external to HBase. The behavior that I see when I kick off the tests is this: [UPDATE], 0, 4765 [UPDATE], 1, 248 [UPDATE], 2, 0 [UPDATE], 3, 0 [UPDATE], 4, 0 Basically, it kicks off a large number of inserts and HBase grinds to a halt. Some number of the writes end up getting inserted (usually around ~50), but then everything stops. Here's the behavior I see with the region servers: npin-172-16-12-203.np.local:60030 1311956094792 requests=50, regions=1, usedHeap=151, maxHeap=16358 npin-172-16-12-204.np.local:60030 1311956094776 requests=5, regions=2, usedHeap=157, maxHeap=16358 npin-172-16-12-205.np.local:60030 1311956093804 requests=0, regions=0, usedHeap=134, maxHeap=16358 npin-172-16-12-206.np.local:60030 1311956093809 requests=0, regions=0, usedHeap=134, maxHeap=16358 npin-172-16-12-207.np.local:60030 1311956094799 requests=0, regions=0, usedHeap=134, maxHeap=16358 Total: servers: 5 requests=55, regions=3 I did thread dumps on both the masters and region servers during this time and did not see anything interesting. I'm using 0.90.3-CDH3U1. Anyone have a suggestion on where to look next?
