The cpu load on each of the nodes never goes above 1 and very little if any swap is in use.
The 4MB is our boundary case - the actual data will always be smaller. This was just a encapsulated test that reproduces our issue. I will re-test with the CMS collector and logging enabled and reply back. Thanks, elsif > Most of these exceptions look related to overloaded servers (GC pauses > causing timeouts, high IO wait tripping up the datanodes, etc). Have > you turned on GC logging? Also, are you swapping on these nodes? > > Check out the performance tuning page here: > > http://wiki.apache.org/hadoop/PerformanceTuning > > The WrongRegionException at the end could be a fault but it's hard to > know without seeing the entire context and knowing what the cluster > was up to at that point. > > Performance can degrade as the JVMs fill up, get more and more > fragmented, and the GC gets slower. > > Also, you are inserting 4MB values? Those are fairly large, at the > upper-end of what you would want to put into HBase. Is this your > actual use case? At the least you'll want to increase your region > size (otherwise you're going to have at most 64 rows per region, often > less), but also consider if HBase is the right place to store 4MB values. > > Hope that helps. > > JG >
