As stack says, but more strongly, if you have 4+ cores then you
definitely want to turn off incremental mode. Is there a reason you're
running your RS with 2GB given that you have 8GB of total memory? I'd
up it to 4GB, after I did that on our production cluster things ran much
more smoothly with CMS.
I'd also drop your swappiness to 0, I've not heard a good argument for
when we ever want to swap on an HBase/Hadoop cluster. If you end up
swapping, you're going to start seeing some weird behavior and very slow
GC runs, and likely killing off regionservers as ZK times out and
assumes the RS is dead.
stack wrote:
"-XX:+CMSIncrementalMode" is our default but its for nodes with 2 or less
CPUs according to
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html. You
might try without this.
But I am surprising that the node(5) which has 8CPU cores and 4GB RAM, 6
SATA-RAID1, has problem.
avg-cpu: %user %nice %system %iowait %steal %idle
7.46 0.00 3.28 23.11 0.00 66.15
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz
avgqu-sz await svctm %util
sda 84.83 25.12 485.57 2.49 53649.75 220.90 110.38
9.20 18.85 2.04 99.53
dm-0 0.00 0.00 0.00 25.12 0.00 201.00 8.00
0.01 0.27 0.01 0.02
dm-1 0.00 0.00 570.90 2.49 53655.72 19.90 93.61
10.74 18.72 1.74 99.53
It seems the disk I/O is very busy.
Yeah. Whats writing? Can you tell? Is it NN or ZK node?
St.Ack
