@JG and @stack Helpful!
runing RS with 2GB is because we have a heterogeneous node(the slave-5), which has only 4GB RAM. Now, I temporarily removed this node from the cluster. Then we got the ~2ms random-read now. It is fine now. Thank you very much. On Wed, Aug 19, 2009 at 2:52 AM, Jonathan Gray <[email protected]> wrote: > As stack says, but more strongly, if you have 4+ cores then you definitely > want to turn off incremental mode. Is there a reason you're running your RS > with 2GB given that you have 8GB of total memory? I'd up it to 4GB, after I > did that on our production cluster things ran much more smoothly with CMS. > > I'd also drop your swappiness to 0, I've not heard a good argument for when > we ever want to swap on an HBase/Hadoop cluster. If you end up swapping, > you're going to start seeing some weird behavior and very slow GC runs, and > likely killing off regionservers as ZK times out and assumes the RS is dead. > > > > stack wrote: > >> "-XX:+CMSIncrementalMode" is our default but its for nodes with 2 or less >> CPUs according to >> http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html. You >> might try without this. >> >> >> But I am surprising that the node(5) which has 8CPU cores and 4GB RAM, 6 >> >>> SATA-RAID1, has problem. >>> >>> avg-cpu: %user %nice %system %iowait %steal %idle >>> 7.46 0.00 3.28 23.11 0.00 66.15 >>> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz >>> avgqu-sz await svctm %util >>> sda 84.83 25.12 485.57 2.49 53649.75 220.90 110.38 >>> 9.20 18.85 2.04 99.53 >>> dm-0 0.00 0.00 0.00 25.12 0.00 201.00 8.00 >>> 0.01 0.27 0.01 0.02 >>> dm-1 0.00 0.00 570.90 2.49 53655.72 19.90 93.61 >>> 10.74 18.72 1.74 99.53 >>> >>> It seems the disk I/O is very busy. >>> >>> >> Yeah. Whats writing? Can you tell? Is it NN or ZK node? >> >> St.Ack >> >>
