Thanks Peter. When I looked at the benchmark client machine, it was not under any stress in terms of disk or CPU. But test machines are connected through 10/100 mbps switch port (not gigabit). Can this be a bottleneck?
Thanks Baskar ----- Original Message ----- From: Peter Schuller To: user@cassandra.apache.org Sent: Thursday, April 28, 2011 2:34 AM Subject: Re: Performance tests using stress testing tool > a) I am not seeing cpu usage more than 10pct. Sounds like the benchmarking client is bottlenecking. > In some of the forums, i see > that 8 cpu 32 gb is considered as good sweet spot for cassandra. Is this > true? Seems reasonable in a very general sense, but of course varies with use-case. > Also when would I see real cpu spikes. At this moment looks like 4 > core is more than sufficient. In general, the more requests and columns you read and write, the more you'll be bottlenecking on CPU. The larger individual columns (and thus fewer columns), the more you'll be bound on disk instead. In your case the bottleneck seems to be the benchmark I think. > B) iostat -x is reporting avgqueue size of around 0.25 and await time of > around 30 ms. What would be acceptable queue size and await time? Any avgqueuesize significantly below 1 is generally good. For close to 1 or higher than one, it will depend on your access pattern, latency demands, and the nature of your storage device (e.g., SSD:s, RAID:s can sustain concurrent I/O). To simplify, there is some maximum amount of I/O requests that your storage device will service concurrently. For a normal disk, this is 1 request (I'm ignoring optimizations due to TCQ/NCQ which can be significant sometimes). As long as you're below the saturation point it's mostly about statistics and varying I/O patterns causing latency. The less saturated you are, the better your average latency will be. Once you're *above* saturation latency goes haywire as you don't service as many I/O requests as are coming in. There is a grey area in between where latency will be very sensitive to smallish changes in I/O load but aggregate throughput remaining below what can be sustained. -- / Peter Schuller
