RE: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

The other problem is: if I keep mixed write and read (e.g, 8 write threads plus 7 read threads) against the 2-nodes cluster continuously, the read latency will go up gradually (along with the size of Cassandra data file), and at the end it will become ~40ms (up from ~20ms) even with only

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

On Tue, Feb 16, 2010 at 2:32 AM, Dr. Martin Grabmüller martin.grabmuel...@eleven.de wrote: In my tests I have observed that good read latency depends on keeping the number of data files low. In my current test setup, I have stored 1.9 TB of data on a single node, which is in 21 data files,

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

Dumped 50mil records into my 2-node cluster overnight, made sure that there's not many data files (around 30 only) per Martin's suggestion. The size of the data directory is 63GB. Now when I read records from the cluster the read latency is still ~44ms, --there's no write happening during the

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

One more thoughts about Martin's suggestion: is it possible to put the data files into multiple directories that are located in different physical disks? This should help to improve the i/o bottleneck issue. Has anybody tested the row-caching feature in trunk (shoot for 0.6?)? -Weijun On Tue,

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

On Tue, Feb 16, 2010 at 11:50 AM, Weijun Li weiju...@gmail.com wrote: Dumped 50mil records into my 2-node cluster overnight, made sure that there's not many data files (around 30 only) per Martin's suggestion. The size of the data directory is 63GB. Now when I read records from the cluster

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

On Tue, Feb 16, 2010 at 11:56 AM, Weijun Li weiju...@gmail.com wrote: One more thoughts about Martin's suggestion: is it possible to put the data files into multiple directories that are located in different physical disks? This should help to improve the i/o bottleneck issue. Yes, you can

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

Thanks for for DataFileDirectory trick and I'll give a try. Just noticed the impact of number of data files: node A has 13 data files with read latency of 20ms and node B has 27 files with read latency of 60ms. After I ran nodeprobe compact on node B its read latency went up to 150ms. The read

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

On Tue, Feb 16, 2010 at 12:16 PM, Weijun Li weiju...@gmail.com wrote: Thanks for for DataFileDirectory trick and I'll give a try. Just noticed the impact of number of data files: node A has 13 data files with read latency of 20ms and node B has 27 files with read latency of 60ms. After I ran

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

throughput but high read latency ( 100ms)? Thanks for for DataFileDirectory trick and I'll give a try. Just noticed the impact of number of data files: node A has 13 data files with read latency of 20ms and node B has 27 files with read latency of 60ms. After I ran nodeprobe compact on node B its

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

To: cassandra-user@incubator.apache.org Subject: Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)? Thanks for for DataFileDirectory trick and I'll give a try. Just noticed the impact of number of data files: node A has 13 data files with read latency of 20ms and node

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

the full row content. -Original Message- From: Weijun Li weiju...@gmail.com Sent: Tuesday, February 16, 2010 12:16pm To: cassandra-user@incubator.apache.org Subject: Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)? Thanks for for DataFileDirectory trick

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

Message- From: Weijun Li weiju...@gmail.com Sent: Tuesday, February 16, 2010 12:16pm To: cassandra-user@incubator.apache.org Subject: Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)? Thanks for for DataFileDirectory trick and I'll give a try. Just

RE: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

-Original Message- From: Jonathan Ellis [mailto:jbel...@gmail.com] Sent: Sunday, February 14, 2010 6:22 PM To: cassandra-user@incubator.apache.org Subject: Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)? are you i/o bound? what is your on-disk data set size? what

Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

Hello, I saw some Cassandra benchmark reports mentioning read latency that is less than 50ms or even 30ms. But my benchmark with 0.5 doesn’t seem to support that. Here’s my settings: Nodes: 2 machines. 2x2.5GHZ Xeon Quad Core (thus 8 cores), 8GB RAM ReplicationFactor=2

Re: Cassandra benchmark shows OK throughput but high read latency ( 100ms)?

are you i/o bound? what is your on-disk data set size? what does iostats tell you? http://spyced.blogspot.com/2010/01/linux-performance-basics.html do you have a lot of pending compactions? (tpstats will tell you) have you increased KeysCachedFraction? On Sun, Feb 14, 2010 at 8:18 PM, Weijun