Unbalanced read latency among nodes in a cluster

2010-02-19 Thread Weijun Li
I setup a two cassandra clusters with 2 nodes each. Both use random
partitioner. It's strange that for each cluster, one node has much shortter
read latency than the other one

This is the info of one of the cluster:

Node A: read count 77302, data file 41GB, read latency 58180, io saturation
100%
Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
35%.

I first started node A, then ran B to join the cluster. Both machines have
exactly the same hardware and OS. The test client randomly pick a node to
write and it worked fine for the other cluster.

Address   Status Load
Range  Ring

169400792707028208569145873749456918214
10.xxx Up 38.39 GB
103633195217832666843316719920043079797|--|
10.xxx Up 24.22 GB
169400792707028208569145873749456918214|--|

For both clusters, whichever node that took more reads (with larger data
file) owns the much worse read latency.

What's the algorithm that cassandra use to split token when a new node is
joining? What could cause this unbalanced read latency issue? How can I fix
this? How to make sure all nodes get evenly distributed data and traffic?

-Weijun


Re: Unbalanced read latency among nodes in a cluster

2010-02-19 Thread Jonathan Ellis
http://wiki.apache.org/cassandra/Operations

On Fri, Feb 19, 2010 at 3:03 PM, Weijun Li weiju...@gmail.com wrote:
 I setup a two cassandra clusters with 2 nodes each. Both use random
 partitioner. It's strange that for each cluster, one node has much shortter
 read latency than the other one

 This is the info of one of the cluster:

 Node A: read count 77302, data file 41GB, read latency 58180, io saturation
 100%
 Node B: read count 488753, data file 26GB, read latency 5822 , io saturation
 35%.

 I first started node A, then ran B to join the cluster. Both machines have
 exactly the same hardware and OS. The test client randomly pick a node to
 write and it worked fine for the other cluster.

 Address   Status Load
 Range  Ring

 169400792707028208569145873749456918214
 10.xxx Up 38.39 GB
 103633195217832666843316719920043079797    |--|
 10.xxx Up 24.22 GB
 169400792707028208569145873749456918214    |--|

 For both clusters, whichever node that took more reads (with larger data
 file) owns the much worse read latency.

 What's the algorithm that cassandra use to split token when a new node is
 joining? What could cause this unbalanced read latency issue? How can I fix
 this? How to make sure all nodes get evenly distributed data and traffic?

 -Weijun