I setup a two cassandra clusters with 2 nodes each. Both use random partitioner. It's strange that for each cluster, one node has much shortter read latency than the other one
This is the info of one of the cluster: Node A: read count 77302, data file 41GB, read latency 58180, io saturation 100% Node B: read count 488753, data file 26GB, read latency 5822 , io saturation 35%. I first started node A, then ran B to join the cluster. Both machines have exactly the same hardware and OS. The test client randomly pick a node to write and it worked fine for the other cluster. Address Status Load Range Ring 169400792707028208569145873749456918214 10.xxxxxxx Up 38.39 GB 103633195217832666843316719920043079797 |<--| 10.xxxxxxx Up 24.22 GB 169400792707028208569145873749456918214 |-->| For both clusters, whichever node that took more reads (with larger data file) owns the much worse read latency. What's the algorithm that cassandra use to split token when a new node is joining? What could cause this unbalanced read latency issue? How can I fix this? How to make sure all nodes get evenly distributed data and traffic? -Weijun
