I have a DC of 4 nodes that must be expanded to accommodate an expected growth in data. Since the DC is not using vnodes, we have decided to set up a new DC with vnodes enabled, start using the new DC, and decommission the old DC.
Both DCs have 4 nodes. The idea is to add additional nodes to the new DC later on. The servers in both DCs are very similar: quad-core machines with 8GB. We have bootstrapped/rebuilt the nodes in the new DC. When that finished, the nodes in the new DC were showing little CPU activity, as you would expect, because they are receiving writes from the other DC. So far, so good. Then we switched the clients from the old DC to the new DC. The CPU load on all nodes in the new DC immediately rose to excessively high levels (15 - 25), which made the servers effectively unavailable. The load did not drop structurally within 20 minutes, so we had to switch the clients back to the old DC. Then the load dropped again. What can be the reason for the high CPU loads on the new nodes? Performance test shows that the servers in the new DC perform slightly better (both IO and CPU) than the servers in the old DC. I did not see anything abnormal in the Cassandra logs, like garbage collection warnings. I also did not see any strange things in the tpstats. The only difference I'm aware of between the old and new DC is the use of vnodes. Any help is appreciated! Thanks, Tom
