Cassandra needs distributed regression testing. The current unit tests and so-called 'system' tests are great, but limited in scope for a distributed system. As pointed by others very recently, this can only help towards the goal of having bullet-proof releases (this is obviously not enough, but it's needed all the same).
We have an attempt at distributed tests in-tree but it never caught up. Amongst the reason, is probably the fact that they are using whirr and thus require an EC2 or Rackspace account. To try to solve that problem, Datastax has developed a small distributed test framework called cassandra-dtest (dtest for short). It is written in python and uses nosetests and ccm (https://github.com/pcmanus/ccm). It is open-source and available at https://github.com/riptano/cassandra-dtest. The number of tests is yet limited but already far exceed those of the in-tree distributed tests. It include in particular multi-DC tests and upgrade tests. This is very much young code and all inputs will be greatly appreciated. The clear goal however is that unless someone has anything better to propose, we use this as the default distributed test framework for Cassandra. Which means getting into the habit of requiring a regression test (unit or distributed) with each bug fix. -- Sylvain
