On 11/30/2011 01:02 AM, Jay Kreps wrote: > I think this is one of the core problems of this type of system--how to do > good, realistic distributed testing. I have not seen anyone really succeed > except via the "throw production load at it and see" approach so I would be > interested to hear people's thoughts.
Cassandra has the same problem and has gone through several iterations of distributed test systems. The most recent is here: https://github.com/riptano/cassandra-dtest Time will tell if this attempt is more successful. The obvious prerequisite for something like this is a set of scripts that make it easy to set up Kafka clusters of different topologies for testing. Another thing they do well is having a solid stress/perf tool distributed with releases, everyone agrees it gives decent results, easy to hack if you want to test something slightly different and thus it's widely used.
