[openstack-dev] Multi-node testing (for redis and others...)

2014-11-17 Thread Joshua Harlow

Hi guys,

A recent question came up about how do we test better with redis for 
tooz. I think this question is also relevant for ceilometer (and other 
users of redis) and in general applies to the whole of openstack as the 
larger system is what people run (I hope not everyone just runs devstack 
on a single-node and that's where they stop, ha).


The basic question is that redis (or zookeeper) have (and typically are) 
ways to be setup with multi-node instances (for example with redis + 
sentinel or zookeeper in multi-node configurations, or the newly 
released redis clustering...). It seems though that our testing 
infrastructure is setup to do the basics of tests (which isn't bad, but 
does have its limits), and this got me thinking on what would be needed 
to actually test these multi-node configurations of things like redis 
(configured in sentinel mode, or redis in clustering mode) in a 
realistic manner that tests 'common' failure patterns (net splits for 
example).


I guess we can split it up into 3 or 4 (or more questions).

1. How do we get a multi-node configuration (of say redis) setup in the 
first place, configured so that all nodes are running and sentinel (for 
example) is running as expected?
2. How do we then inject failures into this setup to ensure that the 
applications and clients built ontop of those systems reliably handle 
these type of injected failures (something like 
https://github.com/aphyr/jepsen or similar?).
3. How do we analyze those results (for when #2 doesn't turn out to work 
as expected) in a meaningful manner, so that we can then turn those 
experiments into more reliable software?


Anyone else have any interesting ideas for this?

-Josh

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Multi-node testing (for redis and others...)

2014-11-17 Thread Joe Gordon
On Mon, Nov 17, 2014 at 1:06 PM, Joshua Harlow harlo...@outlook.com wrote:

 Hi guys,

 A recent question came up about how do we test better with redis for tooz.
 I think this question is also relevant for ceilometer (and other users of
 redis) and in general applies to the whole of openstack as the larger
 system is what people run (I hope not everyone just runs devstack on a
 single-node and that's where they stop, ha).


https://review.openstack.org/#/c/106043/23



 The basic question is that redis (or zookeeper) have (and typically are)
 ways to be setup with multi-node instances (for example with redis +
 sentinel or zookeeper in multi-node configurations, or the newly released
 redis clustering...). It seems though that our testing infrastructure is
 setup to do the basics of tests (which isn't bad, but does have its
 limits), and this got me thinking on what would be needed to actually test
 these multi-node configurations of things like redis (configured in
 sentinel mode, or redis in clustering mode) in a realistic manner that
 tests 'common' failure patterns (net splits for example).

 I guess we can split it up into 3 or 4 (or more questions).

 1. How do we get a multi-node configuration (of say redis) setup in the
 first place, configured so that all nodes are running and sentinel (for
 example) is running as expected?
 2. How do we then inject failures into this setup to ensure that the
 applications and clients built ontop of those systems reliably handle these
 type of injected failures (something like https://github.com/aphyr/jepsen
 or similar?).
 3. How do we analyze those results (for when #2 doesn't turn out to work
 as expected) in a meaningful manner, so that we can then turn those
 experiments into more reliable software?

 Anyone else have any interesting ideas for this?

 -Josh

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev