> On 21 Nov 2014, at 17:15, Aleksandr Didenko <adide...@mirantis.com> wrote:
> Hi,
> following our docs/workflow plus writing rspec tests for every new option we 
> add/modify in our manifests could help with regressions. For example:
>       • we add new keystone config option in openstack::keystone class - 
> keystone_config {'cache/backend': value => 'keystone.cache.memcache_pool';}
>       • we create new test for openstack::keystone class, something like this:
>               • should 
> contain_keystone_config("cache/backend").with_value('keystone.cache.memcache_pool')
> So with such test, if for some reason we lose 
> keystone_config("cache/backend") option, 'rake spec' would alert us about it 
> right away and we'll get "-1" from CI. Of course we should also implement 
> 'rake spec' CI gate for this.
> But from the other hand, if someone changes option in manifests and updates 
> rspec tests accordingly, then such commit will pass 'rake spec' test and we 
> can still lose some specific option.
> > We should speed up development of some modular testing framework that will 
> > check that corresponding change affects only particular pieces.
> Such test would not catch this particular regressions with "keystone_config 
> {'cache/backend': value => 'keystone.cache.memcache_pool';}", because even 
> with regression (i.e. dogpile backend) keystone was working OK. It has passed 
> several BVTs and custom system tests, because 'dogpile' cache backend was 
> working just fine while all memcached servers are up and running. So it looks 
> like we need some kind of tests that will ensure that particular config 
> options (or particular puppet resources) have some particular values (like 
> "backend = keystone.cache.memcache_pool" in [cache] block of keystone.conf).
> So I would go with rspec testing for specific resources but I would write 
> them in 'openstack' module. Those tests should check that needed 
> (nova/cinder/keystone/glance)_config resources have needed values in the 
> puppet catalog. Since we're not going to sync 'openstack' module with the 
> upstream, such tests will remain intact until we change them, and they won't 
> be affected by other modules sync/merge (keystone, cinder, nova, etc).

I totally agree, but we need to remember to introduce tests in separate 
commits, otherwise loosing commit ID we would also lose tests ;)

Also I’m just wondering how do we keep upstream modules in our repo? They are 
not submodules, so how is it organized?

Tomasz 'Zen' Napierala
Sr. OpenStack Engineer

OpenStack-dev mailing list

Reply via email to