Hi,
I must apologise as I believe I was the culprit on at least one of these
problems.
Do we have this process documented somewhere other than email
specifically for puppet changes?
Kind regards,
Josh
Rackspace Australia
On 7/3/14 4:03 AM, James E. Blair wrote:
Hi,
We've got a lot of people interested in working on puppet, which is
great. I'm hoping that as a result, we'll have a much cleaner and more
re-usable system. But at the moment, the primary purpose of what's in
the openstack-infra/config repo is to run the systems for the OpenStack
project. We need to make these changes with a minimum amount of
disruption.
Our current testing for puppet changes is sub-standard. We're even
unable to fully verify that puppet syntax is correct before actually
running it. That means that many simple puppet changes are potentially
breaking for running OpenStack servers. Our experience has backed this
up. I've had to do two emergency reverts of puppet changes this
morning.
So, until such a time as we've refactored the puppet repo enough to have
some reasonable separation and safety, as well as a higher standard of
testing, I'd like to remind everyone of the policy for approving puppet
changes[1] (after they have had sufficient review):
* Someone, generally the person authoring the change, should be
prepared to monitor systems (using puppetboard) for any anomalous
behavior. If there's a problem, they should take immediate
corrective action, either fixing the bug or proposing a revert and
collecting immediate approvals.
* If you are the author, you should self-approve the change and
monitor the results yourself.
* If you are not the author but are nonetheless prepared to monitor
the results in their stead, then you can approve it.
* If you are not the author but can not commit to monitoring the
change, then you should not approve the change. Instead, if the
requisite +2 code review votes have been left, you should leave a
comment inviting the author to ping a member of infra-core in
#openstack-infra when they are prepared to monitor the results of
the change.
Generally, people submitting puppet changes are genuinely interested in
making sure they work and don't break systems. We're a pretty small
team, and this lets everyone help out to the best of their abilities.
Hopefully we'll get to a better place soon and we can blindly approve
changes with confidence that if they pass tests they will work. But
we're not there yet.
[1] I don't mean all config repo changes, just changes to the config
repo that could break puppet -- generally changes that are modifying
code in one of puppet's languages.
-Jim
_______________________________________________
OpenStack-Infra mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
_______________________________________________
OpenStack-Infra mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
