Glad to see you weighed in on this. -d On Sat, Aug 1, 2015 at 3:50 PM, Matt Fischer <m...@mattfischer.com> wrote:
> Agree that you guys are way over thinking this. You don't need to rotate > keys at exactly the same time, we do it in within a one or two hours > typically based on how our regions are setup. We do it with puppet, puppet > runs on one keystone node at a time and drops the keys into place. The > actual rotation and generation we handle with a script that then proposes > the new key structure as a review which is then approved and deployed via > the normal process. For this process I always drop keys 0, 1, 2 into place, > I'm not bumping the numbers like the normal rotations do. > > We had also considered ansible which would be perfect for this, but that > makes our ability to setup throw away environments with a single click a > bit more complicated. If you don't have that requirement, a simple ansible > script is what you should do. > > > On Sat, Aug 1, 2015 at 3:41 PM, Clint Byrum <cl...@fewbar.com> wrote: > >> Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700: >> > On Saturday 01 August 2015 16:27:17 bdobre...@mirantis.com wrote: >> > > I suggest to use pacemaker multistate clone resource to rotate and >> > rsync >> > > fernet tokens from local directories across cluster nodes. The >> resource >> > > prototype is described here >> > > https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros: >> > Pacemaker >> > > will care about CAP/split-brain stuff for us, we just design rotate >> and >> > > rsync logic. Also no shared FS/DB involved but only Corosync CIB - to >> > store >> > > few internal resource state related params, not tokens. Cons: Keystone >> > > nodes hosting fernet tokens directories must be members of pacemaker >> > > cluster. Also custom OCF script should be created to implement this. >> __ >> > > Regards, >> > > Bogdan Dobrelya. >> > > IRC: bogdando >> > >> > Looks complex. >> > >> > I suggest this kind of bash or python script, running on Fuel master >> node: >> > >> > 0. Check that all controllers are online; >> > 1. Go to one of the controllers, rotate keys there; >> > 2. Fetch key 0 from there; >> > 3. For each other controller rotate keys there and put the 0-key >> instead of >> > their new 0-key. >> > 4. If any of the nodes fail to get new keys (because they went offline >> or for >> > some other reason) revert the rotate (move the key with the biggest >> index >> > back to 0). >> > >> > The script can be launched by cron or by button in Fuel. >> > >> > I don't see anything critically bad if one rotation/sync event fails. >> > >> >> This too is overly complex and will cause failures. If you replace key 0, >> you will stop validating tokens that were encrypted with the old key 0. >> >> You simply need to run rotate on one, and then rsync that key repository >> to all of the others. You _must not_ run rotate again until you rsync to >> all of the others, since the key 0 from one rotation becomes the primary >> token encrypting key going forward, so you need it to get pushed out to >> all nodes as 0 first. >> >> Don't over think it. Just read http://lbragstad.com/?p=133 and it will >> remain simple. >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev