On 06/08/2017 01:34 PM, Lance Bragstad wrote:
After digging into etcd a bit, one place this might be help deployer
experience would be the handling of fernet keys for token encryption in
keystone. Currently, all keys used to encrypt and decrypt tokens are
kept on disk for each keystone node in the deployment. While simple, it
requires operators to perform rotation on a single node and then push,
or sync, the new key set to the rest of the nodes. This must be done in
lock step in order to prevent early token invalidation and inconsistent
token responses.
An alternative would be to keep the keys in etcd and make the fernet
bits pluggable so that it's possible to read keys from disk or etcd
(pending configuration). The advantage would be that operators could
initiate key rotations from any keystone node in the deployment (or
using etcd directly) and not have to worry about distributing the new
key set. Since etcd associates metadata to the key-value pairs, we might
be able to simplify the rotation strategy as well.
Interesting, I had the mis-conception that "fernet" keys no longer
required any server-side storage (how is "kept-on-disk" now
implemented?) . We've had continuous issues with the pre-fernet
Keystone tokens filling up databases, even when operators were correctly
expunging old tokens; some environments just did so many requests that
the keystone-token table still blew up to where MySQL can no longer
delete from it without producing a too-large transaction for Galera.
So after all the "finally fernet solves this problem" we propose, hey
lets put them *back* in the database :). That's great. But, lets
please not leave "cleaning out old tokens" as some kind of
cron/worry-about-it-later thing. that was a terrible architectural
decision, with apologies to whoever made it. if you're putting some
kind of "we create an infinite, rapidly growing,
turns-to-garbage-in-30-seconds" kind of data in a database, removing
that data robustly and ASAP needs to be part of the process.
On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <[email protected]
<mailto:[email protected]>> wrote:
On 06/08/2017 12:47 AM, Joshua Harlow wrote:
So just out of curiosity, but do people really even know what
etcd is good for? I am thinking that there should be some
guidance from folks in the community as to where etcd should be
used and where it shouldn't (otherwise we just all end up in a
mess).
So far I've seen a proposal of etcd3 as a replacement for memcached
in keystone, and a new dogpile connector was added to oslo.cache to
handle referring to etcd3 as a cache backend. This is a really
simplistic / minimal kind of use case for a key-store.
But, keeping in mind I don't know anything about etcd3 other than
"it's another key-store", it's the only database used by Kubernetes
as a whole, which suggests it's doing a better job than Redis in
terms of "durable". So I wouldn't be surprised if new / existing
openstack applications express some gravitational pull towards using
it as their own datastore as well. I'll be trying to hang onto
the etcd3 track as much as possible so that if/when that happens I
still have a job :).
Perhaps a good idea to actually give examples of how it should
be used, how it shouldn't be used, what it offers, what it
doesn't... Or at least provide links for people to read up on this.
Thoughts?
Davanum Srinivas wrote:
One clarification: Since
https://pypi.python.org/pypi/etcd3gw
<https://pypi.python.org/pypi/etcd3gw> just
uses the HTTP API (/v3alpha) it will work under both
eventlet and
non-eventlet environments.
Thanks,
Dims
On Wed, Jun 7, 2017 at 6:47 AM, Davanum
Srinivas<[email protected] <mailto:[email protected]>> wrote:
Team,
Here's the update to the base services resolution from
the TC:
https://governance.openstack.org/tc/reference/base-services.html
<https://governance.openstack.org/tc/reference/base-services.html>
First request is to Distros, Packagers, Deployers,
anyone who
installs/configures OpenStack:
Please make sure you have latest etcd 3.x available in your
environment for Services to use, Fedora already does, we
need help in
making sure all distros and architectures are covered.
Any project who want to use etcd v3 API via grpc, please
use:
https://pypi.python.org/pypi/etcd3
<https://pypi.python.org/pypi/etcd3> (works only for
non-eventlet services)
Those that depend on eventlet, please use the etcd3
v3alpha HTTP API using:
https://pypi.python.org/pypi/etcd3gw
<https://pypi.python.org/pypi/etcd3gw>
If you use tooz, there are 2 driver choices for you:
https://github.com/openstack/tooz/blob/master/setup.cfg#L29
<https://github.com/openstack/tooz/blob/master/setup.cfg#L29>
https://github.com/openstack/tooz/blob/master/setup.cfg#L30
<https://github.com/openstack/tooz/blob/master/setup.cfg#L30>
If you use oslo.cache, there is a driver for you:
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
<https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33>
Devstack installs etcd3 by default and points cinder to it:
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
<http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3>
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
<http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356>
Review in progress for keystone to use etcd3 for caching:
https://review.openstack.org/#/c/469621/
<https://review.openstack.org/#/c/469621/>
Doug is working on proposal(s) for oslo.config to store some
configuration in etcd3:
https://review.openstack.org/#/c/454897/
<https://review.openstack.org/#/c/454897/>
So, feel free to turn on / test with etcd3 and report
issues.
Thanks,
Dims
--
Davanum Srinivas :: https://twitter.com/dims
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
<http://[email protected]?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
<http://[email protected]?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
