On 06/08/2017 01:34 PM, Lance Bragstad wrote:
After digging into etcd a bit, one place this might be help deployer experience would be the handling of fernet keys for token encryption in keystone. Currently, all keys used to encrypt and decrypt tokens are kept on disk for each keystone node in the deployment. While simple, it requires operators to perform rotation on a single node and then push, or sync, the new key set to the rest of the nodes. This must be done in lock step in order to prevent early token invalidation and inconsistent token responses.

An alternative would be to keep the keys in etcd and make the fernet bits pluggable so that it's possible to read keys from disk or etcd (pending configuration). The advantage would be that operators could initiate key rotations from any keystone node in the deployment (or using etcd directly) and not have to worry about distributing the new key set. Since etcd associates metadata to the key-value pairs, we might be able to simplify the rotation strategy as well.

Interesting, I had the mis-conception that "fernet" keys no longer required any server-side storage (how is "kept-on-disk" now implemented?) . We've had continuous issues with the pre-fernet Keystone tokens filling up databases, even when operators were correctly expunging old tokens; some environments just did so many requests that the keystone-token table still blew up to where MySQL can no longer delete from it without producing a too-large transaction for Galera.

So after all the "finally fernet solves this problem" we propose, hey lets put them *back* in the database :). That's great. But, lets please not leave "cleaning out old tokens" as some kind of cron/worry-about-it-later thing. that was a terrible architectural decision, with apologies to whoever made it. if you're putting some kind of "we create an infinite, rapidly growing, turns-to-garbage-in-30-seconds" kind of data in a database, removing that data robustly and ASAP needs to be part of the process.






On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <mba...@redhat.com <mailto:mba...@redhat.com>> wrote:



    On 06/08/2017 12:47 AM, Joshua Harlow wrote:

        So just out of curiosity, but do people really even know what
        etcd is good for? I am thinking that there should be some
        guidance from folks in the community as to where etcd should be
        used and where it shouldn't (otherwise we just all end up in a
        mess).


    So far I've seen a proposal of etcd3 as a replacement for memcached
    in keystone, and a new dogpile connector was added to oslo.cache to
    handle referring to etcd3 as a cache backend.  This is a really
    simplistic / minimal kind of use case for a key-store.

    But, keeping in mind I don't know anything about etcd3 other than
    "it's another key-store", it's the only database used by Kubernetes
    as a whole, which suggests it's doing a better job than Redis in
    terms of "durable".   So I wouldn't be surprised if new / existing
    openstack applications express some gravitational pull towards using
    it as their own datastore as well.    I'll be trying to hang onto
    the etcd3 track as much as possible so that if/when that happens I
    still have a job :).





        Perhaps a good idea to actually give examples of how it should
        be used, how it shouldn't be used, what it offers, what it
        doesn't... Or at least provide links for people to read up on this.

        Thoughts?

        Davanum Srinivas wrote:

            One clarification: Since
            https://pypi.python.org/pypi/etcd3gw
            <https://pypi.python.org/pypi/etcd3gw> just
            uses the HTTP API (/v3alpha) it will work under both
            eventlet and
            non-eventlet environments.

            Thanks,
            Dims


            On Wed, Jun 7, 2017 at 6:47 AM, Davanum
            Srinivas<dava...@gmail.com <mailto:dava...@gmail.com>>  wrote:

                Team,

                Here's the update to the base services resolution from
                the TC:
                https://governance.openstack.org/tc/reference/base-services.html
                
<https://governance.openstack.org/tc/reference/base-services.html>

                First request is to Distros, Packagers, Deployers,
                anyone who
                installs/configures OpenStack:
                Please make sure you have latest etcd 3.x available in your
                environment for Services to use, Fedora already does, we
                need help in
                making sure all distros and architectures are covered.

                Any project who want to use etcd v3 API via grpc, please
                use:
                https://pypi.python.org/pypi/etcd3
                <https://pypi.python.org/pypi/etcd3> (works only for
                non-eventlet services)

                Those that depend on eventlet, please use the etcd3
                v3alpha HTTP API using:
                https://pypi.python.org/pypi/etcd3gw
                <https://pypi.python.org/pypi/etcd3gw>

                If you use tooz, there are 2 driver choices for you:
                https://github.com/openstack/tooz/blob/master/setup.cfg#L29
                <https://github.com/openstack/tooz/blob/master/setup.cfg#L29>
                https://github.com/openstack/tooz/blob/master/setup.cfg#L30
                <https://github.com/openstack/tooz/blob/master/setup.cfg#L30>

                If you use oslo.cache, there is a driver for you:
                
https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
                
<https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33>

                Devstack installs etcd3 by default and points cinder to it:
                
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
                
<http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3>
                
http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
                
<http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356>


                Review in progress for keystone to use etcd3 for caching:
                https://review.openstack.org/#/c/469621/
                <https://review.openstack.org/#/c/469621/>

                Doug is working on proposal(s) for oslo.config to store some
                configuration in etcd3:
                https://review.openstack.org/#/c/454897/
                <https://review.openstack.org/#/c/454897/>

                So, feel free to turn on / test with etcd3 and report
                issues.

                Thanks,
                Dims

-- Davanum Srinivas :: https://twitter.com/dims





        
__________________________________________________________________________
        OpenStack Development Mailing List (not for usage questions)
        Unsubscribe:
        openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
        <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
        http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>


    __________________________________________________________________________
    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
    <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe>
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
    <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Reply via email to