Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

2017-03-06 Thread Jeffrey Zhang 
On Tue, Mar 7, 2017 at 2:09 AM, Matt Fischer  wrote:

> I don't think it would cause an issue if every controller rotated all at
> once. The issues are more along the lines of rotating to key C when there
> are tokens out there that are encrypted with keys A and B. In other words
> over-rotation. As long as your keys are properly staged, do the rotation
> all at once or space them out, should not make any difference.
>

​The issue is "at once".
It takes some time to rotate and distribute the keys. There is one case
that.
controller A and controller B generate a new different keys. Then they copy
the ​key to other by using rsync.

A: 0 1 2 3
B: 0' 1' 2 3

When distributing, the 0/0' and 1/1' may be overrode(rsync hold the delete
file handler and copy it to other one). it will lead to

A: 0' 1' 2 3
B: 0 1 2 3

next rotation, it may become

A: 0' 1' 2' 3
B: 0 1 2 3

after distribute , it become

A: 0 1 2 3
B: 0' 1' 2' 3

Next rotation and distribute, issue happen.

This is a small probability， but it still possible.


-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

2017-03-06 Thread Jeffrey Zhang 
On Mon, Mar 6, 2017 at 6:05 PM, Paul Bourke  wrote:

> Two initial ideas:
>
> We could create a specific ansible task to rotate the keys, and document
> that operator should set up a cron job on the deployment node to run this
> periodically.
>
> We could also look at making use of VRRP (keepalived). Potentially the
> cron job could run on every controller, but only take action if it
> identifies it's the one with the VIP.
>
> The second seems preferable to me as it requires no additional effort on
> the part of the operator. Maybe there's problems with this though that I'm
> not thinking of.
>
> -Paul
>

​Thanks Paul, ​

​second seems better. We can implement a file lock to ensure only one
rotate and distribute process is running at the same time.




-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

2017-03-06 Thread Matt Fischer 
I don't think it would cause an issue if every controller rotated all at
once. The issues are more along the lines of rotating to key C when there
are tokens out there that are encrypted with keys A and B. In other words
over-rotation. As long as your keys are properly staged, do the rotation
all at once or space them out, should not make any difference.


On Sun, Mar 5, 2017 at 10:52 PM, Jeffrey Zhang 
wrote:

> fix subject typo
>
> On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang 
> wrote:
>
>> Kolla have support keystone fernet keys. But there are still some
>> topics worth to talk.
>>
>> The key issue is key distribution. Kolla's solution is like
>>
>> * there is a task run frequently by cronjob to check whether
>>   the key should be rotate. This is controlled by
>>   `fernet_token_expiry` variable
>> * When key rotate is required, the task in cron job will generate a
>>   new key by using `keystone-manage fernet-rotate` and distribute all
>>   keys in /etc/keystone/fernet-keys folder to other by using
>>   `rsync --delete`
>>
>> one issue is: there is no global lock in rotate and distribute steps.
>> above command is ran on all controllers. it may cause issues if
>> all controllers run this at the same time.
>>
>> Since we are using Ansible as deployment tools. there is not daemon
>> agent at all to keep rotate and distribution atomic. Is there any
>> easier way to implement a global lock?
>>
>> possible solution:
>> 1. configure cron job with different time on each controller
>> 2. implement a global lock? ( no idea how )
>>
>> [0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html
>>
>> --
>> Regards,
>> Jeffrey Zhang
>> Blog: http://xcodest.me
>>
>
>
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

2017-03-06 Thread Paul Bourke 

Two initial ideas:

We could create a specific ansible task to rotate the keys, and document 
that operator should set up a cron job on the deployment node to run 
this periodically.


We could also look at making use of VRRP (keepalived). Potentially the 
cron job could run on every controller, but only take action if it 
identifies it's the one with the VIP.


The second seems preferable to me as it requires no additional effort on 
the part of the operator. Maybe there's problems with this though that 
I'm not thinking of.


-Paul

On 06/03/17 05:52, Jeffrey Zhang wrote:

fix subject typo

On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang mailto:zhang.lei@gmail.com>> wrote:

Kolla have support keystone fernet keys. But there are still some
topics worth to talk.

The key issue is key distribution. Kolla's solution is like

* there is a task run frequently by cronjob to check whether
  the key should be rotate. This is controlled by
  `fernet_token_expiry` variable
* When key rotate is required, the task in cron job will generate a
  new key by using `keystone-manage fernet-rotate` and distribute all
  keys in /etc/keystone/fernet-keys folder to other by using
  `rsync --delete`

one issue is: there is no global lock in rotate and distribute steps.
above command is ran on all controllers. it may cause issues if
all controllers run this at the same time.

Since we are using Ansible as deployment tools. there is not daemon
agent at all to keep rotate and distribution atomic. Is there any
easier way to implement a global lock?

possible solution:
1. configure cron job with different time on each controller
2. implement a global lock? ( no idea how )

[0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html


--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me 




--
Regards,
Jeffrey Zhang
Blog: http://xcodest.me 


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla][keystone] better way to rotate and distribution keystone fernet keys in container env

2017-03-05 Thread Jeffrey Zhang 
fix subject typo

On Mon, Mar 6, 2017 at 12:28 PM, Jeffrey Zhang 
wrote:

> Kolla have support keystone fernet keys. But there are still some
> topics worth to talk.
>
> The key issue is key distribution. Kolla's solution is like
>
> * there is a task run frequently by cronjob to check whether
>   the key should be rotate. This is controlled by
>   `fernet_token_expiry` variable
> * When key rotate is required, the task in cron job will generate a
>   new key by using `keystone-manage fernet-rotate` and distribute all
>   keys in /etc/keystone/fernet-keys folder to other by using
>   `rsync --delete`
>
> one issue is: there is no global lock in rotate and distribute steps.
> above command is ran on all controllers. it may cause issues if
> all controllers run this at the same time.
>
> Since we are using Ansible as deployment tools. there is not daemon
> agent at all to keep rotate and distribution atomic. Is there any
> easier way to implement a global lock?
>
> possible solution:
> 1. configure cron job with different time on each controller
> 2. implement a global lock? ( no idea how )
>
> [0] https://docs.openstack.org/admin-guide/identity-fernet-token-faq.html
>
> --
> Regards,
> Jeffrey Zhang
> Blog: http://xcodest.me
>



-- 
Regards,
Jeffrey Zhang
Blog: http://xcodest.me
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev