Re: [openstack-dev] [Keystone] Memcache user token index and PKI tokens
On 16/07/2013, at 1:10 AM, Adam Young ayo...@redhat.com wrote: On 07/15/2013 04:06 AM, Kieran Spear wrote: Hi all, I want to backport the fix for the Token List in Memcache can consume an entire memcache page bug[1] to Grizzly, but I had a couple of questions: 1. Why do we need to store the entire token data in the usertoken-userid key? This data always seems to be hashed before indexing into the 'token-tokenid' keys anyway. The size of the memcache data for a user's token list currently grows by 4k every time a new PKI token is created. It doesn't take long to hit 1MB at this rate even with the above fix. Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit. In this case it looks like we're taking a hit from both, since the PKI token id from the user token index is retrieved, then hashed and then that key is used to retrieve the token from the tokens-%s page anyway. 2. Every time it creates a new token, Keystone loads each token from the user's token list with a separate memcache call so it can throw it away if it's expired. This seems excessive. Is it anything to worry about? If it just checked the first two tokens you'd get the same effect on a longer time scale. I guess part of the answer is to decrease our token expiry time, which should mitigate both issues. Failing that we'd consider moving to the SQL backend. HOw about doing both? But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well. No silver bullet, I am afraid. I think we're going to stick with memcache for now (the devil we know :)). With (1) and (2) fixed and the token expiration time tweaked I think memcache will do okay. Kieran Cheers, Kieran [1] https://bugs.launchpad.net/keystone/+bug/1171985 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Memcache user token index and PKI tokens
On Tue, Jul 16, 2013 at 4:01 AM, Kieran Spear kisp...@gmail.com wrote: On 16/07/2013, at 1:10 AM, Adam Young ayo...@redhat.com wrote: On 07/15/2013 04:06 AM, Kieran Spear wrote: Hi all, I want to backport the fix for the Token List in Memcache can consume an entire memcache page bug[1] to Grizzly, but I had a couple of questions: 1. Why do we need to store the entire token data in the usertoken-userid key? This data always seems to be hashed before indexing into the 'token-tokenid' keys anyway. The size of the memcache data for a user's token list currently grows by 4k every time a new PKI token is created. It doesn't take long to hit 1MB at this rate even with the above fix. Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit. In this case it looks like we're taking a hit from both, since the PKI token id from the user token index is retrieved, then hashed and then that key is used to retrieve the token from the tokens-%s page anyway. 2. Every time it creates a new token, Keystone loads each token from the user's token list with a separate memcache call so it can throw it away if it's expired. This seems excessive. Is it anything to worry about? If it just checked the first two tokens you'd get the same effect on a longer time scale. I guess part of the answer is to decrease our token expiry time, which should mitigate both issues. Failing that we'd consider moving to the SQL backend. HOw about doing both? But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well. No silver bullet, I am afraid. I think we're going to stick with memcache for now (the devil we know :)). With (1) and (2) fixed and the token expiration time tweaked I think memcache will do okay. Kieran Cheers, Kieran [1] https://bugs.launchpad.net/keystone/+bug/1171985 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Hi Kieran, I've looked into the potential bug you described and it appears that there has been a change in the master branch to support the idea of pluggable token providers (much better implementation than the driver being responsible for the token itself). This change modified how the memcache driver stored the IDs, and performed the CMS hashing function when the manager returned the token_id to the driver, instead of in-line within the driver. The original fix should have been correct in hashing the PKI token to the short-form ID. Your fix to simply hash the tokens is the correct one and more closely mirrors how the original fix was implemented. If you are interested in the reviews that implement the new pluggable provider(s): https://review.openstack.org/#/c/33858/ (V3) and https://review.openstack.org/#/c/34421/ (V2.0). Going with the shorter TTL on the Tokens is a good idea for various reasons depending on the token driver. I know that the SQL driver (provided you cleanup expired tokens) has worked well for my company, but I want to move to the memcache driver soon. Cheers, Morgan Fainberg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Memcache user token index and PKI tokens
On 17/07/2013, at 12:26 AM, Morgan Fainberg m...@metacloud.com wrote: On Tue, Jul 16, 2013 at 4:01 AM, Kieran Spear kisp...@gmail.com wrote: On 16/07/2013, at 1:10 AM, Adam Young ayo...@redhat.com wrote: On 07/15/2013 04:06 AM, Kieran Spear wrote: Hi all, I want to backport the fix for the Token List in Memcache can consume an entire memcache page bug[1] to Grizzly, but I had a couple of questions: 1. Why do we need to store the entire token data in the usertoken-userid key? This data always seems to be hashed before indexing into the 'token-tokenid' keys anyway. The size of the memcache data for a user's token list currently grows by 4k every time a new PKI token is created. It doesn't take long to hit 1MB at this rate even with the above fix. Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit. In this case it looks like we're taking a hit from both, since the PKI token id from the user token index is retrieved, then hashed and then that key is used to retrieve the token from the tokens-%s page anyway. 2. Every time it creates a new token, Keystone loads each token from the user's token list with a separate memcache call so it can throw it away if it's expired. This seems excessive. Is it anything to worry about? If it just checked the first two tokens you'd get the same effect on a longer time scale. I guess part of the answer is to decrease our token expiry time, which should mitigate both issues. Failing that we'd consider moving to the SQL backend. HOw about doing both? But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well. No silver bullet, I am afraid. I think we're going to stick with memcache for now (the devil we know :)). With (1) and (2) fixed and the token expiration time tweaked I think memcache will do okay. Kieran Cheers, Kieran [1] https://bugs.launchpad.net/keystone/+bug/1171985 Hi Kieran, I've looked into the potential bug you described and it appears that there has been a change in the master branch to support the idea of pluggable token providers (much better implementation than the driver being responsible for the token itself). This change modified how the memcache driver stored the IDs, and performed the CMS hashing function when the manager returned the token_id to the driver, instead of in-line within the driver. The original fix should have been correct in hashing the PKI token to the short-form ID. Your fix to simply hash the tokens is the correct one and more closely mirrors how the original fix was implemented. If you are interested in the reviews that implement the new pluggable provider(s): https://review.openstack.org/#/c/33858/ (V3) and https://review.openstack.org/#/c/34421/ (V2.0). Going with the shorter TTL on the Tokens is a good idea for various reasons depending on the token driver. I know that the SQL driver (provided you cleanup expired tokens) has worked well for my company, but I want to move to the memcache driver soon. Thanks for the info. Good to know this is fixed on master. Spotted one possible upgrade issue in the V3 patch which I've submitted a bug for: https://bugs.launchpad.net/keystone/+bug/1202050 And a bug for getting my one-line fix into stable/grizzly: https://bugs.launchpad.net/keystone/+bug/1202053 Cheers, Kieran Cheers, Morgan Fainberg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Keystone] Memcache user token index and PKI tokens
Hi all, I want to backport the fix for the Token List in Memcache can consume an entire memcache page bug[1] to Grizzly, but I had a couple of questions: 1. Why do we need to store the entire token data in the usertoken-userid key? This data always seems to be hashed before indexing into the 'token-tokenid' keys anyway. The size of the memcache data for a user's token list currently grows by 4k every time a new PKI token is created. It doesn't take long to hit 1MB at this rate even with the above fix. 2. Every time it creates a new token, Keystone loads each token from the user's token list with a separate memcache call so it can throw it away if it's expired. This seems excessive. Is it anything to worry about? If it just checked the first two tokens you'd get the same effect on a longer time scale. I guess part of the answer is to decrease our token expiry time, which should mitigate both issues. Failing that we'd consider moving to the SQL backend. Cheers, Kieran [1] https://bugs.launchpad.net/keystone/+bug/1171985 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Keystone] Memcache user token index and PKI tokens
On 07/15/2013 04:06 AM, Kieran Spear wrote: Hi all, I want to backport the fix for the Token List in Memcache can consume an entire memcache page bug[1] to Grizzly, but I had a couple of questions: 1. Why do we need to store the entire token data in the usertoken-userid key? This data always seems to be hashed before indexing into the 'token-tokenid' keys anyway. The size of the memcache data for a user's token list currently grows by 4k every time a new PKI token is created. It doesn't take long to hit 1MB at this rate even with the above fix. Yep. The reason, though, is that we either take a memory/storage hit (store the whole token) or a performance hit (reproduce the token data) and we've gone for the storage hit. 2. Every time it creates a new token, Keystone loads each token from the user's token list with a separate memcache call so it can throw it away if it's expired. This seems excessive. Is it anything to worry about? If it just checked the first two tokens you'd get the same effect on a longer time scale. I guess part of the answer is to decrease our token expiry time, which should mitigate both issues. Failing that we'd consider moving to the SQL backend. HOw about doing both? But if you move to the sql backend, rememeber to periodically clean up the token table, or you will have storage issues there as well. No silver bullet, I am afraid. Cheers, Kieran [1] https://bugs.launchpad.net/keystone/+bug/1171985 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
