** Description changed: [ Impact ] Validating an application credential token is very slow, taking at least 400ms+ in a simple devstack environment, 5-10x longer than validating a user/password project token. The primary bottleneck during a token validation request (/v3/auth/tokens) is that token.roles is evaluated at least 5 times. validate_token is called twice, first during RBAC to populate the subject token context and again to actually validate the token. Each call to validate_token then called token.roles twice because it first checks if it is None, before calling it again to use the result. Lastly token.roles is evaluated a fifth time during render_token_response_from_model. Each evaluation of token.roles calls through _get_application_credential_roles into list_role_assignments which then makes multiple round-trip SQL queries to the database. Unlike the related get_roles_for_user_and_project function, none of these calls are currently cached/memoized. We memoize list_role_assignments to get the same-speedup. Reduce the number of token.roles calls to only 3 by storing and re-using the token.roles result in validate_token, then memoize list_role_assignments so the 2nd and 3rd call fetch from the cache instead of repeating many SQL queries. This provides a substantial performance improvement bringing validation time in-line with user/password tokens. This bug is being opened to track the Ubuntu SRU of this fix, which was merged upstream without a corresponding bug report: https://review.opendev.org/c/openstack/keystone/+/880456 - [ Test Plan ] source novarc export TOKEN_userpass=$(openstack token issue -f value -c id) openstack application credential delete test1 eval $(openstack application credential create -f shell -c id -c secret test1) export OS_APPLICATION_CREDENTIAL_ID=${id} export OS_APPLICATION_CREDENTIAL_SECRET=${secret} export OS_AUTH_TYPE="v3applicationcredential" unset OS_PASSWORD OS_PROJECT_DOMAIN_NAME OS_PROJECT_NAME OS_USER_DOMAIN_NAME OS_USERNAME OS_PROJECT_DOMAIN_ID OS_TENANT_NAME OS_USER_DOMAIN_ID export TOKEN_appcred=$(openstack token issue -f value -c id) - time curl ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token: - ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H "Accept- - Encoding: gzip" -H "Accept: application/json" + curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H "X-Auth- + Token: ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_userpass}" -H + "Accept-Encoding: gzip" -H "Accept: application/json" --write-out + "%{time_total}\n" - time curl ${OS_AUTH_URL}/auth/tokens -H "X-Auth-Token: - ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H "Accept- - Encoding: gzip" -H "Accept: application/json" + curl --silent --output /dev/null ${OS_AUTH_URL}/auth/tokens -H "X-Auth- + Token: ${TOKEN_userpass}" -H "X-Subject-Token: ${TOKEN_appcred}" -H + "Accept-Encoding: gzip" -H "Accept: application/json" --write-out + "%{time_total}\n" # Timing is approximately ~0.340s tor token1, ~0.061s for token2 - [ Where problems could occur ] Improving this performance required two changes. The first simply removed immediately duplicate calls to expensive SQL queries, regression in that part of the code is less likely. The second change however was to start caching the result of some queries, and ensuring correct cache invalidation is an important concern. A thorough audit of locations where cache invalidation is required was taken and those locations were addressed. Additionally the upstream unit tests fortunately already tested these code paths and also highlighted the lack of cache invalidation, and was resolved once cache invalidation was in place. This patch has been merged upstream for over 18 months since the Caracal release and in-use in a relevant production environment for the same amount of time. I conducted a search of all commits since then and have not found any fixes either for this commit or related code. This reduces regression potential. We are back porting this to 4 previous OpenStack releases. This was also done upstream and the upstream tests all pass.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2115144 Title: [SRU] Improve application credential validation speed To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/keystone/+bug/2115144/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
