Reviewed: https://review.openstack.org/644774 Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=1192f185a5fd2fa6177655f157146488a3de81d1 Submitter: Zuul Branch: master
commit 1192f185a5fd2fa6177655f157146488a3de81d1 Author: Morgan Fainberg <[email protected]> Date: Fri Mar 22 12:35:16 2019 -0700 Pass `flush_on_reconnect` to memcache pooled backend If a memcache server disappears and then reconnects when multiple memcache servers are used (specific to the python-memcached based backends) it is possible that the server will contain stale data. The default is now to supply the ``flush_on_reconnect`` optional argument to the backend. This means that when the service connects to a memcache server, it will flush all cached data in the server. The pooled backend is more likely to run into issues with this as it does not explicitly use a thread.local for the client. The non-pooled backend was not touched, it is not the recommended production use-case. See the help from python-memcached: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3 closes-bug: #1819957 ** Changed in: oslo.cache Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1819957 Title: Caching with stale data when a server disconnects due to network partition and reconnects Status in OpenStack Identity (keystone): Invalid Status in keystonemiddleware: Triaged Status in oslo.cache: Fix Released Status in OpenStack Security Advisory: Won't Fix Bug description: The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows: 1* Password changes/user changes may be reverted for the cache TTL 1a* User may get locked out if PCI-DSS is on and the password change happens during the network partition. 2* Grant changes may be reverted for the cache TTL 3* Resources (all types) may become "undeleted" for the cache TTL 4* Tokens (KSM) may become valid again during the cache TTL As noted in the python-memcached library: @param flush_on_reconnect: optional flag which prevents a scenario that can cause stale data to be read: If there's more than one memcached server and the connection to one is interrupted, keys that mapped to that server will get reassigned to another. If the first server comes back, those keys will map to it again. If it still has its data, get()s can read stale data that was overwritten on another server. This flag is off by default for backwards compatibility. The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning). This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1819957/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
