On 5/8/09 10:15 PM, Doug Williams wrote:
2) Some of you may have noticed problems about an hour after the
restart. Some important objects expire from cache after an hour. Since
there was a huge influx of objects expiring 1 hour after the restart
presumably together there were problems as the database began to be
overworked. We're doing a post-mortem now to determine how to better
keep the cache hot even after objects are cached nearly simultaneously
after restarts like we experienced today.
Ah, the stampeding herd cold start design problem ...
The simplest solution is to add some jitter - i.e., don't expire things
exactly after 3600 seconds. Set the expiration to be X +/- N, where X
is your lifetime and N is the jitter which might be small like 10 or
large like 300, computed per object usually at the time it's cached.
Over time, given a reasonable PRNG, you should have sufficient drift
that cache expiration will remain random but on a cold start you won't
have all your hot objects expiring at exactly the same time every X period.
--
Dossy Shiobara | [email protected] | http://dossy.org/
Panoptic Computer Network | http://panoptic.com/
"He realized the fastest way to change is to laugh at your own
folly -- then you can let go and quickly move on." (p. 70)
