Yeah, we're a (very) big site and we'd definitely not be in good shape if memcached died at any point during peak traffic hours. We can (and do) bounce/flush during off-peak times, but memcached is really a cornerstone of our scalability. That said, we almost never write code that dies completely if memcached isn't around (though we have a few special hacks that I know I've at least described to Trond and others in conference calls). In any spot where I had a query like the one above, I'd write the code such that the application only checks memcached for the value. If it's not there, it fails gracefully. Only the background job ever performs the query and updates the cache. This means that no user ever has to wait 20 seconds for a page to load and you avoid having any sort of a herd effect-- I can only imagine what would happen to the DB if this "20 seconds to execute" query got performed by multiple request threads at once.
On Wed, Aug 5, 2009 at 1:51 PM, dormando <[email protected]> wrote: > > Yes, well... For most big sites I'd actually insist that it's okay if you > disappear from the internet if too many of your memcached instances go > away. Losing 10-50% might be enough to kill you and that's okay. Memcached > has been around long enough and integrated tight enough that it *is* a > critical service. Losing the typical number of common failures, or issuing > a slow rolling restart, shouldn't cause you to go offline. Losing a huge > sack all at once might make you limp severely or fail temporarily. > > For very small sites with just 1-2 instances, I'm not too sure what to > recommend. Consider making sure you at least have 2-4 instances and that > you can survive the loss of one, even if you end up running multiple on > one box. > > -Dormando > > On Wed, 5 Aug 2009, Josef Finsel wrote: > > > Or, to put it another way, memcached is *not a persistent data store nor > > should it be treated as one. If your application will fail if memcached > is > > not running, odds are you are using memcached incorrectly.* > > > > On Wed, Aug 5, 2009 at 1:36 PM, dormando <[email protected]> wrote: > > > > > > > > What happens when we release 1.4.1? > > > > > > Your site really should be able to tolerate at least the brief loss of > a > > > single memcached instance. > > > > > > It's definitely good to update the cache asyncronously where possible, > and > > > all of this stuff stuff (a lot is detailed in the FAQ too). I'd just > > > really like to push the point that these techniques help reduce common > > > load. They can even help a lot in emergencies when you find a page > taking > > > 50 seconds to load for no good reason. > > > > > > However, relying on it will lead to disaster. It allows you to create > > > sites which cannot recover on their own after blips in cache, upgrades, > > > hardware failure, etc. > > > > > > If you have objects that really do take umpteen billions of > milliseconds > > > to load, you should really consider rendering data (or just the really > > > slow components of it) into summary tables. I like using memcached to > take > > > queries that run in a few tens of milliseconds or less, and make them > > > typically run in 0.1ms. If they take 20+ seconds something is horribly > > > wrong. Even 1 full second is something you should be taking very > > > seriously. > > > > > > -Dormando > > > > > > On Wed, 5 Aug 2009, Edward Goldberg wrote: > > > > > > > > > > > I use the term "Cache Warmer" for task like this, where I keep a key > > > > (or keys) warm all of the time. > > > > > > > > The best way to keep the load on the DB constant is to Warm the Cache > > > > at a well defined rate. > > > > > > > > If you wait for a "request" to update the cache, then you may loose > > > > control over the exact moment of the load. > > > > > > > > The idea is to make a list of the items that need to be warmed, and > > > > then do that list at a safe rate and time. > > > > > > > > Very long TTL values can be un-safe. Old values can lead to code > > > > errors that are not expected days later. > > > > > > > > Edward M. Goldberg > > > > http://myCloudWatcher.com/ > > > > > > > > > > > > On Wed, Aug 5, 2009 at 9:44 AM, dormando<[email protected]> wrote: > > > > > > > > > > Also consider optimizing the page so it doesn't take 20 seconds to > > > render > > > > > - memcached should help you under load, and magnify capacity, but > > > > > shouldn't be used as a crutch for poor design. > > > > > > > > > > -Dormando > > > > > > > > > > On Wed, 5 Aug 2009, Adam Lee wrote: > > > > > > > > > >> Run a cron job that executes the query and updates the cache at an > > > interval > > > > >> shorter than the expiration time for the cached item. > > > > >> > > > > >> On Wed, Aug 5, 2009 at 11:38 AM, Haes <[email protected]> wrote: > > > > >> > > > > >> > > > > > >> > Hi, > > > > >> > > > > > >> > I'm using memcached together with Django to speed up the > database > > > > >> > queries. One of my Django views (page) uses a query that takes > over > > > 20 > > > > >> > sec. Normally this query hits the memcached cache and the data > is > > > > >> > served almost instantly. The problem now is that if the cache > > > expires, > > > > >> > the next person accessing this page will have to wait about 20 > > > seconds > > > > >> > for it to load which is not really acceptable for me. > > > > >> > > > > > >> > Is there a way to update the memcached data before it times out, > in > > > a > > > > >> > way that this query always hits the cache? > > > > >> > > > > > >> > Thanks for any hints. > > > > >> > > > > > >> > Cheers. > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> -- > > > > >> awl > > > > >> > > > > > > > > > > > > > > > > > > > > -- > > "If you see a whole thing - it seems that it's always beautiful. Planets, > > lives... But up close a world's all dirt and rocks. And day to day, > life's a > > hard job, you get tired, you lose the pattern." > > Ursula K. Le Guin > > > > What's different about data in the cloud? http://www.azuredba.com > > > > http://www.finsel.com/words,-words,-words.aspx (My blog) - > > http://www.finsel.com/photo-gallery.aspx (My Photogallery) - > > http://www.reluctantdba.com/dbas-and-programmers/blog.aspx (My > Professional > > Blog) > > > -- awl
