Hi there! I thought It would be useful for some people out there to mention a "case of successful" on the Persistent branch of Varnish Cache on a large production scale. Ours is the replacement of the so-long-time-appreciated Squid Cache for delivering (mostly) user-provided content, like Images and other kinds of documents, but static objects like Javascripts and style sheets belonging to our platform, OLX, as well.
- TL;DR Version The development around the persistent branch rocks, *big time*! - Full Version // *Disclaimer* // Our configuration is not what one could call "conventional", and we can't really afford loosing large amounts of cache in short periods of time. - The Business Our company, OnLine eXchange or simply OLX for short, has established its core around the Free Classifieds business, enabling Sellers and Buyers to perform peer-to-peer selling transactions with no commission required to any parts. Our portal allows sellers to upload images for better describing their items into our platform and being more attractive to potential buyers and having better chances of selling them. Just like Craiglist does in the US or Allegro in Poland. Although some particular countries are more popular than others, we have a large amount of traffic world-wide, due to an extensive net of operations and localization. To sum up, this translates into having to tune every single piece of software we run out there for literally delivering several gbps of traffic out to the internet. - Massive Delivering Even though we heavily rest on content delivery networks (CDNs) for performing last-mile optimizations, geo-caching and network optimized delivery, they have limits on what they can handle and they also run caching rotation algorithms over objects as well (say, something like LRU). The do all the time, more often that you do actually want. On the other hand, because of their geo-distribution nature, in reality several request are performed from different locations before they effectively cache objects for some small period of time. It might be true that we are on average offloaded +80% of the traffic, but again, in reality even that small portion has average 400 mbits/s, with frequent 500mbps spikes in business-as-usual levels. - Ancient History We used to run this internal caching tier with Squid Cache backing the cache up, with the help of tons of optimizations and an extremely experimental backend called Cyclic Object storage system or COSS. This very very experimental backend made use of a very granular storage configurations to get the most out of every IOP we could save to our loaded storage backends. Due to its immaturity, some particular operations sucked, like a *30 minutes downtime* per instance by restarting the squid engine because of COSS data rebuild & consistency checks. We also rested very heavily on a sibling caching relation ship configuration by using HTCP, on a way to maximize the caching availability and saving some objects to get to the origin, which, as I said, was heavily loaded almost all the time. Squid is a great tool and believe me when I say, we loved it with all the downsides it had, we have had it for several years up to very recently, we knew it backwards and forwards until then. - Modern History Things started to change for the worse recently when our long-standing-loaded storage backend tier, featuring a vendor supposedly standing at the Storage Top Five ranking, could not face our business-as-usual operations any longer. Performance started to fall apart in pieces once we started to hit +95% on this well-known storage provider's solution CPU usage, unable to serve objects in form and shape once 95% was reached. We were forced to start diving into new and radical alternatives on a very short term. We have been using Varnish internally since some years now for boosting our SOLR backend and some other HTTP caching roles, but not for delivering static content up to that moment. Now was the time to give it a chance, "for real" (we have several gbps of traffic internally too, but you know what I mean). - First Steps We started by using the same malloc backend configurations as we were using on this other areas were Varnish was deployed, with some performance tunes around sessions (VARNISH_SESSION_LINGER) and threads capacity (VARNISH_MAX_THREADS &&VARNISH_THREAD_POOLS && VARNISH_THREAD_TIMEOUT). The server profiles handling this tasks were some huge boxes with 128M RAM, but since the 50-ish terabytes dataset didn't actually fit on RAM, once all the memory was allocated we started to suffer same random panics at random periods of time. Unable to replicate them on demand or to produce any debugging due to the amount of traffic this huge devils handled on production, things started to get really ugly really fast. - The Difficult Choice At this point we started to consider every possible option available out there, even switching away to other caching alternatives, like trafficserver, that provided persistence for cached objects, and we decided to give the persistent backend a shoot, but, there was a catch: the persistent backend was (and also currently is) considered experimental (so did COSS, btw!). Effectively, as for what 3.0.4 release on the main stable branch respects, the persistent backend had many bugs that raised up within minutes, many of them produced panics that crashed the child process, but, fortunately for us, the persistence itself worked so well that the first times went through totally unnoticed, which, compared with experiencing a 100mb cache melt down was just an amazing improvement by itself. Now things started to look better for Varnish. Of course, we learned that in fact we were loosing some cached objects by some broken silos in the way, but, doing a side-by-side comparison things looked different as night and day, and the best was yet to come. We were advised, in case we were to stick to a persistent backend, on using the persistent development branch, giving that more improvements were developed in there and major stability changes were introduced. But, think again, proposing something that has the "experimental" and "on development" tags hanging from it usually sells horribly to the management people on the other side of the table. - Summary At the end, with the help of a hash-based balancing algorithm at the load balancing tier in front of our Varnish caches, we were able to *almost cut half* of the CPU usage at our storage solution tier, that is by geting 60% insead of +90% of CPU usage, something similar to a sunny day for a walk through the park, even for serving the +2.000 request/second arriving at our datacenters. We got there by offloading content on a *up to +70% cache hits*, something that was totally unconceivable for anyone at the very beginning of the migration, giving that we used to get less than 30% in the past with Squid. We were able to get up to this point with lots of patience and research, but particularly with the help of the Varnish core development that constantly supported us at the IRC channels and mailing lists. Thanks a lot guys, you and Varnish rock, big time! A happy user! Dererk -- BOFH excuse #274: It was OK before you touched it.
signature.asc
Description: OpenPGP digital signature
_______________________________________________ varnish-misc mailing list [email protected] https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
