On Sat, Nov 21, 2009 at 1:31 AM, Eric Bowman <[email protected]> wrote: > Thanks -- very useful and helpful. > > cheers, > Eric > of course my equation was wrong though, should be: working_set / optimal_size = N
> David Birdsong wrote: >> On Fri, Nov 20, 2009 at 2:19 PM, Eric Bowman <[email protected]> wrote: >> >>> Hi, >>> >>> Apologies if this has been hashed out before. I did some googling, and >>> read the faq, but I could have been more thorough... ;) >>> >>> I'm considering using Varnish to handle caching for a mapping >>> application. After reading >>> http://varnish.projects.linpro.no/wiki/ArchitectNotes, it seems like >>> Varnish is maybe not a good choice for this. In short I need to cache >>> something like 500,000,000 files that take up about 2TB of storage. >>> >>> Using more 1975 technologies, one of the challenges has been how to >>> distribute these across the file system without putting too many files >>> per directory. We have a solution we kind of like, and there are others >>> out there. >>> >>> My impression is that we would start to put a big strain on Varnish and >>> the OS using it in the standard way. But maybe I'm wrong. Or, is there >>> a way to plugin a backend to manage this storage, without getting into >>> the vm-thrash from which Squid suffers? >>> >>> Thanks for any advice -- Varnish gets such good press I'd really love if >>> it were straightforward to use it in this case. >>> >>> -Eric >>> >> a straight forward way to store an unlimited amount of data is to find >> the optimal cache storage capacity per varnish instance then: >> >> optimal_size / working_set = N >> >> where N is the number of varnish instances you need to run. >> >> then put a layer 7 switch in front of the pool of varnish instances, >> hashing on the requests. >> >> works like a charm. >> >> finding optimal storage amount per varnish requires turning the knobs: >> - tuning VM >> - tuning kernel for high network traffic >> - balancing between big and fast storage medium >> random reads will skyrocket, minimize writing to storage while >> serving if possible (pregenerate your working set, dont let anything >> expire between generating ) >> ..and test >> >> >>> Eric Bowman >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> [email protected] >>> http://projects.linpro.no/mailman/listinfo/varnish-misc >>> >>> > > > -- > Eric Bowman > Boboco Ltd > [email protected] > http://www.boboco.ie/ebowman/pubkey.pgp > +35318394189/+353872801532 > > _______________________________________________ varnish-misc mailing list [email protected] http://projects.linpro.no/mailman/listinfo/varnish-misc
