]] Eric Bowman Hi,
| I'm considering using Varnish to handle caching for a mapping | application. After reading | http://varnish.projects.linpro.no/wiki/ArchitectNotes, it seems like | Varnish is maybe not a good choice for this. In short I need to cache | something like 500,000,000 files that take up about 2TB of storage. | | Using more 1975 technologies, one of the challenges has been how to | distribute these across the file system without putting too many files | per directory. We have a solution we kind of like, and there are others | out there. Hashing on the file name should solve this easily enough, or maybe even better, hash on the hash of the file name, so you have «somefile» where the md5sum of the file name is c21641b4fc25d6d558bf130659d56811. Given how md5 works, and say you want to end up with about 1000 files per directory, you need four or five levels of hashing, so that file would live in c/2/1/6/4/somefile. Five levels give you an average of 476 files per directory. (You can of course use another hash than md5, and it's fine to use it here since we only do it to get a good distribution, not because of any kind of security requirements.) | My impression is that we would start to put a big strain on Varnish and | the OS using it in the standard way. But maybe I'm wrong. Or, is there | a way to plugin a backend to manage this storage, without getting into | the vm-thrash from which Squid suffers? We use a hash internally already, so assuming you make the hash size, for instance 39916801 (a prime number that's not too far from 1/10 of your total number of objects), it should work. Or you could use -h critbit instead, which should scale better, but few people use it so far, so it might well have some bugs. Alternatively, use a hashing load balancer in front and have a bunch of Varnish machines each serving their part of the URL space, like David Birdsong suggested. It'd be interesting to hear your experiences once you get this going. :-) Regards, -- Tollef Fog Heen Redpill Linpro -- Changing the game! t: +47 21 54 41 73 _______________________________________________ varnish-misc mailing list [email protected] http://projects.linpro.no/mailman/listinfo/varnish-misc
