On Mon, Jun 09, 2008 at 05:15:33PM +0200, Florian Lohoff wrote: > http://silicon-verl.de/home/flo/tmp/tahbench-20080609.tgz > > to play with. Its by no means optimized or fixes all problems but it > should show that the millions of little files approach is a performance > problem. BTW: With 4K blocks in the filesystem one wastes an average > of 2K per file which with my working set (~140K files) this sums up to > 280Mbyte wasted space on disk but even worse - worse cache footprint. > With 16kbyte average file size 12% of disk space is wasted because of > the last block beeing statistically half empty.
Just as a coincidence - Slashdot is running a story about the Facebook photo storage. Basically they want to do the same as this. Storing all the photos in large chunks with external index. They even encode the start and length of the file to serve within the url (and some security measures). http://beta.flowgram.com/f/p.html#2qi3k8eicrfgkv So - if they start serving 450TByte with 6.5 Billion photos like this it cant be all so wrong. Flo -- Florian Lohoff [EMAIL PROTECTED] +49-171-2280134 Those who would give up a little freedom to get a little security shall soon have neither - Benjamin Franklin
signature.asc
Description: Digital signature
_______________________________________________ Tilesathome mailing list [email protected] http://lists.openstreetmap.org/cgi-bin/mailman/listinfo/tilesathome
