Re: how to...accelarate randon access to millions of images?
Am Sonntag 16 März 2008 15:54:42 schrieben Sie: Sascha Ottolski [EMAIL PROTECTED] writes: Now my question is: what kind of hardware would I need? Lots of RAM seems to be obvious, what ever a lot may be...What about the disk subsystem? Should I look into something like RAID-0 with many disk to push the IO-performance? First things first: instead of a few large disks, you want lots of small fast ones - 36 GB or 72 GB 10,000 RPM disks - to maximize bandwidth. There are two ways you might condfigure your storage: one is to place all the disks in a RAID-0 array and use a single file system and storage file on top of that. The alternative is to have a separate file system and storage file on each disk. I honestly don't know which will be the fastest; if you have a chance to run some benchmarks, I'd love to see your numbers and your conclusion. Dag, thanks for your hints. Just curious, how would i tell the varnish-process that I have several filesystem to put the cache on? I had the feeling that you give exactly one directory as option. Cheers, Sascha Note that even if a single RAID-0 array turns out to be the fastest option, you may have to compromise and split your disks into two arrays, unless you find a RAID controller that can handle the number of disks you need. DES ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: how to...accelarate randon access to millions of images?
Michael, thanks a lot for taking the time to give me such a detailed answer. please see my replies below. Am Sonntag 16 März 2008 18:00:42 schrieb Michael S. Fischer: On Fri, Mar 14, 2008 at 1:37 PM, Sascha Ottolski [EMAIL PROTECTED] wrote: The challenge is to server 20+ million image files, I guess with up to 1500 req/sec at peak. A modern disk drive can service 100 random IOPS (@ 10ms/seek, that's reasonable). Without any caching, you'd need 15 disks to service your peak load, with a bit over 10ms I/O latency (seek + read). The files tend to be small, most of them in a range of 5-50 k. Currently the image store is about 400 GB in size (and growing every day). The access pattern is very random, so it will be very unlikely that any size of RAM will be big enough... Are you saying that the hit ratio is likely to be zero? If so, consider whether you want to have caching turned on the first place. There's little sense buying extra RAM if it's useless to you. well, wo far I have analyzed the webserver logs of one week. this indicates that indeed there would be at least some cache hits. we have about 20 mio. images on our storage, and in one week about 3.5 mio images were repeadetly requested. to be more precise: 272,517,167 requests made to a total of 7,489,059 different URLs 3,226,150 URLs were requested at least 10 times, accounting for 257,306,351 repeated request so, if I made my analysis not to lousy, I guess there is quite a opportunity that a cache will help. roughly, the currently 20 mio images use 400 GB of storage; so 3.5 mio images may account for 17.5% of 400 GB ~70 GB. well, but 70 GB RAM is still a lot. but may be a mix of enough RAM and fast disks may be the way to go; may be in addition to a content based load-balancing to several caches (say, one for thumbnails, one for larger size images). currently, at peak times we only serve about 350 images/sec, due to the bottleneck of the storage backend. so the targed of 1500 req/sec may be bit of wishful thinking, as I don't know what the real peak would look like without the bottleneck; may very well be more like 500-1000 req/sec; but of course I'd like to leave room for growth :-) Thanks a lot, Sascha Now my question is: what kind of hardware would I need? Lots of RAM seems to be obvious, what ever a lot may be...What about the disk subsystem? Should I look into something like RAID-0 with many disk to push the IO-performance? You didn't say what your failure tolerance requirements were. Do you care if you lose data? Do you care if you're unable to serve some requests while a machine is down? well, it's a cache, after all. the real image store is in place and high available and backed up and all the like. but, the webservers can't get the images fast enough of the storage. we just enabled apache's mod_cache, which seems to help a bit, but I suspect a dedicated tool like varnish could perform better (plus, you don't get any runtime information to learn how efficient the apache cache is). Consider dividing up your image store onto multiple machines. Not only would you get better performance, but you would be able to survive hardware failures with fewer catastropic effects (i.e., you'd lose only 1/n of service). If I were designing such a service, my choices would be: (1) 4 machines, each with 4-disk RAID 1 (fast, but dangerous) (2) 4 machines, each with 5-disk RAID 5 (safe, fast reads, but slow writes for your file size - also, RAID 5 should be battery backed, which adds cost) (3) 4 machines, each with 4-disk RAID 10 (will meet workload requirement, but won't handle peak load in degraded mode) (4) 5 machines, each with 4-disk RAID 10 (5) 9 machines, each with 2-disk RAID 0 Multiply each of these machine counts by 2 if you want to be resilient to failures other than disk failures. You can then put a Varnish proxy layer in front of your image storage servers, and direct incoming requests to the appropriate backend server. --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: how to...accelarate randon access to millions of images?
On Fri, Mar 14, 2008 at 1:37 PM, Sascha Ottolski [EMAIL PROTECTED] wrote: The challenge is to server 20+ million image files, I guess with up to 1500 req/sec at peak. well, wo far I have analyzed the webserver logs of one week. this indicates that indeed there would be at least some cache hits. we have about 20 mio. images on our storage, and in one week about 3.5 mio images were repeadetly requested. to be more precise: I just wonder why would you use varnish? Is there any extensive database or other processing power you need to complete before serving the image? Wouldn't you be better of using some kind of Content Delivery Network, possibly using shards of your image store? Serving static files using a http-accelerator seems odd to me. Greetings Christoph ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: how to...accelarate randon access to millions of images?
In message [EMAIL PROTECTED], C. H andel writes: Finally, I would advice you guys to seriously look at flash-disk drives. The virtual elimination of seektime is just what you want from a web server or cache. Having Flash Drives for 400GB of content could kill some budgets ;) It's a price performance issue: I think the 32GB 2.5 Mtron is in the $1K area, so you'd need about 15 of those + some carriers and controllers. Call it $20K in total. For that you get a compact disk system with virtually no seek-time, a power dissipation of around 40W and no disk-crashes. I know people who nearly cry when they realize that is possible :-) Poul-Henning -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED] | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: how to...accelarate randon access to millions of images?
On Fri, Mar 14, 2008 at 1:37 PM, Sascha Ottolski [EMAIL PROTECTED] wrote: The challenge is to server 20+ million image files, I guess with up to 1500 req/sec at peak. A modern disk drive can service 100 random IOPS (@ 10ms/seek, that's reasonable). Without any caching, you'd need 15 disks to service your peak load, with a bit over 10ms I/O latency (seek + read). The files tend to be small, most of them in a range of 5-50 k. Currently the image store is about 400 GB in size (and growing every day). The access pattern is very random, so it will be very unlikely that any size of RAM will be big enough... Are you saying that the hit ratio is likely to be zero? If so, consider whether you want to have caching turned on the first place. There's little sense buying extra RAM if it's useless to you. Now my question is: what kind of hardware would I need? Lots of RAM seems to be obvious, what ever a lot may be...What about the disk subsystem? Should I look into something like RAID-0 with many disk to push the IO-performance? You didn't say what your failure tolerance requirements were. Do you care if you lose data? Do you care if you're unable to serve some requests while a machine is down? Consider dividing up your image store onto multiple machines. Not only would you get better performance, but you would be able to survive hardware failures with fewer catastropic effects (i.e., you'd lose only 1/n of service). If I were designing such a service, my choices would be: (1) 4 machines, each with 4-disk RAID 1 (fast, but dangerous) (2) 4 machines, each with 5-disk RAID 5 (safe, fast reads, but slow writes for your file size - also, RAID 5 should be battery backed, which adds cost) (3) 4 machines, each with 4-disk RAID 10 (will meet workload requirement, but won't handle peak load in degraded mode) (4) 5 machines, each with 4-disk RAID 10 (5) 9 machines, each with 2-disk RAID 0 Multiply each of these machine counts by 2 if you want to be resilient to failures other than disk failures. You can then put a Varnish proxy layer in front of your image storage servers, and direct incoming requests to the appropriate backend server. --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
Re: how to...accelarate randon access to millions of images?
On Sun, Mar 16, 2008 at 10:02 AM, Michael S. Fischer [EMAIL PROTECTED] wrote: I don't know why I'm having such a problem with this. Sigh! I think I got it right this time. If I were designing such a service, my choices would be: Corrections: (1) 4 machines, each with 4-disk RAID 0 (fast, but dangerous) (2) 4 machines, each with 5-disk RAID 5 (safe, fast reads, but slow writes for your file size - also, RAID 5 should be battery backed, which adds cost) (3) 4 machines, each with 4-disk RAID 10 (will meet workload requirement, but won't handle peak load in degraded mode) (4) 5 machines, each with 4-disk RAID 10 (5) 9 machines, each with 2-disk RAID 1 --Michael ___ varnish-misc mailing list varnish-misc@projects.linpro.no http://projects.linpro.no/mailman/listinfo/varnish-misc
