The general consensus here is right - use a cache. This is useful on any system that generates tiles, not just GAE. Hopefully your traffic tends to predominantly hit a core set of tiles.
Make sure that you have a Cache-Control header with a reasonable max-age. Nothing will work right without this. GAE has a built-in edge cache with undocumented behavior and no guarantee of service. That's the first layer of defense. If you already have Cache-Control set then this isn't good enough. Try CloudFlare. It's free, documented, and you will be able to see charts of exactly how effective the cache is. If CF isn't effective enough, look around at other CDNs. You could set up a varnish instance yourself somewhere but you need to look carefully at the bandwidth costs; it doesn't do any good to push the cost around if you're still paying the same amount per gigabyte. A quick web search turned up these guys advertising $0.06/gb: http://www.scaleengine.com/esc/ (Referenced from http://stackoverflow.com/questions/72369/whats-the-best-cdn-for-image-hosting-on-a-high-volume-web-site. Also numerous recommendations for MaxCDN, already mentioned) Jeff On Wed, Aug 22, 2012 at 11:03 AM, Sameer Lodha <[email protected]> wrote: > I would suggest Cloudflare as they have no limits on bandwidth utilization & > the bandwidth is completely free. You can make do with there free plan. > Setup is trivial as well. > > > On Wed, Aug 22, 2012 at 6:50 PM, Richard Watson <[email protected]> > wrote: >> >> Some options: >> >> 1) At the very least, ensure Google's Edge Cache is able to cache your >> images by adjusting cache control. Search this forum for some thoughts on >> how. >> >> 2) You could try Google's PageSpeed service, although it could take time >> to set up (if they accept you): >> https://developers.google.com/speed/pagespeed/service >> >> 3) Cloudflare.com, which should reduce the load significantly, and is >> fairly quick to set up. No changes to your code. >> >> 4) Put the images on Amazon's S3 and CloudFront, but that'll mean some >> changes to publish images to it. >> >> 5) Use another CDN, as already mentioned. >> >> >> On Wednesday, August 22, 2012 1:09:34 PM UTC+2, noiv wrote: >>> >>> Thanks Rob, Richard, >>> >>> the image tiles are directly served from GAE and only outgoing bandwidth >>> limits capacity. I'm going to reach out in Winter for sponsoring, but the >>> question is how to survive next 8 weeks? >>> >>> If I understand you correctly there might be an option to put a provider >>> with unlimited bandwidth in front, so GAE serves each image only once. How >>> could I start that? >>> >>> -- Torsten >>> >>> >>> >>> >>> >>> >>> >>> On Wednesday, August 22, 2012 11:55:13 AM UTC+2, Rob Coops wrote: >>>> >>>> The big question for me is where are you serving these images from? >>>> If you are serving them directly from NASA servers or from an >>>> alternative source then you would most likely see very little traffic as >>>> most of it will be just URI's pointing to the images. There are a lot of >>>> hosting companies out there that claim to deliver unlimited bandwidth (not >>>> true I am sure, but worth giving it a shot). GAE seems to have been purely >>>> at delivering functionality actual content should be served from other >>>> locations if you want to keep bill within reason. >>>> >>>> If you are already showing people the images from an location other then >>>> GAE you will have to choose to limit the amount of data people can consume >>>> (only a static image no zooming, panning or any other fancy stuff) >>>> informing >>>> visitors that you are unable to afford this functionality due to the high >>>> bandwidth demands. If that still won't do it you could attempt to find some >>>> company willing to sponsor your efforts, unfortunately the nature of GAE >>>> means that portability is not to great so moving to an alternative host >>>> will >>>> be difficult at best. >>>> >>>> I have seen many projects that got sponsorship from companies or >>>> government organisations to allow them to continue to provide the unique >>>> information to viewers. >>>> >>>> On Wed, Aug 22, 2012 at 11:40 AM, Richard Watson <[email protected]> >>>> wrote: >>>>> >>>>> Which resources are being hit the hardest? Outgoing bandwidth? >>>>> >>>>> >>>>> On Wednesday, August 22, 2012 6:51:45 AM UTC+2, Torsten Becker wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> since two years I’m running a blog at GAE focusing the Arctic and as >>>>>> an unique feature a Google Map with daily high resolution Arctic >>>>>> satellite >>>>>> images from NASA was included. The NASA images need to be tiled, cached >>>>>> and >>>>>> served and this process runs on GAE, too. >>>>>> >>>>>> Usually interest is low during dark Arctic Winter and rises in >>>>>> September the time sea ice reaches its minimum extent. No problem so far >>>>>> with the free quota on outgoing bandwidth. >>>>>> >>>>>> This year is different: Latest Arctic storm reduced sea ice extent by >>>>>> a million square kilometers in a week and public interest was so high >>>>>> free >>>>>> quota was exhausted 5 hours after reset. >>>>>> >>>>>> Now - end of August - it is absolutely clear that this year will or >>>>>> has already broke all records in terms of sea ice minimum and makes a >>>>>> major >>>>>> step direction ice free-ness. When in a few years the Arctic lacks sea >>>>>> ice >>>>>> completely in September, it will change weather pattern all over the >>>>>> northern hemisphere - one explanation of accelerating public interests. >>>>>> >>>>>> I’d like to mention the project is ad free and totally beyond any >>>>>> economic interests. All I want is to keep it running and give everybody >>>>>> on >>>>>> the planet the chance to see with his own eyes how dramatic the >>>>>> situation in >>>>>> the Arctic is. True color satellite images are free of interpretation >>>>>> and do >>>>>> not lead to discussions whether there is sea ice or not. >>>>>> >>>>>> Here is the thing: If I enable billing to satisfy the need for pure >>>>>> information I’m bankrupt next month. If not 99% of the users are going to >>>>>> see nothing, get frustrated and possibly never come back. >>>>>> >>>>>> So my best option is to close the site now. >>>>>> >>>>>> What do you think? >>>>>> >>>>>> -- Torsten >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Google App Engine" group. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msg/google-appengine/-/OTYSfYfbDgoJ. >>>>> >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/google-appengine?hl=en. >>>> >>>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/google-appengine/-/6B2A1AnEMhYJ. >> >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
