Actually, there are two HTTP methods that must be supported by the service: HEAD (to first obtain the actual content length) and GET (with Range headers).
Google Storage for Developers is definitely a service where my approach works. Here are some relevant links: - http://code.google.com/apis/storage/docs/reference-methods.html#headobject - http://code.google.com/apis/storage/docs/reference-headers.html#range I believe Apache Tomcat offers the Range Header functionality over static content, but not sure about the HEAD method. I was able to implement this logic in a javax.servlet.Filter over dynamic content. The main drawback (and the reason why I'm not generally using this approach) is scalability. I may be able to go beyond the 1MB per URLFetch call but there will be an upper bound, and I don't know how to determine that consistently. On Jul 27, 1:47 pm, "Ikai L (Google)" <[email protected]> wrote: > It doesn't seem like a hack at all, actually. The problem, I think, is that > many web servers do NOT support it. Do you have a list of services that do? > > Otherwise, nice find! > > On Tue, Jul 27, 2010 at 12:34 PM, hector rovira > <[email protected]>wrote: > > > > > I found a way to get around the 1MB limit on the URLFetchService, but > > it depends on the HTTP Range header... the web service you are calling > > must support that header. My problem was that I needed to request > > some data from a bucket on Google Storage for Developers (which > > supports Range header queries). > > > So let's say my original request was for 20MB worth of data, which > > hits the 1MB limit throwing a ResponseTooLargeException. I "split" my > > original request into 20 separate requests each with a different range > > (0-1MB, 1MB-2MB, etc...). Then I submitted the 20 requests in > > parallel using URLFetchService.fetchAsynch. Each request would take > > less than 10secs and come back with less than 1MB. Finally, I > > reassembled the responses into a single ByteArrayOutputStream and > > proceeded as if I had made a single request. > > > This was a simple way for me to process a request for an analysis that > > ran on GAE, but where the data was stored externally. I agree that if > > you are trying to serve files you should offer the Blobstore API, or > > simply send a redirect to the actual service holding the data. > > > If this smells like a hack, it probably is. > > > On Jul 27, 11:45 am, "Ikai L (Google)" <[email protected]> wrote: > > > 10 second limit for URLfetch, I mean. You can serve larger data sizes > > using > > > the Blobstore API: > > > >http://code.google.com/appengine/docs/python/blobstore/ > > > > On Tue, Jul 27, 2010 at 11:45 AM, Ikai L (Google) <[email protected]> > > wrote: > > > > > No, that's the limit. Also be aware that there's a 10 second limit. > > > > > On Tue, Jul 27, 2010 at 2:02 AM, Ahmed Emir KAMMOUN < > > > > [email protected]> wrote: > > > > >> Hello, > > > > >> I want to know if there is a possibility to have a more big amount of > > > >> request and response size. > > > >> Actually, I find that it is limited to 1 MB. > > > > >>http://code.google.com/intl/fr-FR/appengine/docs/java/urlfetch/overvi. > > .. > > > > >> Regards, > > > > >> -- > > > >> You received this message because you are subscribed to the Google > > Groups > > > >> "Google App Engine" group. > > > >> To post to this group, send email to > > [email protected]. > > > >> To unsubscribe from this group, send email to > > > >> [email protected]<google-appengine%[email protected]> > > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > >> . > > > >> For more options, visit this group at > > > >>http://groups.google.com/group/google-appengine?hl=en. > > > > > -- > > > > Ikai Lan > > > > Developer Programs Engineer, Google App Engine > > > > Blog:http://googleappengine.blogspot.com > > > > Twitter:http://twitter.com/app_engine > > > > Reddit:http://www.reddit.com/r/appengine > > > > -- > > > Ikai Lan > > > Developer Programs Engineer, Google App Engine > > > Blog:http://googleappengine.blogspot.com > > > Twitter:http://twitter.com/app_engine > > > Reddit:http://www.reddit.com/r/appengine > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%[email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
