I found a way to get around the 1MB limit on the URLFetchService, but it depends on the HTTP Range header... the web service you are calling must support that header. My problem was that I needed to request some data from a bucket on Google Storage for Developers (which supports Range header queries).
So let's say my original request was for 20MB worth of data, which hits the 1MB limit throwing a ResponseTooLargeException. I "split" my original request into 20 separate requests each with a different range (0-1MB, 1MB-2MB, etc...). Then I submitted the 20 requests in parallel using URLFetchService.fetchAsynch. Each request would take less than 10secs and come back with less than 1MB. Finally, I reassembled the responses into a single ByteArrayOutputStream and proceeded as if I had made a single request. This was a simple way for me to process a request for an analysis that ran on GAE, but where the data was stored externally. I agree that if you are trying to serve files you should offer the Blobstore API, or simply send a redirect to the actual service holding the data. If this smells like a hack, it probably is. On Jul 27, 11:45 am, "Ikai L (Google)" <[email protected]> wrote: > 10 second limit for URLfetch, I mean. You can serve larger data sizes using > the Blobstore API: > > http://code.google.com/appengine/docs/python/blobstore/ > > On Tue, Jul 27, 2010 at 11:45 AM, Ikai L (Google) <[email protected]> wrote: > > > > > No, that's the limit. Also be aware that there's a 10 second limit. > > > On Tue, Jul 27, 2010 at 2:02 AM, Ahmed Emir KAMMOUN < > > [email protected]> wrote: > > >> Hello, > > >> I want to know if there is a possibility to have a more big amount of > >> request and response size. > >> Actually, I find that it is limited to 1 MB. > > >>http://code.google.com/intl/fr-FR/appengine/docs/java/urlfetch/overvi... > > >> Regards, > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%[email protected]> > >> . > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Ikai Lan > > Developer Programs Engineer, Google App Engine > > Blog:http://googleappengine.blogspot.com > > Twitter:http://twitter.com/app_engine > > Reddit:http://www.reddit.com/r/appengine > > -- > Ikai Lan > Developer Programs Engineer, Google App Engine > Blog:http://googleappengine.blogspot.com > Twitter:http://twitter.com/app_engine > Reddit:http://www.reddit.com/r/appengine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
