With the backends I can't display data to use when he ask for them - he can't wait next hour for getting the prices of books as he wants them now. So I need to run the backend all the item and process price requests as soon as possible (and as there is 5+ million books I can't do a pre-fetch of some books on backend and then shut it down).
I will try to implement that async url fetch for batch of requests, that can by actually fast - just some suppliers are really slow and some of them fast so maybe I could also do two queues (one for slow response and one for fast) so the user get info from fast responsive suppliers quickly and doesnt need to wait) - the only issue is that technically I should process the pull queue from backend so that means I would need it to run all the time ;/ (I really need to display the prices as soon as possible and can't wait with processing). I will try to lower the latency so GAE can spin up more instance and will see how it affect the instance time. Thank you guys! On Sep 14, 1:57 pm, Rishi Arora <[email protected]> wrote: > My app is nearly identical to yours - several concurrent URL fetches are > performed to "gather" content. And when users access my site, that content > is nicely formatted for them for display. My solution - part of if has > already been suggested by Jeff - use async URL fetch. Spinning an instance > for 5 to 10 seconds waiting for your supplier's website to return book > pricing data is a waste of resources. So, whenever you need to do a URL > fetch, consider deferring it by queueing it up in a pull queue. Then once > you have enough deferred (10 is a good number), then call URL fetch > simultaneously for all the 10 requests. Another optimization is to use by > the 9 free hours of backend time. I agree you can't have the backend > running all the time. So, wake up the backend by a cron job that runs once > every hour. This will incur a minimum cost of 15 minutes per hour = 6 > instance hours - which is under the free quota. Each time your backend > wakes up, it looks up all the URL fetch requests deferred in the last hour, > and processes them. My app does exactly this, and it takes me about 45 > seconds to fetch and process all data for ~300 URL fetches every hour. > > If you are attempting to stay within the free quota, absolutely use the > backend hours in any way you can. it'll be a pity to not use those free 9 > instance hours. > > > > > > > > On Tue, Sep 13, 2011 at 8:43 PM, Tim Hoffman <[email protected]> wrote: > > Hi > > > You could submit the request via ajax back to your appengine app > > and it can then do an async requuest on all the urls, . > > > In your case you have some of the info already and have to fetch some of > > it. > > So it might be two ajax calls, one to get the list of books, the result is > > book prices for stuff you know, plus an indicator of the books that a > > further request > > will be required, your front end can then display the details you have, > > submit another > > ajax request to appengine to fetch results for the books you currently have > > no info on. > > Which can then async urlfetch the rest of the details. > > > This way user gets some info straight away and you get to keep you requests > > to a minimum > > and fill in the results later. > > > Just a thought ;-) > > > T > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To view this discussion on the web visit > >https://groups.google.com/d/msg/google-appengine/-/3dA05F9-QDsJ. > > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
