I have used AppStats to optimize my datastore API calls : I'm using batch gets as much as possible and async puts to update my datastore cache.
When memcache is empty, I think I do about 10 datastore API calls to build a standard page and 2 async puts to update the cache. What is strange is that is will run perfectly fine most of the time with decent times (~300ms) and suddenly for unknow reasons some requests will hang for 30s and generate DeadlineExceededExceptions. I'll try to give you some more precise numbers as soon as possible. Thanx again for your help. Francois On Wed, May 18, 2011 at 18:23, Robert Kluin <[email protected]> wrote: > Hey Francois, > I'm not sure about datastore throttling. I know a few months ago I > was hitting a short-term throttle, but it was at high write rates, > something on the order of many thousands of API calls per minute as I > recall. I've heard that may have been lifted now, but I can't > confirm. > > It really sounds like you need to look into batching your datastore > operations to reduce the number of RPCs. Using Appstats to examine > the calls being made should help you. > > > Robert > > > > > On Wed, May 18, 2011 at 11:53, Francois MASUREL <[email protected]> wrote: > > Hi Robert, > > Thanx for your answer. > > Do you confirm that there is some kind of throttling for datastore usage > ? > > I am really interested in knowing if such limitations exist. > > As all my HTML page elements are dynamically loaded from datastore when > not > > cached (images, CSS, etc.), there are for sure quite a few datastore > > requests being done when memcache is cleared as all these elements need > to > > be reloaded on the client side almost simultaneously. > > What is the best solution to circumvent such problems ? > > Thanx again for your help. > > > > Francois > > > > > > On Wed, May 18, 2011 at 17:35, Robert Kluin <[email protected]> > wrote: > >> > >> Hi Francois, > >> You need to use Appstats. It sounds like you're probably doing a > >> lot of datastore operations, perhaps fetching a very large number of > >> entities or lots of operations in serial. > >> > >> http://code.google.com/appengine/docs/java/tools/appstats.html > >> > >> > >> > >> Robert > >> > >> > >> > >> On Wed, May 18, 2011 at 09:28, Francois MASUREL <[email protected]> > wrote: > >> > I have the problem on both MS and HR applications. It doesn't seem to > >> > make > >> > a difference. > >> > > >> > Francois > >> > > >> > > >> > > >> > On Wed, May 18, 2011 at 15:26, mooose <[email protected]> > >> > wrote: > >> >> > >> >> Btw: I am using Python and not Java as Francois. Looks like the > >> >> datastore > >> >> brings back all these DeallineExceededExceptions. > >> >> Switching to HR seems not easily done at the moment or are there any > >> >> bulk > >> >> helpers for the transfer? > >> >> Siegfried > >> >> > >> >> -- > >> >> You received this message because you are subscribed to the Google > >> >> Groups > >> >> "Google App Engine" group. > >> >> To post to this group, send email to > [email protected]. > >> >> To unsubscribe from this group, send email to > >> >> [email protected]. > >> >> For more options, visit this group at > >> >> http://groups.google.com/group/google-appengine?hl=en. > >> > > >> > -- > >> > You received this message because you are subscribed to the Google > >> > Groups > >> > "Google App Engine" group. > >> > To post to this group, send email to > [email protected]. > >> > To unsubscribe from this group, send email to > >> > [email protected]. > >> > For more options, visit this group at > >> > http://groups.google.com/group/google-appengine?hl=en. > >> > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]. > >> For more options, visit this group at > >> http://groups.google.com/group/google-appengine?hl=en. > >> > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group at > > http://groups.google.com/group/google-appengine?hl=en. > > > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
