Greg, If you are having issues with long write times, there are two common options you may consider: smaller batches and transactional tasks.
I have some code that does batch processing, I set it up to search for the maximum batch size that is currently succeeding. Basically I slowly increase my batch size until I am getting deadline exceeded warnings then I drop back down adjust my 'growth factor' and repeat the processes. This seems to keep my batches running very close to the upper limit without too many deadline exceeded exceptions. The other approach I have used it to offload some of the writes to other background tasks. I write a group of entities, then send another handler another block of entities to write. This has been working well for me too. Robert On Thu, Jul 15, 2010 at 5:22 PM, Greg Tracy <[email protected]> wrote: > > I just started using AppStats for the first time - very nice package! > However, it didn't reveal any glaring problems. But it did help > validate my belief that I was using the memcache effectively. > > I've been using the 'quota' package to find costly operations, and I > did discover a bug in my measurement. I was "starting the clock" in > the wrong spot so I have found some new areas to optimize. Will work > on that now... but I'm still not confident I can compensate for the > long write times. > > Thanks! > > > On Jul 14, 10:46 pm, Robert Kluin <[email protected]> wrote: >> If you have not already, take a look at AppStats. Perhaps you can >> find a way to improve performance. You may also want to read through >> some of the datastore articles in the docs to be sure you are >> minimizing resource usage. >> >> Robert >> >> >> >> On Wed, Jul 14, 2010 at 7:24 PM, Greg Tracy <[email protected]> wrote: >> >> > Not for me... I set a budget and immediately started paying. >> >> > Not thrilled about this and may need to start a new thread asking for >> > advice on datastore contention... >> >> > Thanks. >> >> > On Jul 14, 5:57 pm, Nate Bauernfeind <[email protected]> >> > wrote: >> >> I have been. >> >> >> On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy <[email protected]> wrote: >> >> > Does this imply that I can go over the CPU Time quota and not pay for >> >> > it? >> >> >> > On Jul 14, 5:26 pm, Nate Bauernfeind <[email protected]> >> >> > wrote: >> >> > > The DataStore CPU time is included in total CPU time. To take >> >> > > advantage >> >> > of >> >> > > "free datastore cpu time" you need to increase your cpu-usage quota. >> >> > > When >> >> > I >> >> > > loaded a bunch of initial test data into the datastore I got tons of >> >> > > deadline exceeded errors, though I assumed it was cause I was trying >> >> > > to >> >> > > stuff a lot of data in there all at once. >> >> >> > > Nate >> >> >> > > On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy <[email protected]> wrote: >> >> >> > > > I've been adding some new datastore-intensive features to an app and >> >> > > > am blowing through the "CPU Time" quota. While my understanding is >> >> > > > that the "Datastore CPU Time" quotas have been lifted while the App >> >> > > > Engine team continues to work on the performance, I can't figure out >> >> > > > why these are mutually exclusive metrics in my app. >> >> >> > > > When I measure the cpu cycles being consumed in the new features, it >> >> > > > is almost exclusively in the db.put() call. In fact, 50% of the >> >> > > > time, >> >> > > > I'm getting DeadlineExceeded errors before the put call even >> >> > > > returns. >> >> >> > > > Is the "Datastore CPU Time" also counted in the "CPU Time"? If so, >> >> > > > are >> >> > > > there plans to extend the quota limits there as well? >> >> >> > > > I'm interested in finding some resources that track performance >> >> > > > metrics on App Engine. I don't know what to expect in terms of >> >> > > > performance when I store, for example, 400 entities in the data >> >> > > > store. >> >> > > > Are there resources where folks are contributing their metrics for >> >> > > > others to see? >> >> >> > > > Thanks. >> >> >> > > > -- >> >> > > > You received this message because you are subscribed to the Google >> >> > Groups >> >> > > > "Google App Engine" group. >> >> > > > To post to this group, send email to >> >> > > > [email protected] >> >> > . >> >> > > > To unsubscribe from this group, send email to >> >> > > > [email protected]<google-appengine%2Bunsubscrib >> >> > > > [email protected]><google-appengine%2Bunsubscrib >> >> > [email protected]> >> >> > > > . >> >> > > > For more options, visit this group at >> >> > > >http://groups.google.com/group/google-appengine?hl=en. >> >> >> > -- >> >> > You received this message because you are subscribed to the Google >> >> > Groups >> >> > "Google App Engine" group. >> >> > To post to this group, send email to [email protected]. >> >> > To unsubscribe from this group, send email to >> >> > [email protected]<google-appengine%2Bunsubscrib >> >> > [email protected]> >> >> > . >> >> > For more options, visit this group at >> >> >http://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
