Hey Greg, Yeah puts are relatively expensive. Doing a batch put is about the only way to reduce CPU time, but of course that does basically nothing for API time.
I always like to see benchmarks and related discussions. Look forward to your results. Robert On Jul 20, 2010, at 13:51, Greg Tracy <[email protected]> wrote: > > Thanks for the input, Robert. > > After more performance tuning, I've been able to eliminate the > DeadlineExceeded errors. But I still have a hard time scaling the app > since the put calls are so expensive. Over thousands of request > handler calls, I quickly eat up CPU quota (free and paid). > > Going back to the original question, I'm still unsure how the quota > math is working. I'm in the process of creating an-to-end performance > model of these calls and will post the results. > > > > On Jul 15, 4:40 pm, Robert Kluin <[email protected]> wrote: >> Greg, >> If you are having issues with long write times, there are two common >> options you may consider: smaller batches and transactional tasks. >> >> I have some code that does batch processing, I set it up to search >> for the maximum batch size that is currently succeeding. Basically I >> slowly increase my batch size until I am getting deadline exceeded >> warnings then I drop back down adjust my 'growth factor' and repeat >> the processes. This seems to keep my batches running very close to >> the upper limit without too many deadline exceeded exceptions. >> >> The other approach I have used it to offload some of the writes to >> other background tasks. I write a group of entities, then send >> another handler another block of entities to write. This has been >> working well for me too. >> >> Robert >> >> >> >> On Thu, Jul 15, 2010 at 5:22 PM, Greg Tracy <[email protected]> wrote: >> >>> I just started using AppStats for the first time - very nice package! >>> However, it didn't reveal any glaring problems. But it did help >>> validate my belief that I was using the memcache effectively. >> >>> I've been using the 'quota' package to find costly operations, and I >>> did discover a bug in my measurement. I was "starting the clock" in >>> the wrong spot so I have found some new areas to optimize. Will work >>> on that now... but I'm still not confident I can compensate for the >>> long write times. >> >>> Thanks! >> >>> On Jul 14, 10:46 pm, Robert Kluin <[email protected]> wrote: >>>> If you have not already, take a look at AppStats. Perhaps you can >>>> find a way to improve performance. You may also want to read through >>>> some of the datastore articles in the docs to be sure you are >>>> minimizing resource usage. >> >>>> Robert >> >>>> On Wed, Jul 14, 2010 at 7:24 PM, Greg Tracy <[email protected]> wrote: >> >>>>> Not for me... I set a budget and immediately started paying. >> >>>>> Not thrilled about this and may need to start a new thread asking for >>>>> advice on datastore contention... >> >>>>> Thanks. >> >>>>> On Jul 14, 5:57 pm, Nate Bauernfeind <[email protected]> >>>>> wrote: >>>>>> I have been. >> >>>>>> On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy <[email protected]> wrote: >>>>>>> Does this imply that I can go over the CPU Time quota and not pay for >>>>>>> it? >> >>>>>>> On Jul 14, 5:26 pm, Nate Bauernfeind <[email protected]> >>>>>>> wrote: >>>>>>>> The DataStore CPU time is included in total CPU time. To take advantage >>>>>>> of >>>>>>>> "free datastore cpu time" you need to increase your cpu-usage quota. >>>>>>>> When >>>>>>> I >>>>>>>> loaded a bunch of initial test data into the datastore I got tons of >>>>>>>> deadline exceeded errors, though I assumed it was cause I was trying to >>>>>>>> stuff a lot of data in there all at once. >> >>>>>>>> Nate >> >>>>>>>> On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy <[email protected]> wrote: >> >>>>>>>>> I've been adding some new datastore-intensive features to an app and >>>>>>>>> am blowing through the "CPU Time" quota. While my understanding is >>>>>>>>> that the "Datastore CPU Time" quotas have been lifted while the App >>>>>>>>> Engine team continues to work on the performance, I can't figure out >>>>>>>>> why these are mutually exclusive metrics in my app. >> >>>>>>>>> When I measure the cpu cycles being consumed in the new features, it >>>>>>>>> is almost exclusively in the db.put() call. In fact, 50% of the time, >>>>>>>>> I'm getting DeadlineExceeded errors before the put call even returns. >> >>>>>>>>> Is the "Datastore CPU Time" also counted in the "CPU Time"? If so, are >>>>>>>>> there plans to extend the quota limits there as well? >> >>>>>>>>> I'm interested in finding some resources that track performance >>>>>>>>> metrics on App Engine. I don't know what to expect in terms of >>>>>>>>> performance when I store, for example, 400 entities in the data store. >>>>>>>>> Are there resources where folks are contributing their metrics for >>>>>>>>> others to see? >> >>>>>>>>> Thanks. >> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>>>> "Google App Engine" group. >>>>>>>>> To post to this group, send email to [email protected] >>>>>>> . >>>>>>>>> To unsubscribe from this group, send email to >>>>>>>>> [email protected]<google-appengine%2Bunsubscrib >>>>>>>>> [email protected]><google-appengine%2Bunsubscrib >>>>>>> [email protected]> >>>>>>>>> . >>>>>>>>> For more options, visit this group at >>>>>>>>> http://groups.google.com/group/google-appengine?hl=en. >> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups >>>>>>> "Google App Engine" group. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> To unsubscribe from this group, send email to >>>>>>> [email protected]<google-appengine%2Bunsubscrib >>>>>>> [email protected]> >>>>>>> . >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/google-appengine?hl=en. >> >>>>> -- >>>>> You received this message because you are subscribed to the Google Groups >>>>> "Google App Engine" group. >>>>> To post to this group, send email to [email protected]. >>>>> To unsubscribe from this group, send email to >>>>> [email protected]. >>>>> For more options, visit this group >>>>> athttp://groups.google.com/group/google-appengine?hl=en. >> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Google App Engine" group. >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group >>> athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
