Circling back on this thread to report on a couple of outcomes. First, from a numbers standpoint... I saw massive improvement when I started doing batch puts. - On average, a batch put of 472 relatively small entities takes 2,200 megacycles - On average, a batch put of 363 entities takes 1,900 megacycles
Second, total CPU cycles for this job (of parsing a text file, creating entities, and storing them) is a fraction of the API cycle time. 3,000 v. 98,000 megacycles Also, I did a write-up on my experience with this analysis process. If you're interested, check it out and share your thoughts and experiences. http://www.gregtracy.com/adventures-in-performance-tuning-on-google-ap Now... how do I manage the API time?! On Jul 21, 2:26 am, Robert Kluin <[email protected]> wrote: > Hey Greg, > Yeah puts are relatively expensive. Doing a batch put is about the only way > to reduceCPUtime, but of course that does basically nothing forAPItime. > > I always like to see benchmarks and related discussions. Look forward to > your results. > > Robert > > On Jul 20, 2010, at 13:51, Greg Tracy <[email protected]> wrote: > > > > > > > Thanks for the input, Robert. > > > After more performance tuning, I've been able to eliminate the > > DeadlineExceeded errors. But I still have a hard time scaling the app > > since the put calls are so expensive. Over thousands of request > > handler calls, I quickly eat upCPUquota (free and paid). > > > Going back to the original question, I'm still unsure how the quota > > math is working. I'm in the process of creating an-to-end performance > > model of these calls and will post the results. > > > On Jul 15, 4:40 pm, Robert Kluin <[email protected]> wrote: > >> Greg, > >> If you are having issues with long write times, there are two common > >> options you may consider: smaller batches and transactional tasks. > > >> I have some code that does batch processing, I set it up to search > >> for the maximum batch size that is currently succeeding. Basically I > >> slowly increase my batch size until I am getting deadline exceeded > >> warnings then I drop back down adjust my 'growth factor' and repeat > >> the processes. This seems to keep my batches running very close to > >> the upper limit without too many deadline exceeded exceptions. > > >> The other approach I have used it to offload some of the writes to > >> other background tasks. I write a group of entities, then send > >> another handler another block of entities to write. This has been > >> working well for me too. > > >> Robert > > >> On Thu, Jul 15, 2010 at 5:22 PM, Greg Tracy <[email protected]> wrote: > > >>> I just started using AppStats for the first time - very nice package! > >>> However, it didn't reveal any glaring problems. But it did help > >>> validate my belief that I was using the memcache effectively. > > >>> I've been using the 'quota' package to find costly operations, and I > >>> did discover a bug in my measurement. I was "starting the clock" in > >>> the wrong spot so I have found some new areas to optimize. Will work > >>> on that now... but I'm still not confident I can compensate for the > >>> long write times. > > >>> Thanks! > > >>> On Jul 14, 10:46 pm, Robert Kluin <[email protected]> wrote: > >>>> If you have not already, take a look at AppStats. Perhaps you can > >>>> find a way to improve performance. You may also want to read through > >>>> some of the datastore articles in the docs to be sure you are > >>>> minimizing resource usage. > > >>>> Robert > > >>>> On Wed, Jul 14, 2010 at 7:24 PM, Greg Tracy <[email protected]> wrote: > > >>>>> Not for me... I set a budget and immediately started paying. > > >>>>> Not thrilled about this and may need to start a new thread asking for > >>>>> advice on datastore contention... > > >>>>> Thanks. > > >>>>> On Jul 14, 5:57 pm, Nate Bauernfeind <[email protected]> > >>>>> wrote: > >>>>>> I have been. > > >>>>>> On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy <[email protected]> wrote: > >>>>>>> Does this imply that I can go over theCPUTime quota and not pay for > >>>>>>> it? > > >>>>>>> On Jul 14, 5:26 pm, Nate Bauernfeind <[email protected]> > >>>>>>> wrote: > >>>>>>>> The DataStoreCPUtime is included in totalCPUtime. To take advantage > >>>>>>> of > >>>>>>>> "free datastorecputime" you need to increase yourcpu-usage quota. > >>>>>>>> When > >>>>>>> I > >>>>>>>> loaded a bunch of initial test data into the datastore I got tons of > >>>>>>>> deadline exceeded errors, though I assumed it was cause I was trying > >>>>>>>> to > >>>>>>>> stuff a lot of data in there all at once. > > >>>>>>>> Nate > > >>>>>>>> On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy <[email protected]> wrote: > > >>>>>>>>> I've been adding some new datastore-intensive features to an app and > >>>>>>>>> am blowing through the "CPUTime" quota. While my understanding is > >>>>>>>>> that the "DatastoreCPUTime" quotas have been lifted while the App > >>>>>>>>> Engine team continues to work on the performance, I can't figure out > >>>>>>>>> why these are mutually exclusive metrics in my app. > > >>>>>>>>> When I measure thecpucyclesbeing consumed in the new features, it > >>>>>>>>> is almost exclusively in the db.put() call. In fact, 50% of the > >>>>>>>>> time, > >>>>>>>>> I'm getting DeadlineExceeded errors before the put call even > >>>>>>>>> returns. > > >>>>>>>>> Is the "DatastoreCPUTime" also counted in the "CPUTime"? If so, are > >>>>>>>>> there plans to extend the quota limits there as well? > > >>>>>>>>> I'm interested in finding some resources that track performance > >>>>>>>>> metrics on App Engine. I don't know what to expect in terms of > >>>>>>>>> performance when I store, for example, 400 entities in the data > >>>>>>>>> store. > >>>>>>>>> Are there resources where folks are contributing their metrics for > >>>>>>>>> others to see? > > >>>>>>>>> Thanks. > > >>>>>>>>> -- > >>>>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups > >>>>>>>>> "Google App Engine" group. > >>>>>>>>> To post to this group, send email to > >>>>>>>>> [email protected] > >>>>>>> . > >>>>>>>>> To unsubscribe from this group, send email to > >>>>>>>>> [email protected]<google-appengine%2Bunsubscrib > >>>>>>>>> [email protected]><google-appengine%2Bunsubscrib > >>>>>>> [email protected]> > >>>>>>>>> . > >>>>>>>>> For more options, visit this group at > >>>>>>>>>http://groups.google.com/group/google-appengine?hl=en. > > >>>>>>> -- > >>>>>>> You received this message because you are subscribed to the Google > >>>>>>> Groups > >>>>>>> "Google App Engine" group. > >>>>>>> To post to this group, send email to > >>>>>>> [email protected]. > >>>>>>> To unsubscribe from this group, send email to > >>>>>>> [email protected]<google-appengine%2Bunsubscrib > >>>>>>> [email protected]> > >>>>>>> . > >>>>>>> For more options, visit this group at > >>>>>>>http://groups.google.com/group/google-appengine?hl=en. > > >>>>> -- > >>>>> You received this message because you are subscribed to the Google > >>>>> Groups "Google App Engine" group. > >>>>> To post to this group, send email to [email protected]. > >>>>> To unsubscribe from this group, send email to > >>>>> [email protected]. > >>>>> For more options, visit this group > >>>>> athttp://groups.google.com/group/google-appengine?hl=en. > > >>> -- > >>> You received this message because you are subscribed to the Google Groups > >>> "Google App Engine" group. > >>> To post to this group, send email to [email protected]. > >>> To unsubscribe from this group, send email to > >>> [email protected]. > >>> For more options, visit this group > >>> athttp://groups.google.com/group/google-appengine?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
