Hi Greg. Nice write-up.
Robert On Thu, Aug 12, 2010 at 8:25 AM, Greg Tracy <[email protected]> wrote: > > Circling back on this thread to report on a couple of outcomes. > > First, from a numbers standpoint... I saw massive improvement when I > started doing batch puts. > - On average, a batch put of 472 relatively small entities takes 2,200 > megacycles > - On average, a batch put of 363 entities takes 1,900 megacycles > > Second, total CPU cycles for this job (of parsing a text file, > creating entities, and storing them) is a fraction of the API cycle > time. 3,000 v. 98,000 megacycles > > Also, I did a write-up on my experience with this analysis process. If > you're interested, check it out and share your thoughts and > experiences. > > http://www.gregtracy.com/adventures-in-performance-tuning-on-google-ap > > Now... how do I manage the API time?! > > > On Jul 21, 2:26 am, Robert Kluin <[email protected]> wrote: >> Hey Greg, >> Yeah puts are relatively expensive. Doing a batch put is about the only >> way to reduceCPUtime, but of course that does basically nothing forAPItime. >> >> I always like to see benchmarks and related discussions. Look forward to >> your results. >> >> Robert >> >> On Jul 20, 2010, at 13:51, Greg Tracy <[email protected]> wrote: >> >> >> >> >> >> > Thanks for the input, Robert. >> >> > After more performance tuning, I've been able to eliminate the >> > DeadlineExceeded errors. But I still have a hard time scaling the app >> > since the put calls are so expensive. Over thousands of request >> > handler calls, I quickly eat upCPUquota (free and paid). >> >> > Going back to the original question, I'm still unsure how the quota >> > math is working. I'm in the process of creating an-to-end performance >> > model of these calls and will post the results. >> >> > On Jul 15, 4:40 pm, Robert Kluin <[email protected]> wrote: >> >> Greg, >> >> If you are having issues with long write times, there are two common >> >> options you may consider: smaller batches and transactional tasks. >> >> >> I have some code that does batch processing, I set it up to search >> >> for the maximum batch size that is currently succeeding. Basically I >> >> slowly increase my batch size until I am getting deadline exceeded >> >> warnings then I drop back down adjust my 'growth factor' and repeat >> >> the processes. This seems to keep my batches running very close to >> >> the upper limit without too many deadline exceeded exceptions. >> >> >> The other approach I have used it to offload some of the writes to >> >> other background tasks. I write a group of entities, then send >> >> another handler another block of entities to write. This has been >> >> working well for me too. >> >> >> Robert >> >> >> On Thu, Jul 15, 2010 at 5:22 PM, Greg Tracy <[email protected]> wrote: >> >> >>> I just started using AppStats for the first time - very nice package! >> >>> However, it didn't reveal any glaring problems. But it did help >> >>> validate my belief that I was using the memcache effectively. >> >> >>> I've been using the 'quota' package to find costly operations, and I >> >>> did discover a bug in my measurement. I was "starting the clock" in >> >>> the wrong spot so I have found some new areas to optimize. Will work >> >>> on that now... but I'm still not confident I can compensate for the >> >>> long write times. >> >> >>> Thanks! >> >> >>> On Jul 14, 10:46 pm, Robert Kluin <[email protected]> wrote: >> >>>> If you have not already, take a look at AppStats. Perhaps you can >> >>>> find a way to improve performance. You may also want to read through >> >>>> some of the datastore articles in the docs to be sure you are >> >>>> minimizing resource usage. >> >> >>>> Robert >> >> >>>> On Wed, Jul 14, 2010 at 7:24 PM, Greg Tracy <[email protected]> wrote: >> >> >>>>> Not for me... I set a budget and immediately started paying. >> >> >>>>> Not thrilled about this and may need to start a new thread asking for >> >>>>> advice on datastore contention... >> >> >>>>> Thanks. >> >> >>>>> On Jul 14, 5:57 pm, Nate Bauernfeind <[email protected]> >> >>>>> wrote: >> >>>>>> I have been. >> >> >>>>>> On Wed, Jul 14, 2010 at 5:37 PM, Greg Tracy <[email protected]> wrote: >> >>>>>>> Does this imply that I can go over theCPUTime quota and not pay for >> >>>>>>> it? >> >> >>>>>>> On Jul 14, 5:26 pm, Nate Bauernfeind <[email protected]> >> >>>>>>> wrote: >> >>>>>>>> The DataStoreCPUtime is included in totalCPUtime. To take advantage >> >>>>>>> of >> >>>>>>>> "free datastorecputime" you need to increase yourcpu-usage quota. >> >>>>>>>> When >> >>>>>>> I >> >>>>>>>> loaded a bunch of initial test data into the datastore I got tons of >> >>>>>>>> deadline exceeded errors, though I assumed it was cause I was >> >>>>>>>> trying to >> >>>>>>>> stuff a lot of data in there all at once. >> >> >>>>>>>> Nate >> >> >>>>>>>> On Wed, Jul 14, 2010 at 5:24 PM, Greg Tracy <[email protected]> >> >>>>>>>> wrote: >> >> >>>>>>>>> I've been adding some new datastore-intensive features to an app >> >>>>>>>>> and >> >>>>>>>>> am blowing through the "CPUTime" quota. While my understanding is >> >>>>>>>>> that the "DatastoreCPUTime" quotas have been lifted while the App >> >>>>>>>>> Engine team continues to work on the performance, I can't figure >> >>>>>>>>> out >> >>>>>>>>> why these are mutually exclusive metrics in my app. >> >> >>>>>>>>> When I measure thecpucyclesbeing consumed in the new features, it >> >>>>>>>>> is almost exclusively in the db.put() call. In fact, 50% of the >> >>>>>>>>> time, >> >>>>>>>>> I'm getting DeadlineExceeded errors before the put call even >> >>>>>>>>> returns. >> >> >>>>>>>>> Is the "DatastoreCPUTime" also counted in the "CPUTime"? If so, are >> >>>>>>>>> there plans to extend the quota limits there as well? >> >> >>>>>>>>> I'm interested in finding some resources that track performance >> >>>>>>>>> metrics on App Engine. I don't know what to expect in terms of >> >>>>>>>>> performance when I store, for example, 400 entities in the data >> >>>>>>>>> store. >> >>>>>>>>> Are there resources where folks are contributing their metrics for >> >>>>>>>>> others to see? >> >> >>>>>>>>> Thanks. >> >> >>>>>>>>> -- >> >>>>>>>>> You received this message because you are subscribed to the Google >> >>>>>>> Groups >> >>>>>>>>> "Google App Engine" group. >> >>>>>>>>> To post to this group, send email to >> >>>>>>>>> [email protected] >> >>>>>>> . >> >>>>>>>>> To unsubscribe from this group, send email to >> >>>>>>>>> [email protected]<google-appengine%2Bunsubscrib >> >>>>>>>>> [email protected]><google-appengine%2Bunsubscrib >> >>>>>>> [email protected]> >> >>>>>>>>> . >> >>>>>>>>> For more options, visit this group at >> >>>>>>>>>http://groups.google.com/group/google-appengine?hl=en. >> >> >>>>>>> -- >> >>>>>>> You received this message because you are subscribed to the Google >> >>>>>>> Groups >> >>>>>>> "Google App Engine" group. >> >>>>>>> To post to this group, send email to >> >>>>>>> [email protected]. >> >>>>>>> To unsubscribe from this group, send email to >> >>>>>>> [email protected]<google-appengine%2Bunsubscrib >> >>>>>>> [email protected]> >> >>>>>>> . >> >>>>>>> For more options, visit this group at >> >>>>>>>http://groups.google.com/group/google-appengine?hl=en. >> >> >>>>> -- >> >>>>> You received this message because you are subscribed to the Google >> >>>>> Groups "Google App Engine" group. >> >>>>> To post to this group, send email to [email protected]. >> >>>>> To unsubscribe from this group, send email to >> >>>>> [email protected]. >> >>>>> For more options, visit this group >> >>>>> athttp://groups.google.com/group/google-appengine?hl=en. >> >> >>> -- >> >>> You received this message because you are subscribed to the Google >> >>> Groups "Google App Engine" group. >> >>> To post to this group, send email to [email protected]. >> >>> To unsubscribe from this group, send email to >> >>> [email protected]. >> >>> For more options, visit this group >> >>> athttp://groups.google.com/group/google-appengine?hl=en. >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
