Hi Jeff On Dec 18, 11:33 pm, Jeff S <[email protected]> wrote: > Hi Matija, > > Glad that we could be of help in simplifying your app design (I recall > our conversation on IRC). It does seem like some misunderstandings > have cropped up about CPU utilization, measurement, etc. and we've > tried to make the CPU usage measurements more clear. > > There were a couple of points I thought I should chime in on below, to > try to reduce confusion for those that are new to these design points. > > On Dec 18, 3:58 am, Matija <[email protected]> wrote: > > > > > This morning I did some final testing and because app engine team > > introduced 'quota details' page with some more explanation > > inhttp://code.google.com/appengine/docs/quotas.htmllinknow I am > > thinking that I created a little bit more complicated code than I > > needed. Just because they introduced suggestion warnings in admin > > console log with yellow and red colors plus some warning sign from the > > beginning. > > > I am using app engine last half year and I have read almost every > > article, watched every interesting video and constantly read app > > engine group discussions. In the last half year every now and then > > they explained more and more about high cpu quota, difference between > > api cpu and our code cpu usage and difference in its presentation over > > the admin console. > > > My final test was simple: > > > class Test(webapp.RequestHandler): > > > def get(self): > > for i in range(200): > > t = Testna(podatak = str(i)) > > t.put() > > > self.response.out.write('radi') > > It's great that you ran a test, but I think the above may be too > simple to accurately measure performance for many people's apps (might > be accurate for the simple counter case). The inserted entities here > are very simple. Things like the size and structure of the data, > number and complexity of indexes, entity group relationships, etc. can > all effect the datastore CPU usage, response time, and/or likelyhood > of contention. I'd go as far as to say it is impossible to make a > blanket statement about how many writes can be performed per request > without understanding the "shape" of the data.
I understand connection between size and structure of the data, indexes, etc. with overall CPU usage. Actually I did start test with one write, two, four, eight, 100 and finally with 12 times 200 write operations. Simple one write operation (no indexes, low size table etc.) cost around 470 ms-cpu. Two 940 ms-cpu, 4 1880 ms-cpu, 8 3750 ms- cpu, 100 23400 ms-cpu and 200 write operation request 46600 ms-cpu. And as I noticed under 10 write operations there was proportional cpu time with write operation number at about 470 ms per write operation. With more data in table, few indexes and of course some more code than a simple datastore api we couldn't have more than one datastore write operation per request to stay under 1000 ms-cpu limit, if datastore api isn't directly excluded from high cpu quota. Nobody should have too many write operation because multiuser environment (which web applications are) have extreme contention problem and actually great scalability solutions are in a way contention bypass solutions. I am sure that somebodies one write operation request could have big contention problem in a poorly designed system, not to mention how it would be in more write operations request. If this is not true why do we even bother with shard counter principle! > > Results were like some religious revelation for me: > > >http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9...... > > > Although they stated over and over (not at the beginning but lately) > > these results, I needed proof. And now I have it. > > > I will not be punished (in quota sense) for more than one datastore > > write operation. During my test I had no high cpu quota usage. There > > was some warning but I think that they are there when we are near 8s > > limit per request, and when we break this limit, but even then this > > was not counted as high cpu quota. > > The picture of the logs which you posted did indicate that the request > was using too much runtime CPU ("This request used a high amount of > CPU, and was roughly 2.9 times over the average request CPU > limit...."). At the moment there is a limit of two high CPU requests > per minute, and you can bank up to sixty "credits" for high runtime > CPU. If you had run this test continually I suspect you would have > begun to see quota denials. > (Seehttp://code.google.com/appengine/kb/general.html#highcpu > for more information.) Actually I think that here is something else involved. I did 12 test with 200 write operation request. Six were with no warning message, one with warning message(..was roughly 1.1 times over..) and five were DeadlineExceededError error (i did some parallel request to test contention in combination with near request time limit). Two out of this five had also warning message (...roughly 2.9 times over..., ...roughly 1.0 times over...). But during this test I never saw any high cpu quota usage in quota details (other quota data did change). And here I went to do more tests. 23 requests with 200 write operation. http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281440712941963666 Indeed, when ever I noticed 'This request used a high amount of CPU...' message there was high cpu quota usage. I didn't noticed first time because high cpu quota is replenished pretty fast for one web site user. 11 request out of this 23 have had high cpu warning (...roughly 1.0 times...). Big question is why other 12 request didn't. This time I didn't get any DeadlineExceededError error. Then I did 20 requests with 100 write operation. There was two high cpu usage warning and even one 'InternalError: internal error' (sorry for that, but that message did bring me memory on ms sql server :)). And finally I did 100 requests with 10 write operation. Ctrl+tab, Ctrl +t, ctrl+v, enter and F5 can do miracle in no time. No high cpu usage warning at all. Funny but now I noticed also that overall cpu time per write operation is now 235 ms. Is this result of some positive scalability growth or simple small datastore usage during USA night time. Interesting time zone difference impact between USA and Europe day time. Conclusion is that I can be safe with more than one datastore write operation request, but I need to take care for possible contention situations, that can evolve even in one datastore write operation requests. Tnx, Matija. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
