Hi Matija, Glad that we could be of help in simplifying your app design (I recall our conversation on IRC). It does seem like some misunderstandings have cropped up about CPU utilization, measurement, etc. and we've tried to make the CPU usage measurements more clear.
There were a couple of points I thought I should chime in on below, to try to reduce confusion for those that are new to these design points. On Dec 18, 3:58 am, Matija <[email protected]> wrote: > This morning I did some final testing and because app engine team > introduced 'quota details' page with some more explanation > inhttp://code.google.com/appengine/docs/quotas.htmllink now I am > thinking that I created a little bit more complicated code than I > needed. Just because they introduced suggestion warnings in admin > console log with yellow and red colors plus some warning sign from the > beginning. > > I am using app engine last half year and I have read almost every > article, watched every interesting video and constantly read app > engine group discussions. In the last half year every now and then > they explained more and more about high cpu quota, difference between > api cpu and our code cpu usage and difference in its presentation over > the admin console. > > My final test was simple: > > class Test(webapp.RequestHandler): > > def get(self): > for i in range(200): > t = Testna(podatak = str(i)) > t.put() > > self.response.out.write('radi') It's great that you ran a test, but I think the above may be too simple to accurately measure performance for many people's apps (might be accurate for the simple counter case). The inserted entities here are very simple. Things like the size and structure of the data, number and complexity of indexes, entity group relationships, etc. can all effect the datastore CPU usage, response time, and/or likelyhood of contention. I'd go as far as to say it is impossible to make a blanket statement about how many writes can be performed per request without understanding the "shape" of the data. > > Results were like some religious revelation for me: > > http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9...http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9... > > Although they stated over and over (not at the beginning but lately) > these results, I needed proof. And now I have it. > > I will not be punished (in quota sense) for more than one datastore > write operation. During my test I had no high cpu quota usage. There > was some warning but I think that they are there when we are near 8s > limit per request, and when we break this limit, but even then this > was not counted as high cpu quota. The picture of the logs which you posted did indicate that the request was using too much runtime CPU ("This request used a high amount of CPU, and was roughly 2.9 times over the average request CPU limit...."). At the moment there is a limit of two high CPU requests per minute, and you can bank up to sixty "credits" for high runtime CPU. If you had run this test continually I suspect you would have begun to see quota denials. (See http://code.google.com/appengine/kb/general.html#highcpu for more information.) > > I understand contention problem and even user experience with too slow > response, but I will not be punished, temporary baned or something > like that. Contention will actually effect more than just the response time. Failed transactions are automatically retried, consuming datastore CPU, so this is another limit which you could hit. Eventually, a transaction or write may fail (and raise an exception) after unsuccessful retries. > > This is so important for me because I mostly need more write operation > only for administrative purpose. Not 200 writes but let's say 10 or > 20. With bigtable system we are creating super redundant datastore, > but when I need to correct (for some reason) that data, now I know > that I don't have to do this in thousand 'one write' operation step, > than in a 50 (or less) '20 write' operation request. If I do this when > there is low usage (night for example) I could even bypass some > contention problem. In general, the idea is to minimize the number of concurrent writes to a particular entity or entity group. I wasn't completely sure about what you meant by low usage in the above. Low usage of your app by users? > > Second reason was lack of model count ability. So when I needed to use > additional counter model with shard principle for avoiding contention > problem. But I needed two write operation. In my fear of 'punishment' > i did this in two connected ajax requests. If first request (write > operation) succeeded I will send second request to increment count. > With this approach I was under 1000 mcycles cpu usage (with datastore > api), but this also introduced new problems. What if second request > has not succeeded. What now. > > Now I have wrong count data. With this more complicated approach I had > problem and even I have thought that app engine could not be used for > serious purpose where this discrepancy in data is not an option. Last > five years I was lead developer for hospital information system so you > can understand my fear for error data. Counting comments on web page, > poll data or something like data can have 99% precision but something > else can't. Of course I am not developing now something serious like > hospital information system but it takes time to switch from 100% > desktop application data to 99% web application data. > > And now I have found out that this was not necessary and that I can > have 100% data precision in connection with user perception of > succeeded or not succeeded operation. I can have two, four, six or > more write operation in one request only if I stay under 8(or 9?) > seconds time request limit because contention can impact only on > request time limit. > > All these high cpu fuss (in connection with write operations, of > course) were just design warnings and not some forcing statement. While in some cases these warnings do not indicate that your app will be immediately blocked on quotas, they can be an indicator of a design problem. The CPU threshold of 1000 megacycles applies to runtime CPU, so the colored warning numbers which include API CPU usage do not always need to be under 1000, but the combined numbers are an indicator of the overall CPU utilization in a request. Happy coding, Jeff > > Or am I wrong again ? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
