Hi Peter, On Sat, Sep 12, 2009 at 12:33 AM, Peter Liu <[email protected]> wrote:
> > Thanks for explaining how api cpu usage is calculated. > > I haven't enable billing yet, so my quota is low, but every time I > invoke a background task that do batch commits, the api_cpu is in red, > warning me the quota might be reached. > The message you're seeing is, unfortunately, slightly misleading. In the past, there was a hard limit on how much CPU an individual request could use. That is no longer the case, so consider these messages to be warnings that your page is 'expensive', nothing more. -Nick Johnson > Most of the my service calls involve 1+ datastore call, that force the > api_cpu be 80%+ of all cpu. That means api_cpu quota is my bottleneck, > and I must improve/avoid api_cpu to be more scalable. > > Right now I am doing performance optimization, logging every api call > and try to eliminates them, and observe the following behavior and > wondering if others see the same. > > 1. The api_cpu of batch put is more sensitive to the number of > entities stored than the actual size of the payload. For example, > storing 10 small entities 1k each) might take as much api_cpu to store > a single 100k entity. > 2. Using batch put save more on the delay of the call, but doesn't > save much on cpu quota. > 3. 1-N owned relationship (JDO specific?) is expensive (I see query on > child (list of entities) when a put is done on the parent) > 4. With JDO, whenever it feels the need to do a query before a put, it > wraps the query+put with a transaction (begin and commit), these extra > 3 api calls take more api_cpu. > > These makes me to believe that, to optimize api_cpu, > > A. If child objects are small and will not be searched, they should be > modeled as a list of serializable items instead of a 1-n owned > relationship. This is aligned with #1 and #2 and avoid some #3 and #4. > > B. It saves api_cpu to stuff logically related entities into a wider > schema to avoid query and batch put. > > It will be very helpful if someone can provide a rough formula on how > api_cpu is consumed, given the number of entities, entity size, etc... > > If anyone here want to share optimization tips on api_cpu, please > advice. :) > > Thanks! > > On Sep 11, 11:17 am, "Nick Johnson (Google)" <[email protected]> > wrote: > > Hi Peter, > > > > On Fri, Sep 11, 2009 at 2:18 AM, Peter Liu <[email protected]> wrote: > > > > > I see similar results. The total api cpu time is much higher then > > > actual time spent on the actual call. I use a profiling api delegate > > > that logs every api call duration, and the total api cpu reported on > > > admin console is usually at least 2 times higher. > > > > This is because an API call may involve multiple machines. A batch put, > for > > example, can include operations on multiple tabletservers, which are all > > executed simultaneously. > > > > -Nick Johnson > > > > > > > > > > > > > On Sep 9, 2:05 pm, Lec <[email protected]> wrote: > > > > I cannot confirm what it was like previous to 1.2.5 release, but I > > > > think I am seeing something similar. Although with task queues my > > > > workers do data processing then store the data in datastore, i am NOT > > > > seeingcpuproblems with workers. I only see problems with the > > > > datastore in the main threads where i do not use workers. Did anyone > > > > have this kind of experience? > > > > > > L > > > > > > On Sep 9, 3:54 am, herbie <[email protected]> wrote: > > > > > > > It seems I'm not the only one. I was starting to think I was > imagining > > > > > it! Thanks for supporting this thread. > > > > > Would anyone fro Google like to comment? > > > > > > > On Sep 8, 11:01 pm, Robert Kluin <[email protected]> wrote: > > > > > > > > I have some code that is now using moreCPUas well. > > > > > > > > Previously: > > > > > > 1530cpu_ms 616api_cpu_ms > > > > > > 1404cpu_ms 995api_cpu_ms > > > > > > 1104cpu_ms 695api_cpu_ms > > > > > > > > Now: > > > > > > 4619cpu_ms 4133api_cpu_ms > > > > > > 4619cpu_ms 4133api_cpu_ms (yes, it is exactly the same) > > > > > > > > That is unchanged code. Same exact data. > > > > > > > > Robert > > > > > > > > On Tue, Sep 8, 2009 at 3:23 PM, bFlood <[email protected]> > wrote: > > > > > > > > > I just ran a batch delete using the Task queue. It grabs the > next > > > 50 > > > > > > > keys for a Kind and then calls db.delete(keys), so fairly > simple > > > > > > > stuff. here's some example results in the log: > > > > > > > > > 865ms 1032cpu_ms 952api_cpu_ms > > > > > > > 1058ms 1040cpu_ms 952api_cpu_ms > > > > > > > 947ms 49947cpu_ms 49869api_cpu_ms <--??? > > > > > > > 1425ms 1035cpu_ms 952api_cpu_ms > > > > > > > 1674ms 41181cpu_ms 41094api_cpu_ms > > > > > > > > > any thoughts? something seems wrong to me > > > > > > > > > cheers > > > > > > > brian > > > > > > > > > On Sep 8, 8:56 am, bFlood <[email protected]> wrote: > > > > > > > > ok, I was able to check over my code again and even with > rolling > > > back > > > > > > > > small changes, the largeCPUincreases are still there. at this > > > point, > > > > > > > > I have to agree with herbie's findings as well. It would be > nice > > > if > > > > > > > > Google could weigh in on this troubling issue > > > > > > > > > > cheers > > > > > > > > brian > > > > > > > > > > On Sep 8, 4:51 am, herbie <[email protected]> wrote: > > > > > > > > > > > On Sep 8, 12:07 am, Stephen <[email protected]> wrote: > > > > > > > > > > > > OK, but because api_cpu_ms is 96% of the total, then > cpu_ms > > > is also > > > > > > > > > > almost 3x higher? The spike is showing up in the cpu_ms? > > > > > > > > > > > Yes in total the cpu_ms has gone up by nearly 3x too. > > > > > > > > > > > But as I understand it cpu_ms is the totalcpuusage for the > > > request > > > > > > > > > and api_cpu_ms is thecpuusage by GAE api calls. So the > > > difference > > > > > > > > > between the two is thecpuusage of my non api code. This > > > difference > > > > > > > > > hasn’t increased because the code hasn’t changed. > > > > > > > > > > > But yes, the newhighvalue for api_cpu_ms directly affects > my > > > quota > > > > > > > > > because it makes the vast majority of cpu_ms. So we do pay > for > > > > > > > > > api_cpu_ms ! So for example if Google makes a change to > > > db.put() > > > > > > > > > (or any api call) so that it uses morecpu, we will be > billed > > > for > > > > > > > > > morecpuusage even if our code hasn’t changed. > > > > > > > > > > > As my code/ indexes hasn’t changed and the api_cpu_ms has > shot > > > up the > > > > > > > > > obvious conclusion is that an api/datastore change has > caused > > > it? > > > > > > > > > > > But there may be another ‘good’ reason for it, which I > can’t > > > think > > > > > > > > > of, but as I’m going to have to pay for the increase in > > > api_cpu_ms, > > > > > > > > > I would really appreciate it if someone at Google could > help. > > > > > > > > > > > On Sep 8, 12:07 am, Stephen <[email protected]> wrote: > > > > > > > > > > > > On Sep 7, 8:57 pm, herbie <[email protected]> wrote: > > > > > > > > > > > > > On Sep 7, 6:50 pm, Stephen <[email protected]> wrote: > > > > > > > > > > > > > > What about cpu_ms, is that also higher for requests > which > > > write > > > > > > > to the > > > > > > > > > > > > data store? > > > > > > > > > > > > > No, not in relation to api_cpu_ms. For the request > that > > > does the > > > > > > > most > > > > > > > > > > > writing to the datastore api_cpu_ms accounts for 96% of > the > > > total > > > > > > > > > > > cpu_ms value!. The so request handler does not much > more > > > than > > > > > > > create > > > > > > > > > > > new entities in the datastore. > > > > > > > > > > > > OK, but because api_cpu_ms is 96% of the total, then > cpu_ms > > > is also > > > > > > > > > > almost 3x higher? The spike is showing up in the cpu_ms? > > > > > > > > > > > > cpu_ms is billed for, so if you have billing enabled you > are > > > being > > > > > > > > > > overcharged. > > > > > > > > > > > > You could try asking for a refund here: > > > > >http://code.google.com/support/bin/request.py?contact_type=AppEngineB. > .. > > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > > -- Nick Johnson, Developer Programs Engineer, App Engine Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: 368047 --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
