On Wed, Oct 5, 2011 at 2:27 AM, Rishi Arora <[email protected]>wrote:
> I don't think backend CPU time counts against your front-end instance CPU > hours quota. Backends are purely billed based on uptime - and I think this > is true for both current and new pricing (starting November 1). But you > mentioned something about datastore CPU? It is likely that the most of your > billing is because of this. That's correct - datastore operations are currently billed by charging for CPU time. If you do any datastore operations on a backend, they'll show up as regular CPU hours. -Nick Johnson > > Few more questions: Is the reason for 20 backend instances that you want > to execute all your 20 deferred tasks parallely? You mentioned you have > ~4500 tasks. How long does each one take, and how often does each one need > to execute in a day? Lets assume that neither of these tasks are sensitive > to latency, and can be executed at any time during the entire day. If each > task takes 10 seconds on average, and needs to execute, for example, 6 times > a day... that's a total of 4500 * 10 * 6 / 3600 = 75 instance hours. You > should try to "schedule" your backends yourself so that you only pay for 75 > instance hours. > > If you allow 20 instances to get created, then at some point all these 20 > instances will complete their work, and then idle for 15 minutes (or at > least billed for 15 minutes of idle time after the last task completes > processing), before they're shutdown. You'll be wasting 5 instance hours > each time this happens. I think your focus should be to minimize your > instance hours by minimizing the number of parallel instances you allow > running. In my calculation above you only need a total of 75 instance > hours, and so you should set "instances" to 3 or 4 in backends.yaml. > > A yet another way of doing this in a more controlled fashion is by using > "pull" queues instead of enqueueing tasks on the regular "push" type task > queues. You can enqueue all your 4500 tasks on a single "pull" queue, and > all your backends will constantly run pulling tasks out of pull queues and > executing them, until the pull queue is empty. Then they can be woken up > again by a cron job to go check the pull queue again. > > Lastly, any cost you're incurring because of Datastore CPU hours in the > current pricing model, or because of Datastore writes in the new pricing > model - those can't be avoided. You will incur those costs regardless of > the context of execution of your tasks - front-end or backend. > > Hope this helps. > > > On Tue, Oct 4, 2011 at 9:33 AM, someone1 <[email protected]> wrote: > >> Hello, >> >> Thank you for the replies. >> >> The tasks are indeed being run on my backend as I am being charged for >> backend usage (I don't use backends in any other way otherwise). I >> also see the _ah/deferred logs on my backend ID but not on my app. The >> CPU time being used seems to correspond to the Datastore CPU time, are >> the two currently linked? Even under the new pricing scheme, would I >> be billed for my main app up-time when executing deferred tasks on my >> backend? What part of my main app is considered under use when >> executing tasks on the backend? >> >> The max number of instances allowed on any backend is 20. My backend >> is setup as dynamic B1 with 20 instances. I let Google determine how >> many instances needed to be up and running when I queue up my tasks >> (usually all 20 run for 20-30 minutes each). >> >> Again, I'd just like to know what part of my main app is being used >> during the backend operation that is eating up my CPU. I really >> haven't coded anything else on my app except for this data mining >> portion which should all be run on the backend now. >> >> Thanks, >> Prateek >> >> On Oct 4, 8:38 am, Rishi Arora <[email protected]> wrote: >> > I think deferred tasks is an excellent use case for backends. That's >> how I >> > use my backend as well. Can you confirm from your logs that your tasks >> are >> > indeed being processed on the backend? In the drop down for app >> versions, >> > there's a special "version" which is named after your backend. Select >> that >> > to check your logs specific to the backend. Also, I'm assuming the >> reason >> > you're blowing through your budget is because you're spanning out >> multiple, >> > possibly hundreds of instances. Can you find out how many instances get >> > spawned for your deferred tasks? Can you find out how many backend >> > instances are being spawned, if the backend is indeed being used for >> your >> > tasks? Finally, when you configured your backend, what did you set as >> your >> > "instances" parameter in backends.yaml? I don't know what the default >> is, >> > but it is likely "unlimited". In your case, a instance of 1 or 2 sounds >> > sufficient, but you'll have to play around with that, based on how much >> > queueing occurs for your tasks. >> > >> > >> > >> > >> > >> > >> > >> > On Tue, Oct 4, 2011 at 12:06 AM, Gerald Tan <[email protected]> >> wrote: >> > > I believe CPU time will no longer be billable after the new pricing is >> out >> > >> > > -- >> > > You received this message because you are subscribed to the Google >> Groups >> > > "Google App Engine" group. >> > > To view this discussion on the web visit >> > >https://groups.google.com/d/msg/google-appengine/-/Crry-7yTG4QJ. >> > >> > > To post to this group, send email to >> [email protected]. >> > > To unsubscribe from this group, send email to >> > > [email protected]. >> > > For more options, visit this group at >> > >http://groups.google.com/group/google-appengine?hl=en. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]. >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Nick Johnson, Developer Programs Engineer, App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
