This is a template for what I hope is how to best handle a low priory
task with very little impact on the system if it fails. It just cleans
up expired data and should not build up. Any other feedback is
appreciated.
<queue>
<name>deletedata</name>
<rate>20/s</rate>
<bucket-size>40</bucket-size>
<max-concurrent-requests>50</max-concurrent-requests>
<task-retry-limit>1</task-retry-limit>
</queue>
On Mar 7, 7:40 am, Benjamin <[email protected]> wrote:
> I did not - i'm definitely going to add that, thank Dale. I re-
> enabled the queue the other day and it's been behaving as expected, so
> it was definitely some external factor that made it spin out of
> control.
>
> My project is a cloud based data historian onwww.nimbits.comwhere
> users attach arduino devices to the services and record second by
> second data, so when they delete a data point I need to chew through
> the obsolete values that were recorded to that point. I ran out of
> quota on a day where a user deleted a point with millions of recorded
> values on it. It was supposed to just be a small background task.
>
> On Mar 6, 11:44 am, Dale <[email protected]> wrote:
>
>
>
>
>
>
>
> > Did you set max_concurrent_requests for that queue? Perhaps there was
> > some latency on the datastore and your queue was adding 5 tasks each
> > second but all the tasks were still running until the 10 s datastore
> > timeout and then retrying. In that case you would get 50 tasks
> > executing in parallel all querying 1000 records. Could explain the
> > high CPU usage.
>
> > Dale
>
> > On Mar 5, 5:33 pm, Benjamin <[email protected]> wrote:
>
> > > Hi Nick,
>
> > > The app id is nimbits1 (nimbits1.appspot.com)
>
> > > The spike occurred around 12pm EST on March 3rd. 95% of the cpu being
> > > used at the time was
> > > Task Queues > deletedata which is set to 5/sec with a bucket size of
> > > 5.
>
> > > Thanks!
>
> > > On Mar 4, 4:21 pm, Nicholas Verne <[email protected]> wrote:
>
> > > > Could you give us your app id and the approximate time of the gobbling
> > > > of CPU so we can investigate?
>
> > > > Thanks,
>
> > > > Nick
>
> > > > On Sat, Mar 5, 2011 at 5:49 AM, Benjamin <[email protected]> wrote:
> > > > > I have a Task that's purpose is to chew away at data that needs to be
> > > > > deleted. I have millions of records that need to go, so the task uses
> > > > > the low level api and key only queries to grab 1000 delete them and
> > > > > restart the task until the remaining count is zero.
>
> > > > > The task is set to 5/sec with a bucket size of 5. The other day it
> > > > > skyrocketed my CPU usage and maxed out my 5$ quota (400 CPU seconds /
> > > > > second, i think the lights in New York dimmed). Am i missing some
> > > > > setting here? How do i keep a task queue chugging along at a fixed
> > > > > rate without a burst like that, regardless of how many tasks are in
> > > > > the queue?
>
> > > > > Thanks!
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "Google App Engine" group.
> > > > > To post to this group, send email to
> > > > > [email protected].
> > > > > To unsubscribe from this group, send email to
> > > > > [email protected].
> > > > > For more options, visit this group
> > > > > athttp://groups.google.com/group/google-appengine?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
