I agree that it is dangerous to blindly catch all errors. However, in
this case you could catch and log the full trace. That way you get
your logging info, but you don't get dupes.
try:
# do ... lots ... of ... things
raise Exception('Ka Powww!')
except:
logging.exception('A task crapped out.')
Robert
On Tue, Feb 21, 2012 at 15:02, James Gilliam <[email protected]> wrote:
> I usually don't usually catch all errors because it hides diagnostic
> info ... but since I can't limit the retries, i usually get the error
> twice ...
>
> On Feb 21, 11:20 am, Jeff Schnitzer <[email protected]> wrote:
>> If you don't want a task to retry, catch any errors and simply return
>> normally as if the task completed.
>>
>> Jeff
>>
>>
>>
>>
>>
>>
>>
>> On Tue, Feb 21, 2012 at 2:15 PM, James Gilliam <[email protected]> wrote:
>> > one other task queue question ...
>>
>> > i have ...
>>
>> > retry_parameters:
>> > task_retry_limit: 1
>>
>> > is there any way i can say no retries?
>>
>> > a while back, i tried task_retry_limit: 0 but didn't work
>>
>> > thanks
>>
>> > On Feb 21, 10:56 am, Robert Kluin <[email protected]> wrote:
>> > > Why not leave the rate at 1/s, but set max_concurrent_requests to 1 as
>> > > Nick suggested?
>>
>> > > On Tue, Feb 21, 2012 at 12:56, James Gilliam <[email protected]>
>> > wrote:
>> > > > Yes ... I saw the 5 per sec in the admin ...
>>
>> > > > Also, I just realized why I think gae created so many instances when i
>> > > > had the queue at 1/s ... I had assumed the tasks would take much less
>> > > > than a second (which was true at the beginning); but as the number of
>> > > > entity writes continued, the latency for the writes got larger and
>> > > > eventually took longer than a second ... at which point gae started
>> > > > new instances to keep up with the 1 sec rate ... i had assumed it
>> > > > would not consider starting a new task until the previous one in the
>> > > > queue finished -- this appears to not be the semantics
>>
>> > > > My solution is to slow down the tasks even further ... to say 15 /
>> > > > min ... so that new instances will not be created ...
>>
>> > > > On Feb 19, 12:21 pm, Nicholas Verne <[email protected]> wrote:
>> > > >> James,
>>
>> > > >> The misreporting of number of tasks in a queue is a known bug that
>> > > >> we're working on. It causes undue alarm for our users.
>>
>> > > >> How do you know that GAE changed the target rate to 5 / sec ? Did you
>> > > >> observe this in the admin console?
>>
>> > > >> The queue's rate is the rate at which tasks are dispatched to your
>> > > >> app. If your queue is set to run 1 / sec and your task latency is
>> > > >> longer than 1 second, you should expect to have more instances spin
>> > > >> up.
>>
>> > > >> If you wish to prevent more than one task running at a time, set your
>> > > >> queue's max_concurrent_requests to 1.
>>
>> > > >> Nick Verne
>>
>> > > >> On Mon, Feb 20, 2012 at 3:35 AM, James Gilliam <[email protected]>
>> > wrote:
>> > > >> > I am using task queue in python.
>>
>> > > >> > I added 10,000 tasks for a queue set to run at 1/sec and a bucket of
>> > > >> > 1 ... they were short tasks and I didn't want them to run
>> > > >> > immediately ... I didn't even want them to run multiples per second
>> > > >> > even though the latency was low ... it was ok if it took hours as I
>> > > >> > knew it would at 1/sec. I figured it would create a second instance
>> > > >> > as the site was continuing to run ... but I was pretty surprised
>> > with
>> > > >> > what actually happen.
>>
>> > > >> > At first the queue started processing fine ... approx 60 tasks
>> > running
>> > > >> > per minute ...
>>
>> > > >> > Then GAE decided to run them at a faster rate, change the target
>> > rate
>> > > >> > to 5 / sec and created lots of new instances ...
>>
>> > > >> > When there were about 3000 tasks left ... GAE went crazy and said
>> > > >> > there were nearly 200,000 tasks in the queue ...
>>
>> > > >> > As far as I could tell, the number of tasks being reported was just
>> > an
>> > > >> > anomaly ... but it was alarming given that GAE was creating
>> > instances
>> > > >> > like crazy.
>>
>> > > >> > The tasks would add new records (sometimes) and I noticed that the
>> > > >> > latency of these tasks increased the longer the tasks ran. At first
>> > > >> > just 200ms ... by the end nearly 1400 ms.
>>
>> > > >> > So --
>>
>> > > >> > 1. The most important question -- Why did GAE not respect my desire
>> > to
>> > > >> > run the queue at 1 / sec ? Is there a way I can stop GAE from
>> > > >> > overriding my wishes?
>>
>> > > >> > 2. Was the latency increasing because the records being added were
>> > > >> > slowing new records from being added?
>>
>> > > >> > 3. The invalid number of tasks seems like a harmless bug since new
>> > > >> > phantom tasks were not actually created (as far as I know).
>>
>> > > >> > --
>> > > >> > You received this message because you are subscribed to the Google
>> > Groups "Google App Engine" group.
>> > > >> > To post to this group, send email to
>> > [email protected].
>> > > >> > To unsubscribe from this group, send email to
>> > [email protected].
>> > > >> > For more options, visit this group athttp://
>> > groups.google.com/group/google-appengine?hl=en.
>>
>> > > > --
>> > > > You received this message because you are subscribed to the Google
>> > Groups "Google App Engine" group.
>> > > > To post to this group, send email to [email protected]
>> > .
>> > > > To unsubscribe from this group, send email to
>> > [email protected].
>> > > > For more options, visit this group athttp://
>> > groups.google.com/group/google-appengine?hl=en.
>>
>> > --
>> > You received this message because you are subscribed to the Google Groups
>> > "Google App Engine" group.
>> > To post to this group, send email to [email protected].
>> > To unsubscribe from this group, send email to
>> > [email protected].
>> > For more options, visit this group at
>> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
