I'd say your breakage is happening when trying to add 1000 tasks to the queue at in one request. (It's probably timing out.) To add 1000 tasks to a queue in 30 seconds would require the entire process of adding one task and getting a response that the task was successfully added to take 30ms.. that seems doable (and clearly is since your crons aren't always failing).. but I'd guess sometimes it just takes too long.
I'm still confused though.. you say the cron adds 1000 tasks to the queue.. then.. presumably.. the tasks execute at the same time .. but you say that a given cron job and its 1000 tasks complete within 4 minutes.. but that can't happen since the tasks can only take 30 seconds a piece (at most).. 4 minutes is impossible.. I'm guessing that your cron job is staggering the added tasks using the countdown property? You should probably do some batching in there.. either lower the number of tasks you need to fire off by modifying the task url to caclulate and put your entities in batches of 10 (thus, you'd only need to add 100 tasks)... or if you just don't want to do that, chain your tasks.. so the cron adds 100 tasks to the queue.. and each of those tasks is responsible for adding the next batch of 100.. and that batch is responsible for the 100 after it.. etc. OR, just break it up into more cron jobs.. but I really don't like that solution. On Fri, Feb 26, 2010 at 4:19 PM, Marc Provost <[email protected]> wrote: > Thanks for your quick reply! > > I parse several external sources (around 8) and from each source I > need to update the same 1000 entities (most of them already existing, > creation of new entities is rare). For each data source, I schedule a > cron job which spawns 1000 tasks (with attached data) and each of them > will update a single entity. I found by trial and error that app > engine was behaving better the shorter the tasks. So, when I say very > much parallelized, I mean spawning as many tasks as I can for each > cron job, each of them as small as possible. Since I have more > independent tasks running in parallel, my cron jobs execute faster. In > addition, I schedule my cron jobs apart so that they don't overlap, > but this should not matter, as I use the same queue which is limited > at 5 tasks per second. > > So, in summary, I have 8 cron jobs and each cron job spawns 1000 > tasks. A given cron job and its children tasks terminates in 3-4 > minutes at most. The cron jobs are separated so that 2 crons jobs > never execute together. > > Marc > > > > > > On Feb 26, 3:47 pm, Eli Jones <[email protected]> wrote: > > How many is "a bunch"? Also, You say "they are all very much > parallelized" > > but then you say that you've scheduled them 10 minutes apart and they > don't > > overlap.. those two statements are contradictory, please explain more > > clearly your cron-taskqueu setup and how it works and what exactly it is > > doing. > > > > When you say that the cron jobs "spawn tasks that write to one entity > > each".. what do you mean? The cron job is there to fire of the initial > > task.. and that task runs once, putting one entity and that's it? > > > > If so, why are you having these tasks only put one entity at a time.. > > instead of creating multiple entities and putting them in batches? Does > > each task put() new entities? or are they sometimes putting an entity > that > > may already exist? > > > > More info is more better for help. > > > > On Fri, Feb 26, 2010 at 3:35 PM, Marc Provost <[email protected]> > wrote: > > > Ok, here's my situation: > > > > > * I use the java implementation and my app id is poolfana. > > > * I have a bunch of cron jobs scheduled at night (Eastern Time) > > > * They are all very much parallelized. I am being very strict: they > > > spawn tasks that only write to one entity each. Each tasks will > > > execute in a few hundred ms. > > > * A given cron job and its spawned tasks will terminate in a few > > > minutes at most. > > > * I have scheduled each cron job at least 10 minutes apart, so they do > > > not overlap. > > > * In my dashboard, my max request per second is 3. The max limit is > > > supposed to be 30. > > > * My problem? The cron jobs fail sporadically (marked as "failed" in > > > the dashboard) with this error: > > > > > "Request was aborted after waiting too long to attempt to service your > > > request. Most likely, this indicates that you have reached your > > > simultaneous dynamic request limit. This is almost always due to > > > excessively high latency in your app. Please see > > >http://code.google.com/appengine/docs/quotas.htmlfor more details." > > > > > There is an issue for this problem: > > >http://code.google.com/p/googleappengine/issues/detail?id=2396 > > > > > It was starred 50+ times, but it was not acknowledged yet by the > > > google team. I'm writing this post to discuss potential workarounds, > > > potential misuses of the API with the google team or other people that > > > might have solved this problem. What else can I do? Is it a problem on > > > the google side or I'm I doing something wrong? Right now, I need to > > > re-execute the cron jobs manually everyday... > > > > > Thank you! > > > Marc > > > > > -- > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > [email protected]<google-appengine%[email protected]> > <google-appengine%[email protected]<google-appengine%[email protected]> > > > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
