I am not sure what the maximum add rate actually is, but I have be able to sustain adding thousands of tasks a minute for fairly long bursts (perhaps 30 minutes to an hour). Sometimes the queue backs up a bit, but I usually do not have issues adding and running 2,000 tasks / minute. At this point I have not tried to push it much beyond that.
Robert On Fri, Oct 29, 2010 at 16:37, Matija <[email protected]> wrote: > I must say that this exception happens very seldom, but we are still > at developing stage with not realistic testing data. I am not saying > that it is constant problem and maybe it is by design. It resolves > very easy because next same task execution runs successfully as it > should by the app engine task design. > > Problem is that we don't know at what task adding rate exception could > happen. Maybe instead of tree like chain of task we would create > single contiguous list with task by task execution when it is > acceptable. Especially if tasks are fighting with front web request > for same instances and add to save average latency. > > On Oct 29, 9:03 pm, Robert Kluin <[email protected]> wrote: >> I do not think you mentioned the fact you are using a wrapper for the >> task-queue until now. Have you tried saving a very simple entity >> inside the transaction causing a problem to verify that it is not an >> issue with the other library? >> >> Even if you are not using a wrapper, I would try to save a simple >> entity and see if your error goes away. >> >> Robert >> >> >> >> >> >> >> >> On Fri, Oct 29, 2010 at 03:12, Matija <[email protected]> wrote: >> > Thank you for your suggestions. To be honest I know that I can put >> > them in queue via batch put, but we are using Vince Bonfanti's >> > deferred task queue helper (slightly enhanced for our internal use >> > with transactions). It has no support for batch putting, but probably >> > we could add support to it and try it. But as you probably already >> > know, entities batch put doesn't guarantee that every entity will >> > successfully be saved or none (except in transaction), we can presume >> > that tasks also would not. So problem is probably somewhere in api >> > service for task queue. >> >> > But with batch put our task request would need less time to finish and >> > with less request latency we could get more instances. Hm... with more >> > instances we could get more transactional task queuing contention >> > problems. >> >> > I think that it would be nice that somebody from App Engine team >> > explains how can we avoid it, if it is possible. Maybe this is app >> > engine design problem and we should not worry about it, like knowing >> > that for one put statement in let's say every 8000 (I read some >> > article) you can expect not to finish because of its distributed >> > nature. Problem is that they are quiet and because of it we suspect >> > everything. >> >> > On Oct 29, 12:39 am, hawkett <[email protected]> wrote: >> >> Yet another consideration - you need to be aware that when you raise >> >> multiple tasks in quick succession, it is possible (likely even) that >> >> they will execute concurrently. If each task is trying to update the >> >> same entity, they may very well compete with each other, and enter a >> >> contention/retry deadlock. >> >> >> On Oct 28, 11:33 pm, hawkett <[email protected]> wrote: >> >> >> > One possible consideration - when you raise multiple tasks, do you do >> >> > so in a single call to the task queue API (batch put) or multiple >> >> > calls? If the latter, a great test would be to change to batch put and >> >> > see if the problem goes away. >> >> >> > On Oct 28, 11:29 pm, hawkett <[email protected]> wrote: >> >> >> > > Hi Matija, >> >> >> > > I can see your use case - you want to make sure that either all >> >> > > tasks get raised, or none. I can also see your worry - that tasks are >> >> > > actually being *fired* before the transaction is complete. I agree >> >> > > this would be a completely incorrect situation. I also transactionally >> >> > > raise multiple tasks, but I always have a datastore operation in >> >> > > conjunction (and use python) - I haven't encountered this issue of >> >> > > tasks being fired before the transaction completes, but I wonder if >> >> > > google has made the assumption that a transaction will always contain >> >> > > a datastore operation, and perhaps their logic is misbehaving if it >> >> > > does not? >> >> >> > > An example of how this might happen is if their logic is waiting for a >> >> > > counter of outstanding db operations to reach 0 before raising tasks. >> >> > > This would be flawed logic. Thanks, >> >> >> > > Colin >> >> >> > > On Oct 28, 8:21 am, Matija <[email protected]> wrote: >> >> >> > > > I am not saving any entities but I am putting more than one task in >> >> > > > queue. It depends of the data that current task is processing. Let's >> >> > > > say two tasks (always less then five). >> >> >> > > > At the end of 'transaction' I will have two tasks in queue or none. >> >> >> > > > This usually works, but sometimes I get this >> >> > > > ConcurrentModificationException exception. Complete application >> >> > > > chunk >> >> > > > works eventually because task with execute again and will finish >> >> > > > correctly, but it would be nice to understand this exception so >> >> > > > that I >> >> > > > can avoid it. >> >> >> > > > On Oct 28, 1:58 am, Tim Hoffman <[email protected]> wrote: >> >> >> > > > > If you are not saving any entities why are you using a >> >> > > > > transaction to >> >> > > > > put the task into the queue. >> >> >> > > > > The idea of a queuing a task transactionally is that the task >> >> > > > > doesn't >> >> > > > > get fired if the transaction fails (ie you save of entities >> >> > > > > fails). >> >> >> > > > > Seems completely redundant >> >> >> > > > > Rgds >> >> >> > > > > T >> >> >> > > > > On Oct 21, 11:18 pm, Matija <[email protected]> wrote: >> >> >> > > > > > Hi, >> >> > > > > > I am >> >> > > > > > usinghttp://code.google.com/appengine/docs/java/taskqueue/overview.html#Ta... >> >> > > > > > way to enqueue two tasks in transaction. But I am not saving any >> >> > > > > > datastore entities. I am using objectify. >> >> >> > > > > > I got this message: >> >> >> > > > > > java.util.ConcurrentModificationException: too much contention >> >> > > > > > on >> >> > > > > > these datastore entities. please try again. >> >> > > > > > at >> >> > > > > > com.google.appengine.api.datastore.DatastoreApiHelper.translateError(Datast >> >> > > > > > oreApiHelper.java: >> >> > > > > > 39) >> >> > > > > > at >> >> > > > > > com.google.appengine.api.datastore.DatastoreApiHelper.makeSyncCall(Datastor >> >> > > > > > eApiHelper.java: >> >> > > > > > 75) >> >> > > > > > at >> >> > > > > > com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(Transaction >> >> > > > > > Impl.java: >> >> > > > > > 44) >> >> > > > > > at >> >> > > > > > com.google.appengine.api.datastore.TransactionImpl.makeSyncCall(Transaction >> >> > > > > > Impl.java: >> >> > > > > > 52) >> >> > > > > > at >> >> > > > > > com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.j >> >> > > > > > ava: >> >> > > > > > 64) >> >> >> > > > > > When this happened only one task needed to be added to queue, >> >> > > > > > so why >> >> > > > > > this error ? >> >> > -- >> > You received this message because you are subscribed to the Google Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]. >> > For more options, visit this group >> > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
