Hi, I wonder if anyone can help with this conundrum. I've populated a task queue with 3.5million tasks which I wish to run. Each task is idempotent and built as a tiny package of work to run on the datastore in the form of a get by key, a second query, some cpu processing following by a put and a delete.
All of this is wrapped by a try/catch block which catches all exceptions (and logs them). Depending on the type of the exception, the task is either re-tried or failed. Of these 3.5 million tasks, approximately 2 million have now successfully run. The queue is processing these tasks at a rate of 25/s. The problem I appear to be having is that on some occasions (yesterday afternoon, around 6-7pm GMT) the datastore starts failing with either a DatastoreTimeoutException or a DatastoreFailureException, neither of which give any more detail in the exception message. The result of these failures (and I assume, the length of time between task start and task failure) is that the queue ends up running thousands of tasks simultaneously and the corresponding latency plummets. This causes new instances to spool up. Under normal operation (i.e. the 2 million tasks that successfully ran), our instance count goes into the low hundreds (100-150) - a not unreasonable number. When the datastore starts failing, this jumps into the 900-1000 range. Each of these instances each have to be paid for and each task failure is in itself is racking up a lot of CPU time and datastore reads. Therefore - is there a better way of dealing with each Datastore timeout, other than manually pausing the queue until the issue has resolved itself? Also, if any indication as to what the failures might mean that would be really useful too! Thanks in advance for any help! Ed -- You received this message because you are subscribed to the Google Groups "Google App Engine for Java" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine-java/-/MNVU85TGgOAJ. To post to this group, send email to google-appengine-java@googlegroups.com. To unsubscribe from this group, send email to google-appengine-java+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/google-appengine-java?hl=en.
