I have experienced exactly the same problem since two weeks ago. I have a system with 100 restaurantes and 1000 users working without problem, but suddenly, since two weeks ago, every day at rush hour, the tasks in the task queue have experience a long delay in their start, two weeks before was only 1 or 2 seconds, now if 15 to 60 seconds, impacting the user experience and usability. I have to modify all the code with taskqueues and changing it to urlfetch request async qithout waiting for the rpc. The worst is that the task are causing a dealine exeeded 123 error, losing between 50 to 1000 request every day. The task and my proceses are very fast, the last only from 50ms to 3 seconds, not more, but I get a lot of them with 60000ms and more and get cancelled without starting (i have logging.debug message at the very begiining of every task/process that never get executed). I have 2 idel instances and all the settings to increase the instances without restriction. The start time of my instacnes is only 1 second, there are no special processes in the booting. I have 6 modules, separated modules for the task and the problem is affecting the module that call the task.add (not the module that executes the task). I made all the changes to avoid datastore contention, I deactivated the logs, I use a lot the memcache, I changed the F1 instances to F2, ant this error continues. And it APPEARED TWO WEEKS AGO. I have 1 year and a half runing my app, without problems, and suddenly this problem appeared.
This is a HUUUUGE problem, it make me thing about quit GAE and search another provider if I don't get a solution and explanation soon. My customers are very angry and everyday I loose requests (deadline exceeded error, I receive 2 or 3 phone calls). My system is mission critical and 1 munute without answers to request make my customer lose money. If any google engineer wants more detailed, I kindly offer all my support, I'm engineer with near 30 years experience and 3 years working with GAE Python and many Google Tools (Angular, Firebase, GCP, etc). I can bet that the problem is more related to the platform than to my code. Rene Marty El martes, 1 de diciembre de 2015, 13:03:25 (UTC-3), Jim escribió: > > Hello, > > Has anyone else noticed a lot of latency in Task Queues within the past > few days? We've noticed that our push queues are backing up with > thousands, sometimes tens of thousands of tasks. Nothing has changed in > our app or traffic profile. Over the past few days we've seen several > instances where the queues just seem to stop "pushing" tasks and they fill > up. We can go in and manually trigger individual tasks to run; tasks are > not generating errors, they're just not running automatically like they > should based on our configuration. Eventually things seem to catch up > again. Anyone else seeing similar behavior? > > Jim > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/aad9c4d4-5789-4715-bdd6-5e327753691a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
