Hey Thanasis, Could you email me directly with your project ID and task queue names, along with timeframes in recent days when this has occurred? Or if it isn't occurring at present, send an email immediately upon viewing it in future? You could also create a Public Issue Tracker issue <http://code.google.com/p/googleappengine/issues/list> with label " Type-Production". We should be able to look into this to determine what's going on.
Cheers, Nick Cloud Platform Community Support On Wednesday, October 12, 2016 at 3:13:01 PM UTC-4, Thanasis Delenikas wrote: > > Hello Nick, > > No, they do not relate with the processing time. The later is quite > constant - not very fast, but constant. > > Tasks just waited (and waited and waited) in the queue without getting > processed. When they did find their way in the processing handlers, they > took the same time as always. > > The graphs you see skyrocketed not because the processing time went nuts, > but because GAE "forgot" to consume the tasks. > > This is a budget-enabled project - but GAE made no attempt to launch even > one more instance in order to take some tasks off the queues. Queues are > set quite high: Token Bucket:500 and Rate: 10/s > > Since the last post, I've moved my project to the European DCs (I was > planning to do that anyway, since it's closer to me). > I am experiencing similar issues here as well. I mean, I've seen tasks > staying in the queue for 5 minutes without any obvious reason. On other > occasions, queue consumption is instant. > > Thanks. > > > On Monday, October 10, 2016 at 11:32:32 PM UTC+3, Nick (Cloud Platform > Support) wrote: >> >> Hey Thanasis, >> >> Does that latency correspond to the time spent in the handlers that deal >> with the task queue request itself? Or, if not, what underlies the metric? >> >> Cheers, >> >> Nick >> Cloud Platform Community Support >> >> On Friday, October 7, 2016 at 3:55:04 PM UTC-4, Thanasis Delenikas wrote: >>> >>> Hi Nick / Google, >>> >>> During the last 1-2 weeks, I've had the same problem. I've *never* >>> encountered a backlog on my queues (load is small and processing generally >>> fast). Then, suddenly and out of the blue, with *no* application or other >>> changes, I get the behavior displayed on the attached graph (graph from >>> Stackdriver, queue latency, 1month graph). It really crippled me! >>> >>> I don't believe that this issue is related to my project. It must be a >>> generic issue that you should be aware of. I've even tried with different >>> AppEngine instance classes, but this "delay" had nothing to do with the >>> instance "power". >>> >>> I don't think opening tickets will do any good. I've done it in the >>> past, when I reported an issue with task queue rate (Issue #12620, >>> acknowledged since December 17, 2015). Resolution -> NONE. >>> >>> I really love the AppEngine concept and development environment. But I >>> doubt Google's devotion to it and that makes me really sad. >>> >>> >>> On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform >>> Support) wrote: >>>> >>>> Hey Pasha, >>>> >>>> I would suggest posting a more detailed technical report to the Public >>>> Issue Tracker <http://code.google.com/p/googleappengine/issues/list>. >>>> The Public Issue Tracker is much better suited to issues like this, while >>>> this forum generally should be geared toward high-level discussion of the >>>> platform and services, design patterns, etc. >>>> >>>> You should try to include some or all of the following: >>>> >>>> 1. your project ID >>>> 2. the task queue names >>>> 3. their configurations >>>> 4. the code for and/or a description of the task handler >>>> 5. the time-frames when the issue is observed to occur within the past >>>> few days (since time is of the essence in making sure the logs still exist >>>> in the system) >>>> 6. a description of the average load and the frequency distribution of >>>> different sized loads >>>> 7. the frequency distribution of response latencies for the task >>>> handlers >>>> >>>> This will enable us to get started on reasoning about the issue and >>>> looking into it. Feel free to post a link to the thread here once you've >>>> done that. I'll be happy to assist. >>>> >>>> Cheers, >>>> >>>> Nick >>>> Cloud Platform Community Support >>>> >>>> On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov >>>> wrote: >>>>> >>>>> Hi Nick, >>>>> I have the same issue with queue not executing tasks at rate set in >>>>> configuration. It looks really strange because the execution works >>>>> correctly most of the days, but once per week the queue does not execute >>>>> at >>>>> correct rate. I experience the issue on Sep 30th and today Oct 6. >>>>> >>>>> I tried different configuration settings, but the configuration >>>>> changes did affect the real execution rate. The execution is way too slow >>>>> than set in configuration. >>>>> >>>>> Best regards, >>>>> Pasha >>>>> >>>>> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform >>>>> Support) wrote: >>>>>> >>>>>> Thanks for the update. Barring a very low setting for "max concurrent >>>>>> tasks", I don't see why this could occur, so it appears this could be a >>>>>> production issue. In that case, I'd recommend opening a Public Issue >>>>>> Tracker <http://code.google.com/p/googleappengine/issues/list> >>>>>> thread with label "Restrict-View-EditIssue" (to protect your >>>>>> information) >>>>>> and the following information: >>>>>> >>>>>> * your app id >>>>>> * the name of the queue >>>>>> * the queue settings >>>>>> * some queue statistics >>>>>> >>>>>> We may be able to determine the root cause of this at that point. >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> Nick >>>>>> Cloud Platform Community Support >>>>>> >>>>>> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote: >>>>>>> >>>>>>> Just got an alert again. The task rate is 25 tasks/sec with a bucket >>>>>>> size of 10. Average latency of the handlers is varying depending on the >>>>>>> task, but anywhere between 1 and 8 seconds. >>>>>>> >>>>>>> The thing that I'm seeing is that the task queue shows 120 tasks in >>>>>>> the queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem >>>>>>> right to me, and it looks like these tasks are just not being executed. >>>>>>> I >>>>>>> see nothing in the logs of my task handlers that indicate that a >>>>>>> request is >>>>>>> being executed. >>>>>>> >>>>>>> Obviously this task execution thing is pretty essential to my app. >>>>>>> >>>>>>> Hugo >>>>>>> >>>>>>> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud >>>>>>> Platform Support) wrote: >>>>>>>> >>>>>>>> Thanks for coming back to update. Feel free to post again with the >>>>>>>> information mentioned if you notice anything in the future. >>>>>>>> >>>>>>>> Best wishes, >>>>>>>> >>>>>>>> Nick >>>>>>>> Cloud Platform Community Support >>>>>>>> >>>>>>>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote: >>>>>>>>> >>>>>>>>> It looks like the queues are back to normal. I've setup alerts in >>>>>>>>> stack driver that triggered for a while but are resolved now. Will >>>>>>>>> report >>>>>>>>> back if that changes. >>>>>>>>> >>>>>>>>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via >>>>>>>>> Google App Engine <google-a...@googlegroups.com> wrote: >>>>>>>>> >>>>>>>>>> Hey Hugo, >>>>>>>>>> >>>>>>>>>> Without knowing more about A) the configuration of the queue, B) >>>>>>>>>> the rate of task creation, C) the latency statistics on the task >>>>>>>>>> handler, >>>>>>>>>> it's difficult to say much on why this might be happening. If you >>>>>>>>>> could >>>>>>>>>> gather all those types of information, myself or someone else may be >>>>>>>>>> able >>>>>>>>>> to leave a more insightful comment. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> Nick >>>>>>>>>> Cloud Platform Community Support >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote: >>>>>>>>>>> >>>>>>>>>>> I'm seeing tasks in my queue that are there for 6 minutes >>>>>>>>>>> without any tasks running at all. There doesn't seem to be much I >>>>>>>>>>> can do >>>>>>>>>>> about this which is really frustrating. >>>>>>>>>>> >>>>>>>>>>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote: >>>>>>>>>>>> >>>>>>>>>>>> Since last night or so I'm seeing this issue where tasks are >>>>>>>>>>>> being queued up for some time and not being executed on an app >>>>>>>>>>>> that has >>>>>>>>>>>> been running like this for quite a while. This results in a huge >>>>>>>>>>>> pile of >>>>>>>>>>>> tasks that get backlogged for my app. >>>>>>>>>>>> >>>>>>>>>>>> Cloud status dashboard says everything is OK. Anyone else >>>>>>>>>>>> seeing this? >>>>>>>>>>>> >>>>>>>>>>>> Hugo >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to a topic >>>>>>>>>> in the Google Groups "Google App Engine" group. >>>>>>>>>> To unsubscribe from this topic, visit >>>>>>>>>> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe >>>>>>>>>> . >>>>>>>>>> To unsubscribe from this group and all its topics, send an email >>>>>>>>>> to google-appengi...@googlegroups.com. >>>>>>>>>> To post to this group, send email to google-a...@googlegroups.com >>>>>>>>>> . >>>>>>>>>> Visit this group at >>>>>>>>>> https://groups.google.com/group/google-appengine. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to google-appengine+unsubscr...@googlegroups.com. To post to this group, send email to google-appengine@googlegroups.com. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/65d0da0a-377c-426e-8394-03e6562dac00%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
