Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['David Charles Martinez' via Google App Engine]

Hello Govind,

Google Issue Tracker is the right place for these types of issues and with 
the provided details we can track it and troubleshoot it accordingly since 
Google Groups is meant for conceptual questions and discussions and this 
will require proper troubleshooting. In addition, I would like to inform 
you that this issue is currently being investigated.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/74ae877e-23fe-4d85-b4cc-b9b06f3cf3f1%40googlegroups.com.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Govind Rai]

Hi Thanasis,

How were you able to resolve your problem?

Govind

On Wednesday, October 12, 2016 at 12:13:01 PM UTC-7, Thanasis Delenikas 
wrote:
>
> Hello Nick,
>
> No, they do not relate with the processing time. The later is quite 
> constant - not very fast, but constant.
>
> Tasks just waited (and waited and waited) in the queue without getting 
> processed. When they did find their way in the processing handlers, they 
> took the same time as always.
>
> The graphs you see skyrocketed not because the processing time went nuts, 
> but because GAE "forgot" to consume the tasks.
>
> This is a budget-enabled project - but GAE made no attempt to launch even 
> one more instance in order to take some tasks off the queues. Queues are 
> set quite high: Token Bucket:500 and Rate: 10/s
>
> Since the last post, I've moved my project to the European DCs (I was 
> planning to do that anyway, since it's closer to me).
> I am experiencing similar issues here as well. I mean, I've seen tasks 
> staying in the queue for 5 minutes without any obvious reason. On other 
> occasions, queue consumption is instant.
>
> Thanks.
>
>
> On Monday, October 10, 2016 at 11:32:32 PM UTC+3, Nick (Cloud Platform 
> Support) wrote:
>>
>> Hey Thanasis,
>>
>> Does that latency correspond to the time spent in the handlers that deal 
>> with the task queue request itself? Or, if not, what underlies the metric?
>>
>> Cheers,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Friday, October 7, 2016 at 3:55:04 PM UTC-4, Thanasis Delenikas wrote:
>>>
>>> Hi Nick / Google,
>>>
>>> During the last 1-2 weeks, I've had the same problem. I've *never* 
>>> encountered a backlog on my queues (load is small and processing generally 
>>> fast). Then, suddenly and out of the blue, with *no* application or other 
>>> changes, I get the behavior displayed on the attached graph (graph from 
>>> Stackdriver, queue latency, 1month graph). It really crippled me!
>>>
>>> I don't believe that this issue is related to my project. It must be a 
>>> generic issue that you should be aware of. I've even tried with different 
>>> AppEngine instance classes, but this "delay" had nothing to do with the 
>>> instance "power".
>>>
>>> I don't think opening tickets will do any good. I've done it in the 
>>> past, when I reported an issue with task queue rate (Issue #12620, 
>>> acknowledged since December 17, 2015). Resolution -> NONE.
>>>
>>> I really love the AppEngine concept and development environment. But I 
>>> doubt Google's devotion to it and that makes me really sad.
>>>
>>>
>>> On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform 
>>> Support) wrote:

 Hey Pasha,

 I would suggest posting a more detailed technical report to the Public 
 Issue Tracker . 
 The Public Issue Tracker is much better suited to issues like this, while 
 this forum generally should be geared toward high-level discussion of the 
 platform and services, design patterns, etc.

 You should try to include some or all of the following:

 1. your project ID
 2. the task queue names 
 3. their configurations
 4. the code for and/or a description of the task handler
 5. the time-frames when the issue is observed to occur within the past 
 few days (since time is of the essence in making sure the logs still exist 
 in the system)
 6. a description of the average load and the frequency distribution of 
 different sized loads
 7. the frequency distribution of response latencies for the task 
 handlers

 This will enable us to get started on reasoning about the issue and 
 looking into it. Feel free to post a link to the thread here once you've 
 done that. I'll be happy to assist.

 Cheers,

 Nick
 Cloud Platform Community Support

 On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov 
 wrote:
>
> Hi Nick,
> I have the same issue with queue not executing tasks at rate set in 
> configuration. It looks really strange because the execution works 
> correctly most of the days, but once per week the queue does not execute 
> at 
> correct rate. I experience the issue on  Sep 30th and today Oct 6.
>
> I tried different configuration settings, but the configuration 
> changes did affect the real execution rate. The execution is way too slow 
> than set in configuration.
>
> Best regards,
> Pasha
>
> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform 
> Support) wrote:
>>
>> Thanks for the update. Barring a very low setting for "max concurrent 
>> tasks", I don't see why this could occur, so it appears this could be a 
>> production issue. In that case, I'd recommend opening a Public Issue 
>> Tracker  
>> thread with 

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nicolas (Google Cloud Platform Support)' via Google App Engine]

Hi,

 

Thanks for reporting this and thanks for opening the issue tracker ticket, 
that was the right thing to do.

 

Further communications will be made in this thread 
, however we cannot 
promise an ETA for the resolution of this issue.


On Tuesday, June 18, 2019 at 12:18:58 PM UTC-4, Javier Martin wrote:
>
> I'm experiencing the same problem. 
> Every 30-45 minutes, about 10-20 tasks in queue are just ignored for some 
> minutes. It takes about 7-9 minutes for the queue to dispatch them. All the 
> rest of tasks are dispatched normally : (
> I already filed an issue on the tracker 
> https://issuetracker.google.com/issues/135471809
> Did you find any solution?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6e6581cd-6ec8-4e44-9e5d-39828b469c4b%40googlegroups.com.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Javier Martin]

I'm experiencing the same problem. 
Every 30-45 minutes, about 10-20 tasks in queue are just ignored for some 
minutes. It takes about 7-9 minutes for the queue to dispatch them. All the 
rest of tasks are dispatched normally : (
I already filed an issue on the 
tracker https://issuetracker.google.com/issues/135471809
Did you find any solution?

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/a1ba3588-48c4-40b8-876c-f17818b061e1%40googlegroups.com.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Ying Li' via Google App Engine]

For production issue, please post a proper report on our Public Issue 
Tracker [1] so we can investigate into it. Google Groups is generally used 
for simple product inquires and discussion and not actual issue reports, 
thanks.

[1] 
https://cloud.google.com/support/docs/issue-trackers#search_for_or_create_bugs_and_feature_requests_by_product

On Friday, May 24, 2019 at 4:52:15 PM UTC-4, Alan Gregg wrote:
>
> I find this and I struggle to upload things usually because memory wont 
> let me put through things.
>
> Yet you have places saying you can upload and make apps with very little 
> memory and storage.
>
>
>
> On Fri, 24 May 2019, 21:30 Asim Pereira,  wrote:
>
>> I know this issue was raised years ago, but I am facing the exact issue 
>> since quite many months.
>> I have tried various queue configurations, different instance classes, 
>> memory optimization but to no avail.
>> I even broke my backend and move a part of it to Frontend and Queues, but 
>> even the Queues backed by frontend stalls (apart from the backend)
>>
>> What solution worked for you guys?
>>
>> On Tuesday, May 10, 2016 at 12:24:33 PM UTC+5:30, Hugo Visser wrote:
>>>
>>> Since last night or so I'm seeing this issue where tasks are being 
>>> queued up for some time and not being executed on an app that has been 
>>> running like this for quite a while. This results in a huge pile of tasks 
>>> that get backlogged for my app.
>>>
>>> Cloud status dashboard says everything is OK. Anyone else seeing this?
>>>
>>> Hugo
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/d66b56dc-5e5f-4336-bdb9-688a7afc9589%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ef8b59a3-43ce-4030-95bf-a2aa73f12d9d%40googlegroups.com.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Alan Gregg]

I find this and I struggle to upload things usually because memory wont let
me put through things.

Yet you have places saying you can upload and make apps with very little
memory and storage.



On Fri, 24 May 2019, 21:30 Asim Pereira,  wrote:

> I know this issue was raised years ago, but I am facing the exact issue
> since quite many months.
> I have tried various queue configurations, different instance classes,
> memory optimization but to no avail.
> I even broke my backend and move a part of it to Frontend and Queues, but
> even the Queues backed by frontend stalls (apart from the backend)
>
> What solution worked for you guys?
>
> On Tuesday, May 10, 2016 at 12:24:33 PM UTC+5:30, Hugo Visser wrote:
>>
>> Since last night or so I'm seeing this issue where tasks are being queued
>> up for some time and not being executed on an app that has been running
>> like this for quite a while. This results in a huge pile of tasks that get
>> backlogged for my app.
>>
>> Cloud status dashboard says everything is OK. Anyone else seeing this?
>>
>> Hugo
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/d66b56dc-5e5f-4336-bdb9-688a7afc9589%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CANF_uCTHG9yg2Kb5omGSTjBpes0J1-qBG60kSPsf8Kzq0CumhQ%40mail.gmail.com.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Thanasis,

Could you email me directly with your project ID and task queue names, 
along with timeframes in recent days when this has occurred? Or if it isn't 
occurring at present, send an email immediately upon viewing it in future? 
You could also create a Public Issue Tracker issue 
 with label "
Type-Production". We should be able to look into this to determine what's 
going on.

Cheers,

Nick
Cloud Platform Community Support

On Wednesday, October 12, 2016 at 3:13:01 PM UTC-4, Thanasis Delenikas 
wrote:
>
> Hello Nick,
>
> No, they do not relate with the processing time. The later is quite 
> constant - not very fast, but constant.
>
> Tasks just waited (and waited and waited) in the queue without getting 
> processed. When they did find their way in the processing handlers, they 
> took the same time as always.
>
> The graphs you see skyrocketed not because the processing time went nuts, 
> but because GAE "forgot" to consume the tasks.
>
> This is a budget-enabled project - but GAE made no attempt to launch even 
> one more instance in order to take some tasks off the queues. Queues are 
> set quite high: Token Bucket:500 and Rate: 10/s
>
> Since the last post, I've moved my project to the European DCs (I was 
> planning to do that anyway, since it's closer to me).
> I am experiencing similar issues here as well. I mean, I've seen tasks 
> staying in the queue for 5 minutes without any obvious reason. On other 
> occasions, queue consumption is instant.
>
> Thanks.
>
>
> On Monday, October 10, 2016 at 11:32:32 PM UTC+3, Nick (Cloud Platform 
> Support) wrote:
>>
>> Hey Thanasis,
>>
>> Does that latency correspond to the time spent in the handlers that deal 
>> with the task queue request itself? Or, if not, what underlies the metric?
>>
>> Cheers,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Friday, October 7, 2016 at 3:55:04 PM UTC-4, Thanasis Delenikas wrote:
>>>
>>> Hi Nick / Google,
>>>
>>> During the last 1-2 weeks, I've had the same problem. I've *never* 
>>> encountered a backlog on my queues (load is small and processing generally 
>>> fast). Then, suddenly and out of the blue, with *no* application or other 
>>> changes, I get the behavior displayed on the attached graph (graph from 
>>> Stackdriver, queue latency, 1month graph). It really crippled me!
>>>
>>> I don't believe that this issue is related to my project. It must be a 
>>> generic issue that you should be aware of. I've even tried with different 
>>> AppEngine instance classes, but this "delay" had nothing to do with the 
>>> instance "power".
>>>
>>> I don't think opening tickets will do any good. I've done it in the 
>>> past, when I reported an issue with task queue rate (Issue #12620, 
>>> acknowledged since December 17, 2015). Resolution -> NONE.
>>>
>>> I really love the AppEngine concept and development environment. But I 
>>> doubt Google's devotion to it and that makes me really sad.
>>>
>>>
>>> On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform 
>>> Support) wrote:

 Hey Pasha,

 I would suggest posting a more detailed technical report to the Public 
 Issue Tracker . 
 The Public Issue Tracker is much better suited to issues like this, while 
 this forum generally should be geared toward high-level discussion of the 
 platform and services, design patterns, etc.

 You should try to include some or all of the following:

 1. your project ID
 2. the task queue names 
 3. their configurations
 4. the code for and/or a description of the task handler
 5. the time-frames when the issue is observed to occur within the past 
 few days (since time is of the essence in making sure the logs still exist 
 in the system)
 6. a description of the average load and the frequency distribution of 
 different sized loads
 7. the frequency distribution of response latencies for the task 
 handlers

 This will enable us to get started on reasoning about the issue and 
 looking into it. Feel free to post a link to the thread here once you've 
 done that. I'll be happy to assist.

 Cheers,

 Nick
 Cloud Platform Community Support

 On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov 
 wrote:
>
> Hi Nick,
> I have the same issue with queue not executing tasks at rate set in 
> configuration. It looks really strange because the execution works 
> correctly most of the days, but once per week the queue does not execute 
> at 
> correct rate. I experience the issue on  Sep 30th and today Oct 6.
>
> I tried different configuration settings, but the configuration 
> changes did affect the real execution rate. The execution is way too slow 
> than set in configuration.
>
> Best regards,
> Pasha
>

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Thanasis Delenikas]

Hello Nick,

No, they do not relate with the processing time. The later is quite 
constant - not very fast, but constant.

Tasks just waited (and waited and waited) in the queue without getting 
processed. When they did find their way in the processing handlers, they 
took the same time as always.

The graphs you see skyrocketed not because the processing time went nuts, 
but because GAE "forgot" to consume the tasks.

This is a budget-enabled project - but GAE made no attempt to launch even 
one more instance in order to take some tasks off the queues. Queues are 
set quite high: Token Bucket:500 and Rate: 10/s

Since the last post, I've moved my project to the European DCs (I was 
planning to do that anyway, since it's closer to me).
I am experiencing similar issues here as well. I mean, I've seen tasks 
staying in the queue for 5 minutes without any obvious reason. On other 
occasions, queue consumption is instant.

Thanks.


On Monday, October 10, 2016 at 11:32:32 PM UTC+3, Nick (Cloud Platform 
Support) wrote:
>
> Hey Thanasis,
>
> Does that latency correspond to the time spent in the handlers that deal 
> with the task queue request itself? Or, if not, what underlies the metric?
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
> On Friday, October 7, 2016 at 3:55:04 PM UTC-4, Thanasis Delenikas wrote:
>>
>> Hi Nick / Google,
>>
>> During the last 1-2 weeks, I've had the same problem. I've *never* 
>> encountered a backlog on my queues (load is small and processing generally 
>> fast). Then, suddenly and out of the blue, with *no* application or other 
>> changes, I get the behavior displayed on the attached graph (graph from 
>> Stackdriver, queue latency, 1month graph). It really crippled me!
>>
>> I don't believe that this issue is related to my project. It must be a 
>> generic issue that you should be aware of. I've even tried with different 
>> AppEngine instance classes, but this "delay" had nothing to do with the 
>> instance "power".
>>
>> I don't think opening tickets will do any good. I've done it in the past, 
>> when I reported an issue with task queue rate (Issue #12620, acknowledged 
>> since December 17, 2015). Resolution -> NONE.
>>
>> I really love the AppEngine concept and development environment. But I 
>> doubt Google's devotion to it and that makes me really sad.
>>
>>
>> On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform 
>> Support) wrote:
>>>
>>> Hey Pasha,
>>>
>>> I would suggest posting a more detailed technical report to the Public 
>>> Issue Tracker . 
>>> The Public Issue Tracker is much better suited to issues like this, while 
>>> this forum generally should be geared toward high-level discussion of the 
>>> platform and services, design patterns, etc.
>>>
>>> You should try to include some or all of the following:
>>>
>>> 1. your project ID
>>> 2. the task queue names 
>>> 3. their configurations
>>> 4. the code for and/or a description of the task handler
>>> 5. the time-frames when the issue is observed to occur within the past 
>>> few days (since time is of the essence in making sure the logs still exist 
>>> in the system)
>>> 6. a description of the average load and the frequency distribution of 
>>> different sized loads
>>> 7. the frequency distribution of response latencies for the task handlers
>>>
>>> This will enable us to get started on reasoning about the issue and 
>>> looking into it. Feel free to post a link to the thread here once you've 
>>> done that. I'll be happy to assist.
>>>
>>> Cheers,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>> On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov 
>>> wrote:

 Hi Nick,
 I have the same issue with queue not executing tasks at rate set in 
 configuration. It looks really strange because the execution works 
 correctly most of the days, but once per week the queue does not execute 
 at 
 correct rate. I experience the issue on  Sep 30th and today Oct 6.

 I tried different configuration settings, but the configuration changes 
 did affect the real execution rate. The execution is way too slow than set 
 in configuration.

 Best regards,
 Pasha

 On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform 
 Support) wrote:
>
> Thanks for the update. Barring a very low setting for "max concurrent 
> tasks", I don't see why this could occur, so it appears this could be a 
> production issue. In that case, I'd recommend opening a Public Issue 
> Tracker  thread 
> with label "Restrict-View-EditIssue" (to protect your information) and 
> the 
> following information: 
>
> * your app id
> * the name of the queue
> * the queue settings
> * some queue statistics
>
> We may be able to determine the root cause of this at that 

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Thanasis,

Does that latency correspond to the time spent in the handlers that deal 
with the task queue request itself? Or, if not, what underlies the metric?

Cheers,

Nick
Cloud Platform Community Support

On Friday, October 7, 2016 at 3:55:04 PM UTC-4, Thanasis Delenikas wrote:
>
> Hi Nick / Google,
>
> During the last 1-2 weeks, I've had the same problem. I've *never* 
> encountered a backlog on my queues (load is small and processing generally 
> fast). Then, suddenly and out of the blue, with *no* application or other 
> changes, I get the behavior displayed on the attached graph (graph from 
> Stackdriver, queue latency, 1month graph). It really crippled me!
>
> I don't believe that this issue is related to my project. It must be a 
> generic issue that you should be aware of. I've even tried with different 
> AppEngine instance classes, but this "delay" had nothing to do with the 
> instance "power".
>
> I don't think opening tickets will do any good. I've done it in the past, 
> when I reported an issue with task queue rate (Issue #12620, acknowledged 
> since December 17, 2015). Resolution -> NONE.
>
> I really love the AppEngine concept and development environment. But I 
> doubt Google's devotion to it and that makes me really sad.
>
>
> On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform 
> Support) wrote:
>>
>> Hey Pasha,
>>
>> I would suggest posting a more detailed technical report to the Public 
>> Issue Tracker . 
>> The Public Issue Tracker is much better suited to issues like this, while 
>> this forum generally should be geared toward high-level discussion of the 
>> platform and services, design patterns, etc.
>>
>> You should try to include some or all of the following:
>>
>> 1. your project ID
>> 2. the task queue names 
>> 3. their configurations
>> 4. the code for and/or a description of the task handler
>> 5. the time-frames when the issue is observed to occur within the past 
>> few days (since time is of the essence in making sure the logs still exist 
>> in the system)
>> 6. a description of the average load and the frequency distribution of 
>> different sized loads
>> 7. the frequency distribution of response latencies for the task handlers
>>
>> This will enable us to get started on reasoning about the issue and 
>> looking into it. Feel free to post a link to the thread here once you've 
>> done that. I'll be happy to assist.
>>
>> Cheers,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov wrote:
>>>
>>> Hi Nick,
>>> I have the same issue with queue not executing tasks at rate set in 
>>> configuration. It looks really strange because the execution works 
>>> correctly most of the days, but once per week the queue does not execute at 
>>> correct rate. I experience the issue on  Sep 30th and today Oct 6.
>>>
>>> I tried different configuration settings, but the configuration changes 
>>> did affect the real execution rate. The execution is way too slow than set 
>>> in configuration.
>>>
>>> Best regards,
>>> Pasha
>>>
>>> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform 
>>> Support) wrote:

 Thanks for the update. Barring a very low setting for "max concurrent 
 tasks", I don't see why this could occur, so it appears this could be a 
 production issue. In that case, I'd recommend opening a Public Issue 
 Tracker  thread 
 with label "Restrict-View-EditIssue" (to protect your information) and the 
 following information: 

 * your app id
 * the name of the queue
 * the queue settings
 * some queue statistics

 We may be able to determine the root cause of this at that point. 

 Sincerely,

 Nick
 Cloud Platform Community Support

 On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:
>
> Just got an alert again. The task rate is 25 tasks/sec with a bucket 
> size of 10. Average latency of the handlers is varying depending on the 
> task, but anywhere between 1 and 8 seconds.
>
> The thing that I'm seeing is that the task queue shows 120 tasks in 
> the queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem 
> right to me, and it looks like these tasks are just not being executed. I 
> see nothing in the logs of my task handlers that indicate that a request 
> is 
> being executed. 
>
> Obviously this task execution thing is pretty essential to my app.
>
> Hugo
>
> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
> Support) wrote:
>>
>> Thanks for coming back to update. Feel free to post again with the 
>> information mentioned if you notice anything in the future. 
>>
>> Best wishes,
>>
>> Nick
>> Cloud Platform Community Support

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Thanasis Delenikas]

Hi Nick / Google,

During the last 1-2 weeks, I've had the same problem. I've *never* 
encountered a backlog on my queues (load is small and processing generally 
fast). Then, suddenly and out of the blue, with *no* application or other 
changes, I get the behavior displayed on the attached graph (graph from 
Stackdriver, queue latency, 1month graph). It really crippled me!

I don't believe that this issue is related to my project. It must be a 
generic issue that you should be aware of. I've even tried with different 
AppEngine instance classes, but this "delay" had nothing to do with the 
instance "power".

I don't think opening tickets will do any good. I've done it in the past, 
when I reported an issue with task queue rate (Issue #12620, acknowledged 
since December 17, 2015). Resolution -> NONE.

I really love the AppEngine concept and development environment. But I 
doubt Google's devotion to it and that makes me really sad.


On Friday, October 7, 2016 at 1:45:15 AM UTC+3, Nick (Cloud Platform 
Support) wrote:
>
> Hey Pasha,
>
> I would suggest posting a more detailed technical report to the Public 
> Issue Tracker . The 
> Public Issue Tracker is much better suited to issues like this, while this 
> forum generally should be geared toward high-level discussion of the 
> platform and services, design patterns, etc.
>
> You should try to include some or all of the following:
>
> 1. your project ID
> 2. the task queue names 
> 3. their configurations
> 4. the code for and/or a description of the task handler
> 5. the time-frames when the issue is observed to occur within the past few 
> days (since time is of the essence in making sure the logs still exist in 
> the system)
> 6. a description of the average load and the frequency distribution of 
> different sized loads
> 7. the frequency distribution of response latencies for the task handlers
>
> This will enable us to get started on reasoning about the issue and 
> looking into it. Feel free to post a link to the thread here once you've 
> done that. I'll be happy to assist.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
> On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov wrote:
>>
>> Hi Nick,
>> I have the same issue with queue not executing tasks at rate set in 
>> configuration. It looks really strange because the execution works 
>> correctly most of the days, but once per week the queue does not execute at 
>> correct rate. I experience the issue on  Sep 30th and today Oct 6.
>>
>> I tried different configuration settings, but the configuration changes 
>> did affect the real execution rate. The execution is way too slow than set 
>> in configuration.
>>
>> Best regards,
>> Pasha
>>
>> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform 
>> Support) wrote:
>>>
>>> Thanks for the update. Barring a very low setting for "max concurrent 
>>> tasks", I don't see why this could occur, so it appears this could be a 
>>> production issue. In that case, I'd recommend opening a Public Issue 
>>> Tracker  thread 
>>> with label "Restrict-View-EditIssue" (to protect your information) and the 
>>> following information: 
>>>
>>> * your app id
>>> * the name of the queue
>>> * the queue settings
>>> * some queue statistics
>>>
>>> We may be able to determine the root cause of this at that point. 
>>>
>>> Sincerely,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:

 Just got an alert again. The task rate is 25 tasks/sec with a bucket 
 size of 10. Average latency of the handlers is varying depending on the 
 task, but anywhere between 1 and 8 seconds.

 The thing that I'm seeing is that the task queue shows 120 tasks in the 
 queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right 
 to 
 me, and it looks like these tasks are just not being executed. I see 
 nothing in the logs of my task handlers that indicate that a request is 
 being executed. 

 Obviously this task execution thing is pretty essential to my app.

 Hugo

 On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
 Support) wrote:
>
> Thanks for coming back to update. Feel free to post again with the 
> information mentioned if you notice anything in the future. 
>
> Best wishes,
>
> Nick
> Cloud Platform Community Support
>
> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>>
>> It looks like the queues are back to normal. I've setup alerts in 
>> stack driver that triggered for a while but are resolved now. Will 
>> report 
>> back if that changes.
>>
>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via 
>> Google App Engine  wrote:

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nick (Cloud Platform Support)' via Google App Engine]

Hey Pasha,

I would suggest posting a more detailed technical report to the Public 
Issue Tracker . The 
Public Issue Tracker is much better suited to issues like this, while this 
forum generally should be geared toward high-level discussion of the 
platform and services, design patterns, etc.

You should try to include some or all of the following:

1. your project ID
2. the task queue names 
3. their configurations
4. the code for and/or a description of the task handler
5. the time-frames when the issue is observed to occur within the past few 
days (since time is of the essence in making sure the logs still exist in 
the system)
6. a description of the average load and the frequency distribution of 
different sized loads
7. the frequency distribution of response latencies for the task handlers

This will enable us to get started on reasoning about the issue and looking 
into it. Feel free to post a link to the thread here once you've done that. 
I'll be happy to assist.

Cheers,

Nick
Cloud Platform Community Support

On Thursday, October 6, 2016 at 5:26:56 PM UTC-4, Pasha Seliverstov wrote:
>
> Hi Nick,
> I have the same issue with queue not executing tasks at rate set in 
> configuration. It looks really strange because the execution works 
> correctly most of the days, but once per week the queue does not execute at 
> correct rate. I experience the issue on  Sep 30th and today Oct 6.
>
> I tried different configuration settings, but the configuration changes 
> did affect the real execution rate. The execution is way too slow than set 
> in configuration.
>
> Best regards,
> Pasha
>
> On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform 
> Support) wrote:
>>
>> Thanks for the update. Barring a very low setting for "max concurrent 
>> tasks", I don't see why this could occur, so it appears this could be a 
>> production issue. In that case, I'd recommend opening a Public Issue 
>> Tracker  thread 
>> with label "Restrict-View-EditIssue" (to protect your information) and the 
>> following information: 
>>
>> * your app id
>> * the name of the queue
>> * the queue settings
>> * some queue statistics
>>
>> We may be able to determine the root cause of this at that point. 
>>
>> Sincerely,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:
>>>
>>> Just got an alert again. The task rate is 25 tasks/sec with a bucket 
>>> size of 10. Average latency of the handlers is varying depending on the 
>>> task, but anywhere between 1 and 8 seconds.
>>>
>>> The thing that I'm seeing is that the task queue shows 120 tasks in the 
>>> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
>>> me, and it looks like these tasks are just not being executed. I see 
>>> nothing in the logs of my task handlers that indicate that a request is 
>>> being executed. 
>>>
>>> Obviously this task execution thing is pretty essential to my app.
>>>
>>> Hugo
>>>
>>> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
>>> Support) wrote:

 Thanks for coming back to update. Feel free to post again with the 
 information mentioned if you notice anything in the future. 

 Best wishes,

 Nick
 Cloud Platform Community Support

 On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>
> It looks like the queues are back to normal. I've setup alerts in 
> stack driver that triggered for a while but are resolved now. Will report 
> back if that changes.
>
> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
> App Engine  wrote:
>
>> Hey Hugo,
>>
>> Without knowing more about A) the configuration of the queue, B) the 
>> rate of task creation, C) the latency statistics on the task handler, 
>> it's 
>> difficult to say much on why this might be happening. If you could 
>> gather 
>> all those types of information, myself or someone else may be able to 
>> leave 
>> a more insightful comment.
>>
>> Cheers,
>>
>> Nick
>> Cloud Platform Community Support
>>
>>
>> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>>>
>>> I'm seeing tasks in my queue that are there for 6 minutes without 
>>> any tasks running at all. There doesn't seem to be much I can do about 
>>> this 
>>> which is really frustrating.
>>>
>>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:

 Since last night or so I'm seeing this issue where tasks are being 
 queued up for some time and not being executed on an app that has been 
 running like this for quite a while. This results in a huge pile of 
 tasks 
 that get backlogged for my app.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Pasha Seliverstov]

Hi Nick,
I have the same issue with queue not executing tasks at rate set in 
configuration. It looks really strange because the execution works 
correctly most of the days, but once per week the queue does not execute at 
correct rate. I experience the issue on  Sep 30th and today Oct 6.

I tried different configuration settings, but the configuration changes did 
affect the real execution rate. The execution is way too slow than set in 
configuration.

Best regards,
Pasha

On Friday, May 13, 2016 at 11:44:57 AM UTC-7, Nick (Cloud Platform Support) 
wrote:
>
> Thanks for the update. Barring a very low setting for "max concurrent 
> tasks", I don't see why this could occur, so it appears this could be a 
> production issue. In that case, I'd recommend opening a Public Issue 
> Tracker  thread 
> with label "Restrict-View-EditIssue" (to protect your information) and the 
> following information: 
>
> * your app id
> * the name of the queue
> * the queue settings
> * some queue statistics
>
> We may be able to determine the root cause of this at that point. 
>
> Sincerely,
>
> Nick
> Cloud Platform Community Support
>
> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:
>>
>> Just got an alert again. The task rate is 25 tasks/sec with a bucket size 
>> of 10. Average latency of the handlers is varying depending on the task, 
>> but anywhere between 1 and 8 seconds.
>>
>> The thing that I'm seeing is that the task queue shows 120 tasks in the 
>> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
>> me, and it looks like these tasks are just not being executed. I see 
>> nothing in the logs of my task handlers that indicate that a request is 
>> being executed. 
>>
>> Obviously this task execution thing is pretty essential to my app.
>>
>> Hugo
>>
>> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
>> Support) wrote:
>>>
>>> Thanks for coming back to update. Feel free to post again with the 
>>> information mentioned if you notice anything in the future. 
>>>
>>> Best wishes,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:

 It looks like the queues are back to normal. I've setup alerts in stack 
 driver that triggered for a while but are resolved now. Will report back 
 if 
 that changes.

 On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
 App Engine  wrote:

> Hey Hugo,
>
> Without knowing more about A) the configuration of the queue, B) the 
> rate of task creation, C) the latency statistics on the task handler, 
> it's 
> difficult to say much on why this might be happening. If you could gather 
> all those types of information, myself or someone else may be able to 
> leave 
> a more insightful comment.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
>
> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>>
>> I'm seeing tasks in my queue that are there for 6 minutes without any 
>> tasks running at all. There doesn't seem to be much I can do about this 
>> which is really frustrating.
>>
>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>>>
>>> Since last night or so I'm seeing this issue where tasks are being 
>>> queued up for some time and not being executed on an app that has been 
>>> running like this for quite a while. This results in a huge pile of 
>>> tasks 
>>> that get backlogged for my app.
>>>
>>> Cloud status dashboard says everything is OK. Anyone else seeing 
>>> this?
>>>
>>> Hugo
>>>
>> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to 
> google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Hugo Visser]

This has been happening on and off but today it's really off the charts. 
Currently have 28k tasks and none of them are executed. I filed #13118 for 
that.

On Friday, May 13, 2016 at 8:44:57 PM UTC+2, Nick (Cloud Platform Support) 
wrote:
>
> Thanks for the update. Barring a very low setting for "max concurrent 
> tasks", I don't see why this could occur, so it appears this could be a 
> production issue. In that case, I'd recommend opening a Public Issue 
> Tracker  thread 
> with label "Restrict-View-EditIssue" (to protect your information) and the 
> following information: 
>
> * your app id
> * the name of the queue
> * the queue settings
> * some queue statistics
>
> We may be able to determine the root cause of this at that point. 
>
> Sincerely,
>
> Nick
> Cloud Platform Community Support
>
> On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:
>>
>> Just got an alert again. The task rate is 25 tasks/sec with a bucket size 
>> of 10. Average latency of the handlers is varying depending on the task, 
>> but anywhere between 1 and 8 seconds.
>>
>> The thing that I'm seeing is that the task queue shows 120 tasks in the 
>> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
>> me, and it looks like these tasks are just not being executed. I see 
>> nothing in the logs of my task handlers that indicate that a request is 
>> being executed. 
>>
>> Obviously this task execution thing is pretty essential to my app.
>>
>> Hugo
>>
>> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
>> Support) wrote:
>>>
>>> Thanks for coming back to update. Feel free to post again with the 
>>> information mentioned if you notice anything in the future. 
>>>
>>> Best wishes,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:

 It looks like the queues are back to normal. I've setup alerts in stack 
 driver that triggered for a while but are resolved now. Will report back 
 if 
 that changes.

 On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
 App Engine  wrote:

> Hey Hugo,
>
> Without knowing more about A) the configuration of the queue, B) the 
> rate of task creation, C) the latency statistics on the task handler, 
> it's 
> difficult to say much on why this might be happening. If you could gather 
> all those types of information, myself or someone else may be able to 
> leave 
> a more insightful comment.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
>
> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>>
>> I'm seeing tasks in my queue that are there for 6 minutes without any 
>> tasks running at all. There doesn't seem to be much I can do about this 
>> which is really frustrating.
>>
>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>>>
>>> Since last night or so I'm seeing this issue where tasks are being 
>>> queued up for some time and not being executed on an app that has been 
>>> running like this for quite a while. This results in a huge pile of 
>>> tasks 
>>> that get backlogged for my app.
>>>
>>> Cloud status dashboard says everything is OK. Anyone else seeing 
>>> this?
>>>
>>> Hugo
>>>
>> -- 
> You received this message because you are subscribed to a topic in the 
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to 
> google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
>  
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/c982de60-4fb9-4d33-ab47-fd3cf160be78%40googlegroups.com.
For more options, visit 

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nick (Cloud Platform Support)' via Google App Engine]

Thanks for the update. Barring a very low setting for "max concurrent 
tasks", I don't see why this could occur, so it appears this could be a 
production issue. In that case, I'd recommend opening a Public Issue Tracker 
 thread with label 
"Restrict-View-EditIssue" (to protect your information) and the following 
information: 

* your app id
* the name of the queue
* the queue settings
* some queue statistics

We may be able to determine the root cause of this at that point. 

Sincerely,

Nick
Cloud Platform Community Support

On Friday, May 13, 2016 at 2:02:04 PM UTC-4, Hugo Visser wrote:
>
> Just got an alert again. The task rate is 25 tasks/sec with a bucket size 
> of 10. Average latency of the handlers is varying depending on the task, 
> but anywhere between 1 and 8 seconds.
>
> The thing that I'm seeing is that the task queue shows 120 tasks in the 
> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
> me, and it looks like these tasks are just not being executed. I see 
> nothing in the logs of my task handlers that indicate that a request is 
> being executed. 
>
> Obviously this task execution thing is pretty essential to my app.
>
> Hugo
>
> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
> Support) wrote:
>>
>> Thanks for coming back to update. Feel free to post again with the 
>> information mentioned if you notice anything in the future. 
>>
>> Best wishes,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>>>
>>> It looks like the queues are back to normal. I've setup alerts in stack 
>>> driver that triggered for a while but are resolved now. Will report back if 
>>> that changes.
>>>
>>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
>>> App Engine  wrote:
>>>
 Hey Hugo,

 Without knowing more about A) the configuration of the queue, B) the 
 rate of task creation, C) the latency statistics on the task handler, it's 
 difficult to say much on why this might be happening. If you could gather 
 all those types of information, myself or someone else may be able to 
 leave 
 a more insightful comment.

 Cheers,

 Nick
 Cloud Platform Community Support


 On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>
> I'm seeing tasks in my queue that are there for 6 minutes without any 
> tasks running at all. There doesn't seem to be much I can do about this 
> which is really frustrating.
>
> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>>
>> Since last night or so I'm seeing this issue where tasks are being 
>> queued up for some time and not being executed on an app that has been 
>> running like this for quite a while. This results in a huge pile of 
>> tasks 
>> that get backlogged for my app.
>>
>> Cloud status dashboard says everything is OK. Anyone else seeing this?
>>
>> Hugo
>>
> -- 
 You received this message because you are subscribed to a topic in the 
 Google Groups "Google App Engine" group.
 To unsubscribe from this topic, visit 
 https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
 .
 To unsubscribe from this group and all its topics, send an email to 
 google-appengi...@googlegroups.com.
 To post to this group, send email to google-a...@googlegroups.com.
 Visit this group at https://groups.google.com/group/google-appengine.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/757eb42f-5192-4ff3-94d7-1d98b08340be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Hugo Visser]

Hitting the "run now" button makes the task run by the way. I don't see any 
relation to error rates on the task handler instances either. Looks like 
they are just stuck there waiting for execution.

On Friday, May 13, 2016 at 8:02:04 PM UTC+2, Hugo Visser wrote:
>
> Just got an alert again. The task rate is 25 tasks/sec with a bucket size 
> of 10. Average latency of the handlers is varying depending on the task, 
> but anywhere between 1 and 8 seconds.
>
> The thing that I'm seeing is that the task queue shows 120 tasks in the 
> queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
> me, and it looks like these tasks are just not being executed. I see 
> nothing in the logs of my task handlers that indicate that a request is 
> being executed. 
>
> Obviously this task execution thing is pretty essential to my app.
>
> Hugo
>
> On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
> Support) wrote:
>>
>> Thanks for coming back to update. Feel free to post again with the 
>> information mentioned if you notice anything in the future. 
>>
>> Best wishes,
>>
>> Nick
>> Cloud Platform Community Support
>>
>> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>>>
>>> It looks like the queues are back to normal. I've setup alerts in stack 
>>> driver that triggered for a while but are resolved now. Will report back if 
>>> that changes.
>>>
>>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
>>> App Engine  wrote:
>>>
 Hey Hugo,

 Without knowing more about A) the configuration of the queue, B) the 
 rate of task creation, C) the latency statistics on the task handler, it's 
 difficult to say much on why this might be happening. If you could gather 
 all those types of information, myself or someone else may be able to 
 leave 
 a more insightful comment.

 Cheers,

 Nick
 Cloud Platform Community Support


 On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>
> I'm seeing tasks in my queue that are there for 6 minutes without any 
> tasks running at all. There doesn't seem to be much I can do about this 
> which is really frustrating.
>
> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>>
>> Since last night or so I'm seeing this issue where tasks are being 
>> queued up for some time and not being executed on an app that has been 
>> running like this for quite a while. This results in a huge pile of 
>> tasks 
>> that get backlogged for my app.
>>
>> Cloud status dashboard says everything is OK. Anyone else seeing this?
>>
>> Hugo
>>
> -- 
 You received this message because you are subscribed to a topic in the 
 Google Groups "Google App Engine" group.
 To unsubscribe from this topic, visit 
 https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
 .
 To unsubscribe from this group and all its topics, send an email to 
 google-appengi...@googlegroups.com.
 To post to this group, send email to google-a...@googlegroups.com.
 Visit this group at https://groups.google.com/group/google-appengine.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
  
 
 .
 For more options, visit https://groups.google.com/d/optout.

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/25f4084e-0759-4c46-aaba-63bb90f9541b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Hugo Visser]

Just got an alert again. The task rate is 25 tasks/sec with a bucket size 
of 10. Average latency of the handlers is varying depending on the task, 
but anywhere between 1 and 8 seconds.

The thing that I'm seeing is that the task queue shows 120 tasks in the 
queue, 0 running, oldest task ETA 6 minutes ago. That doesn't seem right to 
me, and it looks like these tasks are just not being executed. I see 
nothing in the logs of my task handlers that indicate that a request is 
being executed. 

Obviously this task execution thing is pretty essential to my app.

Hugo

On Wednesday, May 11, 2016 at 11:34:34 PM UTC+2, Nick (Cloud Platform 
Support) wrote:
>
> Thanks for coming back to update. Feel free to post again with the 
> information mentioned if you notice anything in the future. 
>
> Best wishes,
>
> Nick
> Cloud Platform Community Support
>
> On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>>
>> It looks like the queues are back to normal. I've setup alerts in stack 
>> driver that triggered for a while but are resolved now. Will report back if 
>> that changes.
>>
>> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google 
>> App Engine  wrote:
>>
>>> Hey Hugo,
>>>
>>> Without knowing more about A) the configuration of the queue, B) the 
>>> rate of task creation, C) the latency statistics on the task handler, it's 
>>> difficult to say much on why this might be happening. If you could gather 
>>> all those types of information, myself or someone else may be able to leave 
>>> a more insightful comment.
>>>
>>> Cheers,
>>>
>>> Nick
>>> Cloud Platform Community Support
>>>
>>>
>>> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:

 I'm seeing tasks in my queue that are there for 6 minutes without any 
 tasks running at all. There doesn't seem to be much I can do about this 
 which is really frustrating.

 On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>
> Since last night or so I'm seeing this issue where tasks are being 
> queued up for some time and not being executed on an app that has been 
> running like this for quite a while. This results in a huge pile of tasks 
> that get backlogged for my app.
>
> Cloud status dashboard says everything is OK. Anyone else seeing this?
>
> Hugo
>
 -- 
>>> You received this message because you are subscribed to a topic in the 
>>> Google Groups "Google App Engine" group.
>>> To unsubscribe from this topic, visit 
>>> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
>>> .
>>> To unsubscribe from this group and all its topics, send an email to 
>>> google-appengi...@googlegroups.com .
>>> To post to this group, send email to google-a...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
>>>  
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/29d999c4-10fd-4c5c-9f13-fb185aa3be32%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

['Nick (Cloud Platform Support)' via Google App Engine]

Thanks for coming back to update. Feel free to post again with the 
information mentioned if you notice anything in the future. 

Best wishes,

Nick
Cloud Platform Community Support

On Wednesday, May 11, 2016 at 4:33:36 PM UTC-4, Hugo Visser wrote:
>
> It looks like the queues are back to normal. I've setup alerts in stack 
> driver that triggered for a while but are resolved now. Will report back if 
> that changes.
>
> On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google App 
> Engine  wrote:
>
>> Hey Hugo,
>>
>> Without knowing more about A) the configuration of the queue, B) the rate 
>> of task creation, C) the latency statistics on the task handler, it's 
>> difficult to say much on why this might be happening. If you could gather 
>> all those types of information, myself or someone else may be able to leave 
>> a more insightful comment.
>>
>> Cheers,
>>
>> Nick
>> Cloud Platform Community Support
>>
>>
>> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>>>
>>> I'm seeing tasks in my queue that are there for 6 minutes without any 
>>> tasks running at all. There doesn't seem to be much I can do about this 
>>> which is really frustrating.
>>>
>>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:

 Since last night or so I'm seeing this issue where tasks are being 
 queued up for some time and not being executed on an app that has been 
 running like this for quite a while. This results in a huge pile of tasks 
 that get backlogged for my app.

 Cloud status dashboard says everything is OK. Anyone else seeing this?

 Hugo

>>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "Google App Engine" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to 
>> google-appengine+unsubscr...@googlegroups.com.
>> To post to this group, send email to google-appengine@googlegroups.com.
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
>>  
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7e46b65e-5360-42e2-a3a9-6a3c6c9492ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Task queue not executing tasks / backlogged

[Hugo Visser]

It looks like the queues are back to normal. I've setup alerts in stack
driver that triggered for a while but are resolved now. Will report back if
that changes.

On Wed, May 11, 2016, 21:08 'Nick (Cloud Platform Support)' via Google App
Engine  wrote:

> Hey Hugo,
>
> Without knowing more about A) the configuration of the queue, B) the rate
> of task creation, C) the latency statistics on the task handler, it's
> difficult to say much on why this might be happening. If you could gather
> all those types of information, myself or someone else may be able to leave
> a more insightful comment.
>
> Cheers,
>
> Nick
> Cloud Platform Community Support
>
>
> On Tuesday, May 10, 2016 at 4:54:00 AM UTC-4, Hugo Visser wrote:
>>
>> I'm seeing tasks in my queue that are there for 6 minutes without any
>> tasks running at all. There doesn't seem to be much I can do about this
>> which is really frustrating.
>>
>> On Tuesday, May 10, 2016 at 8:54:33 AM UTC+2, Hugo Visser wrote:
>>>
>>> Since last night or so I'm seeing this issue where tasks are being
>>> queued up for some time and not being executed on an app that has been
>>> running like this for quite a while. This results in a huge pile of tasks
>>> that get backlogged for my app.
>>>
>>> Cloud status dashboard says everything is OK. Anyone else seeing this?
>>>
>>> Hugo
>>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Google App Engine" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/google-appengine/MYxDL8sihN4/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/370b31f3-e1fa-4a2f-9287-652aeac0500c%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAMSc4X2ntazGDjxzG4yqS9_ENECNtFRVrAbopW6Jrf4b_SeAeg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.