Again, the rate setting of default queue is changed to 5 QPS for both of my apps. This kind of random behavior is not acceptable if GAE would like to be a serious cloud platform.
On Thursday, October 18, 2012 7:57:56 PM UTC-7, gks wrote: > > Following are the answers to your questions: > 1. Rate: Both maximum rate and enforced rate were changed to 5 QPS. I was > only using default queue and set 20 QPS on it. > 2. Task: All my tasks run on frontends > 3. Admin Log: Yes, I am the only developer uploading changes and there is > no activity in admin log when the setting was changed. > > In addition, when I tried to change the setting to 100 QPS this morning in > order to drain the queue, it was always changed back to 5 QPS about 5 > minutes later after the deployment. Finally I found a workaround to create > another queue with the same setting which stopped the madness of > automatically configuration change. > > Actually task queue has been causing service interruption to me in the > past 2 days. Yesterday, the default queue suddenly stopped accepting any > task with name, which I had to workaround in order to bring my service back > to normal state. I was too tired to write a post here about the issue after > I finished live site support yesterday. Since you are you investigating, > would you please also look into this issue as well? I really need the > feature to add task with name so that I can avoid duplicated task > processing. > > On Thursday, October 18, 2012 7:11:39 PM UTC-7, nverne wrote: >> >> Could you please give us some more details: >> >> Which rate did you see go down? The Maximum Rate or the Enforced Rate? >> >> Were your tasks running on backends or frontends? >> >> Do your admin logs indicate that you were the only developer uploading >> changes? >> >> On Fri, Oct 19, 2012 at 12:59 PM, nischalshetty <[email protected]>wrote: >> >>> We are facing problems with the task queue as well. The tasks shot up to >>> millions and everything stopped working for us! >>> >>> >>> On Thursday, October 18, 2012 11:34:23 PM UTC+5:30, gks wrote: >>>> >>>> And the task queue of my app aos-prod-1 is lying about the # of tasks >>>> in queue since it always says 100, which is impossible given my customers >>>> usage pattern. >>>> >>>> On Thursday, October 18, 2012 10:19:09 AM UTC-7, gks wrote: >>>>> >>>>> After I changed the processing rate to 100QPS to drain the queue, it >>>>> was changed back to 5QPS just now. Can anyone from Google start looking >>>>> into this issue and stop changing my task queue setting? >>>>> >>>>> On Thursday, October 18, 2012 9:42:17 AM UTC-7, gks wrote: >>>>>> >>>>>> I just found task queue processing rate of my apps was changed to >>>>>> 5qps last night without any deployment from me! It caused a major >>>>>> service >>>>>> interruption to my customers since a lot of tasks were delayed. >>>>>> >>>>>> There is no way this change came from my team since everyone was >>>>>> sleeping when the change happened. And there is no corresponding >>>>>> deployment >>>>>> activity from admin log. Can Google provide an explanation on why it >>>>>> happened, and what can be done to prevent it from happening again? >>>>>> >>>>>> My AppID are: aos-prod-1, aos-prod-2 >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google App Engine" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/google-appengine/-/SIo8pCc0-WYJ. >>> >>> To post to this group, send email to [email protected]. >>> To unsubscribe from this group, send email to >>> [email protected]. >>> For more options, visit this group at >>> http://groups.google.com/group/google-appengine?hl=en. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/M5g2CKz1A80J. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
