Okay, I figured it out. It’s that old datastore bug where it throws timeout exceptions when you try to iterate on a .all() with a lot of items.
I’m using the workaround that I only iterate over some of them, then I set a cursor, do the .all() again, move forward to the cursor, and then keep going. Usually that works, but sometimes it will still throw a timeout exception, so I have it in a task so it gets retried. Something must have changed in the datastore, so the workaround was consistently NOT working around the problem. So the task ran over and over, eventually hitting some undocumented quota limit. I just lowered the number of items I’ll go through, and that seems to have gotten the workaround working again. -Joshua > On Jun 14, 2016, at 3:37 PM, Joshua Smith <[email protected]> wrote: > > I’ve tracked down the thing that threw the first over quota exception. It was > a nightly cron that does a lot of data roll-ups. But I don’t understand why > that would throw a quota exception. I have no spending limits set. You > mention per-minute safety limits. What are those? The specific quota is > probably datastore reads. > > My app basically does the same thing every night. Why would this suddenly be > too much? > >> On Jun 14, 2016, at 3:13 PM, 'Nicholas (Google Cloud Support)' via Google >> App Engine <[email protected] >> <mailto:[email protected]>> wrote: >> >> According to the Status Dashboard <https://status.cloud.google.com/>, there >> were no known issues regarding quotas yesterday at all. >> >> As shown here >> <https://cloud.google.com/appengine/docs/quotas#Safety_Quotas_and_Billable_Quotas>, >> various quotas may be encountered by a given application. Even with >> billing enabled, one can still encounter OverQuotaErrors by reaching >> spending limits >> <https://cloud.google.com/appengine/docs/python/console/#billing> or >> per-minute safety limits. If quota error are encountered today, you can >> view them in the Developers Console by going to the App Engine dashboard and >> selecting Quotas. If encountered prior to today, you can select VIEW USAGE >> HISTORY. Here, you can view many months worth of daily usage reports. >> Though it won't reveal usage over quotas, you can see from the previous page >> how much usage is considered over quota and compare that way. >> >> In addition, the errors encountering should be searchable in your logs. The >> article When a Resource is Depleted >> <https://cloud.google.com/appengine/docs/quotas#When_a_Resource_is_Depleted> >> demonstrates how to catch such exceptions at runtime. This can be used in >> conjunction with admin emails to be made aware of such occurrences as soon >> as they happen in the future. >> >> Hope this helps! >> >> On Tuesday, June 14, 2016 at 9:32:27 AM UTC-4, Joshua Smith wrote: >> One of my apps threw a few hundred “Over Quota” errors last night. >> >> However, the app has billing enabled, I’m not aware of any quotas being >> enforced, and the bills show negligible usage. Was there some kind of an >> outage last night that google didn’t report? So far all I can find in the >> logs are the stack traces of the specific requests that were denied, but I >> can’t figure out why google thought my app was over quota. However, the new >> console makes finding pretty much anything pretty much impossible, so it’s >> entirely possible the information I need is there and I just can’t find it. >> >> Advice on how to proceed? >> >> -Joshua >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <mailto:[email protected]>. >> To post to this group, send email to [email protected] >> <mailto:[email protected]>. >> Visit this group at https://groups.google.com/group/google-appengine >> <https://groups.google.com/group/google-appengine>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/google-appengine/78f541d0-5ad9-4952-91d3-e0b9c9ed023f%40googlegroups.com >> >> <https://groups.google.com/d/msgid/google-appengine/78f541d0-5ad9-4952-91d3-e0b9c9ed023f%40googlegroups.com?utm_medium=email&utm_source=footer>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/40C26F9B-80D1-4F13-866B-B039A1458E2A%40gmail.com. For more options, visit https://groups.google.com/d/optout.
