Re: [google-appengine] Spending LImits Going Away :(

['yananc' via Google App Engine]

Hello Joshua,

Thank you for your response. I agree with you on the point that caching 
won’t be of much help if repeat hits are quite limited. Besides the 
recommendations aforementioned, I suggest you to open tech support tickets 
with GCP support so we could be able to dig deep into your projects for any 
possible improvements.

@Linus, thank you for sharing your experience. In such a scenario, setting 
the max number of instances for autoscaling could greatly help, which is 
part of the recommendations 
 to control App 
Engine-related costs. 


On Thursday, August 27, 2020 at 11:32:26 AM UTC-4 Joshua Smith wrote:

> On this particular topic of Cloud Datastore Read Operations being the cost 
> driver, unfortunately caching doesn’t help my case.
>
> This site has about 60,000 individual meetings listed, and I like having 
> *useful* *well-behaved* crawlers find all those meetings so people can use 
> search engines to find meetings that happened in various towns. But what 
> keeps happening is some new useless/poorly-behaved crawler will decide to 
> read all 60,000 of those meetings as fast as it possibly can. (Ignoring the 
> crawl-delay.) Caching in that situation would just add *more* cost, since 
> there are no repeat hits from a crawler.
>
> I periodically look at my access logs, find peaks, look at the requests, 
> and figure out what new bot needs to be added to my robots.txt disallow 
> list. And sometimes I have to add a firewall rule because that bot isn’t 
> obeying robots.txt at all.
>
> -Joshua
>
> On Aug 26, 2020, at 4:47 PM, 'yananc' via Google App Engine <
> google-a...@googlegroups.com> wrote:
>
> Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a 
> possible solution is to leverage cache mechanism to avoid excessive 
> operations. You may find more information from the topic [2].
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/f6b45f06-f6e1-48d2-8fb2-b33f91e01324n%40googlegroups.com.

Re: [google-appengine] Spending LImits Going Away :(

[Joshua Smith]

On this particular topic of Cloud Datastore Read Operations being the cost 
driver, unfortunately caching doesn’t help my case.

This site has about 60,000 individual meetings listed, and I like having 
*useful* *well-behaved* crawlers find all those meetings so people can use 
search engines to find meetings that happened in various towns. But what keeps 
happening is some new useless/poorly-behaved crawler will decide to read all 
60,000 of those meetings as fast as it possibly can. (Ignoring the 
crawl-delay.) Caching in that situation would just add *more* cost, since there 
are no repeat hits from a crawler.

I periodically look at my access logs, find peaks, look at the requests, and 
figure out what new bot needs to be added to my robots.txt disallow list. And 
sometimes I have to add a firewall rule because that bot isn’t obeying 
robots.txt at all.

-Joshua

> On Aug 26, 2020, at 4:47 PM, 'yananc' via Google App Engine 
>  wrote:
> 
> Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a 
> possible solution is to leverage cache mechanism to avoid excessive 
> operations. You may find more information from the topic [2].

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/068BD581-9C44-4E59-A235-61DD0239FFBA%40gmail.com.

Re: [google-appengine] Spending LImits Going Away :(

[Linus Larsen]

True story

One day in December some year ago customers called in complaining about our 
service wasn't responding. A quick look in the console we could see that we 
had gone over our spending limit. Why? Digging deeper we could see that we 
had 1000+ instances running (standard, java, autoscale, and we hadn't 
specified maximum instances). After some trouble shooting and help with 
Google support we found the reason. It turned out that we had hit the roof 
of a socket _connect quota, for each request we get to our service we send 
a pub/sub message, in the pub/sub library we used it was not possible to 
batch send messages so each message sent was literally a socket_connect.

After hitting the socket _connect quota limit the calls to pub/sub where 
timing out, and the timeout value was too high (we were using the library 
default value), so the autoscaler scheduler started a new instance in a 
forever loop.

When this happen everything froze and we couldn't kill instances more 
rapidly than the scheduler spun up a new one, kind of a moment 22 situation.

In order to get it fixed we had to ask Google to bump the socket_connect 
quota, then we had to rise our spending limit to get everything working 
again. 

In a situation like this the only thing that wasn't getting us bankrupt was 
the spending limit,  the irony here the suggestion about using "Budget 
Alerts, Pub/Sub, and Cloud Functions when your costs exceed the threshold" 
won't work if your not allowed to call pub/sub because of a stupid 
socket_connect quota.

We probably stumbled on an edge case with a lot of combinations of usage of 
old libraries / apis, and hidden quota limits (I can't even find the 
socket_connect quota anymore on the quota pages in console), however my 
point here is spending limit is the last outpost we have to actually to set 
a cost limit and it's sad seeing it going away just as the other 
dismounting of what to be a great service.
 


 

onsdag 26 augusti 2020 kl. 22:47:44 UTC+2 skrev yananc:

> Hello Joshua,
>
> The budget alert will be triggered once your costs rise above the 
> threshold you specify. However, as Alexis has explained, there are various 
> factors that might affect ‘how quickly’ you will receive the alert.
>
> Same as the email you shared, the documentation [1] provides details on 
> how to manage App Engine costs. Specifically, besides mechanisms such as 
> setting the max number of instances, Budget Alerts, Pub/Sub, and Cloud 
> Functions could be used to automatically disable your app when your costs 
> exceed the threshold you specify. The documentation also provides steps on 
> how to implement it with sample codes.
>
> Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a 
> possible solution is to leverage cache mechanism to avoid excessive 
> operations. You may find more information from the topic [2].
>
> Hope it helps.
>
> [1]: https://cloud.google.com/appengine/docs/managing-costs
>
> [2]: 
> https://stackoverflow.com/questions/12939376/google-app-engine-too-many-datastore-read-operations
>
> On Tuesday, August 25, 2020 at 6:51:49 PM UTC-4 Luca de Alfaro wrote:
>
>> Yes, at least, we can hard limit the number of active instances: see 
>> https://cloud.google.com/appengine/docs/standard/python3/config/appref
>> So if every active instance has a limited rate of use of backend services 
>> (like datastore), and there are no services accessible except via an 
>> appengine instance (e.g., no GCS direct bandwidth), in practice we can put 
>> a bound using that. 
>>
>> Luca
>>
>> On Tue, Aug 25, 2020 at 3:45 PM Luca de Alfaro  wrote:
>>
>>> I concur with the worry.  Is there any _technical_ reason why it is a 
>>> good idea to do away with a spending limit?  Can we get an instance limit 
>>> instead? 
>>> This is suddenly making standard non-scalable systems on AWS look better 
>>> than appengine! 
>>>
>>> On Tue, Aug 25, 2020 at 9:03 AM Joshua Smith  
>>> wrote:
>>>
 Once again last night, my wallet was saved when a runaway bot chewed up 
 my site’s whole daily spending limit. I got an email from a user, set up a 
 firewall rule, and goosed my budget to get things going again.

 I’m *very* concerned about Google’s decision to remove this feature. 
 Offering a cloud service that bills by usage without having a way to limit 
 the spend shifts an unreasonable amount of risk onto the subscriber.

 I’ve set up budget alerts, as suggested, but I’m concerned that:

 - What if my bill shoots up really fast? How quickly is this alert 
 going to go out?

 - What if I am away from the computer (remember when we used to be able 
 to leave our houses? good times… good times…)?

 I run this particular site as a not-for-profit social good. (It’s a 
 site that small town governments use to post their meetings.) I make 
 *no* money on it.

 I’d be perfectly happy to handle this with self-set quotas on something 

Re: [google-appengine] Spending LImits Going Away :(

['yananc' via Google App Engine]

Hello Joshua,

The budget alert will be triggered once your costs rise above the threshold 
you specify. However, as Alexis has explained, there are various factors 
that might affect ‘how quickly’ you will receive the alert.

Same as the email you shared, the documentation [1] provides details on how 
to manage App Engine costs. Specifically, besides mechanisms such as 
setting the max number of instances, Budget Alerts, Pub/Sub, and Cloud 
Functions could be used to automatically disable your app when your costs 
exceed the threshold you specify. The documentation also provides steps on 
how to implement it with sample codes.

Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a 
possible solution is to leverage cache mechanism to avoid excessive 
operations. You may find more information from the topic [2].

Hope it helps.

[1]: https://cloud.google.com/appengine/docs/managing-costs

[2]: 
https://stackoverflow.com/questions/12939376/google-app-engine-too-many-datastore-read-operations

On Tuesday, August 25, 2020 at 6:51:49 PM UTC-4 Luca de Alfaro wrote:

> Yes, at least, we can hard limit the number of active instances: see 
> https://cloud.google.com/appengine/docs/standard/python3/config/appref
> So if every active instance has a limited rate of use of backend services 
> (like datastore), and there are no services accessible except via an 
> appengine instance (e.g., no GCS direct bandwidth), in practice we can put 
> a bound using that. 
>
> Luca
>
> On Tue, Aug 25, 2020 at 3:45 PM Luca de Alfaro  wrote:
>
>> I concur with the worry.  Is there any _technical_ reason why it is a 
>> good idea to do away with a spending limit?  Can we get an instance limit 
>> instead? 
>> This is suddenly making standard non-scalable systems on AWS look better 
>> than appengine! 
>>
>> On Tue, Aug 25, 2020 at 9:03 AM Joshua Smith  
>> wrote:
>>
>>> Once again last night, my wallet was saved when a runaway bot chewed up 
>>> my site’s whole daily spending limit. I got an email from a user, set up a 
>>> firewall rule, and goosed my budget to get things going again.
>>>
>>> I’m *very* concerned about Google’s decision to remove this feature. 
>>> Offering a cloud service that bills by usage without having a way to limit 
>>> the spend shifts an unreasonable amount of risk onto the subscriber.
>>>
>>> I’ve set up budget alerts, as suggested, but I’m concerned that:
>>>
>>> - What if my bill shoots up really fast? How quickly is this alert going 
>>> to go out?
>>>
>>> - What if I am away from the computer (remember when we used to be able 
>>> to leave our houses? good times… good times…)?
>>>
>>> I run this particular site as a not-for-profit social good. (It’s a site 
>>> that small town governments use to post their meetings.) I make *no* money 
>>> on it.
>>>
>>> I’d be perfectly happy to handle this with self-set quotas on something 
>>> other than dollars. For example, in my case the budget-buster is always 
>>> “Cloud Datastore Read Operations.” If I could set a cap on that one thing, 
>>> it’d give me the protection I need.
>>>
>>> -Joshua
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to google-appengi...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/79fa52cd-f974-42af-8706-eac3e1409123n%40googlegroups.com.

Re: [google-appengine] Spending LImits Going Away :(

['yananc' via Google App Engine]

*Hello Joshua,The budget alert will be triggered once your costs rise above 
the threshold you specify. However, as Alexis has explained, there are 
various factors that might affect ‘how quickly’ you will receive the 
alert.Same as the email you shared, the documentation [1] provides details 
on how to manage App Engine costs. Specifically, besides mechanisms such as 
setting the max number of instances, Budget Alerts, Pub/Sub, and Cloud 
Functions could be used to automatically disable your app when your costs 
exceed the threshold you specify. The documentation also provides steps on 
how to implement it with sample codes.Back to the issue of ‘Cloud Datastore 
Read Operations’ being too high, a possible solution is to leverage cache 
mechanism to avoid excessive operations. You may find more information from 
the topic [2].Hope it helps.[1]: 
https://cloud.google.com/appengine/docs/managing-costs 
[2]: 
https://stackoverflow.com/questions/12939376/google-app-engine-too-many-datastore-read-operations
 
*

On Tuesday, August 25, 2020 at 6:51:49 PM UTC-4 Luca de Alfaro wrote:

> Yes, at least, we can hard limit the number of active instances: see 
> https://cloud.google.com/appengine/docs/standard/python3/config/appref
> So if every active instance has a limited rate of use of backend services 
> (like datastore), and there are no services accessible except via an 
> appengine instance (e.g., no GCS direct bandwidth), in practice we can put 
> a bound using that. 
>
> Luca
>
> On Tue, Aug 25, 2020 at 3:45 PM Luca de Alfaro  wrote:
>
>> I concur with the worry.  Is there any _technical_ reason why it is a 
>> good idea to do away with a spending limit?  Can we get an instance limit 
>> instead? 
>> This is suddenly making standard non-scalable systems on AWS look better 
>> than appengine! 
>>
>> On Tue, Aug 25, 2020 at 9:03 AM Joshua Smith  
>> wrote:
>>
>>> Once again last night, my wallet was saved when a runaway bot chewed up 
>>> my site’s whole daily spending limit. I got an email from a user, set up a 
>>> firewall rule, and goosed my budget to get things going again.
>>>
>>> I’m *very* concerned about Google’s decision to remove this feature. 
>>> Offering a cloud service that bills by usage without having a way to limit 
>>> the spend shifts an unreasonable amount of risk onto the subscriber.
>>>
>>> I’ve set up budget alerts, as suggested, but I’m concerned that:
>>>
>>> - What if my bill shoots up really fast? How quickly is this alert going 
>>> to go out?
>>>
>>> - What if I am away from the computer (remember when we used to be able 
>>> to leave our houses? good times… good times…)?
>>>
>>> I run this particular site as a not-for-profit social good. (It’s a site 
>>> that small town governments use to post their meetings.) I make *no* money 
>>> on it.
>>>
>>> I’d be perfectly happy to handle this with self-set quotas on something 
>>> other than dollars. For example, in my case the budget-buster is always 
>>> “Cloud Datastore Read Operations.” If I could set a cap on that one thing, 
>>> it’d give me the protection I need.
>>>
>>> -Joshua
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to google-appengi...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b225afc2-640e-40e8-afb9-9a672b4a8b55n%40googlegroups.com.

Re: [google-appengine] Spending LImits Going Away :(

[Luca de Alfaro]

Yes, at least, we can hard limit the number of active instances: see
https://cloud.google.com/appengine/docs/standard/python3/config/appref
So if every active instance has a limited rate of use of backend services
(like datastore), and there are no services accessible except via an
appengine instance (e.g., no GCS direct bandwidth), in practice we can put
a bound using that.

Luca

On Tue, Aug 25, 2020 at 3:45 PM Luca de Alfaro  wrote:

> I concur with the worry.  Is there any _technical_ reason why it is a good
> idea to do away with a spending limit?  Can we get an instance limit
> instead?
> This is suddenly making standard non-scalable systems on AWS look better
> than appengine!
>
> On Tue, Aug 25, 2020 at 9:03 AM Joshua Smith 
> wrote:
>
>> Once again last night, my wallet was saved when a runaway bot chewed up
>> my site’s whole daily spending limit. I got an email from a user, set up a
>> firewall rule, and goosed my budget to get things going again.
>>
>> I’m *very* concerned about Google’s decision to remove this feature.
>> Offering a cloud service that bills by usage without having a way to limit
>> the spend shifts an unreasonable amount of risk onto the subscriber.
>>
>> I’ve set up budget alerts, as suggested, but I’m concerned that:
>>
>> - What if my bill shoots up really fast? How quickly is this alert going
>> to go out?
>>
>> - What if I am away from the computer (remember when we used to be able
>> to leave our houses? good times… good times…)?
>>
>> I run this particular site as a not-for-profit social good. (It’s a site
>> that small town governments use to post their meetings.) I make *no* money
>> on it.
>>
>> I’d be perfectly happy to handle this with self-set quotas on something
>> other than dollars. For example, in my case the budget-buster is always
>> “Cloud Datastore Read Operations.” If I could set a cap on that one thing,
>> it’d give me the protection I need.
>>
>> -Joshua
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to google-appengine+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAPgu2u%2BjqXRjMMny-5f%3DNr0OXYXZgi6w_Rm7jwfJDS8B54EixA%40mail.gmail.com.

Re: [google-appengine] Spending LImits Going Away :(

[Luca de Alfaro]

I concur with the worry.  Is there any _technical_ reason why it is a good
idea to do away with a spending limit?  Can we get an instance limit
instead?
This is suddenly making standard non-scalable systems on AWS look better
than appengine!

On Tue, Aug 25, 2020 at 9:03 AM Joshua Smith 
wrote:

> Once again last night, my wallet was saved when a runaway bot chewed up my
> site’s whole daily spending limit. I got an email from a user, set up a
> firewall rule, and goosed my budget to get things going again.
>
> I’m *very* concerned about Google’s decision to remove this feature.
> Offering a cloud service that bills by usage without having a way to limit
> the spend shifts an unreasonable amount of risk onto the subscriber.
>
> I’ve set up budget alerts, as suggested, but I’m concerned that:
>
> - What if my bill shoots up really fast? How quickly is this alert going
> to go out?
>
> - What if I am away from the computer (remember when we used to be able to
> leave our houses? good times… good times…)?
>
> I run this particular site as a not-for-profit social good. (It’s a site
> that small town governments use to post their meetings.) I make *no* money
> on it.
>
> I’d be perfectly happy to handle this with self-set quotas on something
> other than dollars. For example, in my case the budget-buster is always
> “Cloud Datastore Read Operations.” If I could set a cap on that one thing,
> it’d give me the protection I need.
>
> -Joshua
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CAPgu2uLMSxFBrPy%2BrLvGLhWqp2XYBW%2BWtCOqFTtAj7sDhhtCMg%40mail.gmail.com.

Re: [google-appengine] Spending LImits Going Away :(

[Joshua Smith]

Attaching an email from Google, which should answer your first question.

Your answer to my first question is a bit vague (“prompt”) but I’ll accept it.

The answer to the second question misses the point. I can get the notification 
while I’m away from home, but I can’t do much of anything about it. I’d be 
going to the web browser on my phone and trying to do what I can from there, 
but… yikes.

Anyway, here’s the email from Google...

Configure alternative cost management method(s) for App Engine projects by July 
24, 2021.
Hello,

In December 2019, we removed the ability to create new spending limits 

 in App Engine. We’ve since rolled out new tools to help you manage costs in 
App Engine, and will be removing all spending limits on July 24, 2021. Review 
our new set of cost management tools 

 and choose the mechanism that best suits your needs.

What do I need to know?

While App Engine has evolved, the spending limit functionality has not. We’re 
replacing this feature because it doesn’t cover App Engine costs related to 
newer capabilities, or services like App Engine Flex or Cloud Build. The new 
Google Cloud cost management tools will help you control costs associated with 
a broad range of resources, including App Engine.

What do I need to do?

If you don’t need a cost management mechanism for App Engine, no action is 
required.

To manage App Engine-related costs, implement any of the following mechanisms:

Specify the maximum number of instances 

Create budget alerts 

Disable your app manually 

Disable your app programmatically 

Learn more about these cost-management mechanisms by reviewing Managing App 
Engine costs 
.

Your affected projects are below:

linktotweet (linktotweet) 

toptweetsbot (toptweetsbot) 

towngovernment-hr (towngovernment-hr) 

corpus-delicti (corpus-delicti) 

How can I get help?

If you have any questions or require assistance, please reply to this email to 
contact Google Cloud Support.

Thanks for choosing App Engine.



> On Aug 25, 2020, at 3:45 PM, 'Alexis (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> Hello Joshua,
> 
> I'll try to help the best I can. I've added a question below, along with some 
> answers too.
> 
> Questions:
> - Which feature did Google remove? What was this feature called and where was 
> it shown in the GUI? I'm asking so that we can try to maybe do a feature 
> request or see why it was removed.
> 
> To answer your question, the alerts should be prompt unless there is an 
> outage or some other exceptional circumstance. However, keep in mind that we 
> do not have control over public communication channel delays such as SMS, 
> email, etc... Double-checking in the GUI could tell you if it was already 
> sent out or not. Delays can also happen if alerts have multiple conditions 
> and one of them hasn't been met yet. See full article here[1] for more 
> details about latency possibilities.
> 
> If you're away from the computer, you have notification options here[2], 
> called "channels".
> 
> 

[google-appengine] Spending LImits Going Away :(

[Joshua Smith]

Once again last night, my wallet was saved when a runaway bot chewed up my 
site’s whole daily spending limit. I got an email from a user, set up a 
firewall rule, and goosed my budget to get things going again.

I’m very concerned about Google’s decision to remove this feature. Offering a 
cloud service that bills by usage without having a way to limit the spend 
shifts an unreasonable amount of risk onto the subscriber.

I’ve set up budget alerts, as suggested, but I’m concerned that:

- What if my bill shoots up really fast? How quickly is this alert going to go 
out?

- What if I am away from the computer (remember when we used to be able to 
leave our houses? good times… good times…)?

I run this particular site as a not-for-profit social good. (It’s a site that 
small town governments use to post their meetings.) I make no money on it.

I’d be perfectly happy to handle this with self-set quotas on something other 
than dollars. For example, in my case the budget-buster is always “Cloud 
Datastore Read Operations.” If I could set a cap on that one thing, it’d give 
me the protection I need.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com.