Re: [google-appengine] New weird error.

[Joshua Smith]

As suggested, I created a private issue, since they wanted some app details. So 
you wouldn't be able to see the issue even if I shared the link.

But I'll post back here if any progress is made. It's still happening, although 
not too much...



> On Mar 28, 2022, at 3:56 AM, 'Lluis Munoz Ladron de Guevara' via Google App 
> Engine  wrote:
> 
> Hi, 
> 
> Could you please share the Issue Tracker URL? 
> It might help other users who experience this issue. 
> 
> On Wednesday 23 March 2022 at 20:22:24 UTC+1 Joshua Smith wrote:
> 
>> On Mar 23, 2022, at 3:07 PM, 'Amit Sinha' via Google App Engine 
>> > > wrote:
>> 
>> Hi Joshua,
>> 
>> As you mentioned, this started to happen recently, could you please confirm 
>> if you still have this issue?
>> 
> 
> Error reporting says it is still happening. Most recent was about 3 hours 
> ago. (Note that nothing changed in our code, so this appears to be a problem 
> at the infrastructure level.)
> 
> 
>>  I was trying to find any recently reported issues related to the error, but 
>> it seems like nothing has been reported yet. By mentioning the NDB datastore 
>> , are you referring to this [0]?
>> 
> 
> Yes
>> After doing some research on this error, it seems like a very generic error 
>> statement. I can see a couple of other reports with the same types of 
>> “AttributeError” [1]. So it could be hard to find the solution without 
>> investigating in detail. At this point, I would like to note that Google 
>> Group is intended for very general purpose questions. If you believe this 
>> could be an error related to the library, you could report this to Issue 
>> Tracker by creating a private issue [4] with all details (project ID , App 
>> Engine version name etc). In general troubleshooting coding errors are best 
>> to report in Stackoverflow [5] where it can be visible to other developers 
>> to get the best feedback.   
>> 
> 
> Usually when I see infrastructure-level issues I first report them here to 
> see if they are something common lots of people are seeing, before I open an 
> issue tracker about something google's already working on.
> 
> But if your guys aren't working on this, then I'll go ahead and do [4].
> 
>> 
>> [0] https://cloud.google.com/appengine/docs/standard/python/ndb 
>> <https://cloud.google.com/appengine/docs/standard/python/ndb>
>> [1] 
>> https://stackoverflow.com/questions/8949252/why-do-i-get-attributeerror-nonetype-object-has-no-attribute-something
>>  
>> <https://stackoverflow.com/questions/8949252/why-do-i-get-attributeerror-nonetype-object-has-no-attribute-something>
>> [2] 
>> https://stackoverflow.com/questions/54474655/error-nonetype-object-has-no-attribute-call
>>  
>> <https://stackoverflow.com/questions/54474655/error-nonetype-object-has-no-attribute-call>
>> [3] https://github.com/googleapis/google-cloud-python/issues/10619 
>> <https://github.com/googleapis/google-cloud-python/issues/10619>
>> [4] https://cloud.google.com/support/docs/issue-trackers 
>> <https://cloud.google.com/support/docs/issue-trackers>
>> [5] https://cloud.google.com/support/docs/stackexchange 
>> <https://cloud.google.com/support/docs/stackexchange>
>> 
>> On Wednesday, March 23, 2022 at 9:00:56 AM UTC-4 Joshua Smith wrote:
>> (See full stack trace at the end)
>> 
>> Anyone else seeing datastore errors over the last few hours? Most recent was 
>> 2.5 hours ago...
>> 
>> The line that's throwing the exception (srv/main.py:989) is:
>> 
>> if WebMQueueModel.gql("WHERE status = :1", "running").get():
>> 
>> My best guess is that there's some intermittent issue with the NDB datastore 
>> which is bubbling up as this exception.
>> 
>> It's happening the same time as this:
>> 
>> 
>> 
>> 
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/e8f7523e-4798-4f4e-a560-bf2426ac744fn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/e8f7523e-4798-4f4e-a560-bf2426ac744fn%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" 

Re: [google-appengine] New weird error.

[Joshua Smith]

> On Mar 23, 2022, at 3:07 PM, 'Amit Sinha' via Google App Engine 
>  wrote:
> 
> Hi Joshua,
> 
> As you mentioned, this started to happen recently, could you please confirm 
> if you still have this issue?
> 
Error reporting says it is still happening. Most recent was about 3 hours ago. 
(Note that nothing changed in our code, so this appears to be a problem at the 
infrastructure level.)

>  I was trying to find any recently reported issues related to the error, but 
> it seems like nothing has been reported yet. By mentioning the NDB datastore 
> , are you referring to this [0]?
> 
Yes
> After doing some research on this error, it seems like a very generic error 
> statement. I can see a couple of other reports with the same types of 
> “AttributeError” [1]. So it could be hard to find the solution without 
> investigating in detail. At this point, I would like to note that Google 
> Group is intended for very general purpose questions. If you believe this 
> could be an error related to the library, you could report this to Issue 
> Tracker by creating a private issue [4] with all details (project ID , App 
> Engine version name etc). In general troubleshooting coding errors are best 
> to report in Stackoverflow [5] where it can be visible to other developers to 
> get the best feedback.   
> 
Usually when I see infrastructure-level issues I first report them here to see 
if they are something common lots of people are seeing, before I open an issue 
tracker about something google's already working on.

But if your guys aren't working on this, then I'll go ahead and do [4].
> 
> [0] https://cloud.google.com/appengine/docs/standard/python/ndb
> 
> [1] 
> https://stackoverflow.com/questions/8949252/why-do-i-get-attributeerror-nonetype-object-has-no-attribute-something
>  
> <https://stackoverflow.com/questions/8949252/why-do-i-get-attributeerror-nonetype-object-has-no-attribute-something>
> [2] 
> https://stackoverflow.com/questions/54474655/error-nonetype-object-has-no-attribute-call
>  
> <https://stackoverflow.com/questions/54474655/error-nonetype-object-has-no-attribute-call>
> [3] https://github.com/googleapis/google-cloud-python/issues/10619 
> <https://github.com/googleapis/google-cloud-python/issues/10619>
> [4] https://cloud.google.com/support/docs/issue-trackers
> 
> [5] https://cloud.google.com/support/docs/stackexchange
> 
> 
> 
> On Wednesday, March 23, 2022 at 9:00:56 AM UTC-4 Joshua Smith wrote:
> (See full stack trace at the end)
> 
> Anyone else seeing datastore errors over the last few hours? Most recent was 
> 2.5 hours ago...
> 
> The line that's throwing the exception (srv/main.py:989) is:
> 
> if WebMQueueModel.gql("WHERE status = :1", "running").get():
> 
> My best guess is that there's some intermittent issue with the NDB datastore 
> which is bubbling up as this exception.
> 
> It's happening the same time as this:
> 
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/e8f7523e-4798-4f4e-a560-bf2426ac744fn%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/e8f7523e-4798-4f4e-a560-bf2426ac744fn%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/033274FD-CF43-456B-9D1C-541001541A90%40gmail.com.

Re: [google-appengine] No Search Solution?! WOW.

[Joshua Smith]

Okay, that's hilarious.

Users: Can we send email?
Google: Nah. Sending email is too hard. You have to use a 3rd party for that.
Users: Aren't you the same company that makes gmail, the most popular email 
service on the planet?
Google: Yeah. What's your point?
Users: Never mind. Can we search for text in our databases?
Google: Nah. Search is too hard. You have to use a 3rd party for that.
Users: Wait. Is this the same google that is literally the largest search 
provider on the planet?
Google: Yeah, that's us.
Users: But search is too hard to provide in your cloud services platform.
Google: Right. You can do a feature request for it.
Users: Do you implement features we request?
Google: Generally, no. But we do take away features like mail and search, and 
then you can request that we give them back.
Users: Which you will never do.
Google: Correct.

The mind boggles.

> On Mar 8, 2022, at 9:58 AM, 'Lluis Munoz Ladron de Guevara' via Google App 
> Engine  wrote:
> 
> I found this documentation 
> 
>  regarding how to have a full text search in Firestore.
> 
> Additionally I found this feature request 
>  which asks for full text search 
> in the Firestore UI. If you would like that feature to be added to GCP I 
> encourage you to indicate that you are affected by it by pressing the star 
> button. 
> Please bear in mind that users feedback is an important metric to track how 
> relevant a feature request is, more information about this can be found here 
> .
>  
> 
> On Monday 7 March 2022 at 19:58:27 UTC+1 Jim wrote:
> As I'm sure you are aware,  GAE Standard has the Search service, but this is 
> being phased out and no replacement is available on GAE Flex.  
> 
> The lack of an integrated Search offering kinda blows a hole in the entire 
> "server-less platform" vision.
> 
> On Monday, March 7, 2022 at 10:05:57 AM UTC-6 barrado wrote:
> Hi,
> 
> As per now there isn't any built-in search solution for Cloud Firestore. But 
> you can submit a feature request for the product team to consider this 
> functionality.
> 
> On Sunday, February 27, 2022 at 1:18:51 AM UTC+1 kaan...@gmail.com <> wrote:
> The title says it all, I was checking the documents to see what the current 
> intended approach is, assumed Firestore has a searchable field for full text 
> search at this point
> 
> But nope, not only that, no direct solution at all
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/d95d6c33-031e-4f17-a09e-d31f3ce7d593n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/918DC4EC-EA48-420D-BB74-B26A52D9904D%40gmail.com.

[google-appengine] Re: Error -27?

[Joshua Smith]

Seems to have cleared itself up.

> On Jan 31, 2022, at 10:03 AM, Joshua Smith  wrote:
> 
> Anyone else getting these? I'm getting them hitting /admin sorts of pages in 
> a Python27 runtime app. There's nothing in my logs...
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/94833BD2-FDEE-4F84-AAA9-FFAA0864DA86%40gmail.com.

[google-appengine] Firewalling AWS

[Joshua Smith]

I have a small site that I run on a not-for-profit basis.

Periodically I need to update robots.txt or add firewall rules to shut down bad 
actors who beat the crap out of the site running up my instance costs.

Lately, I've been getting slammed by instances running on AWS. They are mostly 
making HEAD requests, which makes me think it's some kind of crawler, but it 
uses regular browser user agents and doesn't respect my robots rules.

There's no legitimate reason for AWS to browse my site, so I just add a 
firewall rule, right? Trouble is, AWS has 6,462 different IPV4 address ranges, 
and this crawler is constantly jumping between them.

Any costs that I have are paid out of my own pocket. So I'm looking for 
suggestions that don't require MORE subscriptions (like CloudFlare or 
something).

Any ideas?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1772968C-ED0C-49D9-BA35-AB7F2CA2AD1E%40gmail.com.

Re: [google-appengine] A request is suspended for 6 hours, then completes

[Joshua Smith]

Did you check for a time.sleep(2160) in the code?

:)

> On Dec 1, 2021, at 11:59 AM, Patrice Bertrand 
>  wrote:
> 
> Here is a strange thing we are seeing repeatedly.   We have a batch job 
> processing, handled on a GAE backend service, which ends up taking more than 
> 6 hours, with only 6 minutes of actual processing.   The request is started 
> by a Cloud Task.
> 
> In one case, as an example, the request is started at 15:58:00, it runs (with 
> some records in the stdout log) until 15:59:36, then it is 'suspended', i.e. 
> there is not a single record in the stdout log although this job does write 
> on every loop).  Then suddenly we have records in the stdout log again, 
> starting at 21:57:24, six hours later, and the handling of the request ends 
> at 22:01:56.Ultimately, it looks like the request has been running for a 
> total of 1mn 36 seconds plus 4mn 32 seconds = 6mn 8 seconds, but with a six 
> hour suspension in the middle.
> 
> In the stdout log, it is the same trace id all along, and we don't see any 
> error occuring at the time of the suspension. 
> 
> Does this ring a bell to anymone ?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/b005e3f3-5686-44f3-bd24-b61cb4f30980n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0B7BDC68-F4F3-42F4-AAA8-8BCF2075F3FA%40gmail.com.

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I'll put it on my to-do list to make a mock application that just does this 
hammer-on-startup thing. That seems pretty easy. (And it'll be interesting to 
see if a trivial mock app takes 5s to start up, or if there's some import I'm 
doing that's weighing things down.)

-Joshua

> On Nov 5, 2021, at 6:39 PM, Jason Collins  wrote:
> 
> Yes, it is the same scheduler (this is all on GAE Standard, right?) Of 
> course, that scheduler has lots of knobs.
> 
> If you can build a repro, can you work with support? If anything, the 
> concurrency of GAE Python 3 should be better than GAE Python 2.
> 
> On Fri, Nov 5, 2021 at 3:36 PM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and instance 
> class. (I do have it in my 2.7 app, of course.)
> 
> I'll try playing with max concurrent threads, but if I bump it from 10 (the 
> default) to 16, that really just changes it from 6 request to start an 
> instance, to 9. Hard to see how that would make any difference.
> 
> As I said, the 2.7 only starts one more instance when I hammer it, and it 
> doesn't end up giving it any work. That's very different behavior from the 
> 3.7 app which starts a bunch of instances and spreads the work to all of them 
> (resulting in all the requests taking at least 5 seconds to fulfill, since 
> the instance takes 5s to start up). This is clearly visible in the logs I 
> included in the first message.
> 
> You're positive the scheduler is the same?
> 
> -Joshua
> 
>> On Nov 5, 2021, at 5:08 PM, Jason Collins > <mailto:jason.a.coll...@gmail.com>> wrote:
>> 
>> The scheduler is the same.
>> 
>> This might have to do with `threadsafe` in app.yaml, which is complicated 
>> and confusing. I'd suggest removing `threadsafe` if it's present, then try 
>> setting the max_concurrent_requests to something higher (like 2x your number 
>> of workers): 
>> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>>  
>> <https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests>
>> 
>> 
>> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith > <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> I figured out from the logs (since it prints a message for each worker it 
>> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
>> with defaults has no effect.
>> 
>> However, I did some experiments and I think I can characterize the 
>> difference between 2.7 and 3.7 when it comes to scaling.
>> 
>> When first landing on my app's page it shows a list of thumbnails. These 
>> come from the datastore, with each thumbnail making a HTTP request (GET 
>> /image?which=1, GET /image?which=2, etc.). If we:
>> 
>> 1. Query for all of these at once so the server gets hammered with 
>> simultaneous requests:
>>  - 2.7 starts one extra instance, and while that's happening the images 
>> are all served by the instance that already existed; the new instance never 
>> actually serves any of those image requests.
>>  - 3.7 starts a lot of instances; it dumps a bunch of requests to each 
>> of them (one per worker thread, I presume), all of which are going to take 5 
>> seconds because that's how long it takes each instance to start.
>> 
>> The net result is the user of the 2.7 app sees all the images load right 
>> away, while the 3.7 app gets them over the course of 15 seconds or so, and 
>> sometimes there are timeout errors.
>> 
>> 2. Query for these with a delay that's long enough to avoid overlapping 
>> requests:
>>  - Neither starts extra instances.
>> 
>> The user sees all the images load pretty quick, but not quite as quick as 
>> 2.7 with the first approach.
>> 
>> My conclusion is that the change is that the old scheduler wouldn't use an 
>> instance until it was running, whereas the new scheduler queues up requests 
>> for instances that are still starting up. The old scheduler doesn't start a 
>> new instance until the one it started most recently is running, whereas the 
>> new scheduler starts a new instance every time the request queue fills up 
>> for existing instances.
>> 
>> If you can serve a burst of requests in the time it take an instance to 
>> start, the old scheduler will get all those served. But the new scheduler 
>> will choke and perform very badly.
>> 
>> I'm not saying that the old scheduler was better. It just handled this 
>> particular case (cold start with a firehose) a lot better. Once the two 
>> syst

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I don't have threadsafe in my 3.7 app's app.yaml. Just runtime and instance 
class. (I do have it in my 2.7 app, of course.)

I'll try playing with max concurrent threads, but if I bump it from 10 (the 
default) to 16, that really just changes it from 6 request to start an 
instance, to 9. Hard to see how that would make any difference.

As I said, the 2.7 only starts one more instance when I hammer it, and it 
doesn't end up giving it any work. That's very different behavior from the 3.7 
app which starts a bunch of instances and spreads the work to all of them 
(resulting in all the requests taking at least 5 seconds to fulfill, since the 
instance takes 5s to start up). This is clearly visible in the logs I included 
in the first message.

You're positive the scheduler is the same?

-Joshua

> On Nov 5, 2021, at 5:08 PM, Jason Collins  wrote:
> 
> The scheduler is the same.
> 
> This might have to do with `threadsafe` in app.yaml, which is complicated and 
> confusing. I'd suggest removing `threadsafe` if it's present, then try 
> setting the max_concurrent_requests to something higher (like 2x your number 
> of workers): 
> https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests
>  
> <https://cloud.google.com/appengine/docs/standard/python/config/appref#max_concurrent_requests>
> 
> 
> On Fri, Nov 5, 2021 at 1:48 PM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> I figured out from the logs (since it prints a message for each worker it 
> starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
> with defaults has no effect.
> 
> However, I did some experiments and I think I can characterize the difference 
> between 2.7 and 3.7 when it comes to scaling.
> 
> When first landing on my app's page it shows a list of thumbnails. These come 
> from the datastore, with each thumbnail making a HTTP request (GET 
> /image?which=1, GET /image?which=2, etc.). If we:
> 
> 1. Query for all of these at once so the server gets hammered with 
> simultaneous requests:
>   - 2.7 starts one extra instance, and while that's happening the images 
> are all served by the instance that already existed; the new instance never 
> actually serves any of those image requests.
>   - 3.7 starts a lot of instances; it dumps a bunch of requests to each 
> of them (one per worker thread, I presume), all of which are going to take 5 
> seconds because that's how long it takes each instance to start.
> 
> The net result is the user of the 2.7 app sees all the images load right 
> away, while the 3.7 app gets them over the course of 15 seconds or so, and 
> sometimes there are timeout errors.
> 
> 2. Query for these with a delay that's long enough to avoid overlapping 
> requests:
>   - Neither starts extra instances.
> 
> The user sees all the images load pretty quick, but not quite as quick as 2.7 
> with the first approach.
> 
> My conclusion is that the change is that the old scheduler wouldn't use an 
> instance until it was running, whereas the new scheduler queues up requests 
> for instances that are still starting up. The old scheduler doesn't start a 
> new instance until the one it started most recently is running, whereas the 
> new scheduler starts a new instance every time the request queue fills up for 
> existing instances.
> 
> If you can serve a burst of requests in the time it take an instance to 
> start, the old scheduler will get all those served. But the new scheduler 
> will choke and perform very badly.
> 
> I'm not saying that the old scheduler was better. It just handled this 
> particular case (cold start with a firehose) a lot better. Once the two 
> systems are running at scale, I wouldn't expect there to be much difference.
> 
> -Joshua
> 
> 
>> On Nov 5, 2021, at 3:29 PM, Joshua Smith > <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> 
>> I think it's telling me that since my P3.7 app has a F4_1G instance, I 
>> should be configuring it to run gunicorn with 8 workers. I don't see any 
>> information about the default number of workers. I guess I can do an 
>> experiment to see what explicitly setting that does...
>> 
>>> On Nov 5, 2021, at 3:17 PM, Jason Collins >> <mailto:jason.a.coll...@gmail.com>> wrote:
>>> 
>>> It sounds like there might be some differences due to concurrency between 
>>> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
>>> processes started for Python 3 here: 
>>> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>>>  
>>> <https://cloud.google.com/appengine/docs/standard/python3/ru

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I figured out from the logs (since it prints a message for each worker it 
starts) that 3.7 is starting 8 workers by default. So adding the entrypoint 
with defaults has no effect.

However, I did some experiments and I think I can characterize the difference 
between 2.7 and 3.7 when it comes to scaling.

When first landing on my app's page it shows a list of thumbnails. These come 
from the datastore, with each thumbnail making a HTTP request (GET 
/image?which=1, GET /image?which=2, etc.). If we:

1. Query for all of these at once so the server gets hammered with simultaneous 
requests:
- 2.7 starts one extra instance, and while that's happening the images 
are all served by the instance that already existed; the new instance never 
actually serves any of those image requests.
- 3.7 starts a lot of instances; it dumps a bunch of requests to each 
of them (one per worker thread, I presume), all of which are going to take 5 
seconds because that's how long it takes each instance to start.

The net result is the user of the 2.7 app sees all the images load right away, 
while the 3.7 app gets them over the course of 15 seconds or so, and sometimes 
there are timeout errors.

2. Query for these with a delay that's long enough to avoid overlapping 
requests:
- Neither starts extra instances.

The user sees all the images load pretty quick, but not quite as quick as 2.7 
with the first approach.

My conclusion is that the change is that the old scheduler wouldn't use an 
instance until it was running, whereas the new scheduler queues up requests for 
instances that are still starting up. The old scheduler doesn't start a new 
instance until the one it started most recently is running, whereas the new 
scheduler starts a new instance every time the request queue fills up for 
existing instances.

If you can serve a burst of requests in the time it take an instance to start, 
the old scheduler will get all those served. But the new scheduler will choke 
and perform very badly.

I'm not saying that the old scheduler was better. It just handled this 
particular case (cold start with a firehose) a lot better. Once the two systems 
are running at scale, I wouldn't expect there to be much difference.

-Joshua


> On Nov 5, 2021, at 3:29 PM, Joshua Smith  wrote:
> 
> I think it's telling me that since my P3.7 app has a F4_1G instance, I should 
> be configuring it to run gunicorn with 8 workers. I don't see any information 
> about the default number of workers. I guess I can do an experiment to see 
> what explicitly setting that does...
> 
>> On Nov 5, 2021, at 3:17 PM, Jason Collins > <mailto:jason.a.coll...@gmail.com>> wrote:
>> 
>> It sounds like there might be some differences due to concurrency between 
>> 2.7 and 3.7/8/9. There are some notes on how to tweak the number of worker 
>> processes started for Python 3 here: 
>> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>>  
>> <https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices>
>> 
>> 
>> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform Support) 
>> wrote:
>> There is no official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently", as there is no change 
>> in scaling behavior, and no different implementation. This is an assumption 
>> while we investigate the situation described by Joshua above. I could not 
>> find a similar issue in the Public Issue Tracker 
>> <https://issuetracker.google.com/>, so the difference in scaling between 
>> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
>> settings in the project, or application-specific factors. Such issue are 
>> difficult to approach in a publicly-accessible thread as this one, as 
>> investigation needs particulars such as project ID, and access to other 
>> private data. To have this issue properly addressed, it's better to open a 
>> support case 
>> <https://cloud.google.com/support/docs/manage-cases#creating_cases> or an 
>> issue in the Public Issue Tracker <https://issuetracker.google.com/>. 
>> 
>> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) 
>> wrote:
>> Hello David,
>> 
>> >There’s not an official way of forcing the scaling to behave like it used 
>> >to be on python 2.7.
>> Is there an official description, somewhere, of "how the scaling has 
>> changed" or "how it has been implemented differently" ?
>> Or this change just a side effect of technical choices that were made in the 
>> new runtimes?
>> 
>> Thanks,
>> 
>>

Re: [google-appengine] Updates from the Google App Engine team (Fall 2021)

[Joshua Smith]

I think it's telling me that since my P3.7 app has a F4_1G instance, I should 
be configuring it to run gunicorn with 8 workers. I don't see any information 
about the default number of workers. I guess I can do an experiment to see what 
explicitly setting that does...

> On Nov 5, 2021, at 3:17 PM, Jason Collins  wrote:
> 
> It sounds like there might be some differences due to concurrency between 2.7 
> and 3.7/8/9. There are some notes on how to tweak the number of worker 
> processes started for Python 3 here: 
> https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices
>  
> <https://cloud.google.com/appengine/docs/standard/python3/runtime#entrypoint_best_practices>
> 
> 
> On Friday, 5 November 2021 at 11:50:52 UTC-7 George (Cloud Platform Support) 
> wrote:
> There is no official description, somewhere, of "how the scaling has changed" 
> or "how it has been implemented differently", as there is no change in 
> scaling behavior, and no different implementation. This is an assumption 
> while we investigate the situation described by Joshua above. I could not 
> find a similar issue in the Public Issue Tracker 
> <https://issuetracker.google.com/>, so the difference in scaling between 
> Python 2 and 3 is likely a one-time occurrence, and caused by idiosyncratic 
> settings in the project, or application-specific factors. Such issue are 
> difficult to approach in a publicly-accessible thread as this one, as 
> investigation needs particulars such as project ID, and access to other 
> private data. To have this issue properly addressed, it's better to open a 
> support case 
> <https://cloud.google.com/support/docs/manage-cases#creating_cases> or an 
> issue in the Public Issue Tracker <https://issuetracker.google.com/>. 
> 
> On Friday, 05 November 2021 at 05:50:42 UTC-4 Nicolas Fonrose (Teevity) wrote:
> Hello David,
> 
> >There’s not an official way of forcing the scaling to behave like it used to 
> >be on python 2.7.
> Is there an official description, somewhere, of "how the scaling has changed" 
> or "how it has been implemented differently" ?
> Or this change just a side effect of technical choices that were made in the 
> new runtimes?
> 
> Thanks,
> 
> On Thursday, November 4, 2021 at 9:56:43 PM UTC+1 David (Cloud Platform 
> Support) wrote:
> Hello,
> 
> Other than applying changes within your own application so it will go easy at 
> startup, you can try using warmup requests 
> <https://cloud.google.com/appengine/docs/standard/python3/configuring-warmup-requests>
>  along with tweaking the min_idle_instances element 
> <https://cloud.google.com/appengine/docs/standard/python3/config/appref#scaling_elements>
>  in your app.yaml, in order to reduce request and response latency during the 
> time when your app's code is being loaded to a newly created instance.
> 
> There’s not an official way of forcing the scaling to behave like it used to 
> be on python 2.7. However, this type of feedback can be passed to the App 
> Engine engineering team in the form of a feature request. You can create such 
> a feature request here 
> <https://issuetracker.google.com/issues/new?component=187191=1162953>.
>  The App Engine engineering team would then evaluate it and decide whether it 
> could be implemented or not.
> On Thursday, November 4, 2021 at 11:38:52 AM UTC-4 Joshua Smith wrote:
> Thanks for sending out this update. I did, indeed, miss most of this news.
> 
> This item, in particular, is awesome:
> 
> 
>> Extending support for App Engine bundled services 
>> <https://cloud.google.com/blog/products/serverless/support-for-app-engine-services-in-second-generation-runtimes>
>>  (Sep 2021) 
> 
> This sounds like it will make it so much easier to migrate to Python 3.7.
> 
> One thing I've noticed is that my older apps on 2.7 seem to handle peak 
> scaling a lot better than my newer apps on 3.7. For example, if I have a web 
> page that hits a 2.7 app with 100 REST calls at startup (bad design, but it 
> happens), the old app serves them all eventually. But if I do the same thing 
> in a 3.7 app, it's likely to choke and fail a bunch of those requests with 
> these:
> 
> 
> 
> The specific pattern is that after the first couple requests, it spins up a 
> new instance. That takes 5 seconds to serve its first request (simple app, so 
> I guess that's just GAE overhead). Then it spins up a couple more. Then I 
> start getting those errors on some of the requests, because 15s have passed. 
> In this capture the blue ones are new instances spinning up, and the orange 
> ones are timeout errors (note the 15s time on thos

Re: [google-appengine] App Engine costs rocket in recent months?

[Joshua Smith]

The thing that I've noticed is that the amount of bot traffic hitting my sites 
has been steadily increasing. I've addressed this through a combination of 
robots.txt and blocking certain IP address blocks in the firewall (I block the 
ones that don't respect crawl-delay or don't respect robots.txt).

It's a never-ending battle against these bots that all seem to be building 
their own google-scale web databases.

If your costs are coming from frontend instances, the simple solution to that 
is to just limit the maximum number of those. Although that means when you do 
get a traffic spike, users will suffer.

-Joshua

> On Sep 2, 2021, at 10:48 AM, Tapir  wrote:
> 
> 
> The costs of my projects rocketed from $5 to $50 in the previous months.
> Anyone else also got the same experience?
> 
> My websites are very small.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/e122d075-4147-4151-b156-693c21484afdn%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ACCBBA77-EABA-45CF-8E83-EA43DFFF0280%40gmail.com.

Re: [google-appengine] Weird Error Message

[Joshua Smith]

That makes sense. But in this case, I don't have google-api-core or 
google-cloud-core in my requirements.txt at all, so I guess it's being dragged 
in by something else?

This is what I have:

Flask==1.1.1
google-cloud-ndb
google-cloud-storage
google-cloud-logging
google-api-python-client
google-cloud-tasks
googleapis_common_protos
oauth2client
httplib2
sendgrid
pillow
pytz

So does this mean I need to downgrade some of those standard google- packages? 
And if so, how do I know what version to downgrade to?

Also, is this documented anywhere? I'd think if I'm using a bunch of google 
packages, google would be careful to make sure they latest versions of them all 
work together. And if they don't, there would be a document saying exactly what 
versions they want apps to use.

-Joshua

> On Aug 25, 2021, at 12:55 PM, NoCommandLine  
> wrote:
> 
> >>> I don't have any explicit version numbers in my requirements.txt file.  
> >>> <<<
> 
> When you don't have explicit version numbers, the system will install the 
> most recent. Sometimes the most recent version of one package causes issues 
> in another package and you are thus advised to stick with a lower version or 
> range of lower versions (i.e. a version is pinned). This is the warning 
> message you are getting here
> 
> In your example, your system has automatically installed google-api-core 
> 2.0.0 but your version of  google-cloud-datastore needs something lower than 
> 2.0.0 but a minimum of 1.14.0
> 
> To fix it, you should set explicit version numbers for the packages 
> mentioned, using the range or version mentioned in the error message you 
> received. You'll have to pick something that works for you. For example, you 
> can have google-api-core 1.14.0 in your requirements.txt file but this 
> assumes that there is nothing in your code that needs a feature that is in a 
> higher release
> 
>. NoCommandLine ..
>  https://nocommandline.com <https://nocommandline.com/>
> 
> A GUI for Google App Engine
> 
> On Tuesday, August 24, 2021 at 12:46:49 PM UTC-7 Joshua Smith wrote:
> I'm up to date with gcloud, and I'm getting this warning when I deploy:
> 
> WARNING: Found incompatible dependencies: "
> google-cloud-datastore 1.15.3 has requirement 
> google-api-core[grpc]<2.0.0dev,>=1.14.0,
>  but you have google-api-core 2.0.0.\n
> google-cloud-datastore 1.15.3 has requirement 
> google-cloud-core<2.0dev,>=1.4.0,
>  but you have google-cloud-core 2.0.0."
> 
> I don't have any explicit version numbers in my requirements.txt file. What 
> does this error mean, and how do I fix it?
> 
> -Joshua
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/80de2521-0ead-4362-b64c-97b204c93d69n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/80de2521-0ead-4362-b64c-97b204c93d69n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/52E39D66-E583-4F8F-A872-8E46ADEA3C24%40gmail.com.

[google-appengine] Weird Error Message

[Joshua Smith]

I'm up to date with gcloud, and I'm getting this warning when I deploy:

WARNING: Found incompatible dependencies: "
google-cloud-datastore 1.15.3 has requirement 
google-api-core[grpc]<2.0.0dev,>=1.14.0,
 but you have google-api-core 2.0.0.\n
google-cloud-datastore 1.15.3 has requirement google-cloud-core<2.0dev,>=1.4.0,
 but you have google-cloud-core 2.0.0."

I don't have any explicit version numbers in my requirements.txt file. What 
does this error mean, and how do I fix it?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7A0721E4-6833-4158-8FB8-E24C537EE622%40gmail.com.

Re: [google-appengine] Uploads to Blobstore failing

[Joshua Smith]

Search the list archives last October for this subject "Uploads to Legacy 
Blobstore are Failing with 500, nothing in the logs"

I ended up giving up and switching to new blobstore, but at the end of the 
thread, I think someone did identify the permission magic to fix the problem.

-Joshua

> On Mar 24, 2021, at 12:10 PM, Shaun Budhram  wrote:
> 
> Also for reference, this is what the permissions on the the bucket look like. 
>  I think these look correct, but I'm not sure?  These were created 
> automatically (except for my name, which I added as a test)
> 
> 
> 
> On Wednesday, March 24, 2021 at 8:15:04 AM UTC-7 Shaun Budhram wrote:
> Hi,
> 
> I have an older application using the Blobstore API.  I think I inadvertently 
> upgraded to Google Cloud Storage when I used 'gcloud' to deploy the app.  A 
> new storage bucket .appspot.com  was created.
> 
> The problem I'm having is that whenever my app uses 
> 'blobstore.create_upload_url' to create an upload URL, the the 
> upload/redirect to the handler is now failing.  I searched the forum and 
> found this post:
> 
> https://groups.google.com/g/google-appengine/c/0-U4xe0Dp0M/m/Lxc9AAw2AQAJ 
> 
> 
> This line in particular:
> 
> It created a new app-id.appspot.com  bucket but 
> my app engine service account did not have permissions. Once I gave it 
> permission it started working again with the blobstore items showing up in 
> the new "default" bucket.
> 
> This suggests that my app needs permission to the storage bucket, which 
> didn't happen automatically.  
> 
> My question is, is this a code-level change that I need to perform when 
> creating the upload url, or is this a configuration somewhere in the web 
> console?  Does anyone know how I can link permissions between this storage 
> bucket and my app engine service account for this app? 
> 
> Any help is greatly appreciated.  Thanks!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/546775b5-e326-4bed-a4bc-0d74adead697n%40googlegroups.com
>  
> .
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/A02BF1F5-640D-4610-8A7F-ACBCB11C9DC8%40gmail.com.

Re: [google-appengine] login: admin in python37

[Joshua Smith]

That seems easy enough, but there's something I don't understand. Where does 
the accessing user's identity come into play?

If I follow the directions the SO answer links to, I end up with something like 
this:

def adminPermission():
from googleapiclient import discovery
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
service = discovery.build('cloudresourcemanager', 'v1', 
credentials=credentials)
resource = "my-project-name"
test_iam_permissions_request_body = {
"permissions": [
"resourcemanager.projects.get"
]
}
request = service.projects().testIamPermissions(resource=resource, 
body=test_iam_permissions_request_body)
response = request.execute()
return len(response.get("permissions",[])) == 1

But all I'm checking in that code is whether my project has project permission, 
not whether the user making the request has project permission. If I hit a URL 
from curl with no auth that is returning True, since of course, my project has 
permission to get itself.

How do I get the credentials of the user accessing the URL?

-Joshua

> On Jan 28, 2021, at 10:56 AM, 'Emil' via Google App Engine 
>  wrote:
> 
> The builtin solution is no longer available, you can test iam permission in 
> your code as explained here <https://stackoverflow.com/a/52055488/12232507> 
> if it is feasible for you.
> 
> On Wednesday, January 27, 2021 at 8:28:10 PM UTC+1 Joshua Smith wrote:
> Interesting. I'll make that my fallback plan if nobody has an idea that 
> simply reproduces the old behavior (which was the exact behavior everyone 
> needed, by the way; why does Google always insist on replacing perfectly good 
> things with insanely complicated things that lack the one thing we all need?)
> 
> 
>> On Jan 27, 2021, at 2:13 PM, 'Charlie Engelke' via Google App Engine 
>> > > wrote:
>> 
> 
>> You can enable and configure IAP independently for different services, so if 
>> you can put all the admin functions in a separate service, that could do it.
>> 
>> On Wednesday, January 27, 2021 at 8:49:57 AM UTC-8 Joshua Smith wrote:
>> That does look super easy, but as you pointed out, it applies to the whole 
>> app. I just want pages in my /admin section to require auth.
>> 
>> 
>>> On Jan 27, 2021, at 11:44 AM, 'Jose V' via Google App Engine 
>>> > wrote:
>>> 
>> 
>>> Just in case it helps, you can also easily implement IAP for App Engine 
>>> <https://cloud.google.com/iap/docs/app-engine-quickstart>. The only 
>>> drawback would be that it secures the entire application, not specific 
>>> endpoints, which I am not sure is what you require in your scenario
>>> 
>>> On Monday, January 25, 2021 at 7:21:37 PM UTC+1 Joshua Smith wrote:
>>> I'm hoping someone on this list has encountered this, and can say "Yeah, I 
>>> ran into that. Here's how I dealt with it..."
>>> 
>>> I'm looking for an easy step-by-step guide to just get the exact behavior 
>>> that Python 2.7 had.
>>> 
>>> -Joshua
>>> 
>>> 
>>>> On Jan 25, 2021, at 1:05 PM, 'Elliott (Cloud Platform Support)' via Google 
>>>> App Engine > wrote:
>>>> 
>>> 
>>>> Hello Joshua,
>>>> 
>>>> I understand that you would like an easy way to implement authentication 
>>>> because you may not continue to use login: admin. I was able to confirm 
>>>> this. First, I would like to apologize for the inconvenience. There is no 
>>>> easy way other than to implement one of the options listed in this 
>>>> document 
>>>> <https://cloud.google.com/appengine/docs/standard/python3/authenticating-users>.
>>>> 
>>>> You are presented with some options including Firebase Authentication, 
>>>> Google Sign-In and OAuth 2.0 and OpenID Connect. Each of these 
>>>> technologies are described in the document.
>>>> 
>>>> I would like your thoughts on the recommended ways so that we may find a 
>>>> solution that agrees with you.
>>>> 
>>>> I hope that we may now have enough to elaborate on this discussion.
>>>> 
>>>> 
>>>> On Friday, January 22, 2021 at 1:07:16 PM UTC-5 Joshua Smith wrote:
>>>> In my quest to figure out how to keep using Google App Engine when 
>>>> Python27 eventually goes away, I've just run into yet another case where 
>>>> something simple seems to have been replaced with a nightmare of 
>>>

Re: [google-appengine] login: admin in python37

[Joshua Smith]

Interesting. I'll make that my fallback plan if nobody has an idea that simply 
reproduces the old behavior (which was the exact behavior everyone needed, by 
the way; why does Google always insist on replacing perfectly good things with 
insanely complicated things that lack the one thing we all need?)

> On Jan 27, 2021, at 2:13 PM, 'Charlie Engelke' via Google App Engine 
>  wrote:
> 
> You can enable and configure IAP independently for different services, so if 
> you can put all the admin functions in a separate service, that could do it.
> 
> On Wednesday, January 27, 2021 at 8:49:57 AM UTC-8 Joshua Smith wrote:
> That does look super easy, but as you pointed out, it applies to the whole 
> app. I just want pages in my /admin section to require auth.
> 
> 
>> On Jan 27, 2021, at 11:44 AM, 'Jose V' via Google App Engine 
>> > > wrote:
>> 
> 
>> Just in case it helps, you can also easily implement IAP for App Engine 
>> <https://cloud.google.com/iap/docs/app-engine-quickstart>. The only drawback 
>> would be that it secures the entire application, not specific endpoints, 
>> which I am not sure is what you require in your scenario
>> 
>> On Monday, January 25, 2021 at 7:21:37 PM UTC+1 Joshua Smith wrote:
>> I'm hoping someone on this list has encountered this, and can say "Yeah, I 
>> ran into that. Here's how I dealt with it..."
>> 
>> I'm looking for an easy step-by-step guide to just get the exact behavior 
>> that Python 2.7 had.
>> 
>> -Joshua
>> 
>> 
>>> On Jan 25, 2021, at 1:05 PM, 'Elliott (Cloud Platform Support)' via Google 
>>> App Engine > wrote:
>>> 
>> 
>>> Hello Joshua,
>>> 
>>> I understand that you would like an easy way to implement authentication 
>>> because you may not continue to use login: admin. I was able to confirm 
>>> this. First, I would like to apologize for the inconvenience. There is no 
>>> easy way other than to implement one of the options listed in this document 
>>> <https://cloud.google.com/appengine/docs/standard/python3/authenticating-users>.
>>> 
>>> You are presented with some options including Firebase Authentication, 
>>> Google Sign-In and OAuth 2.0 and OpenID Connect. Each of these technologies 
>>> are described in the document.
>>> 
>>> I would like your thoughts on the recommended ways so that we may find a 
>>> solution that agrees with you.
>>> 
>>> I hope that we may now have enough to elaborate on this discussion.
>>> 
>>> 
>>> On Friday, January 22, 2021 at 1:07:16 PM UTC-5 Joshua Smith wrote:
>>> In my quest to figure out how to keep using Google App Engine when Python27 
>>> eventually goes away, I've just run into yet another case where something 
>>> simple seems to have been replaced with a nightmare of complexity 
>>> <https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrating-services#user_authentication>.
>>> 
>>> In my old app.yaml, I had this:
>>> 
>>> - url: /admin/.*
>>>   script: main.app
>>>   secure: always
>>>   login: admin
>>> 
>>> Unfortunately, python37 doesn't support login: admin any more (!). I'm 
>>> facing a mountain of documentation detailing a bunch of different ways I 
>>> can do authentication now.
>>> 
>>> Stack overflow is no help at all in simplifying this.
>>> 
>>> Anyone here have advice on the easiest possible way to get the old Python27 
>>> behavior that you have to be logged in as the app administrator in order to 
>>> hit a certain URL?
>>> 
>>> This isn't for ensuring crons are only run by cron. That seems pretty easy 
>>> by looking at headers.
>>> 
>>> This is for when you have administrative functions that only the developers 
>>> need access to, and I'm looking for the easiest way to ensure a URL is only 
>>> accessible to those particular people.
>>> 
>>> In case it matters, I'm using Flask.
>>> 
>>> -Joshua
>>> 
>>> 
>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to google-appengi...@googlegroups.com <>.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/05ac62c4-6c44-4e7b-8068-1601d6a4eef0n%40googleg

Re: [google-appengine] login: admin in python37

[Joshua Smith]

That does look super easy, but as you pointed out, it applies to the whole app. 
I just want pages in my /admin section to require auth.

> On Jan 27, 2021, at 11:44 AM, 'Jose V' via Google App Engine 
>  wrote:
> 
> Just in case it helps, you can also easily implement IAP for App Engine 
> <https://cloud.google.com/iap/docs/app-engine-quickstart>. The only drawback 
> would be that it secures the entire application, not specific endpoints, 
> which I am not sure is what you require in your scenario
> 
> On Monday, January 25, 2021 at 7:21:37 PM UTC+1 Joshua Smith wrote:
> I'm hoping someone on this list has encountered this, and can say "Yeah, I 
> ran into that. Here's how I dealt with it..."
> 
> I'm looking for an easy step-by-step guide to just get the exact behavior 
> that Python 2.7 had.
> 
> -Joshua
> 
> 
>> On Jan 25, 2021, at 1:05 PM, 'Elliott (Cloud Platform Support)' via Google 
>> App Engine > > wrote:
>> 
> 
>> Hello Joshua,
>> 
>> I understand that you would like an easy way to implement authentication 
>> because you may not continue to use login: admin. I was able to confirm 
>> this. First, I would like to apologize for the inconvenience. There is no 
>> easy way other than to implement one of the options listed in this document 
>> <https://cloud.google.com/appengine/docs/standard/python3/authenticating-users>.
>> 
>> You are presented with some options including Firebase Authentication, 
>> Google Sign-In and OAuth 2.0 and OpenID Connect. Each of these technologies 
>> are described in the document.
>> 
>> I would like your thoughts on the recommended ways so that we may find a 
>> solution that agrees with you.
>> 
>> I hope that we may now have enough to elaborate on this discussion.
>> 
>> 
>> On Friday, January 22, 2021 at 1:07:16 PM UTC-5 Joshua Smith wrote:
>> In my quest to figure out how to keep using Google App Engine when Python27 
>> eventually goes away, I've just run into yet another case where something 
>> simple seems to have been replaced with a nightmare of complexity 
>> <https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrating-services#user_authentication>.
>> 
>> In my old app.yaml, I had this:
>> 
>> - url: /admin/.*
>>   script: main.app
>>   secure: always
>>   login: admin
>> 
>> Unfortunately, python37 doesn't support login: admin any more (!). I'm 
>> facing a mountain of documentation detailing a bunch of different ways I can 
>> do authentication now.
>> 
>> Stack overflow is no help at all in simplifying this.
>> 
>> Anyone here have advice on the easiest possible way to get the old Python27 
>> behavior that you have to be logged in as the app administrator in order to 
>> hit a certain URL?
>> 
>> This isn't for ensuring crons are only run by cron. That seems pretty easy 
>> by looking at headers.
>> 
>> This is for when you have administrative functions that only the developers 
>> need access to, and I'm looking for the easiest way to ensure a URL is only 
>> accessible to those particular people.
>> 
>> In case it matters, I'm using Flask.
>> 
>> -Joshua
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/05ac62c4-6c44-4e7b-8068-1601d6a4eef0n%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/05ac62c4-6c44-4e7b-8068-1601d6a4eef0n%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/86c21824-14b2-4bd7-97e7-d4756227f046n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/86c21824-14b2-4bd7-97e7-d4756227f046n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/ACFA0F11-AB14-47A2-A801-804572037972%40gmail.com.

Re: [google-appengine] login: admin in python37

[Joshua Smith]

I'm hoping someone on this list has encountered this, and can say "Yeah, I ran 
into that. Here's how I dealt with it..."

I'm looking for an easy step-by-step guide to just get the exact behavior that 
Python 2.7 had.

-Joshua

> On Jan 25, 2021, at 1:05 PM, 'Elliott (Cloud Platform Support)' via Google 
> App Engine  wrote:
> 
> Hello Joshua,
> 
> I understand that you would like an easy way to implement authentication 
> because you may not continue to use login: admin. I was able to confirm this. 
> First, I would like to apologize for the inconvenience. There is no easy way 
> other than to implement one of the options listed in this document 
> <https://cloud.google.com/appengine/docs/standard/python3/authenticating-users>.
> 
> You are presented with some options including Firebase Authentication, Google 
> Sign-In and OAuth 2.0 and OpenID Connect. Each of these technologies are 
> described in the document.
> 
> I would like your thoughts on the recommended ways so that we may find a 
> solution that agrees with you.
> 
> I hope that we may now have enough to elaborate on this discussion.
> 
> 
> On Friday, January 22, 2021 at 1:07:16 PM UTC-5 Joshua Smith wrote:
> In my quest to figure out how to keep using Google App Engine when Python27 
> eventually goes away, I've just run into yet another case where something 
> simple seems to have been replaced with a nightmare of complexity 
> <https://cloud.google.com/appengine/docs/standard/python/migrate-to-python3/migrating-services#user_authentication>.
> 
> In my old app.yaml, I had this:
> 
> - url: /admin/.*
>   script: main.app
>   secure: always
>   login: admin
> 
> Unfortunately, python37 doesn't support login: admin any more (!). I'm facing 
> a mountain of documentation detailing a bunch of different ways I can do 
> authentication now.
> 
> Stack overflow is no help at all in simplifying this.
> 
> Anyone here have advice on the easiest possible way to get the old Python27 
> behavior that you have to be logged in as the app administrator in order to 
> hit a certain URL?
> 
> This isn't for ensuring crons are only run by cron. That seems pretty easy by 
> looking at headers.
> 
> This is for when you have administrative functions that only the developers 
> need access to, and I'm looking for the easiest way to ensure a URL is only 
> accessible to those particular people.
> 
> In case it matters, I'm using Flask.
> 
> -Joshua
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/05ac62c4-6c44-4e7b-8068-1601d6a4eef0n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/05ac62c4-6c44-4e7b-8068-1601d6a4eef0n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/90827D18-4CCA-407F-92D4-3C2485FC0EBE%40gmail.com.

[google-appengine] login: admin in python37

[Joshua Smith]

In my quest to figure out how to keep using Google App Engine when Python27 
eventually goes away, I've just run into yet another case where something 
simple seems to have been replaced with a nightmare of complexity 
.

In my old app.yaml, I had this:

- url: /admin/.*
  script: main.app
  secure: always
  login: admin

Unfortunately, python37 doesn't support login: admin any more (!). I'm facing a 
mountain of documentation detailing a bunch of different ways I can do 
authentication now.

Stack overflow is no help at all in simplifying this.

Anyone here have advice on the easiest possible way to get the old Python27 
behavior that you have to be logged in as the app administrator in order to hit 
a certain URL?

This isn't for ensuring crons are only run by cron. That seems pretty easy by 
looking at headers.

This is for when you have administrative functions that only the developers 
need access to, and I'm looking for the easiest way to ensure a URL is only 
accessible to those particular people.

In case it matters, I'm using Flask.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6FA029D9-2ADB-4EFA-847B-56A1C15C33B1%40gmail.com.

Re: [google-appengine] Question about dispatch & backends

[Joshua Smith]

FYI, I've solved this puzzle.

The problem wasn't that the B8 instance was timing out in less than 60 seconds.

The problem was that I had the B8 set to max-instance = 1, so if two concurrent 
API requests came in, the dispatcher had nowhere to send the second request. It 
would hang around awhile, waiting for the B8 to free up, and eventually time 
out.

So the only "bug" here is that the error message from the dispatcher could be 
improved, by mentioning that the *reason* it timed out isn't that the server 
took too long, but rather that max-instances is too low.

-Joshua

> On Dec 14, 2020, at 11:26 AM, Linus Larsen  wrote:
> 
> I think I have seen some documentation way back that it was not possible to 
> serve user facing requests from a B instance, instead you
> were supposed to dispatch the request to a TaskQueue and use some sort of 
> client polling in order to get notified when the work on B
> instance was completed. 
> 
> Cannot find it anymore, the documentation has changed so much you don't 
> really know what's the truth anymore.
> 
> / Linus  
> 
> On Mon, Dec 14, 2020 at 3:13 PM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> I'm thinking you didn't read my initial message carefully, so let me try 
> again.
> 
> Yes, I'm using the legacy runtime with the 1 minute limit on frontends.
> 
> Yes, I'm using the B8 with basic scaling for my API calls, which should not 
> have that limit.
> 
> Yes, I'm using URL routing via dispatch.yaml, and have confirmed the B8 
> instance is handling the request by checking the logs.
> 
> The request is nonetheless failing because it's taking more than one minute. 
> 
> That should not be happening, according to the documentation. Unless there's 
> another 1 minute limit somewhere else that I'm not aware of.
> 
> -Joshua
> 
>> On Dec 13, 2020, at 8:58 PM, 'Manpreet Sidhu (Google Cloud Support)' via 
>> Google App Engine > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> Even though your frontend uses automatic scaling, the work is being done on 
>> the B8 instance that is responsible for running your API service.
>> 
>> Based on the screenshot that you attached, the B8 instance has to return in 
>> 24 hours. This is due to the fact that the B8 instance is only capable of 
>> Manual and Basic scaling[0].
>> 
>> Can it be safe to assume that you are using a Legacy Runtime as that is 
>> where the limit of 1 minute is enforced. As an alternative, you can try 
>> routing with URLs[1] or look into implementing the use of Task Queues as 
>> that has a timeout of 10 minutes.
>> 
>> [0]: https://cloud.google.com/appengine/docs/standard#instance_classes 
>> <https://cloud.google.com/appengine/docs/standard#instance_classes>
>> [1]: 
>> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls
>>  
>> <https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls>
>> On Saturday, December 12, 2020 at 10:11:28 AM UTC-5 Joshua Smith wrote:
>> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>>  
>> <https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout>
>> 
>> My frontend service uses automatic scaling, so it has a 1 minute limit to 
>> respond. But my API service is a B8 with this:
>> 
>> 
>> 
>> which means that it should not be subjected to that limit. Yet when I send 
>> requests to it via the dispatch, it seems to be enforcing at 1 minute limit 
>> regardless.
>> 
>> 
>> 
>> 
>>> On Dec 11, 2020, at 4:44 PM, 'Elliott (Cloud Platform Support)' via Google 
>>> App Engine > wrote:
>>> 
>> 
>>> Hello Joshua,
>>> 
>>> I understand that your program uses APIs that need more than 60 seconds to 
>>> complete but even though you are using B8 instances in App Engine, your 
>>> program times out past the 60 seconds. I’m assuming you are using App 
>>> Engine Standard.
>>> 
>>> To find supporting documentation for you, can you tell me more about the 
>>> "gotta get it done in 60 seconds" rule? Can you please give examples of 
>>> this?
>>> 
>>> 
>>> On Friday, December 11, 2020 at 12:01:48 PM UTC-5 Joshua Smith wrote:
>>> I have an app with a frontend UX that queries some backend APIs for 
>>> information. Some of these APIs need more than 60 seconds to complete their 
>>> work.
>>> 
>>> I *thought* I could solve this by doing the following:
>>&g

Re: [google-appengine] Issue 175627114: 60 second deadline being enforced in case where the documentation says it does not apply

[Joshua Smith]

The "Re-open" button is disabled.

> On Jan 4, 2021, at 10:04 AM, Cameron Brown  wrote:
> 
> If you can, just reopen the bug.
> 
> -C
> 
> On Mon, 4 Jan 2021, 14:16 Joshua Smith,  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> Process question. The issue tracker failed to send me a notification when 
> someone from Google asked a question, and since I didn't reply, they closed 
> the ticket. I've answered the question now, and asked them to re-open the 
> ticket. Is that going to work? Or do I need to start over with a new issue?
> 
> There's been no response for over a week, but maybe that's normal with the 
> holiday?
> 
> -Joshua
> 
>> Begin forwarded message:
>> 
>> From: mailto:buganizer-sys...@google.com>>
>> 
>> https://issuetracker.google.com/issues/175627114 
>> <https://issuetracker.google.com/issues/175627114>
>> 
>> Changed
>> status:  New  →  Not Reproducible
>> 
>> hj...@google.com <http://google.com/> added comment #3 
>> <https://issuetracker.google.com/issues/175627114#comment3>:
>> Hello, we haven’t received any reply from you in the past days, we hope that 
>> this is not an issue for you, I will proceed to close this case.
>> 
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/57F1817E-8282-4ACC-B359-D4FD3C2EAC81%40gmail.com
>  
> <https://groups.google.com/d/msgid/google-appengine/57F1817E-8282-4ACC-B359-D4FD3C2EAC81%40gmail.com?utm_medium=email_source=footer>.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/CAA5HY84CotfhuxKAeBx1LgbqCk5mn9-iaJCKYR-fpdrboX9-aw%40mail.gmail.com
>  
> <https://groups.google.com/d/msgid/google-appengine/CAA5HY84CotfhuxKAeBx1LgbqCk5mn9-iaJCKYR-fpdrboX9-aw%40mail.gmail.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2358A165-684E-4D90-B8BB-D3515DE60E63%40gmail.com.

[google-appengine] Fwd: Issue 175627114: 60 second deadline being enforced in case where the documentation says it does not apply

[Joshua Smith]

Process question. The issue tracker failed to send me a notification when 
someone from Google asked a question, and since I didn't reply, they closed the 
ticket. I've answered the question now, and asked them to re-open the ticket. 
Is that going to work? Or do I need to start over with a new issue?

There's been no response for over a week, but maybe that's normal with the 
holiday?

-Joshua

> Begin forwarded message:
> 
> From: 
> 
> https://issuetracker.google.com/issues/175627114 
> 
> 
> Changed
> status:  New  →  Not Reproducible
> 
> hj...@google.com  added comment #3 
> :
> Hello, we haven’t received any reply from you in the past days, we hope that 
> this is not an issue for you, I will proceed to close this case.
> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/57F1817E-8282-4ACC-B359-D4FD3C2EAC81%40gmail.com.

Re: [google-appengine] Question about dispatch & backends

[Joshua Smith]

I've created the issue in the tracker.

> On Dec 14, 2020, at 3:35 PM, 'Elliott (Cloud Platform Support)' via Google 
> App Engine  wrote:
> 
> Hello Joshua,
> 
> I’m sorry the conversation in this thread is going on and you do not have the 
> answer you need.  You indicated that the request is failing because it's 
> taking more than one minute and that is not how the behavior is supposed to 
> be according to documentation.
> 
> Because Google Groups is meant for general discussions, the next step is to 
> either report a common issue here 
> <https://issuetracker.google.com/issues/new?component=187191=0>, 
> since I could not find one at this time or go through regular support 
> channels here <https://cloud.google.com/support> to get 1 on 1 support.
> 
> I know this is frustrating after all the exchanges but one these options 
> would be the best way to move forward.
> 
> 
> On Monday, December 14, 2020 at 11:49:57 AM UTC-5 Joshua Smith wrote:
> When I hit the B8 with a request to its appspot.com <http://appspot.com/> 
> address, those happily run for more than a minute. The issue seems to be 
> related to dispatching. (I can't just have my front-end connect to the 
> appspot address, because I need some cookies to be present, so cross-domain 
> is out.)
> 
> -Joshua
> 
> 
>> On Dec 14, 2020, at 11:26 AM, Linus Larsen > > wrote:
>> 
> 
>> I think I have seen some documentation way back that it was not possible to 
>> serve user facing requests from a B instance, instead you
>> were supposed to dispatch the request to a TaskQueue and use some sort of 
>> client polling in order to get notified when the work on B
>> instance was completed. 
>> 
>> Cannot find it anymore, the documentation has changed so much you don't 
>> really know what's the truth anymore.
>> 
>> / Linus  
>> 
>> On Mon, Dec 14, 2020 at 3:13 PM Joshua Smith > > wrote:
>> I'm thinking you didn't read my initial message carefully, so let me try 
>> again.
>> 
>> Yes, I'm using the legacy runtime with the 1 minute limit on frontends.
>> 
>> Yes, I'm using the B8 with basic scaling for my API calls, which should not 
>> have that limit.
>> 
>> Yes, I'm using URL routing via dispatch.yaml, and have confirmed the B8 
>> instance is handling the request by checking the logs.
>> 
>> The request is nonetheless failing because it's taking more than one minute. 
>> 
>> That should not be happening, according to the documentation. Unless there's 
>> another 1 minute limit somewhere else that I'm not aware of.
>> 
>> -Joshua
>> 
>>> On Dec 13, 2020, at 8:58 PM, 'Manpreet Sidhu (Google Cloud Support)' via 
>>> Google App Engine >> > wrote:
>>> 
>>> Even though your frontend uses automatic scaling, the work is being done on 
>>> the B8 instance that is responsible for running your API service.
>>> 
>>> Based on the screenshot that you attached, the B8 instance has to return in 
>>> 24 hours. This is due to the fact that the B8 instance is only capable of 
>>> Manual and Basic scaling[0].
>>> 
>>> Can it be safe to assume that you are using a Legacy Runtime as that is 
>>> where the limit of 1 minute is enforced. As an alternative, you can try 
>>> routing with URLs[1] or look into implementing the use of Task Queues as 
>>> that has a timeout of 10 minutes.
>>> 
>>> [0]: https://cloud.google.com/appengine/docs/standard#instance_classes 
>>> <https://cloud.google.com/appengine/docs/standard#instance_classes>
>>> [1]: 
>>> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls
>>>  
>>> <https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls>
>>> On Saturday, December 12, 2020 at 10:11:28 AM UTC-5 Joshua Smith wrote:
>>> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>>>  
>>> <https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout>
>>> 
>>> My frontend service uses automatic scaling, so it has a 1 minute limit to 
>>> respond. But my API service is a B8 with this:
>>> 
>>> 
>>> 
>>> which means that it should not be subjected to that limit. Yet when I send 
>>> requests to it via the dispatch, it seems to be enforcing at 1 minute limit 
>>> regardless.
>>> 
>>> 
>>> 
>>> 
>>>> On Dec 11, 2020, at 4:44 PM, 'Elliott (Cloud Platform S

Re: [google-appengine] Question about dispatch & backends

[Joshua Smith]

When I hit the B8 with a request to its appspot.com <http://appspot.com/> 
address, those happily run for more than a minute. The issue seems to be 
related to dispatching. (I can't just have my front-end connect to the appspot 
address, because I need some cookies to be present, so cross-domain is out.)

-Joshua

> On Dec 14, 2020, at 11:26 AM, Linus Larsen  wrote:
> 
> I think I have seen some documentation way back that it was not possible to 
> serve user facing requests from a B instance, instead you
> were supposed to dispatch the request to a TaskQueue and use some sort of 
> client polling in order to get notified when the work on B
> instance was completed. 
> 
> Cannot find it anymore, the documentation has changed so much you don't 
> really know what's the truth anymore.
> 
> / Linus  
> 
> On Mon, Dec 14, 2020 at 3:13 PM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> I'm thinking you didn't read my initial message carefully, so let me try 
> again.
> 
> Yes, I'm using the legacy runtime with the 1 minute limit on frontends.
> 
> Yes, I'm using the B8 with basic scaling for my API calls, which should not 
> have that limit.
> 
> Yes, I'm using URL routing via dispatch.yaml, and have confirmed the B8 
> instance is handling the request by checking the logs.
> 
> The request is nonetheless failing because it's taking more than one minute. 
> 
> That should not be happening, according to the documentation. Unless there's 
> another 1 minute limit somewhere else that I'm not aware of.
> 
> -Joshua
> 
>> On Dec 13, 2020, at 8:58 PM, 'Manpreet Sidhu (Google Cloud Support)' via 
>> Google App Engine > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> Even though your frontend uses automatic scaling, the work is being done on 
>> the B8 instance that is responsible for running your API service.
>> 
>> Based on the screenshot that you attached, the B8 instance has to return in 
>> 24 hours. This is due to the fact that the B8 instance is only capable of 
>> Manual and Basic scaling[0].
>> 
>> Can it be safe to assume that you are using a Legacy Runtime as that is 
>> where the limit of 1 minute is enforced. As an alternative, you can try 
>> routing with URLs[1] or look into implementing the use of Task Queues as 
>> that has a timeout of 10 minutes.
>> 
>> [0]: https://cloud.google.com/appengine/docs/standard#instance_classes 
>> <https://cloud.google.com/appengine/docs/standard#instance_classes>
>> [1]: 
>> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls
>>  
>> <https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls>
>> On Saturday, December 12, 2020 at 10:11:28 AM UTC-5 Joshua Smith wrote:
>> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>>  
>> <https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout>
>> 
>> My frontend service uses automatic scaling, so it has a 1 minute limit to 
>> respond. But my API service is a B8 with this:
>> 
>> 
>> 
>> which means that it should not be subjected to that limit. Yet when I send 
>> requests to it via the dispatch, it seems to be enforcing at 1 minute limit 
>> regardless.
>> 
>> 
>> 
>> 
>>> On Dec 11, 2020, at 4:44 PM, 'Elliott (Cloud Platform Support)' via Google 
>>> App Engine > wrote:
>>> 
>> 
>>> Hello Joshua,
>>> 
>>> I understand that your program uses APIs that need more than 60 seconds to 
>>> complete but even though you are using B8 instances in App Engine, your 
>>> program times out past the 60 seconds. I’m assuming you are using App 
>>> Engine Standard.
>>> 
>>> To find supporting documentation for you, can you tell me more about the 
>>> "gotta get it done in 60 seconds" rule? Can you please give examples of 
>>> this?
>>> 
>>> 
>>> On Friday, December 11, 2020 at 12:01:48 PM UTC-5 Joshua Smith wrote:
>>> I have an app with a frontend UX that queries some backend APIs for 
>>> information. Some of these APIs need more than 60 seconds to complete their 
>>> work.
>>> 
>>> I *thought* I could solve this by doing the following:
>>> 
>>> 1. Run the code on a B8 instance with a different service name
>>> 2. Set up dispatch.yaml to direct the incoming API requests to that service:
>>> 
>>> dispatch:
>>> 
>>> - url: "*/api/*"
>>>   ser

Re: [google-appengine] Production problem this day on get_service_account_name()

[Joshua Smith]

I saw one of those in my logs, too. Perhaps it was related to the global outage 
google had in account management earlier today.

> On Dec 14, 2020, at 7:58 AM, Patrice Bertrand 
>  wrote:
> 
> Hello,
> Today, for the first time, we experienced an api failure in production:   
> between 12:50 and 13:50 CET all calls to get_service_account_name() failed, 
> with no other information than "Internal Error".   
> I am surprised I cannot find anything about this incident in the "Cloud 
> Platform Status" information panel.
> Is it just me ?   But then why would it be just today and just for one hour ?
> Thank you for any relevant information.
> Patrice
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/ef085948-55bf-4150-b670-a1071b0d6c81n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/63DE0291-DD93-433E-B4AD-A12891B10338%40gmail.com.

Re: [google-appengine] Question about dispatch & backends

[Joshua Smith]

I'm thinking you didn't read my initial message carefully, so let me try again.

Yes, I'm using the legacy runtime with the 1 minute limit on frontends.

Yes, I'm using the B8 with basic scaling for my API calls, which should not 
have that limit.

Yes, I'm using URL routing via dispatch.yaml, and have confirmed the B8 
instance is handling the request by checking the logs.

The request is nonetheless failing because it's taking more than one minute. 

That should not be happening, according to the documentation. Unless there's 
another 1 minute limit somewhere else that I'm not aware of.

-Joshua

> On Dec 13, 2020, at 8:58 PM, 'Manpreet Sidhu (Google Cloud Support)' via 
> Google App Engine  wrote:
> 
> Even though your frontend uses automatic scaling, the work is being done on 
> the B8 instance that is responsible for running your API service.
> 
> Based on the screenshot that you attached, the B8 instance has to return in 
> 24 hours. This is due to the fact that the B8 instance is only capable of 
> Manual and Basic scaling[0].
> 
> Can it be safe to assume that you are using a Legacy Runtime as that is where 
> the limit of 1 minute is enforced. As an alternative, you can try routing 
> with URLs[1] or look into implementing the use of Task Queues as that has a 
> timeout of 10 minutes.
> 
> [0]: https://cloud.google.com/appengine/docs/standard#instance_classes 
> <https://cloud.google.com/appengine/docs/standard#instance_classes>
> [1]: 
> https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls
>  
> <https://cloud.google.com/appengine/docs/standard/python/how-requests-are-routed#urls>
> On Saturday, December 12, 2020 at 10:11:28 AM UTC-5 Joshua Smith wrote:
> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>  
> <https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout>
> 
> My frontend service uses automatic scaling, so it has a 1 minute limit to 
> respond. But my API service is a B8 with this:
> 
> 
> 
> which means that it should not be subjected to that limit. Yet when I send 
> requests to it via the dispatch, it seems to be enforcing at 1 minute limit 
> regardless.
> 
> 
> 
> 
>> On Dec 11, 2020, at 4:44 PM, 'Elliott (Cloud Platform Support)' via Google 
>> App Engine > > wrote:
>> 
> 
>> Hello Joshua,
>> 
>> I understand that your program uses APIs that need more than 60 seconds to 
>> complete but even though you are using B8 instances in App Engine, your 
>> program times out past the 60 seconds. I’m assuming you are using App Engine 
>> Standard.
>> 
>> To find supporting documentation for you, can you tell me more about the 
>> "gotta get it done in 60 seconds" rule? Can you please give examples of this?
>> 
>> 
>> On Friday, December 11, 2020 at 12:01:48 PM UTC-5 Joshua Smith wrote:
>> I have an app with a frontend UX that queries some backend APIs for 
>> information. Some of these APIs need more than 60 seconds to complete their 
>> work.
>> 
>> I *thought* I could solve this by doing the following:
>> 
>> 1. Run the code on a B8 instance with a different service name
>> 2. Set up dispatch.yaml to direct the incoming API requests to that service:
>> 
>> dispatch:
>> 
>> - url: "*/api/*"
>>   service: reporting
>> 
>> - url: "*/*"
>>   service: default
>> 
>> The dispatching is working correctly. I see this in my logs for an /api hit:
>> 
>> 
>> However, it's timing out:
>> 
>> 
>> 
>> My understanding was that a B8 instance didn't have the "gotta get it done 
>> in 60 seconds" rule that frontend instance types did.
>> 
>> What am I missing?
>> 
>> -Joshua
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/7209920b-7b59-4dd4-84a6-e722e7df462bn%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/7209920b-7b59-4dd4-84a6-e722e7df462bn%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to go

[google-appengine] Question about dispatch & backends

[Joshua Smith]

I have an app with a frontend UX that queries some backend APIs for 
information. Some of these APIs need more than 60 seconds to complete their 
work.

I *thought* I could solve this by doing the following:

1. Run the code on a B8 instance with a different service name
2. Set up dispatch.yaml to direct the incoming API requests to that service:

dispatch:

- url: "*/api/*"
  service: reporting

- url: "*/*"
  service: default

The dispatching is working correctly. I see this in my logs for an /api hit:


However, it's timing out:



My understanding was that a B8 instance didn't have the "gotta get it done in 
60 seconds" rule that frontend instance types did.

What am I missing?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4F4943B1-BE85-434C-A819-33BB353CFD8C%40gmail.com.

Re: [google-appengine] Google App Engine Cost

[Joshua Smith]

I more than a dozen App Engine apps (using cloud datastore and standard (not 
flex) GAE), and in my experience, the only non-negligible costs are:

1. Instances
2. Datastore reads

Instances tracks your request rate. For example, I have a very traditional 
web/db python app, and it usually has about 1 F1 instance per 0.1 requests/sec. 
(So if the request rate reached 0.5rps, then I'd have 5 instances.) This app 
has several hundred daily users, and it almost never needs more than one 
instance. (The first instance is free.) The only time my request rate goes up 
is when a bot ignores the crawl-delay in my robots.txt, or when somebody's 
calendar widget goes bonkers and starts hammering my ICS handler. I have alerts 
set up against budget, so when this happens I can investigate and block them 
with a firewall rule.

As for datastore reads, that's ridiculously hard to predict. It depends so much 
on what your app does, and how aggressively you cache frequently-accessed 
stuff. But looking at all my apps together, instances are usually half the 
bill, and datastore reads is the other half. So here's a quick estimator for 
monthly cost, based on my experience.

$ = 2 * ( ( floor(10 * RPS) - 1 ) * $36 )

RPS is requests per second, and you are probably over-estimating that by an 
order of magnitude.

-Joshua

> On Dec 7, 2020, at 5:15 PM, Stesca com  wrote:
> 
> I am looking into the Google AppEngine for deploying one application. The 
> price calculator is painting a very scary picture, so I decided to ask this 
> here.
> 
> My question is from all of you who have deployed their applications onto the 
> AppEngine and have load from moderate to heavy and the instances run 24/7. 
> 
> What is your average cost per month :
> Only AppEngine
> Full solution with Cloud SQL, Storage & Firebase etc.
> I would highly appreciate your help.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/536027e1-7f1c-4281-a1d6-14cfd2357da0n%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/881BBC18-C88A-43ED-9E7F-6FAC00990EF3%40gmail.com.

Re: [google-appengine] Google Translate API & Permissions

[Joshua Smith]

This problem is more systemic than that. The documentation lacks a step-by-step 
how-to. I understand that there are a lot of options, but I'm pretty sure my 
use case is very common:

1. I have a google app engine app
2. I want to call the translation API from that app

I figured out how to do that from the translation API docs, but I lacked 
permission.

I could not find any step by step instructions for giving permission. I found 
this:

https://cloud.google.com/translate/docs/setup

which pointed me to this:

https://cloud.google.com/iam/docs/understanding-roles

at which point a normal human is going to be ready to give up. Look at that 
second page! It's an endless stream of gobledegook.

After getting the pointer from Amit that I need to add the role to IAM, I went 
to IAM & Admin in my console, and selected "roles". Spoiler alert: this is not 
where you do this.

Amit mentioned "service accounts" so I went looking for that. I chose the 
"Service Accounts" section. I see my service account, and it has a "..." 
Actions menu. None of those actions are about adding roles.

I click on the link for my service account, and see there's a tab for 
"permissions". Doesn't seem to be anywhere I can give it permissions there.

Honestly, I've just given up again. I know that by fumbling around in the 
console I eventually ended up on a page where it wanted me to type an email 
address. And I eventually figured out that it wanted the email address of the 
service account. But for the life of me, I can't find that now.

You need to write a how-to for giving an app engine app permission to access an 
API. Or if such a thing already exists, you need to elevate it to make it 
findable.

-Joshua

> On Dec 4, 2020, at 2:17 PM, wesley chun  wrote:
> 
> Thanks for your feedback Joshua. Can you take a screenshot of the Cloud 
> Console where you think there should be a message or that is misleading, then 
> go to the appropriate page in the documentation you think additional 
> messaging would help, and click on the "Send feedback" button in the upper 
> right corner of that page? The tool will also let you highlight where on the 
> docs page your messaging should go. This will greatly help the team analyze 
> your feedback and take appropriate action as necessary.
> 
> Cheers,
> --Wesley
> 
> On Fri, Nov 20, 2020 at 5:33 AM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> That worked. Thank you so much.
> 
> You guys need to work on your documentation. I never could have figured that 
> out myself. And the process of adding that role to the service account was 
> also weird. (The UX is asking for the email of a new user, when what I need 
> to add is a service account that isn’t a new user, and isn’t even really an 
> email.)
> 
> -Joshua
> 
>> On Nov 19, 2020, at 2:19 PM, 'Amit Sinha' via Google App Engine 
>> > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> Hello Joshua, could you try to add the “Cloud Translation API User” role in 
>> the service account from IAM? As of this [1] documentation, it includes the 
>> permission that showing in the error.
>> 
>> [1] 
>> https://cloud.google.com/iam/docs/understanding-roles#cloud-translation-roles
>>  
>> <https://cloud.google.com/iam/docs/understanding-roles#cloud-translation-roles>
>> On Monday, November 16, 2020 at 2:23:03 PM UTC-5 Joshua Smith wrote:
>> I don’t think Google could have come up with a more confusing and convoluted 
>> system for API permission management if they tried.
>> 
>> So I’ve enabled the “cloud translation” API within my project.
>> 
>> I have some Python (2.7, old school) code that goes:
>> 
>> credentials = 
>> ServiceAccountCredentials.from_json_keyfile_name(CLIENT_SECRETS_FILE, 
>> scopes=['https://www.googleapis.com/auth/cloud-platform’ 
>> <https://www.googleapis.com/auth/cloud-platform%E2%80%99>])
>> http_auth = credentials.authorize(Http())
>> service = build("translation", "v3", http=http_auth)
>> service.projects().translateText(parent=“projects/my-project-id-here",body={"contents":"bonjour",
>>  "targetLanguageCode":"en"}).execute()
>> 
>> And I get:
>> 
>> googleapiclient.errors.HttpError: > https://translation.googleapis.com/v3/projects/ 
>> <https://translation.googleapis.com/v3/projects/>my-project-id-here:translateText?alt=json
>>  returned "Cloud IAM permission 'cloudtranslate.generalModels.predict' 
>> denied.">
>> 
>> So it seems like I need to check some box somewhere to make this work, but I 
>> can’t figure out where.
>> 
>>

Re: [google-appengine] Google Translate API & Permissions

[Joshua Smith]

That worked. Thank you so much.

You guys need to work on your documentation. I never could have figured that 
out myself. And the process of adding that role to the service account was also 
weird. (The UX is asking for the email of a new user, when what I need to add 
is a service account that isn’t a new user, and isn’t even really an email.)

-Joshua

> On Nov 19, 2020, at 2:19 PM, 'Amit Sinha' via Google App Engine 
>  wrote:
> 
> Hello Joshua, could you try to add the “Cloud Translation API User” role in 
> the service account from IAM? As of this [1] documentation, it includes the 
> permission that showing in the error.
> 
> [1] 
> https://cloud.google.com/iam/docs/understanding-roles#cloud-translation-roles
> 
> On Monday, November 16, 2020 at 2:23:03 PM UTC-5 Joshua Smith wrote:
> I don’t think Google could have come up with a more confusing and convoluted 
> system for API permission management if they tried.
> 
> So I’ve enabled the “cloud translation” API within my project.
> 
> I have some Python (2.7, old school) code that goes:
> 
> credentials = 
> ServiceAccountCredentials.from_json_keyfile_name(CLIENT_SECRETS_FILE, 
> scopes=['https://www.googleapis.com/auth/cloud-platform’ 
> <https://www.googleapis.com/auth/cloud-platform%E2%80%99>])
> http_auth = credentials.authorize(Http())
> service = build("translation", "v3", http=http_auth)
> service.projects().translateText(parent=“projects/my-project-id-here",body={"contents":"bonjour",
>  "targetLanguageCode":"en"}).execute()
> 
> And I get:
> 
> googleapiclient.errors.HttpError:  https://translation.googleapis.com/v3/projects/ 
> <https://translation.googleapis.com/v3/projects/>my-project-id-here:translateText?alt=json
>  returned "Cloud IAM permission 'cloudtranslate.generalModels.predict' 
> denied.">
> 
> So it seems like I need to check some box somewhere to make this work, but I 
> can’t figure out where.
> 
> The service account in the client secrets file is working fine when I use it 
> to hit Google Analytics from the same app. And in fact, my code to access GA 
> is almost identical except the scope and service call.
> 
> Any ideas?
> 
> -Joshua
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/38511647-af55-4440-b8a9-47a2d26e1f42n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/38511647-af55-4440-b8a9-47a2d26e1f42n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/96693B33-8943-4948-8444-7F9ABAAAB70D%40gmail.com.

[google-appengine] Google Translate API & Permissions

[Joshua Smith]

I don’t think Google could have come up with a more confusing and convoluted 
system for API permission management if they tried.

So I’ve enabled the “cloud translation” API within my project.

I have some Python (2.7, old school) code that goes:

credentials = 
ServiceAccountCredentials.from_json_keyfile_name(CLIENT_SECRETS_FILE, 
scopes=['https://www.googleapis.com/auth/cloud-platform’])
http_auth = credentials.authorize(Http())
service = build("translation", "v3", http=http_auth)
service.projects().translateText(parent=“projects/my-project-id-here",body={"contents":"bonjour",
 "targetLanguageCode":"en"}).execute()

And I get:

googleapiclient.errors.HttpError: https://translation.googleapis.com/v3/projects/my-project-id-here:translateText?alt=json
 returned "Cloud IAM permission 'cloudtranslate.generalModels.predict' denied.">

So it seems like I need to check some box somewhere to make this work, but I 
can’t figure out where.

The service account in the client secrets file is working fine when I use it to 
hit Google Analytics from the same app. And in fact, my code to access GA is 
almost identical except the scope and service call.

Any ideas?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/DB3B5905-5B08-4A6C-8CC9-789E61E49712%40gmail.com.

Re: [google-appengine] GAE Outage

[Joshua Smith]

No errors in my projects. I’d recommend trying a re-deploy of your existing 
code, with a new version number.

> On Nov 13, 2020, at 8:25 AM, Rob Curtis  wrote:
> 
> Busy experiencing this issue now. Anyone else experiencing 500 errors where 
> instances aren't starting? We haven't changed any code, it just started 
> happening.
> 
> On Monday, November 9, 2020 at 3:58:05 PM UTC+2 Joshua Smith wrote:
> I don’t think so, since nobody responded with a +1 and the whole thing was 
> never acknowledged by Google.
> 
> What I ended up doing was re-deploying with a new version number. The problem 
> went away when I did that, although I don’t know for sure if that means I 
> fixed it, or if it just happened to resolve itself at the same time.
> 
> 
>> On Nov 8, 2020, at 12:14 PM, Mog Obahor > > wrote:
>> 
> 
>> Was this a fleet wide issue across all users that use GAE?
>> 
>> On Thursday, October 22, 2020 at 8:51:36 PM UTC-5 Joshua Smith wrote:
>> Seems to have resolved itself. Looks like the outage started at 7pm EDT and 
>> ended just after 9pm EDT...
>> 
>> 
>> 
>> 
>>> On Oct 22, 2020, at 9:07 PM, Joshua Smith > wrote:
>>> 
>> 
>>> My app is getting a lot of this error:
>>> 
>> 
>>> 
>> 
>>> 
>>> It started at 8:30. I tried updating my version and migrating traffic and 
>>> that helped a little, but it seems the problem is still happening on and 
>>> off.
>>> 
>>> -Joshua
>>> 
>>> 
>>> 
>> 
>> 
> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/6ca163dd-0e1c-4e09-b602-09ba5ce74b3an%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/6ca163dd-0e1c-4e09-b602-09ba5ce74b3an%40googlegroups.com?utm_medium=email_source=footer>.
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/07c5879d-061d-44c9-bb79-1be840027a35n%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/07c5879d-061d-44c9-bb79-1be840027a35n%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/71CA972F-7DB1-40DF-A1BE-DB4C29B56E45%40gmail.com.

Re: [google-appengine] GAE Outage

[Joshua Smith]

I don’t think so, since nobody responded with a +1 and the whole thing was 
never acknowledged by Google.

What I ended up doing was re-deploying with a new version number. The problem 
went away when I did that, although I don’t know for sure if that means I fixed 
it, or if it just happened to resolve itself at the same time.

> On Nov 8, 2020, at 12:14 PM, Mog Obahor  wrote:
> 
> Was this a fleet wide issue across all users that use GAE?
> 
> On Thursday, October 22, 2020 at 8:51:36 PM UTC-5 Joshua Smith wrote:
> Seems to have resolved itself. Looks like the outage started at 7pm EDT and 
> ended just after 9pm EDT...
> 
> 
> 
> 
>> On Oct 22, 2020, at 9:07 PM, Joshua Smith > > wrote:
>> 
> 
>> My app is getting a lot of this error:
>> 
> 
>> 
> 
>> 
>> It started at 8:30. I tried updating my version and migrating traffic and 
>> that helped a little, but it seems the problem is still happening on and off.
>> 
>> -Joshua
>> 
>> 
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/6ca163dd-0e1c-4e09-b602-09ba5ce74b3an%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/6ca163dd-0e1c-4e09-b602-09ba5ce74b3an%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6812116A-1DDF-4F6E-8E56-8C15454F81B0%40gmail.com.

[google-appengine] GAE Outage

[Joshua Smith]

My app is getting a lot of this error:



It started at 8:30. I tried updating my version and migrating traffic and that 
helped a little, but it seems the problem is still happening on and off.

-Joshua



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/8A6AF475-A943-4422-9A76-1770273DBC9A%40gmail.com.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Indeed, there is no such account with any permission on that bucket.

So there you have it. That’s probably what happened. (Doesn’t matter anymore 
for me, since I’m now using cloudstorage for uploads, not blobstore.)

Google really botched this transition from appcfg to gcloud.

-Joshua

> On Oct 9, 2020, at 4:42 AM, 'Jose V' via Google App Engine 
>  wrote:
> 
> Hello,
> 
> I have seen previous occurrences in which deploying with "gcloud app deploy" 
> led to 50X errors when uploading to blobstore. It was indeed due to the App 
> Engine Service account lacking permissions on the newly created Cloud Storage 
> bucket.
> Note that by default, the Service account used by App Engine has a name like 
> "[PROJECT-ID]@appspot.gserviceaccount.com" and the Cloud Storage Bucket will 
> have a name like "[PROJECT-ID].appspot.com <http://appspot.com/>"
> 
> Could you confirm that the App Engine service account has write permissions 
> for the bucket? See Cloud Storage Roles [1] for reference.
> 
> Regards
> [1]: https://cloud.google.com/storage/docs/access-control/iam-roles 
> <https://cloud.google.com/storage/docs/access-control/iam-roles>
> 
> On Thursday, October 8, 2020 at 7:23:05 PM UTC+2 Will Reiher wrote:
> I don't remember what the error code was anymore. I just knew that something 
> was f'd. It was really a hunch that that new bucket was involved and that if 
> my app didn't have permission it might be the answer. I was prepared for a 
> very long day but after the permission change it all just started working 
> again - with the blobs showing up in that new bucket.
> 
> Most of my stuff is using Cloud Storage except for user uploads. I'm working 
> to replace that with a UUID based storage name so that I can maintain the 
> ability to allow for same-named files to be uploaded.
> 
> Of course today Cloud Task UI is not available and it wants me to enable the 
> API (I've been using it for months now) Nervous that my queues will be erased 
> if I enable it.
> 
> On Thursday, October 8, 2020 at 10:03:22 AM UTC-7 Joshua Smith wrote:
> Do you recall if you got 500 errors? I’d expect that to generate 403’s.
> 
> Regardless, that does sound like the kind of thing that happened.
> 
> Switching to use cloudstorage instead of blobstore wasn’t too hard, so that’s 
> the workaround I’d recommend for anyone else who hits this issue.
> 
> I’m mostly just annoyed that google has switched to a support model where 
> they won’t provide timely support unless you pay them $100/month. That’s 
> greedy and stupid.
> 
> -Joshua
> 
> 
>> On Oct 8, 2020, at 12:58 PM, Will Reiher > 
>> wrote:
>> 
> 
>> I had an issue a few weeks back. It was due to my transition to gcloud 
>> deploy. It created a new app-id.appspot.com <http://app-id.appspot.com/> 
>> bucket but my app engine service account did not have permissions. Once I 
>> gave it permission it started working again with the blobstore items showing 
>> up in the new "default" bucket.
>> 
>> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
>> 
>> Hi,
>> 
>> I have deployed this Blobstore API sample 
>> <https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/blobstore/api>
>>  service and it works well and allows me to successfully upload images. You 
>> could try to add further logging statements in your app to know what is 
>> causing your issue. 
>> 
>> On the other hand, I recommend you to fully migrate you app to Python3 since 
>> Python 2 is no longer supported by the community and use Cloud Storage 
>> instead of Blobstore.
>> 
>> And also note that appcfg tooling was shuted down 
>> <https://cloud.google.com/appengine/docs/deprecations> on August 30th, 2020.
>> 
>> 
>> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
>> Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
>> couple weeks ago.
>> 
>> I did use curl and it revealed nothing. Just a terse 500 with no content 
>> from the server.
>> 
>> 
>>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus > 
>>> wrote:
>>> 
>> 
>>> Hey Joshua,
>>> 
>>> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
>>> think that running “gcloud app deploy” introduced the issue, can you try 
>>> redeploying the application using appcfg.py? 
>>> 
>>> You could also use curl commands to troubleshoot the upload to blobstore.. 
>>> 
>>> Thanks,
>>> 
>>> Char 

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Do you recall if you got 500 errors? I’d expect that to generate 403’s.

Regardless, that does sound like the kind of thing that happened.

Switching to use cloudstorage instead of blobstore wasn’t too hard, so that’s 
the workaround I’d recommend for anyone else who hits this issue.

I’m mostly just annoyed that google has switched to a support model where they 
won’t provide timely support unless you pay them $100/month. That’s greedy and 
stupid.

-Joshua

> On Oct 8, 2020, at 12:58 PM, Will Reiher  wrote:
> 
> I had an issue a few weeks back. It was due to my transition to gcloud 
> deploy. It created a new app-id.appspot.com bucket but my app engine service 
> account did not have permissions. Once I gave it permission it started 
> working again with the blobstore items showing up in the new "default" bucket.
> 
> On Thursday, October 8, 2020 at 3:13:28 AM UTC-7 barrado wrote:
> 
> Hi,
> 
> I have deployed this Blobstore API sample 
> <https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/standard/blobstore/api>
>  service and it works well and allows me to successfully upload images. You 
> could try to add further logging statements in your app to know what is 
> causing your issue. 
> 
> On the other hand, I recommend you to fully migrate you app to Python3 since 
> Python 2 is no longer supported by the community and use Cloud Storage 
> instead of Blobstore.
> 
> And also note that appcfg tooling was shuted down 
> <https://cloud.google.com/appengine/docs/deprecations> on August 30th, 2020.
> 
> 
> On Wednesday, October 7, 2020 at 4:17:44 PM UTC+2 Joshua Smith wrote:
> Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
> couple weeks ago.
> 
> I did use curl and it revealed nothing. Just a terse 500 with no content from 
> the server.
> 
> 
>> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus > 
>> wrote:
>> 
> 
>> Hey Joshua,
>> 
>> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
>> think that running “gcloud app deploy” introduced the issue, can you try 
>> redeploying the application using appcfg.py? 
>> 
>> You could also use curl commands to troubleshoot the upload to blobstore.. 
>> 
>> Thanks,
>> 
>> Char 
>> 
>> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith > wrote:
>> The problem persists.
>> 
>> It occurs to me that last week was the first time I had to make a small 
>> update to the app in a while, and I did that using “gcloud app deploy”  
>> instead of appcfg.py for the first time on that project.
>> 
>> Is it possible that pushing the app using gcloud introduced a problem 
>> uploading to blobstore? I wouldn’t think they’d be related, but the timing 
>> is suspicious.
>> 
>> Downloading from blobstore still works. It’s only uploads that are failing.
>> 
>> My users are getting upset, and your lack of feedback or an ETA is not 
>> helping.
>> 
>> -Joshua
>> 
>>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>>> > wrote:
>>> 
>> 
>>> Hi, Joshua
>>> 
>>> Thank you for reporting this issue. It seems you may be affected by an 
>>> internal issue reported about App Engine standard applications returning 
>>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>>> instances in us-central1 region. This issue is externalized on the GCP 
>>> Status Page as you can find more information on the Status board[1].
>>> 
>>> I apologize for the delay or the impact this issue may be causing. For 
>>> swifter responses or help on issues that are urgent to you, I think it is 
>>> best to reach out directly to the GCP Support Engineers[2]. 
>>> 
>>> Let us know if your application is not running in the us-central1 region.  
>>> 
>>> Thank you. 
>>> 
>>> [1]https://status.cloud.google.com/incident/appengine/20007 
>>> <https://status.cloud.google.com/incident/appengine/20007>
>>> [2]https://cloud.google.com/support-hub 
>>> <https://cloud.google.com/support-hub>
>>> 
>>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>>> The silence here is deafening, but my users are howling.
>>> 
>>> Google: Please address this issue immediately! It’s been down at least 24 
>>> hours!
>>> 
>>> I created an issue in public issue tracker with the same info you see here: 
>>> 170195261 <https://issuetracker.google.com/issues/170195261>
>>> 
>&

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Unfortunately, I used gcloud app deploy because appcfg.py stopped working a 
couple weeks ago.

I did use curl and it revealed nothing. Just a terse 500 with no content from 
the server.

> On Oct 6, 2020, at 4:14 PM, Charounson Saintilus  
> wrote:
> 
> Hey Joshua,
> 
> I don’t think I’m gonna be of much help here, but wanted to mention: if you 
> think that running “gcloud app deploy” introduced the issue, can you try 
> redeploying the application using appcfg.py? 
> 
> You could also use curl commands to troubleshoot the upload to blobstore.. 
> 
> Thanks,
> 
> Char 
> 
> On Tue, Oct 6, 2020 at 3:40 PM Joshua Smith  <mailto:mrjoshuaesm...@gmail.com>> wrote:
> The problem persists.
> 
> It occurs to me that last week was the first time I had to make a small 
> update to the app in a while, and I did that using “gcloud app deploy”  
> instead of appcfg.py for the first time on that project.
> 
> Is it possible that pushing the app using gcloud introduced a problem 
> uploading to blobstore? I wouldn’t think they’d be related, but the timing is 
> suspicious.
> 
> Downloading from blobstore still works. It’s only uploads that are failing.
> 
> My users are getting upset, and your lack of feedback or an ETA is not 
> helping.
> 
> -Joshua
> 
>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>> > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
> 
>> Hi, Joshua
>> 
>> Thank you for reporting this issue. It seems you may be affected by an 
>> internal issue reported about App Engine standard applications returning 
>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>> instances in us-central1 region. This issue is externalized on the GCP 
>> Status Page as you can find more information on the Status board[1].
>> 
>> I apologize for the delay or the impact this issue may be causing. For 
>> swifter responses or help on issues that are urgent to you, I think it is 
>> best to reach out directly to the GCP Support Engineers[2]. 
>> 
>> Let us know if your application is not running in the us-central1 region.  
>> 
>> Thank you. 
>> 
>> [1]https://status.cloud.google.com/incident/appengine/20007 
>> <https://status.cloud.google.com/incident/appengine/20007>
>> [2]https://cloud.google.com/support-hub 
>> <https://cloud.google.com/support-hub>
>> 
>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>> The silence here is deafening, but my users are howling.
>> 
>> Google: Please address this issue immediately! It’s been down at least 24 
>> hours!
>> 
>> I created an issue in public issue tracker with the same info you see here: 
>> 170195261 <https://issuetracker.google.com/issues/170195261>
>> 
>> I highly doubt I’m the only one experiencing this.
>> 
>> -Joshua
>> 
>> 
>>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > wrote:
>>> 
>>> There is a production issue in my app. (I’m happy to create a private 
>>> production issue ticket, but I couldn’t figure out where to do that.)
>>> 
>>> My app is still on Python 2.7 and uses the legacy blobstore.
>>> 
>>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>>> been working fine for several years.
>>> 
>>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>> 
>>> As of this morning, attempts to upload to the blobstore are resulting in 
>>> error 500 and I don’t see the request in my logs.
>>> 
>>> I simulated one of these uploads using curl, but that produced no insight:
>>> 
>>> * Using HTTP2, server supports multi-use
>>> * Connection state changed (HTTP/2 confirmed)
>>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>>> len=0
>>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>>> > Host: —redacted— 
>>> > User-Agent: curl/7.64.1
>>> > Accept: */*
>>> > Content-Length: 12881
>>> > Content-Type: multipart/form-data; 
>>> > boundary=c1bce33de7c355c9
>>> > 
>>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>>> * We are completely uploaded and fine
>>> < HTTP/2 500 
>>> < x-guploader-uploadid: 
>>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>>> < content-length: 0

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

Sigh.

I’ve gone ahead and re-coded my app to use cloudstorage instead of blobstore 
for new uploads. So this is no longer critical.

(Thankfully, downloads have continued to work. If you break blobstore 
downloads, that’ll be a real problem for me.)

The idea of using a cloud provider is they:

1. Keep the infrastructure working, and
2. Respond quickly when there are outages.

If I have to spend $100/month to get that level of service, it changes the 
economics completely.

I’m very disappointed in GCS right now.

-Joshua

> On Oct 6, 2020, at 3:39 PM, Joshua Smith  wrote:
> 
> The problem persists.
> 
> It occurs to me that last week was the first time I had to make a small 
> update to the app in a while, and I did that using “gcloud app deploy”  
> instead of appcfg.py for the first time on that project.
> 
> Is it possible that pushing the app using gcloud introduced a problem 
> uploading to blobstore? I wouldn’t think they’d be related, but the timing is 
> suspicious.
> 
> Downloading from blobstore still works. It’s only uploads that are failing.
> 
> My users are getting upset, and your lack of feedback or an ETA is not 
> helping.
> 
> -Joshua
> 
>> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>> > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> Hi, Joshua
>> 
>> Thank you for reporting this issue. It seems you may be affected by an 
>> internal issue reported about App Engine standard applications returning 
>> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
>> instances in us-central1 region. This issue is externalized on the GCP 
>> Status Page as you can find more information on the Status board[1].
>> 
>> I apologize for the delay or the impact this issue may be causing. For 
>> swifter responses or help on issues that are urgent to you, I think it is 
>> best to reach out directly to the GCP Support Engineers[2]. 
>> 
>> Let us know if your application is not running in the us-central1 region.  
>> 
>> Thank you. 
>> 
>> [1]https://status.cloud.google.com/incident/appengine/20007 
>> <https://status.cloud.google.com/incident/appengine/20007>
>> [2]https://cloud.google.com/support-hub 
>> <https://cloud.google.com/support-hub>
>> 
>> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
>> The silence here is deafening, but my users are howling.
>> 
>> Google: Please address this issue immediately! It’s been down at least 24 
>> hours!
>> 
>> I created an issue in public issue tracker with the same info you see here: 
>> 170195261 <https://issuetracker.google.com/issues/170195261>
>> 
>> I highly doubt I’m the only one experiencing this.
>> 
>> -Joshua
>> 
>> 
>>> On Oct 5, 2020, at 11:50 AM, Joshua Smith >> > wrote:
>>> 
>>> There is a production issue in my app. (I’m happy to create a private 
>>> production issue ticket, but I couldn’t figure out where to do that.)
>>> 
>>> My app is still on Python 2.7 and uses the legacy blobstore.
>>> 
>>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>>> been working fine for several years.
>>> 
>>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>>> 
>>> As of this morning, attempts to upload to the blobstore are resulting in 
>>> error 500 and I don’t see the request in my logs.
>>> 
>>> I simulated one of these uploads using curl, but that produced no insight:
>>> 
>>> * Using HTTP2, server supports multi-use
>>> * Connection state changed (HTTP/2 confirmed)
>>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>>> len=0
>>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>>> > Host: —redacted— 
>>> > User-Agent: curl/7.64.1
>>> > Accept: */*
>>> > Content-Length: 12881
>>> > Content-Type: multipart/form-data; 
>>> > boundary=c1bce33de7c355c9
>>> > 
>>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>>> * We are completely uploaded and fine
>>> < HTTP/2 500 
>>> < x-guploader-uploadid: 
>>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>>> < content-length: 0
>>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>>> < server: UploadServer
>>> < content-type: text/

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

The problem persists.

It occurs to me that last week was the first time I had to make a small update 
to the app in a while, and I did that using “gcloud app deploy”  instead of 
appcfg.py for the first time on that project.

Is it possible that pushing the app using gcloud introduced a problem uploading 
to blobstore? I wouldn’t think they’d be related, but the timing is suspicious.

Downloading from blobstore still works. It’s only uploads that are failing.

My users are getting upset, and your lack of feedback or an ETA is not helping.

-Joshua

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> Hi, Joshua
> 
> Thank you for reporting this issue. It seems you may be affected by an 
> internal issue reported about App Engine standard applications returning 
> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
> instances in us-central1 region. This issue is externalized on the GCP Status 
> Page as you can find more information on the Status board[1].
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 
> 
> Let us know if your application is not running in the us-central1 region.  
> 
> Thank you. 
> 
> [1]https://status.cloud.google.com/incident/appengine/20007 
> <https://status.cloud.google.com/incident/appengine/20007>
> [2]https://cloud.google.com/support-hub <https://cloud.google.com/support-hub>
> 
> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
> The silence here is deafening, but my users are howling.
> 
> Google: Please address this issue immediately! It’s been down at least 24 
> hours!
> 
> I created an issue in public issue tracker with the same info you see here: 
> 170195261 <https://issuetracker.google.com/issues/170195261>
> 
> I highly doubt I’m the only one experiencing this.
> 
> -Joshua
> 
> 
>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > > wrote:
>> 
>> There is a production issue in my app. (I’m happy to create a private 
>> production issue ticket, but I couldn’t figure out where to do that.)
>> 
>> My app is still on Python 2.7 and uses the legacy blobstore.
>> 
>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>> been working fine for several years.
>> 
>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>> 
>> As of this morning, attempts to upload to the blobstore are resulting in 
>> error 500 and I don’t see the request in my logs.
>> 
>> I simulated one of these uploads using curl, but that produced no insight:
>> 
>> * Using HTTP2, server supports multi-use
>> * Connection state changed (HTTP/2 confirmed)
>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>> len=0
>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>> > Host: —redacted— 
>> > User-Agent: curl/7.64.1
>> > Accept: */*
>> > Content-Length: 12881
>> > Content-Type: multipart/form-data; 
>> > boundary=c1bce33de7c355c9
>> > 
>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>> * We are completely uploaded and fine
>> < HTTP/2 500 
>> < x-guploader-uploadid: 
>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>> < content-length: 0
>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>> < server: UploadServer
>> < content-type: text/html; charset=UTF-8
>> < 
>> * Connection #0 to host mytowngovernment.org <http://mytowngovernment.org/> 
>> left intact
>> * Closing connection 0
>> 
>> Although perhaps that uploadid might help you diagnose?
>> 
>> Please advise.
>> 
>> -Joshua
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9EC69CDD-FDB4-4CEA-AFA2-BC03919C300A%40gmail.com.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 

My reading of that the cases page is that I’d have to pay you $100/month for 
the privilege of reporting outages that way. (The free tier says it’s 
read-only.)

Am I missing something?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/87208FE3-1BB7-400A-8F76-BDB9946B9EBE%40gmail.com.

Re: [google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

It pre-dated that issue, and it is continuing despite the larger outage being 
resolved.

Please address this ASAP.

Thanks.

-Joshua

> On Oct 6, 2020, at 11:47 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> Hi, Joshua
> 
> Thank you for reporting this issue. It seems you may be affected by an 
> internal issue reported about App Engine standard applications returning 
> elevated HTTP 500 errors. This issue is observed for App Engine  Standard 
> instances in us-central1 region. This issue is externalized on the GCP Status 
> Page as you can find more information on the Status board[1].
> 
> I apologize for the delay or the impact this issue may be causing. For 
> swifter responses or help on issues that are urgent to you, I think it is 
> best to reach out directly to the GCP Support Engineers[2]. 
> 
> Let us know if your application is not running in the us-central1 region.  
> 
> Thank you. 
> 
> [1]https://status.cloud.google.com/incident/appengine/20007 
> <https://status.cloud.google.com/incident/appengine/20007>
> [2]https://cloud.google.com/support-hub <https://cloud.google.com/support-hub>
> 
> On Tuesday, October 6, 2020 at 10:58:39 AM UTC-4 Joshua Smith wrote:
> The silence here is deafening, but my users are howling.
> 
> Google: Please address this issue immediately! It’s been down at least 24 
> hours!
> 
> I created an issue in public issue tracker with the same info you see here: 
> 170195261 <https://issuetracker.google.com/issues/170195261>
> 
> I highly doubt I’m the only one experiencing this.
> 
> -Joshua
> 
> 
>> On Oct 5, 2020, at 11:50 AM, Joshua Smith > > wrote:
>> 
>> There is a production issue in my app. (I’m happy to create a private 
>> production issue ticket, but I couldn’t figure out where to do that.)
>> 
>> My app is still on Python 2.7 and uses the legacy blobstore.
>> 
>> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
>> been working fine for several years.
>> 
>> The last successful one was: 02/Oct/2020:11:37:23 -0400
>> 
>> As of this morning, attempts to upload to the blobstore are resulting in 
>> error 500 and I don’t see the request in my logs.
>> 
>> I simulated one of these uploads using curl, but that produced no insight:
>> 
>> * Using HTTP2, server supports multi-use
>> * Connection state changed (HTTP/2 confirmed)
>> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
>> len=0
>> * Using Stream ID: 1 (easy handle 0x7fa064808200)
>> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
>> > Host: —redacted— 
>> > User-Agent: curl/7.64.1
>> > Accept: */*
>> > Content-Length: 12881
>> > Content-Type: multipart/form-data; 
>> > boundary=c1bce33de7c355c9
>> > 
>> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
>> * We are completely uploaded and fine
>> < HTTP/2 500 
>> < x-guploader-uploadid: 
>> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
>> < content-length: 0
>> < date: Mon, 05 Oct 2020 15:38:41 GMT
>> < server: UploadServer
>> < content-type: text/html; charset=UTF-8
>> < 
>> * Connection #0 to host mytowngovernment.org <http://mytowngovernment.org/> 
>> left intact
>> * Closing connection 0
>> 
>> Although perhaps that uploadid might help you diagnose?
>> 
>> Please advise.
>> 
>> -Joshua
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/9e37a2e5-ef32-481c-87f2-fff0f807c5dan%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/17DE773C-4DAF-491C-890B-58DA53512FB3%40gmail.com.

[google-appengine] Re: Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

The silence here is deafening, but my users are howling.

Google: Please address this issue immediately! It’s been down at least 24 hours!

I created an issue in public issue tracker with the same info you see here: 
170195261 <https://issuetracker.google.com/issues/170195261>

I highly doubt I’m the only one experiencing this.

-Joshua

> On Oct 5, 2020, at 11:50 AM, Joshua Smith  wrote:
> 
> There is a production issue in my app. (I’m happy to create a private 
> production issue ticket, but I couldn’t figure out where to do that.)
> 
> My app is still on Python 2.7 and uses the legacy blobstore.
> 
> I use blobstore.create_upload_url('/upload’) to populate a form. This has 
> been working fine for several years.
> 
> The last successful one was: 02/Oct/2020:11:37:23 -0400
> 
> As of this morning, attempts to upload to the blobstore are resulting in 
> error 500 and I don’t see the request in my logs.
> 
> I simulated one of these uploads using curl, but that produced no insight:
> 
> * Using HTTP2, server supports multi-use
> * Connection state changed (HTTP/2 confirmed)
> * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: 
> len=0
> * Using Stream ID: 1 (easy handle 0x7fa064808200)
> > POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
> > Host: —redacted— 
> > User-Agent: curl/7.64.1
> > Accept: */*
> > Content-Length: 12881
> > Content-Type: multipart/form-data; 
> > boundary=c1bce33de7c355c9
> > 
> * Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
> * We are completely uploaded and fine
> < HTTP/2 500 
> < x-guploader-uploadid: 
> ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
> < content-length: 0
> < date: Mon, 05 Oct 2020 15:38:41 GMT
> < server: UploadServer
> < content-type: text/html; charset=UTF-8
> < 
> * Connection #0 to host mytowngovernment.org <http://mytowngovernment.org/> 
> left intact
> * Closing connection 0
> 
> Although perhaps that uploadid might help you diagnose?
> 
> Please advise.
> 
> -Joshua
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/B333BF3B-5C67-41AF-95FB-A6903DE81359%40gmail.com.

[google-appengine] Uploads to Legacy Blobstore are Failing with 500, nothing in the logs

[Joshua Smith]

There is a production issue in my app. (I’m happy to create a private 
production issue ticket, but I couldn’t figure out where to do that.)

My app is still on Python 2.7 and uses the legacy blobstore.

I use blobstore.create_upload_url('/upload’) to populate a form. This has been 
working fine for several years.

The last successful one was: 02/Oct/2020:11:37:23 -0400

As of this morning, attempts to upload to the blobstore are resulting in error 
500 and I don’t see the request in my logs.

I simulated one of these uploads using curl, but that produced no insight:

* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa064808200)
> POST /_ah/upload/AMmfu6YC1hHSvQPFczp—redacted—Pt3zS4LmedW5GR/ HTTP/2
> Host: —redacted— 
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Length: 12881
> Content-Type: multipart/form-data; 
> boundary=c1bce33de7c355c9
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
* We are completely uploaded and fine
< HTTP/2 500 
< x-guploader-uploadid: 
ABg5-UyVR-AC3_kY-A3vD45EgWUWJjzynvU_S0rDjhhsIlyJsDmYRU-ZrjSLe9Pppe7eCd-S9dHAv_7dVcsvfMj5wS0
< content-length: 0
< date: Mon, 05 Oct 2020 15:38:41 GMT
< server: UploadServer
< content-type: text/html; charset=UTF-8
< 
* Connection #0 to host mytowngovernment.org left intact
* Closing connection 0

Although perhaps that uploadid might help you diagnose?

Please advise.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9D0D7E6F-5B1E-4A36-8BE8-60BFA030BF61%40gmail.com.

Re: [google-appengine] Spending LImits Going Away :(

[Joshua Smith]

On this particular topic of Cloud Datastore Read Operations being the cost 
driver, unfortunately caching doesn’t help my case.

This site has about 60,000 individual meetings listed, and I like having 
*useful* *well-behaved* crawlers find all those meetings so people can use 
search engines to find meetings that happened in various towns. But what keeps 
happening is some new useless/poorly-behaved crawler will decide to read all 
60,000 of those meetings as fast as it possibly can. (Ignoring the 
crawl-delay.) Caching in that situation would just add *more* cost, since there 
are no repeat hits from a crawler.

I periodically look at my access logs, find peaks, look at the requests, and 
figure out what new bot needs to be added to my robots.txt disallow list. And 
sometimes I have to add a firewall rule because that bot isn’t obeying 
robots.txt at all.

-Joshua

> On Aug 26, 2020, at 4:47 PM, 'yananc' via Google App Engine 
>  wrote:
> 
> Back to the issue of ‘Cloud Datastore Read Operations’ being too high, a 
> possible solution is to leverage cache mechanism to avoid excessive 
> operations. You may find more information from the topic [2].

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/068BD581-9C44-4E59-A235-61DD0239FFBA%40gmail.com.

Re: [google-appengine] Spending LImits Going Away :(

[Joshua Smith]

about latency possibilities.
> 
> If you're away from the computer, you have notification options here[2], 
> called "channels".
> 
> [1] 
> https://cloud.google.com/monitoring/alerts/concepts-indepth#notification-latency
> [2] 
> https://cloud.google.com/monitoring/support/notification-options#creating_channels
> 
> On Tuesday, August 25, 2020 at 12:03:33 PM UTC-4 Joshua Smith wrote:
> Once again last night, my wallet was saved when a runaway bot chewed up my 
> site’s whole daily spending limit. I got an email from a user, set up a 
> firewall rule, and goosed my budget to get things going again.
> 
> I’m very concerned about Google’s decision to remove this feature. Offering a 
> cloud service that bills by usage without having a way to limit the spend 
> shifts an unreasonable amount of risk onto the subscriber.
> 
> I’ve set up budget alerts, as suggested, but I’m concerned that:
> 
> - What if my bill shoots up really fast? How quickly is this alert going to 
> go out?
> 
> - What if I am away from the computer (remember when we used to be able to 
> leave our houses? good times… good times…)?
> 
> I run this particular site as a not-for-profit social good. (It’s a site that 
> small town governments use to post their meetings.) I make no money on it.
> 
> I’d be perfectly happy to handle this with self-set quotas on something other 
> than dollars. For example, in my case the budget-buster is always “Cloud 
> Datastore Read Operations.” If I could set a cap on that one thing, it’d give 
> me the protection I need.
> 
> -Joshua
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/f77ae31b-61bd-4251-bb5d-4809076b70fan%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/f77ae31b-61bd-4251-bb5d-4809076b70fan%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7664FA4C-11CE-430F-9427-398EFA670B92%40gmail.com.

[google-appengine] Spending LImits Going Away :(

[Joshua Smith]

Once again last night, my wallet was saved when a runaway bot chewed up my 
site’s whole daily spending limit. I got an email from a user, set up a 
firewall rule, and goosed my budget to get things going again.

I’m very concerned about Google’s decision to remove this feature. Offering a 
cloud service that bills by usage without having a way to limit the spend 
shifts an unreasonable amount of risk onto the subscriber.

I’ve set up budget alerts, as suggested, but I’m concerned that:

- What if my bill shoots up really fast? How quickly is this alert going to go 
out?

- What if I am away from the computer (remember when we used to be able to 
leave our houses? good times… good times…)?

I run this particular site as a not-for-profit social good. (It’s a site that 
small town governments use to post their meetings.) I make no money on it.

I’d be perfectly happy to handle this with self-set quotas on something other 
than dollars. For example, in my case the budget-buster is always “Cloud 
Datastore Read Operations.” If I could set a cap on that one thing, it’d give 
me the protection I need.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/FC0F0C74-0D40-48DF-8919-208202A9B1A8%40gmail.com.

Re: [google-appengine] Network error when attempting to fetch resource

[Joshua Smith]

Look at the network activity debugger in your browser, or try hitting the URL 
using curl -v from the terminal.

That will generally help you sort out what headers are being included.

The rule you included would only add those headers to images, not your JSON 
stuff, but maybe that was just an example?

FWIW, I think it’s way more natural to do this in the python code:

  self.response.headers["Access-Control-Allow-Origin"] = "*"


> On May 11, 2020, at 10:03 AM, Nutchapol Dendumrongsup 
>  wrote:
> 
> Hello fellow developers and data scientists. I have a question about a 
> problem with accessing the API deployed on the google app engine from the 
> javascript app deployed on Heroku. 
> 
> When I access my API link on google chrome, it works properly. However, when 
> I use my javascript app deployed on Heroku to call my API link, it sometimes 
> does not work and return me the network error. 
> 
> I have followed the instruction on 
> "https://cloud.google.com/appengine/docs/standard/python3/config/appref; to 
> enable the CORS access on my app API  in the app.ymal file on google app 
> engine as the following  
> 
> runtime: python37instance_class: F4_1Ghandlers:
> - url: /images
>   static_dir: static/images
>   http_headers:
> Access-Control-Allow-Origin: ‘*’
> 
> Similarly, I have also enabled the CORS access on my javascript app deployed 
> on Heroku as shown in the picture attached. However, the problem still 
> persists. I am not sure if it has to do with the way we set up our code or 
> the google app engine server itself. If there is anything I could do besides 
> what I have done I would love to hear that as well. Thanks a lot!
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/0a5daf81-3a94-4fe7-b5bb-0edfcb8a83f9%40googlegroups.com
>  
> .
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/F759CF66-FA1B-4800-BC59-60205B3A99D0%40gmail.com.

Re: [google-appengine] Re: Alternative to IN (Filter)

[Joshua Smith]

Do a whole lot of queries in tasks and collect the results together. That’s all 
an IN filter on a nosql dB would do behind the scenes anyway. If that doesn’t 
work for you get more clever in your schema to include something you can filter 
on to get you a superset. 

> On May 10, 2020, at 4:52 AM, hr lumiins  wrote:
> 
> 
> Hi Team,
> 
> Can i can support from google cloud team  or someone who can solve this 
> issue. ?
> 
> 
>> On Friday, May 8, 2020 at 10:28:40 PM UTC+2, Skm Villa wrote:
>> Hi Team,
>> 
>> My code : 
>> 
>> List cells;
>> return ofy().load().type(User.class).filter("geocells in", 
>> cells).limit(50).list();
>> 
>> Error 
>> [e~taxidealsnl/standard-project:dev.426523627618302404].: 2020-05-08 
>> 11:48:52.783  INFO 1 --- [Request0C1E463F] 
>> c.l.m.c.ExceptionHandlerController   : Error ==The Cloud Datastore SDK 
>> does not currently support 'IN' filters
>> 
>> Could you pls alterrnative solution for above code.
>> 
>> it is complicated 
>> 
>> I have 100K records, i cant scan whole DB for it...
>> https://cloud.google.com/appengine/docs/standard/python/ndb/queries#neq_and_in
>> 
>> Thx.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/233518b1-6957-424d-87d7-ee8ce51b2d1d%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/E9F9DB76-DD16-4944-AC47-4922B7501C06%40gmail.com.

Re: [google-appengine] Any more migration stories?

[Joshua Smith]

I have several mission critical services running in P2.7. About 13,000 lines of 
python code.

To get a feel for what P3.7 would be like, I used the new environment to create 
a couple new services. It was not fun. Everything about the new environment is 
10x more complicated, working with a local database is well neigh impossible, 
and the new code patterns are different enough from the old code patterns, that 
I’m looking at a line-by-line rewrite.

For example, to delete something from google.appengine.ext.db

model.delete()

To delete something from google.cloud.ndb

model.key.delete()

It’s a small change, but since Python is not compiled, nothing is going to help 
me find that other than rigorous search and runtime errors.

All told, I think migrating all my services from P2.7 to P3.7 is going to take 
at least a couple person-months of labor. It’s a nightmare.

Google completely botched this. They should have created polyfills so we can 
just deal with the few language change issues, but not have to switch to an 
entirely different set of services.

-Joshua

> On May 5, 2020, at 4:17 PM, 'Charlie Engelke' via Google App Engine 
>  wrote:
> 
> I'd still like to here anyone's stories involving migrating their App Engine 
> apps from Python 2.7 to Python 3.7. Have you migrated yet? Planning to do so 
> soon? Not going to move to the new platform?
> 
> And any particular pain points? Things that would have helped? 
> 
> Thanks,
> 
> Charlie
> 
> On Friday, May 1, 2020 at 2:18:33 PM UTC-7 Charlie Engelke wrote:
> Have you recently migrated from App Engine for Python 2.7 to App Engine for 
> Python 3.7? Are you working on migrating now, or planning to soon? Or maybe 
> you are using App Engine for Python 2.7 and aren't planning such a migration.
> 
> In each of those cases, I'd like to hear how things went or are going for 
> you, pain points you encounter, things that went easily, and anything else 
> you have to share. I'm working on creating some new code samples to help with 
> this, and your stories will be a big help.
> 
> Thanks,
> 
> Charlie
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/850cd6fa-77a4-436d-8d7e-40eab67f84eb%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6A942BC1-FD7E-4D35-816A-3B1B53F75886%40gmail.com.

Re: [google-appengine] Diving into App Engine is soooooooooooo hard now - My analysis

[Joshua Smith]

> On Apr 19, 2020, at 1:01 AM, MdeA  wrote:
> 
> Now I'm just wondering what will happen to all my web apps running on GAE 
> after July, 2020.

Short answer: They will continue to work as they have been.

We need to migrate from Py2.7 to Py3.7 and the new stack, but google is giving 
us years to get it done.

At least, that how I read the notices google has sent out.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0F5EE449-D458-486A-91B4-430D5B6B8DBB%40gmail.com.

[google-appengine] Outage?

[Joshua Smith]

The dashboard shows all is well, but I’m getting errors when I try to deploy an 
Python2.7 app using the old appcfg.py

HTTP Error 500: Internal Server Error. Aborting. 
Error 500: --- begin server output ---
Server ErrorA server error has occurred.
--- end server output ---


When I look at Activity in my dashboard (hoping to see what the error actually 
is), that’s reporting an error as well:



Just me? Or is this widespread?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1C77E9F1-D2B7-466E-AE9B-CACF4A1CC52F%40gmail.com.

Re: [google-appengine] Diving into App Engine is soooooooooooo hard now - My analysis

[Joshua Smith]

It’s interesting to look at what Google is doing with Firebase.

1. They are encouraging us to move more of the processing into the client, 
which while often architecturally foolish, does eliminate the need for some of 
the PaaS stuff.

2. They have a serverless/cloud functions thing over there, which overlaps a 
lot with the PaaS stuff we’ve been able to do in GAE. I haven’t dug into that 
too much, but on the surface it looks like the spaghetti antipattern.

Given that this is google, and that it’s PaaS all over again, I think the risk 
of putting any eggs in the Firebase basket seems almost unbearably high. Is 
Google going to turn their back on Firebase when the next shiny object 
distracts them? Why should we believe that any service google offers will last 
more than their committed 3 years?

I’m super tired of having to go back and rewrite my critical business 
infrastructure every few years.

(Google’s competitors in the PaaS space don’t inspire any more confidence, so 
this isn’t so much a google problem as a cloud problem, I think.)

-Joshua

> On Mar 20, 2020, at 12:59 PM, Patrice B  wrote:
> 
> 
> Hi guys, 
> 
> A few months back, I had a similar reflexion on current trends regarding 
> AppEngine, which I called "the end of paas", which could be called "end of 
> serverless" in a way
> 
> https://groups.google.com/forum/?utm_medium=email_source=footer#!search/paas|sort:date/google-appengine/DHMMv5r8qjU/A-VO3PXPDwAJ
> 
> It's not very long, so I paste it here, since it is quite similar to what is 
> being said here.
> 
> It seems most of the services that made AppEngine a proper Platorm as a 
> Service are now scheduled to be shut down, and users are advised to migrate.  
>  Migrate search to ElasticSearch, migrate memcache to Redis, and maybe at 
> some point we'll be asked to migrate Ndb to MongoDB and GCS to whatever.   
> I'm not complaining about the way the process is handled actually, there is 
> enough time to consider the options and work on a migration scenario, there 
> is no imminent deadline, at least for the moment.   But I'm wondering what 
> went wrong with the PaaS approach, and is it officially dead.
> 
> Is this the end of GAE as a PaaS ?   I truly believed PaaS was the future of 
> cloud architectures:  stop thinking in terms of servers, start thinking in 
> terms of services.   When I started working with AppEngine, I dreamed of CPU 
> as a service, with no server granularity, and I was disappointed to find I 
> still had to worry about servers, starting up a new server instance, choosing 
> what type of instance would be best, a scaling strategy, etc.   I was 
> expecting servers as a service, i.e. serving my requests without me ever 
> thinking in terms of servers.   But at least there was Ndb, search, memcache, 
> GCS and a few more.
> 
> Now it seems all of these are on their way out, which makes me wonder: was 
> there something wrong with the concept of PaaS itself, or is it just that 
> these products didn't gain traction, and are now too costly to maintain with 
> regards to their user base ?   Actually, the one thing that was wrong with 
> Paas from the very beginning was that it would lock a project into a given 
> cloud.   That was a risk to be reckoned for users, but it could have been 
> seen as a feature for the cloud provider.   Now is it for this reason that 
> the mentionned services didn't make it ?  Because users would have been wary 
> of being locked in, and for that reason would prefer to use leading products 
> deployed on leased servers ?One thing is for sure: once an application 
> has migrated to more standard services, it will not be tied to GCP anymore.
> 
> There was a major benefit to the PaaS concept:  it was very cheap for 
> startups.   Deploying ElasticSearch on a the smallest possible cluster will 
> start at around $200 a month, while the search usage of a small application 
> could cost less than $10.   Same for the shared memcache service offered by 
> AppEngine.   Now you are having to pay for running servers all night with 
> very little transactions to handle. 
> 
> I'd be happy to hear your thoughts on this matter ? 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/6edf3459-a4d0-4ea4-9282-86cdb6a2672f%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to 

Re: [google-appengine] Diving into App Engine is soooooooooooo hard now - My analysis

[Joshua Smith]

Anyone tried Beanstalk? Is it any better?

I’ve recently had a run-in with Azure where I had to store some stuff for a 
customer in their S3-like blob store service. It was a nightmare. Insanely 
over-engineeered to the point that it becomes impossible to figure out how to 
do the simplest things. Much like GAE’s trajectory.

Ima coin a new law: All PAAS services will expand until they are 
indistinguishable from IAAS.

> On Mar 16, 2020, at 2:46 AM, Nickolas Daskalou  wrote:
> 
> Same experience for us.
> 
> 10 years ago App Engine was a revolution. Now it's almost revolting.
> 
> Agree with the hard to set up development environment for the newer App 
> Engine installments.
> 
> Other pain points too, like no shared Memcache replacement (we're told NDB 
> for Python 3 is fully ready, when in reality a lot of existing apps which 
> rely on the built in feature of the free shared Memcache will either suffer 
> performance issues and/or an increase in billing cost if they switch over).
> 
> Google are having a hard time working out how to crack the cloud computing 
> market. Enterprises go with Azure because Microsoft are already heavily 
> embedded and sell the business fluff well, and AWS have a massive head start 
> on startups due to a larger existing user base and service offering because 
> they were first in the game.
> 
> Nick
> 
> On Mon, 16 Mar 2020, 4:35 am Matthew Parkes,  > wrote:
> I have to totally agree with this.  My experience has been the same coming 
> from AppEngine Standard python 2.7 I decided to try and migrate one of my 
> apps to Python3.7 but was extremely frustrated and put off with the whole 
> experience.  I lost a lot of nice framework that made decisions easy and 
> straight forward to actually just get functionality working. (Endpoints API, 
> User management, Data management, Admin datastore).  I found it very hard to 
> get an actual development environment setup where I could rapidly develop and 
> test, see data in the database etc.
> 
> The local testing has become very difficult. It's sad that while google 
> appears to say "Oh AppEngine so much more flexible for you now" sounds like a 
> step forward, when really it's more like 1 step forward, 5 steps backwards.
> 
> Disappointed,
> 
> Matt
> 
> On Sunday, 15 March 2020 08:48:56 UTC-7, Brian Ruuska wrote:
> Glad to hear I'm not the only one disappointed with it all. I started using 
> App Engine when it was in beta, and launched a one-person business when it 
> went GA. It's so overwhelming attempting to port my product to the new 
> environments that it's much more than a one-person effort. And with all the 
> land mines and missing functionality I decided to just close the business 
> last year. They seem to be more interested in serving big enterprises, and us 
> little guys are just left behind. :-(
> 
> On Saturday, March 14, 2020 at 7:22:15 AM UTC-5, Kaan Soral wrote:
> I'll whine a lot, but there's a concentrated and crisp TL;DR in the end
> 
> 
> 
> I've been using App Engine I guess for 10 years now? It's hard for me too, 
> but also, I'm preparing to open source my current project, at each step I 
> question everything, I was going to open source my game so young 
> developers/kids/gamers could dive into networked game development easier 
> (It's an MMORPG on Google Cloud, my dream was that open sourcing it could 
> enable fast experimentation with the genre), but as it is, at the current 
> state of App Engine, the bottleneck is definitely all the diversified and 
> separated products, separate emulators (missing ones), separate environment 
> variables for everything etc. - I can't imagine a happy scenario where one 
> could just develop an App Engine app locally without spending weeks trying to 
> understand what's what first - Back in the day, I think the Launcher GUI and 
> the simplicity of it all, got us all hooked (This is coming from a CLI user)
> 
> 
> 
> Things used to be as simple as "python2.7 /sdk/dev_appserver.py 
> --storage_path=/storage/ --blobstore_path=/blobstore/ 
> --datastore_path=/storage/db.rdbms --host=0.0.0.0 --port=8080 /path 
> --require_indexes" - We could test EVERYTHING locally, now we can almost test 
> nothing locally
> 
> 
> 
> It seems that the python2.7 that's currently in the process of being 
> deprecated was the last of it's kind
> 
> 
> 
> Instead of refactoring everything for Python 3, I decided to move onto NodeJS 
> instead, the work done, the documentation, the depth of it all, on the 
> surface, is very exciting, until you dive in
> 
> 
> 
> First of all, at each step, you're faced with initially arbitrary decisions 
> that's going to eliminate you, flexible or standard? - does it really matter, 
> I don't think so, firestore in datastore mode or firestore? oh my god, it's 
> like each option is designed to torture and drive away a potential user
> 
> 
> 
> As a long time App Engine / Datastore user, I've simplified my data design 

[google-appengine] Migrating Python 2 to 3: images

[Joshua Smith]

I need to read and resize an image in my google app engine app. In my python2.7 
code, this is simply:

from google.appengine.api import images
image = images.resize(image, width=1024, output_encoding=images.JPEG, 
correct_orientation=images.CORRECT_ORIENTATION)

I can’t find anything the migration guide that suggests an alternative to this 
API (but several places say I need to find one). So helpful.

What does this august body recommend I use to replace the above two lines of 
code? I certainly don’t want to have to use some web-based micro-service for 
such a simple thing.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/DA44AE42-4D42-47C1-BE0B-9726516FF865%40gmail.com.

Re: [google-appengine] Can't I set a specific request timeout ? 24 hours for running an HTTP requets seems crazy.

[Joshua Smith]

Relying on the infrastructure to police your algorithm’s implementation doesn’t 
make much sense.

If you’re doing some operation with a loop, you can simply add a check every 
few iterations and abort if it’s taking too long.

Using timers might be an option, depending on the language you are using. But 
being explicit within the algorithm is probably a better engineering practice.

> On Jan 31, 2020, at 3:40 AM, Patrice B  wrote:
> 
> Thank you for your time.   I do understand that task queues and cron jobs may 
> need a longer timeout value. 
> 
> My question really is this : is there no way to configure an instance in such 
> a way that HTTP requests are killed on timeout after say 10 minutes ? 
> 
> And if there is no way to achieve this, as seems to be the case, then how 
> would you suggest to protect an instance against a request that would keep 
> running forever ?Should be start a timer and attempt to handle timeout 
> manually ? 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/cbc75d76-591f-4f58-bff3-42d6fe0b8b4c%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/DAC3FA17-71F1-43F1-94FB-99FF06F54B29%40gmail.com.

Re: [google-appengine] Can't I set a specific request timeout ? 24 hours for running an HTTP requets seems crazy.

[Joshua Smith]

HTTP requests service task queues and cron jobs, so while it would be crazy to 
think a web browser would still be around, it’s perfectly normal to put 
compute-intensive cron jobs into an HTTP request handler that might need many 
hours to run.

> On Jan 30, 2020, at 10:14 AM, Patrice B  wrote:
> 
> Regarding request timeout, I find here 
> https://cloud.google.com/appengine/docs/standard/python/how-instances-are-managed#timeout
>  that a timeout value of 10 minutes is used for "automatic scaling" and a 
> value of 24 hours is used in basic and manual scaling.And indeed, I have 
> observed requests running for hours and hours, with important impact on the 
> instance.
> 
> Is there no way for me to define a specific timeout value other than 10 
> minutes or 24 hours ?24 hours, which applies to HTTP requests as well as 
> task queue tasks,  is a crazy value for web requests, who would wait 24 hours 
> for an HTTP response ?  
> 
> This absence of a reasonable limit has major implications:  while the user 
> probably moved on and left the page after a few minutes, the request may be 
> running for hours, thus slowing the whole instance. 
> 
> It would be very useful to be able to specify one's own timeout value.   And 
> if that is impossible, then 10 minutes for HTTP requests is certainly a 
> better choice than 24 hours. 
> 
> Does anyone else have this kind of issue ? 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/f411f311-5ef0-47d5-ade7-2cb53830224c%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1F102CC1-005D-476D-B427-673EE381554A%40gmail.com.

Re: [google-appengine] Python 3 standard environment data storage

[Joshua Smith]

Answering my own questions, for the record...

> 
> 
> 
> Questions:
> 
> - Does it take a while for the Datastore console notice there’s a datastore 
> on a new project?

Yup. It takes a while but eventually the console starts showing the database.

> - Is there an extra step in the python3.7 environment beyond including an 
> index.yaml file in my deployment?

Yup:

gcloud datastore indexes create index.yaml

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0FE4DC08-CCE1-4E2E-A77C-DABE66EAF5C1%40gmail.com.

Re: [google-appengine] Python 3 standard environment data storage

[Joshua Smith]

I moved my test app to the cloud and I’m able to read and write data using 
Cloud NDB. However, the console still shows my database as empty, and it seems 
to be ignoring my index.yaml file (or maybe the indexes are just taking a 
really long time to build, but I can’t tell because the console says I have no 
indexes).

Questions:

- Does it take a while for the Datastore console notice there’s a datastore on 
a new project?
- Is there an extra step in the python3.7 environment beyond including an 
index.yaml file in my deployment?

Also, I’ve figured out how to use sendgrid instead of the built-in email like I 
had in P2.7, which was surprisingly painless and easy. But is there a 
replacement for this:

from google.appengine.ext import ereporter
ereporter.register_logger()

That thing is invaluable when you have lots of apps. Is there something similar 
for P3.7 that will send me exception summaries daily?

I’m familiar with the “Cloud Console” phone app, which will do notifications. 
But in my experience you can’t rely on that, because sometimes it spontaneously 
stops notifying.

-Joshua

> On Jan 16, 2020, at 11:34 AM, 'Pierre-Yves Blain' via Google App Engine 
>  wrote:
> 
> Firestore in Datastore mode is indeed recommended for databases that will be 
> used primarily by App Engine apps. Firestore in Native mode could also be an 
> interesting option to look into as it represents the next major version of 
> Datastore. You can take a look at the differences here [1] and see what's 
> most applicable for your use case. If you prioritize a feel closer to what 
> you have been using over the additional features provided by Native mode, 
> than Datastore mode may be more appropriate for your use case.
> 
> The Cloud NDB client library is meant to replace App Engine NDB for customers 
> migrating projects/apps to Python 3. As explained in our documentation [2], 
> this library will not support new features of Firestore in Datastore mode. As 
> such, new apps/projects should use the Datastore mode client library [3] 
> instead of Cloud NDB. Reading and writing data from Cloud NDB will allow your 
> Python 2 and Python 3 apps to use the same databases, however the product 
> managing those databases will now be Cloud Firestore in Datastore mode.
> 
> Finally, products in Beta mode have usually spent 6 months in the Alpha phase 
> where the majority of testing was done. Products in Beta are therefore 
> generally more robust, publicly announced, and generally feature complete 
> although this might change based on customer feedback. You can read more 
> about the different phases here [4]. It is worth noting that SLAs do not 
> apply for products in that phase, so you may verify the Firestore SLA [5] and 
> see if you require it for your projects. If required, the Datastore mode 
> Client Libraries [6] may be a better option as it is in GA phase.
> 
> [1] https://cloud.google.com/datastore/docs/firestore-or-datastore 
> <https://cloud.google.com/datastore/docs/firestore-or-datastore> 
> [2] 
> https://cloud.google.com/appengine/docs/standard/python3/migrating-to-cloud-ndb
>  
> <https://cloud.google.com/appengine/docs/standard/python3/migrating-to-cloud-ndb>
>  
> [3] 
> https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-usage-python
>  
> <https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-usage-python>
> [4] https://cloud.google.com/products/#product-launch-stages 
> <https://cloud.google.com/products/#product-launch-stages>
> [5] https://cloud.google.com/firestore/sla 
> <https://cloud.google.com/firestore/sla> 
> [6] 
> https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-install-python
>  
> <https://cloud.google.com/datastore/docs/reference/libraries#client-libraries-install-python>
> 
> On Wednesday, January 15, 2020 at 3:21:00 PM UTC-5, Joshua Smith wrote:
> I’ve been using AppEngine for about a decade, currently my apps are all 
> Python 2.7, Standard environment, google.appengine.ext.db
> 
> I need to create a new app, so I figured it’s a good time to start learning 
> all the technologies that have replaced everything I’ve been using.
> 
> I’m trying to understand what database technology I should use. Since I’m 
> eventually going to have to migrate all my existing apps to Python 3, I’d 
> like to use the database that is closest to what I’ve been using.
> 
> From my reading of the docs, I think I can use “Cloud Firestore in Datastore 
> Mode” and interface to that using “Cloud NDB” which provides an API very 
> similar to “ndb” which is something I’ve never used, but looks darn similar 
> to “db”.
> 
> Does that make sense? I see that there are “this is beta software” warnings 
> around t

Re: [google-appengine] Python 3 standard environment data storage

[Joshua Smith]

Running locally, I’m using this datastore emulator:

https://cloud.google.com/datastore/docs/tools/datastore-emulator 


It works, but it is SOOO SLW. It takes literally 10-20 seconds to respond 
to a simple query.

Is this a known issue? Is there a fix?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/8369E74C-B58D-4A76-9E3A-06FA28A432E7%40gmail.com.

Re: [google-appengine] Python 3 standard environment data storage

[Joshua Smith]

My understanding is that yes, 2.7 will eventually go *poof* like the old pre-HR 
DB implementation went *poof*. However, it’s a long way off. They haven’t 
issued a deprecation warning yet, and as a core technology, google's habit is 
to give you 3 years to move off of it after deprecation.

The launcher, though, is another story. That’s going away this summer, and will 
be sorely missed. I cannot imagine why they think having to type mysterious 
incantations into a terminal is preferable to clicking “run” or “deploy”

I managed to get the new 3.7 app I’m using as a test case running, but it took 
me nearly an hour to figure out all the proper steps. (The step-by-step 
walk-throughs are nice, but they are misleading. For example, if you use a 
database (who doesn’t?) you need to install a bunch of other stuff and do a 
whole bunch of other cryptic things to get a test database running or your 
simple “hello world” will crash with a page of error messages.)

-Joshua

> On Jan 16, 2020, at 2:28 PM, Kaan Soral  wrote:
> 
> I see a lot of questions of this nature lately, good that we're slowly 
> starting to migrate/consider migrating as a community
> 
> A bit hijacking the thread; Do we really need to migrate old apps? I have a 
> lot of apps that I keep alive just to keep them alive, they are keepsakes at 
> this point, I don't want to kill them, their income roughly offsets their 
> costs, so for me, the question is keep or kill, and I'd like to keep
> 
> Is there any point where old technologies / Python 2.7 is going to become 
> unsupported?
> 
> (This Python 2.7 thing is such a bummer too, I certainly wish we could have a 
> Google Python 2.7 fork and keep using it indefinitely :)
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/da5bc76b-c9e9-484d-b32f-4e4a0e9162ec%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/5D39EB09-16C2-4D67-ABE0-71D5ECF20094%40gmail.com.

[google-appengine] Python 3 standard environment data storage

[Joshua Smith]

I’ve been using AppEngine for about a decade, currently my apps are all Python 
2.7, Standard environment, google.appengine.ext.db

I need to create a new app, so I figured it’s a good time to start learning all 
the technologies that have replaced everything I’ve been using.

I’m trying to understand what database technology I should use. Since I’m 
eventually going to have to migrate all my existing apps to Python 3, I’d like 
to use the database that is closest to what I’ve been using.

>From my reading of the docs, I think I can use “Cloud Firestore in Datastore 
>Mode” and interface to that using “Cloud NDB” which provides an API very 
>similar to “ndb” which is something I’ve never used, but looks darn similar to 
>“db”.

Does that make sense? I see that there are “this is beta software” warnings 
around the Cloud NDB part. Should I be worried about that?

Since this is a new project, I don’t have any data migration to worry about. 
But if I choose this path when porting an existing project, is the underlying 
database the same database? Can I be reading and writing the same datastore 
from a Python 2.7 google.appengine.ext.db app and a Python 3.7 
google.cloud.datastore app at the same time?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/D06E859A-8233-4387-840E-FC86FBC7E43C%40gmail.com.

Re: [google-appengine] Design Advice

[Joshua Smith]

This isn’t accessing analytics, it’s accessing Firebase. I just had some 
working code from Analytics that I used as a starting point.

Anyway, I figured it out. The trick is that I needed to create a new service 
account private key over on the Firebase side, download client secrets from 
there, and then use those secrets in the authenticated calls to firebase.

-Joshua

> On Jan 14, 2020, at 1:16 PM, 'Pierre-Yves Blain' via Google App Engine 
>  wrote:
> 
> Is the Analytics API enabled? Also, was the service account email address 
> used to add a user to the analytics account you want to access via the API?
> You can follow this detailed quickstart documentation [1] with explicit steps 
> to accomplish the authorization.
> 
> [1] 
> https://developers.google.com/analytics/devguides/config/mgmt/v3/quickstart/service-py
> 
> On Friday, January 10, 2020 at 4:09:23 PM UTC-5, Joshua Smith wrote:
> And a related issue. I have some code that this app successfully uses to 
> connect to the google analytics API. I tweaked that code for firebase as 
> follows:
> 
>   from oauth2client.service_account import ServiceAccountCredentials
>   from httplib2 import Http
>   credentials = ServiceAccountCredentials.from_json_keyfile_name(
>   CLIENT_SECRETS, 
> scopes=['https://www.googleapis.com/auth/firebase.database', 
> 'https://www.googleapis.com/auth/userinfo.email 
> <https://www.googleapis.com/auth/firebase.database',+'https://www.googleapis.com/auth/userinfo.email>'])
>   http = Http()
>   http_auth = credentials.authorize(http)
> 
> But that gives 401 errors. I would guess that I need to enabled an API on the 
> console, but I can’t figure out which one to enable. I enabled “Firebase 
> Storage” but that didn’t help.
> 
> What API do I need to enable to get that scope authorized?
> 
> -Joshua
> 
>> On Jan 10, 2020, at 3:19 PM, Joshua Smith > <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> 
>> This looks very promising, and I’m playing with it now. Getting the 
>> permissions set up is hard, though.
>> 
>> At first, I simply tried to create a firebase project associated with the 
>> GAE project that’s going to dump data there. I was able to do that, but it 
>> won’t let me create a realtime database because my GAE project has a 
>> database of its own.
>> 
>> So then I tried creating a brand new thing. And that works, except I can’t 
>> figure out how to give my GAE project permission to read/write to it. If I 
>> try to add it’s “@appspot.gserviceaccount.com 
>> <http://appspot.gserviceaccount.com/>” email as a User with Editor 
>> permission, it just gives an error.
>> 
>> Any idea how I can give my GAE project permission to authenticate against a 
>> different Firebase project?
>> 
>> -Joshua
>> 
>>> On Jan 10, 2020, at 9:51 AM, 'Olu' via Google App Engine 
>>> >> <mailto:google-appengine@googlegroups.com>> wrote:
>>> 
>>> I think using Firebase Realtime Database to send real-time updates might be 
>>> applicable in your setup and it works well with App Engine(Standard). I 
>>> suggest you go through this doc[1].
>>> 
>>> [1]https://cloud.google.com/solutions/using-firebase-real-time-events-app-engine
>>>  
>>> <https://cloud.google.com/solutions/using-firebase-real-time-events-app-engine>
>>> 
>>> -- 
>>> You received this message because you are subscribed to the Google Groups 
>>> "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to google-appengine+unsubscr...@googlegroups.com 
>>> <mailto:google-appengine+unsubscr...@googlegroups.com>.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/602eb9d0-d46f-45cd-91dd-a59eddd0ef88%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/google-appengine/602eb9d0-d46f-45cd-91dd-a59eddd0ef88%40googlegroups.com?utm_medium=email_source=footer>.
>> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/936d054a-fb51-4fbf-b379-52692a8c1f8c%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/936d054a-fb51-4fbf-b379-52692a8c1f8c%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6ECCCA7F-D116-4E03-AC1C-585CC3248856%40gmail.com.

Re: [google-appengine] Design Advice

[Joshua Smith]

And a related issue. I have some code that this app successfully uses to 
connect to the google analytics API. I tweaked that code for firebase as 
follows:

  from oauth2client.service_account import ServiceAccountCredentials
  from httplib2 import Http
  credentials = ServiceAccountCredentials.from_json_keyfile_name(
  CLIENT_SECRETS, 
scopes=['https://www.googleapis.com/auth/firebase.database', 
'https://www.googleapis.com/auth/userinfo.email'])
  http = Http()
  http_auth = credentials.authorize(http)

But that gives 401 errors. I would guess that I need to enabled an API on the 
console, but I can’t figure out which one to enable. I enabled “Firebase 
Storage” but that didn’t help.

What API do I need to enable to get that scope authorized?

-Joshua

> On Jan 10, 2020, at 3:19 PM, Joshua Smith  wrote:
> 
> This looks very promising, and I’m playing with it now. Getting the 
> permissions set up is hard, though.
> 
> At first, I simply tried to create a firebase project associated with the GAE 
> project that’s going to dump data there. I was able to do that, but it won’t 
> let me create a realtime database because my GAE project has a database of 
> its own.
> 
> So then I tried creating a brand new thing. And that works, except I can’t 
> figure out how to give my GAE project permission to read/write to it. If I 
> try to add it’s “@appspot.gserviceaccount.com” email as a User with Editor 
> permission, it just gives an error.
> 
> Any idea how I can give my GAE project permission to authenticate against a 
> different Firebase project?
> 
> -Joshua
> 
>> On Jan 10, 2020, at 9:51 AM, 'Olu' via Google App Engine 
>> > <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> I think using Firebase Realtime Database to send real-time updates might be 
>> applicable in your setup and it works well with App Engine(Standard). I 
>> suggest you go through this doc[1].
>> 
>> [1]https://cloud.google.com/solutions/using-firebase-real-time-events-app-engine
>>  
>> <https://cloud.google.com/solutions/using-firebase-real-time-events-app-engine>
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengine+unsubscr...@googlegroups.com 
>> <mailto:google-appengine+unsubscr...@googlegroups.com>.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/602eb9d0-d46f-45cd-91dd-a59eddd0ef88%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/602eb9d0-d46f-45cd-91dd-a59eddd0ef88%40googlegroups.com?utm_medium=email_source=footer>.
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/C8182E97-7F5B-4F5B-92D8-70338A917C86%40gmail.com.

Re: [google-appengine] Design Advice

[Joshua Smith]

This looks very promising, and I’m playing with it now. Getting the permissions 
set up is hard, though.

At first, I simply tried to create a firebase project associated with the GAE 
project that’s going to dump data there. I was able to do that, but it won’t 
let me create a realtime database because my GAE project has a database of its 
own.

So then I tried creating a brand new thing. And that works, except I can’t 
figure out how to give my GAE project permission to read/write to it. If I try 
to add it’s “@appspot.gserviceaccount.com” email as a User with Editor 
permission, it just gives an error.

Any idea how I can give my GAE project permission to authenticate against a 
different Firebase project?

-Joshua

> On Jan 10, 2020, at 9:51 AM, 'Olu' via Google App Engine 
>  wrote:
> 
> I think using Firebase Realtime Database to send real-time updates might be 
> applicable in your setup and it works well with App Engine(Standard). I 
> suggest you go through this doc[1].
> 
> [1]https://cloud.google.com/solutions/using-firebase-real-time-events-app-engine
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/602eb9d0-d46f-45cd-91dd-a59eddd0ef88%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/E571FF44-373A-41A4-B62F-B4069A801024%40gmail.com.

[google-appengine] Design Advice

[Joshua Smith]

I have a big app running on AppEngine standard, python 2.7.

It’s tracking analytics info from dozens of apps in the field, and I’m thinking 
about adding a “real time” feature, similar to what google analytics has. A web 
client would show inbound hits on a map as they occur, for example.

While the incoming data has some scale (not massive, typically about 4 
instances), there will never be a lot of people viewing the output. 2 or 3 at a 
time, at most. Usually 0.

I’m wondering whether folks in this group have advice on the best approach to 
send a real-time data flow into the web pages.

- WebSockets seems logical, but it looks like GAE standard doesn’t support them
- I could run a websockets server somewhere else (EC2?) and somehow (?) send 
messages from GAE to that for distribution
- I looked at gcloud PubSub, but that seems to be more of a server->server tool 
than a way to send messages to web apps
- I looked at gcloud message, but that seems ill suited to use in web pages
- Polling sounds expensive, since I’d need to create lots and lots of temporary 
database items for the messages

Is there some other tool or technique out there that I should investigate?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/1BD53305-A562-44EC-92E6-8C5331FEF958%40gmail.com.

Re: [google-appengine] Controlling costs

[Joshua Smith]

So the theory is that if you set max_idle_instances to 0 it gets ignored, but 
at 1 it is obeyed?

That’s plausible.

I wish google would document this.

Anyway, I’ll try this and report back…

-Joshua

> On Dec 30, 2019, at 4:41 PM, Vitaly Bogomolov  wrote:
> 
> my advice is the same.
> 
> automatic_scaling:
>   max_idle_instances: 1
>   min_idle_instances: 1
>   max_concurrent_requests: 20

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/002452C3-D7CE-4C50-A9FF-81DDCA586332%40gmail.com.

Re: [google-appengine] Controlling costs

[Joshua Smith]

Thank you both!

I love when the documentation is wrong :)

I’m trying:

automatic_scaling:
  max_idle_instances: 0
  max_pending_latency: 200ms
  max_concurrent_requests: 20

So far, the site is still quite responsive, and there are not a bunch of idle 
instances lying around. So yay!

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6FDBAFF6-EE7F-4AEA-8708-DBDCF060B4DE%40gmail.com.

Re: [google-appengine] gcloud app deploy is failing

[Joshua Smith]

Confirmed that my deploy is now working. A little heads-up would be nice before 
you take another shot at that experiment. I don’t remember opting-in to being a 
guinea pig.

And yes, I assumed it had something to do with that cloud build stuff. I guess 
I understand why you are doing it, but this transformation of Google App Engine 
from a PaaS offering to an IaaS offering is horrible. It used to be we had a 
simple Launcher app that made development and deployment of an app super easy. 
Now I’m dealing with a half dozen “services,” trying to ensure all the service 
accounts have the right permissions, and having to learn entire stacks to do 
something that used to be an API call.

There is a real market need for a PaaS offering, which is what GAE used to be. 
Now it’s gotten so complicated that I might as well be spinning up a container 
in EC2. You have completely destroyed your competitive advantage, and there is 
vacuum that someone (beanstalk maybe?) is going to step into.

It reminds me of the the way Microsoft Access solved a real problem for 
millions of users, yet Microsoft repeatedly tried to kill the thing to drag 
people to SQL Server, which was such overkill for 99% of those users.

Once again, Google is listening to their engineers instead of their customers. 
It’s a shame. GAE was great. This gcloud mess is the opposite of great.

-Joshua

> On Dec 13, 2019, at 8:46 AM, 'joshuamo' via Google App Engine 
>  wrote:
> 
> App Engine will be moving towards using Cloud Build for the Standard 
> environment[1], matching the other Serverless offerings.  Good catch noticing 
> the Cloud Build aspect here.  We were rolling this out at a low percentage 
> and this is what seems to have triggered the issue you've experienced.
> 
> For now, we've rolled back the experiment, so you should be able to deploy 
> again without issue.
> 
> [1] https://cloud.google.com/appengine/docs/standard/payment-instrument
> 
> On Thursday, December 12, 2019 at 7:54:37 AM UTC-6, Joshua Smith wrote:
> So this problem persists, which I guess means it’s not an outage, but rather 
> something got messed up by google on my account.
> 
> Please advise on how I can get this issue rectified. To summarize, I’m 
> running gcloud app deploy and it is reporting a 404 error trying to read the 
> manifest of the cloud bucket where it is staging things.
> 
> I looked at the issue tracker, but that doesn’t appear to be geared toward 
> production problems any more.
> 
> Opening a ticket appears to have a $100/month price tag attached to it, which 
> seems onerous considering how much my company is paying google every month 
> for these services, and considering google broke my account.
> 
> What’s the right course of action here?
> 
> -Joshua
> 
>> On Dec 11, 2019, at 4:28 PM, Joshua Smith > <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> 
>> Digging into the GAE “Activity” logs, I see some new stuff about google 
>> cloud builder that I haven’t seen before.
>> 
>> Perhaps something went sideways in the transition to that new technology?
>> 
>> All these extra service accounts and whatnot that got created and show up in 
>> IAM are ridiculously complicated. I wouldn’t know where to begin diagnosing 
>> that.
>> 
>> What’s the right way to get someone from google to figure out how they 
>> messed up my account?
>> 
>> -Joshua
>> 
>>> On Dec 11, 2019, at 3:33 PM, Joshua Smith >> <mailto:mrjoshuaesm...@gmail.com>> wrote:
>>> 
>>> I’m trying to update a website deployed in appengine. I’ve made sure my 
>>> cloud SDK is all up to date. It’s a big site, with about 8000 files. I’ve 
>>> been using the same deploy command in this same folder for years. I’m 
>>> getting this cryptic error:
>>> 
>>> Updating service [default]...failed.
>>>
>>> ERROR: (gcloud.app.deploy) Error Response: [5] failed to fetch metadata 
>>> from GCR, with reason: generic::not_found: failed to fetch metadata from 
>>> GCR for image 
>>> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>>>  
>>> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>>>  with reason: generic::not_found: fetchImageMetadata failed for image 
>>> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>>>  
>>> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>>>  reason: generic::not_found: failed to fetch manifest from GCR (via 
>>> gcr.FetchManifest): generic::not

[google-appengine] gcloud app deploy is failing

[Joshua Smith]

So this problem persists, which I guess means it’s not an outage, but rather 
something got messed up by google on my account.

Please advise on how I can get this issue rectified. To summarize, I’m running 
gcloud app deploy and it is reporting a 404 error trying to read the manifest 
of the cloud bucket where it is staging things.

I looked at the issue tracker, but that doesn’t appear to be geared toward 
production problems any more.

Opening a ticket appears to have a $100/month price tag attached to it, which 
seems onerous considering how much my company is paying google every month for 
these services, and considering google broke my account.

What’s the right course of action here?

-Joshua

> On Dec 11, 2019, at 4:28 PM, Joshua Smith  wrote:
> 
> Digging into the GAE “Activity” logs, I see some new stuff about google cloud 
> builder that I haven’t seen before.
> 
> Perhaps something went sideways in the transition to that new technology?
> 
> All these extra service accounts and whatnot that got created and show up in 
> IAM are ridiculously complicated. I wouldn’t know where to begin diagnosing 
> that.
> 
> What’s the right way to get someone from google to figure out how they messed 
> up my account?
> 
> -Joshua
> 
>> On Dec 11, 2019, at 3:33 PM, Joshua Smith > <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> 
>> I’m trying to update a website deployed in appengine. I’ve made sure my 
>> cloud SDK is all up to date. It’s a big site, with about 8000 files. I’ve 
>> been using the same deploy command in this same folder for years. I’m 
>> getting this cryptic error:
>> 
>> Updating service [default]...failed. 
>>   
>> ERROR: (gcloud.app.deploy) Error Response: [5] failed to fetch metadata from 
>> GCR, with reason: generic::not_found: failed to fetch metadata from GCR for 
>> image 
>> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>>  
>> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>>  with reason: generic::not_found: fetchImageMetadata failed for image 
>> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>>  
>> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>>  reason: generic::not_found: failed to fetch manifest from GCR (via 
>> gcr.FetchManifest): generic::not_found: error fetching 
>> "kaoncom-hr/app-engine-tmp/app/ttl-2h/manifests/8927ab89-5aed-4a70-9969-9ed1927844ec"
>>  : generic::not_found: got HTTP/404 response, wanted HTTP/200
>> 
>> The guid-looking thing changes every deploy. kaoncom-hr is the name of the 
>> app.
>> 
>> I checked the dashboards and google is not reporting an outages.
>> 
>> Any ideas?
>> 
>> -Joshua
>> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/934C1882-9618-4E1C-B6E8-3562240A94E2%40gmail.com.

[google-appengine] Re: Outage?

[Joshua Smith]

Digging into the GAE “Activity” logs, I see some new stuff about google cloud 
builder that I haven’t seen before.

Perhaps something went sideways in the transition to that new technology?

All these extra service accounts and whatnot that got created and show up in 
IAM are ridiculously complicated. I wouldn’t know where to begin diagnosing 
that.

What’s the right way to get someone from google to figure out how they messed 
up my account?

-Joshua

> On Dec 11, 2019, at 3:33 PM, Joshua Smith  wrote:
> 
> I’m trying to update a website deployed in appengine. I’ve made sure my cloud 
> SDK is all up to date. It’s a big site, with about 8000 files. I’ve been 
> using the same deploy command in this same folder for years. I’m getting this 
> cryptic error:
> 
> Updating service [default]...failed.  
>  
> ERROR: (gcloud.app.deploy) Error Response: [5] failed to fetch metadata from 
> GCR, with reason: generic::not_found: failed to fetch metadata from GCR for 
> image 
> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>  
> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>  with reason: generic::not_found: fetchImageMetadata failed for image 
> us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec
>  
> <http://us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec>,
>  reason: generic::not_found: failed to fetch manifest from GCR (via 
> gcr.FetchManifest): generic::not_found: error fetching 
> "kaoncom-hr/app-engine-tmp/app/ttl-2h/manifests/8927ab89-5aed-4a70-9969-9ed1927844ec"
>  : generic::not_found: got HTTP/404 response, wanted HTTP/200
> 
> The guid-looking thing changes every deploy. kaoncom-hr is the name of the 
> app.
> 
> I checked the dashboards and google is not reporting an outages.
> 
> Any ideas?
> 
> -Joshua
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/C6668FFB-C7C2-4E61-9072-58D1FE568F28%40gmail.com.

[google-appengine] Outage?

[Joshua Smith]

I’m trying to update a website deployed in appengine. I’ve made sure my cloud 
SDK is all up to date. It’s a big site, with about 8000 files. I’ve been using 
the same deploy command in this same folder for years. I’m getting this cryptic 
error:

Updating service [default]...failed.
   
ERROR: (gcloud.app.deploy) Error Response: [5] failed to fetch metadata from 
GCR, with reason: generic::not_found: failed to fetch metadata from GCR for 
image 
us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec,
 with reason: generic::not_found: fetchImageMetadata failed for image 
us.gcr.io/kaoncom-hr/app-engine-tmp/app/ttl-2h:8927ab89-5aed-4a70-9969-9ed1927844ec,
 reason: generic::not_found: failed to fetch manifest from GCR (via 
gcr.FetchManifest): generic::not_found: error fetching 
"kaoncom-hr/app-engine-tmp/app/ttl-2h/manifests/8927ab89-5aed-4a70-9969-9ed1927844ec"
 : generic::not_found: got HTTP/404 response, wanted HTTP/200

The guid-looking thing changes every deploy. kaoncom-hr is the name of the app.

I checked the dashboards and google is not reporting an outages.

Any ideas?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/618664D9-08A8-4321-ACE3-D68620905166%40gmail.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Joshua Smith]

Google Cloud Storage is like S3 with cloudfront built-in.

You could absolutely migrate that database to GCS and then access those objects 
via https or via API.

> On Aug 22, 2019, at 1:17 PM, ALT-EMAIL Virilo Tejedor 
>  wrote:
> 
> thanks Joshua... has GAE something similar to cloudfront?
> 
> I could move the S3 bucket to GAE and use the GAE's cloudfront?
> 
> I've no credits for AWS
> 
> On Thursday, August 22, 2019 at 7:12:34 PM UTC+2, Joshua Smith wrote:
> GAE isn’t going to serve those images any faster than S3 does (unless they’re 
> in glacier storage). 
> 
> What you propose won’t be possible because of limits on the size of GAE apps 
> (10K files, max file sizes, etc.) 
> 
> If you are reading the images more than once, putting a cloudfront 
> distribution between your app and the S3 bucket might help. 
> 
> On Aug 22, 2019, at 8:53 AM, ALT-EMAIL Virilo Tejedor  <>> wrote: 
> > 
> > Hi all, 
> > 
> > I'd like to create a static web server to store almost 1 TB of images. 
> > 
> > It is an opensource dataset that I'd like to use to train a Deep Learning 
> > model. 
> > 
> > I have free usage of GPUs and Internet conexion in another plattform, but 
> > they don't provide me 1 TB storage. 
> > 
> > I've also 600$ credits in Google Cloud, I was wondering if there was an 
> > easy way to create something to feed with images the server in the other 
> > plattform. 
> > 
> > The datasource is available as an AWS bucket.  I tried to connect the GPU 
> > machine directly to the ASW bucket via awscli, but it is too much slow.  
> > Like if the bucket were thought for a complete sync but not for coninuous 
> > random access to the files. 
> > 
> > I've though two possible approaches: 
> > 
> > - Execute a python script in GAE to download the dataset and to 
> > create a GAE web server: 
> > https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
> >  
> > <https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website>
> >  
> > 
> > - Execute a python script in GAE to download the dataset and to 
> > create a Google Cloud CDN. 
> > 
> > Do you think any of this approaches are valid to feed the model during the 
> > training? 
> > 
> > I'm a newbie in GAE and any help, starting point or idea will be very 
> > wellcomed. 
> > 
> > Thanks in advance 
> > 
> > -- 
> > You received this message because you are subscribed to the Google Groups 
> > "Google App Engine" group. 
> > To unsubscribe from this group and stop receiving emails from it, send an 
> > email to google-a...@googlegroups.com <>. 
> > To view this discussion on the web visit 
> > https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com
> >  
> > <https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com>.
> >  
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/6dc6f001-f27e-4206-96e0-dca3d1f3fbee%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/6dc6f001-f27e-4206-96e0-dca3d1f3fbee%40googlegroups.com?utm_medium=email_source=footer>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/AEB097FC-DF59-4D68-BEFE-89578069F0FF%40gmail.com.

Re: [google-appengine] Using GAE to store images for other machine out of GAE

[Joshua Smith]

GAE isn’t going to serve those images any faster than S3 does (unless they’re 
in glacier storage).

What you propose won’t be possible because of limits on the size of GAE apps 
(10K files, max file sizes, etc.)

If you are reading the images more than once, putting a cloudfront distribution 
between your app and the S3 bucket might help.

On Aug 22, 2019, at 8:53 AM, ALT-EMAIL Virilo Tejedor  
wrote:
> 
> Hi all,
> 
> I'd like to create a static web server to store almost 1 TB of images.
> 
> It is an opensource dataset that I'd like to use to train a Deep Learning 
> model.
> 
> I have free usage of GPUs and Internet conexion in another plattform, but 
> they don't provide me 1 TB storage.
> 
> I've also 600$ credits in Google Cloud, I was wondering if there was an easy 
> way to create something to feed with images the server in the other plattform.
> 
> The datasource is available as an AWS bucket.  I tried to connect the GPU 
> machine directly to the ASW bucket via awscli, but it is too much slow.  Like 
> if the bucket were thought for a complete sync but not for coninuous random 
> access to the files.
> 
> I've though two possible approaches:
> 
>   - Execute a python script in GAE to download the dataset and to create 
> a GAE web server: 
> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
> 
>   - Execute a python script in GAE to download the dataset and to create 
> a Google Cloud CDN.
> 
> Do you think any of this approaches are valid to feed the model during the 
> training?
> 
> I'm a newbie in GAE and any help, starting point or idea will be very 
> wellcomed.
> 
> Thanks in advance
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/dbd0a8f8-859b-4f50-a108-80b21e27267f%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/FAE6108D-A359-414F-BF7E-0C2E3DE81EF8%40gmail.com.

Re: [google-appengine] Source Code download from GAE Flex

[Joshua Smith]

The cloud SDK seems to do a compare of local files to those in cloud storage 
when you do a deploy. Would it be possible for Sai to access that bucket 
directly and download the objects back to his machine? The file names might be 
garbage, but it’d be relatively easy to fix that manually.

-Joshua

> On Aug 20, 2019, at 1:16 PM, 'Nicolas (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> Hi Sai,
> 
> It used to be possible to download your source code with the appcfg.py tools 
> but unfortunately this tool is now deprecated and it’s replacement, the Cloud 
> SDK   does not support this feature.
> 
> There is a feature request for this, and you can follow their progress here 
> , although there is no ETA for 
> when it will be implemented.
> 
> On Monday, August 19, 2019 at 6:26:58 PM UTC-4, Sai M wrote:
> Hello,
> 
> By accident i have lost the data on my personal laptop where i have kept the 
> Source code of the project i have deployed on Google App Flex.  Unfortunately 
> i don't have the backup as it happened by accident when i was trying to 
> upgrade my OS.
> 
> The Project ID is..."smartcc".
> 
> Can you pls. help on instructions of how to down load the latest application 
> source code deployed on Google App Flex env please.
> 
> Thanks a ton in advance.
> 
> Best Regards
> Sai
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/e52822af-78fd-45dc-97d1-4a72915fe6ef%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/891ECA4A-88C5-4A59-B131-55872CA05929%40gmail.com.

Re: [google-appengine] Header names after september 30 (Load balancers)

[Joshua Smith]

Wait. What?

Are you saying this bit of server code isn’t going to work after September 30?

class CountryHandler(webapp2.RequestHandler):
  def get(self):
self.response.headers["Content-Type"] = "text/plain"
self.response.out.write(self.request.headers["X-AppEngine-Country"])

Or is the issue only on the client side?

Also, I didn’t get that email. I get emails all the time from google about 
stuff they are deprecating (the most recent being a stern warning that I need 
to stop using the lovely “Launcher” app and instead use the terrible 
command-line interface; a development about which I am not at all pleased). Why 
wouldn’t I have gotten this one?

-Joshua

> On Aug 15, 2019, at 4:31 AM, Carl Smith  wrote:
> 
> You can test to see how your application handles lowercase by passing this 
> header through also.
> 
> x-goog-downcase-all-headers: test
> 
> This will make all of the headers lowercase
> 
> On Tuesday, August 13, 2019 at 10:30:50 PM UTC+1, Santiago Del Valle wrote:
> We got this email
> 
> After September 30, HTTP(S) Load Balancers will convert HTTP/1.1 header names 
> to lowercase in the request and response directions; header values will not 
> be affected.
> As header names are case-insensitive, this change will not affect clients and 
> servers that follow the HTTP/1.1 specification (including all popular web 
> browsers and open source servers). Similarly, as HTTP/2 and QUIC protocols 
> already require lowercase header names, traffic arriving at load balancers 
> over these protocols will not be affected. However, we recommend testing 
> projects that use custom clients or servers prior to the rollout to ensure 
> minimal impact.
> 
> However I do not understand if this will affect headers injected by google, 
> like "HTTP_X_APPENGINE_COUNTRY" which we use on our project, is this going to 
> change or will it remain unaffected?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/c498388c-6784-4b83-ab0e-e4c0305fc553%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/A8212D2D-8807-474D-93EB-FA5B2A965815%40gmail.com.

Re: [google-appengine] we need to disable weak ciphers on our GAE standard Java 8 for PCI compliance

[Joshua Smith]

Maybe if you front your site with CloudFlare, they can do something like that?

> On Jun 25, 2019, at 1:59 PM, 'Nicolas (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> Hi Robert,
>  
> Currently it is not possible to disable this cipher suite for a custom domain 
> however a feature request  
> have already been open for the implementation of this.
>  
> That being said, it is possible to file a ticket  
> 
>  with GCP support and ask them to do this for your domain.
> 
> 
> Please keep in mind there are no ETAs or timeline for a resolution or the 
> implementation of the feature request.
> 
> 
> On Tuesday, June 25, 2019 at 1:37:06 PM UTC-4, Robert Dyas wrote:
> bump - any idea here?
> 
> On Monday, June 24, 2019 at 5:08:34 PM UTC-4, Robert Dyas wrote:
> We are using a custom domain... PCI security check flagged this issue. We 
> need to disable TLS1.0, DES, 3DES, SSLv3, SSLv3.
> How do we do this?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/d2202e0a-e922-401b-91fb-4badf32dffb7%40googlegroups.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/32893B73-FC6A-4B83-96F9-0EEB5C9162C3%40gmail.com.

Re: [google-appengine] Re: Getting GAE to use GZIP encoding in responses

[Joshua Smith]

I didn’t get an answer in either place. But my experience has been that this 
list tends to produce answers whereas SO is just asking into the void.

> On Apr 4, 2019, at 5:42 PM, 'Nicolas (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> 
> Hi Joshua, 
> 
> Thank you for posting here however I can see on your StackOverflow thread 
> that the issue is resolved for you but would like to know the root cause of 
> your issue. 
> 
> As this seems to be a bit more of a technical question you will probably have 
> better answers from the community by posting on StackOverflow as Google 
> Groups is intended for general discussion.
> 
> 
>> On Thursday, March 14, 2019 at 9:05:48 AM UTC-4, Joshua Smith wrote:
>> Is there a trick to convince GAE to GZIP responses? The docs say it’s 
>> supposed to be automatic, but it isn’t happening. Details in this SO 
>> question…
>> 
>> https://stackoverflow.com/questions/55145439/how-can-i-serve-gzip-encoded-images-from-google-app-engine
>> 
>> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/911a8289-5997-4e48-824b-8f2ea5264fb2%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/DF8EF245-23E6-4E02-954A-1A5A72BF6E15%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Getting GAE to use GZIP encoding in responses

[Joshua Smith]

Is there a trick to convince GAE to GZIP responses? The docs say it’s supposed 
to be automatic, but it isn’t happening. Details in this SO question…

https://stackoverflow.com/questions/55145439/how-can-i-serve-gzip-encoded-images-from-google-app-engine
 



-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/85167501-F688-4D8C-A95F-051153F66D35%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Domain Puzzle

[Joshua Smith]

I’m not going to pretend I have any idea how that spooky magic worked, but you 
solved my problem!

I’ll buy you a beer next time I’m near your googleplex.

-Joshua

> On Jan 14, 2019, at 4:56 PM, 'Tiago (Google Cloud Platform Support)' via 
> Google App Engine  wrote:
> 
> Hello Joshua,
> 
> The error message you provided indicates that the domain is currently in use 
> by some other project. You can add custom domains 
> <https://cloud.google.com/appengine/docs/standard/python/mapping-custom-domains#adding_a_custom_domain_for_your_application>
>  using the Cloud Console, the Cloud SDK or through the App Engine APIs. In 
> your particular case, the best way to re-add the domain to this particular 
> project is to use the App Engine APIs since it gives you the additional 
> option to override any existing mappings for the domain in question 
> (assuming, of course, that the domain ownership has already been verified via 
> the Webmaster Central portal 
> <https://www.google.com/webmasters/verification/verification?domain=mytowngovernment.org>).
> 
> Here’s a brief set of instructions on how to use the API: 
> 1. Visit the API Explorer 
> <https://developers.google.com/apis-explorer/#search/admin%20api/appengine/v1/appengine.apps.domainMappings.create>
>  and authorize OAuth requests to be performed for your account
> 2. Enter your App ID under appsId, set overrideStrategy to OVERRIDE, and 
> provide the domain in the body request as an “id” property. 
> 
> You can enable SSL support afterwards in the Cloud Console by going to App 
> Engine > Settings > Custom Domains, selecting the domain in question and 
> clicking on “Enable managed security”.
> 
> As for the G Suite mapping, in order to remove a primary domain in the Admin 
> Console, a second one has to be added and verified first. Then you can make 
> the newly added domain your secondary, and later remove the original domain. 
> This operation can cause issues with some G Suite services though, so I would 
> suggest going for the API and Cloud Shell routes first. 
> 
> 
> 
> 
> On Tuesday, January 8, 2019 at 10:38:11 AM UTC-5, Joshua Smith wrote:
> I have a site that I set up ages ago, back when you had to have gsuite to map 
> domains to google app engine.
> 
> On the GSuite console, it redirects the naked domain to www, as one did back 
> then. The redirect monkey is stupid, though, in that it discards the path 
> part of the URL.
> 
> Now I’d like to support the naked domain directly on GAE, but it won’t let me 
> because it says that the naked domain is already being used.
> 
> I can’t remove the mapping in gsuite, because there’s no button to remove it. 
> Only a button to change it (and there’s no “delete” option there).
> 
> Helpful screenshots attached.
> 
> Has anybody who’s been around forever like me managed to make the transition 
> to modern naked domains successfully?
> 
> -Joshua
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To post to this group, send email to google-appengine@googlegroups.com 
> <mailto:google-appengine@googlegroups.com>.
> Visit this group at https://groups.google.com/group/google-appengine 
> <https://groups.google.com/group/google-appengine>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/d6900f1f-88e5-4008-8f1f-62c55502f51c%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/d6900f1f-88e5-4008-8f1f-62c55502f51c%40googlegroups.com?utm_medium=email_source=footer>.
> For more options, visit https://groups.google.com/d/optout 
> <https://groups.google.com/d/optout>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/912DE869-E37D-41C0-BAA7-18C1FE29B971%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] GAE python 2.7 end of life

[Joshua Smith]

SendGrid has a daily limit of 100 emails in the free tier. My app predates the 
severe GAE mail limits, and thus can send more than that every day without 
incurring charges.

So if you guys drop mail in Python 3, you’re going to add $15/month in cost to 
my application. My app is a free service provided to support open government 
and accountability, which means that cost is going to come out of my pocket.

Please reconsider.

-Joshua

> On Jan 11, 2019, at 9:25 AM, 'George (Cloud Platform Support)' via Google App 
> Engine  wrote:
> 
> Hi Vitaly, 
> 
> In addition to sending email, SendGrid can receive email or make sense of the 
> email you’ve already sent using webhooks. You may check related detail on the 
> "Sending Emails with SendGrid" documentation page 
> .
>  I am sure that most liked GAE services will be made available in a form or 
> another with Python 3. 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/7d78b4e8-556a-48d8-a819-5897f0bca883%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6C1CE5A1-C187-4C49-A04A-BA43DD7A3E6A%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Socket Timeout errors on URLFetch calls

[Joshua Smith]

URLFetch is a little flaky in app engine. Always has been, kind of like mail. I 
suspect rolling your own fetch using URL connections wouldn’t help at all.

Over the many years I’ve been using GAE, I’ve noticed that both mail and url 
fetch errors tend to cluster. Like storms that pass through, causing lots of 
errors, and then move on and everything is good for a few weeks.

-Joshua

> On Mar 29, 2018, at 11:59 AM, Hugo Visser  wrote:
> 
> My app makes heavy use of URLFetch to get content and I'm seeing socket time 
> out exceptions on App Engine standard (Java). Am I the only one experiencing 
> this? Would ditching URLFetch for normal url connections fix this?
> 
> Thanks,
> 
> Hugo
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/7c5a6697-9a73-4a83-8ce6-8b5953a5b69a%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7C664552-EEA8-4E88-8333-859D3CC6EB73%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: GAE / Datastore issues right now?

[Joshua Smith]

Wouldn’t it be cool if the status page reflected it, or they sent a 
notification like they are supposed to?

> On Feb 15, 2018, at 3:21 PM, 'Yannick (Cloud Platform Support)' via Google 
> App Engine  wrote:
> 
> Cloud Platform support is aware of this issue, but I do not have any 
> information to communicate at this time.
> 
> On Thursday, February 15, 2018 at 3:07:48 PM UTC-5, Thanasis Delenikas wrote:
> Is anybody else experiencing issues with GAE (Java) in the last 10 minutes or 
> so?
> 
> I see a ridiculous number of ConcurrentModificationException  - my app is 
> almost dead, nothing moves. All datastore ops seem to throw exceptions.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/361012f1-abff-461e-b36c-497ce89b9efe%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/B168F9FE-0B85-48AE-9284-F4A61214092C%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Change to the way GAE counts MB?

[Joshua Smith]

MB is short for megabyte, which is defined as 1,232,896 bytes, not 1,000,000 
bytes.

Also, as I said, this file was fine until yesterday. Google arbitrarily lowered 
the limit and did not update the documentation or even bother to tell anyone.



> On Jan 25, 2018, at 11:57 PM, 'Katayoon (Cloud Platform Support)' via Google 
> App Engine  wrote:
> 
> Based on App Engine Quotas documentation 
> , no single static data file 
> may be larger than 32MB. So, Instead of 32x1024x1024, it's just 3200 
> bytes, which is less than your file size.
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/0c4d2af2-9dda-4b50-908c-fa245657fb54%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/F8AFE82F-53D5-4301-BB7F-D714B30D8183%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Change to the way GAE counts MB?

[Joshua Smith]

I use GAE to host a PHP site, and there’s a file that’s been there about a year 
with no issues. It’s 32674472 bytes.

The file size limit, of course, is 32MB which is higher than that, assuming GAE 
defines a 1 MB = 1024 * 1024.

But I’m getting this error on deployment:

Details: [
  [
{
  "@type": "type.googleapis.com/google.rpc.ResourceInfo",
  "description": "File 
https://storage.googleapis.com/staging.kaoncom-hr.appspot.com/1ced1565690281184e66ebd4fa0a666264a28249
 is too big (32674472 bytes). Max file size is 3200 bytes.",
  "resourceName": 
"https://storage.googleapis.com/staging.kaoncom-hr.appspot.com/1ced1565690281184e66ebd4fa0a666264a28249;,
  "resourceType": "file"
}
  ]
]

Looks like google has changed to counting MB the way disk drive people count 
them.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/6936AD46-81F4-4548-8EAB-5C574DCA3BD5%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Logs Viewer is Down

[Joshua Smith]

I’m not able to see any logs on any of my GAE projects, in Safari or Chrome.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/91CD55AD-9C3C-4281-B096-45CB5529B0CE%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Deployment Errors

[Joshua Smith]

I created an issue. I hope this is the right place for production issues. It 
looks very different from where I used to post production issues:

https://issuetracker.google.com/issues/68317095

> On Oct 26, 2017, at 11:33 AM, Joshua Smith <mrjoshuaesm...@gmail.com> wrote:
> 
> When I look at google storage, the file that it’s complaining it cannot see 
> is in fact not there.
> 
> So I guess the problem is that deploy thinks the file is there, and doesn’t 
> push it, but it’s not there for some reason and so when it tries to move 
> everything in the manifest from staging to live, it is missing one or more 
> files.
> 
> Seems like what I need is a “force” option to get “gcloud app deploy” to 
> upload files that it doesn’t think it needs to upload. But I don’t see 
> anything like that in the docs.
> 
> From the logs…
> 
> 2017-10-26 11:15:08,308 DEBUGrootSkipping upload of 
> [catalog/tours/core/pdfjs/web/images/secondarytoolbarbutton-rotate...@2x.png]
> 
> 'catalog/tours/core/pdfjs/web/images/secondarytoolbarbutton-rotate...@2x.png’:
>  {'sourceUrl': 
> 'https://storage.googleapis.com/staging.kaoncom-hr.appspot.com/84f034b57d24f7371d1a605bd12afa1f71b50f1b',
>  'sha1Sum': '84f034b57d24f7371d1a605bd12afa1f71b50f1b 
> <https://storage.googleapis.com/staging.kaoncom-hr.appspot.com/84f034b57d24f7371d1a605bd12afa1f71b50f1b',%20'sha1Sum':%20'84f034b57d24f7371d1a605bd12afa1f71b50f1b>'},
> 
> But
> 
> 
> So the question is, how can I convince gcloud app deploy to not skip files?
> 
> -Joshua
> 
>> On Oct 26, 2017, at 10:40 AM, Joshua Smith <mrjoshuaesm...@gmail.com 
>> <mailto:mrjoshuaesm...@gmail.com>> wrote:
>> 
>> Google’s dashboard shows everything green, but I’m getting deployment 
>> errors. Anyone else?
>> 
>> Updating service [default]...failed. 
>>   
>> ERROR: (gcloud.app.deploy) Error Response: [3] Errors were encountered while 
>> copying files to App Engine.
>> 
>> Details: [
>>   [
>> {
>>   "@type": "type.googleapis.com/google.rpc.ResourceInfo 
>> <http://type.googleapis.com/google.rpc.ResourceInfo>",
>>   "description": "Error reading from Google Cloud Storage.",
>>   "resourceName": "https://storage.googleapis.com/ 
>> <https://storage.googleapis.com/>{REDACTED}",
>>   "resourceType": "file"
>> }
>>   ]
>> ]
>> 
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/756E2E2F-31DC-41D0-88D9-277B7D663D8C%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Deployment Errors

[Joshua Smith]

When I look at google storage, the file that it’s complaining it cannot see is 
in fact not there.

So I guess the problem is that deploy thinks the file is there, and doesn’t 
push it, but it’s not there for some reason and so when it tries to move 
everything in the manifest from staging to live, it is missing one or more 
files.

Seems like what I need is a “force” option to get “gcloud app deploy” to upload 
files that it doesn’t think it needs to upload. But I don’t see anything like 
that in the docs.

>From the logs…

2017-10-26 11:15:08,308 DEBUGrootSkipping upload of 
[catalog/tours/core/pdfjs/web/images/secondarytoolbarbutton-rotate...@2x.png]

'catalog/tours/core/pdfjs/web/images/secondarytoolbarbutton-rotate...@2x.png’:
 {'sourceUrl': 
'https://storage.googleapis.com/staging.kaoncom-hr.appspot.com/84f034b57d24f7371d1a605bd12afa1f71b50f1b',
 'sha1Sum': '84f034b57d24f7371d1a605bd12afa1f71b50f1b'},

But


So the question is, how can I convince gcloud app deploy to not skip files?

-Joshua

> On Oct 26, 2017, at 10:40 AM, Joshua Smith <mrjoshuaesm...@gmail.com> wrote:
> 
> Google’s dashboard shows everything green, but I’m getting deployment errors. 
> Anyone else?
> 
> Updating service [default]...failed.  
>  
> ERROR: (gcloud.app.deploy) Error Response: [3] Errors were encountered while 
> copying files to App Engine.
> 
> Details: [
>   [
> {
>   "@type": "type.googleapis.com/google.rpc.ResourceInfo 
> <http://type.googleapis.com/google.rpc.ResourceInfo>",
>   "description": "Error reading from Google Cloud Storage.",
>   "resourceName": "https://storage.googleapis.com/ 
> <https://storage.googleapis.com/>{REDACTED}",
>   "resourceType": "file"
> }
>   ]
> ]
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/E0F744FC-B832-410C-BEC1-A5574D384C71%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Deployment Errors

[Joshua Smith]

Google’s dashboard shows everything green, but I’m getting deployment errors. 
Anyone else?

Updating service [default]...failed.

ERROR: (gcloud.app.deploy) Error Response: [3] Errors were encountered while 
copying files to App Engine.

Details: [
  [
{
  "@type": "type.googleapis.com/google.rpc.ResourceInfo",
  "description": "Error reading from Google Cloud Storage.",
  "resourceName": "https://storage.googleapis.com/ 
{REDACTED}",
  "resourceType": "file"
}
  ]
]

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/A685722A-8DF7-48CD-9CE1-F3BC8FD745B6%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Mail API Quotas

[Joshua Smith]

Turning off the limit, and changing the pricing for usage that they are already 
tracking could not possibly be a “massive waste of engineering.” It should be 
trivial. Maybe 10 minutes was an exaggeration. 15 minutes?

And I would spend a penny per email when I need to send 1,000. Thankfully, I’ve 
been here long enough that I’m grandfathered in with a much higher quota.

Also, it isn’t always trivial to send through those other services when there 
are attachments or other complexities involved.

-Joshua

> On Mar 31, 2017, at 10:15 AM, Jeff Schnitzer <j...@infohazard.org> wrote:
> 
> Nobody who sends email at volume will spend a penny per email. For 1MM 
> messages per month that would be $10k. At Sendgrid that’s $500.
> 
> It’s trivial to enqueue a task that sends an email to any of the hundred 
> other commodity email services on the internet, most of which give you 
> deliverability reporting and fancy graphs and custom IP addresses and 
> whatnot. They run on razor-thin margins and more companies are getting out of 
> the business than into it (eg Mandrill, Messagebus). It would be a massive 
> waste of engineering.
> 
> Jeff
> 
> On Fri, Mar 31, 2017 at 6:56 AM, Joshua Smith <mrjoshuaesm...@gmail.com 
> <mailto:mrjoshuaesm...@gmail.com>> wrote:
> The existing mail API makes sense for things like “ereporter” that 
> automatically sends the administrator a daily email summarizing unhandled 
> exceptions.
> 
> It also works fine for low-volume cases, like password authentication for 
> apps that only get a dozen or so new users a day.
> 
> It’s quite absurd that google doesn’t offer a high-volume email service, but 
> that doesn’t mean it shouldn’t at least offer a low-volume one.
> 
> And yes, billing a penny, or even a few cents, for every outbound email is an 
> obvious solution to the spam concern. I have no idea why they don’t just do 
> that. It’d probably take them like 10 minutes to implement.
> 
> -Joshua
> 
>> On Mar 31, 2017, at 5:44 AM, Bookingforce CTO <c...@bookingforce.io 
>> <mailto:c...@bookingforce.io>> wrote:
>> 
>> Hello there,
>>  
>>   being CTO of a company that is betting on Google Infrastructure when I 
>> found out about the limitation to 100 recipients, at first I thought "ah, I 
>> have to increase that quota", then, after asking (twice) to support team, I 
>> felt like "what's wrong with GAE ?" 
>> 
>> The only fact that more than 100 emails is referred as "high volume" is 
>> almost disturbing... we are using GAE to build cloud applications that scale 
>> up using multiple instances but still we can't contact our customers via 
>> email if they are more than 100 ? If you want to prevent spam, bill each 
>> email, we may be willing to pay as much as we pay for other services. 
>> 
>> If you just don't want to offer a serious service about email, you should 
>> think to discontinue the Mail API, like this it makes no sense.
>> 
>> Cheers,
>>   Luca 
>> 
>> On Wednesday, October 26, 2016 at 7:49:05 PM UTC+2, Nick (Cloud Platform 
>> Support) wrote:
>> The Public Issue Tracker thread brought our attention has gotten tracked 
>> now, thanks to PK. We'll update it with any progress.
>> 
>> Now, as much as I'm sure we all appreciate Jeff's kind offer, I don't think 
>> it will be necessary to pay him $1000, since Cloud Platform Community 
>> Support are here to do a similar job at no cost. As said in other threads on 
>> this topic, if anybody at any time has issues migrating from any service to 
>> any other, has questions about design patterns, has questions about the use 
>> of our APIs or how to refactor their code, etc., this forum is the perfect 
>> place for such higher-level discussion and we would be happy to discuss in 
>> depth. 
>> 
>> We never want users to feel left alone, and asking a question once has the 
>> benefit that all other users in a similar situation can read and learn from 
>> the thread. This doesn't fit under the umbrella of "platform issues / 
>> feature requests" (Public Issue Tracker) or "a specific error / stack trace" 
>> (Stack Overflow), so it would be fine to post such threads in this forum.
>> 
>> As an additional, final comment, I'll say that we do pay close attention to 
>> the requests for services / service improvements that users bring forward, 
>> whether here or in the Public Issue Tracker. We've taken the feedback from 
>> many users on the Mail API and bulk sending and while we don't promise any 
>> concrete action given this feedback, know that we have taken it int

Re: [google-appengine] Mail API Quotas

[Joshua Smith]

The existing mail API makes sense for things like “ereporter” that 
automatically sends the administrator a daily email summarizing unhandled 
exceptions.

It also works fine for low-volume cases, like password authentication for apps 
that only get a dozen or so new users a day.

It’s quite absurd that google doesn’t offer a high-volume email service, but 
that doesn’t mean it shouldn’t at least offer a low-volume one.

And yes, billing a penny, or even a few cents, for every outbound email is an 
obvious solution to the spam concern. I have no idea why they don’t just do 
that. It’d probably take them like 10 minutes to implement.

-Joshua

> On Mar 31, 2017, at 5:44 AM, Bookingforce CTO <c...@bookingforce.io> wrote:
> 
> Hello there,
>  
>   being CTO of a company that is betting on Google Infrastructure when I 
> found out about the limitation to 100 recipients, at first I thought "ah, I 
> have to increase that quota", then, after asking (twice) to support team, I 
> felt like "what's wrong with GAE ?" 
> 
> The only fact that more than 100 emails is referred as "high volume" is 
> almost disturbing... we are using GAE to build cloud applications that scale 
> up using multiple instances but still we can't contact our customers via 
> email if they are more than 100 ? If you want to prevent spam, bill each 
> email, we may be willing to pay as much as we pay for other services. 
> 
> If you just don't want to offer a serious service about email, you should 
> think to discontinue the Mail API, like this it makes no sense.
> 
> Cheers,
>   Luca 
> 
> On Wednesday, October 26, 2016 at 7:49:05 PM UTC+2, Nick (Cloud Platform 
> Support) wrote:
> The Public Issue Tracker thread brought our attention has gotten tracked now, 
> thanks to PK. We'll update it with any progress.
> 
> Now, as much as I'm sure we all appreciate Jeff's kind offer, I don't think 
> it will be necessary to pay him $1000, since Cloud Platform Community Support 
> are here to do a similar job at no cost. As said in other threads on this 
> topic, if anybody at any time has issues migrating from any service to any 
> other, has questions about design patterns, has questions about the use of 
> our APIs or how to refactor their code, etc., this forum is the perfect place 
> for such higher-level discussion and we would be happy to discuss in depth. 
> 
> We never want users to feel left alone, and asking a question once has the 
> benefit that all other users in a similar situation can read and learn from 
> the thread. This doesn't fit under the umbrella of "platform issues / feature 
> requests" (Public Issue Tracker) or "a specific error / stack trace" (Stack 
> Overflow), so it would be fine to post such threads in this forum.
> 
> As an additional, final comment, I'll say that we do pay close attention to 
> the requests for services / service improvements that users bring forward, 
> whether here or in the Public Issue Tracker. We've taken the feedback from 
> many users on the Mail API and bulk sending and while we don't promise any 
> concrete action given this feedback, know that we have taken it into account 
> and we want to thank you for bringing up how you view things, what you'd like 
> to see, etc.
> 
> Cheers,
> 
> Nick
> Cloud Platform Community Support
> 
> On Tuesday, October 18, 2016 at 10:42:26 PM UTC-4, Jeff Schnitzer wrote:
> Since this thread keeps coming up… I’ll make you all an offer: For $1k I’ll 
> migrate your GAE app to Sendgrid, Mailgun, or whatever other email service 
> you want. Assuming your code isn’t spaghetti, it will probably take me an 
> hour. It should take you about the same or less.
> 
> This is sooo much kvetching about something that is so trivial to fix.
> 
> Jeff
> 
> On Tue, Oct 18, 2016 at 9:27 AM, PK <p...@ <>gae123.com <http://gae123.com/>> 
> wrote:
> +1 for Joshua’s comment. 
> 
> Nick, wrt "We always encourage users to make use of the Public Issue Tracker 
> “ the issue is issue 10560 
> <https://code.google.com/p/googleappengine/issues/detail?id=10560> stuck in 
> state “New” for 2.5 years like 100s maybe thousands other issues in the 
> public tracker.
> 
> PK
> p...@ <>gae123.com <http://gae123.com/>
> 
> 
> 
> 
>> On Oct 18, 2016, at 8:50 AM, Joshua Smith <mrjoshu...@ <>gmail.com 
>> <http://gmail.com/>> wrote:
>> 
>> Nick,
>> 
>> I know you are stuck in the apologist position, and there’s nothing you can 
>> do about it, but come on…
>> 
>> It is positively absurd that GAE doesn’t include any bulk email capability. 
>> If you guys are concerned about spammers abusing the s

[google-appengine] Re: Deployment Issues

[Joshua Smith]

And just like that, the problem disappeared, and it’s all working now.

> On Mar 30, 2017, at 10:58 AM, Joshua Smith <mrjoshuaesm...@gmail.com> wrote:
> 
> I got a weird error when I tried to deploy. I updated to the latest Python 
> Launcher code, and tried again, and it said I need to rollback.
> 
> This was the result:
> 
> $ appcfg.py rollback .
> 10:54 AM Application: kaon-log
> 10:54 AM Host: appengine.google.com <http://appengine.google.com/>
> 10:54 AM Rolling back the update.
> Error 404: --- begin server output ---
> 
> 
> 
> 404 Not Found
> 
> 
> Error: Not Found
> The requested URL 
> /api/appversion/rollback?app_id=kaon-logforce_rollback=0version=58
>  was not found on this server.
> 
> 
> --- end server output ---
> 
> 
> Is there an outage?
> 
> -Joshua
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/F6C0FCAF-0D8E-466D-83EF-909B832C22F9%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Deployment Issues

[Joshua Smith]

I got a weird error when I tried to deploy. I updated to the latest Python 
Launcher code, and tried again, and it said I need to rollback.

This was the result:

$ appcfg.py rollback .
10:54 AM Application: kaon-log
10:54 AM Host: appengine.google.com
10:54 AM Rolling back the update.
Error 404: --- begin server output ---



404 Not Found


Error: Not Found
The requested URL 
/api/appversion/rollback?app_id=kaon-logforce_rollback=0version=58
 was not found on this server.


--- end server output ---


Is there an outage?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/06FB8122-7773-4127-9D83-7BDB0D7EE41C%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Google Code issue trackers read-only

[Joshua Smith]

…after the migration, the issue tracker will return to being write-only.

;)


> On Mar 2, 2017, at 12:22 PM, 'Jesse Scherer (Google Cloud Support)' via 
> Google App Engine  wrote:
> 
> Greetings all,
> 
> We're marking the Google Code App Engine project read-only in order to 
> perform the migration to Google Issue Tracker 
> , which will replace 
>  Google Code as 
> the backend for public-facing issue trackers. While some mechanics will 
> change, all content will be preserved. When we're done, we'll set up 
> redirects so that old links to Google Code forward to to the corresponding 
> Issue Tracker issue.
> 
> I'll post again when we're back!
> 
> Regards,
> Jesse
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/3a2e6afb-1712-40f7-9c7d-890618e13bea%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/B4C50225-72FD-4307-9A6F-D068759044A1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Switching from Backends to Modules

[Joshua Smith]

Thanks. I did eventually find that.

Google cloud is getting to be a lot like Microsoft. Everything is documented, 
but there are so many docs it’s impossible to find what you are looking for.

> On Feb 24, 2017, at 4:20 PM, 'Adam (Cloud Platform Support)' via Google App 
> Engine <google-appengine@googlegroups.com> wrote:
> 
> In your case to deploy the 'reporting' service you would use:
> 
> appcfg.py update reporting.yaml
> 
> You can list any number of .yaml files for services to deploy after 
> 'appcfg.py update'. The usage is given in the documentation under Deploying 
> multiple service applications 
> <https://cloud.google.com/appengine/docs/standard/python/tools/uploadinganapp#to_deploy_multiple_services>.
> 
> On Friday, February 24, 2017 at 10:46:40 AM UTC-5, Joshua Smith wrote:
> I have a process I need to let run longer than 10 minutes, and I have been 
> using “backends” for this.
> 
> I saw the deprecation warning, so I decided to try making it into a module 
> instead. The documentation on this is a complete mess. Hey google: You need 
> to go clean this up. There are a whole lot of different pages that describe 
> bits and pieces of the puzzle, but I can’t find anything that tells me what I 
> actually need to do. It doesn’t help that backends became modules became 
> services. SMDH.
> 
> Anyway, I found the article that told me how to generate a .yaml for my 
> module based on my old backends.yaml and my app.yaml. My module is called 
> “reporting” so I have reporting.yaml.
> 
> After making that and deleting my backends.yaml, I clicked deploy on the GAE 
> launcher, but that didn’t do anything about my module.
> 
> So I guess I need to run appcfg.py manually? But I have no idea what 
> arguments to give it.
> 
> I just want to push the latest code up, and then I will hit the module with a 
> URL.
> 
> I used to do this with: appcfg.py backends  update
> 
> What command should I issue to get the same result with my “reporting” module?
> 
> -Joshua
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> <mailto:google-appengine+unsubscr...@googlegroups.com>.
> To post to this group, send email to google-appengine@googlegroups.com 
> <mailto:google-appengine@googlegroups.com>.
> Visit this group at https://groups.google.com/group/google-appengine 
> <https://groups.google.com/group/google-appengine>.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/697eccc5-b17a-4852-8b09-12627d347d10%40googlegroups.com
>  
> <https://groups.google.com/d/msgid/google-appengine/697eccc5-b17a-4852-8b09-12627d347d10%40googlegroups.com?utm_medium=email_source=footer>.
> For more options, visit https://groups.google.com/d/optout 
> <https://groups.google.com/d/optout>.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4872BB30-48FF-4245-8599-20FD67874BB0%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: OAuth2 server-to-server on Google App Engine

[Joshua Smith]

Nevermind. I found this:

https://developers.google.com/identity/protocols/OAuth2ServiceAccount 
<https://developers.google.com/identity/protocols/OAuth2ServiceAccount>

The code is pretty simple:

class GetGAHandler(webapp2.RequestHandler):
  def get(self):
from oauth2client.service_account import ServiceAccountCredentials
from httplib2 import Http
from apiclient.discovery import build
credentials = ServiceAccountCredentials.from_json_keyfile_name(
  CLIENT_SECRETS, 
scopes=['https://www.googleapis.com/auth/analytics.readonly'])
http_auth = credentials.authorize(Http())
service = build("analytics", "v3", http=http_auth)

Getting the google-api-python-client installed in a local folder was a 
nightmare, and convincing GAE to use it wasn’t obvious. However, once I sorted 
through all that, the above code works like a champ.

> On Feb 24, 2017, at 1:27 PM, Joshua Smith <mrjoshuaesm...@gmail.com> wrote:
> 
> I’m trying to access the Google Analytics API from a Google App Engine app. I 
> got this working a long time ago, before Analytics was one of the APIs you 
> could use server-to-server.
> 
> So I have code that works with an administrator user in the middle of the 
> transaction. I used the decorator from oauth2client.appengine import 
> oauth2decorator_from_clientsecrets
> 
> What I’d like to do is get this working so my google app engine app (written 
> in Python) can just talk to google analytics without a person in the loop. I 
> can’t find an example of that.
> 
> Since all the google services seem to work the same way, I probably don’t 
> need an example of connecting to analytics. Rather, just an example of how to 
> connect to any google service without a user in the middle would be great.
> 
> The following code doesn’t work, but hopefully will give you an idea of what 
> I’m trying to do. It is failing because there’s no callback set up.
> 
> class StorageDuck():
>  def put(self, content):
>console.log("Pretending to put: "+content)
> 
> class GetGAHandler(webapp2.RequestHandler):
>  def get(self):
>from apiclient.discovery import build
>from oauth2client import client
>from oauth2client import file
>from oauth2client import tools
>http = httplib2.Http(memcache)
>service = build("analytics", "v3", http=http)
>flow = client.flow_from_clientsecrets(
>  CLIENT_SECRETS,
>  scope='https://www.googleapis.com/auth/analytics.readonly',
>  message="")
>credentials = tools.run(flow, StorageDuck())
>chttp = credentials.authorize(http=httplib2.Http())
> 
> Any pointers?
> 
> -Joshua
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/473D4DE5-7EAA-42CB-81D5-EA3A53FC1AE6%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] OAuth2 server-to-server on Google App Engine

[Joshua Smith]

I’m trying to access the Google Analytics API from a Google App Engine app. I 
got this working a long time ago, before Analytics was one of the APIs you 
could use server-to-server.

So I have code that works with an administrator user in the middle of the 
transaction. I used the decorator from oauth2client.appengine import 
oauth2decorator_from_clientsecrets

What I’d like to do is get this working so my google app engine app (written in 
Python) can just talk to google analytics without a person in the loop. I can’t 
find an example of that.

Since all the google services seem to work the same way, I probably don’t need 
an example of connecting to analytics. Rather, just an example of how to 
connect to any google service without a user in the middle would be great.

The following code doesn’t work, but hopefully will give you an idea of what 
I’m trying to do. It is failing because there’s no callback set up.

class StorageDuck():
  def put(self, content):
console.log("Pretending to put: "+content)

class GetGAHandler(webapp2.RequestHandler):
  def get(self):
from apiclient.discovery import build
from oauth2client import client
from oauth2client import file
from oauth2client import tools
http = httplib2.Http(memcache)
service = build("analytics", "v3", http=http)
flow = client.flow_from_clientsecrets(
  CLIENT_SECRETS,
  scope='https://www.googleapis.com/auth/analytics.readonly',
  message="")
credentials = tools.run(flow, StorageDuck())
chttp = credentials.authorize(http=httplib2.Http())

Any pointers?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/AD42F353-F9AC-4433-B6F2-4EF9863448C3%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Switching from Backends to Modules

[Joshua Smith]

I have a process I need to let run longer than 10 minutes, and I have been 
using “backends” for this.

I saw the deprecation warning, so I decided to try making it into a module 
instead. The documentation on this is a complete mess. Hey google: You need to 
go clean this up. There are a whole lot of different pages that describe bits 
and pieces of the puzzle, but I can’t find anything that tells me what I 
actually need to do. It doesn’t help that backends became modules became 
services. SMDH.

Anyway, I found the article that told me how to generate a .yaml for my module 
based on my old backends.yaml and my app.yaml. My module is called “reporting” 
so I have reporting.yaml.

After making that and deleting my backends.yaml, I clicked deploy on the GAE 
launcher, but that didn’t do anything about my module.

So I guess I need to run appcfg.py manually? But I have no idea what arguments 
to give it.

I just want to push the latest code up, and then I will hit the module with a 
URL.

I used to do this with: appcfg.py backends  update

What command should I issue to get the same result with my “reporting” module?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9DCE5E00-1BE4-4B33-8C0F-D96870359019%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] How do I obtain a CSR?

[Joshua Smith]

Honestly, someone with “very little technical knowledge” should not attempt to 
work with SSL certificates.

HTTPS is about setting up a secure website and letting the people who use that 
website know that you are doing things correctly to protect them.

You should find someone who understands SSL (and web security in general) to 
take care of this for you.

Think of it like trying to repair the airbag on your car by yourself. It’s just 
a bad idea.

-Joshua

> On Jan 24, 2017, at 8:19 AM, Liz Furber  wrote:
> 
> I have very little technical knowledge. I have bought an SSL certificate 
> online for our charity's website and regularly receive emails from the CA 
> asking me for the CSR. I have no idea how to get this from Google Cloud 
> Platform. The webpage - 
> https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl#obtaining_a_certificate
>  - gives the relevant information I am sure, but I am already lost at section 
> 1a. Can anyone help with a step by step guide on how I do this? I bought the 
> certificate in mid December and really need to get on with this. Any help 
> would be amazing. Thank you.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to google-appengine+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to google-appengine@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/google-appengine 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/google-appengine/39d9-9e83-4fdb-a174-7350b0ebdf38%40googlegroups.com
>  
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/2C558281-AA25-4637-ABBA-C8264BA878C1%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Migrating from G-Suite to Direct Domain Mapping

[Joshua Smith]

I have an old service that I set up back when you had to use G-Suite (then 
called Google Apps) to map a domain name to an appengine service.

I’m paying $5/mo for that G-Suite one-seat license, even though I don’t use it.

What is the procedure to use to stop using G-Suite to assign a domain to a 
service and instead map it directly?

After I change that, I’ll cancel the G-Suite subscription, and hopefully not 
break everything.

Pointers?

-Joshua

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/DD4EC40D-96FF-422F-9F3A-093163BDA840%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Is Google App Engine Launcher being deprecated? Why??

[Joshua Smith]

Yes, obviously I know all that, because I said I’m writing scripts for every 
project. So instead of having a nice GUI that does it all for me, I have to 
create and maintain a dozen scripts. This is progress?

> With these operations, you could make your own custom user interface for 
> managing your projects, applications and services.

Wait. What?

Does google actually believe that instead of providing an easy-to-use interface 
for setting up and managing applications and services, enterprises want to 
build their OWN GUIs to do this?

Do you guys ever actually talk to your customers?



> On Dec 26, 2016, at 5:06 PM, 'Nicholas (Google Cloud Support)' via Google App 
> Engine <google-appengine@googlegroups.com> wrote:
> 
> Thanks for the feedback.
> 
> Regarding the switch to different projects, one can use the global flag 
> --project=my_project_id <https://cloud.google.com/sdk/gcloud/reference/> for 
> a given command to be performed with a specific context.  Alternatively, one 
> could use gcloud config set project my_project_id 
> <https://cloud.google.com/sdk/gcloud/reference/config/set> to achieve a 
> longer term effect.  This will remain in effect until changed.
> 
> The default behavior to promote a new deployment can be changed per 
> deployment using the --no-promote 
> <https://cloud.google.com/sdk/gcloud/reference/app/deploy> flag.  If you wish 
> deployments not to be promoted by default, you can use gcloud config set 
> app/promote_by_default false 
> <https://cloud.google.com/sdk/gcloud/reference/app/deploy>.  This will remain 
> the new default behavior unless changed or using the --promote flag when 
> deploying.
> 
> Depending on the needs of your company, you may want to consider the App 
> Engine Admin API 
> <https://cloud.google.com/appengine/docs/admin-api/getting-started/>.  These 
> APIs allow you to create applications, create/list/delete service versions, 
> etc. programmatically.  With these operations, you could make your own custom 
> user interface for managing your projects, applications and services.
> 
> On Wednesday, December 21, 2016 at 3:36:57 PM UTC-5, Joshua Smith wrote:
> Having used both, the GUI is much better in two cases:
> 
> 1. When you have a lot of projects, it’s nice to have the GUI to just click 
> and update, vs having to root around finding the right script to run.
> 
> 2. I could train a non-programmer to push an update to a web app using the 
> GUI. Not so using the command line.
> 
> Also, the whole setup for the command line is really focused on the idea that 
> you’ll have only one project, which is pretty silly. So you end up having to 
> write simple scripts as wrappers around the command line stuff, just to pass 
> project names.
> 
> And one other issue with the command line that was really just a violation of 
> the principle of least surprise: deploying should NOT route all traffic to 
> the version you just pushed. That’s insane. Whoever thought that was a good 
> default clearly has no experience developing real applications. Not a big 
> deal to work around because I have to write the aforementioned wrapper 
> scripts anyway, but… seriously?
> 
> -Joshua
> 
>> On Dec 21, 2016, at 2:39 PM, 'Nicholas (Google Cloud Support)' via Google 
>> App Engine <google-appengine@googlegroups.com 
>> <mailto:google-appengine@googlegroups.com>> wrote:
>> 
>> I am not aware of any plans to create a GUI for the Cloud SDK.
>> Are there particular features of the launcher that are unavailable or 
>> difficult to use with gcloud command line tool?
>> Are there any particular pain points to the gcloud command line tool?
>> 
>> On Wednesday, December 21, 2016 at 1:31:10 PM UTC-5, George Bittmann wrote:
>> The App Engine Launcher was largely how I was able to sell my team on using 
>> App Engine.
>> 
>> Nicholas do you know if there are plans to create a GUI product for App 
>> Engine using the new Cloud SDK?
>> 
>> Knowing one way or the other will be helpful.
>> 
>> 
>> 
>> 
>> On Monday, December 19, 2016 at 4:54:41 PM UTC-5, Nicholas (Google Cloud 
>> Support) wrote:
>> Thank you for voicing your concerns.
>> 
>> I can confirm that we are strongly moving towards having App Engine a 
>> component of the Cloud SDK but there are no public plans to deprecate the 
>> App Engine SDKs.
>> 
>> If you wish to have multiple services running using dev_appserver.py, you 
>> can do so by by running multiple instances of them.  As stated in the Local 
>> Development Server Options 
>> <https://cloud.google.com/appengine/docs/python/tools/local-devserver-command