[google-appengine] Re: Microservices and instance costs

[Ani Hatzis]

Hi Richard,

I'm not sure I understand what you mean by "base costs" and "on top of 
instance hours". In standard environment you pay for instance hours in the 
App Engine sandbox, and as they can scale down to 0 instances, and 28 hours 
of F1 are free per day, you may even pay nothing. In flexible environment 
you pay for the virtual machines used by your app. I was referring to 
standard environment.

The price for every instance is a multiple of the cheapest class (F1/B1), 
e.g. F2/B2 costs twice as much as F1/B1. But in your billing, all instance 
hours are converted to F1/B1 hours and billed with $0.05 per hour.

You probably expect your bill to look like this for all services in this 
app:

   - 10 instance hours *F1* x $0.05 = $0.50
   - 3 instance hours *F2* x $0.10 = $0.30

But your billing will actually show all classes converted to F1/B1. 1 hour 
of F2 equals 2 hours of F1:

   - *16* instance hours x $0.05 = $0.80

Whether you use one service or multiple services doesn't change the 
pricing. But there can be cases where your app may *consume* more instance 
hours because the instances can't be shared across all services.

Service 1: has one instance of F1 running 24/7, and handles the low traffic 
coming to it. That's 24 hours.
Service 2: performs a job every 1 hour and thus needs to spin up its own F1 
instance occasionally, that's another few hours per day.

If you would have a monolithic app (i.e. 1 service), all request handlers 
would be available in the same instance, and thus the chances are better 
that you would get away with a single instance for everything together. 
Hence the better utilization of available (and billed) instances.

But there can be also other cases, where using multiple services would save 
you cost. For example if only a few request handlers (e.g. background 
tasks) need 1GB memory, then you would put them into their own service with 
a bigger class and put all other handlers into a separate service with a 
cheaper instance class, rather than serving everything with the expensive 
class.

Either way, for billing all instance hours in your app are lumped together 
and converted to F1/B1 prices. In this regard, think about the services 
like having multiple "apps" billed together in the same project.

Well, this explanation hopefully didn't make it worse for you Just ask 
again.
Best,
Ani


On Monday, August 20, 2018 at 5:29:03 PM UTC+2, Richard Cheesmar wrote:
>
> Hi, Ani, 
>
> Thanks for reply, but does that mean I am therefore paying the base cost 
> for each instance of each service on top of instance hours.
>
>
> On Monday, August 20, 2018 at 6:11:16 PM UTC+3, Ani Hatzis wrote:
>>
>> Hi Richard,
>>
>> instances are not shared across services, each service has its own 
>> instances (you can define different scaling and instances classes per 
>> service).
>>
>> In your billing, you will not see the instance hours broken down to 
>> services, but for all services of the project together.
>>
>> This also means that if you have low traffic per service you may see 
>> under-utilization of instances, compared to a monolithic app with just one 
>> service.
>>
>> Hope that clarified it a little.
>> Ani
>>
>> On Monday, August 20, 2018 at 2:30:24 PM UTC+2, Richard Cheesmar wrote:
>>>
>>>
>>> Regarding App Engine Microservice costs, in the Google docs, it states 
>>> the following
>>>
>>> Costs for instance-hours (the CPU and memory for running your code) are 
>>> not separated for services; all the instance-hours for an entire project 
>>> are lumped together. 
>>>
>>> Can anyone tell me if this implies that for example there is n instances 
>>> on a project that are shared across all microservices or does each 
>>> microservice have a separate instance that is charged at a base rate and 
>>> then on top instance hours used.
>>>
>>> Thanks
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/9b89304d-b8b9-4f8b-817d-0959e06eb30c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Microservices and instance costs

[Ani Hatzis]

Hi Richard,

instances are not shared across services, each service has its own 
instances (you can define different scaling and instances classes per 
service).

In your billing, you will not see the instance hours broken down to 
services, but for all services of the project together.

This also means that if you have low traffic per service you may see 
under-utilization of instances, compared to a monolithic app with just one 
service.

Hope that clarified it a little.
Ani

On Monday, August 20, 2018 at 2:30:24 PM UTC+2, Richard Cheesmar wrote:
>
>
> Regarding App Engine Microservice costs, in the Google docs, it states the 
> following
>
> Costs for instance-hours (the CPU and memory for running your code) are 
> not separated for services; all the instance-hours for an entire project 
> are lumped together. 
>
> Can anyone tell me if this implies that for example there is n instances 
> on a project that are shared across all microservices or does each 
> microservice have a separate instance that is charged at a base rate and 
> then on top instance hours used.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/bfc7b3b8-70e7-4621-9812-ca85ed82f67e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Custom Domain much slower than $PROJECT_ID.appspot.com

[Ani Hatzis]

Hi James,

I believe you are affected by a known issue
. Last status was that
they are working on it, but no ETA.

On Thu, Aug 9, 2018 at 3:41 PM James Kozianski  wrote:

> Hi,
>
> I've set up a custom domain for my service and I find that when accessing
> the service via that domain it is significantly slower than accessing it
> via its $PROJECT_ID.appspot.com address. I have determined that it's not
> an issue with DNS as the difference in latency is still there when using
> the resolved IP addresses directly.
>
> The slowdown is very noticeable: it varies from 2x to 10x slower on the
> custom domain, and the delay occurs in the period between the connection
> being established and the first byte being sent.
>
> Also note my app engine service is running in Sydney.
>
> Is it possible I've misconfigured something? My DNS is CNAMEd to
> ghs.googlehosted.com, but it seems to me that using custom domains for
> routing is simply really slow. Is there anything I can do to improve the
> latency here?
>
> Cheers,
> James
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/78aa47f9-eb0c-4fb2-ab5d-94f678b205bd%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ5js4_eH5waG1gbe7%3DjkMrSS%2BL5v3mJOtQ-d6cBiiVjfg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Can't deploy changes in console: error response 13: An internal error occurred

[Ani Hatzis]

Ask her to try again now (since your post is already two hours old).
However, if the problem persists, ask her to run the command with

gcloud app deploy . --verbosity=debug

to get more information about the potential reason.

There have been reports with this error code and message in this group
previously. If a Mac was used somewhere in the process, the error might be
caused by an "Icon?" file that was accidentally added to the application
code / repository. See this post
.
In general, to avoid this kind of error in the future, it makes sense to
define the skip_files

directive in app.yaml.


On Mon, Jul 23, 2018 at 3:00 PM Victor Victor 
wrote:

> Hi!
>
> I have added a friend in my Google Cloud project and gave her Owner, App
> Engine Admin, App Engine Deployer and App Engine Service Admin rights in
> IAM but when in her console she runs the command "*gcloud app deploy*"
> and tries to deploy the changes she made in her code in order to deploy her
> version, an error response 13: An internal error occurs.
>
> Can someone help us find a solution to this so that she will be able to
> deploy her changes to the live version?
>
>
> 
>
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/62e862b1-0776-42b8-9217-c789b3d798db%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ7eqj9pK8V9rig_yu2%2BSs-eAnisyoXSXAvO%3D57i1ZC31A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Old versions still running (flexible environment)

[Ani Hatzis]

gh they are a bit
>>> higher than expected (still looking into that). I'd submit a billing ticket
>>> if I was sure.
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/6678e51f-cb09-4cac-b4cc-83890975bbbd%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/6678e51f-cb09-4cac-b4cc-83890975bbbd%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Ani Hatzis
Consultant and developer for Google Cloud solutions

<https://www.credential.net/ukfxy03s> <https://www.credential.net/b0y6kwie>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ59RCZF83C7ei%2Bof%2BK4-Y5A0Rtvga3yfuSnOKrD-ot5dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Using GAE to upload images to Google Cloud Storage?

[Ani Hatzis]

Non-user applications can create signed URLs, see Creating Signed URLs with 
a Program 
.
 
You can do that in App Engine, if you want to enforce some kind of business 
logic, or let the external app create the signed URL by itself. With that 
approach the actual upload would go directly into the GCS bucket, without 
GAE in the middle.

However, if you don't need to return the unique ID in the response of the 
upload, you could consider to use a Google Cloud Function (Node.js and 
triggered by events in a GCS bucket, such as after an object is written 
into the bucket). The function gets the metadata of the written object and 
can create the unique ID and can write it to Cloud Datastore, for example, 
or to the custom metadata of the GCS object. For details see GCF's Cloud 
Storage Tutorial 
.

On Thursday, July 12, 2018 at 6:03:20 AM UTC+2, Parth Mishra wrote:
>
> I have a standard environment service that is handling HTTP POST requests 
> of images. It basically just creates a unique ID for them based on some 
> request metadata and then uploads to Cloud Storage.
>
> My question is, is GAE the right choice to handle this? I'll be getting 
> ~70K of these images (less than an MB each) throughout the day with various 
> peak and slow traffic times. I cant' generate a signed URL since the 
> application sending the request is an external, non-user application. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/91f19167-e817-4cf1-8e89-f404d0c9effb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Can app engine host a Node JS worker?

[Ani Hatzis]

Maybe Cloud Functions will be added to Choosing a Computing Option 
 once it gets out 
of beta?

In general I would say, that Cloud Functions (almost no control, focus on 
code) is on the opposite side of Compute Engine (full control, lots of 
DIY). Compared to App Engine, GCF is "even more serverless", probably more 
suitable for small jobs, rather than fully fledged web apps / API backends. 
GAE comes with different scalability options and version/traffic 
management, and I believe that automated testing and deployment of an app 
with dozens of endpoints, maybe hundreds, is easier with such features. 
(Although Testing and CI/CD 
 is possible 
with GCF).

On the other hand, if you want to build just a few functions to accompany 
Firebase, or utility functions for Cloud Storage, or integrate (GCP) 
resources through Pub/Sub and light-weight computing, then GCF might be the 
better option.

With the rise of microservice architecture, GCF might become an excellent 
compute option for microservices, too. However, it's still in beta, and I 
believe that a few features are missing yet that would go a long way here: 
Google-managed SSL with custom domains, builtin client authentication, and 
better control of versions (or traffic migration). AFAIK, App Engine's Task 
Queue API is promoted to a stand-alone product, Memcache, too. Add Storage 
and Datastore to the mix and I see that GCF can become a reasonable 
alternative to GAE in more use-cases.

Of course, a rather different roadmap for GCF could be to somehow merge GCF 
with its cousin from the Google apps family, Google Apps Script, but I 
don't expect that to happen.


On Wednesday, July 11, 2018 at 10:23:13 AM UTC+2, David Komer wrote:
>
> Just to bump this - the linked docs are great for comparing between App 
> Engine Standard and Flexible, but it doesn't really address Cloud Functions.
>
> It would be helpful to have a comparison / best-practices guideline for 
> choosing between Cloud Functions and App Engine
>
> On Saturday, 30 June 2018 21:30:19 UTC+3, George (Cloud Platform Support) 
> wrote:
>>
>> If your intention is to deploy a full-fledged app, you have the choice 
>> between the Standard Environment and Flexible. The "Choosing an App Engine 
>> Environment" online document 
>>  
>> should help you in taking the decision best suited to your needs. In both 
>> cases, the managed environment works transparently in ensuring that your 
>> desired functionality is there. You can configure your app specifying how 
>> its services scale. Instances are created or stopped in response to 
>> traffic. You may have a look at the "How Instances are Managed" page 
>> 
>>  
>> for more information. 
>>
>> Cloud Endpoints is a distributed API management system, not suitable 
>> for a priority job queue. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0ada7b70-d9ba-467e-9acb-f96dea725c1a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Can App Engine Flexible be autoscaled based on Cloud Pub/Sub queue?

[Ani Hatzis]

I haven't tried this myself, but you can run a flexible app in Kubernetes
Engine
,
and for Kubernetes Engine there is a new custom-metric based autoscaling

feature in beta.

On Tue, Jul 3, 2018 at 12:37 AM Parth Mishra  wrote:

> I know you can configure autoscaling based on queue size when scaling a
> Compute Engine instance group, but I'm unsure of how I could replicate this
> behavior in a flexible App Engine based environment. Is there a way to at
> least mimic this scaling behavior?
>
> If it's not possible in GAE, is it possible in a Kubernetes cluster or is
> it only possible when using GCE?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/bc655f90-063d-4fcb-8ffc-4c8505e5cb01%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ6yRcrvXmjA2hOzcdAx0Bm4ENODsHi_dxch%3DjmvT9Mt-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Does App Engine Flexible for Python support concurrent requests?

[Ani Hatzis]

On Tue, Jul 3, 2018 at 12:37 AM Parth Mishra  wrote:

>
> 1. Isn't the deadline exceeded error only for standard App engine or does
> it apply to both? I was testing this today and I could make requests with
> latency return times of much greater than 60s
>

Correct, request timeout limit is 60 seconds for standard and 60 minutes
for flexible (see comparison here

).


> 2. I guess I'm confused on why the app.yaml documentation for Flexible
> omits things such as 'max_concurrent_requests' and 'target_cpu_throughput'.
> Do those concepts of concurrency not apply to flexible? The only way to
> scale in the flexible engine is to reach a target CPU threshold, but what
> if I want to scale based on request latency as well? Somewhat similarly, I
> want to know if I can configure my web framework to accept new requests
> without waiting for an existing one to finish. Does that clear up what I'm
> trying to achieve?
>

Scaling is indeed different between standard and flexible, see short
description here
.
The "target_cpu_throughput
"
setting of standard is "cpu_utilization : target_utilization
"
in flexible though.


>
> On Monday, July 2, 2018 at 2:00:56 PM UTC-4, Kenworth (Google Cloud
> Platform) wrote:
>>
>> 1- I assume the 60-second value was just some random number. This is
>> because the deadline for requests to frontend instances is 60 seconds.
>> Otherwise, you will be hit by DeadlineExceededErrors
>> .
>>
>> 2- Concurrent, by definition, means an event can exist/happen or be done
>> at the same time. This means it does not have to wait for a single task to
>> finish before executing the next one. GAE Flex environment automatically
>> scales your app and down
>>  while balancing the
>> load. Here is an article
>> 
>> explaining the instance behavior depending if the application is set to
>> manual or automatic scaling.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/52f798a0-9ff7-4df1-8551-b8815ee61142%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ5VCp%2B%2BhCEi6w-jVAb2N07b9nBVcAfGiTKtbK2x_XMj8w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Hey after two days I get a message the I cannot deploy my appengine becuase I already got too many version.

[Ani Hatzis]

There is no particular write-protection for a given version. I can think of 
two solutions in your case:

You could use two user accounts, where only one of them has the permission 
to upload new versions, and normally you would use the other (restricted) 
account for your day-to-day work between deployments. In Google Cloud 
Platform you would use roles to manage access, where a role is like a list 
of permissions, that can be assigned to individual user accounts, Google 
groups, all users in a G Suite domain or Cloud Identity domain, or service 
accounts. The pre-defined roles with App Engine 

 already 
provide a separation, see "App Engine Deployer" and also the additional 
information later on that page regarding deployment and traffic management.

If you don't want to handle two accounts, but have some kind of an 
additional step after an (accidental) deployment, then I suggest to utilize 
App Engine's version management and the *gcloud app deploy* options 
"--no-promote" flag and not to stop the previous version. 

*Set default config settings:*
As you might know, you can manage SDK configurations 
 in Cloud SDK  where you 
can store default settings, even multiple configurations. But I assume you 
only have one project and thus only one configuration (the DEFAULT 
configuration). You can just store the defaults in your gcloud 
configuration.

In *gcloud app deploy*, *--promote* or *--no-promote* flag tells App Engine 
whether traffic should be automatically migrated from the previous version 
to the new version. Turn the promote off by default for all your *gcloud 
app deploy* commands in this config: 

$ gcloud config set app/promote_by_default false

Updated property [app/promote_by_default].

The next option will prevent that a previous version is automatically 
stopped when you deploy a new version:

$ gcloud config set app/stop_previous_version false

Updated property [app/stop_previous_version].

Verify that your configuration is correctly set under the *[app]* section 
(that's the one for App Engine specific configurations):

$ gcloud config list

[app]

promote_by_default = False

stop_previous_version = False

You only need to repeat these steps, if you have a new gcloud installation 
or if you want to create a new gcloud configuration.

*Future Deployments in two steps:*

*Step 1:* From now on, only use *gcoud app deploy* without specifying a 
version (*--version*), without the *--pomote* flag, and without the 
*--stop-previous-version*. Just something like this:

gcloud app deploy .

As no *--version* is provided, App Engine will create a new version and 
give it an automatically created version name (looks like an ISO timestamp 
starting with the year, e.g. 20180630). The new version will not be the 
new default version, and the previous version will not stop to serve 
traffic. So the deployment has no effect to your users until now.

*Step 2:* Next in gcloud or in the Web UI:

   - The deployment was accidental: just delete the accidentally created 
   version (typically the newest one)
   - The deployment was intentional: migrate traffic to the new version or 
   make it the new default version

You can occasionally delete very old versions, so you won't hit the 200 
versions limit per app again.

Best regards,
Ani


On Friday, June 29, 2018 at 9:06:29 PM UTC+2, Guy Dviri wrote:
>
> I reply thanks you 2 hours ago , but I don't see the reply in the thread , 
>> google groups UI must change their layout in my opinion , 
>>
> any way I like to ask @Ani one more thing ,  is it possible to block 
> version from updating , let's say that it will be update only with special 
> key,, or something like that , to prevent update by mistake.. 
> do you know some way to do it?
>  
>
 

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d8ba1ba7-5722-486a-84a6-4789da7d39e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: Hey after two days I get a message the I cannot deploy my appengine becuase I already got too many version.

[Ani Hatzis]

Hello,

when you deploy a new App Engine app version and do not specify a specific
version, App Engine will create a new version automatically. You can see
them in the Web UI under *App Engine > Versions* or in the shell:  gcloud
app versions list

Many users utilize versions using traffic split to perform A/B testing or
soft migrations. If something goes wrong during the rollout of a new
feature, they can easily direct traffic back to an older version.

If you do not want to utilize versions, you could just specify the same
version at every deployment of your App Engine app, for example:
gcloud app deploy --version=v123 .

If version *v123* already exists, App Engine will replace it, and you will
not see a growing list of versions. However, if you care for availability
of your app, I would recommend to deploy into separate versions and make
use of this feature.

On Thu, Jun 28, 2018 at 10:38 PM Guy Dviri  wrote:

> no it's just a small app ,not even up to the air  ,  i don't know why i
> have 200 version i only need one..
> what could i doing wrong..
>
> i using endpoint + sql + bigstore .
> i deploy my module using :
>
> console : appenginedeploy
>
> what do you think could trigger this behavior ?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/17eec7eb-5203-4c27-bee0-e922010d38b7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ6RVmBpUF11Mxn6nidihTfnuj4NQJdQqeqrG1ZJZy0mhA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: App Engine Standard: "heroku run" equivalent

[Ani Hatzis]

In context of running the migration from your local machine, do you also 
make use of App Engine's versioning system? See this Stackoverflow answer 
.

On Thursday, June 21, 2018 at 12:37:35 AM UTC+2, rgmenon wrote:
>
> I have a Django application running on App Engine *Standard* env, 
> connected to a CloudSQL DB. I'd like to update the schema (when it changes) 
> from the deployed App Engine instance by executing:
> python manage.py migrate
> from the App Engine instance.
>
> In other words, I'm looking for the App Engine Standard equivalent of the 
> Heroku command:
> heroku run python manage.py migrate
>
> What's the recommended way to accomplish this task on App Engine Standard 
> environment? Currently, I work around the issue by connecting to the DB 
> using cloud_sql_proxy and running the migration from a local machine.
>
> Thank you!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/3237bcd5-df43-4060-aead-8353ec9102d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Problem: Can not get TinyWebDB connected to Google Cloud Platform

[Ani Hatzis]

Dennis, I'm glad it helped. Later on you might possibly also have technical 
questions regarding App Engine for which I recommend Stack Overflow using 
the tag [google-app-engine] 
<https://stackoverflow.com/questions/tagged/google-app-engine>. Questions 
with this tag (and others) are also monitored by the support team of Google 
Cloud.

How was your experience using Cloud Shell on a Chromebook? I'm considering 
to switch once my MBP dies on me.


On Wednesday, June 13, 2018 at 6:06:48 PM UTC+2, Dennis Pike wrote:
>
> Ani ... many thanks.  After 2 days of following so many outdated 
> procedures for connecting tinywebDB to Google Cloud Platform, after you 
> showed me the cloud shell, and with a little more searching the web ... I 
> got it all done and working in 20 minutes.  My mobile app (developed in App 
> Inventor 2) is now talking to the Google Cloud Platform via TinyWebDB.  
> Could not wait to get back here and tell you ...many thanks.   After a 
> coffee and a break I will document this process THAT WORKS so I have it for 
> future reference!  Thank you again. 
>
> On Wednesday, June 13, 2018 at 11:05:43 AM UTC-4, Ani Hatzis wrote:
>>
>> Hi Dennis,
>>
>> I haven't looked into App Inventor for a long time. But luckily you don't 
>> need a Windows machine today, just a browser.
>>
>> You can use Google Cloud Shell <https://cloud.google.com/shell/docs/>, 
>> which is a shell environment for managing your resources hosted on GCP and 
>> it runs in your browser (and so on your Chromebook, too). Among many other 
>> useful things it also comes with Cloud SDK pre-installed. It is just your 
>> own small virtual machine. And it's free for GCP customers.
>>
>> As a new GCP customer also make sure to take advantage of the free 
>> starter credits that you might get offered in the Cloud Console. In order 
>> to avoid unnecessary costs, I recommend that you also set a daily budget 
>> for your App Engine app 
>> <https://cloud.google.com/appengine/docs/standard/python/console/#setting_a_spending_limit>
>>  
>> (once you enabled billing for it) and/or also setup billing alerts 
>> <https://cloud.google.com/appengine/docs/standard/python/console/#managing_billing>
>>  
>> which also works outside of standard environment apps.
>>
>> Cheerio,
>> Ani
>>
>> On Wed, Jun 13, 2018 at 4:41 PM Dennis Pike  wrote:
>>
>>> Hi New to this forum. If not posted in proper place, someone move it. ...
>>> Problem: Can not get TinyWebDB connected to Google Cloud Platform
>>> History: Discovered, invested much time into learning App Inventor 2 (
>>> http://ai2.appinventor.mit.edu).  Learned to use TinyDB. Learned how to 
>>> replace TinyDB with TInyWebDB using their default cloud storage for 
>>> development. (http://appinvtinywebdb.appspot.com/)... understanding 
>>> this is not for a finished deployed production environment mobile app. 
>>> Created a Google Cloud Platform Account. Then a Project. Everything I find 
>>> / read regarding making the TinyWebDB connect to the Google Cloud Platform 
>>> involves downloading Python 2.7.x  ... and Google Cloud SDK. (most of this 
>>> documentation seems out of date). 
>>> Problem: I only have a Chromebook and these downloads are for Mac, 
>>> Windows or Linux.  
>>> More History: Installed Linux on Chrombook via GIThub.com (Chromium OS 
>>> Universal Chroot Environment https://goo.gl/fd3zc). Downloaded Google 
>>> Cloud SDK and Python 2.7.x.  None of the commands I issue work..inside the 
>>> Crouton Environment. Alway say bad command.
>>> Questions: Why is it necessary for me to download and configure all this 
>>> stuff just to get a simple URL for the tinywebDB SourceURL? Do I REALLY 
>>> need to go out an buy a windows machine (ugh) just to get this connection?
>>>
>>> Thanks for any feedback. 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to google-appengi...@googlegroups.com.
>>> To post to this group, send email to google-a...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/d2052ec8-7d10-4b15-9001-b37686cfdae7%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/google-appengine/d2052ec8-7d10-4b15-9001-b37686cfdae7%40goo

[google-appengine] Re: Cloud endpoints are running slow in App Engine Standard Environment

[Ani Hatzis]

Since your app is not in production yet, you could test the latency by 
changing your scaling from auto scaling to basic or manual scaling in 
app.yaml (see Known Issues 
)
 
and compare the latencies of both scaling settings.

I would also compare the traces with a request that was directly sent to 
your API's backend, without Cloud Endpoints. Perhaps the problem is in your 
app, not with Endpoints.

I'm not 100% sure when the trace starts, but as far as I understand the 
documentation, the timeline of a trace starts when the (App Engine) 
application received the incoming request, not when Cloud Endpoints 
received the request (mentioned here 
 and here 
). If 
it's true that Cloud Endpoints latency is not shown in the trace, then I'd 
assume that you would see the same high latency if you send a request 
directly to the API's backend. However, by following the logs of a trace or 
by comparing the trace timestamps with those of the matching logs in App 
Engine and Cloud Endpoints, you should be able to verify whether Endpoints 
latency is shown in a trace or not. (the log in Endpoints 
 
would start before the log in App Engine 

 
and end after the log in App Engine ended)

If the trace is just for your App Engine app, I would try to figure out 
what actually happens in the long block of *untraced* time, i.e. before you 
see the first span, e.g. the first RPC to datastore. I don't know how it is 
with Java standard, but for Python standard, the first RPCs show up after 
maybe 10ms, 20ms - or if it's the first request of a new instance maybe 
after 400ms. You could use Cloud Logging and Cloud Debug to get a better 
understanding why it takes so much time until the first RPC call.

I hope this helps.

Best regards,
Ani

On Tuesday, June 12, 2018 at 9:53:40 AM UTC+2, Azeem Haider wrote:
>
> I'm using Automatic Scaling.
>
> Sorry, in my first post I did not upload stack traces. Check these below
>
>
> 
>
>
> Here is RPCs. And can you please what RPCs and LATENCY time mean here ?
>
>
> 
>
>
> Expanding trace for most user URI.
>
>
> 
>
>
> Here is trace detail of user request that I mention in my question, if you 
> see it fetch from memory but still function take around 2 sec
>
>
> 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0d95dfd3-6e5c-4272-88a4-79fa5f32964e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Problem: Can not get TinyWebDB connected to Google Cloud Platform

[Ani Hatzis]

Hi Dennis,

I haven't looked into App Inventor for a long time. But luckily you don't
need a Windows machine today, just a browser.

You can use Google Cloud Shell <https://cloud.google.com/shell/docs/>,
which is a shell environment for managing your resources hosted on GCP and
it runs in your browser (and so on your Chromebook, too). Among many other
useful things it also comes with Cloud SDK pre-installed. It is just your
own small virtual machine. And it's free for GCP customers.

As a new GCP customer also make sure to take advantage of the free starter
credits that you might get offered in the Cloud Console. In order to avoid
unnecessary costs, I recommend that you also set a daily budget for your
App Engine app
<https://cloud.google.com/appengine/docs/standard/python/console/#setting_a_spending_limit>
(once you enabled billing for it) and/or also setup billing alerts
<https://cloud.google.com/appengine/docs/standard/python/console/#managing_billing>
which also works outside of standard environment apps.

Cheerio,
Ani

On Wed, Jun 13, 2018 at 4:41 PM Dennis Pike  wrote:

> Hi New to this forum. If not posted in proper place, someone move it. ...
> Problem: Can not get TinyWebDB connected to Google Cloud Platform
> History: Discovered, invested much time into learning App Inventor 2 (
> http://ai2.appinventor.mit.edu).  Learned to use TinyDB. Learned how to
> replace TinyDB with TInyWebDB using their default cloud storage for
> development. (http://appinvtinywebdb.appspot.com/)... understanding this
> is not for a finished deployed production environment mobile app. Created a
> Google Cloud Platform Account. Then a Project. Everything I find / read
> regarding making the TinyWebDB connect to the Google Cloud Platform
> involves downloading Python 2.7.x  ... and Google Cloud SDK. (most of this
> documentation seems out of date).
> Problem: I only have a Chromebook and these downloads are for Mac, Windows
> or Linux.
> More History: Installed Linux on Chrombook via GIThub.com (Chromium OS
> Universal Chroot Environment https://goo.gl/fd3zc). Downloaded Google
> Cloud SDK and Python 2.7.x.  None of the commands I issue work..inside the
> Crouton Environment. Alway say bad command.
> Questions: Why is it necessary for me to download and configure all this
> stuff just to get a simple URL for the tinywebDB SourceURL? Do I REALLY
> need to go out an buy a windows machine (ugh) just to get this connection?
>
> Thanks for any feedback.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/d2052ec8-7d10-4b15-9001-b37686cfdae7%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/d2052ec8-7d10-4b15-9001-b37686cfdae7%40googlegroups.com?utm_medium=email_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>


-- 
Ani Hatzis
Google Cloud consultant and developer

<https://www.credential.net/ukfxy03s> <https://www.credential.net/b0y6kwie>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ7BwvSr61AcMfc3E%2B6vrGqNg39W26thsEaAKx6guogqWw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Cloud endpoints are running slow in App Engine Standard Environment

[Ani Hatzis]

Hi,

in your app's request_log (under Stackdriver Logging), do the requests 
reflect the high latency in the field protoPayload.latency? Or is the 
latency issue isolated to Endpoints itself (those are logged to 
request_latency_in_ms 
 in 
Endpoints 
logs 
)? 
The latter includes the time taken by Endpoints and the backend (your app) 
together.

And what type of scaling is defined in app.yaml? Maybe you are affected by 
one of the known issues. From the Cloud Endpoints documentation 

 
for standard environment:

Automatic scaling instances do not support background threads, which are 
> used by the Endpoints Management Framework. The Framework will still 
> function, but with potentially reduced performance.


If you use automatic scaling, maybe you could try if the performance 
improves with basic or manual scaling. I have used Endpoints with automatic 
scaling before and haven't noticed such latency issues as in your case 
(Python standard environment though).

Best regards,
Ani
 
On Monday, June 11, 2018 at 12:45:50 PM UTC+2, Azeem Haider wrote:
>
> I'm using App Engine Standard Environment with Google Cloud Endpoints v2 
> for java.
> Using objectify for DataStore.
>
> I just make a simple call but it takes a lot more time to get the results. 
> There are very little users those are using my Endpoints, It's in the 
> testing phase.
>
> EndPoint Function: 
>
> public User getUser(@Named("userId") Long userId){
> return ofy().load().key(Key.create(User.class, userId)).now();
> }
>
>
> It Just a very simple function no authentication call etc. Just Put the 
> userId and get the result. There are only a few users almost 5.
>
> When I call from API Explorer. Same Endpoint function running, again and 
> again. Execute without OAuth
>
> First Call:  time to execute: 8652 ms     May be delay 
> due to start new instance because App Engine shut down instance when there 
> is no traffic
> Second Call:     time to execute: 7625 ms   
> Third Call      time to execute: 4665 ms 
> Fourth Call -  time to execute: 2873 ms
>  
> After this rest of all call are between in 4000 ms - 2000 ms  Only few 
> time again it goes around to 7000 ms.
>
>
> But When I call it from Android using Client Library. It takes too much 
> time to get a result even more than API Explorer.
>
>  Activity Code: ( Using AsyncTask )
>
> doInBackground(Long... params){
> 
> Zeem.Builder builder = new 
> Zeem.Builder(AndroidHttp.newCompatibleTransport(), new JacksonFactory(), 
> null);
> Zeem zeem = builder.build();
>
> zeem.userApi.getUser(5720147234914304).execute(); // 
> 5720147234914304 is userId
> 
> }
>
>
> First time when I run this is it takes more time I think some kind of SSL.
>
> Here is the Log When I first time Run.
>
>
> D/libc-netbsd: [getaddrinfo]: hostname=my-project-id.appspot.com; 
> servname=(null); cache_mode=(null), netid=0; mark=0
>[getaddrinfo]: ai_addrlen=0; ai_canonname=(null); 
> ai_flags=4; ai_family=0
> D/NativeCrypto: ssl=0xae85ba00 NativeCrypto_SSL_do_handshake fd=0xaef31480 
> shc=0xaef31484 timeout_millis=2 client_mode=1 npn=0x0
> doing handshake ++
> D/NativeCrypto: ssl=0xae85ba00 info_callback where=0x10 ret=1
> ssl=0xae85ba00 handshake start in UNKWN  before/connect 
> initialization
> ssl=0xae85ba00 info_callback calling handshakeCompleted
> ssl=0xae85ba00 info_callback completed
> D/NativeCrypto: ssl=0xae85ba00 info_callback where=0x1001 ret=1
> ssl=0xae85ba00 SSL_connect:UNKWN  before/connect 
> initialization
> ssl=0xae85ba00 info_callback ignored
> D/NativeCrypto: ssl=0xae85ba00 info_callback where=0x1001 ret=1
> ssl=0xae85ba00 SSL_connect:23WCHA SSLv2/v3 write client 
> hello A
> ssl=0xae85ba00 info_callback ignored
> D/NativeCrypto: ssl=0xae85ba00 info_callback where=0x1002 ret=-1
> D/NativeCrypto: ssl=0xae85ba00 SSL_connect:error exit in 23RSHA SSLv2/v3 
> read server hello A
> ssl=0xae85ba00 info_callback ignored
> doing handshake -- ret=-1
> ssl=0xae85ba00 NativeCrypto_SSL_do_handshake ret=-1 
> errno=11 sslError=2 timeout_millis=2
> D/NativeCrypto: doing handshake ++
> ssl=0xae85ba00 info_callback where=0x1001 ret=1
> ssl=0xae85ba00 SSL_connect:3RSH_A SSLv3 read server hello A
> ssl=0xae85ba00 info_callback ignored
> D/NativeCrypto: ssl=0xae85ba00 info_callback where=0x1002 ret=-1
> ssl=0xae85ba00 SSL_connect:error exit in 3RSC_A SSLv3 read 
> server certificate A
>  

Re: [google-appengine] Exporting data from google cloud catastrophe to google sheets via api

[Ani Hatzis]

If I understand correctly, you want to export certain parts of your data
from Cloud SQL (MySQL 2nd generation) to Google Sheets, and users should be
able to manipulate the exported data in the sheets. I assume by
"manipulating" you refer to pivot tables, diagrams and so on. Do you want
to stream the data in realtime to Sheets? Or do you want to export the data
occasionally, e.g. manually, or just every few hours or days? Is the data
changed exclusively through your App Engine app or are there clients that
will write directly into your Cloud SQL database? And also, do you want to
export full sets of data, e.g. replacing an old sheet or exporting data to
a new sheet, or would it make sense to append new data to the sheet? Do you
need to re-export updated data into sheets? Do you need a strongly
consistent view of the data in the sheet? How much data is exported into
how many sheets and how often? And how should the access of users be
controlled? Do all Sheets users have the same permissions and do all of
them access the same data? And which programming language of the available
Google App Engine standard environments have you picked?

There are many ways to perform such exports, depending on the answers to
the above questions, and also your preferred programming language and tools.

   - One obvious of course is to manually export SQL data into a CSV file
    or
   similar and import the file into a spreadsheet. Also see Best Practices
   for Importing and Exporting Data
   .
   - If you are looking for an automated process: Inside your App Engine
   app, you could create background tasks that send the data to the Sheets API
   frequently or whenever data is changed or on a certain user request. This
   would consume GAE instance hours and is also subject to some GAE quotas and
   limits, e.g. for outbound requests. Also see Sheets API usage limits
   . The advantage is that
   task queues are integrated with your environment, so you can stick with
   your language and tools and test all pieces together on the local
   development server.
   - Or you could have a Google Apps Script web-app that pulls data
   directly from Cloud SQL and writes them into a spreadsheet. Most Google
   Apps Script quotas and limits are per user account. This route might make
   sense if you are already familiar with Google Apps Script or at least
   JavaScript, and if you want a deeper Google Drive integration.
   - Compared to a Google Apps Script *web-app*, Cloud Functions could be
   the better option, if you also plan to support export targets outside of
   Google Drive.
   - If you want to improve the user experience for Google Drive users: You
   also can integrate a Sheets add-on that users can install, that would pull
   the data from Cloud SQL into their current sheet. The add-on could have a
   sidebar with a form so the users could enter data filters before the data
   is pulled.
   - There are a few more options that come to mind though.

Ani



On Fri, May 25, 2018 at 3:08 PM Richard Bernstein 
wrote:

> Ok I am pretty much going to use the standard implementation rather than
> flexible. I will have a need to export certain parts of my data to google
> sheets to allow users to manipulate data. Is there any existing limitation
> on using standard/mysql/googlesheets api together?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/c1d22698-477d-4765-b80a-9b9e569ff0aa%40googlegroups.com
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ4KH%3DfvOG5MQULeZ%3D6L7iDMYo0XpzL9_sDT%2BhUnxoPn6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: gcloud beta app deploy : This deployment has too many files

[Ani Hatzis]

Although I'm not sure whether this is the right place to discuss the
trouble-shooting, here some things to try. My apologies if I repeat things
you already have tried.

   1. cd into the directory, where the app.yaml file of your application is
   located
   2. With the "." and the debug level (see gcloud Reference
   <https://cloud.google.com/sdk/gcloud/reference/>):
   gcloud beta app deploy . --verbosity=debug
   3. Next the command will prompt you to confirm the deployment and shows
   a few values. In the first lines you should see values for "descriptor" and
   "source". "descriptor" should be the absolute path to the app.yaml file,
   and "source" to the parent directory of the app.yaml file.
   4. The --verbosity=debug flag will also print more information (that
   otherwise is only written to the gcloud log file). It will show every
   file that is skipped and every file that is uploaded during deployment
   (with full paths). After the skipping part, the "Uploading ..." part should
   only mention 24 files or so:
   ╔╗╠═
   Uploading 1 file to Google Cloud Storage ═╣╚INFO: Uploading
   [/home/REDACTED/shipping-backend/app.yaml] to
   [8a185114176aa5cdd7b0e1233915bb2bf03aeb33]
   (Since it isn't my first deployment of your app it uploads just the one
   file that I have changed)

   The supposed 10,000 files would hopefully will show up here or right
   before that, and then you should get a better idea what is causing the
   error. This answer <https://stackoverflow.com/a/42471654/1549523> suggests
   to check the directory for hidden content (e.g. *.git*, *.venv*,
   anything from your IDE). This other answer
   <https://stackoverflow.com/a/42451932/1549523> suggests to check if you
   have (circular) symlinks in that directory.

If you are 100% sure the directory contains only the 42 items that you have
shared with me:

   1. You can evaluate whether you have the same error if you don't use the
   beta command. For avoiding the Endpoints beta feature of a managed rollout
   strategy, we would need to change the app.yaml file as described in
this Troubleshooting
   App Engine Flexible Deployment
   
<https://cloud.google.com/endpoints/docs/openapi/troubleshoot-aeflex-deployment>
   .
   2. Search for the current config_id of your deployed Endpoints API
   service
   gcloud endpoints services describe [PROJECT_ID].appspot.com
   3. Very likely there is one serviceConfig. Copy the value of "id:",
   something like 2018-05-19r0
   4. Add the value as config_id to your app.yaml and change the value of
   rollout_strategy from "managed" to "fixed", so it looks something like
   this:
   endpoints_api_service:
 name: "[PROJECT_ID].appspot.com"
 config_id: "2018-05-19r0"
 rollout_strategy: fixed
   #  rollout_strategy: managed
   5. Run the normal command:
   gcloud app deploy . --verbosity=debug

There is still the probability that your gcloud installation or setup is
"kaput", which you could verify with a new installation and setup as
explained in Before you begin
<https://cloud.google.com/endpoints/docs/openapi/get-started-app-engine#before-you-begin>
in
Cloud Endpoints > OpenAPI > Getting Started with Endpoints on App Engine
Flexible Environment
<https://cloud.google.com/endpoints/docs/openapi/get-started-app-engine>.
If this doesn't help, then it might be a bug, and I would file a bug-report
in the Public Issue Tracker <https://issuetracker.google.com/>.

For any other trouble-shooting, I would recommend to create your own
question on Stack Overflow. GCP support monitors certain tags, such as
*google-app-engine*. Also include the output of gcloud version, and --maybe
redacted for secrets and project IDs-- the output of gcloud info - Don't
forget to a post the link to your question here, too.

Good luck
Ani

On Sat, May 19, 2018 at 10:23 PM Esther Escobodo <escobodoest...@gmail.com>
wrote:

>
>
> yes
>
> On Sat, May 19, 2018, 12:46 PM Esther Escobodo <escobodoest...@gmail.com>
> wrote:
>
>>
>>
>> yes
>>
>> On Sat, May 19, 2018, 12:45 PM Ani Hatzis <anihat...@gmail.com> wrote:
>>
>>> Hey Lourdu Rajan,
>>>
>>> gcloud endpoints services deploy openapi.yaml
>>> Everything fine as expected.
>>>
>>> gcloud beta app deploy
>>> at some point confirmed the upload of 24 files:
>>> ╔╗╠═
>>> Uploading 24 files to Google Cloud Storage ═╣
>>> ╚╝File
>>> upload done.Updating service [default] (this may take several
>>> minutes)...done.
>>>
>>> My understanding

Re: [google-appengine] Re: gcloud beta app deploy : This deployment has too many files

[Ani Hatzis]

Hey Lourdu Rajan,

gcloud endpoints services deploy openapi.yaml
Everything fine as expected.

gcloud beta app deploy
at some point confirmed the upload of 24 files:
╔╗╠═ Uploading
24 files to Google Cloud Storage ═╣
╚╝File upload
done.Updating service [default] (this may take several minutes)...done.

My understanding is that the error occurs before you get this message right?

However, the deploy fails later where it reads the Cloud SQL connection (I
don't have one yet, I'll fix this later).
ERROR: (gcloud.beta.app.deploy) Error Response: [13] Invalid Cloud SQL
name: 

On Sat, May 19, 2018 at 6:50 PM 'Kenworth (Google Cloud Platform)' via
Google App Engine  wrote:

> @Lourdu
>
> You can try GitHub , Bitbucket
> , or similar repository tools. Preferably,
> post publicly only a skeleton app that would reproduce the issue.
>
> You mentioned files are < 250 kb but how many files do you have on your
> application? For example in Linux you can issue the following command to
> count the files on a directory:
>
> ls directory | wc -l
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/f71cfa13-4b96-4cea-b2b5-9fa2b92352a7%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ5iDjwHVNfGTu54iwWp64An2smvA%3Dqy9v76FF2m1bbvdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] gcloud beta app deploy : This deployment has too many files

[Ani Hatzis]

Ok, beta makes sense then and maybe it's really just a bug. If your code is 
public I could try to deploy myself into a temporary project and see if the 
deployment gives me the same error. Have you ever used Cloud Shell 
<https://cloud.google.com/shell/docs/features>? It makes reproducing such 
steps very easy and it comes with pre-installed tools. Just to be sure the 
problem is not related to the local environment, e.g. outdated or broken 
Cloud SDK.

Regards,
Ani


On Thursday, May 17, 2018 at 5:07:42 PM UTC+2, Lourdu rajan wrote:
>
> Hi Ani Hatzis,
>
> Thanks for your reply.
>
> I am trying to publish my code as Endpoint Services in Google Flex 
> Environment. Currently Endpoint backend deploy methodlogy is in beta 
> version. So "gcloud beta app deploy" command used for deploying my code 
> into the Flex environment. 
>
> I also tried to publish the code with specifying the app.yaml file in the 
> gcloud command (i.e. gcloud beta app deploy app.yaml), but still got the 
> same issue.
>
> If you would like, then I may be share the code.
>
> Also implemented below items inthis code,
>
> - Swagger Docs
> - OpenAPI file for Endpoint
> - OAuth Security in OpenAPI file as Security Definition
> - Encode/Decode for Database credentials in app.yaml
> - Swagger Project structure
>
> Do you think any of these item may cause for this issue?, because your 
> sample code from Git 
> <https://github.com/GoogleCloudPlatform/nodejs-docs-samples/tree/master/endpoints/getting-started>works
>  
> fine for me. 
>
> Regards
> Lourdu Rajan G
>
> On Thu, May 17, 2018 at 3:39 AM, Ani Hatzis <anih...@gmail.com 
> > wrote:
>
>> Hi,
>>
>> well, that doesn't sound like 10,000+ files. If it's a bug you could file 
>> a report in the Public Issue Tracker <https://issuetracker.google.com/>. 
>> Is there a particular reason that you use gcloud *beta* app deploy 
>> instead of gcloud app deploy? The first one gives you the commands that 
>> are still in beta, the latter one the commands that are GA. And my last 
>> question: Do you have used gcloud beta app deploy with nothing else? I 
>> believe that if you don't specify the location, e.g. gcloud app deploy 
>> src/app.yaml, the command will try to automatically detect where the app 
>> is, and maybe it gets confused and tries to upload some other path with 
>> 10,000+ files.
>>
>> Good luck,
>> Ani
>>
>> On Wednesday, May 16, 2018 at 9:11:36 PM UTC+2, Lourdu rajan wrote:
>>>
>>> Hi Ani Hatzis,
>>>
>>> Thanks for your reply.
>>>
>>> I don't have any process like file storage or file processing. Just 
>>> trying to deploy my code into the Google app engine flexible environment, 
>>> which 
>>> has basic functionality and it's reference. Like, app.yaml, openapi.yaml, 
>>> app.js, db.js, package.json and node modules (reference). I checked the 
>>> file size of each files, its all just below 250 kB.
>>>
>>> Any other solutions/suggestions please.
>>>
>>> Regards,
>>> Lourdu Rajan G
>>>
>>> On Wed 16 May, 2018, 12:02 AM Ani Hatzis, <anih...@gmail.com> wrote:
>>>
>>>> Does your app deployment indeed include more than 10,000 files? You 
>>>> cannot deploy an app with more than 10,000 files. The deployment is 
>>>> supposed to contain mainly code and configuration files. You probably want 
>>>> to also host a bunch of other files (images, PDFs etc.) and have added 
>>>> them 
>>>> to the deployment, I guess. You would not include them in the deployment. 
>>>> Instead you can host these files in Cloud Storage 
>>>> <https://cloud.google.com/storage/docs/> or Firebase Hosting 
>>>> <https://firebase.google.com/products/hosting/>.
>>>>
>>>> On Tue, May 15, 2018 at 7:01 PM Lourdu rajan <lourd...@gmail.com> 
>>>> wrote:
>>>>
>>>>> I got the below error, when I tried to deploy my Google Cloud Flex 
>>>>> Environment Endpoint project (NodeJs) through gcloud command (gcloud beta 
>>>>> app deploy).
>>>>>
>>>>> ERROR: (gcloud.beta.app.deploy) INVALID_ARGUMENT: This deployment has 
>>>>> too many files. New versions are limited to 1 files for this app.- 
>>>>> '@type': type.googleapis.com/google.rpc.BadRequest fieldViolations:  
>>>>> - description: This deployment has too many files. 
>>>>>
>>>>> -- 
>>>>> You received this message because you are

Re: [google-appengine] gcloud beta app deploy : This deployment has too many files

[Ani Hatzis]

Hi,

well, that doesn't sound like 10,000+ files. If it's a bug you could file a 
report in the Public Issue Tracker <https://issuetracker.google.com/>. Is 
there a particular reason that you use gcloud *beta* app deploy instead of 
gcloud 
app deploy? The first one gives you the commands that are still in beta, 
the latter one the commands that are GA. And my last question: Do you have 
used gcloud beta app deploy with nothing else? I believe that if you don't 
specify the location, e.g. gcloud app deploy src/app.yaml, the command will 
try to automatically detect where the app is, and maybe it gets confused 
and tries to upload some other path with 10,000+ files.

Good luck,
Ani

On Wednesday, May 16, 2018 at 9:11:36 PM UTC+2, Lourdu rajan wrote:
>
> Hi Ani Hatzis,
>
> Thanks for your reply.
>
> I don't have any process like file storage or file processing. Just trying 
> to deploy my code into the Google app engine flexible environment, which 
> has basic functionality and it's reference. Like, app.yaml, openapi.yaml, 
> app.js, db.js, package.json and node modules (reference). I checked the 
> file size of each files, its all just below 250 kB.
>
> Any other solutions/suggestions please.
>
> Regards,
> Lourdu Rajan G
>
> On Wed 16 May, 2018, 12:02 AM Ani Hatzis, <anih...@gmail.com > 
> wrote:
>
>> Does your app deployment indeed include more than 10,000 files? You 
>> cannot deploy an app with more than 10,000 files. The deployment is 
>> supposed to contain mainly code and configuration files. You probably want 
>> to also host a bunch of other files (images, PDFs etc.) and have added them 
>> to the deployment, I guess. You would not include them in the deployment. 
>> Instead you can host these files in Cloud Storage 
>> <https://cloud.google.com/storage/docs/> or Firebase Hosting 
>> <https://firebase.google.com/products/hosting/>.
>>
>> On Tue, May 15, 2018 at 7:01 PM Lourdu rajan <lourd...@gmail.com 
>> > wrote:
>>
>>> I got the below error, when I tried to deploy my Google Cloud Flex 
>>> Environment Endpoint project (NodeJs) through gcloud command (gcloud beta 
>>> app deploy).
>>>
>>> ERROR: (gcloud.beta.app.deploy) INVALID_ARGUMENT: This deployment has 
>>> too many files. New versions are limited to 1 files for this app.- 
>>> '@type': type.googleapis.com/google.rpc.BadRequest fieldViolations:  - 
>>> description: This deployment has too many files. 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Google App Engine" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to google-appengi...@googlegroups.com .
>>> To post to this group, send email to google-a...@googlegroups.com 
>>> .
>>> Visit this group at https://groups.google.com/group/google-appengine.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/google-appengine/83d04d8f-8642-4f28-a55d-4c5ae89b1f08%40googlegroups.com
>>>  
>>> <https://groups.google.com/d/msgid/google-appengine/83d04d8f-8642-4f28-a55d-4c5ae89b1f08%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Google App Engine" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to google-appengi...@googlegroups.com .
>> To post to this group, send email to google-a...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/google-appengine.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/google-appengine/CALdDuQ5KaTRA7f8RS%3D5yo%3DekvqgtZ9dSSrA9rSsKCSpisHZ6CA%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/google-appengine/CALdDuQ5KaTRA7f8RS%3D5yo%3DekvqgtZ9dSSrA9rSsKCSpisHZ6CA%40mail.gmail.com?utm_medium=email_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/add00054-e4c6-47fc-a7e7-129d78323192%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: read that only Python, Java, and Go are supported

[Ani Hatzis]

That was true a few years ago. Today PHP is supported by both, the standard 
environment and the flexible environment 
 of App Engine. Look into the 
comparison of the two PHP environments to pick the best one for your app. 
Generally speaking, while the standard environment is more restrictive than 
the flexible environment, it also needs less effort to manage and operate 
the app. However, you can combine both environments in the same App Engine 
app (project), or even different languages, where you put each into one 
service (module) and deploy them together. For starters, it should be 
easier though to pick just one language and environment and stick with it 
until you feel comfortable with the platform.

On Wednesday, May 16, 2018 at 5:06:08 AM UTC+2, Richard Bernstein wrote:
>
> in Google Cloud App engine. Can I assume that my php app won't be able to 
> run? Or is there a workaround?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7e3cc4ad-e453-4b5c-a30b-bc8d9e9634b5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] gcloud beta app deploy : This deployment has too many files

[Ani Hatzis]

Does your app deployment indeed include more than 10,000 files? You cannot
deploy an app with more than 10,000 files. The deployment is supposed to
contain mainly code and configuration files. You probably want to also host
a bunch of other files (images, PDFs etc.) and have added them to the
deployment, I guess. You would not include them in the deployment. Instead
you can host these files in Cloud Storage
 or Firebase Hosting
.

On Tue, May 15, 2018 at 7:01 PM Lourdu rajan  wrote:

> I got the below error, when I tried to deploy my Google Cloud Flex
> Environment Endpoint project (NodeJs) through gcloud command (gcloud beta
> app deploy).
>
> ERROR: (gcloud.beta.app.deploy) INVALID_ARGUMENT: This deployment has too
> many files. New versions are limited to 1 files for this app.- '@type':
> type.googleapis.com/google.rpc.BadRequest fieldViolations:  -
> description: This deployment has too many files.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/83d04d8f-8642-4f28-a55d-4c5ae89b1f08%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ5KaTRA7f8RS%3D5yo%3DekvqgtZ9dSSrA9rSsKCSpisHZ6CA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: What is the best way to manage env vars in App Engine (flexible environment)?

[Ani Hatzis]

Hi Mike,

I believe you can use Cloud KMS (Key Management Service) to store private 
keys in key rings and then encrypt the secrets before they go into your 
files. Privileged users with the required permissions can use the private 
keys to locally encrypt the secrets and copy the encrypted values manually 
into app.yaml whenever new secrets must be added to the app. At runtime / 
startup the app will authenticate with KMS and get the key to decrypt the 
environment variables. Additional environment variables would store the 
information needed to retrieve the private key (project, keyring, 
key-name). So the source code in the repository and the one deployed to GAE 
production will not contain any plain secrets. And you can have fine-grain 
control who has access to the private keys, separately of the source code.

There is the How-to: Storing secrets 
 that shows how to use 
Cloud KMS to store secrets in Cloud Storage, but the concept should work 
with local app.yaml or other files, too.

If you prefer to use Cloud Storage, I suggest to use a project specific 
bucket (restricted access of course), and put all secrets (encrypted or 
plain) into a path with the version ID of the app, so the app can 
programmatically get the secrets appropriate for the current version. Since 
each version will have its own folder with secrets, e.g. secret-v7-4/, you 
can still roll-back traffic migration safely if something goes wrong with a 
new version in production. If the bucket isn't used for anything else you 
could automate this work-flow by a Cloud Function that is triggered by new 
(or updated) secret folders and then for example could trigger the app to 
load the new variables (sending an HTTP request to the app's task handler 
or so). See this Cloud Storage Tutorial 
 for such a 
background function. Unless of course you already use a CI/CD pipeline 
where you could integrate something like this.

In any case, since the encryption/decryption part is a potential cause for 
failure (e.g. typo in key-ring name, missing secrets files, IAM 
misconfiguration), the tests should cover this, too.


On Monday, May 14, 2018 at 9:23:48 PM UTC+2, Mike Hardy wrote:
>
> Bumping this. We've been storing our env_variables in app.yaml but we're 
> adding more team members and we don't want to expose these variables to 
> everyone. Does anyone have a suggestion for storing secrets in App Engine 
> Flexible?
>
> On Saturday, September 9, 2017 at 7:20:44 AM UTC-7, Alex G wrote:
>>
>> Moving our apps from Heroku to App Engine has been great, but there are a 
>> few things that we miss. In particular, having a way to define config 
>> variables  in Cloud 
>> Console would be great. We have tried a couple of approaches but none are 
>> optimal:
>>
>>- Place config within env_variables 
>>
>> 
>>  
>>in the app.yaml file. Not ideal to store secrets since anyone deploying 
>> the 
>>app has access to the app.yaml and those secrets.
>>- Place config in Cloud Storage. Hard to edit, prone to errors (e.g. 
>>breaking the JSON file that might contain them) and it doesn't trigger an 
>>app restart with the updated config (I believe Heroku restarts the app 
>>every time you change a config variable).
>>
>> Are there any ideas on implementing a similar feature in App Engine?
>> Is there a better way of currently handling those config vars?
>>
>> Thanks
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/31617464-4301-46a7-960e-d98574d9d7a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Transaction across Java appengine datastore and cloud sql

[Ani Hatzis]

It seems your code-base has grown into a quite huge monolithic app. If you 
haven't done yet, you might look into this: Microservices Architecture on 
Google App Engine 
.
 
You could expand your app by putting new models / handlers into new apps 
using this pattern, and eventually also use a different storage product for 
them. It also depends on your existing app and the roadmap, how well you 
can separate the new parts without refactoring the old app.

You cannot do a transaction across Cloud Datastore and Cloud SQL or Cloud 
Spanner.

You also should keep in mind that every product comes with different 
advantages and disadvantages and you need to figure out which trade-offs 
make most sense for your technical and business requirements. Considering a 
mature app that has to cover many different use-cases, maybe you will pick 
more than one storage product to meet all of them.

And although I don't know if that's the case here, worth to mention: I 
often have seen a (composite) index used in Datastore only for user-facing 
searches where Search API would have made much more sense. Refactoring such 
cases can be quite easy if your app is the only client writing into its GCD.

Good luck!

On Sunday, May 13, 2018 at 9:45:56 AM UTC+2, Rajesh Gupta wrote:
>
> Hi,
> We are running out of indexes on appengine datastore.  We have 10-20 
> remaining.
> No time for doing refactoring and to regain the indexes. 
>
> One thought is to use Cloud sql or Cloud spanner for further development.  
> There are other reasons also to look for alternatives to datastore. 
> (finding talent, restrictions of datastore on aggregates queries, no easy 
> way to do migrations or schema changes)
>
> Is there a way to do a transaction across datastore and cloud sql or cloud 
> spanner
>
> What are the recommended methods to further expand dev.
>
> Rajesh
> *www.GainERP.com *
> *ERP & Field Service on Google Cloud*
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/4bce3e58-b220-4783-8971-7d2f821b947f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: security of web service with Google App Engine + Cloud Endpoints

[Ani Hatzis]

Yes, Deniz, I believe so, but just to be sure we are talking about the same 
"user ID" or more precise, how it is extracted: For Cloud Endpoints 
handlers in App Engine the implementation can be really simple and safe. If 
you implement the handler in Node.js (flexible environment) your handler 
reads the request's HTTP header X-Endpoint-API-UserInfo 
<https://cloud.google.com/endpoints/docs/openapi/authenticating-users#receiving_auth_results_in_your_api>
 to 
retrieve current user information as JSON object, including the user ID. I 
assume that's what you intent to implement and you can rely on the user 
information retrieved from this header (see earlier reply by Katayoon). If 
you intent to use the Authorization header instead, read my PS below. You 
might want your implementation to also check if the hosted domain is the 
same as a particular G Suite domain though, or apply whatever custom access 
control you need additionally (roles, custom claims etc.).

Other than that, as always keep tight control of user access to your GCP 
resources (via IAM in Cloud Console) and an eye on Audit Log 
<https://cloud.google.com/logging/docs/audit/>, and you should be safe.

Ani

PS: Theoretically, your handler could (instead of X-Endpoint-API-UserInfo) 
access the original HTTP Authorization header of the request (if I remember 
correctly, Cloud Endpoints ESP/OpenAPI makes it available to the handler) 
to validate the ID token and extract user information. But safely 
validating a JSON Web Token (JWT) is not trivial 
<https://cloud.google.com/endpoints/docs/openapi/troubleshoot-jwt-openapi>, 
hence that anyone could send a fake ID token to your endpoint. Therefor I 
assume that's not what you plan to do. Handlers outside of Cloud Endpoints 
or even outside of App Engine might need to (partially) implement their own 
token validation, but I would avoid this wherever I can. For everyone who 
is interested: a very general platform-agnostic documentation is provided 
by Google Identity Platform: OpenID Connect - Validating an ID Token 
<https://developers.google.com/identity/protocols/OpenIDConnect#validatinganidtoken>
.


On Tuesday, May 1, 2018 at 6:38:20 PM UTC+2, Deniz Yazar wrote:
>
> Hi Ani,
>
> I already have firebase database rules but I am only worried about the 
> security of Google App Engine and Cloud Endpoints side. I am wondering is 
> it really this simple to achieve security. I get the ID token of the user 
> in my API running on google app engine. Than I use admin privileges to get 
> info of that user ID I extracted from the token. I need to make sure that 
> userID cannot be manipulated in the token. Can I be sure that the id I 
> extract from the token is safe?
>
> On Monday, April 30, 2018 at 8:46:02 AM UTC-4, Ani Hatzis wrote:
>>
>> Hi Deniz,
>>  
>>
>>> allow users to get information about their account
>>>
>>
>> Is the only data shown to the user the information that has been 
>> extracted from the current user's ID token?
>>
>> Or do you store additional custom user information that your endpoints 
>> will show to the current user, e.g. their user profile? If you have more 
>> user-related data in Firebase than what is already managed by Firebase Auth 
>> (in Firebase Users), you might (as usual) need rules in FB to prevent 
>> reading/writing other users' custom data in Realtime Database (or Cloud 
>> Firestore). In Firebase, all authenticated users can read/write the entire 
>> database by default, unless you setup rules to restrict access. And the 
>> access works outside of your App Engine app, so you would need the FB rules 
>> as additional protection when clients access FB directly.
>>
>> Ani
>>  
>>
>> On Sunday, April 29, 2018 at 6:19:05 PM UTC+2, Deniz Yazar wrote:
>>>
>>> I am developing a web service(server) for my web application to allow 
>>> users to get information about their account. After looking at every 
>>> amazing Google Cloud product, I am thinking of doing the following to have 
>>> a secure server on Google Cloud:
>>>
>>>1. Having clients login with Firebase Auth on web app
>>>2. Add security specifications in Cloud Endpoints to allow only 
>>>firebase logged in users to access* specific paths (*running on a 
>>>node.js server on Google App Engine)
>>>3. Extract the ID token received on the server by decoding it to 
>>>return users only their data
>>>   - Is simply having firebase security rules in the openAPI file in 
>>>   Cloud Endpoints enough to make sure only users logged in can access 
>>> their 
>>>   *own* data?
>>>
>>> I'm also limiting acc

[google-appengine] Re: sending email at a later date

[Ani Hatzis]

In App Engine, you can use tasks in Push Queues 
. 
When you create the task you provide a task option 
,
 
either countdownMillis or etaMillis, that specify the absolute earliest 
time at which the task should run. For pull tasks, no worker can lease a 
task before the time has arrived. The pattern would be to create one push 
task for each notification to be sent and define the countdown or eta 
accordingly, and the payload for email address and related data, and/or one 
for SMS and related data. Once the time has passed, task queue will direct 
the task to your task handler. The task handler will then look into the 
payload and accordingly send notifications using other services. However, 
be aware that you need appropriate task queue configuration for your kind 
of app and work-load, including sudden bursts, and also how to handle 
errors (with-out spamming users in case of errors). There is not guarantee 
that the task will be executed exactly at the specified time (for example, 
if too many tasks are executed for this queue or for the total number of 
instances allowed for the app), execution might happen later. But there is 
a guarantee that the task will run at least once. However, it would make 
sense to track each task back to a certain event or reason in your app. So, 
if a scheduled appointment is changed, the task should be deleted / 
replaced accordingly.

A different approach would be to define pending notifications as Datastore 
entities and trigger their handling by scheduled tasks 
 (cron 
jobs). For example, every 2 minutes (more or less), a cron-job task is 
executed. The cron-job task itself just creates a push task to execute 
immediately. That push task handler will query your Datastore for 
"PendingNotification" entities that have an indexed property "runAfter" 
(datetime), filtered and sorted by this property, where the timestamp is 
already in the past. For each entity in the query, the task handler will 
send the notification accordingly and then delete the entity (so it doesn't 
come up at the next query). That task could do this in batches, for example 
for 10 notifications per execution. If the query would give more results, 
enqueue the same task again. Else just return 200 and quit. Most likely, 
you are already familiar with such patterns if you do batch migrations in 
administrative tasks. If at some point an invoice due date or a scheduled 
service appointment is changed, your app picks the corresponding 
PendingNotification entity and deletes or updates it accordingly.

The key advantages of task queues for apps in App Engine with task queue 
support:

   1. they support namespaces (you probably already use namespaces 
   

 
   for a multi-tenant SaaS app
   2. you can test push queues in the local development server 
   

   
If you want a similar approach but moving the actual email or SMS sending 
workload out of the App Engine task handlers, you might want to look into Cloud 
Pub/Sub , a server-less event-driven 
message system. Although App Engine has a Task Queue REST API, it's 
probably easier in Pub/Sub to integrate with many other clients. Basically, 
clients (like a Cloud Function that will send emails or SMS), subscribes to 
a topic ("notifications"). Once your task handler publishes a message (e.g. 
"send invoice #123 now to u...@example.com"), the client gets the message 
and does its job. So, instead of your final task handler connecting to a 
mail provider or SMS provider, it just publishes a message to a Pub/Sub 
topic and is done, and other parts of your system pick that message and 
continue from there. This combination would reduce instance hours in your 
App Engine app, but then add other costs, e.g. for Pub/Sub messaging, and 
also for Cloud Functions or whatever you want to use to actually send the 
notifications. You probably also need to put a little more thought into 
testing and deployments.

Regards,
Ani

On Friday, May 4, 2018 at 5:11:58 AM UTC+2, Rajesh Gupta wrote:
>
> HI,
>
> We are in erp & field service management SAAS
>
> We use java on appengine.
>
> We need email reminders that needs to be sent our later dates.
>
> For example, for the invoice due date, a reminder has to be sent 1 day 
> early
>
> In field service SAAS, we need to send reminders as SMS 30 min, prior the 
> appointment given to the customer.
>
> What cloud components can be used here  or what should be the design for 
> performing the events at later date   
>
>
> -- 
> Regards,
> Rajesh
> 

[google-appengine] Re: security of web service with Google App Engine + Cloud Endpoints

[Ani Hatzis]

Hi Deniz,
 

> allow users to get information about their account
>

Is the only data shown to the user the information that has been extracted 
from the current user's ID token?

Or do you store additional custom user information that your endpoints will 
show to the current user, e.g. their user profile? If you have more 
user-related data in Firebase than what is already managed by Firebase Auth 
(in Firebase Users), you might (as usual) need rules in FB to prevent 
reading/writing other users' custom data in Realtime Database (or Cloud 
Firestore). In Firebase, all authenticated users can read/write the entire 
database by default, unless you setup rules to restrict access. And the 
access works outside of your App Engine app, so you would need the FB rules 
as additional protection when clients access FB directly.

Ani
 

On Sunday, April 29, 2018 at 6:19:05 PM UTC+2, Deniz Yazar wrote:
>
> I am developing a web service(server) for my web application to allow 
> users to get information about their account. After looking at every 
> amazing Google Cloud product, I am thinking of doing the following to have 
> a secure server on Google Cloud:
>
>1. Having clients login with Firebase Auth on web app
>2. Add security specifications in Cloud Endpoints to allow only 
>firebase logged in users to access* specific paths (*running on a 
>node.js server on Google App Engine)
>3. Extract the ID token received on the server by decoding it to 
>return users only their data
>   - Is simply having firebase security rules in the openAPI file in 
>   Cloud Endpoints enough to make sure only users logged in can access 
> their 
>   *own* data?
>
> I'm also limiting access to some paths for service accounts, after making 
> that security rule in Cloud Endpoints, isn't there really anything else to 
> do to make sure the system is secure. Is it really this easy to achieve a 
> secure system? After this point, is it just my google account's security 
> thats left to protect?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b263aa45-fa93-4624-87db-766333bd991d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: GAE Intermittently showing '502 Server Error'

[Ani Hatzis]

Hey Suvodeep, I understand the frustration, but from my point of view, I
prefer to pay just for the resources I use and according to the SLA. I
wouldn't want to indirectly pay for free premium support of other customers
(which probably would increase all prices significantly). Anyway, wish you
the best.

On 25 March 2018 at 05:11, Suvodeep Pyne  wrote:

> Hey Jordan
>
> Thanks for that info. Now, take a look at this.
>
>
> 
>
> This is the second time in the past 4 months we faced a P1 issue which
> took 2 days to get resolved. The LB was throwing 502s intermittently while
> the app server was totally healthy. You can check my account for logs. And
> btw, there were others who were complaining about the same issue
> .
>
> Since as you mentioned that this is a forum for general product
> discussions; I want you to note that this is perhaps not the best customer
> experience. I am aware of your support tiers but there should be a better
> interaction model with your customer. We are paying for the service and you
> are answerable to us in case of outages/service disruption.
>
> Regards
>
>
> On Sunday, March 25, 2018 at 4:17:23 AM UTC+5:30, Jordan (Cloud Platform
> Support) wrote:
>>
>> This occurs when your actual application becomes too busy to respond to
>> the nginx server Docker container that sits in front of your application
>> Docker container within the instance. Nginx will send health checks
>> 
>> to your application in order to make sure it is responsive so that it may
>> accept new requests. If your code blocks the main thread for too long,
>> nginx will not receive any response from your app and will assume it is not
>> healthy. This will produce a 502 Bad Gateway error when App Engine asks
>> nginx if the application is ready to accept new requests, and the instance
>> will be restarted in order to make it healthy again.
>>
>> It is therefore recommended to increase the number of instances you have
>> (as I assume you are limiting your max to a very low number like one), and
>> to ensure your code is properly configured to handle concurrent requests,
>> returns quickly, and does not block the main thread for too long. You can
>> also configure the health checks
>> 
>> as to not render your instance as unhealthy so quickly and to give it more
>> time to recover from traffic spikes.
>>
>> - Note that Google Groups is reserved for general product discussions and
>> not for technical support. For further technical support it is recommended
>> to post your detailed questions
>> to Stack Overflow
>>  using the
>> supported Cloud tags.
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/e6ba838b-5def-4a14-8243-
> dda691f4fa64%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ56n%3DFeTNnDGjeC%2BbOoLWUJ-gBNXVA3UaGqmXVmRF8A%2Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Why Domain is Loading very slowly but when opening directly through App Engine Url it is opening very fastly?

[Ani Hatzis]

Is your app hosted in the asia-northeast1 or australia-southeast1 regions?
In this case you might be affected by this issue: 64458939 Custom Domains
latency impact . Star the
issue to get notified of any changes.

On 23 March 2018 at 23:34, Sudhanshu Gaur  wrote:

> I have a Domain which i bought it from GoDaddy.com and i have linked it
> to Google App Engine Flexible(NodeJs) Environment.
>
> Now I when I am opening my Domain then it is sometimes taking more than 5
> secs to load and sometimes I am getting Error
>
> Server Error - 502
>
> But the thing is when i am directly opening my app engine url which is
> https://.appspot.com/ then it is always opening without status
> code 502 and also very fast.
>
> Can anyone please tell me why this is happening.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/0350e6b2-6a38-48d6-a1fc-
> 21201080e37a%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ6VYr7M6K1SVfb5RsTc145xZvWdUPqAEeCHU4FoBpUM9A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How does App Engine affects the cost of using Cloud Datastore?

[Ani Hatzis]

Hello George,

is it still necessary to *create/deploy/enable* an app in order to use a 
new datastore? To validate, I have created a new (empty) project 
.
 
In Cloud Console under *Cloud Datastore > Entities* , after selecting a 
Datastore region, I was able to create an entity. I also could read this 
entity via REST API (API Explorer for projects.lookup 
).
 
I didn't create an app. And there is also no hint or warning in Console 
that prompts me to create an app. Under *App Engine > Dashboard* I'm 
offered the "Your first app" flow.

Maybe the quote relates to features that are not (or were not) available 
without an app. Not sure, if composite index could be managed through the 
API or gcloud or by other means than deploying the index.yaml (or 
equivalent). And the old import & export worked only with an App Engine app 
while the new one doesn't seem to use the task queues and app instances.


On Thursday, 22 March 2018 18:57:50 UTC+1, George (Cloud Platform Support) 
wrote:
>
> Hello Yihang, 
>
> The App Engine application incurs costs when running. Billing begins when 
> an instance starts and ends fifteen minutes after a manual instance shuts 
> down or fifteen minutes after a basic instance has finished processing its 
> last request. You may find more detail on the "App Engine Pricing" page 
> . In other words, this 
> mandatory app would not cost you, if your main application runs on a GCE 
> instance. 
>
> Datastore usage costs are calculated in terms of stored data, entity 
> reads, entity writes, entity deletes, and small operations. None of these 
> involve the active App Engine application, if the only use you make of the 
> Datastore is from your GCE instance. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/edb130d5-2a87-4de5-8bab-e9a6466b63c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Is it still possible to install Google App Engine's "Short Links" tool?

[Ani Hatzis]

Are you looking for this? https://developers.google.com/url-shortener/

On 11 March 2018 at 00:50, Gary McCormick  wrote:

>
> This is such an important tool, but I can't seem to find any documentation
> on how to get it up and running at my new company!
>
> Any thoughts or ideas are appreciated!
>
> Added some screenshots borrowed from the web to illustrate which tool I'm
> referring to.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/adbb6663-590c-4bed-b359-
> 84addd74d2cc%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ47_ZGaqVZLJGAeeBdYsH0mrYUKQ0yP_S4%3DQ-PeCakAcA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Google Compute Engine (managed) Vs Google App Engine Flex

[Ani Hatzis]

Hello Vi,

in case you haven't read this page yet, Choosing a Computing Option
 shows a good
comparison. App Engine Flex and Compute Engine have some overlapping areas
where either of them would be a reasonable choice, but I would think that
if you put a weighted requirements check-list beside both of them you
probably will strongly favor one over the other. The quotas/limits &
pricing pages should also be of interest. I would recommend to go through
some of their quickstarts very early, or use the interactive tutorial in
Cloud Console  / Cloud Shell (no local setup needed at this point).


On 9 March 2018 at 23:49, Vi  wrote:

> Hi all
>
> Which is the difference between Compute Engine managed VMs and App Engine
> Flex?
> How make a choice between these two solutions?
>
> Thanks in advance :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/0d9dc93a-34cf-4b9d-912c-
> 247e5f91eec4%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ7VyyyoHM%2BJAMEbRDvNC-yTQiSuRiZV605tGnvJU%3D0b2g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Unable to update app engine app

[Ani Hatzis]

Well, no, I had no such issue, but that doesn't mean anything. You could 
file a bug report on their public issue tracker 
<https://issuetracker.google.com/issues/new?component=187272=1010421>
.

All the best


On Thursday, 8 March 2018 22:53:48 UTC+1, Nezare Chafni wrote:
>
> I've been using the new SDK for 6 months, nothing special, the problem 
> started with the switch is what I'm saying. Everything succeeds, no errors, 
> my code just never updates.
>
> app.yaml is configured correctly, again, i've been deploying this app with 
> the new sdk. the configuration is correct.
>
> This is a bug on app engine's side, the update goes through successfully 
> the code just never updates. I've tried everything.
>
>
>
> On Thursday, March 8, 2018 at 1:36:48 PM UTC-8, Ani Hatzis wrote:
>>
>> When I used App Engine SDK (appcfg) it could happen that a deployment 
>> failed, so I repeated the deployment a few minutes later. The same is with 
>> Cloud SDK (cloud app deploy). Nothing noteworthy. The new SDK has a little 
>> learning curve, but I believe it is far superior to the App Engine SDK.
>>
>> However, you haven't actually explained what you tried and what exactly 
>> fails. Do you get a time-out, an error message or something? Or what 
>> environment do you use? Does your app have multiple services or just the 
>> default service? Has it worked before with cloud app deploy or is it 
>> your first attempt to deploy with the new SDK? What are your exact steps?
>>
>> Since you asked for ideas: Two things that I first stumbled over as a 
>> long-time appcfg user:
>>
>>1. When you have never deployed before with the new SDK: Beside a 
>>correct gcloud initialization, login and configuration, you would also 
>> need 
>>to remove the version and the app ID from the app.yaml file. The command 
>>will complain about these two if present, but possible to miss the error 
>> I 
>>guess.
>>2. If you already have used the new SDK to deploy successfully: In 
>>gcloud, have you switched to the configuration that has the correct 
>> project 
>>ID and account before you deploy? (gcloud config configurations list 
>>will show you the active configuration and properties)
>>
>> If your problem persists, I suggest to ask on Stack Overflow with more 
>> details though, unless of course it turns out to be an issue specific to 
>> your account or project.
>>
>> Good luck!
>> Ani
>>
>>
>> On Thursday, 8 March 2018 21:25:04 UTC+1, Nezare Chafni wrote:
>>>
>>> I can't update my app engine app This happened several times 
>>> since moving from the appcfg to the new SDK, and its insanely annoying. I 
>>> tried everything. even sanity check debug statements. It's annoying enough 
>>> to not make me wanna use app engine again.
>>>
>>> I tried new versions, overwriting versions, skipping files, removing 
>>> pyc, almost everything I can think!
>>>
>>> Any ideas? Is this happening to anyone else?!
>>>
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/d42e6d90-c66d-494c-9c7f-bc1c80da8c63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Unable to update app engine app

[Ani Hatzis]

When I used App Engine SDK (appcfg) it could happen that a deployment 
failed, so I repeated the deployment a few minutes later. The same is with 
Cloud SDK (cloud app deploy). Nothing noteworthy. The new SDK has a little 
learning curve, but I believe it is far superior to the App Engine SDK.

However, you haven't actually explained what you tried and what exactly 
fails. Do you get a time-out, an error message or something? Or what 
environment do you use? Does your app have multiple services or just the 
default service? Has it worked before with cloud app deploy or is it your 
first attempt to deploy with the new SDK? What are your exact steps?

Since you asked for ideas: Two things that I first stumbled over as a 
long-time appcfg user:

   1. When you have never deployed before with the new SDK: Beside a 
   correct gcloud initialization, login and configuration, you would also need 
   to remove the version and the app ID from the app.yaml file. The command 
   will complain about these two if present, but possible to miss the error I 
   guess.
   2. If you already have used the new SDK to deploy successfully: In 
   gcloud, have you switched to the configuration that has the correct project 
   ID and account before you deploy? (gcloud config configurations list 
   will show you the active configuration and properties)

If your problem persists, I suggest to ask on Stack Overflow with more 
details though, unless of course it turns out to be an issue specific to 
your account or project.

Good luck!
Ani


On Thursday, 8 March 2018 21:25:04 UTC+1, Nezare Chafni wrote:
>
> I can't update my app engine app This happened several times since 
> moving from the appcfg to the new SDK, and its insanely annoying. I tried 
> everything. even sanity check debug statements. It's annoying enough to not 
> make me wanna use app engine again.
>
> I tried new versions, overwriting versions, skipping files, removing pyc, 
> almost everything I can think!
>
> Any ideas? Is this happening to anyone else?!
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/60e1e57e-02ac-466e-b1a3-01a5c7a775f7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: gcloud datastore export

[Ani Hatzis]

You could load the Cloud Datastore export files that you have stored in 
Cloud Storage directly into BigQuery .

You can do this manually or automate it. Yesterday I learned about this shell 
script  that someone has done.

Once the data is in BigQuery, you can not only do the usual queries or Data 
Studio reporting etc., but also export the data, see Exporting Table Data 
 (web 
user-interface, command-line, or API).


On Tuesday, 6 March 2018 10:47:07 UTC+1, alex wrote:
>
> Is there any way how to convert files generated by 
>
>gcloud datastore export
>
> to human readable format (fx csv or json)?
>
> I tried -format, but it is used for formatting command response 
> (logically) 
>
> tx, 
> alex
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/0c0f3fef-330f-4564-921b-6e1af073fbc3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: How to upgrade appengine launcher from 1.9.18

[Ani Hatzis]

The link <https://developers.google.com/accounts/docs/AuthForInstalledApps>
you have provided points to an authentication method that was deprecated in
2012. I also believe that the *--email* and *--oauth2* options are not
supported by the update command (anymore?).

Unfortunately, I have used appcfg.py with the same account for all
projects, so I'm not sure about the best way. Maybe someone on this list
still uses appcfg.py with the same setup as you. If I remember correctly,
running appcfg.py for the first time would prompt you to login through a
Google login page in the browser. The auth token then would be stored in
your home directory and automatically be used for any future update command.

This is the appcfg.py doc
<https://cloud.google.com/appengine/docs/standard/python/tools/appcfg-arguments>.
Look into the Global options
<https://cloud.google.com/appengine/docs/standard/python/tools/appcfg-arguments#global_options>,
*--oauth2_refresh_token=token* As far as I understand, with this option you
can use the token of a different Google account for the current command
execution. The alternative would be to remove / replace the auth token that
was previously stored in your home directory and then running again through
the auth / login.

I actually recommend you to look into *gcloud app deploy* doc
<https://cloud.google.com/sdk/gcloud/reference/app/deploy> and eventually
switch to gcloud. It is much easier to work on different projects with
different accounts, since you can store multiple configurations and just
switch between the configurations before running a command like gcloud app
deploy. You'll have to learn a little bit about gcloud and you need to make
small tweaks in app.yaml before you can deploy them (it will tell you which
one in the errors). But I guess, the time would be spent well.

On 2 March 2018 at 00:23, Kathleen Juliff <mss.ka...@gmail.com> wrote:

> Thank you Ani,
>
> Your reply makes sense. However when I use the command line I get an error.
> Error 404: --- begin server output ---
> https://developers.google.com/accounts/docs/AuthForInstalledApps
>
>  I looked at old posts on this group and have temporarily solved the
> problem using the oauth2 option - as in
> appcfg.py --application= --email= --oauth2 update
> PROJECT_DIR
>
> But I think I must be doing something wrong. Can you point me in the right
> direction?
>
> Thank you.
>
> Kathleen
>
> On Wednesday, February 28, 2018 at 5:20:06 AM UTC-5, Ani Hatzis wrote:
>>
>> Hi Kathleen,
>>
>> on which OS do you run the App Engine launcher?
>>
>> 1.9.18 is very old, maybe from around August 2015. I would suspect that
>> you have an old App Engine SDK 1.9.18 installed (that also came with the
>> GUI launcher), and later you have added the Google Cloud SDK
>> <https://cloud.google.com/sdk/> (a different SDK, that works for most of
>> the GCP products and is much more sophisticated regarding configurations,
>> automation etc.). The Google Cloud SDK has no GUI launcher for App Engine
>> though (and deployments use a "gcloud app deploy" instead of the old
>> command), but the local dev-server and the other GAE parts are the same.
>> gcloud's App Engine component is on 1.9.67 which is right now the most
>> recent version of App Engine.
>>
>> I don't know if you could make the GUI launcher to use the newer
>> dev_appserver.py from Google Cloud SDK instead. Since the GUI launcher
>> of App Engine SDK does merely more than starting and stopping
>> dev_appserver.py this should be easy. I assume you already added confg
>> and auth to gcloud, so you only need to get familiar with the new command
>> line options of gcloud for this and for App Engine deployments.
>>
>> Regards,
>> Ani
>>
>> On Wednesday, 28 February 2018 05:27:56 UTC+1, Kathleen Juliff wrote:
>>>
>>> I tied uninstalling and reinstalling but no difference.
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/google-appengine/f97844c8-1359-42df-a2f0-
> 2387c9e5d0d5%40googlegroups.com
> <https://groups.google.com/d/msgid/google-appengine/f97844c8-1359-42df-a2f0-2387c9e5d0d5%40googlegroups.com?utm_medium=email_source=footer>
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



--

-- 
You received this me

[google-appengine] Re: How to upgrade appengine launcher from 1.9.18

[Ani Hatzis]

Hi Kathleen,

on which OS do you run the App Engine launcher?

1.9.18 is very old, maybe from around August 2015. I would suspect that you 
have an old App Engine SDK 1.9.18 installed (that also came with the GUI 
launcher), and later you have added the Google Cloud SDK 
 (a different SDK, that works for most of 
the GCP products and is much more sophisticated regarding configurations, 
automation etc.). The Google Cloud SDK has no GUI launcher for App Engine 
though (and deployments use a "gcloud app deploy" instead of the old 
command), but the local dev-server and the other GAE parts are the same. 
gcloud's App Engine component is on 1.9.67 which is right now the most 
recent version of App Engine.

I don't know if you could make the GUI launcher to use the newer 
dev_appserver.py from Google Cloud SDK instead. Since the GUI launcher of 
App Engine SDK does merely more than starting and stopping dev_appserver.py 
this 
should be easy. I assume you already added confg and auth to gcloud, so you 
only need to get familiar with the new command line options of gcloud for 
this and for App Engine deployments.

Regards,
Ani

On Wednesday, 28 February 2018 05:27:56 UTC+1, Kathleen Juliff wrote:
>
> I tied uninstalling and reinstalling but no difference.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/55ec407e-5c34-418a-9374-bdb4ea96ea24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Architecture advice for Google Cloud Platform website solution

[Ani Hatzis]

Hello John,

this is a great post. Don't be too concerned about the "rusty" bit or first 
cloud project. There are many new things to learn, typically things that 
will make your life easier, but you will find a lot of your experience 
useful, too.

I have tried to read a fair bit about GPC but find there is not too much 
> out there which shows how a whole solution would be put together - it is 
> mainly individual bits.


It's the same as with non-cloud solutions, the "whole solution" is tailored 
to a specific use-case that probably isn't too useful for others.

As far as I can tell from your description you can put many different 
individual bits together to get your solution. It is much about your 
preferences. Of course, often one product is more suitable for a certain 
task than the other (technically or financially), but your choices seem to 
be reasonable.

As someone else already suggested, you should at least follow through with 
one of the tutorials that are available for GAE apps and maybe also a small 
Google Apps Script project (e.g. one that's bound to a Google Sheet).

For a quick walkthrough of GCP, I recommend to create a Google App Engine 
project, standard environment because those are very much 
"batteries-included" with little to no administration efforts, you have a 
free tier that's great for getting your feet wet, and as you mentioned it 
doesn't cost you if it isn't used for weeks or months. In Cloud Console you 
are offered interactive tutorials after creation that you can directly 
follow through in Cloud Shell, so you don't even need to setup a local 
workspace. Of course, Container Engine and Compute Engine are good compute 
options, too, they give you a lot more flexibility (beyond your 
requirements) but they also come with a lot more administrative overhead. 
>From what I understand your post the trade-off wouldn't be worth it.

The reasoning behind my Google Apps Script suggestion, although it isn't a 
GCP product: you mentioned that your users already have Google accounts / 
Google Drive, you know JavaScript, your usage projections should be fine 
within their per-account limits, its server-less, and you normally pay by 
account per month. The other reason is that not only Google Forms could be 
used, but you also could build the reports in Google Docs, convert to PDF 
and send them per mail. Such things can be implemented rather easily in 
Apps Script, as long as you target Google users. Of course you can combine 
GCP products with G Suite products / Google Apps Script. If you want to 
scale significantly and extend the focus outside of Google Drive users 
however, you might be better off with adding GCP product to the front-end 
part of your architecture and not only relying on G Suite / Apps Script.

Regarding GCP, there are a few pages you probably have already read:

   - Choosing a Computing Option 
   
  - Choosing an App Engine Environment 
  
   - Choosing a Storage Option 

Regarding the products you have picked, your choices are very reasonable. 
Some notes though:


>- Cloud SQL - mqsql 2nd generation (have never used nosql database 
>before - which would be better for this kind of app?) Can this go down to 
>zero instances when not in use?
>
> In Cloud Datastore you don't have instances and I believe that the pricing 
could be more suitable to your needs. Also if you want admin users to 
modify questionaries on their own and you want to store the answers in a 
database (not just in Google Sheets), then you might be better off with 
Datastore. And Datastore supports namespaces for multi-tenancy apps. Given 
a good design (especially regarding small entity-groups, no joins, no 
monotonically increasing IDs) Datastore can scale and perform incredibly 
well. If you consider GAE Python Standard, Datastore is a perfect match 
because of the awesome NDB library which is only available in Python 
Standard. On the other hand, you already have SQL experience. And your 
solution probably will never get close to hitting the Cloud SQL limits 
anyway.


>- Cloud IAM to manage users and what they can do?
>
> Just to clarify... Cloud IAM is not a solution to manage users on 
application level, but for controlling access to your GCP resources. For 
example, you can manage which user can create new projects, change billing 
accounts, access a bucket in Cloud Storage, deploy a GAE app, etc. But you 
would need your own (or 3rd-party) solution to define and control access 
like "Patient X must not access answers provided by patient Y" or 
"Practitioner A must not access data of a patient unless certain 
requirements are met".
 

>
>- Not sure what I would need to get a domain name and SSL up and 
>running?
>
> You can use custom domains with some of the products, the process is 

Re: [google-appengine] Image Detection - I want it to look in my library (bucket) only and identify based on images in my own bucket

[Ani Hatzis]

Yes, the Google Vision API can be integrated with your Google Cloud Storage
bucket (it's also mentioned under "Integrated REST API" in the Product
Overview ).

There is a How-to guide
 that
explains that you can provide an image with the request by

   - uploading the image (as a base64-encoded string)
   - passing the Cloud Storage URI of an image in a bucket
   - or passing a publicly-accessible HTTP/HTTPS URL of an image somewhere
   on the WWW

You can use many different clients to send the request to the Vision API,
e.g. a Python app that is hosted on App Engine could send the request to
the API and store the response in Cloud Datastore (one entity for each
image in the bucket). Since you want to detect only images that are stored
in a bucket, you could also use a Cloud Function
 (JavaScript) that is automatically
triggered every time a new image is created in a bucket (tutorial with code
sample here ).



On Sun, 18 Feb 2018 at 18:13 Dzaner Bojic  wrote:

>
> Can Google Vision API do the following?
>
> I want to upload my own images as the library and assign a value for what
> each one is.
>
> Then when I use the Image Detection, I want it to look in my library
> (bucket) only and identify based on images in my own bucket.  That is it.
>
> Seems like it is using the whole WWW and returning all images via label
> detection, but I want to just use my own bucket.
>
> If you know how, please let me know!
>
> Please help!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/40f92121-11f3-4b16-80dc-05abdd4eb7ff%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ7Xwe8%2BKL%3DkD7Yj-6xe%2BKGR5x_YvmNCNvEFvFYJ7FT8JQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Google App Engine Cost

[Ani Hatzis]

Hi Terry!

Yes, it seems your thinking about this in the correct.

I would like to point to a very different solution though. If you also have
(basic) JavaScript or Google Apps Script experience, you could check out App
Maker  which is part of G Suite.
It allows users in the same domain to create web-apps with little code.
Google also offers G Suite for Nonprofits
. From
what I have read in your post (small web app, 5 users in the same org,
small database, online collaboration) I believe that App Maker could be
more suitable for your needs. I don't use it myself, so I'm not 100%
certain that data changes would be synced automatically between users that
have the same record displayed, but I believe their data bindings supports
this use-case . The
solution is currently in beta.

If you expect that in the future there might be many more users or users
outside of a G Suite domain, App Engine and/or Firebase
 could be the better approach. Firebase
(either with Firebase Realtime Database or the new Firestore) is very
helpful syncing your data between users in real-time (mobile or web). You
can combine Firebase with App Engine or use either product stand-alone.
However, I think real-time collaboration is difficult in App Engine
with-out a nice stack like Firebase or some other push service. And in
standard environment I don't know any way to use web-sockets for that.

If I remember correctly, there is a new web socket feature for App Engine
in development that could help with that aspect. Not sure.

My 2 cents, best wishes
Ani

On Fri, 9 Feb 2018 at 04:49 Terry Caliendo  wrote:

> I'm trying to understand what it would cost to run a fairly simple app on
> Google App Engine.  I've tried the cost calculator, but I'm a bit unsure
> that I'm entering everything correctly (and/or reading the limits of the
> free tier correctly).  The simple app I'm developing is for a non-profit so
> I need to keep the cost extremely low.
>
> *Basic App Info*
> The app will be a specialized database of client information.
>
> The app will be very responsive for the user in that when the user makes a
> change to a field, the browser will send the change to the server after the
> focus moves away from the field (ie. the user won't be hitting a "submit"
> button to save the changes).
>
> Also, in similar fashion to the way google docs works, if two users have
> the same record open at the same time and one user makes an update to a
> field that gets saved (as outlined above), the system will send the change
> to the other user's browser to update the corresponding field.
>
> I'm primarily a php developer and this type of instant interaction is not
> well suited, because each instance of php/cgi is resource intensive.  Thus
> keeping a process alive to create a websocket type setup would put an
> unnecessary strain on the server.  And polling the server every 5 seconds
> via client side "pull" would also be intensive on the server.
>
>
> *Flexible Enviornment*
> Thus I'm was looking at NodeJS as the development platform as I believe it
> would handle websockets with high efficiency.  But it appears as though
> Node JS needs to run in the "Flexible Environment", which does not have a
> free tier and appears as though it would be somewhat costly for the non
> profit at around $20-$40 per month (from articles I've read on the
> internet).
>
>
> *Standard Environment*
> The "Standard Environment" has a free tier that it seems like I may be
> able to keep this app within.  I think I'd probably choose to have the app
> developed in Python, but I'm open to suggestions as to what would be the
> best player for websockets (or server "push") in the Standard Envoronment.
>
>
>
> *App Resource Needs*
> My app will have about 5 users that will use the app on and off throughout
> their day.   Each of them would make approximately 5 changes per session,
> and browse approximately 10 records per session.And lets say each
> worker does 20 sessions per day, and that the sessions each last 10
> minutes.  So, with approx 23 work days per month, there would be about a
> total of 11,500 writes per month and 22,000 reads per month.  The total
> data stored on the database would be small, around 10MB.
>
>
> *Cost*
> From what I'm gathering from the pricing, I'm absolutely in the free tier
> for the database usage, correct?
>
> If I had the app built to run in the Standard Environment, it appears as
> though I can have one F1 instance running 24/7 for free.  At some point
> during the day the 5 users could all be accessing the app at the same time,
> so would one F1 instance (running Python) be able to handle 5 concurrent
> websocket connections with read/writes to the database?
>
> If so it appears as though I could get this app hosted for this 

Re: [google-appengine] Google Cloud Storage API, sometimes we get errors

[Ani Hatzis]

Hello Armen,

if the code is the same, what else is different? Is the solution hosted in
the same project and is the access to the same bucket? Is the
authentication method different or the ACL of the bucket? Maybe you can
share if Cloud Logging is showing more entries before the actual error.


On Wed, 7 Feb 2018 at 15:05 Armen Haroyan  wrote:

> Hello
>
> We have a SaaS based solution, many clients use the same code. But for 1
> client we often get an error
>
> 'Could not load the default credentials. Browse to 
> https://developers.google.com/accounts/docs/application-default-credentials 
> for more information'
>
>
>
> Any ideas why this can happen?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/11f6a3cd-3051-4ec1-9b52-43df6aab95eb%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ4BUCOwtp-m1MTiwxkV1a-FQ0fnKi2hBQLBZZ9z32V1NQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Picture tagging and uploading to Database

[Ani Hatzis]

Hello Henk,

you could also use other GCP products to achieve this workflow, but 
considering your work-load peaks and the described use-cases App Engine 
could be a good platform choice. Here comes the shooting :)

*Incoming emails*
I'm not sure what caveats come for copy of images between Outlook and 
a web UI, but I think it should be possible to automate some of these 
steps. If you don't have a dedicated email address for such messages, 
Exchange / Outlook could submit POST requests to the GAE app and upload the 
images on request of the receiving Outlook user. If all suppliers send 
their messages to a dedicated address (e.g. upl...@example.com), you could 
configure GAE to handle all incoming mail messages and process the images. 
Python standard environment has InboundMailHandler 

 
for doing this. (Note: Although this feature might not be available for all 
supported GAE environments or languages, you can combine multiple 
environments/languages in the same app)

If there are not many suppliers you might consider the burden of managing 
supplier-side users with authorization to upload and tagging images 
directly into your app.

However, the basic steps of the inbound mail handler could be:

   1. Retrieve image data from the email (and ignore all other binaries / 
   attachments?) and move the images into a dedicated Cloud Storage bucket.
   2. For each image, create a "pending tag-job" in Cloud Datastore.
   3. Save mail body and meta data into Cloud Datastore (a very scalable 
   no-sql database) as "email", and link it to all related tag-jobs (i.e. 
   images).
   4. Maybe resize the images (some GAE versions have features for that or 
   use a 3rd party lib) or you could use a Cloud Function that automatically 
   resizes all images that are added to a certain bucket.
   5. Let the app suggest some tags based on the sender of the email (easy 
   with a map of sender addresses and supplier tags), and maybe even the 
   product tags if the emails are supposed to be structured.
   6. Note: There is a maximum size of inbound/outbound mail messages, 
   including attachments: 31.5 MB.

*Tagging*
In a web UI of the same GAE app, authorized users (of your company) can 
browse all pending (or completed) tag-jobs, and validate / edit the product 
and supplier tags that were automatically suggested by the app. (If you are 
lucky, you can train a machine learning product to tag the images by itself 
or at least suggest a smaller set of tags, e.g. "Apple Fuji", together with 
sender addresses it might be possible to reduce the manual tagging effort 
significantly)

*Output*
This sounds a little bit like a shop or catalogue. The web pages for that 
could be done in GAE, maybe with links to Program X pages that also contain 
the prices? I also don't expect any issues if you want to send the output 
in different formats through other channels, at least not if there is some 
kind of HTTP / REST API available. There is also SMS and voice service 
support in some GAE versions. If the images can be public, you could use 
the Cloud Storage URLs for caching (faster and cheaper), also in 
newsletters.

The newsletters however could be a little tricky. GAE supports sending 
mails only in small quotas (if I remember correctly), bulk emails (i.e. 
same mail to many recipients), or by a 3rd party SMTP server. But there is 
so much to consider (bounce handling, opt-in / double opt-in etc.), it 
might be easier to send emails through a mail service provider. Mailgun, 
Mailjet and SendGrid have special pricing and maybe even free tier for GCP 
customers, but any provider should work if your app can use their API.

*Administrative UI*
Maybe also some functions to delete images/emails, mark tag-jobs as 
completed, edit mappings used for auto suggested tags etc. Maybe also an 
import of product / supplier data from Program X (CSV?). User 
authorization. (Authentication probably best via oAuth 2 since there is so 
much that could go wrong)

I hope my 2 cents were helpful.

Groet uit Duitsland.
Ani



On Tuesday, 6 February 2018 17:08:37 UTC+1, Henk Troost wrote:
>
> Hi all,
>
> I'm new to the forum so i will just shoot.
>
>
> Simple question: is what i am asking for possible?
>
> If yes: which way should i be looking? 
>
> I have been googling and searching for ways to upload images to a database 
> and came across google app engine.
>
>
> The background
>
> For my business (export of fresh products) i want to use daily images from 
> my suppliers to enhance our productoffering to our clients.
> These images are coming in through email (we are using exchange server & 
> outlook 2013). 
> At this moment we are having built a custom ERP-solution (Program X): to 
> my knowledge webbased and working under SQL.
> (i am not familiar with programming etc)
>
>
> The idea
> I want to tag image content and upload them to a database. 
> Then i 

[google-appengine] Re: Error: Not Found The requested URL / was not found on this server.

[Ani Hatzis]

Hello Naheez,

unless you have deployed an app.yaml file that is different from the 
tutorial (as described here 
),
 
you probably just forgot to deploy the static file index.html into your 
app's www/ directory as described here 
.
 
Also for this tutorial example, the www/ directory should be in the same 
directory as your app.yaml file. For example:

/my-local-computer/my-app/
/my-local-computer/my-app/app.yaml
/my-local-computer/my-app/www/index.html
/my-local-computer/my-app/www/css/style.css
and so on.


According to the error page, the request actually reached your application, 
so you will be able to investigate the requests in your logs.
In Cloud Console, go to Logging, select your App Engine project (so you can 
see the logging for all services of the app), and search for a request that 
caused this 404 status and the path "/". Optionally, you can enter 
"status:404" in Logging's text search to filter for such logs.

Expand a matching log record and you will see details that probably will 
tell you what exactly went wrong under the hood. The user-facing error 
pages are not very helpful for debugging (and they shouldn't be for 
security reasons).


On Wednesday, 8 November 2017 14:52:00 UTC+1, Naheez Thawfeeg wrote:
>
> Dear all
>
> I am trying to host a HTLML only static website on app engine...The 
> website is www.guestworm.com and I followed this tutorial to upload and 
> deploy 
> my website : 
> https://cloud.google.com/appengine/docs/standard/python/getting-started/hosting-a-static-website
>
> I get the following error when I visit my site after deploying ant it's 
> not working : 
>
> Error: Not FoundThe requested URL / was not found on this server.
>
> Can anyone help fix this issue?
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/954a2b4d-bef0-433e-9c91-bcd7d1d67870%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [google-appengine] Re: [Google Cloud Insiders] URGENT: is GAE Standard memcache down????

[Ani Hatzis]

Hm, no issues here in my apps, eu-west. But only use memcache explicitly
with Datastore for an admin task, but it works either case. Maybe it is the
Capability API, I don't use it here.

On Mon, Nov 6, 2017, 22:32 'Yannick (Cloud Platform Support)' via Google
App Engine,  wrote:

> There effectively is an ongoing Memcache outage, you can follow the incident
> details  for
> periodic updates on this issue as we work on resolving it.
>
>
> On Monday, November 6, 2017 at 4:11:12 PM UTC-5, PK wrote:
>
>> Yes it is here: https://issuetracker.google.com/issues/68946028
>>
>> I also see now a message in the Google Cloud Status Console about this.
>>
>> On Nov 6, 2017, at 1:07 PM, Wade Piotrowski  wrote:
>>
>> Yes, Posting shortly. Please file a support case for the most up-to-date
>> information if you haven't already.
>>
>>
>> On Mon, Nov 6, 2017 at 12:56 PM PK  wrote:
>>
>>> This started returning False 15 minutes ago. My app is down because of
>>> it.
>>>
>>> from google.appengine.api.capabilities import CapabilitySet
>>> print CapabilitySet('memcache', methods=['set']).is_enabled()
>>>
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Google Cloud Insiders" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to google-cloud-insiders+unsubscr...@googlegroups.com.
>>> To post to this group, send email to
>>> google-cloud-insid...@googlegroups.com.
>>> Visit this group at
>>> https://groups.google.com/group/google-cloud-insiders.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/google-cloud-insiders/C170C4A6-5355-428B-B010-4507B5148F3C%40gae123.com
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengine+unsubscr...@googlegroups.com.
> To post to this group, send email to google-appengine@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/google-appengine/ab1bc058-f3d4-4c3f-b4ea-e9951177b2e1%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/CALdDuQ7xM%2ByvvPtZaSxC5PYHx3iQ5sUU%2Bz1OeQm6gPcpVK7bLQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How do I find out what I am being billed for?

[Ani Hatzis]

I'm surprised your app.yaml has no "resources" section defined, because the 
default would be "cpu: 1", which wouldn't add up with the instance core 
hours you have reported from your billing. But the default is also 
"automatic_scaling", so maybe between the moments you have checked, your 
app scaled automatically to many more than the one instance. The chart 
could tell you that. However, just to be safe, it might be worth to ask the 
Billing Support 
 for help 
understanding the reason why instance core hours of the current version ID 
"dev" were or are burning through so fast in your app.


> But I will try specifying the resources. Does this look ok?
>
>
> manual_scaling:
>
>   instances: 1
>
> resources:
>
>   cpu: 2
>
>   memory_gb: 1.4
>
>   disk_size_gb: 10
>
>  
Yes, this looks correct. When you deploy, explicitly state a new version 
ID, so Billing Support has a chance to look into the current version's yaml 
files (or whatever they might need).

Good luck and have fun!

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/b45fc262-b0e8-4772-aff1-b8e009351d77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How do I find out what I am being billed for?

[Ani Hatzis]

On Sunday, 5 November 2017 23:59:42 UTC+1, Peter Camilleri wrote:
>
> Last night at 12:00 I recorded this from the billing page
>
> App Engine Flex Instance Core Hours 8,640.47 Minute $9.92
>
> And this morning I recorded this
>
> App Engine Flex Instance Core Hours 191.79 Hour $13.22
>
> 8600 minutes is about 144 hours
>
> 191 - 144 = 47 hours of time used in 9 hours. So it looks like I've got 5 
> apps running simultaneously which is eating up my credit.
>

OK. In case that there is really just one instance running all the time (as 
your Versions page suggested), it might be that you have the instance 
configure with multiple vCPUs (cores). Flexible environment is charged per 
core hour (see doc here 
). 
So, an instance with four cores running for one (real) hour will be charged 
with 4 core hours. In the 9 hours you had ~ $3.3 costs for core hours, ~ 
$0.37/core hour, depending on your region, we talk about an instance with 
maybe 8 cores. Check your app.yaml file for a "resources" configuration 
(see doc here 
).
 
What does the "cpu" value say? If you want to can copy-paste the section 
here.
 

>
> My versions page looks like this. Could this be where my credit is going?
>
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> I thought that these instances do not consume resources if they are 
> stopped? The lower 6 instances are also created when I execute "gcloud app 
> deploy app.yaml" with no extra arguments. So should I instead be specifying 
> a specific version each time I deploy (e.g. dev or prod) so that I am 
> overwriting the current instance and not creating a new instance each time? 
>
However these are all inside of my "default" service, so I thought they 
> would naturally be overwriting the last deployed version naturally.
>
>
You are right, stopped instances don't consume resources (at least not for 
the VM). What you are seeing in this page are all versions of the app that 
have been deployed until now. When you deploy a version, by default, 
instance of an older version is stopped. A new instance spins up and gets 
all the traffic. So your current version is in the first line, handling 
100% of the traffic, running 1 instance. Below are the older versions (no 
traffic and no instances).

Older versions are kept, so you have the opportunity to roll-back to an 
older version in case your new version shows elevated errors. You can 
delete versions that you don't need anymore though, but in that case you 
can't just roll-back to them.


> Python flexible environment (although this is only because the standard 
> environment tutorials using CloudSQL wouldnt work out of the box.)
>

Have you followed these Cloud SQL instructions of standard environment here 
? If 
you had no other reason for flexible, I would recommend the standard 
environment. Albeit there are many good reasons for choosing flexible 
environment over standard, flexible needs more administrative overhead than 
standard. In standard it's easier to focus on your own code.
 

>
> No worries. I quickly checked and found my instance was not using the 
> cheapest instance type. So that cut my SQL costs by about 80% right there.
>

Great! And add budgets & alerts to your billing account. It's really 
helpful to stay alerted about potential config errors wreaking havocs to 
your credits during your first steps.

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/3ed2b648-f75d-4286-a4a4-3a35e11c6f1d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: How do I find out what I am being billed for?

[Ani Hatzis]

On Sunday, 5 November 2017 17:06:37 UTC+1, Peter Camilleri wrote:
>
>
> I didnt choose an instance class. How do I do this? Isn't that more 
> something you do in the compute engine? Not the app engine?
>
> I have recently tried to specify resources, e.g. only 1 instance, minimal 
> ram, etc. but I dont did this after I found out my costs were unusually high
>

App Engine standard environment also supports different instances classes 

 
with different pricing . You 
define the instance classes (instance_class) and the type of scaling in the 
*app.yaml* files of your GAE app (see An Overview of App Engine 
,
 
How Instances are Managed 
,
 
Managing App Resources 
,
 
and the app.yaml Reference 
). 
If I remember correctly, the defaults will be the cheapest instance class 
(F1) and auto-scaling. There can be many reasons why an instance would be 
automatically started (one of the big selling points of App Engine 
actually), so shutting instances down manually might have no persistent 
effect at all. Also note that for apps in standard environment, the charges 
will always be calculated on base of the F1 price, where the higher 
instance classes are priced as a multiple of the F1 price, i.e. if you have 
used F4, their instance hours will be 4 times of what they actually have 
been, but with the price of F1.

NB: Since each app comes with a free tier of 28 instance hours per day (F1 
or B1), an un-used app should remain for free, unless there is something 
wrong in your configuration as explained in my earlier post. In order to 
learn more about this, it makes sense to check the logging of the app to 
see what requests where handled by the billed instances. Maybe it is 
unexpected traffic from outside (a crawler?), a cron-job that runs every 
now and then, or there is a bug in a task in one of your task-queues and 
the task is retried indefinitely.

On the other hand, App Engine flexible environment are deployed to virtual 
machine types as specified, and the billing is different from standard 
environment, see Pricing - Flexible environment instances 
.

It would help if you could tell us which version of App Engine you have in 
use (programming language, and whether standard or flexible environment). 
Above I have linked to the Python version.
 

>  
>
>> or that there are too many idle instances running, or forgotten 
>> cron-jobs, etc. Best approach is to study all the expenses for the billing 
>> account where the costs were charged and identify the big hitters.
>>
>
> How do I find idle instances? I found some older instances of apps were 
> running so I deleted them. I think that is what you are referring to. IS 
> this normal?
>  
>

You can find active and idle instances on your App Engine > Dashboard 
 
or App Engine > Instances 
.
 
Toggle the version and chart type ("Instances") above the charts, and the 
period between 1 hour and 30 days. It not only shows the instances, but you 
can also access the requests logs for each instance to figure out what is 
causing the instances to not be shut down (as mentioned earlier).
 

> Yea this is where I found out I was paying 50$ for resources. Although I 
> dont find the descriptions to be very helpful, which is why I came here 
> looking for help.
>

If you want, you could copy-paste the lines here. But remove all project 
IDs first and anything else you consider sensitive information.
 

>
> Yes, I also have a cloud SQL instance. But it is literally just 1 database 
> with 3 tables, each with about 4-5 fields (these are from tutorials). So 
> it's realistically a tiny database. but i still got charged 15$ for using 
> that. What I would like is a breakdown of where my money went, because all 
> I know is that I lost 15$ to CloudSQL but have no idea why.
>

Different from Cloud Datastore, Cloud SQL is also charging for instances 
(as their is a managed VM being the database), and maybe also CPU and 
memory if you have dedicated-core instances. See Cloud SQL Pricing 
. Unfortunatley, I'm not very 
familiar with Cloud SQL, so I can't tell how you could save money for 
unused Cloud SQL resources, but probably disabling its instance might at 
least save the instance costs until you actually need it.

-- 
You received this message because you are 

[google-appengine] Re: How do I find out what I am being billed for?

[Ani Hatzis]

Hi Peter,

$75 per week sounds a lot for a non-used tutorial app, that normally should 
stay below the free tier. Maybe there is more than one app running since 
you played with the tutorials. One other explanation would be that you have 
chosen an expensive instance class (e.g. F4, F4G1), or that there are too 
many idle instances running, or forgotten cron-jobs, etc. Best approach is 
to study all the expenses for the billing account where the costs were 
charged and identify the big hitters.

In order to see your detailed expenses in Google Cloud Platform:

   1. Go to Google Cloud Console > Billing 
   
   2. Click on each billing account (probably you only have one listed for 
   all your projects)
   3. On the left navigation under Billing, click on *Transactions*
   4. You will see a list of transactions. Make sure that above the table, 
   you have selected:
  1. Detailed transaction view
  2. All transactions
  3. All time
   5. Each costs item will show you which project has consumed a certain 
   service or resource, the quantity (in hours, counts etc.), and the cost in 
   USD
   6. Some lines will show payments (I assume, the free 300$ credit should 
   appear as the last item at the bottom of the page)

More information here: View Your Cost and Payment History 
.

Most tutorials in the documentation of Google Cloud Platform also include 
how to delete the training assets after the tutorial is completed, so no 
more costs will occur.

However, if you do not want to delete a Google App Engine app, you have two 
options to save costs (at least for not running the app instances):

   - Disable the app until you want to work again with the app. The app 
   will stop serving requests, until re-enabled. Data and state of the app 
   will be kept, but may result in costs (e.g. size of datastore).
   1. In Cloud Console for the project that has the GAE app, go to App 
  Engine > Settings 
  
  2. Click on *Disable application* button
  3. Enter the appp ID and confirm the action
   - Apply a daily billing limit for the app. Once the daily spending limit 
   is reached, the app will stop working, any further requests will be 
   responded with server errors.
  1. In Cloud Console for the project that has the GAE app, go to App 
  Engine > Settings 
  
  2. Click on the Edit button above
  3. Enter a USD value (e.g. 1) into the input field *Daily spending*
  4. Click on Save
   
Both options relate to GAE costs (typically instance hours, some traffic, 
datastore ops), but other costs may incur, let's say, you have created the 
app with a Cloud SQL instance, significant Datastore or Storage size. Just 
check the Transactions page to see which costs were actually charged. The 
documentation provide lots of information on billing, as well as pricing, 
quotas, and free tiers for each product.

Optionally, you can also setup budgets and alerts 
, so you will receive 
an email at certain budget milestones (reset each month). I strongly 
recommend to setup at least one alert for each billing account, so you 
won't be surprised at the end of a billing period in case you forgot to 
delete or disable something.

Ani

On Saturday, 4 November 2017 17:09:33 UTC+1, Peter Camilleri wrote:
>
> I started playing with GAE a couple weeks ago. I've only played around 
> with the tutorials so far. I successfully deployed the bookshelf and visit 
> apps.
>
> I checked my billing today and found that I have burnt through about 1/5th 
> of my free 300$ credit. This seems pretty extreme.
>
> The billing page doesnt really tell me what I've been billed for. My 
> biggest expense is
>
> "Credit 
> FreeTrial:2017-10-22T00:00:00.000-07:002af76aae-dd42-421e--X" 
> which is 50$
>
> Should I be deleting my app engine instance when I'm not using it? 
>
> Even so, these prices seem pretty extreme, it looks like App engine will 
> cost me around $75 a week to host my project. The GAE calculator gives me a 
> significantly smaller estimate.
>
> Where am I losing money?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/7551e3a7-42a2-403d-bd3b-b9b390b53159%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[google-appengine] Re: Datastore storage size

[Ani Hatzis]

Hello John,

Cloud Datastore automatically scales its size up or down, you don't need to 
change anything.

The pie chart is showing the size for each resource (as shown in the table 
on its right) relative to the total size currently taken by this datastore. 
If you hover over the colored segments you will see to which resource the 
segment belongs. In other words, for your project, you will learn that 
almost all data is used by storing the actual entities (in blue), and that 
a much smaller part is used for the built-in indexes (that is, for indexed 
properties), in green. Composite indexes, in your case 0, refers to indexes 
that are composed of multiple indexed properties or ancestor queries.

You can drill down the chart / stats to certain namespaces or kinds. The 
documentation covers more information about Datastore statistics 
 and the Dashboard 
.

Hope that clarified your question.

On Friday, 3 November 2017 14:01:13 UTC+1, John Coleman wrote:
>
> Hello,
>
> We have an GAE application and looking at datastore dashboard, it's 
> showing the attached datastore usage (almost 96% for entities).
> Does the storage automatically scale up or do we need to change quota??
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/google-appengine/dbf25b1b-fa32-47f4-8fb4-7e0e12197a21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.