>From my limited experience of doing ETL-type tasks on App Engine, I'd 
suggest using a "backend instance" with basic scaling if you can (although 
these instances are probably slower than the n1-standard-2, and it 
certainly will have much less memory). This would avoid the complexity of 
managing the lifetime of some other instance, and if you have a bunch of 
existing App Engine standard stuff, is simpler to manage. Backend instances 
can take a long time to process a cron task request, or can start a 
background thread to do the ETL, so you can avoid the request 
timer: https://cloud.google.com/appengine/docs/java/an-overview-of-app-engine

If you really need to run this on a machine with more memory or custom 
code, we've used an App Engine flexible module that we leave running all 
the time. This does cost us some fraction of the instance time, but we 
figured the simplicity of not needing to deal with starting, stopping, and 
waiting for instance boot makes it worth it, and relative to our other 
costs, running a single small VM 24x7 is not significant. We used an AE 
flexible module so that App Engine manages the instance for us, and so 
deploying to it is extremely similar to our other app engine code.

On Sunday, September 18, 2016 at 10:37:19 AM UTC-4, Niklas Andersson wrote:
> I have a bunch of files at Cloud Storage that I need to parse and insert 
> into Cloud SQL once per day.
> The operation takes about an hour on a n1-standard-2 instance, and then 
> ~10mins to insert the data into Cloud SQL.
> My idea of design to solve this problem with GAE is:
>    1. Define a scheduled task in GAE, it does a HTTP request to a URL
>    2. Create a URL handler that can respond to the HTTP req mentioned 
>    above in GAE, it’s main task would be to boot a GCE instance
>    3. The GCE instance would be based on a template where it does a few 
>    things automatically at boot:
>       1. Download the file from Cloud Storage
>       2. Parses it
>       3. Inserts the data into Cloud SQL
>       4. Shuts down
> Would this be the best design to solve the problem?

You received this message because you are subscribed to the Google Groups 
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to google-appengine+unsubscr...@googlegroups.com.
To post to this group, send email to google-appengine@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit 
For more options, visit https://groups.google.com/d/optout.

Reply via email to