Hi Nick
Thank you very much for your reply and thoughts. I appreciate your time! I
tend to use GAE managed vm instead of "vanilla" GAE At the moment. I know
it's beta right now, but I hope you keep it up and I think this way we are
not too much tied to GAE stuff. This also solves some of my "problems" I
described above too.
I'm in a hurry right now, but I'll, if you don't mind, try to ask more
specific and better explained questions later about my current problems.
Regards,
Silvan
On Tuesday, March 8, 2016 at 6:59:31 PM UTC+1, Nick (Cloud Platform
Support) wrote:
>
> Hey Silvan,
>
> These are a whole lot of excellent questions and details. I'll try to
> offer my feedback on several points and address your questions.
>
> Without knowing what your system is, it's hard to tell beyond the abstract
> notion of "separating environments" what your goals should be in
> development. However, you seem to have a strong knowledge of many tools and
> it seems that you'll be able to select tools and services which meet the
> needs you have.
>
> As far as the question of os.getenv('SERVER_SOFTWARE', '').startswith('Google
> App Engine'), I can say that while this is not exactly a "universal best
> practice", there's nothing wrong with it, either. It's just a particularity
> of how to detect the production environment on GAE python. Pretty much
> every web framework in every language will have some idiomatic means of
> determining whether we're in production or development, and choosing
> appropriate database and API connection params / code paths, etc.
> accordingly.
>
> As far as running django management commands locally, there are tons of
> toolchains out there for managing development commands. I just learned
> about Fabric based on your mention of it, and it seems like an excellent
> tool for python developers to do exactly what you need. As for more
> specifics of exactly what kinds of questions can arise in regard to "local
> management of Django", we enter a situation where we need more specifics to
> spur the development of more specific answers. For example, does your
> "local Django" (really the development environment version of your code
> which uses the Django framework) connect to a remote DB? Or a local copy of
> the DB?
>
> Now, as to the question of best practices in migrating production
> databases, this depends largely on two factors: the database technology
> itself and the environment it's deployed to. A MySQL server running on a
> GCE instance is different from Cloud SQL, which is different from
> Datastore, which is different from a MongoDB server, etc. Finally, the
> system which *interacts *with the database is also critical for
> determining the specifics of a migration strategy: will the app require a
> period of read-only activity? Can the migration happen incrementally? Etc.
>
> One strategy for management of databases which you mentioned, used
> especially when you have multiple database instances (as in the case where
> you want to have multiple separate environments on the same
> infrastructure), would be to have an admin instance from which scripts are
> run. One benefit of this is that both control and logging are centralized.
> Logging separately into each database instance to manage it isn't
> necessarily scalable, and is almost impossible or an anti-pattern when
> dealing with primary-replica DB systems, sharding, etc.
>
> With Datastore, often you see *schema* migrations, in which a user's
> "schema" for the schemaless datastore needs to update, in which case the
> relevant elements need to either be incrementally-migrated as they're
> updated / retrieved *or *all at once, through simple iteration with
> cursors, a Map-Reduce job, etc. In addition to this sense of the word
> "migration", often people will backup their Datastore to Cloud Storage
> which basically means a JSON / binary serialization of entities for
> insertion into another app's Datastore (or in more exotic cases, for
> translation to an external RDBMS, etc.).
>
> As to your final post, manage.py from Django and remote_api from App
> Engine can work well together. If you find complexity is increasing,
> something like Fabric, again, is a very valuable tool. You could run Fabric
> off of your "admin instance" in the cloud, while keeping a local copy of
> all scripts and credentials in the rare case that your admin instance needs
> redundancy (of course you can save images of the instance and merely spin
> these up in such a case). Another advantage of having a cloud-based admin
> instance which centralizes your administrative scripts is the more
> consistent uptime than simply running these off your laptop, or a special
> box in your workspace, etc.
>
> I hope some of these thoughts have been helpful in responding to your very
> interesting post. You seem well-placed to develop any solutions required as
> the need for them comes along. A commitment to experimentation and reading
> documentation is indispensable in our era where a proliferation of
> potential solutions for each of the classic problems (and new problems) of
> web systems development, all with great selling-points and specific
> peculiar features, confronts us as developers.
>
> Feel free to use this forum for general, high level discussion as you've
> done here, and feel free to ask questions about any services in particular.
> Thanks for contributing your thoughts and use-case details!
>
> Sincerely,
>
> Nick
> Cloud Platform Community Support
>
>
>
>
> On Monday, March 7, 2016 at 9:11:54 AM UTC-5, Silvan Spross wrote:
>>
>> I'm used to deploy to managed servers through some Fabric scripts (or
>> CI's using them). I also run a project with Docker, where Django, Celery,
>> Redis, RabbitMQ, Postgres and Ngnix are in separate containers.
>>
>> However, now I want to give Google App Engine a try. I don't want to use
>> Google Container Engine or Google App Engine Managed VM for now. I consider
>> them as another option if using plain Google App Engine does not work out.
>>
>> My goals are:
>>
>> 1.
>> *Separated Environments *Development, Stage and Production (where stage
>> and production can be just different app engine projects and so can be
>> considered as "the same").
>>
>> 2. *Using Task Queue* (instead of Celery)
>> Used for async tasks and scheduled ones (cronjob).
>>
>> 3. *Using Memcache* (instead of Redis)
>> For caching keys up to caching templates etc.
>>
>> 4.
>> *Using Cloud Storage (Buckets) as Storage backend *For media files.
>>
>> 5. *Using CloudSQL* (instead of Postgres)
>> I don't care about nonrel for the moment. Using Datastore should be a
>> future goal here.
>>
>> My subject is all about how to setup and separate environments (1). But
>> if someone has feedback or input about another goal, why e.g. not to use
>> it, I'm very happy to hear! Btw. my test project is much alike
>> https://github.com/GoogleCloudPlatform/appengine-django-skeleton for now.
>>
>> What bugs me at the moment:
>>
>> - Load different settings e.g. database based on
>> os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')
>> Is this best practice?
>>
>> - I run my application local with:
>> source env/bin/activate
>> dev_appserver.py app.yaml
>> to have python libraries that use C extensions loaded locally (like
>> mysql) and that
>> google.appengine
>> is available. But how can I run django management commands locally?
>> Whats best practice here?
>>
>> - And the biggest one: Whats best practice to migrate production
>> database? I heard stuff about interacting with the database through a
>> separate GCE instance running scripts. Or is there no best practice
>> because
>> with Datastore "you don't need" migrations ;)
>>
>> I would be very happy about some tipps pointing me to the right
>> direction. Thank you!
>> Silvan
>>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/e61fc87a-5854-412c-98cf-b19a75846e63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
