Hey Silvan,
These are a whole lot of excellent questions and details. I'll try to offer
my feedback on several points and address your questions.
Without knowing what your system is, it's hard to tell beyond the abstract
notion of "separating environments" what your goals should be in
development. However, you seem to have a strong knowledge of many tools and
it seems that you'll be able to select tools and services which meet the
needs you have.
As far as the question of os.getenv('SERVER_SOFTWARE', '').startswith('Google
App Engine'), I can say that while this is not exactly a "universal best
practice", there's nothing wrong with it, either. It's just a particularity
of how to detect the production environment on GAE python. Pretty much
every web framework in every language will have some idiomatic means of
determining whether we're in production or development, and choosing
appropriate database and API connection params / code paths, etc.
accordingly.
As far as running django management commands locally, there are tons of
toolchains out there for managing development commands. I just learned
about Fabric based on your mention of it, and it seems like an excellent
tool for python developers to do exactly what you need. As for more
specifics of exactly what kinds of questions can arise in regard to "local
management of Django", we enter a situation where we need more specifics to
spur the development of more specific answers. For example, does your
"local Django" (really the development environment version of your code
which uses the Django framework) connect to a remote DB? Or a local copy of
the DB?
Now, as to the question of best practices in migrating production
databases, this depends largely on two factors: the database technology
itself and the environment it's deployed to. A MySQL server running on a
GCE instance is different from Cloud SQL, which is different from
Datastore, which is different from a MongoDB server, etc. Finally, the
system which *interacts *with the database is also critical for determining
the specifics of a migration strategy: will the app require a period of
read-only activity? Can the migration happen incrementally? Etc.
One strategy for management of databases which you mentioned, used
especially when you have multiple database instances (as in the case where
you want to have multiple separate environments on the same
infrastructure), would be to have an admin instance from which scripts are
run. One benefit of this is that both control and logging are centralized.
Logging separately into each database instance to manage it isn't
necessarily scalable, and is almost impossible or an anti-pattern when
dealing with primary-replica DB systems, sharding, etc.
With Datastore, often you see *schema* migrations, in which a user's
"schema" for the schemaless datastore needs to update, in which case the
relevant elements need to either be incrementally-migrated as they're
updated / retrieved *or *all at once, through simple iteration with
cursors, a Map-Reduce job, etc. In addition to this sense of the word
"migration", often people will backup their Datastore to Cloud Storage
which basically means a JSON / binary serialization of entities for
insertion into another app's Datastore (or in more exotic cases, for
translation to an external RDBMS, etc.).
As to your final post, manage.py from Django and remote_api from App Engine
can work well together. If you find complexity is increasing, something
like Fabric, again, is a very valuable tool. You could run Fabric off of
your "admin instance" in the cloud, while keeping a local copy of all
scripts and credentials in the rare case that your admin instance needs
redundancy (of course you can save images of the instance and merely spin
these up in such a case). Another advantage of having a cloud-based admin
instance which centralizes your administrative scripts is the more
consistent uptime than simply running these off your laptop, or a special
box in your workspace, etc.
I hope some of these thoughts have been helpful in responding to your very
interesting post. You seem well-placed to develop any solutions required as
the need for them comes along. A commitment to experimentation and reading
documentation is indispensable in our era where a proliferation of
potential solutions for each of the classic problems (and new problems) of
web systems development, all with great selling-points and specific
peculiar features, confronts us as developers.
Feel free to use this forum for general, high level discussion as you've
done here, and feel free to ask questions about any services in particular.
Thanks for contributing your thoughts and use-case details!
Sincerely,
Nick
Cloud Platform Community Support
On Monday, March 7, 2016 at 9:11:54 AM UTC-5, Silvan Spross wrote:
>
> I'm used to deploy to managed servers through some Fabric scripts (or CI's
> using them). I also run a project with Docker, where Django, Celery, Redis,
> RabbitMQ, Postgres and Ngnix are in separate containers.
>
> However, now I want to give Google App Engine a try. I don't want to use
> Google Container Engine or Google App Engine Managed VM for now. I consider
> them as another option if using plain Google App Engine does not work out.
>
> My goals are:
>
> 1.
> *Separated Environments *Development, Stage and Production (where stage
> and production can be just different app engine projects and so can be
> considered as "the same").
>
> 2. *Using Task Queue* (instead of Celery)
> Used for async tasks and scheduled ones (cronjob).
>
> 3. *Using Memcache* (instead of Redis)
> For caching keys up to caching templates etc.
>
> 4.
> *Using Cloud Storage (Buckets) as Storage backend *For media files.
>
> 5. *Using CloudSQL* (instead of Postgres)
> I don't care about nonrel for the moment. Using Datastore should be a
> future goal here.
>
> My subject is all about how to setup and separate environments (1). But if
> someone has feedback or input about another goal, why e.g. not to use it,
> I'm very happy to hear! Btw. my test project is much alike
> https://github.com/GoogleCloudPlatform/appengine-django-skeleton for now.
>
> What bugs me at the moment:
>
> - Load different settings e.g. database based on
> os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine')
> Is this best practice?
>
> - I run my application local with:
> source env/bin/activate
> dev_appserver.py app.yaml
> to have python libraries that use C extensions loaded locally (like
> mysql) and that
> google.appengine
> is available. But how can I run django management commands locally?
> Whats best practice here?
>
> - And the biggest one: Whats best practice to migrate production
> database? I heard stuff about interacting with the database through a
> separate GCE instance running scripts. Or is there no best practice
> because
> with Datastore "you don't need" migrations ;)
>
> I would be very happy about some tipps pointing me to the right direction.
> Thank you!
> Silvan
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/f773e074-cc65-4dcf-8f88-9aa1d8eb5a67%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
