Re: How to update a live / deployed Django project?

[Rok Jaklič]

We are using two git branches, one for development and master. When dev 
branch is good enough, we merge it to master and then we have a script 
which does git pull and restarts uwsgi.

Rok

On Wednesday, February 10, 2016 at 9:50:22 AM UTC+1, Tobias Dacoir wrote:
>
> We've build a small web application using Django 1.8 over the past couple 
> of months and we are going live with version 1.0 soon (one last closed beta 
> test is coming up). 
> So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx 
> and a local MySQL Server.
>
> I've already found some good tutorials on how to properly set it up (
> http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
>  
> ) however what I am mising now is how to handle updates to our code.
>
> For our development server we have a git hook that automatically deploys 
> new commits to the server. However that often requires manual interaction - 
> running manage.py makemigrations etc. Sometimes Django also does not pick 
> up our changed models and we have to manually fiddle with the Database in 
> order to get it back running. We even had times where we lost all our data 
> since it was easier to just start with a new database. 
>
> Obviously this can't happen once we are live and obviously we want 
> downtime to be as small as possible. So is there any guide on how to update 
> code of a live, deployed Django application?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/0f5f95e5-c75f-4028-ab48-0dca2c5660eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Tobias Dacoir]

Thanks for all the great input. I'll definetely look into salt too (only 
worked with puppet so far) but for now I guess fabric will be fine. We are 
using Python 2.7 everywhere anyway. 

Like I said, right now we plan to host it on a single physical machine that 
we get from the university (it's a research project). But I see that nobody 
is using the Apache / WSGI solution that is explained in Django's 
documentation. So I'll also have to look into nginx + gunicorn as well. 
Phew...seems like even more work than actually developing our webapp :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/786c01bc-36d0-45ff-9d33-890db0b5ab50%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Andrew Farrell]

> I have been through several variations on what you are doing.  The
solution which has worked best for me is to use SaltStack to control the
update.

If you are interested in learning SaltStack, I just recently wrote a
step-by-step
tutorial

on
it, targeted for django developers. It walks you through writing a
configuration to deploy a django site (with postgres, gunicorn, and nginx)
on a local virtual machine and then VPS as well as one way of writing
automated tests for the configuration.

If you have feedback on the tutorial or suggestions on how to improve the
learner experience, please let me know.

cheers,
Andrew

On Wed, Feb 10, 2016 at 2:10 PM, Vernon D. Cole 
wrote:

> I have been through several variations on what you are doing.  The
> solution which has worked best for me is to use SaltStack to control the
> update.  It will pull your updates using git, apply as needed, and restart
> your servers. If you are building a new server (for testing or prototype
> usually) it will install everything. The last time I needed to test a new
> version of django it took about two hours to build a whole new prototype
> server on a virtual machine on my laptop.  Version updates on a running
> server take an eyeblink.
>
> I put an example up on github/salt_demo.
>  It needs some work since two
> years ago, but should give you the idea. It will run standalone, but I
> really recommend setting up a Salt master, even if you only have a few
> machines.  It can can really save you -- for example, I have had times when
> an ssh daemon quit working, and I was able to regain control using Salt.
> The demo uses PostgreSQL -- I will send you my mariadb state if you wish.
> --
> Vernon Cole
>
>
> On Wednesday, February 10, 2016 at 1:50:22 AM UTC-7, Tobias Dacoir wrote:
>>
>> We've build a small web application using Django 1.8 over the past couple
>> of months and we are going live with version 1.0 soon (one last closed beta
>> test is coming up).
>> So far we have deployed it on an Ubuntu VM (in Azure) using Apache /
>> Nginx and a local MySQL Server.
>>
>> I've already found some good tutorials on how to properly set it up (
>> http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
>> ) however what I am mising now is how to handle updates to our code.
>>
>> For our development server we have a git hook that automatically deploys
>> new commits to the server. However that often requires manual interaction -
>> running manage.py makemigrations etc. Sometimes Django also does not pick
>> up our changed models and we have to manually fiddle with the Database in
>> order to get it back running. We even had times where we lost all our data
>> since it was easier to just start with a new database.
>>
>> Obviously this can't happen once we are live and obviously we want
>> downtime to be as small as possible. So is there any guide on how to update
>> code of a live, deployed Django application?
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/3aeba092-2834-455e-a5e8-0c15be79d245%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2By5TLbjkiwEcWPmz4AdQF-GpUduLHyx3SmY%2BTNvZyX--WaAyg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Vernon D. Cole]

I have been through several variations on what you are doing.  The solution 
which has worked best for me is to use SaltStack to control the update.  It 
will pull your updates using git, apply as needed, and restart your 
servers. If you are building a new server (for testing or prototype 
usually) it will install everything. The last time I needed to test a new 
version of django it took about two hours to build a whole new prototype 
server on a virtual machine on my laptop.  Version updates on a running 
server take an eyeblink. 

I put an example up on github/salt_demo. 
 It needs some work since two 
years ago, but should give you the idea. It will run standalone, but I 
really recommend setting up a Salt master, even if you only have a few 
machines.  It can can really save you -- for example, I have had times when 
an ssh daemon quit working, and I was able to regain control using Salt.  
The demo uses PostgreSQL -- I will send you my mariadb state if you wish.
--
Vernon Cole


On Wednesday, February 10, 2016 at 1:50:22 AM UTC-7, Tobias Dacoir wrote:
>
> We've build a small web application using Django 1.8 over the past couple 
> of months and we are going live with version 1.0 soon (one last closed beta 
> test is coming up). 
> So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx 
> and a local MySQL Server.
>
> I've already found some good tutorials on how to properly set it up (
> http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
>  
> ) however what I am mising now is how to handle updates to our code.
>
> For our development server we have a git hook that automatically deploys 
> new commits to the server. However that often requires manual interaction - 
> running manage.py makemigrations etc. Sometimes Django also does not pick 
> up our changed models and we have to manually fiddle with the Database in 
> order to get it back running. We even had times where we lost all our data 
> since it was easier to just start with a new database. 
>
> Obviously this can't happen once we are live and obviously we want 
> downtime to be as small as possible. So is there any guide on how to update 
> code of a live, deployed Django application?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/3aeba092-2834-455e-a5e8-0c15be79d245%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Radek Svarz]

Tobias,

I know I will not help you directly, but maybe my setup + process can help 
someone else. I am deploying towards Redhat Openshift 2 PaaS.
I was in the similar situation choosing the best fit option for continuous 
deployment.

Final process:

- Merging the feature code to the master branch and pushing to github.
- Codeship autograbs the code from github, does CI tests.
- If tests pass it pushes to Openshift with hot_deploy setup (i.e. the old 
version is still running during the deploy phase). 
- Once the new version is on Openshift incl. all updated requirements it 
restarts the WSGI process (= very short downtime) and the new version is up.
- TODO some automated smoke tests towards Openshift production

The whole thing lasts a few minutes totally on the background, I am only 
notified of the success / failure.

And I can make some hot fixes through github's web interface :)

note: fabric is not compatible with Python 3, so its nogo for me. 

Radek

On Wednesday, February 10, 2016 at 9:50:22 AM UTC+1, Tobias Dacoir wrote:
>
> We've build a small web application using Django 1.8 over the past couple 
> of months and we are going live with version 1.0 soon (one last closed beta 
> test is coming up). 
> So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx 
> and a local MySQL Server.
>
> I've already found some good tutorials on how to properly set it up (
> http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
>  
> ) however what I am mising now is how to handle updates to our code.
>
> For our development server we have a git hook that automatically deploys 
> new commits to the server. However that often requires manual interaction - 
> running manage.py makemigrations etc. Sometimes Django also does not pick 
> up our changed models and we have to manually fiddle with the Database in 
> order to get it back running. We even had times where we lost all our data 
> since it was easier to just start with a new database. 
>
> Obviously this can't happen once we are live and obviously we want 
> downtime to be as small as possible. So is there any guide on how to update 
> code of a live, deployed Django application?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/c8fcfab2-6427-430c-8c9b-2ba1d05d3fa4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Andreas Kuhne]

The thing is, you should be able to catch those issues when migrating
locally on you development machines. In that case, just fix the migration
and then it should work perfectly in production. You can always add certain
values in the migration scripts themselves.

Regards,

Andréas

2016-02-10 14:51 GMT+01:00 Tobias Dacoir :

> Thanks for sharing your workflow. Sounds simple enough and is more or less
> a scripted version of what I had in mind all along.
>
> About the manual intervention: At some points during development we had to
> provide a one-time manual entry when Django was migrating the database,
> either by adding or removing properties to existing models. This shouldn't
> happen after we go live with 1.0 - hopefully. We always have ideas for new
> features to be build in :)
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/2566cd3d-478b-4276-aba7-3917141ae104%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CALXYUbnxgf_BAbofbevV0Etxn9Dv-zjK3NAh3AdE7JHOHoN2ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Tobias Dacoir]

Thanks for sharing your workflow. Sounds simple enough and is more or less 
a scripted version of what I had in mind all along. 

About the manual intervention: At some points during development we had to 
provide a one-time manual entry when Django was migrating the database, 
either by adding or removing properties to existing models. This shouldn't 
happen after we go live with 1.0 - hopefully. We always have ideas for new 
features to be build in :)

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/2566cd3d-478b-4276-aba7-3917141ae104%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Andreas Kuhne]

Hi,

What I tend to do when deploying is:
1. Create a fabric script that contains a reference to the server that
should be updated (defaults to dev server).
2. Use git archive to zip the contents of a git tag to a local file.
3. Upload the file to the server
4. Stop the uwsgi process
5. Update files
6. Run migrate, collectstatic
7. Start uwsgi process
8. Delete archive file from server.
9. Done!

I really don't see when a migration would need manual updates? If that is
happening, you are probably doing migrations wrong - as someone pointed out
earlier. We are running several production workloads with this
configuration and have never needed to manually apply any migrations -
except when migrating the database from postgres to mysql.

I like the deployment process to be as automated as possible, and usually
not need any manual "help" to get it working (even to the point of being
deployed from a CI server automatically).

Regards,

Andréas

2016-02-10 14:04 GMT+01:00 Tobias Dacoir :

> Thanks for the reply.
>
> Thanks for the advice. We should include the migrate command into our git
> hook as well. But like I said, sometimes it's not always possible as manual
> interaction is required. At least for 1.0 our model should not change much,
> except for future extensions and new models.
>
> We also have a development / test server that is a direct clone of the
> future live server. However I still don't have the workflow laid out.
> Should we use KISS and just upload a zip of our new version and just
> replace the whole folder on the live server with the new code and issue an
> Apache restart afterwards (not even sure that this is needed in order to
> pick up the new code)?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-users+unsubscr...@googlegroups.com.
> To post to this group, send email to django-users@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/a8ae9869-83cb-4023-a884-91ebf8190724%40googlegroups.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CALXYUbmDgRF1hPKPt_kFn%3DWehMVj9vCCZzTy6W2Jzdb8RK0%3DLg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[Tobias Dacoir]

Thanks for the reply.

Thanks for the advice. We should include the migrate command into our git 
hook as well. But like I said, sometimes it's not always possible as manual 
interaction is required. At least for 1.0 our model should not change much, 
except for future extensions and new models.

We also have a development / test server that is a direct clone of the 
future live server. However I still don't have the workflow laid out. 
Should we use KISS and just upload a zip of our new version and just 
replace the whole folder on the live server with the new code and issue an 
Apache restart afterwards (not even sure that this is needed in order to 
pick up the new code)? 

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/a8ae9869-83cb-4023-a884-91ebf8190724%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: How to update a live / deployed Django project?

[James Schneider]

On Feb 10, 2016 12:50 AM, "Tobias Dacoir"  wrote:
>
> We've build a small web application using Django 1.8 over the past couple
of months and we are going live with version 1.0 soon (one last closed beta
test is coming up).
> So far we have deployed it on an Ubuntu VM (in Azure) using Apache /
Nginx and a local MySQL Server.
>
> I've already found some good tutorials on how to properly set it up (
http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
) however what I am mising now is how to handle updates to our code.
>
> For our development server we have a git hook that automatically deploys
new commits to the server. However that often requires manual interaction -
running manage.py makemigrations etc. Sometimes Django also does not pick
up our changed models and we have to manually fiddle with the Database in
order to get it back running. We even had times where we lost all our data
since it was easier to just start with a new database.
>

You need to fix your deployment process before going live. What are you
doing where Django doesn't catch changed models? That's a red flag for me.
Either you're doing something strange with your models, your code is not
updating the way you think it is, or you've uncovered a bug in Django and
should report it. You shouldn't need to run 'makemigrations' anywhere but
in your development environment. Are your migrations carried through as
part of your Git hook update? It sounds like they aren't. If your
development servers are seeing database changes, your production servers
should probably get those same changes in most cases. Squashing migrations
should be performed in development before the final updated deployment
package is out together and then sent to production after testing.

I'd recommend setting up test servers which are clones of your production
servers (maybe even including a snapshot of your live data). Your changes
on your development servers should be tested on the test servers, including
the deployment process. Git alone may not be enough to complete a
deployment (probably not a lot of control in a failure scenario), but
something like Fabric may help. There are lots of ways to deploy though.
Many use mass configuration managers like CFEngine, Puppet, Chef, and
Ansible.

> Obviously this can't happen once we are live and obviously we want
downtime to be as small as possible. So is there any guide on how to update
code of a live, deployed Django application?
>

Depends on the available infrastructure. For a single server, you'll incur
downtime for shutting down the web service, applying migrations, and
bringing things back online. If you have a load balancer in use, you may be
able to pull out a single real server and update it, and put it back with
little or no interruption. Obviously you'll have to account for existing
session and shared databases among servers in that case.

If your database changes are something simple like adding columns or tables
that aren't referenced in existing code, you may be able to perform the
migration live, then update the code, and reload the web server process,
incurring minimal, if any, downtime. Really depends on what your DB changes
are. I would hope that with a 1.0 non-beta release that your DB structure
would remain stable, though.

Test, test, test!

Backup, backup, backup!

-James

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/CA%2Be%2BciViVX730QiFkTCMWW3jSOj%2B_PVJFWbP9jJX8%2BmX-KUvBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

How to update a live / deployed Django project?

[Tobias Dacoir]

We've build a small web application using Django 1.8 over the past couple 
of months and we are going live with version 1.0 soon (one last closed beta 
test is coming up). 
So far we have deployed it on an Ubuntu VM (in Azure) using Apache / Nginx 
and a local MySQL Server.

I've already found some good tutorials on how to properly set it up 
(http://rogueleaderr.com/post/65157477648/the-idiomatic-guide-to-deploying-django-in#disqus_thread
 
) however what I am mising now is how to handle updates to our code.

For our development server we have a git hook that automatically deploys 
new commits to the server. However that often requires manual interaction - 
running manage.py makemigrations etc. Sometimes Django also does not pick 
up our changed models and we have to manually fiddle with the Database in 
order to get it back running. We even had times where we lost all our data 
since it was easier to just start with a new database. 

Obviously this can't happen once we are live and obviously we want downtime 
to be as small as possible. So is there any guide on how to update code of 
a live, deployed Django application?

-- 
You received this message because you are subscribed to the Google Groups 
"Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to django-users+unsubscr...@googlegroups.com.
To post to this group, send email to django-users@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/django-users/9f89dfc1-4fa5-486b-92ae-ace24238dc3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.