Re: Migrating the python-django charm to Ansible
On 6 November 2014 01:03, Michael Nelson michael.nel...@canonical.com wrote: Hi Patrick, On Thu, Nov 6, 2014 at 4:22 AM, Patrick Hetu patrick.h...@gmail.com wrote: [snip] The shim would do things like: * Translating Juju variable to role variable That can be done in the playbook where needed as above. * Sanitize variable like: unit_name, relation_name, etc That *should* be done by the existing charmhelpers (it writes out /etc/ansible/host_vars/localhost on each hook execution with config, relations and some other things like unit name, public and private addresses) It does, but they're not sanitized currently. Would be a good thing to add a sanitized version of local/remote unit names to the ansible host_vars. [snip] Then using tags to dispatch based on the current relation to the corresponding roles. I'm confused - this is what the ansible charmhelpers does right now? Build role for re-usability --- Yes... This have the potential to solve the problem of repetition of some high level tasks in other charm. Like adding ppa, You can add PPAs with the existing apt_repository module (see the example there): http://docs.ansible.com/apt_repository_module.html Yeah, we find 95% or so things we need to do ansible provides OOTB. For example, and relevant to django: http://docs.ansible.com/pip_module.html http://docs.ansible.com/django_manage_module.html We can also include roles from ansible galaxy as needed in a charm, gaining quite a bit of reuse. fetching source code, adding ssl certificate, configuring backup, etc A little bit like what charmhelper is doing right now. Yes! I think making reusable roles for common tasks in juju charms is really worthwhile [1]. We do have a bunch of public shareable roles that Simon Davy and I (mainly) are reusing and improving for some internal services [2]. Please take a look and see if there's things that could be useful - but our use-case is a bit different (for example, the payload role is because we are never pulling sourcecode branches from external repositories during deployments, instead pulling a built code asset from an internal swift container - although there would be no reason why it couldn't be extended to support both). Other roles may be useful (directories-and-permissions for listing readonly/readwrite dirs, or wsgi-app for interfacing with the gunicorn charm. A typical charm which I write for a service these days only does two things in addition to the re-used roles: installs any specific package dependencies and writes out the specific settings/config file and handle any other relations (elasticsearch/postgresql). I posted an example a while back (uses an old version of the shared roles) [3]. Also, I know that Simon has a django shared role that he's about to submit to charm-ansible-roles - it may be worth a look (Simon?), but again, our use-case is a bit different as we're not attempting to write one charm to deploy any django project right now (we did try in the past, but found it was adding unreasonable complexity for us, unless the projects being deployed were very similar). So, the django role is fairly simple, it includes tasks for syncdb, migrate, grantuser, and collectstatic. But it is based on using a migration action via juju run, as we wanted more explicit control of when migrations are run on an update, and on details of our specific use cases (we always have a different db user with elevated privs for doing migrations, the run time db user is very limited) So it needs a bit of work before it's generally consumable outside of our specific uses. Regards the framework charm idea, we found that each of our django services, while mostly similar, had different config options and relations we wanted to expose. We'd previously had experience with a mega-charm (the u1 charm deployed 10 or so different services from the same codebase, it had 42 relations!), and it wasn't that fun. Having to add all possible config options, and all possible relations was messy. Plus, then we'd have some charms that would need to relate to themselves, so you would need to implement both sides of the relation hook in the same charm, which was not my idea of a good time. So we have individual charms for each services, even if running from the same codebase. Each charm supports only the relations and config that it needs[1]. We utilise shared ansible roles as Michael said to reduce the charm to pretty much the bare minimum needed for this specific service: charm config, relations, writing the service config to disk. Most other things are taken care of by the roles. Another way to do this is to use a subordinate, which I think the django charm currently supports? But limitations around subordinates (i.e. not being able to remove relations, juju run not working, etc[1]) has meant we didn't take this route. That, plus the shared roles has worked well for us. HTH -- Simon --
[Review Queue] zend-server
[1] zend-server Following up on a very thorough review performed by Matt Bruzek, I spent some time reviewing the latest changes of zend-server. The charm is really shaping up well, and I’ll be excited to see it included in the Juju Charm Store. The charm deployed successfully if done without its optional relation to MySQL, but that relation, required for clustering, was breaking during my tests. Additionally, whilst unit tests have been added, they failed when run via bundletester. I’ve left my feedback on github for the author, and hope to see more from him soon. https://github.com/afroyd/Zend-Server-juju/issues/6 -- Adam Israel-- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Migrating the python-django charm to Ansible
Specifically, this is not the config.yaml you're talking about, but
the config file which is passed during deployment right? As in,
myconfig.yaml in:
juju deploy --config myconfig.yaml mycharm
yes, that's what I meant.
[1] Using a configuration file without that charm name at the top
was working in the past but now doesn't. Maybe this could be
re-enable without to much negative impact.
That might be worth a juju bug.
Done: https://bugs.launchpad.net/juju-core/+bug/1390525
Let me know about the shims!
Yes, sorry, I forgot to mention the main problem that shim is trying to
solve.
The major reason for shims is because Ansible can't redefined an
existing variable at runtime. Ansible evaluate the yaml as a jinja2 template
first and then run it.
So a shim is an assignement trick to get the right values in the playbook.
For example, I want to use to be able to customize the django_settings
variable
in the charm configuration but set it to the basename of the working_dir in
case it not set:
- name: set settings_module if django_settings != ''
set_fact: shim_settings_module={{ django_settings }}
when: django_settings != ''
- name: set settings_module if django_settings == ''
set_fact: shim_settings_module={{ shim_working_dir | basename
}}.settings
when: django_settings == ''
I've tried, the default() filter, set_fact module and with_first_found
helpers.
They are all great tools for simulating simple if.
But redefining a variable will fail silently or trigger an infinite loop.
You can check it out with this snippet:
- hosts: localhost
vars:
a: abc
tasks:
- debug: msg={{ a }}
- set_fact: a=b
- debug: msg={{ a }}
Then running it with -e to see the problem:
ansible-playbook redefine.yml -i local -e a=xyz
[2] https://github.com/absoludity/charm-ansible-roles
I've checked those roles and wanted to contribute to it but I got blocked.
I found that tagging the tasks with Juju's hook names make it difficult to
produce a reusable role.
Because you will have to set a tag to all your tasks in the role
and always be running untagged task.
This also make the charm specific to Juju.
I've chose to use tags only in the playbook and not in roles.
I use it in the python-django charm like that:
- role: wsgi-app
tags:
- wsgi-relation-joined
- wsgi-relation-changed
- website-relation-joined
- website-relation-changed
wsgi_user: {{ django_uid }}
wsgi_group: {{ django_gid }}
working_dir: {{ shim_working_dir }}
python_path: {{ shim_python_path_list }}
app_label: {{ sanitized_unit_name }}
wsgi_application: wsgi
working_dir: {{ shim_working_dir }}
settings_module: {{ shim_settings_module }}
And for task specific to a role, I filter in the roles with a when:
- name: Open port
when: relations['website']
- name: Reload wsgi
...
when: relations['wsgi']
or a better way could be to include base on relations in the playbook like
this:
- include: wsgi_apps/tasks/relations/wsgi.yml
when: relations['wsgi']
This is far from perfect but I can't see other way to keep the role
flexible but not specific to Juju.
That *should* be done by the existing charmhelpers (it writes out
/etc/ansible/host_vars/localhost on each hook execution with config,
relations and some other things like unit name, public and private
addresses)
It does, but they're not sanitized currently. Would be a good thing to
add a sanitized version of local/remote unit names to the ansible
host_vars.
I have open a bug about that:
https://bugs.launchpad.net/charm-helpers/+bug/1390535
So we have individual charms for each services, even if running from
the same codebase. Each charm supports only the relations and config
that it needs[1]. We utilise shared ansible roles as Michael said to
reduce the charm to pretty much the bare minimum needed for this
specific service: charm config, relations, writing the service config
to disk. Most other things are taken care of by the roles.
Yes that seems to be the way to go. So people would to build a custom
django charm
by adding only pieces that they want in there playbook.
our use-case is a bit different as we're not attempting to
write one charm to deploy any django project right now (we did try in
the past, but found it was adding unreasonable complexity for us,
unless the projects being deployed were very similar).
I think the python-django charm would be there only to show all the
possible features
available by the reusable roles and people would be encorage to build a
charm per site.
Patrick
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
Charm Reviews + Activity
Hi -- This week, I reviewed a test charm addition from Matt and a version bump charm from Charles (which needed information). In reviewing the new apache2 test, I found once annoyance. `juju test` bootstraps and tears down between each step, and with all these new tests added en masse there seems to be a practice emerging of using a '00_setup' script to install dependencies. This in effect inserts an extra bootstrap unnecessarily into the test process. @Marco or others -- thoughts about putting an exception or machinery in to get rid of this? I was going to go through a lot of these other additions basic test additions, but wanted to get the question out there before I did. In other charm related activity, I also worked on getting real bundles created for the Landscape charm (submitted into the ~landscape namespace as of now), will get them further cleaned up next week and submitted for review. Thanks! -- David Britton david.brit...@canonical.com -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Someone fixed the security issue with --debug?
I am comparing the use of streams during the bootstrap of 1.20 and 1.21. I noticed that 1.21 no longer dumps the content of the cloud-init script, which has user credentials and machine keys, implicitly fixing this bug --debug dumps sensitive information to terminal https://bugs.launchpad.net/juju-core/+bug/1289038 If we can guarantee that --debug will never dump the content of the script, agent config, and jenv files, we can mark this bug fixed. Juju CI and also enable --debug for better logs too. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
