Re: Migrating the python-django charm to Ansible
Let me know about the shims!
Well, I've just tested and it looks like it was fixed in Ansible 1.7 at
least for host_vars/...
So much cleaner code to come!
Also, let met know what your thinking about the changes I propose on how
relations
are handle in charm-ansible-roles.
Patrick
2014-11-07 15:33 GMT-05:00 Patrick Hetu patrick.h...@gmail.com:
Specifically, this is not the config.yaml you're talking about, but
the config file which is passed during deployment right? As in,
myconfig.yaml in:
juju deploy --config myconfig.yaml mycharm
yes, that's what I meant.
[1] Using a configuration file without that charm name at the top
was working in the past but now doesn't. Maybe this could be
re-enable without to much negative impact.
That might be worth a juju bug.
Done: https://bugs.launchpad.net/juju-core/+bug/1390525
Let me know about the shims!
Yes, sorry, I forgot to mention the main problem that shim is trying to
solve.
The major reason for shims is because Ansible can't redefined an
existing variable at runtime. Ansible evaluate the yaml as a jinja2
template
first and then run it.
So a shim is an assignement trick to get the right values in the playbook.
For example, I want to use to be able to customize the django_settings
variable
in the charm configuration but set it to the basename of the working_dir in
case it not set:
- name: set settings_module if django_settings != ''
set_fact: shim_settings_module={{ django_settings }}
when: django_settings != ''
- name: set settings_module if django_settings == ''
set_fact: shim_settings_module={{ shim_working_dir | basename
}}.settings
when: django_settings == ''
I've tried, the default() filter, set_fact module and with_first_found
helpers.
They are all great tools for simulating simple if.
But redefining a variable will fail silently or trigger an infinite loop.
You can check it out with this snippet:
- hosts: localhost
vars:
a: abc
tasks:
- debug: msg={{ a }}
- set_fact: a=b
- debug: msg={{ a }}
Then running it with -e to see the problem:
ansible-playbook redefine.yml -i local -e a=xyz
[2] https://github.com/absoludity/charm-ansible-roles
I've checked those roles and wanted to contribute to it but I got blocked.
I found that tagging the tasks with Juju's hook names make it difficult to
produce a reusable role.
Because you will have to set a tag to all your tasks in the role
and always be running untagged task.
This also make the charm specific to Juju.
I've chose to use tags only in the playbook and not in roles.
I use it in the python-django charm like that:
- role: wsgi-app
tags:
- wsgi-relation-joined
- wsgi-relation-changed
- website-relation-joined
- website-relation-changed
wsgi_user: {{ django_uid }}
wsgi_group: {{ django_gid }}
working_dir: {{ shim_working_dir }}
python_path: {{ shim_python_path_list }}
app_label: {{ sanitized_unit_name }}
wsgi_application: wsgi
working_dir: {{ shim_working_dir }}
settings_module: {{ shim_settings_module }}
And for task specific to a role, I filter in the roles with a when:
- name: Open port
when: relations['website']
- name: Reload wsgi
...
when: relations['wsgi']
or a better way could be to include base on relations in the playbook like
this:
- include: wsgi_apps/tasks/relations/wsgi.yml
when: relations['wsgi']
This is far from perfect but I can't see other way to keep the role
flexible but not specific to Juju.
That *should* be done by the existing charmhelpers (it writes out
/etc/ansible/host_vars/localhost on each hook execution with config,
relations and some other things like unit name, public and private
addresses)
It does, but they're not sanitized currently. Would be a good thing to
add a sanitized version of local/remote unit names to the ansible
host_vars.
I have open a bug about that:
https://bugs.launchpad.net/charm-helpers/+bug/1390535
So we have individual charms for each services, even if running from
the same codebase. Each charm supports only the relations and config
that it needs[1]. We utilise shared ansible roles as Michael said to
reduce the charm to pretty much the bare minimum needed for this
specific service: charm config, relations, writing the service config
to disk. Most other things are taken care of by the roles.
Yes that seems to be the way to go. So people would to build a custom
django charm
by adding only pieces that they want in there playbook.
our use-case is a bit different as we're not attempting to
write one charm to deploy any django project right now (we did try in
the past, but found it was adding unreasonable complexity for us,
unless the projects being deployed were very similar).
I think the python-django charm would be there only to show all the
possible features
available by the reusable roles and people would be encorage
Re: Migrating the python-django charm to Ansible
On 6 November 2014 01:03, Michael Nelson michael.nel...@canonical.com wrote: Hi Patrick, On Thu, Nov 6, 2014 at 4:22 AM, Patrick Hetu patrick.h...@gmail.com wrote: [snip] The shim would do things like: * Translating Juju variable to role variable That can be done in the playbook where needed as above. * Sanitize variable like: unit_name, relation_name, etc That *should* be done by the existing charmhelpers (it writes out /etc/ansible/host_vars/localhost on each hook execution with config, relations and some other things like unit name, public and private addresses) It does, but they're not sanitized currently. Would be a good thing to add a sanitized version of local/remote unit names to the ansible host_vars. [snip] Then using tags to dispatch based on the current relation to the corresponding roles. I'm confused - this is what the ansible charmhelpers does right now? Build role for re-usability --- Yes... This have the potential to solve the problem of repetition of some high level tasks in other charm. Like adding ppa, You can add PPAs with the existing apt_repository module (see the example there): http://docs.ansible.com/apt_repository_module.html Yeah, we find 95% or so things we need to do ansible provides OOTB. For example, and relevant to django: http://docs.ansible.com/pip_module.html http://docs.ansible.com/django_manage_module.html We can also include roles from ansible galaxy as needed in a charm, gaining quite a bit of reuse. fetching source code, adding ssl certificate, configuring backup, etc A little bit like what charmhelper is doing right now. Yes! I think making reusable roles for common tasks in juju charms is really worthwhile [1]. We do have a bunch of public shareable roles that Simon Davy and I (mainly) are reusing and improving for some internal services [2]. Please take a look and see if there's things that could be useful - but our use-case is a bit different (for example, the payload role is because we are never pulling sourcecode branches from external repositories during deployments, instead pulling a built code asset from an internal swift container - although there would be no reason why it couldn't be extended to support both). Other roles may be useful (directories-and-permissions for listing readonly/readwrite dirs, or wsgi-app for interfacing with the gunicorn charm. A typical charm which I write for a service these days only does two things in addition to the re-used roles: installs any specific package dependencies and writes out the specific settings/config file and handle any other relations (elasticsearch/postgresql). I posted an example a while back (uses an old version of the shared roles) [3]. Also, I know that Simon has a django shared role that he's about to submit to charm-ansible-roles - it may be worth a look (Simon?), but again, our use-case is a bit different as we're not attempting to write one charm to deploy any django project right now (we did try in the past, but found it was adding unreasonable complexity for us, unless the projects being deployed were very similar). So, the django role is fairly simple, it includes tasks for syncdb, migrate, grantuser, and collectstatic. But it is based on using a migration action via juju run, as we wanted more explicit control of when migrations are run on an update, and on details of our specific use cases (we always have a different db user with elevated privs for doing migrations, the run time db user is very limited) So it needs a bit of work before it's generally consumable outside of our specific uses. Regards the framework charm idea, we found that each of our django services, while mostly similar, had different config options and relations we wanted to expose. We'd previously had experience with a mega-charm (the u1 charm deployed 10 or so different services from the same codebase, it had 42 relations!), and it wasn't that fun. Having to add all possible config options, and all possible relations was messy. Plus, then we'd have some charms that would need to relate to themselves, so you would need to implement both sides of the relation hook in the same charm, which was not my idea of a good time. So we have individual charms for each services, even if running from the same codebase. Each charm supports only the relations and config that it needs[1]. We utilise shared ansible roles as Michael said to reduce the charm to pretty much the bare minimum needed for this specific service: charm config, relations, writing the service config to disk. Most other things are taken care of by the roles. Another way to do this is to use a subordinate, which I think the django charm currently supports? But limitations around subordinates (i.e. not being able to remove relations, juju run not working, etc[1]) has meant we didn't take this route. That, plus the shared roles has worked well for us. HTH -- Simon --
Re: Migrating the python-django charm to Ansible
Specifically, this is not the config.yaml you're talking about, but
the config file which is passed during deployment right? As in,
myconfig.yaml in:
juju deploy --config myconfig.yaml mycharm
yes, that's what I meant.
[1] Using a configuration file without that charm name at the top
was working in the past but now doesn't. Maybe this could be
re-enable without to much negative impact.
That might be worth a juju bug.
Done: https://bugs.launchpad.net/juju-core/+bug/1390525
Let me know about the shims!
Yes, sorry, I forgot to mention the main problem that shim is trying to
solve.
The major reason for shims is because Ansible can't redefined an
existing variable at runtime. Ansible evaluate the yaml as a jinja2 template
first and then run it.
So a shim is an assignement trick to get the right values in the playbook.
For example, I want to use to be able to customize the django_settings
variable
in the charm configuration but set it to the basename of the working_dir in
case it not set:
- name: set settings_module if django_settings != ''
set_fact: shim_settings_module={{ django_settings }}
when: django_settings != ''
- name: set settings_module if django_settings == ''
set_fact: shim_settings_module={{ shim_working_dir | basename
}}.settings
when: django_settings == ''
I've tried, the default() filter, set_fact module and with_first_found
helpers.
They are all great tools for simulating simple if.
But redefining a variable will fail silently or trigger an infinite loop.
You can check it out with this snippet:
- hosts: localhost
vars:
a: abc
tasks:
- debug: msg={{ a }}
- set_fact: a=b
- debug: msg={{ a }}
Then running it with -e to see the problem:
ansible-playbook redefine.yml -i local -e a=xyz
[2] https://github.com/absoludity/charm-ansible-roles
I've checked those roles and wanted to contribute to it but I got blocked.
I found that tagging the tasks with Juju's hook names make it difficult to
produce a reusable role.
Because you will have to set a tag to all your tasks in the role
and always be running untagged task.
This also make the charm specific to Juju.
I've chose to use tags only in the playbook and not in roles.
I use it in the python-django charm like that:
- role: wsgi-app
tags:
- wsgi-relation-joined
- wsgi-relation-changed
- website-relation-joined
- website-relation-changed
wsgi_user: {{ django_uid }}
wsgi_group: {{ django_gid }}
working_dir: {{ shim_working_dir }}
python_path: {{ shim_python_path_list }}
app_label: {{ sanitized_unit_name }}
wsgi_application: wsgi
working_dir: {{ shim_working_dir }}
settings_module: {{ shim_settings_module }}
And for task specific to a role, I filter in the roles with a when:
- name: Open port
when: relations['website']
- name: Reload wsgi
...
when: relations['wsgi']
or a better way could be to include base on relations in the playbook like
this:
- include: wsgi_apps/tasks/relations/wsgi.yml
when: relations['wsgi']
This is far from perfect but I can't see other way to keep the role
flexible but not specific to Juju.
That *should* be done by the existing charmhelpers (it writes out
/etc/ansible/host_vars/localhost on each hook execution with config,
relations and some other things like unit name, public and private
addresses)
It does, but they're not sanitized currently. Would be a good thing to
add a sanitized version of local/remote unit names to the ansible
host_vars.
I have open a bug about that:
https://bugs.launchpad.net/charm-helpers/+bug/1390535
So we have individual charms for each services, even if running from
the same codebase. Each charm supports only the relations and config
that it needs[1]. We utilise shared ansible roles as Michael said to
reduce the charm to pretty much the bare minimum needed for this
specific service: charm config, relations, writing the service config
to disk. Most other things are taken care of by the roles.
Yes that seems to be the way to go. So people would to build a custom
django charm
by adding only pieces that they want in there playbook.
our use-case is a bit different as we're not attempting to
write one charm to deploy any django project right now (we did try in
the past, but found it was adding unreasonable complexity for us,
unless the projects being deployed were very similar).
I think the python-django charm would be there only to show all the
possible features
available by the reusable roles and people would be encorage to build a
charm per site.
Patrick
--
Juju mailing list
Juju@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju
Migrating the python-django charm to Ansible
Since the last UDS, I'm trying to make the python-django charm evolve to use Ansible for is various tasks instead of using pure Python scripts. My previous attempt: lp:~patrick-hetu/charms/precise/python-django/ansible had some problems so I decided to restart from scratch here (work in progress): lp:~patrick-hetu/charms/precise/python-django/ansible_reboot Based on the following observations. Use roles that are not specific to Juju --- If you look at an Ansible role you will probably found a file defaults/main.yml or vars/main.yml. For me it was stunning how close those are to a charm configuration file. So why not grab this opportunity to use the same file (minus the charm name at the top [1]) with both Juju and Ansible and make Juju use the same roles that Ansible community are using by introducing shims in the charm playbook. [1] Using a configuration file without that charm name at the top was working in the past but now doesn't. Maybe this could be re-enable without to much negative impact. Couple the playbook with Juju with a shim - To get the playbook to work with an external role we could use a shim file that translate Juju's dynamic configuration and relation data to roles variables. I was thinking using a shim imported in the pre_tasks section of the charm's playbook that runs on all tags. (see: https://github.com/ansible/ansible/issues/3157) The shim would do things like: * Translating Juju variable to role variable * Sanitize variable like: unit_name, relation_name, etc * Ensure backward compatiblity Then using tags to dispatch based on the current relation to the corresponding roles. Build role for re-usability --- This have the potential to solve the problem of repetition of some high level tasks in other charm. Like adding ppa, fetching source code, adding ssl certificate, configuring backup, etc A little bit like what charmhelper is doing right now. I might not be clear so don't be afraid to ask clarification. I'm looking forward to receive your feedbacks Patrick -- Juju mailing list Juju@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju
Re: Migrating the python-django charm to Ansible
Hi Patrick,
On Thu, Nov 6, 2014 at 4:22 AM, Patrick Hetu patrick.h...@gmail.com wrote:
Since the last UDS, I'm trying to make the python-django charm evolve
to use Ansible for is various tasks instead of using pure Python scripts.
My previous attempt:
lp:~patrick-hetu/charms/precise/python-django/ansible
had some problems so I decided to restart from scratch here (work in
progress):
lp:~patrick-hetu/charms/precise/python-django/ansible_reboot
Based on the following observations.
Use roles that are not specific to Juju
---
If you look at an Ansible role you will probably found a file
defaults/main.yml or vars/main.yml. For me it was stunning how close
those are to a charm configuration file.
Specifically, this is not the config.yaml you're talking about, but
the config file which is passed during deployment right? As in,
myconfig.yaml in:
juju deploy --config myconfig.yaml mycharm
?
So why not grab this opportunity to use the same file
(minus the charm name at the top [1]) with both Juju and Ansible
and make Juju use the same roles that Ansible community are using by
introducing
shims in the charm playbook.
This should be possible without needing to re-use the actual file
itself - if I've understood correctly what you mean: any ansible-based
juju charm can reuse any community role for free, by using a shim to
set the roles expected variable names, based on the charm config?
So looking at the examples in your branch, if your charm has a config
option django-settings (and therefore playbook and templates have {{
django_settings }} available via the current helpers), while an
ansible-community django-project role expects a settings_module
variable, your example shim.yml does:
set_fact: shim_settings_module={{ django_settings }}
and then when you use the role, you pass in the the shim_settings_module:
- role: django-project
tags:
- install
django_settings: {{ shim_settings_module }}
But I don't see the need for the shim at all, when you can just pass
the charm config value straight through:
django_settings: {{ django_settings }}
(or in the general case: django_settings: {{ whatever_my_charm_config_uses }})
I'm not sure whether I'm missing something about what you're meaning
(re-using the actual file), or whether you may have missed that all
the config is already available in the playbook/template contexts via
the existing charmhelper ansible support (unlikely, as you've used it
before I think). Let me know what I'm missing.
[1] Using a configuration file without that charm name at the top
was working in the past but now doesn't. Maybe this could be
re-enable without to much negative impact.
That might be worth a juju bug. I assume it's related to the config
file being used generally to store config for multiple services within
your setup, but I don't see any reason why that should mean that if
the config file without the charm (service) name at the top is used
for deploying a single charm, that juju couldn't still accept it as
the config for that service.
Couple the playbook with Juju with a shim
-
To get the playbook to work with an external role we could use
a shim file that translate Juju's dynamic configuration
and relation data to roles variables.
Sounds great to be able to reuse existing external roles, but let me
know what I'm missing about why a shim is needed to do this.
I was thinking using a shim imported in the pre_tasks section of
the charm's playbook that runs on all tags.
(see: https://github.com/ansible/ansible/issues/3157)
The shim would do things like:
* Translating Juju variable to role variable
That can be done in the playbook where needed as above.
* Sanitize variable like: unit_name, relation_name, etc
That *should* be done by the existing charmhelpers (it writes out
/etc/ansible/host_vars/localhost on each hook execution with config,
relations and some other things like unit name, public and private
addresses)
* Ensure backward compatiblity
Then using tags to dispatch based on the current relation
to the corresponding roles.
Build role for re-usability
---
Yes...
This have the potential to solve the problem of repetition of some
high level tasks in other charm.
Like adding ppa,
You can add PPAs with the existing apt_repository module (see the
example there):
http://docs.ansible.com/apt_repository_module.html
fetching source code, adding ssl certificate,
configuring backup, etc
A little bit like what charmhelper is doing right now.
Yes! I think making reusable roles for common tasks in juju charms is
really worthwhile [1]. We do have a bunch of public shareable roles
that Simon Davy and I (mainly) are reusing and improving for some
internal services [2]. Please take a look and see if there's things
that could be useful - but our use-case is a bit different (for
