Hmm, no replies. Damn.
Here's my solution, in case anyone else has a similar problem.
I realised I can skip the approach of writing to disk, and keep it all in
memory.
So I have a task run by all 'lambda' hosts, which is:
set_fact:
this_host_vars: "{{ hostvars[inventory_hostname] | dict_filter(filter_by)
| to_yaml | from_yaml }}"
Where `dict_filter` is a custom filter I wrote to do
x = {k:v for k,v in d.items() if k in filter_by}
Then on the single main host, I do:
with_items: "{{ groups.lambdas }}"
vars:
to_add: "{{ { item: hostvars[item]['this_host_vars'] } }}"
set_fact:
lambda_confs: "{{ lambda_confs | default({}) | combine(to_add) }}"
For two dozen 'lambdas', it takes a few seconds to do these two tasks.
Regards,
Matt
On Wednesday, April 22, 2020 at 1:31:24 PM UTC+10, Matthew Davis wrote:
>
> I have a use case which is a bit different to most, but Ansible seems to
> do a pretty good job.
> I'm trying to figure out the best way to leverage Ansible's inherent
> concurrency for multiple hosts, when deploying to serverless
> infrastructure, which has no hosts. (Instead just using tasks which do API
> calls on localhost.)
>
> # The Task
>
> I'm building, testing and deploying a project using Ansible.
>
> My target infrastructure is all serverless. I'm deploying code to AWS
> Lambda functions, with CloudFormation. So there are no 'hosts' for Ansible
> to connect to.
>
> I have something currently which works, but is slow and a bit messy.
>
> I've got a big playbook, all connecting to one host, which is 'localhost'
> (connection=local).
> I've got a yaml file with a list of my lambda functions and relevant
> config (e.g. timeout, name, comment, environment variables etc).
>
>
> I've got one role for all my Lambda function stuff.
>
> 1. Use `include_vars` to load in aforementioned variable file
> 2. using `with_dict` and the variable from that file, `pip install`
> the modules each one depends on,
> 3. copy over the code for each Lambda function to where those
> dependencies where installed, also with_dict
> 4. run a unit test for each (shell: python main.py), also with_dict
> 5. zip up each folder, also with_dict
> 6. upload each zip to S3, using with_nested (for each lambda function,
> for each region)
>
> In particular the uploading is slow, because it's in series. I'd like to
> do it in parallel.
>
> Then I have other roles for other things which are not per-lambda. (e.g.
> deploying cloudformation templates, configuring a firewall etc.)
>
> # The problem
>
> Inside that role for Lambda stuff, almost every task has with_dict or
> with_items, or sometimes with_nested (e.g. upload each lambda zip for each
> of multiple regions), in addition to extensive `when` conditions. (I made
> it so I can pass in `-e only_lambda=x` to Ansible, to skip tasks for
> Lambdas I haven't changed.)
> This ends up quite messy.
>
> Overall the whole thing is quite slow too, because it does everything in
> series not parallel. Some things are I/O bound, and could be sped up a lot
> by doing them concurrently.
> e.g. I used `async` to do the upload in parallel.
>
> Another thing that get's messy is that I want to keep that variable file
> with lambda config minimal. I don't want to copy-paste boiler plate for
> each one. (It's already too huge.)
> So each lambda has a field called 'Name'.
> I want to add another field like: `local_zip_name: "{{ build_dir }}/{{
> self.cf_name }}.zip".
> Modifying a dict in Ansible is surprisingly messy. Currently I do this
> using `set_fact`.
> Although it turns out that if you load the variable from a file with `-e
> @file.yaml`, it seems that changes with `set_fact` don't take effect.
> So I have to use an include_vars task instead or loading a var file with
> command line arguments.
>
> # Attempted solution
>
> I'm trying to add each 'lambda function' to static inventory.
> Each one being localhost, but having unique host variables equal to what I
> had in that yaml file.
> This way I can leverage all the work Ansible devs have already done to do
> things concurrently.
>
> It turns out you can omit the IP address, and it defaults to localhost
> anyway, and Ansible will happily work with multiple hosts that are actually
> the exact same host.
> It all just works. Yay!
> I can get rid of all the `with_items`, `with_dict` etc, and just run the
> role against the group of hosts.
> And I can reduce the number of tasks by moving set_fact definitions into
> host definitions.
>
> Then for other tasks which I only need to do once, I run those roles with
> a different host group, that's one Ansible host connecting to localhost.
>
> This host approach also makes it nice to set things like `local_zip_fname`
> (mentioned above). I can set that variable once for the whole group, and
> lazy variable evaluation means it's evaluated differently for each host.
> This means I don't have to worry about dependency issues about which
> variables are defined first, and there's one place for all the default
> variables.
>
> But, there are 2 issues.
>
> 1. I want to reference that `local_zip_fname` variable (defined using
> Jinja2 templating for each Lambda host) from the other host (one host,
> localhost). But `local_zip_fname` is not defined for that other host. So I
> tried the thing where you dig into `host_vars['other_host']['my_var']` to
> get it. But now lazy variable evaluation bites back. The variable I want
> (`local_zip_fname`) includes Jinja2 templating for another variable
> (cf_name), which was present on those other hosts (one per Lambda), but not
> this host.
>
> Question: How can I access variables for other hosts, forcing Jinja2
> evaluation based on that other host's environment?
> The only solution I can think of is to save the variables to disk for each
> of those other hosts (using | to_yaml to force Jinja evaluation), and then
> loading that file from the host I want to read the variables. Is there a
> better way?
>
> The other issue is that the speed improvement is no where near as good as
> I hoped. If I deploy from a beefy machine with 8 CPUs instead of my usual
> 2, it seems to speed up the relevant tasks by a factor of 2. (Not by a
> factor of 8/2). If I try to deploy with my usual 2 CPU, there's no
> noticeable difference in speed.
> It seems that there is a lot of overhead per host, which is taking away
> from the speed increase from converting sequential `with_items` to
> concurrent hosts.
>
> I've tried to optimise things with:
> gathering=no
> or using fact caching,
> and I've already set forks=50 (More than the number of hosts).
> And I enabled pipelining (which probably does nothing for connection=local)
>
> Is there any other way to optimise this? Is there a way to tell Ansible
> that these 30 hosts are actually all the same machine, so it only needs to
> do some things once not 30 times?
>
> Or is there some other approach to solve my problem? Is this the x y
> problem <http://xyproblem.info/>?
>
>
> Thanks,
> Matt
>
> (GH: matt-telstra and mlda065)
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/51a91203-1602-41ec-9ad4-ac32088d11d8%40googlegroups.com.
