Re: [ansible-project] Loopless alternative?
I ended up with the inline jinja for loop.
Also I used a pipe lookup instead of an s3 list task prior to this one.
So now everything is done in a one single task, without noise, and with the
same results.
Thanks everyone, it was really useful and I will keep it in mind for future
reference.
Dick
On Thu, 17 Aug 2023 at 20:30, Vladimir Botka wrote:
> On Thu, 17 Aug 2023 13:11:25 -0400
> Brian Coca wrote:
>
> > see map/select/reject filters .. they are actually loops and normally
> > much simpler than using jinja command syntax ( {% %} ).
>
> Unfortunately, some filters are not *map* friendly. For example, the
> filter *product*
>
> list1|product(list2) .. works fine
> list1|zip(list2)|map('product') ... does not work
>
>
> Details: Given the list
>
> l1:
> - dir: /tmp/test/d1
> sub_dir: [a, b]
> - dir: /tmp/test/d2
> sub_dir: [a, b, c]
>
> the goal is to create the list of products
>
> l2:
> - /tmp/test/d1/a
> - /tmp/test/d1/b
> - /tmp/test/d2/a
> - /tmp/test/d2/b
> - /tmp/test/d2/c
>
> The iteration (the filter *subelements* not used
> to demonstrate the functionality of *product*)
>
> - debug:
> msg: "{{ [item.0]|product(item.1) }}"
> loop: "{{ dirs|zip(sdirs) }}"
> vars:
> dirs: "{{ l1|map(attribute='dir') }}"
> sdirs: "{{ l1|map(attribute='sub_dir') }}"
>
> works as expected. Gives (abridged)
>
> msg:
> - - /tmp/test/d1
> - a
> - - /tmp/test/d1
> - b
>
> msg:
> - - /tmp/test/d2
> - a
> - - /tmp/test/d2
> - b
> - - /tmp/test/d2
> - c
>
> But, the filter *product* doesn't work with *map*
>
> dirs: "{{ l1|map(attribute='dir') }}"
> sdirs: "{{ l1|map(attribute='sub_dir') }}"
> l3: "{{ dirs|zip(sdirs)|map('product') }}"
>
> gives
>
> l3:
> - - - /tmp/test/d1
> - - - a
> - b
> - - - /tmp/test/d2
> - - - a
> - b
> - c
>
> This leaves you with Jinja if you want to avoid the loops in tasks
>
> l3: |
> {% filter from_yaml %}
> {% for i in l1 %}
> {% for s in i.sub_dir %}
> - {{ i.dir }}/{{ s }}
> {% endfor %}
> {% endfor %}
> {% endfilter %}
>
> --
> Vladimir Botka
>
> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-project+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/20230817202944.3f3512c8%40gmail.com
> .
>
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAF8BbLa3GKSz-vwjTdUYN54pM8nv-LaKBG4gQ1q172gv66BY%2Bg%40mail.gmail.com.
Re: [ansible-project] Loopless alternative?
On Thu, 17 Aug 2023 13:11:25 -0400
Brian Coca wrote:
> see map/select/reject filters .. they are actually loops and normally
> much simpler than using jinja command syntax ( {% %} ).
Unfortunately, some filters are not *map* friendly. For example, the
filter *product*
list1|product(list2) .. works fine
list1|zip(list2)|map('product') ... does not work
Details: Given the list
l1:
- dir: /tmp/test/d1
sub_dir: [a, b]
- dir: /tmp/test/d2
sub_dir: [a, b, c]
the goal is to create the list of products
l2:
- /tmp/test/d1/a
- /tmp/test/d1/b
- /tmp/test/d2/a
- /tmp/test/d2/b
- /tmp/test/d2/c
The iteration (the filter *subelements* not used
to demonstrate the functionality of *product*)
- debug:
msg: "{{ [item.0]|product(item.1) }}"
loop: "{{ dirs|zip(sdirs) }}"
vars:
dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"
works as expected. Gives (abridged)
msg:
- - /tmp/test/d1
- a
- - /tmp/test/d1
- b
msg:
- - /tmp/test/d2
- a
- - /tmp/test/d2
- b
- - /tmp/test/d2
- c
But, the filter *product* doesn't work with *map*
dirs: "{{ l1|map(attribute='dir') }}"
sdirs: "{{ l1|map(attribute='sub_dir') }}"
l3: "{{ dirs|zip(sdirs)|map('product') }}"
gives
l3:
- - - /tmp/test/d1
- - - a
- b
- - - /tmp/test/d2
- - - a
- b
- c
This leaves you with Jinja if you want to avoid the loops in tasks
l3: |
{% filter from_yaml %}
{% for i in l1 %}
{% for s in i.sub_dir %}
- {{ i.dir }}/{{ s }}
{% endfor %}
{% endfor %}
{% endfilter %}
--
Vladimir Botka
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/20230817202944.3f3512c8%40gmail.com.
pgpzQmp4GVXoP.pgp
Description: OpenPGP digital signature
Re: [ansible-project] Loopless alternative?
see map/select/reject filters .. they are actually loops and normally
much simpler than using jinja command syntax ( {% %} ).
--
--
Brian Coca
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CACVha7cCi7ZURZdzutxvi5vsCDsUyFrB757n3Q-fTcHFyQo0Tg%40mail.gmail.com.
Re: [ansible-project] Loopless alternative?
On Thu, 17 Aug 2023 07:56:58 -0400
Todd Lewis wrote:
>bn:
>- - basename
> - dev_wss_db
>- - basename
> - dev_wss_db_requests
>- - basename
> - dev_bss_service_database
>- - basename
> - dev_bss_frontend_db
>- - basename
> - dev_mss_db
>
> But I didn't find a way to map that using "community.general.dict" to create
>
>bn:
>- basename: dev_wss_db
>- basename: dev_wss_db_requests
>- basename: dev_bss_service_database
>- basename: dev_bss_frontend_db
>- basename: dev_mss_db
You can always use brute-force Jinja as the last resort. For example,
given the list
bn_list:
- dev_wss_db
- dev_wss_db_requests
- dev_bss_service_database
- dev_bss_frontend_db
- dev_mss_db
the below Jinja creates the list of the hashes
bn: |
{% filter from_yaml %}
{% for basename in bn_list %}
- basename: {{ basename }}
{% endfor %}
{% endfilter %}
As a side-note, this is equivalent to
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"
> This for me is one of the more frustrating things about Jinja
> pipelines. I keep wishing "map" would take arbitrary
> expressions rather than the limited set it's stuck with.
This is very good point. It would be possible to write such a filter.
However, I'm not sure about the security implications.
--
Vladimir Botka
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/20230817170740.038163cc%40gmail.com.
pgpOgqpg7TAic.pgp
Description: OpenPGP digital signature
Re: [ansible-project] Loopless alternative?
Thanks, Vladimir. I had missed the point of "community.general.dict_kv".
I had gotten as far as this:
- set_fact:
bn: |
{{ query('ansible.builtin.nested', ['basename'], (all_objects
| map(attribute='Key')
| map('regex_replace', '^' ~ prefix ~ '\d{10}_(.*)\.pgdump',
'\1'))) }}
which produces:
bn:
- - basename
- dev_wss_db
- - basename
- dev_wss_db_requests
- - basename
- dev_bss_service_database
- - basename
- dev_bss_frontend_db
- - basename
- dev_mss_db
But I didn't find a way to map that using "community.general.dict" to create
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
This for me is one of the more frustrating things about Jinja pipelines.
I keep wishing "map" would take arbitrary expressions rather than the
limited set it's stuck with. So you end up with a fleet of one-off
filters like "community.general.dict_kv" which does what
"community.general.dict" would do if there were an obvious way to turn this:
- - basename
- dev_wss_db
- - basename
- dev_wss_db_requests
- - basename
- dev_bss_service_database
- - basename
- dev_bss_frontend_db
- - basename
- dev_mss_db
into this:
- - - basename
- dev_wss_db
- - - basename
- dev_wss_db_requests
- - - basename
- dev_bss_service_database
- - - basename
- dev_bss_frontend_db
- - - basename
- dev_mss_db
i.e. a "deepen" counterpart to "flatten". But that magical incantation
has so far eluded me.
--
Todd
On 8/16/23 6:46 PM, Vladimir Botka wrote:
Create the list of the hashes
bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"
gives
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
zip the lists and combine the items
backup_object: "{{ all_objects|zip(bn)|map('combine') }}"
--
You received this message because you are subscribed to the Google Groups "Ansible
Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/d783a813-746b-c961-eca7-76ba6bccc6aa%40gmail.com.
Re: [ansible-project] Loopless alternative?
On Thu, 17 Aug 2023 00:46:08 +0200
Vladimir Botka wrote:
> Create the list of the hashes
>
> bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
> bn: "{{ all_objects|
> map(attribute='Key')|
> map('regex_replace', bn_regex, '\\1')|
> map('community.general.dict_kv', 'basename') }}"
>
> gives
>
> bn:
> - basename: dev_wss_db
> - basename: dev_wss_db_requests
> - basename: dev_bss_service_database
> - basename: dev_bss_frontend_db
> - basename: dev_mss_db
There are options. Filter *basename* and get rid of *prefix*
bn_regex: '^\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"
Create the hash in *regex_replace* and use *from_yaml* instead
of *community.general.dict_kv*
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('regex_replace', bn_regex, '{basename: \\1}')|
map('from_yaml') }}"
Instead of *regex_replace* use *splitext* and *split*
bn: "{{ all_objects|
map(attribute='Key')|
map('basename')|
map('splitext')|map('first')|
map('split', '_', 1)|map('last')|
map('community.general.dict_kv', 'basename') }}"
--
Vladimir Botka
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/20230817013426.387bc3d6%40gmail.com.
pgpIJbMNdKKpL.pgp
Description: OpenPGP digital signature
Re: [ansible-project] Loopless alternative?
Create the list of the hashes
bn_regex: '^{{ prefix }}\d{10}_(.*)\.pgdump$'
bn: "{{ all_objects|
map(attribute='Key')|
map('regex_replace', bn_regex, '\\1')|
map('community.general.dict_kv', 'basename') }}"
gives
bn:
- basename: dev_wss_db
- basename: dev_wss_db_requests
- basename: dev_bss_service_database
- basename: dev_bss_frontend_db
- basename: dev_mss_db
zip the lists and combine the items
backup_object: "{{ all_objects|zip(bn)|map('combine') }}"
--
Vladimir Botka
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/20230817004608.1a07c1ff%40gmail.com.
pgpFTDB_f6ZbO.pgp
Description: OpenPGP digital signature
Re: [ansible-project] Loopless alternative?
Here's the best I've got to offer. I tried pipelining filters etc but
couldn't quite get over the hump. So, back to old school loops.
- set_fact:
backup_objects: |
{% set result = [] %}
{% for obj in all_objects %}
{% set _ = result.append({'basename': obj['Key'] |
regex_replace('^' ~ prefix ~ '\\d{10}_(.*)\.pgdump', '\\1')} | combine(obj)) %}
{% endfor %}{{ result }}
The point of the "set _" line is for the side effect of appending
revised objects to the "result" list. Otherwise it's pretty straightforward.
Cheers,
--
Todd
On 8/16/23 10:52 AM, Dick Visser wrote:
Hii
I have a list of dicts, where I want to inject a number of helper keys.
I can do this with set_facts in a loop. Example playbook (hopefully
this displays OK):
---
- name:Add helper keys to list of dicts
hosts:localhost
connection:local
gather_facts:no
tasks:
- set_fact:
backup_objects:"{{ backup_objects|default([])|union(
[
item|combine(
{
'basename':item.Key|regex_replace('^'~ prefix~
'\\d{10}_(.*)\\.pgdump','\\1')
}
)
]
)
}}"
loop:"{{ all_objects}}"
loop_control:
label:"{{ item.Key}}"
- debug:var=backup_objects
vars:
prefix:backup/database/
all_objects:
- ETag:'"d41d8cd98f00b204e9800998ecf8427e"'
Key:backup/database/1689953756_dev_wss_db.pgdump
LastModified:'2023-07-21T15:36:01.000Z'
Owner:
ID:2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size:0
StorageClass:STANDARD
- ETag:'"d41d8cd98f00b204e9800998ecf8427e"'
Key:backup/database/1689953756_dev_wss_db_requests.pgdump
LastModified:'2023-07-21T15:36:08.000Z'
Owner:
ID:2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size:0
StorageClass:STANDARD
- ETag:'"d41d8cd98f00b204e9800998ecf8427e"'
Key:backup/database/1689953756_dev_bss_service_database.pgdump
LastModified:'2023-07-21T15:36:13.000Z'
Owner:
ID:2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size:0
StorageClass:STANDARD
- ETag:'"d41d8cd98f00b204e9800998ecf8427e"'
Key:backup/database/1689953756_dev_bss_frontend_db.pgdump
LastModified:'2023-07-21T15:36:19.000Z'
Owner:
ID:2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size:0
StorageClass:STANDARD
- ETag:'"d41d8cd98f00b204e9800998ecf8427e"'
Key:backup/database/1689953756_dev_mss_db.pgdump
LastModified:'2023-07-21T15:36:25.000Z'
Owner:
ID:2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size:0
StorageClass:STANDARD
This works, but in my real world use case this list is very long, so
there will be a lot of output because of each iteration. I have
already set the loop label to something less noisy.
Is there a way to add keys to a list of dicts, where those new keys
are based on an operation of another key?
thx !
Dick
--
You received this message because you are subscribed to the Google
Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAF8BbLZDy-7ym4pcEJSBuCVkZZ2gPzinj8NFTcNnN6WedH%2BfMQ%40mail.gmail.com
.
--
Todd
--
You received this message because you are subscribed to the Google Groups "Ansible
Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/5bd7bec0-540c-3505-cbfd-75b47e9869ff%40gmail.com.
[ansible-project] Loopless alternative?
Hii
I have a list of dicts, where I want to inject a number of helper keys.
I can do this with set_facts in a loop. Example playbook (hopefully this
displays OK):
---
- name: Add helper keys to list of dicts
hosts: localhost
connection: local
gather_facts: no
tasks:
- set_fact:
backup_objects: "{{ backup_objects|default([]) | union(
[
item | combine(
{
'basename': item.Key|regex_replace('^' ~ prefix ~
'\\d{10}_(.*)\\.pgdump', '\\1')
}
)
]
)
}}"
loop: "{{ all_objects }}"
loop_control:
label: "{{ item.Key }}"
- debug: var=backup_objects
vars:
prefix: backup/database/
all_objects:
- ETag: '"d41d8cd98f00b204e9800998ecf8427e"'
Key: backup/database/1689953756_dev_wss_db.pgdump
LastModified: '2023-07-21T15:36:01.000Z'
Owner:
ID:
2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size: 0
StorageClass: STANDARD
- ETag: '"d41d8cd98f00b204e9800998ecf8427e"'
Key: backup/database/1689953756_dev_wss_db_requests.pgdump
LastModified: '2023-07-21T15:36:08.000Z'
Owner:
ID:
2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size: 0
StorageClass: STANDARD
- ETag: '"d41d8cd98f00b204e9800998ecf8427e"'
Key: backup/database/1689953756_dev_bss_service_database.pgdump
LastModified: '2023-07-21T15:36:13.000Z'
Owner:
ID:
2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size: 0
StorageClass: STANDARD
- ETag: '"d41d8cd98f00b204e9800998ecf8427e"'
Key: backup/database/1689953756_dev_bss_frontend_db.pgdump
LastModified: '2023-07-21T15:36:19.000Z'
Owner:
ID:
2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size: 0
StorageClass: STANDARD
- ETag: '"d41d8cd98f00b204e9800998ecf8427e"'
Key: backup/database/1689953756_dev_mss_db.pgdump
LastModified: '2023-07-21T15:36:25.000Z'
Owner:
ID:
2d8917bbcab5a8e0d3d7f5f39d147cd6de38e883357d7ae16323398c302fe97e
Size: 0
StorageClass: STANDARD
This works, but in my real world use case this list is very long, so there
will be a lot of output because of each iteration. I have already set the
loop label to something less noisy.
Is there a way to add keys to a list of dicts, where those new keys are
based on an operation of another key?
thx !
Dick
--
You received this message because you are subscribed to the Google Groups
"Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ansible-project+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/ansible-project/CAF8BbLZDy-7ym4pcEJSBuCVkZZ2gPzinj8NFTcNnN6WedH%2BfMQ%40mail.gmail.com.
