Chris, please ignore my previous response, which was premature. I had misinterpreted the (perhaps slightly ambiguous) documentation for the slurp module, and thought that it needed to be given a base64-encoded file. Apparently, it does not. If I give it a raw .tar.xz, your ec2.py revision seems to work perfectly, thank you!
Perhaps an example in the documentation for the ec2 module using slurp and user_data would be in order, however, before you submit a pull request for your modifications to be incorporated into Ansible proper? Many thanks again for your solution, let’s hope it can be incorporated into the next Ansible release. On Friday, 26 December 2014 12:19:01 UTC, Daniel Neades wrote: > > Chris, thank you *very* much for looking at that. I have tried your > modified ec2.py but, unfortunately, even using it exactly as you describe, > the user data is still being stored on AWS in its base64 form. From within > the instance: > > # fetch http://169.254.169.254/latest/user-data > > # cat user-data > > /Td6WFoAAATm1rRGAgAhARYAAAB … /sCAAAAAARZWg== > > # base64 -d user-data test.tar.xz > > # tar tvf test.tar.xz > > -rw-r--r-- 0 djn staff 2 Dec 26 10:40 1.txt > > -rw-r--r-- 0 djn staff 2 Dec 26 10:40 2.txt > > -rw-r--r-- 0 djn staff 322 Dec 24 12:03 configinit > > By the way, looking at line 930 of the boto connection.py source file you > referenced, I don’t understand why boto is base64-encoding a parameter that > its own documentation (on lines 774-776) indicates is apparently already to > be base64-encoded. (Note, however, that I am not familiar with the boto > source.) Still, since you are passing in a decoded version of the Ansible > user_data_b64 parameter, I’d have thought what you’ve done ought to work. > Empirically, though, it doesn’t seem to. > > Finally, perhaps I should note that to test this, I dropped your ec2.py > over the top of the existing ec2.py in a stock Ansible 1.8.2 installation > that was installed by Homebrew on my OS X machine. This installation of > Ansible is using boto 2.34.0, I believe. > > Thank you again for your help! > > > On Friday, 26 December 2014 07:55:09 UTC, Chris Church wrote: >> >> One issue may be that boto is encoding unicode user data into utf-8, then >> base64-encoding it: >> >> https://github.com/boto/boto/blob/develop/boto/ec2/connection.py#L927 >> >> >> So, if we can somehow pass a str() instance to boto instead of a >> unicode() instance, it'll skip the utf-8 encoding step. >> >> I've implemented a version of the ec2 module that accepts a user_data_b64 >> parameter as a base64-encoded string of the user data, and decodes it into >> a str() instance before passing to boto: >> >> >> https://github.com/cchurch/ansible-modules-core/blob/f6522309ffaedace2305fd67a5a5721213f59199/cloud/amazon/ec2.py >> >> >> You can read the local binary file into a base64-encoded string using the >> slurp module, then pass that blob to the ec2 module. The slurp module >> needs an absolute path, so you can use role_path to specify a path >> relative to a role, or playbook_dir relative to your top-level playbook. >> >> - local_action: slurp src="{{role_path}}/files/ec2_user_data" >> register: ec2_user_data_b64 >> - ec2: >> user_data_b64: '{{ ec2_user_data_b64.content }}' >> ... >> >> >> >> >> On Wed, Dec 24, 2014 at 11:10 AM, Daniel Neades <[email protected]> >> wrote: >> >>> Incidentally, I also tried using the file lookup plugin on a >>> base64-encoded tar file, and then filtering that with Jinja2’s b64decode >>> filter. That also fails, though I think it does so at a later stage than >>> trying to use the file lookup plugin on a binary file. Using the b64decode >>> filter on a looked-up base64-encoded file, one gets an error whose stack >>> trace ends something like this: >>> >>> File >>> "/usr/local/Cellar/ansible/1.8.2/libexec/lib/python2.7/site-packages/ansible/utils/template.py", >>> >>> line 362, in template_from_string >>> res = jinja2.utils.concat(rf) >>> File "<template>", line 7, in root >>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 0: >>> ordinal not in range(128) >>> >>> I then thought I’d do a little experiment, and provide the user data as >>> a literal (copy-and-pasted) string of base64-encoded data, again filtered >>> with the Jinga2 b64decode filter, like this: >>> >>> user_data: "{{ '/Td6WFoAAATm1rRGA … VZiascRn+wIAAAAABFla' | b64decode }}" >>> >>> That took the file lookup plugin out of the equation, but resulted in >>> the same error. >>> >>> I next tried the pipe lookup plugin (with ‘cat my-binary-file’), but >>> that also fails in the same way as the file lookup plugin. Looking at the >>> source code, the pipe lookup plugin follows the file lookup one and >>> unconditionally tries to decode the output from the command as a UTF-8 >>> encoded string. >>> >>> I provisionally conclude from all this that Ansible generally doesn’t >>> want to handle binary parameter values. That seems limiting. I suppose one >>> solution to my specific problem (without fixing the wider issue) would be >>> for the ec2 module to accept something like a user_data_file parameter (as >>> an alternative to user_data), which would specify a local file whose >>> content was to be used without any processing at all as an opaque user data >>> binary object. In fact, that’s pretty much exactly what I want, as I could >>> give it the FreeBSD configinit .tar file directly. >>> >>> I am still hoping that someone has a solution that will enable binary >>> user data to be provided to an instance. >>> >>> Again, my thanks in advance for any assistance. >>> >>> -- >>> Daniel >>> >>> On Tuesday, 23 December 2014 23:06:38 UTC, Daniel Neades wrote: >>>> >>>> On Saturday, 25 October 2014 20:33:32 UTC+1, Chris Church wrote: >>>>> >>>>> The file lookup plugin is your friend. >>>>> >>>> >>>> It is – if one wishes to supply text-based user data. Unfortunately, I >>>> have a requirement to provide a binary blob of user data, and the file >>>> lookup plugin does not seem to handle that – looking at the source code, >>>> it >>>> expects the file to be a UTF8-encoded text file. >>>> >>>> <snip> >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Ansible Project" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/ansible-project/9708972a-8a15-4819-9837-4a172b9cf608%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/ansible-project/9708972a-8a15-4819-9837-4a172b9cf608%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/c9c36d15-22b2-4384-bf79-a022c1cdb643%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
