Bingo. I was suspecting some kind of isolation thing, because it was so clear that the environment was different when running a playbook than at the command line, even when sudo'd as awx. Many thanks. I'll never get that week back, but I love a mystery solved.
~Bill On Tue, Apr 4, 2017 at 3:59 PM, Matt Davis <[email protected]> wrote: > Tower uses an isolation tech called proot that will often break shared > ticket caches. If you can't wait for Ansible 2.3 (should be released within > the next couple weeks), I'd suggest disabling proot (IIRC it's in > settings.py, but my Tower-fu is getting rusty). > > On Tuesday, April 4, 2017 at 12:48:27 PM UTC-7, William McKenzie wrote: >> >> time is definitely good. >> >> i run ntpdate in my vagrant provisioning script just to be sure. >> >> On Tue, Apr 4, 2017 at 2:20 PM, cupcake <[email protected]> wrote: >> >>> sanity check; is time in sync? windows AD/kerb wont auth if the skew is >>> more than 5 or 10 minutes off. I also saw some weirdness like this recently >>> and a reboot and then kinit again made it work but i think due to another >>> config reason on my part. >>> >>> On Tuesday, April 4, 2017 at 9:09:27 AM UTC-4, William McKenzie wrote: >>>> >>>> I think that's what I'm doing. >>>> >>>> I've tried doing the kinit from the console, doing the kinit in a cron >>>> job, doing the kinit manually in a playbook before running the winrm play >>>> book, and doing it as a local_action in the winrm playbook itself. >>>> >>>> In all cases (except the last one), the kinit succeeds; I can use klist >>>> to see the tickets (logged in as awx user). I can see the credential cache >>>> with the correct owner and attributes in the /tmp directory. I've also >>>> tried using the KEYRING instead of the FILE cache. For whatever reason, the >>>> winrm job is unable to see the credentials in the cache. >>>> >>>> When I run this playbook, it fails: >>>> --- >>>> - name: WinPing >>>> hosts: all >>>> >>>> tasks: >>>> - name: knit >>>> local_action: command echo "xxxxxxx" | kinit -l 7d -r 7d -pf >>>> [email protected] >>>> >>>> - name: ping >>>> win_ping: >>>> >>>> >>>> >>>> >>>> Produces this output. It seems to be running task setup before running >>>> my local action, and setup fails. >>>> >>>> >>>> TASK [setup] ****************************** >>>> ************************************* >>>> 07:39:32 >>>> 8 >>>> Using module file /usr/lib/python2.7/site-packag >>>> es/ansible/modules/core/windows/setup.ps1 >>>> 9 >>>> <louis.home.cartewright.com> ESTABLISH WINRM CONNECTION FOR USER: >>>> [email protected] on PORT 5986 TO louis.home.cartewright.com >>>> 10 >>>> <louis.home.cartewright.com> WINRM CONNECT: transport=kerberos >>>> endpoint=https://louis.home.cartewright.com:5986/wsman >>>> 11 >>>> <louis.home.cartewright.com> WINRM CONNECTION ERROR: >>>> authGSSClientInit() failed: (('Unspecified GSS failure. Minor code may >>>> provide more information', 851968), ("Can't find client principal >>>> [email protected] in cache collection", -1765328243)) >>>> 12 >>>> Traceback (most recent call last): >>>> 13 >>>> File >>>> "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", >>>> line 154, in _winrm_connect >>>> 14 >>>> self.shell_id = protocol.open_shell(codepage=65001) # UTF-8 >>>> 15 >>>> File >>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", >>>> line 132, in open_shell >>>> 16 >>>> res = self.send_message(xmltodict.unparse(req)) >>>> 17 >>>> File >>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", >>>> line 207, in send_message >>>> 18 >>>> return self.transport.send_message(message) >>>> 19 >>>> File >>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", >>>> line 181, in send_message >>>> 20 >>>> prepared_request = self.session.prepare_request(request) >>>> 21 >>>> File "/var/lib/awx/venv/ansible/lib/python2.7/site-packag… >>>> 22 >>>> >>>> >>>> >>>> >>>> On Monday, April 3, 2017 at 4:37:12 PM UTC-5, Matt Davis wrote: >>>>> >>>>> Ansible doesn't manage the tickets for you until Ansible Core 2.3 >>>>> (still in release candidate). Anything earlier, you'll have to do the >>>>> kinit >>>>> on the controller yourself (either via a cron job or as part of your >>>>> playbook with a local action). >>>>> >>>>> On Monday, April 3, 2017 at 7:27:21 AM UTC-7, William McKenzie wrote: >>>>>> >>>>>> Here's my setup: >>>>>> >>>>>> Ansible Tower 3.1.1, Basic License, using the vagrant box, with some >>>>>> post provisioning steps to setup krb5 and join the box to my domain. >>>>>> >>>>>> Kerberos configuration is good. I can kinit, klist, etc. etc. from >>>>>> command line. >>>>>> >>>>>> python winrm is good: >>>>>> >>>>>> vagrant@ansible-tower ~]$ sudo su - awx >>>>>> >>>>>> Last login: Sat Apr 1 23:12:18 JST 2017 on pts/1 >>>>>> >>>>>> >>>>>> *Welcome to Ansible Tower!* >>>>>> >>>>>> Log into the web interface here: etc... >>>>>> >>>>>> -bash-4.2$ cat test.py >>>>>> >>>>>> import sys >>>>>> >>>>>> from winrm.protocol import Protocol >>>>>> >>>>>> HYPERV_SERVER = 'https://louis.home.cartewright.com:5986/wsman' >>>>>> >>>>>> class RM(): >>>>>> >>>>>> >>>>>> def __init__(self): >>>>>> >>>>>> self.win_connect = Protocol(endpoint=HYPERV_SERVER, transport= >>>>>> 'kerberos', server_cert_validation='ignore') >>>>>> >>>>>> >>>>>> def test(self): >>>>>> >>>>>> shell_id = self.win_connect.open_shell() >>>>>> >>>>>> cmd = "dir" >>>>>> >>>>>> command_id = self.win_connect.run_command(shell_id, cmd) >>>>>> >>>>>> output,error_value,exit_status = self.win_connect. >>>>>> get_command_output(shell_id, command_id) >>>>>> >>>>>> self.win_connect.cleanup_command(shell_id, command_id) >>>>>> >>>>>> self.win_connect.close_shell(shell_id) >>>>>> >>>>>> print output >>>>>> >>>>>> >>>>>> def main(): >>>>>> >>>>>> rm = RM() >>>>>> >>>>>> rm.test() >>>>>> >>>>>> if __name__ == '__main__': >>>>>> >>>>>> main() >>>>>> >>>>>> sys.exit() >>>>>> >>>>>> -bash-4.2$ python test.py >>>>>> >>>>>> Volume in drive C is SAMSUNG 512GB SSD >>>>>> >>>>>> Volume Serial Number is 2C8F-7BFA >>>>>> >>>>>> >>>>>> Directory of C:\Users\ansible >>>>>> >>>>>> >>>>>> 03/31/2017 11:04 AM <DIR> . >>>>>> >>>>>> 03/31/2017 11:04 AM <DIR> .. >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Desktop >>>>>> >>>>>> 03/31/2017 11:04 AM <DIR> Documents >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Downloads >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Favorites >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Links >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Music >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Pictures >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Saved Games >>>>>> >>>>>> 07/16/2016 06:47 AM <DIR> Videos >>>>>> >>>>>> 0 File(s) 0 bytes >>>>>> >>>>>> 11 Dir(s) 291,787,771,904 bytes free >>>>>> >>>>>> >>>>>> -bash-4.2$ >>>>>> >>>>>> So now, I manually create some inventory in the default >>>>>> directories/files for ansible (not Tower). Works perfectly: >>>>>> >>>>>> -bash-4.2$ cat /etc/ansible/group_vars/windows.yml >>>>>> >>>>>> ansible_connection: winrm >>>>>> >>>>>> ansible_user: [email protected] >>>>>> >>>>>> ansible_password: R1pflash >>>>>> >>>>>> ansible_winrm_server_cert_validation: ignore >>>>>> >>>>>> >>>>>> -bash-4.2$ >>>>>> >>>>>> -bash-4.2$ ansible windows -m win_ping -v >>>>>> >>>>>> Using /etc/ansible/ansible.cfg as config file >>>>>> >>>>>> louis.home.cartewright.com | SUCCESS => { >>>>>> >>>>>> "changed": false, >>>>>> >>>>>> "ping": "pong" >>>>>> >>>>>> } >>>>>> >>>>>> >>>>>> Now, I create the exact same inventory in tower, exact same >>>>>> credentials ([email protected]) and now matter how I tweak >>>>>> it, always the same thing: >>>>>> >>>>>> Using /etc/ansible/ansible.cfg as config file SSH password: Using >>>>>> module file /usr/lib/python2.7/site-packag >>>>>> es/ansible/modules/core/windows/win_ping.ps1 < >>>>>> louis.home.cartewright.com> ESTABLISH WINRM CONNECTION FOR USER: >>>>>> [email protected] on PORT 5986 TO >>>>>> louis.home.cartewright.com <louis.home.cartewright.com> WINRM >>>>>> CONNECT: transport=kerberos endpoint=https://louis.home.ca >>>>>> rtewright.com:5986/wsman <louis.home.cartewright.com> WINRM >>>>>> CONNECTION ERROR: authGSSClientInit() failed: (('Unspecified GSS failure. >>>>>> Minor code may provide more information', 851968), ("Can't find client >>>>>> principal [email protected] in cache collection", >>>>>> -1765328243)) Traceback (most recent call last): File >>>>>> "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", >>>>>> line 154, in _winrm_connect self.shell_id = >>>>>> protocol.open_shell(codepage=65001) >>>>>> # UTF-8 File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/winrm/protocol.py", line 132, in open_shell >>>>>> res = self.send_message(xmltodict.unparse(req)) File >>>>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", >>>>>> line 207, in send_message return self.transport.send_message(message) >>>>>> File >>>>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", >>>>>> line 181, in send_message prepared_request = >>>>>> self.session.prepare_request(request) >>>>>> File >>>>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/requests/sessions.py", >>>>>> line 394, in prepare_request hooks=merge_hooks(request.hooks, >>>>>> self.hooks), File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/requests/models.py", line 298, in prepare >>>>>> self.prepare_auth(auth, url) File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/requests/models.py", line 500, in >>>>>> prepare_auth r = auth(self) File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/requests_kerberos/kerberos_.py", line 308, >>>>>> in __call__ auth_header = self.generate_request_header(None, host, >>>>>> is_preemptive=True) File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/requests_kerberos/kerberos_.py", line 148, >>>>>> in generate_request_header raise KerberosExchangeError("%s failed: %s" % >>>>>> (kerb_stage, str(error.args))) KerberosExchangeError: authGSSClientInit() >>>>>> failed: (('Unspecified GSS failure. Minor code may provide more >>>>>> information', 851968), ("Can't find client principal >>>>>> [email protected] in cache collection", -1765328243)) < >>>>>> louis.home.cartewright.com> WINRM CONNECT: transport=ssl endpoint= >>>>>> https://louis.home.cartewright.com:5986/wsman < >>>>>> louis.home.cartewright.com> WINRM CONNECTION ERROR: the specified >>>>>> credentials were rejected by the server Traceback (most recent call >>>>>> last): >>>>>> File >>>>>> "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", >>>>>> line 154, in _winrm_connect self.shell_id = >>>>>> protocol.open_shell(codepage=65001) >>>>>> # UTF-8 File "/var/lib/awx/venv/ansible/lib >>>>>> /python2.7/site-packages/winrm/protocol.py", line 132, in open_shell >>>>>> res = self.send_message(xmltodict.unparse(req)) File >>>>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/protocol.py", >>>>>> line 207, in send_message return self.transport.send_message(message) >>>>>> File >>>>>> "/var/lib/awx/venv/ansible/lib/python2.7/site-packages/winrm/transport.py", >>>>>> line 190, in send_message raise InvalidCredentialsError("the specified >>>>>> credentials were rejected by the server") InvalidCredentialsError: the >>>>>> specified credentials were rejected by the server >>>>>> louis.home.cartewright.com | UNREACHABLE! => { "changed": false, >>>>>> "msg": "kerberos: authGSSClientInit() failed: (('Unspecified GSS failure. >>>>>> Minor code may provide more information', 851968), (\"Can't find client >>>>>> principal [email protected] in cache collection\", >>>>>> -1765328243)), ssl: the specified credentials were rejected by the >>>>>> server", >>>>>> "unreachable": true } >>>>>> >>>>>> >>>>>> Just completely unable to find the credentials in the cache, no >>>>>> matter how I do it. I've put the password in the tower credentials, in >>>>>> the >>>>>> inventory, using ASK, no matter what I do, it cannot find credentials. >>>>>> >>>>>> What am I missing? >>>>>> >>>>>> ~Bill >>>>>> >>>>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Ansible Project" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/to >>> pic/ansible-project/-7e0do1u8dk/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/ansible-project/5ca67b34-4b01-4c10-bb3a-5956d24c24ad% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/ansible-project/5ca67b34-4b01-4c10-bb3a-5956d24c24ad%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to a topic in the > Google Groups "Ansible Project" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/ansible-project/-7e0do1u8dk/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/ansible-project/9df03222-6b17-40dc-980d-eb62311ad4e4%40googlegroups. > com > <https://groups.google.com/d/msgid/ansible-project/9df03222-6b17-40dc-980d-eb62311ad4e4%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAC7g-J3aHeN%2BUyzuh8KzMaEj9H04r1t_9ULHOB5nCPPY3o3x5w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
