Ah, mention of ControlMaster makes me think of this: https://groups.google.com/forum/#!searchin/ansible-project/controlpersist/ansible-project/QUdxNK1zEH0/rQKnO827FUgJ
Do you have pipelining enabled? If so, you might be running into the combination of updated CentOS6 openssh + pipelining bug that others are reporting. -Toshio On Fri, Nov 7, 2014 at 11:32 AM, Paul Becker <[email protected]> wrote: > Ok, here's a followup. I added a statement to print the contents of the > result variable in _make_tmp_path (in /runner/__init__.py) and then ran my > playbook. The output is kind of long, but here it is: > > <10.250.1.55> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', > '-o', 'ControlPersist=60s', '-o', > 'ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', > 'KbdInteractiveAuthentication=no', '-o', > 'PreferredAuthentications=publickey', '-o', 'PasswordAuthentication=no', > '-o', 'ConnectTimeout=10', '10.250.1.55', "/bin/sh -c 'mkdir -p > /tmp/ansible-tmp-1415379422.44-263362473697853 && echo > /tmp/ansible-tmp-1415379422.44-263362473697853'"] > result= { > 'stdout': '', > 'stderr': 'OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013\n > debug1: Reading configuration data /etc/ssh/ssh_config\r\n > debug1: Applying options for *\r\n > debug1: auto-mux: Trying existing master\r\n > debug1: Control socket "/root/.ansible/cp/ansible-ssh-10.250.1.55-22-root" > does not exist\r\n > debug2: ssh_connect: needpriv 0\r\n > debug1: Connecting to 10.250.1.55 [10.250.1.55] port 22.\r\n > debug2: fd 3 setting O_NONBLOCK\r\n > debug1: fd 3 clearing O_NONBLOCK\r\n > debug1: Connection established.\r\n > debug3: timeout: 9665 ms remain after connect\r\n > debug1: permanently_set_uid: 0/0\r\n > debug1: identity file /root/.ssh/identity type -1\r\n > debug1: identity file /root/.ssh/identity-cert type -1\r\n > debug3: Not a RSA1 key file /root/.ssh/id_rsa.\r\n > debug2: key_type_from_name: unknown key type \'-----BEGIN\'\r\n > debug3: key_read: missing keytype\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug3: key_read: missing whitespace\r\n > debug2: key_type_from_name: unknown key type \'-----END\'\r\n > debug3: key_read: missing keytype\r\n > debug1: identity file /root/.ssh/id_rsa type 1\r\n > debug1: identity file /root/.ssh/id_rsa-cert type -1\r\n > debug1: identity file /root/.ssh/id_dsa type -1\r\n > debug1: identity file /root/.ssh/id_dsa-cert type -1\r\n > debug1: identity file /root/.ssh/id_ecdsa type -1\r\n > debug1: identity file /root/.ssh/id_ecdsa-cert type -1\r\n > debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 > Debian-4\r\n > debug1: match: OpenSSH_6.0p1 Debian-4 pat OpenSSH*\r\n > debug1: Enabling compatibility mode for protocol 2.0\r\n > debug1: Local version string SSH-2.0-OpenSSH_5.3\r\n > debug2: fd 3 setting O_NONBLOCK\r\n > debug1: SSH2_MSG_KEXINIT sent\r\n > debug3: Wrote 960 bytes for a total of 981\r\n > debug1: SSH2_MSG_KEXINIT received\r\n > debug2: kex_parse_kexinit: > diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\n > debug2: kex_parse_kexinit: > [email protected],[email protected],[email protected],[email protected],ssh-rsa,ssh-dss\r\n > debug2: kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\n > debug2: kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\n > debug2: kex_parse_kexinit: > hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\n > debug2: kex_parse_kexinit: > hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\n > debug2: kex_parse_kexinit: [email protected],zlib,none\r\n > debug2: kex_parse_kexinit: [email protected],zlib,none\r\n > debug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\n > debug2: kex_parse_kexinit: first_kex_follows 0 \r\n > debug2: kex_parse_kexinit: reserved 0 \r\n > debug2: kex_parse_kexinit: > ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\n > debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256\r\ndebug2: > kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\n > debug2: kex_parse_kexinit: > aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]\r\n > debug2: kex_parse_kexinit: > hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\n > debug2: kex_parse_kexinit: > hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96\r\n > debug2: kex_parse_kexinit: none,[email protected]\r\n > debug2: kex_parse_kexinit: none,[email protected]\r\n > debug2: kex_parse_kexinit: \r\n > debug2: kex_parse_kexinit: \r\n > debug2: kex_parse_kexinit: first_kex_follows 0 \r\n > debug2: kex_parse_kexinit: reserved 0 \r\n > debug2: mac_setup: found hmac-md5\r\n > debug1: kex: server->client aes128-ctr hmac-md5 [email protected]\r\n > debug2: mac_setup: found hmac-md5\r\n > debug1: kex: client->server aes128-ctr hmac-md5 [email protected]\r\n > debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent\r\n > debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP\r\n > debug3: Wrote 24 bytes for a total of 1005\r\n > debug2: dh_gen_key: priv key bits set: 125/256\r\n > debug2: bits set: 480/1024\r\ndebug1: SSH2_MSG_KEX_DH_GEX_INIT sent\r\n > debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY\r\n > debug3: Wrote 144 bytes for a total of 1149\r\n > debug3: check_host_in_hostfile: host 10.250.1.55 filename > /root/.ssh/known_hosts\r\n > debug3: check_host_in_hostfile: host 10.250.1.55 filename > /root/.ssh/known_hosts\r\n > debug3: check_host_in_hostfile: match line 1\r\n > debug1: Host \'10.250.1.55\' is known and matches the RSA host key.\r\n > debug1: Found key in /root/.ssh/known_hosts:1\r\n > debug2: bits set: 509/1024\r\n > debug1: ssh_rsa_verify: signature correct\r\n > debug2: kex_derive_keys\r\n > debug2: set_newkeys: mode 1\r\n > debug1: SSH2_MSG_NEWKEYS sent\r\n > debug1: expecting SSH2_MSG_NEWKEYS\r\n > debug3: Wrote 16 bytes for a total of 1165\r\n > debug2: set_newkeys: mode 0\r\n > debug1: SSH2_MSG_NEWKEYS received\r\n > debug1: SSH2_MSG_SERVICE_REQUEST sent\r\n > debug3: Wrote 48 bytes for a total of 1213\r\n > debug2: service_accept: ssh-userauth\r\n > debug1: SSH2_MSG_SERVICE_ACCEPT received\r\n > debug2: key: /root/.ssh/identity ((nil))\r\n > debug2: key: /root/.ssh/id_rsa (0x7ff98d26d620)\r\n > debug2: key: /root/.ssh/id_dsa ((nil))\r\n > debug2: key: /root/.ssh/id_ecdsa ((nil))\r\n > debug3: Wrote 64 bytes for a total of 1277\r\n > debug1: Authentications that can continue: publickey,password\r\n > debug3: start over, passed a different list publickey,password\r\n > debug3: preferred publickey\r\ndebug3: authmethod_lookup publickey\r\n > debug3: remaining preferred: \r\n > debug3: authmethod_is_enabled publickey\r\n > debug1: Next authentication method: publickey\r\n > debug1: Trying private key: /root/.ssh/identity\r\n > debug3: no such identity: /root/.ssh/identity\r\n > debug1: Offering public key: /root/.ssh/id_rsa\r\n > debug3: send_pubkey_test\r\n > debug2: we sent a publickey packet, wait for reply\r\n > debug3: Wrote 368 bytes for a total of 1645\r\n > debug1: Server accepts key: pkalg ssh-rsa blen 277\r\n > debug2: input_userauth_pk_ok: SHA1 fp > 69:be:9f:27:19:27:09:6a:bc:a6:d4:0c:2f:79:d0:58:6d:72:15:2e\r\n > debug3: sign_and_send_pubkey: RSA > 69:be:9f:27:19:27:09:6a:bc:a6:d4:0c:2f:79:d0:58:6d:72:15:2e\r\n > debug1: read PEM private key done: type RSA\r\ndebug3: Wrote 640 bytes for a > total of 2285\r\n > debug1: Enabling compression at level 6.\r\n > debug1: Authentication succeeded (publickey).\r\n > debug1: setting up multiplex master socket\r\n > debug3: muxserver_listen: temporary control path > /root/.ansible/cp/ansible-ssh-10.250.1.55-22-root.b522MvBV0wTcYeIJ\r\n > debug2: fd 4 setting O_NONBLOCK\r\n > debug3: fd 4 is O_NONBLOCK\r\n > debug3: fd 4 is O_NONBLOCK\r\n > debug1: channel 0: new > [/root/.ansible/cp/ansible-ssh-10.250.1.55-22-root.b522MvBV0wTcYeIJ]\r\n > debug3: muxserver_listen: mux listener channel 0 fd 4\r\n > debug1: Entering interactive session.\r\n > debug2: set_control_persist_exit_time: schedule exit in 60 seconds\r\n > ', 'rc': 0} > > > The important part is that rc=0, and stdout = ''. Stdout being empty causes > _make_tmp_path() to return '' because of this line: > > rc = > conn.shell.join_path(utils.last_non_blank_line(result['stdout']).strip(), > '') > > Apparently, something in my initial SSH connection is not entirely right, > causing the tmp var to be empty, which then cascades to PUT having an > incorrect output path, etc... > > I found that if I change ControlMaster=no in my ssh_args (in > /etc/ansible.cfg) that it WORKS every time. I am hoping that there's a clue > here somewhere that someone will benefit from. > > Thanks, > Paul > > -- > You received this message because you are subscribed to the Google Groups > "Ansible Project" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ansible-project/85b477f6-7578-44c8-aff5-56bf00ef2820%40googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Ansible Project" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAG9juEpk%3DEnMVyR76%3DPo3iUReztA%2B%3DJUXvj4F10o_mPCCTZifA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
