-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lewis,
Looking through vcld.log will allow you to see every ssh command that vcld sends to a node. Also, let me explain the process VCL uses for provisioning. We use something we call "hot standby" to help minimize the amount of time it takes for reservations to become ready for users to connect to their reservations. At the end of each reservation, the node is reloaded so that it is almost ready for another user. Then, when another user makes a reservation for it, only a few steps need to be done to finish provisioning it. The image selected to be reloaded on the node depends on what Predictive Reload module is assigned. Previous to 2.4.2, Predictive Reload modules were assigned per management node, meaning all computers managed by a specific management node would be handled the same way. With 2.4.2, it is assigned per computer, meaning each computer can be handled differently. Also, 2.4.2 introduced a new module that doesn't actually reload it, but instead powers it off, and for VMs, unregisters it. All that to say, the behavior you are seeing may be that the node has been reloaded at the end of your reservation to prep for another one. One more point to note is that VCL randomizes the root/admin password when it provisions each node. You can find the password it selects for each reservation in the vcld.log file. Josh On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote: > The VM was powered on as scheduled, I logged on to the VM using admin & PW > supplied. Logged off & the VM stayed powered on after the reservation > expired. Ping to the VM works, ICMP Type 0 Code 0. ICMP Type 3 Code 10 > to Port 22 reply is Host Administratively prohibited. > > From the VM console, on the VMware host, no administrator user is able to > logon. I have added 2 more admin users & they will not work either. The > admin logon supplied from the reservation worked, but ssh was not > active/displayed as a service. > > I have a Wireshark 1.8.10 trace of the private network conversation. I am > researching how to upgrade to the current release of Wireshark, but I have > not found any information that Wireshark will decode SSH. > > Is there any trace to run to determine what is taking place between the > Management Node & the VM? > > Thanks > > Lewis Litchfield > Technology Services – Viking Center Lab > University of South Florida Sarasota-Manatee > 941-359-4231 > > ________________________________________ > From: [email protected] <[email protected]> > Sent: Sunday, May 17, 2015 8:35 AM > To: [email protected] > Subject: PROBLEM -- > localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2 > 12-v0|admin > > failed to add AllowUsers admin to external_sshd_config > ------------------------------------------------------------------------ > time: 2015-05-17 08:35:52 > caller: Linux.pm:delete_user(894) > ( 0) Linux.pm, delete_user (line: 894) > (-1) Linux.pm, sanitize (line: 1177) > (-2) reclaim.pm, call_os_sanitize (line: 271) > (-3) reclaim.pm, process (line: 162) > (-4) vcld, make_new_child (line: 571) > (-5) vcld, main (line: 350) > ------------------------------------------------------------------------ > management node: localhost > reservation PID: 22905 > parent vcld PID: 26123 > > request ID: 44 > reservation ID: 44 > request state/laststate: timeout/reserved > request start time: 2015-05-17 08:15:00 > request end time: 2015-05-17 09:00:00 > for imaging: no > log ID: 51 > > computer: VM7Cent6VCL1 > computer id: 6 > computer type: virtualmachine > computer eth0 MAC address: 00:50:56:3F:40:E3 > computer eth1 MAC address: 00:50:56:3F:40:ED > computer private IP address: 192.168.10.48 > computer public IP address: 131.247.29.132 > computer in block allocation: no > provisioning module: VCL::Module::Provisioning::VMware::VMware > > vm host: VM7 > vm host ID: 2 > vm host computer ID: 5 > vm profile: VMware ESXi - local storage > vm profile VM path: datastore1 > vm profile repository path: <undefined> > vm profile datastore path: datastore1 > vm profile disk type: shared > > image: vmwarelinux-vm7c2testvcl212-v0 > image display name: vm7c2testvcl2 > image ID: 12 > image revision ID: 12 > image size: 5762 MB > use Sysprep: yes > root access: yes > image owner ID: 1 > image owner affiliation: Local > image revision date created: 2015-05-01 09:43:57 > image revision production: yes > OS module: VCL::Module::OS::Linux > > user: admin > user name: vcl admin > user ID: 1 > user affiliation: Local > ------------------------------------------------------------------------ > > RECENT LOG ENTRIES FOR THIS PROCESS: > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)| executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3: > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', > |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...): > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 > 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2 > seconds before making next SSH attempt > 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| attempt 2/3: executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3: > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', > |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...): > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 > 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4 > seconds before making next SSH attempt > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| attempt 3/3: executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3: > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel admin', > |exit status: 255, output: 22905|44:44|timeout| ssh output (/usr/sbin/...): > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > |22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output: > |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm, > |run_ssh_command (line: 5043) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)| executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > |22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 1/3: > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm, > |run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 > 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 2 > seconds before making next SSH attempt > 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| attempt 2/3: executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > |22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 2/3: > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm, > |run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 > 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping for 4 > seconds before making next SSH attempt > 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| attempt 3/3: executing SSH command on VM7Cent6VCL1: > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > |22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt 3/3: > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) utils.pm, > |run_ssh_command (line: 5006) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > |22905|44:44|timeout| ---- WARNING ---- > |22905|44:44|timeout| 2015-05-17 > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to run > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1, exit > |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...): ssh: > |connect to host VM7Cent6VCL1 port 22: No route to host > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043) > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > 2015-05-17 > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre > ss(1641)|attempting to retrieve private IP address for computer: > VM7Cent6VCL1 2015-05-17 > 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15 > lines from file on localhost: '/etc/hosts' 2015-05-17 > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre > ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48 2015-05-17 > 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows > were returned from database select 2015-05-17 > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21 > 21)|image owner id: 1 2015-05-17 > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte > mpting to retrieve and store data for user: user.id = '1' 2015-05-17 > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data > has been retrieved for user: admin (id: 1) - -- - ------------------------------- Josh Thompson VCL Developer North Carolina State University my GPG/PGP key can be found at pgp.mit.edu All electronic mail messages in connection with State business which are sent to or received by this account are subject to the NC Public Records Law and may be disclosed to third parties. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme gQWeyHN1fzwbqDZ0Vw5aW+o= =I6Ta -----END PGP SIGNATURE-----
