Thanks, your explanation clears up a lot of things for me. I am still looking for some good documentation/book that explains Apache VCL from beginning to end, IE Apache VCL for Dummies.
I will setup some more test & examine the vcld.log file more closely, it is Greek to me. But the more I dig into Apache VCL the clearer things become. It looks like I need to rethink my plans about how to use Apache VCL. I was under the impression that the virtual machine would be powered on for the reservation & powered off after the reservation was over. If I understand correctly, using 2.3.2, a user making a reservation may or may not receive the same virtual machine each time, but the functionality is the same? Thanks Again. Lewis On Mon, May 18, 2015 at 7:20 AM, Josh Thompson <[email protected]> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Lewis, > > Looking through vcld.log will allow you to see every ssh command that vcld > sends to a node. > > Also, let me explain the process VCL uses for provisioning. We use > something > we call "hot standby" to help minimize the amount of time it takes for > reservations to become ready for users to connect to their reservations. > At > the end of each reservation, the node is reloaded so that it is almost > ready > for another user. Then, when another user makes a reservation for it, > only a > few steps need to be done to finish provisioning it. The image selected > to be > reloaded on the node depends on what Predictive Reload module is assigned. > Previous to 2.4.2, Predictive Reload modules were assigned per management > node, meaning all computers managed by a specific management node would be > handled the same way. With 2.4.2, it is assigned per computer, meaning > each > computer can be handled differently. Also, 2.4.2 introduced a new module > that > doesn't actually reload it, but instead powers it off, and for VMs, > unregisters it. > > All that to say, the behavior you are seeing may be that the node has been > reloaded at the end of your reservation to prep for another one. > > One more point to note is that VCL randomizes the root/admin password when > it > provisions each node. You can find the password it selects for each > reservation in the vcld.log file. > > Josh > > On Sun May 17 2015 1:09:04 PM Litchfield, Elwin L. wrote: > > The VM was powered on as scheduled, I logged on to the VM using admin & > PW > > supplied. Logged off & the VM stayed powered on after the reservation > > expired. Ping to the VM works, ICMP Type 0 Code 0. ICMP Type 3 Code 10 > > to Port 22 reply is Host Administratively prohibited. > > > > From the VM console, on the VMware host, no administrator user is able > to > > logon. I have added 2 more admin users & they will not work either. The > > admin logon supplied from the reservation worked, but ssh was not > > active/displayed as a service. > > > > I have a Wireshark 1.8.10 trace of the private network conversation. I > am > > researching how to upgrade to the current release of Wireshark, but I > have > > not found any information that Wireshark will decode SSH. > > > > Is there any trace to run to determine what is taking place between the > > Management Node & the VM? > > > > Thanks > > > > Lewis Litchfield > > Technology Services – Viking Center Lab > > University of South Florida Sarasota-Manatee > > 941-359-4231 > > > > ________________________________________ > > From: [email protected] <[email protected]> > > Sent: Sunday, May 17, 2015 8:35 AM > > To: [email protected] > > Subject: PROBLEM -- > > > localhost|44:44|timeout|Linux.pm|VM7Cent6VCL1>VM7|vmwarelinux-vm7c2testvcl2 > > 12-v0|admin > > > > failed to add AllowUsers admin to external_sshd_config > > ------------------------------------------------------------------------ > > time: 2015-05-17 08:35:52 > > caller: Linux.pm:delete_user(894) > > ( 0) Linux.pm, delete_user (line: 894) > > (-1) Linux.pm, sanitize (line: 1177) > > (-2) reclaim.pm, call_os_sanitize (line: 271) > > (-3) reclaim.pm, process (line: 162) > > (-4) vcld, make_new_child (line: 571) > > (-5) vcld, main (line: 350) > > ------------------------------------------------------------------------ > > management node: localhost > > reservation PID: 22905 > > parent vcld PID: 26123 > > > > request ID: 44 > > reservation ID: 44 > > request state/laststate: timeout/reserved > > request start time: 2015-05-17 08:15:00 > > request end time: 2015-05-17 09:00:00 > > for imaging: no > > log ID: 51 > > > > computer: VM7Cent6VCL1 > > computer id: 6 > > computer type: virtualmachine > > computer eth0 MAC address: 00:50:56:3F:40:E3 > > computer eth1 MAC address: 00:50:56:3F:40:ED > > computer private IP address: 192.168.10.48 > > computer public IP address: 131.247.29.132 > > computer in block allocation: no > > provisioning module: VCL::Module::Provisioning::VMware::VMware > > > > vm host: VM7 > > vm host ID: 2 > > vm host computer ID: 5 > > vm profile: VMware ESXi - local storage > > vm profile VM path: datastore1 > > vm profile repository path: <undefined> > > vm profile datastore path: datastore1 > > vm profile disk type: shared > > > > image: vmwarelinux-vm7c2testvcl212-v0 > > image display name: vm7c2testvcl2 > > image ID: 12 > > image revision ID: 12 > > image size: 5762 MB > > use Sysprep: yes > > root access: yes > > image owner ID: 1 > > image owner affiliation: Local > > image revision date created: 2015-05-01 09:43:57 > > image revision production: yes > > OS module: VCL::Module::OS::Linux > > > > user: admin > > user name: vcl admin > > user ID: 1 > > user affiliation: Local > > ------------------------------------------------------------------------ > > > > RECENT LOG ENTRIES FOR THIS PROCESS: > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 871) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4902)| > executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 1/3: > > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel > admin', > > |exit status: 255, output: 22905|44:44|timeout| ssh output > (/usr/sbin/...): > > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 > > 08:35:40|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping > for 2 > > seconds before making next SSH attempt > > 2015-05-17 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| > attempt 2/3: executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 2/3: > > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel > admin', > > |exit status: 255, output: 22905|44:44|timeout| ssh output > (/usr/sbin/...): > > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 > > 08:35:42|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping > for 4 > > seconds before making next SSH attempt > > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| > attempt 3/3: executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'/usr/sbin/userdel admin' 2>&1 22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 3/3: > > |failed to execute SSH command on VM7Cent6VCL1: '/usr/sbin/userdel > admin', > > |exit status: 255, output: 22905|44:44|timeout| ssh output > (/usr/sbin/...): > > |ssh: connect to host VM7Cent6VCL1 port 22: No route to host > > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > |22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to > run > > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key > > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'/usr/sbin/userdel admin' 2>&1, exit status: 255, output: > > |22905|44:44|timeout| ssh output (/usr/sbin/...): ssh: connect to host > > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) > utils.pm, > > |run_ssh_command (line: 5043) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 881) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4902)| > executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > > |22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 1/3: > > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) > utils.pm, > > |run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 > > 08:35:46|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping > for 2 > > seconds before making next SSH attempt > > 2015-05-17 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| > attempt 2/3: executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > > |22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 2/3: > > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) > utils.pm, > > |run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 > > 08:35:48|22905|44:44|timeout|utils.pm:run_ssh_command(4894)|sleeping > for 4 > > seconds before making next SSH attempt > > 2015-05-17 08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(4906)| > attempt 3/3: executing SSH command on VM7Cent6VCL1: > > |22905|44:44|timeout| /usr/bin/ssh -i /etc/vcl/vcl.key -o > > |StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1 > > |22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5006)|attempt > 3/3: > > |failed to execute SSH command on VM7Cent6VCL1: 'sed -i -e "/^AllowUsers > > |.*/d" /etc/ssh/external_sshd_config', exit status: 255, output: > > |22905|44:44|timeout| ssh output (sed -i -e ...): ssh: connect to host > > |VM7Cent6VCL1 port 22: No route to host 22905|44:44|timeout| ( 0) > utils.pm, > > |run_ssh_command (line: 5006) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > |22905|44:44|timeout| ---- WARNING ---- > > |22905|44:44|timeout| 2015-05-17 > > |08:35:52|22905|44:44|timeout|utils.pm:run_ssh_command(5043)|failed to > run > > |SSH command after 3 attempts, command: /usr/bin/ssh -i /etc/vcl/vcl.key > > |-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o > > |ConnectionAttempts=1 -o ConnectTimeout=3 -l root -p 22 -x VM7Cent6VCL1 > > |'sed -i -e "/^AllowUsers .*/d" /etc/ssh/external_sshd_config' 2>&1, exit > > |status: 255, output: 22905|44:44|timeout| ssh output (sed -i -e ...): > ssh: > > |connect to host VM7Cent6VCL1 port 22: No route to host > > |22905|44:44|timeout| ( 0) utils.pm, run_ssh_command (line: 5043) > > |22905|44:44|timeout| (-1) Linux.pm, delete_user (line: 890) > > |22905|44:44|timeout| (-2) Linux.pm, sanitize (line: 1177) > > |22905|44:44|timeout| (-3) reclaim.pm, call_os_sanitize (line: 271) > > |22905|44:44|timeout| (-4) reclaim.pm, process (line: 162) > > |22905|44:44|timeout| (-5) vcld, make_new_child (line: 571) > > > > 2015-05-17 > > > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre > > ss(1641)|attempting to retrieve private IP address for computer: > > VM7Cent6VCL1 2015-05-17 > > 08:35:52|22905|44:44|timeout|OS.pm:get_file_contents(1897)|retrieved 15 > > lines from file on localhost: '/etc/hosts' 2015-05-17 > > > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_computer_private_ip_addre > > ss(1697)|returning IP address from /etc/hosts file: 192.168.10.48 > 2015-05-17 > > 08:35:52|22905|44:44|timeout|utils.pm:is_inblockrequest(5760)|zero rows > > were returned from database select 2015-05-17 > > > 08:35:52|22905|44:44|timeout|DataStructure.pm:get_image_affiliation_name(21 > > 21)|image owner id: 1 2015-05-17 > > > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1412)|atte > > mpting to retrieve and store data for user: user.id = '1' 2015-05-17 > > > 08:35:52|22905|44:44|timeout|DataStructure.pm:retrieve_user_data(1475)|data > > has been retrieved for user: admin (id: 1) > - -- > - ------------------------------- > Josh Thompson > VCL Developer > North Carolina State University > > my GPG/PGP key can be found at pgp.mit.edu > > All electronic mail messages in connection with State business which > are sent to or received by this account are subject to the NC Public > Records Law and may be disclosed to third parties. > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2 > > iD8DBQFVWcsOV/LQcNdtPQMRAtezAJ0Sw6V2YJ9msiQ8o50Rwbe21fgkQACeKeme > gQWeyHN1fzwbqDZ0Vw5aW+o= > =I6Ta > -----END PGP SIGNATURE----- > >
