Hello, We have a small VCL system serving 200 images on 8 VMware servers. Our system seems to operate pretty well, except that we occasionally get reservation failures. The servers (web, mgt, db) do not appear overloaded, max at about 10% utilization. Out of about 200 reservations (roughly simultaneous after a block allocation), about 4 reservations fail. These failures usually indicate "failed to update private IP address". I have included some of the failure entries from vcld.log below. What can we do to improve the reliability and performance of our VCL system ?
Thanks, Al Quiros Florida International University |4443|2053:2053|new| 2012-01-17 10:24:25|4443|2053:2053|new|OS.pm:update_public_ip_address(608)|failed to retrieve dynamic public IP address from vclimg9 2012-01-17 10:24:25|4443|2053:2053|new|utils.pm:insertloadlog(3875)|inserted computer=20, dynamicDHCPaddress, failed to retrieve dynamic public IP address from vclimg9 |4443|2053:2053|new| 2012-01-17 10:24:25|4443|2053:2053|new|State.pm:reservation_failed(213)|reservation failed on vclimg9: failed to update private IP address |4443|2053:2053|new| ( 0) State.pm, reservation_failed (line: 213) 2012-01-17 10:24:26|4443|2053:2053|new|utils.pm:insertloadlog(3875)|inserted computer=20, failed, failed to update private IP address 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(216)|inserted computerloadlog entry 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(224)|updated log ending value to 'failed', logid=474 2012-01-17 10:24:26|4443|2053:2053|new|utils.pm:update_computer_state(2033)|computer 20 state updated to: failed 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(235)|computer vclimg9 (20) state set to failed 2012-01-17 10:24:26|4443|2053:2053|new|utils.pm:update_request_state(1991)|request 2053 state updated to: failed, laststate to: new 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(248)|set request state to 'failed'/'new' 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(257)|vclimg9 in blockcomputers table 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(258)|removed vclimg9 from blockcomputers table 2012-01-17 10:24:26|4443|2053:2053|new|State.pm:reservation_failed(269)|exiting 1 |32292|1990:1990|reload| 2012-01-17 10:26:19|32292|1990:1990|reload|OS.pm:wait_for_response(465)|failed to connect to vclimg148 via SSH after 600 seconds |32292|1990:1990|reload| 2012-01-17 10:26:19|32292|1990:1990|reload|VMware.pm:load(419)|failed to perform OS post-load tasks on VM vclimg148 on VM host: idp06.fiu.edu |32292|1990:1990|reload| 2012-01-17 10:26:19|32292|1990:1990|reload|new.pm:reload_image(623)|vmwarelinux-LinuxBase19-v0 failed to load on vclimg148, returning 2012-01-17 10:26:19|32292|1990:1990|reload|utils.pm:insertloadlog(3875)|inserted computer=166, loadimagefailed, vmwarelinux-LinuxBase19-v0 failed to load on vclimg148 |32292|1990:1990|reload| 2012-01-17 10:26:19|32292|1990:1990|reload|new.pm:process(295)|failed to load vclimg148 with vmwarelinux-LinuxBase19-v0 |32292|1990:1990|reload| 2012-01-17 10:26:19|32292|1990:1990|reload|State.pm:reservation_failed(213)|reservation failed on vclimg148: process failed after trying to load or make available |32292|1990:1990|reload| ( 0) State.pm, reservation_failed (line: 213) 2012-01-17 10:26:20|32292|1990:1990|reload|utils.pm:insertloadlog(3875)|inserted computer=166, failed, process failed after trying to load or make available 2012-01-17 10:26:20|32292|1990:1990|reload|State.pm:reservation_failed(216)|inserted computerloadlog entry 2012-01-17 10:26:20|32292|1990:1990|reload|utils.pm:update_computer_state(2033)|computer 166 state updated to: failed 2012-01-17 10:26:20|32292|1990:1990|reload|State.pm:reservation_failed(235)|computer vclimg148 (166) state set to failed 2012-01-17 10:26:20|32292|1990:1990|reload|utils.pm:update_request_state(1991)|request 1990 state updated to: failed, laststate to: reload 2012-01-17 10:26:20|32292|1990:1990|reload|State.pm:reservation_failed(248)|set request state to 'failed'/'reload' 2012-01-17 10:26:20|32292|1990:1990|reload|State.pm:reservation_failed(266)|vclimg148 is NOT in blockcomputers table 2012-01-17 10:26:20|32292|1990:1990|reload|State.pm:reservation_failed(269)|exiting 1 2012-01-17 10:26:20|2218|1990:1990|failed|vcld:main(252)|request deleted |6463|2090:2090|new| 2012-01-17 10:28:47|6463|2090:2090|new|Linux.pm:get_network_configuration(2635)|failed to determine the public interface name |6463|2090:2090|new| 2012-01-17 10:28:47|6463|2090:2090|new|Linux.pm:get_public_ip_address(2736)|failed to retrieve public network configuration |6463|2090:2090|new| 2012-01-17 10:28:47|6463|2090:2090|new|OS.pm:update_public_ip_address(608)|failed to retrieve dynamic public IP address from vclimg128 2012-01-17 10:28:47|6463|2090:2090|new|utils.pm:insertloadlog(3875)|inserted computer=146, dynamicDHCPaddress, failed to retrieve dynamic public IP address from vclimg128 |6463|2090:2090|new| (-3) State.pm, reservation_failed (line: 213) |6463|2090:2090|new| 2012-01-17 10:28:47|6463|2090:2090|new|State.pm:reservation_failed(213)|reservation failed on vclimg128: failed to update private IP address |6463|2090:2090|new| ( 0) State.pm, reservation_failed (line: 213) 2012-01-17 10:28:48|6463|2090:2090|new|utils.pm:insertloadlog(3875)|inserted computer=146, failed, failed to update private IP address 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(216)|inserted computerloadlog entry 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(224)|updated log ending value to 'failed', logid=511 2012-01-17 10:28:48|6463|2090:2090|new|utils.pm:update_computer_state(2033)|computer 146 state updated to: failed 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(235)|computer vclimg128 (146) state set to failed 2012-01-17 10:28:48|6463|2090:2090|new|utils.pm:update_request_state(1991)|request 2090 state updated to: failed, laststate to: new 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(248)|set request state to 'failed'/'new' 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(257)|vclimg128 in blockcomputers table 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(258)|removed vclimg128 from blockcomputers table 2012-01-17 10:28:48|6463|2090:2090|new|State.pm:reservation_failed(269)|exiting 1 |32289|1989:1989|reload| 2012-01-17 10:31:45|32289|1989:1989|reload|OS.pm:wait_for_response(465)|failed to connect to vclimg48 via SSH after 600 seconds |32289|1989:1989|reload| 2012-01-17 10:31:45|32289|1989:1989|reload|VMware.pm:load(419)|failed to perform OS post-load tasks on VM vclimg48 on VM host: idp02.fiu.edu |32289|1989:1989|reload| 2012-01-17 10:31:45|32289|1989:1989|reload|new.pm:reload_image(623)|vmwarelinux-LinuxBase19-v0 failed to load on vclimg48, returning 2012-01-17 10:31:45|32289|1989:1989|reload|utils.pm:insertloadlog(3875)|inserted computer=59, loadimagefailed, vmwarelinux-LinuxBase19-v0 failed to load on vclimg48 |32289|1989:1989|reload| 2012-01-17 10:31:45|32289|1989:1989|reload|new.pm:process(295)|failed to load vclimg48 with vmwarelinux-LinuxBase19-v0 |32289|1989:1989|reload| 2012-01-17 10:31:45|32289|1989:1989|reload|State.pm:reservation_failed(213)|reservation failed on vclimg48: process failed after trying to load or make available |32289|1989:1989|reload| ( 0) State.pm, reservation_failed (line: 213)
