Oh well, I will then update to 4.3.2 anyway, and hope to see the problem go away...
THanks guys On 10 March 2015 at 22:45, Loic Lambiel <loic.lamb...@exoscale.ch> wrote: > Hi, > > Maybe the same issue we faced here: https://github.com/exoscale/ > cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd > > The patch has not been yet submitted upstream. I'll do it once it is > running on our production. > > Anyway, you may easily test it by performing the small change directly > into the script. > > Hope that helps. > > Regards, > > Loic Lambiel > > > On 10/03/15 19:40, Andrija Panic wrote: > >> Hi guys, >> >> Kernel is Elrepo 3.10.67 (CentOS 6.6) >> Libvirt manually compiled 1.2.3 with RBD support. >> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD support. >> >> And here is the output, with debug mode, from Agent. >> >> At the begining I see agent deleting routes for 169.254.x.x network...: >> >> Basically, It seems like the >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> is >> failing, not being able to connect to socket... >> And I could not ping VR (169.254.2.99)... >> >> LOG: >> >> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-4:) Processing command: >> com.cloud.agent.api.GetVmStatsCommand >> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd , MgmtId: >> 161344838950, via: 788, Ver: v1, Flags: 100111, >> [{"com.cloud.agent.api.StartCommand":{"vm":{"id": >> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1," >> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam": >> 268435456,"arch":"x86_64","os":"Debian >> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8 >> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP name=r-2191-VM >> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter >> disable_rp_filter=true","rebootOnCrash":false," >> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm": >> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{" >> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"}," >> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{" >> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{" >> uuid":"d042e003-5278-461d-8a35-3b1261164f93"," >> volumeType":"ROOT","dataStore":{"org.apache.cloudstack. >> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d- >> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":" >> cephmon.domaina.net","path":"cloudstack","port":6789,"url":"RBD:// >> cephmon.domaina.net/cloudstack/?ROLE=Primary& >> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6 >> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003- >> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r- >> 2191-VM","accountId":2,"format":"RAW","id":2857," >> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":" >> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_ >> details":{"managed":"false","storagePort":"6789","storageHost":" >> cephmon.domaina.net >> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0," >> networkRateMbps":-1,"defaultNic":false,"uuid":"d27a7aad-2dcc-45c7-bf30- >> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0"," >> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63"," >> broadcastType":"LinkLocal","type":"Control"," >> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180"," >> executeInSequence":false,"wait":0}},{"com.cloud.agent. >> api.check.CheckSshCommand":{"ip":"169.254.2.99","port": >> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait": >> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{" >> router.name >> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{" >> com.cloud.agent.api.PlugNicCommand":{"nic":{" >> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid" >> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146. >> 62","netmask":"255.255.255.128","gateway":"5.226.146.1"," >> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":" >> Public","broadcastUri":"vlan://untagged","isolationUri":" >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"}," >> instanceName":"r-2191-VM","vmType":"DomainRouter","wait": >> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{" >> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62"," >> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP": >> false,"broadcastUri":"vlan://untagged","vlanGateway":"5. >> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":" >> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public"," >> networkName":"breth1-500"}],"accessDetails":{"router.guest. >> ip":"5.226.146.62","zone.network.type":"Advanced"," >> router.name >> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{" >> com.cloud.agent.api.routing.SetSourceNatCommand":{" >> ipAddress":{"accountId":2,"publicIp":"5.226.146.62"," >> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP": >> false,"broadcastUri":"vlan://untagged","vlanGateway":"5. >> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":" >> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public"," >> networkName":"breth1-500"},"add":true,"accessDetails":{" >> zone.network.type":"Advanced"," >> router.name":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}] } >> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-1:) Processing command: >> com.cloud.agent.api.StartCommand >> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-5:) Processing command: >> com.cloud.agent.api.GetStorageStatsCommand >> 2015-03-10 16:44:01,747{GMT} DEBUG [kvm.resource.KVMGuestOsMapper] >> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian >> GNU/Linux 7(64-bit) >> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:) Processing command: >> com.cloud.agent.api.NetworkUsageCommand >> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Executing: >> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh >> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g >> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Execution is successful. >> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId: >> 161344838950, via: 788, Ver: v1, Flags: 10, >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM"," >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":""," >> wait":0}}] >> } >> 2015-03-10 16:44:11,055{GMT} DEBUG [kvm.resource.BridgeVifDriver] >> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null] >> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script] >> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route | grep >> 169.254.0.0/16 >> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script] >> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del >> 169.254.0.0/16 >> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.KVMGuestOsMapper] >> (agentRequest-Handler-1:) Can't find the mapping of guest os: Debian >> GNU/Linux 7(64-bit) >> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) starting r-2191-VM: <domain type='kvm'> >> <name>r-2191-VM</name> >> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid> >> <description>Debian GNU/Linux 7(64-bit)</description> >> <clock offset='utc'> >> </clock> >> <features> >> <pae/> >> <apic/> >> <acpi/> >> </features> >> <devices> >> <emulator>/usr/libexec/qemu-kvm</emulator> >> <interface type='bridge'> >> <source bridge='cloud0'/> >> <mac address='0e:00:a9:fe:02:63'/> >> <model type='virtio'/> >> </interface> >> <serial type='pty'> >> <target port='0'/> >> </serial> >> <graphics type='vnc' autoport='yes' listen='10.44.253.180' >> passwd='e9e24eb0b1ff8ec2'/> >> <disk device='disk' type='network'> >> <driver name='qemu' type='raw' cache='none' /> >> <source protocol='rbd' >> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'> >> <host name='cephmon.domain.internal' port='6789'/> >> </source> >> <auth username='cloudstack'> >> <secret type='ceph' uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/> >> </auth> >> <target dev='vda' bus='virtio'/> >> </disk> >> <disk device='cdrom' type='file'> >> <driver name='qemu' type='raw' cache='none' /> >> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/> >> <target dev='hdc' bus='ide'/> >> </disk> >> <console type='pty'> >> <target port='0'/> >> </console> >> <input type='tablet' bus='usb'/> >> <channel type='unix'> >> <source mode='bind' path='/var/lib/libvirt/qemu/r-2191-VM.agent'/> >> <target type='virtio' name='r-2191-VM.vport'/> >> <address type='virtio-serial'/> >> </channel> >> </devices> >> <memory>262144</memory> >> <devices> >> <memballoon model='none'/> >> </devices> >> <vcpu>1</vcpu> >> <os> >> <type arch='x86_64' machine='pc'>hvm</type> >> <boot dev='cdrom'/> >> <boot dev='hd'/> >> </os> >> <cputune> >> <shares>166</shares> >> </cputune> >> <cpu></cpu><on_reboot>restart</on_reboot> >> <on_poweroff>destroy</on_poweroff> >> <on_crash>destroy</on_crash> >> </domain> >> >> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:11,495{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:14,501{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-10:) >> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt, >> continuing >> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-10:) >> Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i >> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m >> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180 >> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-10:) >> Execution is successful. >> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:) >> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: >> 11, >> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_ >> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":" >> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs >> 18.domaina.net >> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "}},"_gatewayAccessible":true,"_vnetAccessible":true," >> hostType":"Routing","hostId":788,"wait":0}}] >> } >> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-3:) >> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950, via: 788, >> Ver: v1, Flags: 100010, >> [{"com.cloud.agent.api.PingAnswer":{"_command":{" >> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}] >> } >> 2015-03-10 16:44:19,502{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:22,509{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:27,510{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:27,541{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:30,548{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-3:) Processing command: >> com.cloud.agent.api.GetHostStatsCommand >> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top -b -n 1|grep >> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle >> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Execution is successful. >> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Executing: /bin/bash -c freeMem=$(free|grep >> cache:|awk '{print $4}');echo $freeMem >> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Execution is successful. >> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep Mem:|awk >> '{print $2}' >> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-3:) Execution is successful. >> 2015-03-10 16:44:35,549{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:38,556{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:43,557{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:43,588{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:46,594{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-4:) Processing command: >> com.cloud.agent.api.GetVmStatsCommand >> 2015-03-10 16:44:51,595{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:44:54,600{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:44:59,601{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:44:59,628{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:02,634{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:07,635{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:10,640{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-5:) Processing command: >> com.cloud.agent.api.GetStorageStatsCommand >> 2015-03-10 16:45:15,641{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Execution is successful. >> 2015-03-10 16:45:15,667{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-11:) >> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in libvirt, >> continuing >> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-11:) >> Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i >> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m >> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180 >> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor] >> (Thread-11:) >> Execution is successful. >> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent] (UgentTask-5:) >> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver: v1, Flags: >> 11, >> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_ >> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":" >> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs >> 18.domaina.net >> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1699-1788-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1607-1683-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1683-2156-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1716-1823-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1569-1551-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-1683-1964-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net >> "}},"_gatewayAccessible":true,"_vnetAccessible":true," >> hostType":"Routing","hostId":788,"wait":0}}] >> } >> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent] (Agent-Handler-2:) >> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950, via: 788, >> Ver: v1, Flags: 100010, >> [{"com.cloud.agent.api.PingAnswer":{"_command":{" >> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}] >> } >> 2015-03-10 16:45:18,672{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:23,673{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:26,680{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:31,682{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Executing: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> >> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource. >> LibvirtComputingResource] >> (Script-5:) Interrupting script. >> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) Timed out: >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl >> -n >> r-2191-VM -p %vpccidr= >> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=% >> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask= >> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >> . Output is: >> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-1:) passcmd failed:timeout >> 2015-03-10 16:45:36,686{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-2:) Processing command: >> com.cloud.agent.api.GetHostStatsCommand >> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top -b -n 1|grep >> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle >> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Execution is successful. >> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Executing: /bin/bash -c freeMem=$(free|grep >> cache:|awk '{print $4}');echo $freeMem >> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Execution is successful. >> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep Mem:|awk >> '{print $2}' >> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource. >> LibvirtComputingResource] >> (agentRequest-Handler-2:) Execution is successful. >> 2015-03-10 16:45:39,692{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:44,693{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Trying to connect to 169.254.2.99 >> 2015-03-10 16:45:47,700{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Could not connect to 169.254.2.99 >> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent] >> (agentRequest-Handler-3:) Processing command: >> com.cloud.agent.api.GetVmStatsCommand >> 2015-03-10 16:45:52,701{GMT} DEBUG >> [resource.virtualnetwork.VirtualRoutingResource] >> (agentRequest-Handler-1:) >> Unable to logon to 169.254.2.99 >> >> On 10 March 2015 at 17:50, ilya musayev <ilya.mailing.li...@gmail.com> >> wrote: >> >> What kernel are you running? Please include Libvirt and Qemu version. >>> >>> >>> On 3/10/15 8:21 AM, Andrija Panic wrote: >>> >>> Hi, >>>> >>>> we are having issues from time to time with starting VR - while creating >>>> VPC. >>>> >>>> Here is the few lines from MGMT log: >>>> >>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl] >>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start vpc [VPC >>>> [621-VPC-delete] due to >>>> com.cloud.exception.AgentUnavailableException: Resource [Host:788] is >>>> unreachable: Host 788: Unable to start instance due to Unable to start >>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not retrying >>>> at >>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart( >>>> VirtualMachineManagerImpl.java:1072) >>>> >>>> >>>> The interesting part is on the Agent itself: >>>> >>>> >>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource. >>>> LibvirtComputingResource] >>>> (agentRequest-Handler-2:) Timed out: >>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc >>>> hviasocket.pl >>>> -n >>>> r-2189-VM -p %vpccidr= >>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=% >>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask= >>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true >>>> . Output is: >>>> >>>> >>>> Interesting stuff is - it's seems somewhat random, 3-4 out of 5 works. >>>> >>>> Any suggestion are really appreciated, since this is live system :( >>>> ACS 4.3.0, KVM, CEPH >>>> >>>> Thanks, >>>> >>>> >>> >> > > > -- Andrija Panić
