Args.... so this was the reason definitively: bash -c ip route - does
not work, and then CS agent deletes 169.254.0.0 route completely...
why - have no damn clue...
I made simple cron, creating route, sleep 30sec, then create route again...
- so at least VR creation will be prolonged for max of 30sec...but will be
created.
Just created bunch of VPCs with 100% sucess...
Any info, please = why is 169.254.0.0 route being deleted ?
On 14 March 2015 at 18:50, Andrija Panic <andrija.pa...@gmail.com> wrote:
> I think I located the problem, sort of:
>
> it seems like ACS agent is deleting route to 168.254.0.0 - so after the
> patchviasocket - agent can not connect to private interface on VM
> (169.254.x.x)
>
> if I try to provision VPC - it waits, and waits, and waits... until I add
> route manually on the particualr host (ip route add 169.254.0.0/16 dev
> cloud0) - and then VPC creation (VR creation) is completeled.
>
> Any idea why is 169.254 route being deleted...: ???
>
>
> From log:
>
> 2015-03-14 18:41:32,187 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: /bin/bash -c ip route | grep
> 169.254.0.0/16
> 2015-03-14 18:41:32,193 DEBUG [utils.script.Script]
> (agentRequest-Handler-5:null) Executing: /bin/bash -c ip route del
> 169.254.0.0/16
> 2015-03-14 18:41:32,648 DEBUG [kvm.resource.LibvirtComputingResource]
> (agentRequest-Handler-5:null) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl
> -n r-2236-VM -p %vpccidr=
> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%template=domP%name=r-2236-VM%eth0ip=169.254.2.56%eth0mask=255.255.0.0%type=vpcrouter%disable_rp_filter=true
> 2015-03-14 18:41:32,676 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-5:null) Trying to connect to 169.254.2.56
> 2015-03-14 18:41:35,682 DEBUG
> [resource.virtualnetwork.VirtualRoutingResource]
> (agentRequest-Handler-5:null) Could not connect to 169.254.2.56
>
> and the first command /bin/bash -c ip route | grep 169.254.0.0/16
> fails in my case if I run it manually on centos
>
> ip route - works, but bash -c ip route does not work
> ACS 4.3.2 now...
>
>
> [root@cs18 ~]# /bin/bash -c ip route | grep 169.254.0.0/16
> Usage: ip [ OPTIONS ] OBJECT { COMMAND | help }
> ip [ -force ] -batch filename
> where OBJECT := { link | addr | addrlabel | route | rule | neigh | ntable
> |
> tunnel | maddr | mroute | mrule | monitor | xfrm |
> token }
> OPTIONS := { -V[ersion] | -s[tatistics] | -d[etails] | -r[esolve] |
> -f[amily] { inet | inet6 | ipx | dnet | link } |
> -o[neline] | -t[imestamp] | -b[atch] [filename] |
> -rc[vbuf] [size]}
>
> You can see this command fails on my CentOS 6.6...
>
> not sure what the hack is happening...
>
> On 14 March 2015 at 17:10, Andrija Panic <andrija.pa...@gmail.com> wrote:
>
>> Yes I understadn that - more of a problem, qemu-img from CentOS 6.5
>> doesnt work snapshots (redhat removed -S flag, etc.), then we need RBD
>> support (for CEPH), which RH does not support in normal RHEL/CentOS, etc.
>>
>> Next cloud deployement - dump the the RHEL/CentOS, and go with Ubuntu
>> 14.04, which does support everythgin out of the box. I'm so frustrated with
>> trying anything new with CentOS/RHEL...simply hurts...
>>
>> On 14 March 2015 at 17:04, Marcus <shadow...@gmail.com> wrote:
>>
>>> Yeah, I don't remember all of the discussion around the minimum
>>> version and why, but each libvirt and qemu has such different behavior
>>> (not all good), that we decided to target a minimum regardless of what
>>> was supplied by the distros. It's just too hard to support them all
>>> with their differing behavior. Open stack ran into the same issues.
>>>
>>> On Sat, Mar 14, 2015 at 9:02 AM, Andrija Panic <andrija.pa...@gmail.com>
>>> wrote:
>>> > Marcus - QEMU is 0.12.x (CentOS 6.4)
>>> >
>>> > but libvirt is compiled 1.2.3
>>> >
>>> > On 14 March 2015 at 16:58, Marcus <shadow...@gmail.com> wrote:
>>> >
>>> >> Oh, this is an unsupported version. should be using libvirt 0.9.4+ and
>>> >> Qemu 1.0+ per the install docs.
>>> >>
>>> >> On Sat, Mar 14, 2015 at 8:56 AM, Andrija Panic <
>>> andrija.pa...@gmail.com>
>>> >> wrote:
>>> >> > Tried the proposed solution - but it still don't work - BTW my qemu
>>> is
>>> >> > 0.12.x - from CentOS 6.4...
>>> >> >
>>> >> > When I manually start the script with all parameters, it simply
>>> hangs, no
>>> >> > output at all, until ctrl + c
>>> >> >
>>> >> > BTW this is with now upgraded ACS to 4.3.2...
>>> >> >
>>> >> > :(
>>> >> >
>>> >> > On 12 March 2015 at 09:24, Andrija Panic <andrija.pa...@gmail.com>
>>> >> wrote:
>>> >> >
>>> >> >> Hi Marcus, will do - I'm just unconfortable changing live things,
>>> but
>>> >> that
>>> >> >> is my problem. Will let you know of the output.
>>> >> >>
>>> >> >> Thanks
>>> >> >>
>>> >> >> On 12 March 2015 at 01:34, ilya musayev <
>>> ilya.mailing.li...@gmail.com>
>>> >> >> wrote:
>>> >> >>
>>> >> >>> Andrija,
>>> >> >>>
>>> >> >>> Have you tried the solution Loic mentioned? Its seems like an
>>> easy one
>>> >> to
>>> >> >>> try..
>>> >> >>>
>>> >> >>> Thanks
>>> >> >>> ilya
>>> >> >>>
>>> >> >>>
>>> >> >>> On 3/11/15 3:54 AM, Andrija Panic wrote:
>>> >> >>>
>>> >> >>>> Oh well, I will then update to 4.3.2 anyway, and hope to see the
>>> >> problem
>>> >> >>>> go
>>> >> >>>> away...
>>> >> >>>>
>>> >> >>>> THanks guys
>>> >> >>>>
>>> >> >>>> On 10 March 2015 at 22:45, Loic Lambiel <
>>> loic.lamb...@exoscale.ch>
>>> >> >>>> wrote:
>>> >> >>>>
>>> >> >>>> Hi,
>>> >> >>>>>
>>> >> >>>>> Maybe the same issue we faced here:
>>> https://github.com/exoscale/
>>> >> >>>>> cloudstack/commit/df3e7533a78c3594cdd39b8b18e6f4a3abcb8cfd
>>> >> >>>>>
>>> >> >>>>> The patch has not been yet submitted upstream. I'll do it once
>>> it is
>>> >> >>>>> running on our production.
>>> >> >>>>>
>>> >> >>>>> Anyway, you may easily test it by performing the small change
>>> >> directly
>>> >> >>>>> into the script.
>>> >> >>>>>
>>> >> >>>>> Hope that helps.
>>> >> >>>>>
>>> >> >>>>> Regards,
>>> >> >>>>>
>>> >> >>>>> Loic Lambiel
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>> On 10/03/15 19:40, Andrija Panic wrote:
>>> >> >>>>>
>>> >> >>>>> Hi guys,
>>> >> >>>>>>
>>> >> >>>>>> Kernel is Elrepo 3.10.67 (CentOS 6.6)
>>> >> >>>>>> Libvirt manually compiled 1.2.3 with RBD support.
>>> >> >>>>>> Qemu 0.12.1.2 - Inktank patched original RHEL 6.4 rpms with RBD
>>> >> >>>>>> support.
>>> >> >>>>>>
>>> >> >>>>>> And here is the output, with debug mode, from Agent.
>>> >> >>>>>>
>>> >> >>>>>> At the begining I see agent deleting routes for 169.254.x.x
>>> >> network...:
>>> >> >>>>>>
>>> >> >>>>>> Basically, It seems like the
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> is
>>> >> >>>>>> failing, not being able to connect to socket...
>>> >> >>>>>> And I could not ping VR (169.254.2.99)...
>>> >> >>>>>>
>>> >> >>>>>> LOG:
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:43:46,976{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-1:) Request:Seq 788-1319632926: { Cmd ,
>>> >> MgmtId:
>>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 100111,
>>> >> >>>>>> [{"com.cloud.agent.api.StartCommand":{"vm":{"id":
>>> >> >>>>>> 2191,"name":"r-2191-VM","type":"DomainRouter","cpus":1,"
>>> >> >>>>>> minSpeed":166,"maxSpeed":1000,"minRam":268435456,"maxRam":
>>> >> >>>>>> 268435456,"arch":"x86_64","os":"Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)","bootArgs":" vpccidr=10.0.0.0/8
>>> >> >>>>>> domain=cs2cloud.internal dns1=8.8.8.8 dns2= template=domP
>>> >> >>>>>> name=r-2191-VM
>>> >> >>>>>> eth0ip=169.254.2.99 eth0mask=255.255.0.0 type=vpcrouter
>>> >> >>>>>> disable_rp_filter=true","rebootOnCrash":false,"
>>> >> >>>>>> enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":
>>> >> >>>>>> false,"vncPassword":"e9e24eb0b1ff8ec2","params":{"
>>> >> >>>>>> memoryOvercommitRatio":"1.0","cpuOvercommitRatio":"6.0"},"
>>> >> >>>>>> uuid":"485587f7-4a04-4d77-bc36-2367f2adf77e","disks":[{"
>>> >> >>>>>> data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"
>>> >> >>>>>> uuid":"d042e003-5278-461d-8a35-3b1261164f93","
>>> >> >>>>>> volumeType":"ROOT","dataStore":{"org.apache.cloudstack.
>>> >> >>>>>> storage.to.PrimaryDataStoreTO":{"uuid":"5b93422e-1a66-353d-
>>> >> >>>>>> 88a8-2203f79b1dc6","id":209,"poolType":"RBD","host":"
>>> >> >>>>>> cephmon.domaina.net
>>> ","path":"cloudstack","port":6789,"url":"RBD://
>>> >> >>>>>> cephmon.domaina.net/cloudstack/?ROLE=Primary&
>>> >> >>>>>> STOREUUID=5b93422e-1a66-353d-88a8-2203f79b1dc6
>>> >> >>>>>> "}},"name":"ROOT-2191","size":2621440000,"path":"d042e003-
>>> >> >>>>>> 5278-461d-8a35-3b1261164f93","volumeId":2857,"vmName":"r-
>>> >> >>>>>> 2191-VM","accountId":2,"format":"RAW","id":2857,"
>>> >> >>>>>> deviceId":0,"hypervisorType":"KVM"}},"diskSeq":0,"path":"
>>> >> >>>>>> d042e003-5278-461d-8a35-3b1261164f93","type":"ROOT","_
>>> >> >>>>>>
>>> details":{"managed":"false","storagePort":"6789","storageHost":"
>>> >> >>>>>> cephmon.domaina.net
>>> >> >>>>>> ","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"
>>> >> >>>>>> networkRateMbps":-1,"defaultNic":false,"uuid":"
>>> >> >>>>>> d27a7aad-2dcc-45c7-bf30-
>>> >> >>>>>> 3302d011da66","ip":"169.254.2.99","netmask":"255.255.0.0","
>>> >> >>>>>> gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:63","
>>> >> >>>>>> broadcastType":"LinkLocal","type":"Control","
>>> >> >>>>>> isSecurityGroupEnabled":false}]},"hostIp":"10.44.253.180","
>>> >> >>>>>> executeInSequence":false,"wait":0}},{"com.cloud.agent.
>>> >> >>>>>> api.check.CheckSshCommand":{"ip":"169.254.2.99","port":
>>> >> >>>>>> 3922,"interval":6,"retries":100,"name":"r-2191-VM","wait":
>>> >> >>>>>>
>>> 0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"
>>> >> >>>>>> router.name
>>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> >> >>>>>> com.cloud.agent.api.PlugNicCommand":{"nic":{"
>>> >> >>>>>> deviceId":1,"networkRateMbps":99999,"defaultNic":true,"uuid"
>>> >> >>>>>> :"fe848dbe-cf4b-473e-8124-62289f86b144","ip":"5.226.146.
>>> >> >>>>>> 62","netmask":"255.255.255.128","gateway":"5.226.146.1","
>>> >> >>>>>> mac":"06:f0:d6:00:01:49","broadcastType":"Vlan","type":"
>>> >> >>>>>> Public","broadcastUri":"vlan://untagged","isolationUri":"
>>> >> >>>>>>
>>> >> vlan://untagged","isSecurityGroupEnabled":false,"name":"breth1-500"},"
>>> >> >>>>>> instanceName":"r-2191-VM","vmType":"DomainRouter","wait":
>>> >> >>>>>> 0}},{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"
>>> >> >>>>>> ipAddresses":[{"accountId":2,"publicIp":"5.226.146.62","
>>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> >> >>>>>> networkName":"breth1-500"}],"accessDetails":{"router.guest.
>>> >> >>>>>> ip":"5.226.146.62","zone.network.type":"Advanced","
>>> >> >>>>>> router.name
>>> >> >>>>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{"
>>> >> >>>>>> com.cloud.agent.api.routing.SetSourceNatCommand":{"
>>> >> >>>>>> ipAddress":{"accountId":2,"publicIp":"5.226.146.62","
>>> >> >>>>>> sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":
>>> >> >>>>>> false,"broadcastUri":"vlan://untagged","vlanGateway":"5.
>>> >> >>>>>> 226.146.1","vlanNetmask":"255.255.255.128","vifMacAddress":"
>>> >> >>>>>> 06:f0:d6:00:01:49","networkRate":99999,"trafficType":"Public","
>>> >> >>>>>> networkName":"breth1-500"},"add":true,"accessDetails":{"
>>> >> >>>>>> zone.network.type":"Advanced","
>>> >> >>>>>> router.name
>>> ":"r-2191-VM","router.ip":"169.254.2.99"},"wait":0}},{}]
>>> >> }
>>> >> >>>>>> 2015-03-10 16:43:50,884{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-1:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.StartCommand
>>> >> >>>>>> 2015-03-10 16:43:58,257{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:01,747{GMT} DEBUG
>>> [kvm.resource.KVMGuestOsMapper]
>>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>>> Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)
>>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.NetworkUsageCommand
>>> >> >>>>>> 2015-03-10 16:44:06,309{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing:
>>> >> >>>>>>
>>> /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh
>>> >> >>>>>> vpc_netusage.sh 169.254.2.1 -l 46.232.180.55 -g
>>> >> >>>>>> 2015-03-10 16:44:06,439{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:06,441{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Seq 788-1319632928: { Ans: , MgmtId:
>>> >> >>>>>> 161344838950, via: 788, Ver: v1, Flags: 10,
>>> >> >>>>>>
>>> >> [{"com.cloud.agent.api.NetworkUsageAnswer":{"routerName":"r-1873-VM","
>>> >> >>>>>>
>>> >> bytesSent":198379,"bytesReceived":7380903,"result":true,"details":"","
>>> >> >>>>>> wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG
>>> [kvm.resource.BridgeVifDriver]
>>> >> >>>>>> (agentRequest-Handler-1:) nic=[Nic:Control-169.254.2.99-null]
>>> >> >>>>>> 2015-03-10 16:44:11,055{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route |
>>> grep
>>> >> >>>>>> 169.254.0.0/16
>>> >> >>>>>> 2015-03-10 16:44:11,060{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,061{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing: /bin/bash -c ip route del
>>> >> >>>>>> 169.254.0.0/16
>>> >> >>>>>> 2015-03-10 16:44:11,064{GMT} DEBUG [utils.script.Script]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG
>>> [kvm.resource.KVMGuestOsMapper]
>>> >> >>>>>> (agentRequest-Handler-1:) Can't find the mapping of guest os:
>>> Debian
>>> >> >>>>>> GNU/Linux 7(64-bit)
>>> >> >>>>>> 2015-03-10 16:44:11,065{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) starting r-2191-VM: <domain
>>> type='kvm'>
>>> >> >>>>>> <name>r-2191-VM</name>
>>> >> >>>>>> <uuid>485587f7-4a04-4d77-bc36-2367f2adf77e</uuid>
>>> >> >>>>>> <description>Debian GNU/Linux 7(64-bit)</description>
>>> >> >>>>>> <clock offset='utc'>
>>> >> >>>>>> </clock>
>>> >> >>>>>> <features>
>>> >> >>>>>> <pae/>
>>> >> >>>>>> <apic/>
>>> >> >>>>>> <acpi/>
>>> >> >>>>>> </features>
>>> >> >>>>>> <devices>
>>> >> >>>>>> <emulator>/usr/libexec/qemu-kvm</emulator>
>>> >> >>>>>> <interface type='bridge'>
>>> >> >>>>>> <source bridge='cloud0'/>
>>> >> >>>>>> <mac address='0e:00:a9:fe:02:63'/>
>>> >> >>>>>> <model type='virtio'/>
>>> >> >>>>>> </interface>
>>> >> >>>>>> <serial type='pty'>
>>> >> >>>>>> <target port='0'/>
>>> >> >>>>>> </serial>
>>> >> >>>>>> <graphics type='vnc' autoport='yes' listen='10.44.253.180'
>>> >> >>>>>> passwd='e9e24eb0b1ff8ec2'/>
>>> >> >>>>>> <disk device='disk' type='network'>
>>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>>> >> >>>>>> <source protocol='rbd'
>>> >> >>>>>> name='cloudstack/d042e003-5278-461d-8a35-3b1261164f93'>
>>> >> >>>>>> <host name='cephmon.domain.internal' port='6789'/>
>>> >> >>>>>> </source>
>>> >> >>>>>> <auth username='cloudstack'>
>>> >> >>>>>> <secret type='ceph'
>>> uuid='5b93422e-1a66-353d-88a8-2203f79b1dc6'/>
>>> >> >>>>>> </auth>
>>> >> >>>>>> <target dev='vda' bus='virtio'/>
>>> >> >>>>>> </disk>
>>> >> >>>>>> <disk device='cdrom' type='file'>
>>> >> >>>>>> <driver name='qemu' type='raw' cache='none' />
>>> >> >>>>>> <source file='/usr/share/cloudstack-common/vms/systemvm.iso'/>
>>> >> >>>>>> <target dev='hdc' bus='ide'/>
>>> >> >>>>>> </disk>
>>> >> >>>>>> <console type='pty'>
>>> >> >>>>>> <target port='0'/>
>>> >> >>>>>> </console>
>>> >> >>>>>> <input type='tablet' bus='usb'/>
>>> >> >>>>>> <channel type='unix'>
>>> >> >>>>>> <source mode='bind'
>>> path='/var/lib/libvirt/qemu/r-2191-VM.agent'/>
>>> >> >>>>>> <target type='virtio' name='r-2191-VM.vport'/>
>>> >> >>>>>> <address type='virtio-serial'/>
>>> >> >>>>>> </channel>
>>> >> >>>>>> </devices>
>>> >> >>>>>> <memory>262144</memory>
>>> >> >>>>>> <devices>
>>> >> >>>>>> <memballoon model='none'/>
>>> >> >>>>>> </devices>
>>> >> >>>>>> <vcpu>1</vcpu>
>>> >> >>>>>> <os>
>>> >> >>>>>> <type arch='x86_64' machine='pc'>hvm</type>
>>> >> >>>>>> <boot dev='cdrom'/>
>>> >> >>>>>> <boot dev='hd'/>
>>> >> >>>>>> </os>
>>> >> >>>>>> <cputune>
>>> >> >>>>>> <shares>166</shares>
>>> >> >>>>>> </cputune>
>>> >> >>>>>> <cpu></cpu><on_reboot>restart</on_reboot>
>>> >> >>>>>> <on_poweroff>destroy</on_poweroff>
>>> >> >>>>>> <on_crash>destroy</on_crash>
>>> >> >>>>>> </domain>
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:11,467{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:11,494{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:11,495{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:14,501{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>> >> >>>>>> libvirt,
>>> >> >>>>>> continuing
>>> >> >>>>>> 2015-03-10 16:44:17,007{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Executing:
>>> >> >>>>>>
>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>> >> >>>>>> -i
>>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> >> >>>>>> 2015-03-10 16:44:17,019{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-10:)
>>> >> >>>>>> Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:17,953{GMT} DEBUG [cloud.agent.Agent]
>>> >> (UgentTask-5:)
>>> >> >>>>>> Sending ping: Seq 788-8: { Cmd , MgmtId: -1, via: 788, Ver:
>>> v1,
>>> >> Flags:
>>> >> >>>>>> 11,
>>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> >> >>>>>> 18.domaina.net
>>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:17,964{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (Agent-Handler-3:)
>>> >> >>>>>> Received response: Seq 788-8: { Ans: , MgmtId: 161344838950,
>>> via:
>>> >> 788,
>>> >> >>>>>> Ver: v1, Flags: 100010,
>>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> >> >>>>>>
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:44:19,502{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:22,509{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:27,510{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:27,512{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:27,540{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:27,541{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:30,548{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:31,378{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:31,379{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c idle=$(top
>>> -b -n
>>> >> >>>>>> 1|grep
>>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> >> >>>>>> 2015-03-10 16:44:31,910{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:31,911{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c
>>> >> freeMem=$(free|grep
>>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:31,916{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Executing: /bin/bash -c free|grep
>>> Mem:|awk
>>> >> >>>>>> '{print $2}'
>>> >> >>>>>> 2015-03-10 16:44:31,920{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-3:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:35,549{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:38,556{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:43,557{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:43,587{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:43,588{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:46,594{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:49,449{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-4:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:44:51,595{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:54,600{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:59,601{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:44:59,602{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:44:59,628{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:02,634{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:07,635{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:10,640{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:12,066{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-5:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetStorageStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:15,641{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:15,667{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:17,021{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Found NFS storage pool 93655746-a9ef-394d-95e9-6e62471dd39f in
>>> >> >>>>>> libvirt,
>>> >> >>>>>> continuing
>>> >> >>>>>> 2015-03-10 16:45:17,022{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Executing:
>>> >> >>>>>>
>>> >> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh
>>> >> >>>>>> -i
>>> >> >>>>>> 10.44.253.13 -p /var/lib/libvirt/PRIMARY -m
>>> >> >>>>>> /mnt/93655746-a9ef-394d-95e9-6e62471dd39f -h 10.44.253.180
>>> >> >>>>>> 2015-03-10 16:45:17,036{GMT} DEBUG [kvm.resource.KVMHAMonitor]
>>> >> >>>>>> (Thread-11:)
>>> >> >>>>>> Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:17,951{GMT} DEBUG [cloud.agent.Agent]
>>> >> (UgentTask-5:)
>>> >> >>>>>> Sending ping: Seq 788-9: { Cmd , MgmtId: -1, via: 788, Ver:
>>> v1,
>>> >> Flags:
>>> >> >>>>>> 11,
>>> >> >>>>>> [{"com.cloud.agent.api.PingRoutingCommand":{"newStates":{},"_
>>> >> >>>>>> hostVmStateReport":{"i-2-1358-VM":{"state":"PowerOn","host":"
>>> >> >>>>>> cs18.domaina.net"},"r-2185-VM":{"state":"PowerOn","host":"cs
>>> >> >>>>>> 18.domaina.net
>>> >> >>>>>> "},"i-293-1948-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-1873-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1699-1788-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"r-2120-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1607-1683-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-2-1779-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"r-2191-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-2156-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-328-1304-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-595-1998-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-10-130-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1716-1823-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-1569-1551-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-10-2153-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "},"i-1683-1964-VM":{"state":"PowerOn","host":"
>>> cs18.domaina.net
>>> >> >>>>>> "},"i-914-1988-VM":{"state":"PowerOn","host":"cs18.domaina.net
>>> >> >>>>>> "}},"_gatewayAccessible":true,"_vnetAccessible":true,"
>>> >> >>>>>> hostType":"Routing","hostId":788,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:45:18,002{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (Agent-Handler-2:)
>>> >> >>>>>> Received response: Seq 788-9: { Ans: , MgmtId: 161344838950,
>>> via:
>>> >> 788,
>>> >> >>>>>> Ver: v1, Flags: 100010,
>>> >> >>>>>> [{"com.cloud.agent.api.PingAnswer":{"_command":{"
>>> >> >>>>>>
>>> hostType":"Routing","hostId":788,"wait":0},"result":true,"wait":0}}]
>>> >> >>>>>> }
>>> >> >>>>>> 2015-03-10 16:45:18,672{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:23,673{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:26,680{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:31,682{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:31,683{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Executing:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>
>>> >> >>>>>> 2015-03-10 16:45:36,684{GMT} WARN [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (Script-5:) Interrupting script.
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} WARN [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) Timed out:
>>> >> >>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>> hviasocket.pl
>>> >> >>>>>> -n
>>> >> >>>>>> r-2191-VM -p %vpccidr=
>>> >> >>>>>> 10.0.0.0/8%domain=cs2cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>> template=domP%name=r-2191-VM%eth0ip=169.254.2.99%eth0mask=
>>> >> >>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>> . Output is:
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-1:) passcmd failed:timeout
>>> >> >>>>>> 2015-03-10 16:45:36,686{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-2:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetHostStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:37,436{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c idle=$(top
>>> -b -n
>>> >> >>>>>> 1|grep
>>> >> >>>>>> Cpu\(s\):|cut -d% -f4|cut -d, -f2);echo $idle
>>> >> >>>>>> 2015-03-10 16:45:37,967{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:37,968{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c
>>> >> freeMem=$(free|grep
>>> >> >>>>>> cache:|awk '{print $4}');echo $freeMem
>>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:37,973{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Executing: /bin/bash -c free|grep
>>> Mem:|awk
>>> >> >>>>>> '{print $2}'
>>> >> >>>>>> 2015-03-10 16:45:37,977{GMT} DEBUG [kvm.resource.
>>> >> >>>>>> LibvirtComputingResource]
>>> >> >>>>>> (agentRequest-Handler-2:) Execution is successful.
>>> >> >>>>>> 2015-03-10 16:45:39,692{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:44,693{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Trying to connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:47,700{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Could not connect to 169.254.2.99
>>> >> >>>>>> 2015-03-10 16:45:51,807{GMT} DEBUG [cloud.agent.Agent]
>>> >> >>>>>> (agentRequest-Handler-3:) Processing command:
>>> >> >>>>>> com.cloud.agent.api.GetVmStatsCommand
>>> >> >>>>>> 2015-03-10 16:45:52,701{GMT} DEBUG
>>> >> >>>>>> [resource.virtualnetwork.VirtualRoutingResource]
>>> >> >>>>>> (agentRequest-Handler-1:)
>>> >> >>>>>> Unable to logon to 169.254.2.99
>>> >> >>>>>>
>>> >> >>>>>> On 10 March 2015 at 17:50, ilya musayev <
>>> >> ilya.mailing.li...@gmail.com>
>>> >> >>>>>> wrote:
>>> >> >>>>>>
>>> >> >>>>>> What kernel are you running? Please include Libvirt and Qemu
>>> >> version.
>>> >> >>>>>>
>>> >> >>>>>>>
>>> >> >>>>>>> On 3/10/15 8:21 AM, Andrija Panic wrote:
>>> >> >>>>>>>
>>> >> >>>>>>> Hi,
>>> >> >>>>>>>
>>> >> >>>>>>>> we are having issues from time to time with starting VR -
>>> while
>>> >> >>>>>>>> creating
>>> >> >>>>>>>> VPC.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Here is the few lines from MGMT log:
>>> >> >>>>>>>>
>>> >> >>>>>>>> 2015-03-10 16:11:50,018 WARN [c.c.n.v.VpcManagerImpl]
>>> >> >>>>>>>> (Job-Executor-27:ctx-84f4fba4 ctx-970acb35) Failed to start
>>> vpc
>>> >> [VPC
>>> >> >>>>>>>> [621-VPC-delete] due to
>>> >> >>>>>>>> com.cloud.exception.AgentUnavailableException: Resource
>>> [Host:788]
>>> >> >>>>>>>> is
>>> >> >>>>>>>> unreachable: Host 788: Unable to start instance due to
>>> Unable to
>>> >> >>>>>>>> start
>>> >> >>>>>>>> VM[DomainRouter|r-2189-VM] due to error in finalizeStart, not
>>> >> >>>>>>>> retrying
>>> >> >>>>>>>> at
>>> >> >>>>>>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(
>>> >> >>>>>>>> VirtualMachineManagerImpl.java:1072)
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> The interesting part is on the Agent itself:
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> 2015-03-10 16:03:28,520{GMT} WARN [kvm.resource.
>>> >> >>>>>>>> LibvirtComputingResource]
>>> >> >>>>>>>> (agentRequest-Handler-2:) Timed out:
>>> >> >>>>>>>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patc
>>> >> >>>>>>>> hviasocket.pl
>>> >> >>>>>>>> -n
>>> >> >>>>>>>> r-2189-VM -p %vpccidr=
>>> >> >>>>>>>> 10.0.0.0/24%domain=cs640cloud.internal%dns1=8.8.8.8%dns2=%
>>> >> >>>>>>>> template=domP%name=r-2189-VM%eth0ip=169.254.2.134%eth0mask=
>>> >> >>>>>>>> 255.255.0.0%type=vpcrouter%disable_rp_filter=true
>>> >> >>>>>>>> . Output is:
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>> Interesting stuff is - it's seems somewhat random, 3-4 out
>>> of 5
>>> >> >>>>>>>> works.
>>> >> >>>>>>>>
>>> >> >>>>>>>> Any suggestion are really appreciated, since this is live
>>> system
>>> >> :(
>>> >> >>>>>>>> ACS 4.3.0, KVM, CEPH
>>> >> >>>>>>>>
>>> >> >>>>>>>> Thanks,
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>>>>
>>> >> >>>>>
>>> >> >>>>>
>>> >> >>>>
>>> >> >>>
>>> >> >>
>>> >> >>
>>> >> >> --
>>> >> >>
>>> >> >> Andrija Panić
>>> >> >>
>>> >> >
>>> >> >
>>> >> >
>>> >> > --
>>> >> >
>>> >> > Andrija Panić
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Andrija Panić
>>>
>>
>>
>>
>> --
>>
>> Andrija Panić
>>
>
>
>
> --
>
> Andrija Panić
>
--
Andrija Panić
