Hi Prapul
On Fri, Sep 18, 2015 at 2:32 PM, prapul cool <prapul_cool2...@yahoo.com.invalid> wrote: > Hi Traiano, > Curious to know which network your using? I selected "Basic" when creating the zone. > Regards, > > Prapul sriram, > > Cloud Architect, > > PSI Cloud Solutions, > > Psiclouds.com. > > > On Friday, 18 September 2015 2:56 PM, Traiano Welcome > <trai...@gmail.com> wrote: > > > On Fri, Sep 18, 2015 at 12:20 PM, Anshul Gangwar > <anshul.gang...@citrix.com> wrote: >> From where you are using Hyper-V Agent? >> > > > This is the Citrix CloudPlatform hyperv agent 4.5.0.0-1 > > > >> Regards, >> Anshul >> >> On 18-Sep-2015, at 2:40 PM, Traiano Welcome >> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: >> >> Hi Anshul >> >> >> On Fri, Sep 18, 2015 at 8:02 AM, Anshul Gangwar >> <anshul.gang...@citrix.com<mailto:anshul.gang...@citrix.com>> wrote: >> Attach some Hyper-V agent log. They will give better idea on what is >> happening on host. >> >> >> Thanks for the feedback. Attached is the paste of the windows event >> manager error we're seeing that relates to the agent. >> I'm not sure if this is the correct log but I can't find any other >> logs written by the hyper-v agent (is there a way to set this up?) >> >> On the controller side, I see an agent directory, but it is empty ... >> How to enable agent logging on the controller side? >> >> >> Regards, >> Anshul >> >> On 17-Sep-2015, at 3:27 PM, Traiano Welcome >> <trai...@gmail.com<mailto:trai...@gmail.com>> wrote: >> >> Hi All >> >> I am trying to configure a zone based on a hyper-v cluster (ms windows >> 2012 hyper-v cluster of two nodes, using SMBv3). >> >> The zone configuration completes successfully, including adding >> primary and secondary SMBv3 storage and adding hyper-v cluster node. >> However, I see that CloudStack is continuously building the system >> VMs, and destroying them in an endless loop, without ever running them >> successfully >> >> What I'm seeing in the hyper-v management console: >> >> -The Cloudstack controller is successfully is launching and >> destroying the system vms: >> - System VMs are created and destroyed continuously (Console Proxy nd >> Secondary Storage VM) (confirmed by the cloudstack manager logs) >> - System VMs appear as stopped in the CloudStack web interface, and >> periodically vanish. >> >> Looking through the logs the VM build process fails with this message: >> >> --- >> . >> . >> >> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230) >> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to >> com.cloud.agent.api.StartCommand fail on exceptionObject reference not >> set to an instance of an object. >> . >> . >> . >> --- >> >> Details of my setup: >> >> 1. Hyper-V Version: Hypervisor version 6.2 >> 2. CloudStack version/distribution: 4.5 >> 3. Hyper-V Host configuration: CPU sockets = 2, RAM = 160GiB, Dual >> 10Gibps NIC (teamed) >> 4. Primary and Secondary Primary Storage Details: SMBv3 on a Windows 2012 >> node >> 5. Zone Network Details: Basic. >> 6. The CloudStack controller is successfully managing VMs running on >> XenServer 6.2. >> >> Attached is a log dump (addresses sanitized) of the whole vm build cycle. >> >> Has anyone seen this kind of issue before when integrating Hyper-V >> with CloudStack ? >> Is there a set of additional things I could try to troubleshoot this, >> or configurations I could try? >> >> Here are the basic troubleshooting tests I'v etried: >> >> - Seeding the hyper-v image to secondary storage: success >> - Mounting the secondary and primary storage from the Windows 2012 >> storage node to the CloudStck controller VM: success >> - Browsing to the agent port via https from the controller to the >> Hyper-V nodes: success >> - Creating VMs manually on the Hyper-V nodes: success >> >> I'm not seeing any network communications, physical hardware or >> hyper-v clusters. I suppose that the fact system VMs are being >> created on Hyper-V means that essentially CloudStack is communicating >> with hyper-V., and controlling it correctly ... The only question is >> why the System VM deployment fails. >> >> I suspect physical network addressing configuration, but I'm not sure >> how to confirm this ... >> >> Many thanks in advance for any assistance! >> Traiano >> >> >> P.S: Some key debug messages in the logs (leading up to the system vms >> being destroyed): >> >> >> -- <sample from /var/log/cloudstack/management/management-server.log> --- >> >> 2015-09-17 04:45:05,382 DEBUG [c.c.a.SecondaryStorageVmAlertAdapter] >> (secstorage-1:ctx-bb312d18) (logid:49567518) New secondary storage vm >> created, zone: lol-hq-zone, secStorageVm: s-2412-VM, public IP: null, >> private IP: null >> . >> . >> 2015-09-17 04:45:07,709 DEBUG [c.c.a.m.AgentManagerImpl] >> (AgentManager-Handler-5:null) (logid:) SeqA 83-346478: Sending Seq >> 83-346478: { Ans: , MgmtId: 117586639591181, via: 83, Ver: v1, >> Flags:100010, >> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] >> } >> . >> . >> 2015-09-17 04:45:07,692 DEBUG [c.c.h.h.r.HypervDirectConnectResource] >> (DirectAgent-293:ctx-de6632d6) (logid:138757fe) Sending cmd to >> https://xx.yy.zz.223:8250/api/HypervResource/com.cloud.agent.api.StartCommand >> cmd >> data:{"vm":{"id":2412,"name":"s-2412VM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"Debian >> GNU/Linux 7(64-bit)","bootArgs":" template=domP type=secstorage >> host=172.16.116.219 port=8250 name=s-2412-VM zone=9 pod=9 >> guid=s-2412","enableHA":false,"limitCpuUse":false,"enableDynamicallyScale >> Vm":false,","params":{},"uuid":"cfe1f3c6-6b1d-4984-8752-67d33a9cbb26","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e069cb94-8f38-4cf6-b39b-459230c2e18a","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"4281fa3a-eb9f-3556-82a6-34fc2b8bbc8b","id":14,"poolType":"SMB","host":"xx.yy.zz.221","path":"/Primary","port":445,"url":"SMB://xx.yy.zz.221/Primary/?ROLE=Primary&STOREUUID=4281fa3a-eb9f-3556-82a6-34fc2b8bbc8b"}},"name":"ROOT-2412","size":0,"path":"e069cb94-8f38-4cf6-b39b-459230c2e18a","volumeId":2246,"vmName":"s-2412-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":2246,"deviceId":0,"hypervisorType":"Hyperv"}},"diskSeq":0,"path":"e069cb94-8f38-4cf6-b39b-459230c2e18a","type":"ROOT >> ","_details":{"managed":"false","storagePort":"445","storageHost":"xx.yy.zz.221","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"2ea5 >> f5f1-8500-4794-91fc-2816ebb171a7","uuid":"a4a46926-8caf-4bee-b1a0-70529c3ecac6","ip":"aa.bb.cc.137","netmask":"255.255.255.0","gateway":"aa.bb.cc.10","mac":"06:86:a6:00:00:0c","dns1":"172.16. >> 100.180","dns2":"mm.nn.oo.171","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"Virtual >> Switch"}, "deviceId":0,"networkRateMbps":-1 >> ,"defaultNic":false,"pxeDisable":true,"nicUuid":"df1eb057-f81a-493e-a2ea-a643238ed6f8","uuid":"e548abca-23da-4568-96e4-3a6aa709566b","ip":"169.254.3.60","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:03:3c","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"f1cdfa32-09ad-4431-97c6-1b237e1e4575","uuid":"d149dd9f-ac98-411d-9fb0-e587852a9df1","ip":"xx.yy.zz.162","netmask":"255.255.255.0","gateway":"xx.yy.zz.10","mac":"06:7f:b0:00:00:3d","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"Virtual >> Switch"},{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"143c7431-49f3-466b-a754-da131 >> d2d61ce","uuid":"25fc42d5-31fe-4ed1-ae66-0ca9d617bfa7","ip":"xx.yy.zz.159","netmask":"255.255.255.0","gateway":"xx.yy.zz.10","mac":"06:ab:8a:00:00:3a","broadcastType":"Native","type":"Storage","i >> SecurityGroupEnabled":false,"name":"Virtual >> Switch"}]},"hostIp":"xx.yy.zz.223","executeInSequence":false,"secondaryStorage":"cifs://xx.yy.zz.221/Secondary?user=username&domain=AHEM","contextMap":{"job":"job-58/job-7170","logid":"138757fe"},"wait":0} >> . >> . >> 2015-09-17 04:45:07,844 DEBUG [c.c.h.h.r.HypervDirectConnectResource] >> (DirectAgent-295:ctx-f5578a04) (logid:d11afe7a) executeRequest >> received response [{"com.cloud.agent.api.StartAnswer":{"vm":{" >> id":1241,"name":"v-1241-VM","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian >> GNU/Linux 7(64-bit)","bootArgs":" tem >> plate\u003ddomP type\u003dconsoleproxy host\u003d172.16.116.219 >> port\u003d8250 name\u003dv-1241-VM zone\u003d9 pod\u003d9 >> guid\u003dProxy.1241 proxy_vm\u003d1241 disable_rp_filter\u003dtrue >> vmpas >> sword\u003dtCUAMd6x eth2ip\u003daa.bb.cc.138 >> eth2mask\u003d255.255.255.0 gateway\u003daa.bb.cc.10 >> eth0ip\u003d169.254.2.35 eth0mask\u003d255.255.0.0 >> eth1ip\u003dxx.yy.zz.163 eth1mask\u003d255 >> .255.255.0 mgmtcidr\u003d172.16.116.0/23 localgw\u003dxx.yy.zz.10 >> internaldns1\u003dxx.yy.zz.180 internaldns2\u003dmm.nn.oo.171 >> dns1\u003dxx.yy.zz.180 dns2\u003dmm.nn.oo.171","enableHA":fal >> se,"limitCpuUse":false,"enableDynamicallyScaleVm":false,,"params":{},"uuid":"b6119681-e6da-4617-a2dd-504e5f82fe4a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0c3 >> d902f-8840-41d7-8c87-df860ea6ffcd","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"587027e4-e766-30fa-8972-d8ed9515170b","id":13,"poolType":"SMB"," >> host":"xx.yy.zz.223","path":"/primary","port":445,"url":"SMB://xx.yy.zz.223/primary/?ROLE\u003dPrimary\u0026STOREUUID\u003d587027e4-e766-30fa-8972-d8ed9515170b"}},"name":"ROOT-1241","size":0,"pat >> h":"0c3d902f-8840-41d7-8c87-df860ea6ffcd","volumeId":1075,"vmName":"v-1241-VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1075,"deviceId":0,"hypervisorType":"Hyperv"}},"diskSeq": >> 0,"path":"0c3d902f-8840-41d7-8c87-df860ea6ffcd","type":"ROOT","_details":{"managed":"false","storagePort":"445","storageHost":"xx.yy.zz.223","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRate >> Mbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"42a9765e-b84f-4199-9446-145df4bc68de","uuid":"a4a46926-8caf-4bee-b1a0-70529c3ecac6","ip":"aa.bb.cc.138","netmask":"255.255.255.0","gatewa >> y":"aa.bb.cc.10","mac":"06:4d:90:00:00:0d","dns1":"xx.yy.zz.180","dns2":"mm.nn.oo.171","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":false >> ,"name":"Virtual >> Switch"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"1cba88d1-5867-4073-a2fd-c8139a8a5a1a","uuid":"e548abca-23da-4568-96e4-3a6aa709566b","ip":"169.254.2.35","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:23","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"bb80257d-1578-4643-95b7-63a72f80703a","uuid":"d149dd9f-ac98-411d-9fb0-e587852a9df1","ip":"xx.yy.zz.163","netmask":"255.255.255.0","gateway":"xx.yy.zz.10","mac":"06:0c:48:00:00:3e","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"Virtual >> Switch"}]},"result":false,"details":"com.cloud.agent.api.StartCommand >> fail on exceptionObject reference not set to an instance of an >> object.","contextMap":{},"wait":0}}] >> . >> . >> . >> 2015-09-17 04:45:07,844 DEBUG [c.c.h.h.r.HypervDirectConnectResource] >> (DirectAgent-295:ctx-f5578a04) (logid:d11afe7a) executeRequest >> received response [{"com.cloud.agent.api.StartAnswer":{"vm":{" >> id":1241,"name":"v-1241-VM","type":"ConsoleProxy","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian >> GNU/Linux 7(64-bit)","bootArgs":" >> template\u003ddomP type\u003dconsoleproxy host\u003d172.16.116.219 >> port\u003d8250 name\u003dv-1241-VM zone\u003d9 pod\u003d9 >> guid\u003dProxy.1241 proxy_vm\u003d1241 disable_rp_filter\u003dtrue >> vmpassword\u003dtCUAMd6x eth2ip\u003daa.bb.cc.138 >> eth2mask\u003d255.255.255.0 gateway\u003daa.bb.cc.10 >> eth0ip\u003d169.254.2.35 eth0mask\u003d255.255.0.0 >> eth1ip\u003dxx.yy.zz.163 eth1mask\u003d255.255.255.0 >> mgmtcidr\u003d172.16.116.0/23 localgw\u003dxx.yy.zz.10 >> internaldns1\u003dxx.yy.zz.180 internaldns2\u003dmm.nn.oo.171 >> dns1\u003dxx.yy.zz.180 >> dns2\u003dmm.nn.oo.171","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,,"params":{},"uuid":"b6119681-e6da-4617-a2dd-504e5f82fe4a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"0c3d902f-8840-41d7-8c87-df860ea6ffcd","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"587027e4-e766-30fa-8972-d8ed9515170b","id":13,"poolType":"SMB","host":"xx.yy.zz.223","path":"/primary","port":445,"url":"SMB://xx.yy.zz.223/primary/?ROLE\u003dPrimary\u0026STOREUUID\u003d587027e4-e766-30fa-8972-d8ed9515170b"}},"name":"ROOT-1241","size":0,"path":"0c3d902f-8840-41d7-8c87-df860ea6ffcd","volumeId":1075,"vmName":"v-124VM","accountId":1,"format":"VHD","provisioningType":"THIN","id":1075,"deviceId":0,"hypervisorType":"Hyperv"}},"diskSeq": >> 0,"path":"0c3d902f-8840-41d7-8c87-df860ea6ffcd","type":"ROOT","_details":{"managed":"false","storagePort":"445","storageHost":"xx.yy.zz.223","volumeSize":"0"}}],"nics":[{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"pxeDisable":true,"nicUuid":"42a9765e-b84f-4199-9446-145df4bc68de","uuid":"a4a46926-8caf-4bee-b1a0-70529c3ecac6","ip":"aa.bb.cc.138","netmask":"255.255.255.0","gateway":"aa.bb.cc.10","mac":"06:4d:90:00:00:0d","dns1":"xx.yy.zz.180","dns2":"mm.nn.oo.171","broadcastType":"Native","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"Virtual >> Switch"},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"1cba88d1-5867-4073-a2fd-c8139a8a5a1a","uuid":"e548abca-23da-4568-96e4-3a6aa709566b","ip":"169.254.2.35","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:23","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"pxeDisable":true,"nicUuid":"bb80257d-1578-4643-95b7-63a72f80703a","uuid":"d149dd9f-ac98-411d-9fb0-e587852a9df1","ip":"xx.yy.zz.163","netmask":"255.255.255.0","gateway":"xx.yy.zz.10","mac":"06:0c:48:00:00:3e","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"Virtual >> Switch"}]},"result":false,"details":"com.cloud.agent.api.StartCommand >> fail on exceptionObject reference not set to an instance of an >> object.","contextMap":{},"wait":0}}] >> . >> . >> . >> 2015-09-17 04:45:07,888 INFO [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-57:ctx-fb0c16f6 job-126/job-7169 ctx-d0465230) >> (logid:d11afe7a) Unable to start VM on Host[-94-Routing] due to >> com.cloud.agent.api.StartCommand fail on exceptionObject reference not >> set to an instance of an object. >> --- <end of sample> --- >> >> <cloudstac-hvagent-log-eventviewer.txt> >> > >
