[ https://issues.apache.org/jira/browse/CLOUDSTACK-1244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13585781#comment-13585781 ]
Rohit Yadav commented on CLOUDSTACK-1244: ----------------------------------------- I tried to see the issue, so in scripts/vm/hypervisor/xenserver/xcposs/patch the systemvm.iso key has the value: systemvm.iso=../../../../../vms,0644,/usr/share/xcp/packages/iso/ This forces the mgmt server to find and scp systemvm.iso from that path and systemvm.iso is not available on that path. I just gave it a path to console-proxy/dist and it worked, but I don't know the impact of changing the path will have for debian/rpm packaging; Further I was unable to find what caused this issue on 4.1 branch, log: INFO [xen.resource.CitrixResourceBase] (1895346684@qtp-110990270-3:) Private Network is Pool-wide network associated with eth0 for host 192.168.56.10 INFO [xen.resource.CitrixResourceBase] (1895346684@qtp-110990270-3:) Guest Network is Pool-wide network associated with eth0 for host 192.168.56.10 INFO [xen.resource.CitrixResourceBase] (1895346684@qtp-110990270-3:) Public Network is Pool-wide network associated with eth0 for host 192.168.56.10 INFO [xen.discoverer.XcpServerDiscoverer] (1895346684@qtp-110990270-3:) Host: devcloud connected with hypervisor type: XenServer. Checking CIDR... INFO [agent.manager.DirectAgentAttache] (1895346684@qtp-110990270-3:) StartupAnswer received 1 Interval = 60 WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../ovsgre WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../network/domr//ipassoc.sh WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../network/domr//getDomRVersion.sh WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../network/domr//getRouterStatus.sh WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../../vms/systemvm.iso WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../network/domr//networkUsage.sh WARN [xen.resource.CitrixResourceBase] (DirectAgent-1:) We cannot locate /Bhaisaab/Work/apache-cloudstack/scripts/vm/hypervisor/xenserver/xcposs/../../../../network/domr//l2tp_vpn.sh INFO [network.security.SecurityGroupListener] (1895346684@qtp-110990270-3:) Received a host startup notification INFO [network.security.SecurityGroupListener] (1895346684@qtp-110990270-3:) Scheduled network rules cleanup, interval=2269 INFO [network.security.SecurityGroupListener] (1895346684@qtp-110990270-3:) Received a host startup notification INFO [cloud.resource.ResourceManagerImpl] (1895346684@qtp-110990270-3:) server resources successfully discovered by XCP Agent > fail to push sysmvm.iso onto xen host > -------------------------------------- > > Key: CLOUDSTACK-1244 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1244 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.1.0 > Reporter: sadhu suresh > Assignee: Abhinandan Prateek > Priority: Blocker > Fix For: 4.1.0 > > > Env: > host :xen6.0.2 > MS:513f8b98e9b7818d0b37c9641d9021b77a0cfc45 > System VMs unable to start due to MS fail to find the systemvm.iso. That > leads to Runtime Exception > steps: > 1.prepared the setup in dev env > 2.Add the zone,pod,cluster and xen host details and enable the zone > INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:) Found a > stopped console proxy, bring it up to running pool. proxy vim id : 2 > WARN [network.element.VpcVirtualRouterElement] (secstorage-1:) Network > Ntwk[200|Public|1] is not associated with any VPC > WARN [network.element.VpcVirtualRouterElement] (consoleproxy-1:) Network > Ntwk[200|Public|1] is not associated with any VPC > WARN [network.element.VpcVirtualRouterElement] (secstorage-1:) Network > Ntwk[202|Control|3] is not associated with any VPC > WARN [network.element.VpcVirtualRouterElement] (secstorage-1:) Network > Ntwk[201|Management|2] is not associated with any VPC > WARN [network.element.VpcVirtualRouterElement] (secstorage-1:) Network > Ntwk[203|Storage|4] is not associated with any VPC > WARN [xen.resource.CitrixResourceBase] (DirectAgent-3:) Catch Exception: > class com.cloud.utils.exception.CloudRuntimeException due to > com.cloud.utils.exception.CloudRuntimeException: can not find systemvmiso > com.cloud.utils.exception.CloudRuntimeException: can not find systemvmiso > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1162) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1307) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:517) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > WARN [xen.resource.CitrixResourceBase] (DirectAgent-3:) Unable to start > s-36-VM due to > com.cloud.utils.exception.CloudRuntimeException: can not find systemvmiso > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.createPatchVbd(CitrixResourceBase.java:1162) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1307) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:517) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > WARN [xen.resource.CitrixResourceBase] (DirectAgent-3:) Unable to clean up > VBD due to > You gave an invalid object reference. The object may have recently been > deleted. The class parameter gives the type of reference given, and the > handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:209) > at com.xensource.xenapi.Connection.dispatch(Connection.java:368) > at > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) > at com.xensource.xenapi.VBD.unplug(VBD.java:1059) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1114) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1377) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:517) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > INFO [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:) Unable to start VM > on Host[-3-Routing] due to Unable to start s-36-VM due to > INFO [xen.resource.CitrixResourceBase] (DirectAgent-4:) VM does not exist on > XenServer4a2aea51-5c83-4176-9be5-79d79318aadb > WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:) > Exception while trying to start secondary storage vim > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[SecondaryStorageVm|s-36-VM]Scope=interface > com.cloud.dc.DataCenter; id=1 > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:728) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:466) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:459) > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80) > at > com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:43) > at sun.reflect.GeneratedMethodAccessor48.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621) > at > org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610) > at > org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:264) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:689) > at > com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:121) > at > com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:52) > at > com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104) > at > com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33) > at > com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81) > at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:679) > Work around: > if you clear the Tags on xen host parameter and readd the host will resolve > the issue > 1.put the registered host in CS in maintenance mode > 2.remove the host form Cloudstack > 3.perform param-clear on "tags" param > eg: > xe host-param-clear param-name=tags uuid=4a2aea51-5c83-4176-9be5-79d79318aadb > 4.re add the host into cloudstack > will resolve the issue .after performing the workaround system vms are up and > running -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira