[
https://issues.apache.org/jira/browse/CLOUDSTACK-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625629#comment-13625629
]
Francois Gaudreault edited comment on CLOUDSTACK-1967 at 4/8/13 6:20 PM:
-------------------------------------------------------------------------
If it's by design, then there is still a bug :) (Stacktrace)
I think we should push for a better integration with the API maintenance call.
I know some are spinning external VMs (not managed by CloudStack), but you
should not do that. You can't really do it with Xen by the way, CS wipes the
stuff that it doesn't know.
was (Author: fgaudreault):
If it's by design, then there is still a bug :) (Stacktrace)
I think we should push for a better integration with the API maintenance call.
I know some as external VMs not managed by CloudStack, but you should not do
that. You can't really do it with Xen by the way, CS wipes the stuff that it
doesn't know.
> Maintenance mode not fully working with VMWare
> ----------------------------------------------
>
> Key: CLOUDSTACK-1967
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1967
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: VMware
> Affects Versions: 4.1.0
> Reporter: Francois Gaudreault
> Attachments: managementlog_maint.log
>
>
> When we turn on maintenance mode on one node of the cluster, the VMs are
> migrating fine, but the host is never fully sent to maintenance mode on the
> VMWare side. We do see bunch of stack traces related to the management
> server loosing connectivity, but machine will be available to ICMP for the
> entire process. The host will appear as Disconnected. This is not a critical
> issue (major tho) since the VMs are migrated properly, but the error should
> be investigated/reproduced.
> See the attachments for logs.
> Catalina.out StackTrace:
> INFO [vmware.resource.VmwareResource] (ClusteredAgentManager Timer:)
> VmwareResource network configuration info. private vSwitch: vSwitch0, public
> vSwitch: vSwitch0, guest network: vSwitch0
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
> host hardware info. {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
> host network info.
> {"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"00:26:55:2b:c0:8a"}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
> startup command:
> {"totalSize":0,"poolInfo":{"uuid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":66896003072},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","wait":0}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
> startup command:
> {"totalSize":0,"poolInfo":{"uuid":"ddd756cb-8592-4473-9482-5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732612788224},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-5a4b68085a05","wait":0}
> INFO [agent.manager.DirectAgentAttache] (AgentTaskPool-6:) StartupAnswer
> received 4 Interval = 60
> INFO [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot)
> Executing resource CheckNetworkCommand
> {"networkInfoList":[{"physicalNetworkId":206,"privateNetworkName":"vSwitch0"},{"physicalNetworkId":207,"storageNetworkName":"vSwitch1"},{"physicalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName":"vSwitch2"}],"wait":0}
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Scheduled
> network rules cleanup, interval=1961
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4]
> is unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
> mode
> WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
> connect host 4 to pool Pool[205|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205]
> is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205]
> is unreachable: Unable to get an answer to the modify storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4]
> is unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
> mode
> WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
> connect host 4 to pool Pool[206|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206]
> is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206]
> is unreachable: Unable to get an answer to the modify storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> INFO [vmware.resource.VmwareResource] (DirectAgent-403:yul01vi13.ops.dot)
> Executing resource ReadyCommand: {"dcId":3,"hostId":4,"wait":0}
> INFO [agent.manager.AgentMonitor] (Thread-5:) Found the following agents
> behind on ping: [8, 4]
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Host 4 is
> disconnecting with event ShutdownRequested
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why
> host 8 has disconnected with event PingTimeout
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state
> determined is Up
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent is
> determined to be up and running
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
