[
https://issues.apache.org/jira/browse/CLOUDSTACK-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ram Ganesh updated CLOUDSTACK-1967:
-----------------------------------
Assignee: Sateesh Chodapuneedi
> Maintenance mode not fully working with VMWare
> ----------------------------------------------
>
> Key: CLOUDSTACK-1967
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1967
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: VMware
> Affects Versions: 4.1.0, 4.2.0
> Reporter: Francois Gaudreault
> Assignee: Sateesh Chodapuneedi
> Attachments: managementlog_maint.log
>
>
> When we turn on maintenance mode on one node of the cluster, the VMs are
> migrating fine, but the host is never fully sent to maintenance mode on the
> VMWare side. We do see bunch of stack traces related to the management
> server loosing connectivity, but machine will be available to ICMP for the
> entire process. The host will appear as Disconnected. This is not a critical
> issue (major tho) since the VMs are migrated properly, but the error should
> be investigated/reproduced.
> See the attachments for logs.
> Catalina.out StackTrace:
> INFO [vmware.resource.VmwareResource] (ClusteredAgentManager Timer:)
> VmwareResource network configuration info. private vSwitch: vSwitch0, public
> vSwitch: vSwitch0, guest network: vSwitch0
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
> host hardware info. {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
> host network info.
> {"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"00:26:55:2b:c0:8a"}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
> startup command:
> {"totalSize":0,"poolInfo":{"uuid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":66896003072},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","wait":0}
> INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
> startup command:
> {"totalSize":0,"poolInfo":{"uuid":"ddd756cb-8592-4473-9482-5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732612788224},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-5a4b68085a05","wait":0}
> INFO [agent.manager.DirectAgentAttache] (AgentTaskPool-6:) StartupAnswer
> received 4 Interval = 60
> INFO [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot)
> Executing resource CheckNetworkCommand
> {"networkInfoList":[{"physicalNetworkId":206,"privateNetworkName":"vSwitch0"},{"physicalNetworkId":207,"storageNetworkName":"vSwitch1"},{"physicalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName":"vSwitch2"}],"wait":0}
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Scheduled
> network rules cleanup, interval=1961
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
> host startup notification
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4]
> is unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
> mode
> WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
> connect host 4 to pool Pool[205|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205]
> is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205]
> is unreachable: Unable to get an answer to the modify storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4]
> is unreachable: Host 4: Unable to send class
> com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
> mode
> WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
> connect host 4 to pool Pool[206|VMFS] due to
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206]
> is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206]
> is unreachable: Unable to get an answer to the modify storage pool command
> at
> com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
> at
> com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
> at
> com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
> at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
> at
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
> at
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
> at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> INFO [vmware.resource.VmwareResource] (DirectAgent-403:yul01vi13.ops.dot)
> Executing resource ReadyCommand: {"dcId":3,"hostId":4,"wait":0}
> INFO [agent.manager.AgentMonitor] (Thread-5:) Found the following agents
> behind on ping: [8, 4]
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Host 4 is
> disconnecting with event ShutdownRequested
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why
> host 8 has disconnected with event PingTimeout
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state
> determined is Up
> INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent is
> determined to be up and running
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
