Francois Gaudreault created CLOUDSTACK-1967:
-----------------------------------------------
Summary: Maintenance mode not fully working with VMWare
Key: CLOUDSTACK-1967
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1967
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: VMware
Affects Versions: 4.1.0
Reporter: Francois Gaudreault
Attachments: managementlog_maint.log
When we turn on maintenance mode on one node of the cluster, the VMs are
migrating fine, but the host is never fully sent to maintenance mode on the
VMWare side. We do see bunch of stack traces related to the management server
loosing connectivity, but machine will be available to ICMP for the entire
process. The host will appear as Disconnected. This is not a critical issue
(major tho) since the VMs are migrated properly, but the error should be
investigated/reproduced.
See the attachments for logs.
Catalina.out StackTrace:
INFO [vmware.resource.VmwareResource] (ClusteredAgentManager Timer:)
VmwareResource network configuration info. private vSwitch: vSwitch0, public
vSwitch: vSwitch0, guest network: vSwitch0
INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
host hardware info. {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}
INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on
host network info.
{"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"00:26:55:2b:c0:8a"}
INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
startup command:
{"totalSize":0,"poolInfo":{"uuid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":66896003072},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","wait":0}
INFO [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage
startup command:
{"totalSize":0,"poolInfo":{"uuid":"ddd756cb-8592-4473-9482-5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732612788224},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-5a4b68085a05","wait":0}
INFO [agent.manager.DirectAgentAttache] (AgentTaskPool-6:) StartupAnswer
received 4 Interval = 60
INFO [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot)
Executing resource CheckNetworkCommand
{"networkInfoList":[{"physicalNetworkId":206,"privateNetworkName":"vSwitch0"},{"physicalNetworkId":207,"storageNetworkName":"vSwitch1"},{"physicalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName":"vSwitch2"}],"wait":0}
INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
host startup notification
INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Scheduled
network rules cleanup, interval=1961
INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
host startup notification
INFO [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a
host startup notification
WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4] is
unreachable: Host 4: Unable to send class
com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
mode
WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
connect host 4 to pool Pool[205|VMFS] due to
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205] is
unreachable: Unable to get an answer to the modify storage pool command
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205] is
unreachable: Unable to get an answer to the modify storage pool command
at
com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
at
com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4] is
unreachable: Host 4: Unable to send class
com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance
mode
WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to
connect host 4 to pool Pool[206|VMFS] due to
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206] is
unreachable: Unable to get an answer to the modify storage pool command
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206] is
unreachable: Unable to get an answer to the modify storage pool command
at
com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
at
com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
at
com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
at
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
at
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
at
com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
INFO [vmware.resource.VmwareResource] (DirectAgent-403:yul01vi13.ops.dot)
Executing resource ReadyCommand: {"dcId":3,"hostId":4,"wait":0}
INFO [agent.manager.AgentMonitor] (Thread-5:) Found the following agents
behind on ping: [8, 4]
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Host 4 is
disconnecting with event ShutdownRequested
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why
host 8 has disconnected with event PingTimeout
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state determined
is Up
INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent is determined
to be up and running
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
