[
https://issues.apache.org/jira/browse/CLOUDSTACK-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan reopened CLOUDSTACK-2428:
---------------------------------------------
Reopening this issue:
Tested with following set up:
Advanced zone set up with 1 cluster having 2 Xenserver - 6.1.0-59235p .
Deploy few Vms.
Pull the network cable of the master host.
Waited for the host to be marked as "Down". This did not happen even agter 1/2
hour.
[root@cs-nightly-ms2 management]# grep "Investigating why host 4 has
disconnected with event PingTimeout" management-server.log
2013-09-17 17:04:11,586 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-1:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:05:11,603 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-6:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:06:11,654 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-16:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:07:11,700 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:08:11,738 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-15:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:09:11,768 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-4:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:10:11,822 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:11:11,865 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-5:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:12:11,921 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-3:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:13:11,941 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-5:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:14:11,974 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-2:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:15:11,988 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-3:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:16:12,017 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-5:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:17:12,045 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-1:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:18:12,074 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-10:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:19:12,111 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-11:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:20:12,139 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-1:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:21:12,176 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-10:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:22:12,206 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-4:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:23:12,240 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-7:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:24:12,271 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-2:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:25:12,302 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-13:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:26:12,343 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-16:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:27:12,365 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-5:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:28:12,397 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-12:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:29:12,426 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-15:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:30:12,448 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-4:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:31:12,481 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-8:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:32:12,519 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-1:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:33:12,553 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-10:null) Investigating why host 4 has disconnected with event
PingTimeout
2013-09-17 17:34:12,591 INFO [agent.manager.AgentManagerImpl]
(AgentTaskPool-9:null) Investigating why host 4 has disconnected with event
PingTimeout
Following exception seen in management server logs:
2013-09-17 17:10:32,416 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-365:null) Logging on as the slave to 10.223.50.195
2013-09-17 17:10:32,417 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-312:null) Seq 4-1945829646: Exception Caught while executing
command
com.cloud.utils.exception.CloudRuntimeException: Unable to create slave
connection to host(89e1f641-cd46-4bf1-a43d-51a894f03ea3) due to org.apache.xm
lrpc.XmlRpcException: Failed to read server's response: No route to host
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:580)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5812)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:2615)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:493)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.apache.xmlrpc.XmlRpcException: Failed to read server's response:
No route to host
at
org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:161)
at
org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143)
at
org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69)
at
org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137)
at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126)
at com.xensource.xenapi.Connection.dispatch(Connection.java:303)
at
com.xensource.xenapi.Session.slaveLocalLoginWithPassword(Session.java:587)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.slaveLocalLoginWithPassword(XenServerConnectionPool.java:688)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:574)
... 14 more
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
at java.net.Socket.connect(Socket.java:546)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:584)
at sun.net.NetworkClient.doConnect(NetworkClient.java:173)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:275)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:332)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:876)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at
sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:979)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at
org.apache.xmlrpc.client.XmlRpcSunHttpTransport.writeRequest(XmlRpcSunHttpTransport.java:104)
at
org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151)
... 24 more
After sometime , the following exception is seen:
2013-09-17 17:11:09,832 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-344:null) Seq 4-1945829667: Exception Caught while executing
command
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
slave 10.223.50.194 to 10.223.50.195 due to org.apache.xmlrpc.XmlRpcException:
Failed to read server's response: No route to host
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5812)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1756)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:549)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-09-17 17:11:09,834 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-344:null) Seq 4-1945829667: Response Received:
2013-09-17 17:11:09,835 DEBUG [agent.transport.Request] (DirectAgent-344:null)
Seq 4-1945829667: Processing: { Ans: , MgmtId: 161197867246747, via: 4, Ver:
v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
Unable to reset master of slave 10.223.50.194 to 10.223.50.195 due to
org.apache.xmlrpc.XmlRpcException: Failed to read server's response: No route
to host","wait":0}}] }
2013-09-17 17:11:09,835 DEBUG [agent.manager.AgentAttache]
(DirectAgent-344:null) Seq 4-1945829667: Unable to find listener.
2013-09-17 17:11:09,938 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-18:null) Catch Exception: com.xensource.xenapi.Types$HostOffline
Host is offline 10.223.50.194 due to You attempted an operation which involves
a host which could not be contacted.
2013-09-17 17:11:09,938 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-18:null) Trying to reset master of slave 10.223.50.194 to
10.223.50.195
2013-09-17 17:11:09,944 DEBUG [xen.resource.XenServerConnectionPool]
(DirectAgent-18:null) localLogout has problem Failed to read server's response:
No route to host
2013-09-17 17:11:09,945 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-18:null) Seq 4-1945829668: Exception Caught while executing command
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
slave 10.223.50.194 to 10.223.50.195 due to org.apache.xmlrpc.XmlRpcException:
Failed to read server's response: No route to host
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5812)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1756)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:549)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Will attach management server logs.
> HA - When the master host is disconnected , the host status contines to
> remain in "Up" state because of
> com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
> slave
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-2428
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2428
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Build from pvaln
> Reporter: Sangeetha Hariharan
> Assignee: Koushik Das
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: logs_7_29, logs.rar
>
>
> 1. Advance zone with 1 cluster with 2 hosts. Create Shared network with
> private vlan.
> 2. Deploy few HA enabled Vms in this network.
> 3. pull network cable for one of the host.
> When cloudstack detects that the host is disconnected , it is not able to out
> the host in disconnected state and start HA for Vms that are HA enabeld,
> I see the following exception in the management server logs:
> 2013-05-09 17:15:55,576 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-267:null) Seq 1-1435828229: Executing request
> 2013-05-09 17:15:55,602 DEBUG [xen.resource.XenServerConnectionPool]
> (DirectAgent-267:null) Catch Exception:
> com.xensource.xenapi.Types$HostOffline Host is offline 10.223.81.62 due to
> You attempted an operation which involves a host which could not be contacted.
> 2013-05-09 17:15:55,603 DEBUG [xen.resource.XenServerConnectionPool]
> (DirectAgent-267:null) Trying to reset master of slave 10.223.81.62 to
> 10.223.81.61
> 2013-05-09 17:16:02,319 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-265:null) can not ping xenserver
> 520d4994-8b1f-4dda-b51d-2ee63750abf6
> 2013-05-09 17:16:02,319 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-265:null) Unable to get current status on 1
> 2013-05-09 17:16:02,321 INFO [agent.manager.AgentManagerImpl]
> (AgentTaskPool-11:null) Investigating why host 1 has disconnected with event
> AgentDisconnected
> 2013-05-09 17:16:02,321 DEBUG [agent.manager.AgentManagerImpl]
> (AgentTaskPool-11:null) checking if agent (1) is alive
> 2013-05-09 17:16:02,323 DEBUG [agent.transport.Request]
> (AgentTaskPool-11:null) Seq 1-1435828294: Sending { Cmd , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 100011,
> [{"CheckHealthCommand":{"wait":50}}] }
> 2013-05-09 17:16:02,323 DEBUG [agent.transport.Request]
> (AgentTaskPool-11:null) Seq 1-1435828294: Executing: { Cmd , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 100011,
> [{"CheckHealthCommand":{"wait":50}}] }
> 2013-05-09 17:16:02,323 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-271:null) Seq 1-1435828294: Executing request
> 2013-05-09 17:16:04,035 DEBUG [agent.manager.AgentAttache]
> (AgentTaskPool-10:null) Seq 6-474349576: Waiting some more time because this
> is the current command
> 2013-05-09 17:16:04,040 DEBUG [xen.resource.XenServerConnectionPool]
> (DirectAgent-268:null) localLogout has problem Failed to read server's
> response: connect timed out
> 2013-05-09 17:16:04,040 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-268:null) Seq 1-1435828292: Exception Caught while executing
> command
> com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
> slave 10.223.81.62 to 10.223.81.61 due to org.apache.xmlrpc.XmlRpcException:
> Failed to read server's response: connect timed out
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5639)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1682)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:524)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-05-09 17:16:04,041 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-268:null) Seq 1-1435828292: Response Received:
> 2013-05-09 17:16:04,041 DEBUG [agent.transport.Request]
> (DirectAgent-268:null) Seq 1-1435828292: Processing: { Ans: , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 10,
> [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Unable to reset master of slave 10.223.81.62 to 10.223.81.61 due to
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect
> timed out","wait":0}}] }
> 2013-05-09 17:16:04,041 DEBUG [agent.transport.Request]
> (AgentTaskPool-5:null) Seq 1-1435828292: Received: { Ans: , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 10, { Answer } }
> 2013-05-09 17:16:04,041 DEBUG [cloud.ha.AbstractInvestigatorImpl]
> (AgentTaskPool-5:null) host (10.223.81.50) cannot be pinged, returning null
> ('I don't know')
> 2013-05-09 17:16:04,041 DEBUG [cloud.ha.UserVmDomRInvestigator]
> (AgentTaskPool-5:null) sending ping from (5) to agent's host ip address
> (10.223.81.50)
> 2013-05-09 17:16:04,043 DEBUG [agent.transport.Request]
> (AgentTaskPool-5:null) Seq 5-2082341067: Sending { Cmd , MgmtId:
> 7647994577963, via: 5, Ver: v1, Flags: 100011,
> [{"PingTestCommand":{"_computingHostIp":"10.223.81.50","wait":20}}] }
> 2013-05-09 17:16:04,043 DEBUG [agent.transport.Request]
> (AgentTaskPool-5:null) Seq 5-2082341067: Executing: { Cmd , MgmtId:
> 7647994577963, via: 5, Ver: v1, Flags: 100011,
> [{"PingTestCommand":{"_computingHostIp":"10.223.81.50","wait":20}}] }
> 2013-05-09 17:16:04,043 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-272:null) Seq 5-2082341067: Executing request
> 2013-05-09 17:16:04,053 DEBUG [xen.resource.XenServerConnectionPool]
> (DirectAgent-91:null) localLogout has problem Failed to read server's
> response: connect timed out
> 2013-05-09 17:16:04,053 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-91:null) Seq 1-1435828293: Exception Caught while executing
> command
> com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
> slave 10.223.81.62 to 10.223.81.61 due to org.apache.xmlrpc.XmlRpcException:
> Failed to read server's response: connect timed out
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5639)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1682)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:524)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-05-09 17:16:04,054 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-91:null) Seq 1-1435828293: Response Received:
> 2013-05-09 17:16:04,054 DEBUG [agent.transport.Request] (DirectAgent-91:null)
> Seq 1-1435828293: Processing: { Ans: , MgmtId: 7647994577963, via: 1, Ver:
> v1, Flags: 10,
> [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Unable to reset master of slave 10.223.81.62 to 10.223.81.61 due to
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect
> timed out","wait":0}}] }
> 2013-05-09 17:16:04,055 DEBUG [agent.transport.Request]
> (AgentTaskPool-7:null) Seq 1-1435828293: Received: { Ans: , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 10, { Answer } }
> 2013-05-09 17:16:04,055 DEBUG [cloud.ha.AbstractInvestigatorImpl]
> (AgentTaskPool-7:null) host (10.223.81.52) cannot be pinged, returning null
> ('I don't know')
> 2013-05-09 17:16:04,055 DEBUG [cloud.ha.UserVmDomRInvestigator]
> (AgentTaskPool-7:null) sending ping from (5) to agent's host ip address
> (10.223.81.52)
> 2013-05-09 17:16:04,057 DEBUG [agent.transport.Request]
> (AgentTaskPool-7:null) Seq 5-2082341068: Sending { Cmd , MgmtId:
> 7647994577963, via: 5, Ver: v1, Flags: 100011,
> [{"PingTestCommand":{"_computingHostIp":"10.223.81.52","wait":20}}] }
> 2013-05-09 17:16:04,057 DEBUG [agent.manager.AgentAttache]
> (AgentTaskPool-14:null) Seq 3-1752367195: Waiting some more time because this
> is the current command
> 2013-05-09 17:16:04,057 DEBUG [agent.transport.Request]
> (AgentTaskPool-7:null) Seq 5-2082341068: Executing: { Cmd , MgmtId:
> 7647994577963, via: 5, Ver: v1, Flags: 100011,
> [{"PingTestCommand":{"_computingHostIp":"10.223.81.52","wait":20}}] }
> 2013-05-09 17:16:04,057 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-91:null) Seq 5-2082341068: Executing request
> 2013-05-09 17:16:05,175 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
> (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
> 2013-05-09 17:16:05,614 DEBUG [xen.resource.XenServerConnectionPool]
> (DirectAgent-267:null) localLogout has problem Failed to read server's
> response: connect timed out
> 2013-05-09 17:16:05,614 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-267:null) Seq 1-1435828229: Exception Caught while executing
> command
> com.cloud.utils.exception.CloudRuntimeException: Unable to reset master of
> slave 10.223.81.62 to 10.223.81.61 due to org.apache.xmlrpc.XmlRpcException:
> Failed to read server's response: connect timed out
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:443)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:661)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:5639)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:7725)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:570)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-05-09 17:16:05,615 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-267:null) Seq 1-1435828229: Response Received:
> 2013-05-09 17:16:05,615 DEBUG [agent.transport.Request]
> (DirectAgent-267:null) Seq 1-1435828229: Processing: { Ans: , MgmtId:
> 7647994577963, via: 1, Ver: v1, Flags: 10,
> [{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
> Unable to reset master of slave 10.223.81.62 to 10.223.81.61 due to
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect
> timed out","wait":0}}] }
> 2013-05-09 17:16:05,704 DEB
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
