Hi, I restartd All hosts (one mgr and xenserver) again.
Following is the error log.
2015-07-20 15:33:49,688 INFO [c.c.u.e.CSExceptionErrorCode]
(StatsCollector-3:ctx-692a5392) Could not find exception:
com.cloud.exception.OperationTimedoutException in error code list for
exceptions
2015-07-20 15:33:49,688 WARN [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Timed out
on null
2015-07-20 15:33:49,689 DEBUG [c.c.a.m.AgentAttache]
(StatsCollector-3:ctx-692a5392) Seq 1-3176445112179752972: Cancelling.
2015-07-20 15:33:49,689 DEBUG [c.c.s.StorageManagerImpl]
(StatsCollector-3:ctx-692a5392) Unable to send storage pool command to
Pool[1|NetworkFilesystem] via 1
com.cloud.exception.OperationTimedoutException: Commands
3176445112179752972 to Host 1 timed out after 3600
at
com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:436)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:433)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:362)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1000)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.lang.Thread.run(Thread.java:745)
2015-07-20 15:33:49,689 INFO [c.c.s.StatsCollector]
(StatsCollector-3:ctx-692a5392) Unable to reach
Pool[1|NetworkFilesystem]
com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:1]
is unreachable: Unable to send command to the pool
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:1010)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:392)
at
com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:406)
at
com.cloud.server.StatsCollector$StorageCollector.runInContext(StatsCollector.java:642)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----------
Cao Tong
On 07/20/2015 02:52 PM, tony_caot...@163.com wrote:
No, no one's IP was changed.
1. In xenserver I can not login systemvms using the internal IP like
'169.254.1.112', There shoud be a bridge network for this right?
it is
gone.
2. I try to delete xenserver host from CS on web, it also failed with
lots of logs like following, then memory is full and mangement down...
2015-07-20 14:47:30,580 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-15:null) Seq 1-7282039122481381399:
Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
192405008094602
2015-07-20 14:47:30,582 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-5:null) Seq 1-7282039122481381399: Forwarding
Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
192405008094602
2015-07-20 14:47:30,583 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-1:null) Seq 1-7282039122481381399: Forwarding
Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
192405008094602
2015-07-20 14:47:30,584 DEBUG [c.c.a.m.ClusteredAgentAttache]
(AgentManager-Handler-14:null) Seq 1-7282039122481381399:
Forwarding Seq
1-7282039122481381399: { Cmd , MgmtId: 104062526015411, via: 1,
Ver: v1,
Flags: 100111, [{"com.cloud.agent.api.MaintainCommand":{"wait":0}}]
} to
192405008094602
I guess that, is there some service or daemons working for CS is
not up
on Xenserver ?
-----------
Cao Tong
On 07/20/2015 02:35 PM, Rajani Karuturi wrote:
Did the management server ip change?
management server ip in the configuration table is used my systemvms.
select * from configuration where name like 'host';
If it changed, correct the value in db and restart systemvms.
~Rajani
On Mon, Jul 20, 2015 at 11:56 AM,<tony_caot...@163.com> wrote:
Hello,
I shutdown my cs-manager and xenserver last weekend, And now the
ssvm
and cpvm is disconnect, thost two was runing on xenserver. so What
should i do right now ?
Please anybody help me and thanks.
In xenserver I found that the three system VMs are not running.
my xenserver seems can not reconnect to CS-manager. and it seams
not
under control of CS.
What is the right steps of shutdown all CS group machines and resume
them?
How can i let my xenserver reconnected ?
Thanks,
--
-----------
Cao Tong
