[
https://issues.apache.org/jira/browse/CLOUDSTACK-4228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kelven Yang reassigned CLOUDSTACK-4228:
---------------------------------------
Assignee: Kelven Yang
> [VMWARE]NPE & operation timed out issues while migrating volumes across
> different Zone wide primary storages
> ------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4228
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4228
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Storage Controller, VMware
> Affects Versions: 4.2.0
> Reporter: Sailaja Mada
> Assignee: Kelven Yang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: apilog.log, db.dmp, management-server.log
>
>
> Steps:
> 1. Upgraded from 2.2.16 to 4.2 with VMWARE cluster
> 2. Configured new zone with VMWARE cluster of 2 hosts with zone wide primary
> storage
> 3. Added one more Zone wide primary storage to this new zone
> 4. Deploy instance on this zone
> 5. Add new DATA volume and attach to this instance
> 6. Tried to migrate ROOT/DATA volume of this instance from Zone wide primary
> storage 1 to second Zone wide primary storage which is added to the same zone
> Observation:
> 1. NPE & operation timed out issues are observed while migrating volumes
> across different Zone wide primary storage's
> 2. It is observed 3 times when tried about 6 times.
> log & DB dumb is attached . This is a single node management server
> 2013-08-10 00:02:25,453 DEBUG [agent.manager.AgentAttache]
> (StorageManager-Scavenger-4:null) Seq 18-1413808894: Waiting some more time
> because this is the current command
> 2013-08-10 00:02:25,453 INFO [utils.exception.CSExceptionErrorCode]
> (StorageManager-Scavenger-4:null) Could not find exception:
> com.cloud.exception.OperationTimedoutException in error code list for
> exceptions
> 2013-08-10 00:02:25,453 WARN [agent.manager.AgentAttache]
> (StorageManager-Scavenger-4:null) Seq 18-1413808894: Timed out on Seq
> 18-1413808894: { Cmd , MgmtId: 90310994128556, via: 18, Ver: v1, Flags:
> 100111,
> [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
> }
> 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache]
> (StorageManager-Scavenger-4:null) Seq 18-1413808894: Cancelling.
> 2013-08-10 00:02:25,454 DEBUG [agent.manager.AgentAttache]
> (StorageManager-Scavenger-4:null) Seq 18-1413808894: No more commands found
> 2013-08-10 00:02:25,454 DEBUG [cloud.storage.StorageManagerImpl]
> (StorageManager-Scavenger-4:null) Unable to send storage pool command to
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5760aaa2 via 18
> com.cloud.exception.OperationTimedoutException: Commands 1413808894 to Host
> 18 timed out after 3600
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:430)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:486)
> at
> com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:439)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:987)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:438)
> at
> com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:452)
> at
> com.cloud.template.TemplateManagerImpl.evictTemplateFromStoragePool(TemplateManagerImpl.java:793)
> at
> com.cloud.storage.StorageManagerImpl.cleanupStorage(StorageManagerImpl.java:1039)
> at
> com.cloud.storage.StorageManagerImpl$StorageGarbageCollector.run(StorageManagerImpl.java:1335)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request]
> (StorageManager-Scavenger-4:null) Seq 14-647496884: Sending { Cmd , MgmtId:
> 90310994128556, via: 14, Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
> }
> 2013-08-10 00:02:25,456 DEBUG [agent.transport.Request]
> (StorageManager-Scavenger-4:null) Seq 14-647496884: Executing: { Cmd ,
> MgmtId: 90310994128556, via: 14, Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.storage.DestroyCommand":{"volume":{"id":21,"mountPoint":"/cpg_vol/sailaja/upgdvsps3","path":"00399c8254793ab780eb799da277ba80","size":0,"storagePoolType":"NetworkFilesystem","storagePoolUuid":"7ecc6ea4-0e87-3257-90f9-c2e97de50040","deviceId":0},"wait":0}}]
> }
> 2013-08-10 00:02:25,456 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-151:null) Seq 14-647496884: Executing request
> 2013-08-10 00:02:25,457 WARN [agent.manager.DirectAgentAttache]
> (DirectAgent-151:null) Seq 14-647496884: Exception caught
> java.lang.NullPointerException
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:195)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-10 00:02:26,138 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-13:null) SeqA 10-13769: Processing Seq 10-13769: { Cmd
> , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:26,145 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-13:null) SeqA 10-13769: Sending Seq 10-13769: { Ans: ,
> MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:31,705 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-10:null) SeqA 16-11861: Processing Seq 16-11861: { Cmd
> , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:31,712 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-10:null) SeqA 16-11861: Sending Seq 16-11861: { Ans: ,
> MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:36,142 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-11:null) SeqA 10-13770: Processing Seq 10-13770: { Cmd
> , MgmtId: -1, via: 10, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1138,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:36,151 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-11:null) SeqA 10-13770: Sending Seq 10-13770: { Ans: ,
> MgmtId: 90310994128556, via: 10, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-08-10 00:02:36,707 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-14:null) SeqA 16-11862: Processing Seq 16-11862: { Cmd
> , MgmtId: -1, via: 16, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1145,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2013-08-10 00:02:36,710 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-14:null) SeqA 16-11862: Sending Seq 16-11862: { Ans: ,
> MgmtId: 90310994128556, via: 16, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
