[
https://issues.apache.org/jira/browse/CLOUDSTACK-3788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13732840#comment-13732840
]
Fang Wang edited comment on CLOUDSTACK-3788 at 8/7/13 10:25 PM:
----------------------------------------------------------------
Can you reproduce it now?
I looked at the log and the source code, the source code at
XenserverSnapshotStrategy.java, and SnapshotManagerImpl.java
got changed.
>From the log, there is a snpshot got stuck at deleteSnapshot command. It shows
>destroyed in DB, but it failed:
2013-07-23 18:52:33,744 DEBUG [agent.transport.Request] (Job-Executor-66:job-95
= [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Seq 3-1039532574: Sending { Cmd ,
MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/6/28192719-5193-4e92-8b0a-92b66b1f4a6a","volume":{"uuid":"031802d8-c10a-4038-9acf-b86eba54b647","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ef3dea99-0a1b-35b9-87ec-d8cabe7ed9c1","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/campocr-195-103/primarykvm","port":2049}},"name":"ROOT-6","size":147456,"path":"906ec8f4-e625-4943-aedd-aff3444150b9","volumeId":6,"vmName":"i-3-6-JULYLAST","accountId":3,"format":"QCOW2","id":6},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/export/home/chandan/campocr-195-103/secondary/","_role":"Image"}},"vmName":"i-3-6-JULYLAST","name":"Atoms-VM-1_ROOT-6_20130723235146","hypervisorType":"KVM","id":24}},"wait":0}}]
}
2013-07-23 18:52:34,741 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 2-11944: Processing Seq 2-11944: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-07-23 18:52:34,745 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 2-11944: Sending Seq 2-11944: { Ans: ,
MgmtId: 7471666038533, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-07-23 18:52:37,598 DEBUG [agent.transport.Request]
(AgentManager-Handler-3:null) Seq 3-1039532574: Processing: { Ans: , MgmtId:
7471666038533, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2013-07-23 18:52:37,598 DEBUG [agent.transport.Request] (Job-Executor-66:job-95
= [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Seq 3-1039532574: Received: { Ans:
, MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 10, { Answer } }
2013-07-23 18:52:37,611 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-66:job-95 = [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Failed to
delete snapshot:
java.lang.NullPointerException
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:192)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:496)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>From the DB, the 2 snapshots you created were at the same time: 2013-07-23
>23:51:46
I would suggest do hourly and the weekly snapshot at different time slots, not
at the same time.
was (Author: fangw):
Can you reproduce it now?
I looked at the log and the source code, the source code at
XenserverSnapshotStrategy.java, and SnapshotManagerImpl.java
got changed.
>From the log, there is a snpshot got stuck at deleteSnapshot command. It shows
>destroyed in DB, but it failed:
2013-07-23 18:52:33,744 DEBUG [agent.transport.Request] (Job-Executor-66:job-95
= [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Seq 3-1039532574: Sending { Cmd ,
MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/3/6/28192719-5193-4e92-8b0a-92b66b1f4a6a","volume":{"uuid":"031802d8-c10a-4038-9acf-b86eba54b647","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"ef3dea99-0a1b-35b9-87ec-d8cabe7ed9c1","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/chandan/campocr-195-103/primarykvm","port":2049}},"name":"ROOT-6","size":147456,"path":"906ec8f4-e625-4943-aedd-aff3444150b9","volumeId":6,"vmName":"i-3-6-JULYLAST","accountId":3,"format":"QCOW2","id":6},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/export/home/chandan/campocr-195-103/secondary/","_role":"Image"}},"vmName":"i-3-6-JULYLAST","name":"Atoms-VM-1_ROOT-6_20130723235146","hypervisorType":"KVM","id":24}},"wait":0}}]
}
2013-07-23 18:52:34,741 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 2-11944: Processing Seq 2-11944: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": []\n}","wait":0}}] }
2013-07-23 18:52:34,745 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-5:null) SeqA 2-11944: Sending Seq 2-11944: { Ans: ,
MgmtId: 7471666038533, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-07-23 18:52:37,598 DEBUG [agent.transport.Request]
(AgentManager-Handler-3:null) Seq 3-1039532574: Processing: { Ans: , MgmtId:
7471666038533, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2013-07-23 18:52:37,598 DEBUG [agent.transport.Request] (Job-Executor-66:job-95
= [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Seq 3-1039532574: Received: { Ans:
, MgmtId: 7471666038533, via: 3, Ver: v1, Flags: 10, { Answer } }
2013-07-23 18:52:37,611 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-66:job-95 = [ 669b527b-d1ab-4726-9a22-8363aff5dee4 ]) Failed to
delete snapshot:
java.lang.NullPointerException
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:192)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:496)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>From the DB, the 2 snapshots you created were at the same time: 2013-07-23
>23:51:46
I would suggest to do hourly and the weekly snapshot at the different time, not
at the same time slot.
> [KVM] Weekly Snapshot got stuck in "Allocated State"
> ----------------------------------------------------
>
> Key: CLOUDSTACK-3788
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3788
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Snapshot
> Affects Versions: 4.2.0
> Reporter: Chandan Purushothama
> Assignee: Fang Wang
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: management-server.log.2013-07-23.gz,
> mysql_cloudstack_dump.zip
>
>
> Weekly Snapshot stuck in Allocated State:
> mysql> select * from snapshots where name like
> "Atoms-VM-1_ROOT-6_20130723235146";
> +----+----------------+------------+-----------+-----------+------------------+-----------+------+----------------------------------+--------------------------------------+---------------+------------------+--------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | id | data_center_id | account_id | domain_id | volume_id | disk_offering_id
> | status | path | name | uuid
> | snapshot_type | type_description | size | created
> | removed | backup_snap_id | swift_id | sechost_id | prev_snap_id |
> hypervisor_type | version | s3_id |
> +----+----------------+------------+-----------+-----------+------------------+-----------+------+----------------------------------+--------------------------------------+---------------+------------------+--------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> | 24 | 1 | 3 | 1 | 6 | 1
> | Destroyed | NULL | Atoms-VM-1_ROOT-6_20130723235146 |
> 08a0d2aa-9635-41cd-ba54-5367303bceac | 3 | HOURLY |
> 147456 | 2013-07-23 23:51:46 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> | 25 | 1 | 3 | 1 | 6 | 1
> | Allocated | NULL | Atoms-VM-1_ROOT-6_20130723235146 |
> 1e24a056-be38-4b55-845b-a5672b9fa93c | 5 | WEEKLY |
> 147456 | 2013-07-23 23:51:46 | NULL | NULL | NULL |
> NULL | NULL | KVM | 2.2 | NULL |
> +----+----------------+------------+-----------+-----------+------------------+-----------+------+----------------------------------+--------------------------------------+---------------+------------------+--------+---------------------+---------+----------------+----------+------------+--------------+-----------------+---------+-------+
> 2 rows in set (0.04 sec)
> Attached Management Server logs and cloud database dump
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
