[
https://issues.apache.org/jira/browse/CLOUDSTACK-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859014#comment-13859014
]
edison su commented on CLOUDSTACK-5262:
---------------------------------------
sounds like it's related to xapi error:
>From xenserver log, backupsnapshot succeeded:
Nov 25 01:07:23 localhost SM: [20038] Called backupSnapshot with {'isISCSI':
'false', 'localMountPoint':
'/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947',
'secondaryStorageMountPath':
'10.223.110.233:/export/home/sangeetha/felton/xen/secondary', 'backupUuid':
'07f55f61-e1a5-4880-a9a0-37e063e88e39', 'snapshotUuid':
'ee56cec0-632d-469f-9326-d683e9fc1425', 'prevBackupUuid':
'snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71', 'path':
'snapshots/9/73', 'primaryStorageSRUuid':
'69ec4a37-f19c-8951-8b52-ebe3437f4b5e'}
Nov 25 01:07:23 localhost SM: [20038] primarySRPath:
/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e
Nov 25 01:07:23 localhost SM: [20038] ['/usr/bin/vhd-util', 'query', '--debug',
'-p', '-n',
'/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/ee56cec0-632d-469f-9326-d683e9fc1425.vhd']
Nov 25 01:07:23 localhost SM: [20038] pread SUCCESS
Nov 25 01:07:23 localhost SM: [20038] Base copy of snapshotUuid:
ee56cec0-632d-469f-9326-d683e9fc1425 is 3641afa6-419b-447c-a95a-fc61eb12653c
Nov 25 01:07:23 localhost SM: [20038] Base copy path:
/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/3641afa6-419b-447c-a95a-fc61eb12653c.vhd
Nov 25 01:07:23 localhost SM: [20038] The remote snapshots directory has
already been mounted on
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947
Nov 25 01:07:23 localhost SM: [20038] Backups dir
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73
Nov 25 01:07:23 localhost SM: [20038] Back up
3641afa6-419b-447c-a95a-fc61eb12653c to Secondary Storage as
07f55f61-e1a5-4880-a9a0-37e063e88e39
Nov 25 01:07:23 localhost SM: [20038] Starting to copy
/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/3641afa6-419b-447c-a95a-fc61eb12653c.vhd
to
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd
Nov 25 01:07:23 localhost SM: [20038] ['dd',
'if=/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/3641afa6-419b-447c-a95a-fc61eb12653c.vhd',
'of=/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd',
'bs=4M']
Nov 25 01:07:23 localhost SM: [20043] #### VMOPS enter gethostvmstats ####
Nov 25 01:07:24 localhost SM: [20043] #### VMOPS exit gethostvmstats ####
Nov 25 01:07:24 localhost SM: [20050]
nfs-on-slave.check(/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/50b3372b-808e-471a-a7b0-2c9440ce89af.vhd)
Nov 25 01:07:24 localhost SM: [20054]
nfs-on-slave.check(/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/6ba40335-e710-43eb-960a-a5c097e5c2dd.vhd)
Nov 25 01:07:24 localhost SM: [20070]
nfs-on-slave.check(/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/ef005753-44de-444e-99e6-96f67983c64f.vhd)
Nov 25 01:07:25 localhost SM: [20038] pread SUCCESS
Nov 25 01:07:25 localhost SM: [20038] Successfully copied
/var/run/sr-mount/69ec4a37-f19c-8951-8b52-ebe3437f4b5e/3641afa6-419b-447c-a95a-fc61eb12653c.vhd
to
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd
Nov 25 01:07:25 localhost SM: [20038] ['/usr/bin/vhd-util', 'set', '--debug',
'-n',
'/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd',
'-f', 'hidden', '-v', '0']
Nov 25 01:07:25 localhost SM: [20038] pread SUCCESS
Nov 25 01:07:25 localhost SM: [20038] ['/opt/xensource/bin/vhd-util', 'modify',
'-p',
'/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71.vhd',
'-n',
'/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd']
Nov 25 01:07:25 localhost SM: [20038] pread SUCCESS
Nov 25 01:07:25 localhost SM: [20038] Successfully set parent of
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/07f55f61-e1a5-4880-a9a0-37e063e88e39.vhd
to
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71.vhd
but mgt server callhostplugin got exception:
2013-11-25 08:16:16,738 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-a502b0ee) callHostPlugin failed for cmd: backupSnapshot
with args secondaryStorageMountPath:
10.223.110.233:/export/home/sangeetha/felton/xen/secondary, localMountPoint:
/var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947, backupUuid:
07f55f61-e1a5-4880-a9a0-37e063e88e39, path: snapshots/9/73, prevBackupUuid:
snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71, snapshotUuid:
ee56cec0-632d-469f-9326-d683e9fc1425, isISCSI: false, primaryStorageSRUuid:
69ec4a37-f19c-8951-8b52-ebe3437f4b5e, due to HandleInvalid clazz:task,
handle:OpaqueRef:2d1621a0-7c9e-ea80-48ef-c7114ce3a63d
2013-11-25 08:16:16,740 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-364:ctx-a502b0ee) unable to destroy
task(com.xensource.xenapi.Task@4b81265b) on
host(b8c15cde-f075-487b-86bc-9ef564900887) due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
> Few of the snapshot creation from ROOT volume fails when there are
> concurrent snapshots in progress.
> -----------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5262
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5262
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: Build from 4.3
> Reporter: Sangeetha Hariharan
> Assignee: edison su
> Priority: Critical
> Fix For: 4.3.0
>
> Attachments: management-server.rar, xen1-2.rar, xen1.rar, xen2.rar
>
>
> Steps to reproduce the problem:
> Set up - Advanced zone with 2 Xenserver 6.2 hosts
> Deploy 10 Vms in each of the hosts , so we start with 20 Vms.
> We are constantly writing to the ROOT volume ( print timestamp every 1
> minute).
> Start concurrent snapshots for ROOT volumes for all the Vms.
> Few of the snapshot jobs ( 4 of them) are failing due to the following
> exception:
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request]
> (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Sending { Cmd
> , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com), Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
> }
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.t.Request]
> (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Executing: {
> Cmd , MgmtId: 112516401760401, via: 2(Rack3Host23.lab.vmops.com), Ver: v1,
> Flags: 100011,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ee56cec0-632d-469f-9326-d683e9fc1425","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"d3975b00-d9c3-418a-9a39-f09139754a32","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/9/73","volume":{"uuid":"aee2805e-6394-47a6-9360-65829ce61929","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"babe1c7d-3154-3b46-addc-837ca2b7f07d","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.231","path":"/export/home/sangeetha/felton/xen/primary","port":2049,"url":"NetworkFilesystem://10.223.110.231//export/home/sangeetha/felton/xen/primary/?ROLE=Primary&STOREUUID=babe1c7d-3154-3b46-addc-837ca2b7f07d"}},"name":"ROOT-73","size":21474836480,"path":"d1988b67-9f8f-4685-9bfd-a171238e8abd","volumeId":73,"vmName":"i-9-73-MyTestVM","accountId":9,"format":"VHD","id":73,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.233:/export/home/sangeetha/felton/xen/secondary","_role":"Image"}},"vmName":"i-9-73-MyTestVM","name":"TestVM-tiny-host-0ps-0-3_ROOT-73_20131125131535","hypervisorType":"XenServer","id":296,"quiescevm":false}},"executeInSequence":false,"wait":21600}}]
> }
> 2013-11-25 08:16:14,239 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Executing request
> 2013-11-25 08:16:16,738 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-364:ctx-a502b0ee) callHostPlugin failed for cmd: backupSnapshot
> with args secondaryStorageMountPath:
> 10.223.110.233:/export/home/sangeetha/felton/xen/secondary, localMountPoint:
> /var/run/cloud_mount/e8eabfa9-3012-36f4-adf5-895bbf085947, backupUuid:
> 07f55f61-e1a5-4880-a9a0-37e063e88e39, path: snapshots/9/73, prevBackupUuid:
> snapshots/9/73/20ebe402-5c2f-4a61-a0c4-525d97ebfa71, snapshotUuid:
> ee56cec0-632d-469f-9326-d683e9fc1425, isISCSI: false, primaryStorageSRUuid:
> 69ec4a37-f19c-8951-8b52-ebe3437f4b5e, due to HandleInvalid clazz:task,
> handle:OpaqueRef:2d1621a0-7c9e-ea80-48ef-c7114ce3a63d
> 2013-11-25 08:16:16,740 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-364:ctx-a502b0ee) unable to destroy
> task(com.xensource.xenapi.Task@4b81265b) on
> host(b8c15cde-f075-487b-86bc-9ef564900887) due to
> You gave an invalid object reference. The object may have recently been
> deleted. The class parameter gives the type of reference given, and the
> handle parameter echoes the bad value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.Task.destroy(Task.java:616)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:4067)
> at
> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1131)
> at
> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,097 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-2:null) SeqA 3-37675: Processing Seq 3-37675: { Cmd ,
> MgmtId: -1, via: 3, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2013-11-25 08:16:17,100 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-2:null) SeqA 3-37675: Sending Seq 3-37675: { Ans: ,
> MgmtId: 112516401760401, via: 3, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-364:ctx-a502b0ee) Could not copy backupUuid: null from primary
> storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
> 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> 2013-11-25 08:16:17,755 WARN [c.c.h.x.r.XenServerStorageProcessor]
> (DirectAgent-364:ctx-a502b0ee) BackupSnapshot Failed due to Could not copy
> backupUuid: null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to
> secondary storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary
> due to null
> com.cloud.utils.exception.CloudRuntimeException: Could not copy backupUuid:
> null from primary storage 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary
> storage 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at
> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1160)
> at
> com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1320)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:90)
> at
> com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:612)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
> at
> com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Response Received:
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request]
> (DirectAgent-364:ctx-a502b0ee) Seq 2-756819795: Processing: { Ans: , MgmtId:
> 112516401760401, via: 2, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot
> Failed due to Could not copy backupUuid: null from primary storage
> 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
> 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to
> null","wait":0}}] }
> 2013-11-25 08:16:17,756 DEBUG [c.c.a.t.Request]
> (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Seq 2-756819795: Received: {
> Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, { CopyCmdAnswer }
> }
> 2013-11-25 08:16:17,768 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
> Could not copy backupUuid: null from primary storage
> 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
> 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
> at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy160.takeSnapshot(Unknown Source)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
> at
> com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
> at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy232.takeSnapshot(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at
> com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2013-11-25 08:16:17,775 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Job-Executor-165:ctx-bae63a1a ctx-4f637cc6) Take snapshot: 73 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
> at sun.reflect.GeneratedMethodAccessor444.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy160.takeSnapshot(Unknown Source)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1342)
> at
> com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1402)
> at sun.reflect.GeneratedMethodAccessor443.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy232.takeSnapshot(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
> at
> com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot
> Failed due to Could not copy backupUuid: null from primary storage
> 69ec4a37-f19c-8951-8b52-ebe3437f4b5e to secondary storage
> 10.223.110.233:/export/home/sangeetha/felton/xen/secondary due to null
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:138)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:289)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
> ... 42 more
> 2013-11-25 08:16:17,776 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-165:ctx-bae63a1a) Complete async job-631, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for volume 73"}
> mysql> select job_cmd , account_id , job_result, job_result_code, job_status
> from async_job where job_status=2 and
> job_cmd="org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd";
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | job_cmd |
> account_id | job_result
>
> | job_result_code | job_status |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
> 9 |
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for volume
> 73"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
> 9 |
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for volume
> 80"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
> 9 |
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for volume
> 83"} | 530 | 2 |
> | org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd |
> 9 |
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to create snapshot due to an internal error creating snapshot for volume
> 85"} | 530 | 2 |
> +-------------------------------------------------------------------+------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+------------+
> 4 rows in set (0.01 sec)
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
