[
https://issues.apache.org/jira/browse/CLOUDSTACK-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13726272#comment-13726272
]
Thomas O'Dowd commented on CLOUDSTACK-3229:
-------------------------------------------
Looking at the latest SMlog file, I found:
[8042] 2013-08-01 08:37:56.522934 #### VMOPS enter s3 #### ####
[8042] 2013-08-01 08:37:56.701046 #### VMOPS Attempting execution 1/3 of
perform_request ####
[8042] 2013-08-01 08:37:59.025139 #### VMOPS Sent PUT request to
s3.cloudian.com:18080/images/snapshots/ac434bba-1b7e-4e27-adad-04aeaf003935
with headers {'Content-Length': 50430464L, 'Content-MD5':
'cpA3yVjT37paxRTkZaxTsA==', 'Expect': '100-continue', 'Date': 'Thu, 01 Aug 2013
08:37:56 +0000', 'Content-Type': 'application/octet-stream', 'Authorization':
'AWS 00ba9a7f9a8142b070c3:HfmfhJrDPbYSxw41g2uYkG9VzT8='}. Received response
status 200: OK ####
[8042] 2013-08-01 08:37:59.025465 #### VMOPS exit s3 with result true ####
This shows that the PUT snapshot was successfully saved to S3. It shows the 200
response and it exists with result true. So this doesn't seem to be S3 related
issue perhaps?
The next section of SMlog continues like...
[8045] 2013-08-01 08:37:59.582776 #### VMOPS enter deleteSnapshotBackup
####
[8045] 2013-08-01 08:37:59.582856 Calling deleteSnapshotBackup with
{'localMountPoint':
'/var/run/cloud_mount/49ab0a32-3113-3ee2-bb24-3e65acb96ce9', 'backupUUID':
'ac434bba-1b7e-4e27-adad-04aeaf003935.vhd', 'secondaryStorageMountPath':
'192.168.56.10:/opt/storage/cache/', 'path': 'snapshots/2/3'}
[8052] 2013-08-01 08:37:59.823145 lock: acquired
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/sr
[8052] 2013-08-01 08:37:59.823239 sr_detach {'sr_uuid':
'af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3', 'subtask_of':
'DummyRef:|ad99ade6-031a-1388-e065-f3e2891e5dbb|SR.detach', 'args': [],
'host_ref': 'OpaqueRef:0d19f3fa-7f04-543a-d906-aaf478fc574c', 'session_ref':
'OpaqueRef:b4773703-89d0-00e7-d4c5-57a77731d52f', 'device_config': {'SRmaster':
'true', 'serverpath': '/opt/storage/cache/snapshots/2/3', 'server':
'192.168.56.10'}, 'command': 'sr_detach', 'sr_ref':
'OpaqueRef:cd6b5efa-4440-5d26-112b-d3f798ec91ba'}
[8052] 2013-08-01 08:37:59.823514 Aborting GC/coalesce
<8052> 2013-08-01 08:37:59.823564 === SR
af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3: abort ===
[8052] 2013-08-01 08:37:59.823665 lock: tried lock
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/running, acquired: True
(exists: True)
<8052> 2013-08-01 08:37:59.823708 abort: releasing the process lock
[8052] 2013-08-01 08:37:59.823762 lock: released
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/running
[8052] 2013-08-01 08:37:59.823821 ['umount',
'/var/run/sr-mount/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3']
[8052] 2013-08-01 08:37:59.833791 SUCCESS
[8052] 2013-08-01 08:37:59.833986 lock: released
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/sr
[8052] 2013-08-01 08:37:59.834277 lock: closed
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/sr
[8052] 2013-08-01 08:37:59.837456 lock: closed
/var/lock/sm/af1d24b9-76cc-4b5e-ca54-6b9d16f9bcd3/running
[8072] 2013-08-01 08:38:53.250810 #### VMOPS enter gethostvmstats ####
[8072] 2013-08-01 08:38:53.254586 #### VMOPS exit gethostvmstats ####
Now. I'm not sure what this is telling me. It seems to be saying that it
aborted GC/coalesce. I wonder if this is what then causes the operation to fail?
Anyway, perhaps this is because of all the already failed snapshots on my
system. I'm going to clean everything up tomorrow and start fresh to see if it
works. In the mean time if you have any ideas, please go for it :-)
> Object_Store_Refactor - Snapshot fails due to an internal error
> ---------------------------------------------------------------
>
> Key: CLOUDSTACK-3229
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3229
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Affects Versions: 4.2.0
> Environment: chrome on linux
> devcloud
> Cloudian or Amazon S3 Object store
> Reporter: Thomas O'Dowd
> Assignee: John Burwell
> Priority: Blocker
> Attachments: SMlog, SMlog.last_5000_lines.txt
>
>
> Assuming initial devcloud state...
> I added a cache for the S3 storage like this.
> on devcloud machine as root:
> # mkdir /opt/storage/cache
> # vi /etc/exports (and append this line)
> /opt/storage/cache *(rw,no_subtree_check,no_root_squash,fsid=9999)
> # exportfs -a
> On Mgmt server GUI:
> 1. navigate to infrastructure -> secondary storage
> 2. delete the NFS SS.
> 3. add S3 storage for Cloudian (I used 60000 as the timeouts - assuming
> millis). I used the /opt/storage/cache thing as the s3 cache.
> 4. nav to templates
> 5. register a new template (I uploaded tinyLinux again as "mytiny" (5.3
> 64bit)).
> 6. confirm with s3cmd that 2 objects are now on S3.
> --------- s3 objects -------
> template/tmpl/1/1/routing-1/acton-systemvm-02062012.vhd.bz2
> 2013-06-27T03:01:46.203Z None 140616708 "b533e7b65219439ee7fca0146ddd7ffa-27"
> template/tmpl/2/201/201-2-ae9e9409-4c8e-3ad8-a62f-abec7a35fe26/tinylinux.vhd
> 2013-06-27T03:04:06.730Z None 50430464 "4afac316e865adf74ca1a8039fae7399-10"
> --------- s3 objects -------
> 7. I restarted the management server at this point which actually resulted in
> another object on S3.
> --------- the new s3 object -------
> template/tmpl/1/5/tiny Linux/ttylinux_pv.vhd 2013-06-27T03:43:26.494Z None
> 50430464 "4afac316e865adf74ca1a8039fae7399-10"
> --------- the new s3 object -------
> 8. Go to instance and create a new choosing the "mytiny" template which we
> registered.
> 9. launch it after selecting all defaults.
> 10. wait until it starts.
> 11. nav to storage. I see ROOT-8. Click on this to open.
> 12. click the camera to take the snapshot.
> after a pause I get a popup
> "Failed to create snapshot due to an internal error creating snapshot
> for volume 8"
> Also on the mgmt terminal I get the following log entry (only 1):
> INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-8:job-16) VOLSS:
> createSnapshotCmd starts:1372321251009
> If I check the "view snapshots" button under storage, I can however see the
> snapshot. It says its on primary. I'm expecting it to go to secondary storage
> though. Nothing is in the S3 logs and no snapshots.
> If I try to delete that snapshot from here I get this error in the logs:
> ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-12:job-20) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to delete
> snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete
> snapshotshot 4 due to it is not in BackedUp Status
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:513)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:96)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> If I navigate to instance, my instance, and try to take a vm snapshot from
> here, I get a different pop-up which says:
> "There is other active volume snapshot tasks on the instance to which the
> volume is attached, please try again later"
> And I get an exception:
> ERROR [cloud.api.ApiServer] (352129314@qtp-2110413789-32:) unhandled
> exception executing api command: createVMSnapshot
> com.cloud.utils.exception.CloudRuntimeException: There is other active volume
> snapshot tasks on the instance to which the volume is attached, please try
> again later.
> at
> com.cloud.vm.snapshot.VMSnapshotManagerImpl.allocVMSnapshot(VMSnapshotManagerImpl.java:299)
> at
> org.apache.cloudstack.api.command.user.vmsnapshot.CreateVMSnapshotCmd.create(CreateVMSnapshotCmd.java:78)
> at
> com.cloud.api.ApiDispatcher.dispatchCreateCmd(ApiDispatcher.java:101)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:475)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> There are no requests going to the S3 storage for the snap-shotting that I
> can see and its the only secondary storage that I have setup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
