Small update: I was able to get past this error by editing
/etc/xapi.d/plugins/s3xen on the hypervisor, and adding this line to the s3
function:
> filename = "%s.vhd" % filename.replace('/dev/VG_XenStorage-',
> '/var/run/sr-mount/').replace('VHD-', ‘')
It just changes the filename to what it would be set to if isISCSI returned
false in my previous snippet. The IsISCSI function returns true for
SRType.LVM which is what I’d be using with local storage - is that intended?
Thanks,
> On Nov 14, 2014, at 10:16 AM, Justyn Shull <[email protected]> wrote:
>
> Thanks for the tip, Sanjeev. The snapshot is around ~2.5gb.
>
> I found the “s3.singleupload.max.size” parameter and changed it to 0 so that
> multi-part upload is always used, and restarted cs management. So far, I
> am still getting the same error I pasted before:
>
>> Exception: Attempt to put
>> /dev/VG_XenStorage-c2c8fe9f-59af-f4d7-52e1-5853bbf47be8/VHD-fa86335b-806f-4355-944f-856eb41d1ac9
>> that does not exist.
>
> I tried digging through the code a little, and I think I found where the
> filename gets set:
> plugins/hypervisors/xenserver/src/com/cloud/hypervisor/xenserver/resource/XenServerStorageProcessor.java
> 1067: protected String backupSnapshotToS3(final Connection connection,
> final S3TO s3, final String srUuid, final String folder, final String
> snapshotUuid,
> 1068: final Boolean iSCSIFlag, final
> int wait) {
> 1070: final String filename = iSCSIFlag ? "VHD-" + snapshotUuid :
> snapshotUuid + ".vhd";
> 1071: final String dir = (iSCSIFlag ? "/dev/VG_XenStorage-" :
> "/var/run/sr-mount/") + srUuid;
> 1080: parameters.addAll(Arrays.asList("operation", "put",
> "filename", dir + "/" + filename, "iSCSIFlag", iSCSIFlag.toString(),
> "bucket", s3.getBucketName(), "key”,
>
> I haven’t figured out what sets that iSCSIFlag, but it seems like in my case
> it shouldn’t be set if it should be uploading the .vhd file, right?
>
>
>
> On Nov 13, 2014, at 11:46 PM, Sanjeev Neelarapu
> <[email protected]<mailto:[email protected]>> wrote:
>
> Hi,
>
> You have followed correct steps in updating the secondary storage from NFS to
> S3. CS would upload the .vhd created on NFS to S3.
>
> What is the size of the snapshot? It it is >5GB can you enable multipart
> upload and try again?
> You can find the global setting parameter to enable multipart upload.
>
> Thanks,
> Sanjeev
>
> -----Original Message-----
> From: Justyn Shull [mailto:[email protected]]
> Sent: Thursday, November 13, 2014 11:35 PM
> To: [email protected]<mailto:[email protected]>
> Subject: Snapshots not working when using S3 (ceph) ACS 4.4.0
>
> I’m trying to enable object storage (ceph using the S3 radosgw) as a
> secondary store for an existing cloudstack installation, and running into
> some issues. There was already an existing NFS store being used as the
> secondary storage, so I used the updateCloudToUseObjectStore api call (using
> cloud monkey) with these params(keys changed):
>
> update cloudtouseobjectstore name=cephs3
> zoneId=749cde04-531a-4e1f-bfa2-ad7f7854b1f8 url=xxx details[0].key=accesskey
> details[0].value=xxx details[1].key=secretkey details[1].value=xxx
> details[2].key=bucket details[2].value=CLOUDSTACK details[3].key=endpoint
> details[3].value=10.16.33.172 provider=S3
>
> 1) There were no errors from that call, and as far as I can tell it changed
> the old NFS store to ‘ImageCache’ and created the new s3 store in the
> database. I’m not sure what else to check for to see if that was successful,
> or if there are any long-running processes that it triggers..
>
> However, I tried creating a volume snapshot to test and that is where I’m
> running into issues now. Cloudstack appears to create the snapshot on NFS
> (I think this part is normal), but then when it goes to upload the snapshot
> to S3 - it’s using the wrong local path. This is the log from the
> hypervisor (XenServer 6.1.0 w/ local storage):
>
> ###
> 2014-11-13 10:23:17 DEBUG [root] #### VMOPS enter s3 #### ####
> 2014-11-13 10:23:17 DEBUG [root] #### VMOPS Enetered parseArguments with
> args: {'maxErrorRetry': 'null', 'key':
> 'snapshots/4/54/VHD-2862dc6b-c057-4bb4-9d70-b263fc1086c9',
> 'maxSingleUploadSizeInBytes': '5368709120', 'accessKey': ‘xxx', 'bucket':
> 'CLOUDSTACK', 'filename':
> '/dev/VG_XenStorage-85dfb820-d810-716b-89cb-0e1303da2c2b/VHD-2862dc6b-c057-4bb4-9d70-b263fc1086c9',
> 'secretKey': ‘xxx', 'socketTimeout': 'null', 'endPoint': '10.16.33.172',
> 'https': 'false', 'connectionTimeout': 'null', 'operation': 'put',
> 'iSCSIFlag': 'true'} ####
> 2014-11-13 10:23:17 DEBUG [root] #### VMOPS Operation put on file
> /dev/VG_XenStorage-85dfb820-d810-716b-89cb-0e1303da2c2b/VHD-2862dc6b-c057-4bb4-9d70-b263fc1086c9
> from/in bucket CLOUDSTACK key
> snapshots/4/54/VHD-2862dc6b-c057-4bb4-9d70-b263fc1086c9 ####
> 2014-11-13 10:23:17 DEBUG [root] #### VMOPS Traceback (most recent call
> last):
> File "/etc/xapi.d/plugins/s3xen", line 414, in s3
> client.put(bucket, key, filename, maxSingleUploadBytes)
> File "/etc/xapi.d/plugins/s3xen", line 325, in put
> raise Exception(
> Exception: Attempt to put
> /dev/VG_XenStorage-85dfb820-d810-716b-89cb-0e1303da2c2b/VHD-2862dc6b-c057-4bb4-9d70-b263fc1086c9
> that does not exist.
> ####
> 2014-11-13 10:23:17 DEBUG [root] #### VMOPS exit s3 with result false ####
> ###
>
> 2) I’m assuming it should be trying to upload the .vhd it created on the nfs
> store, but I am not sure whether it’s a config issue or some sort of bug
> somewhere that is causing this.
>
> 3) Am I correct in assuming the general snapshot flow should be like this?
> (all on hypervisor) Mount NFS -> create .vhd snapshot from localstorage/lvm
> -> upload .vhd from NFS to S3/objstor -> delete .vhd from nfs
>
> Any help would be appreciated,
>
> Thanks,
>
