Thanks you guys sooo much for the help!!! I got this figured out. I was using swift's --use-slo flag and apparently SoftLayer doesn't support that. It be really great to add to the docs that SLO support is optional (and turned off by default). I was definitely confused by that.
Thanks! -Ben On Thu, Mar 13, 2014 at 11:42 AM, Ben McCann <[email protected]> wrote: > I managed to find out from SoftLayer that they're running 1.7.x > > > On Thu, Mar 13, 2014 at 11:39 AM, Ben McCann <[email protected]> wrote: > >> The segments do appear to be there. >> >> I don't see any X-Object-Manifest header though or anything like that >> >> URL: >> https://dal05.objectstorage.softlayer.net/v1/AUTH_c4fd4142-xxxx-xxxx-xxxx-xxxxxxxxxxxx/mongodb-backups/mongodb_backup_2014.03.09.09.00.03.tar.gz >> Auth Token: AUTH_tkced5xxxxxxxxxxxxxxxxxxxxxxxxxxxx >> Account: AUTH_c4fd4142-xxxx-xxxx-xxxx-xxxxxxxxxxxx >> Container: mongodb-backups >> Object: mongodb_backup_2014.03.09.09.00.03.tar.gz >> Content Type: application/x-tar >> Content Length: 3895 >> Last Modified: Sun, 09 Mar 2014 09:58:09 GMT >> ETag: 0a8ec37fb235ac99e16a726e1b17375b >> Meta Mtime: 1394358254.226578 >> Accept-Ranges: bytes >> X-Timestamp: 1394359089.40223 >> X-Trans-Id: tx912c6e12d63a409aa5362b6075c5f930 >> >> Is this object likely to be stored wrong? Or maybe some misconfiguration >> on their end like a load banalancer stripping out headers or something? >> >> >> On Thu, Mar 13, 2014 at 11:30 AM, Clay Gerrard <[email protected]>wrote: >> >>> No, you can't really turn off DLO and it's been around for awhile (let's >>> assume for now they support it). >>> >>> Can you verify the segments are showing up in the "container_segments" >>> container? Can you verify the "X-Object-Manifest" header on the >>> "container/file.tar.gz" object? >>> >>> "swift stat -v container file.tar.gz" should show it as just >>> "Manifest", something like "container >>> _segments/file.tar.gz/1394610073.993163/104857600/1000000/" >>> >>> Good Luck, >>> >>> -Clay >>> >>> >>> On Thu, Mar 13, 2014 at 11:00 AM, Ben McCann <[email protected]> wrote: >>> >>>> Oh geeze, sorry, looking at those docs, it seems maybe I was mistaken >>>> and am using Dynamic Large Objects because I uploaded the file using swift >>>> -S. Is DLO also optional? Was it added at a certain version? >>>> >>>> Thanks so much for the help! >>>> -Ben >>>> >>>> >>>> On Thu, Mar 13, 2014 at 10:50 AM, Clay Gerrard >>>> <[email protected]>wrote: >>>> >>>>> You should check with your deployer to see what version of Swift they >>>>> are running. >>>>> >>>>> the /info (capabilities) feature was added in 1.11 and I think Havana >>>>> shipped with 1.10 >>>>> >>>>> But I think SLO support has been around since 1.5 which should be in >>>>> Havana, maybe even Grizzly. [1] >>>>> >>>>> Either way SLO support has always been optional (and off by default) - >>>>> so your deployer may only support DLO. [2] >>>>> >>>>> 1. >>>>> https://wiki.openstack.org/wiki/ReleaseNotes/Grizzly#OpenStack_Object_Storage_.28Swift.29 >>>>> 2. >>>>> http://docs.openstack.org/developer/swift/overview_large_objects.html >>>>> >>>>> >>>>> On Thu, Mar 13, 2014 at 10:31 AM, Ben McCann <[email protected]>wrote: >>>>> >>>>>> Oh, you're right, I did miss the -v option. Thanks! >>>>>> >>>>>> The curl command isn't working for me, however. >>>>>> >>>>>> StorageURL: >>>>>> https://dal05.objectstorage.softlayer.net/v1/AUTH_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx >>>>>> >>>>>> $ curl https://dal05.objectstorage.softlayer.net/info >>>>>> Bad URL >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Mar 13, 2014 at 2:59 AM, Kuo Hugo <[email protected]> wrote: >>>>>> >>>>>>> Hi Ben, >>>>>>> >>>>>>> 1. What's the Swift CLI version are you using for now ? >>>>>>> 2. The $>swift stat -v suppose to show you the storage_url. -v >>>>>>> means verbose. >>>>>>> 3. There's another choice via curl >>>>>>> >>>>>>> ~IRON_MAN$>curl -i https://$FQDN_of_AUTH_SERVER/auth/v1.0 -H >>>>>>> "X-auth-user:johndoe" -H "x-auth-key:password_of_john" >>>>>>> HTTP/1.1 200 OK >>>>>>> X-Storage-Url: https://swift.example.com/v1/AUTH_johndoe >>>>>>> X-Auth-Token: AUTH_tk0f4473fa1234567890 >>>>>>> Content-Type: text/html; charset=UTF-8 >>>>>>> Set-Cookie: X-Auth-Token=AUTH_tk0f4473fa1234567890; Path=/ >>>>>>> X-Storage-Token: AUTH_tk0f4473fa1234567890 >>>>>>> Content-Length: 0 >>>>>>> X-Trans-Id: txcfefd5396a0f4f4694707-0053217efd >>>>>>> >>>>>>> >>>>>>> And then you can send a GET to https://swift.example.com/info >>>>>>> >>>>>>> $> curl https://cloud.swiftstack.com/info >>>>>>> >>>>>>> >>>>>>> One more information here, the /info was available since Swift >>>>>>> 1.11.0 >>>>>>> https://github.com/openstack/swift/blob/master/CHANGELOG#L120-L125 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2014-03-13 14:41 GMT+08:00 Ben McCann <[email protected]>: >>>>>>> >>>>>>> Thanks Samuel. That's super helpful. I didn't realize that SLO could >>>>>>>> be turned on/off on the server. I thought it was merely a client-side >>>>>>>> convention. >>>>>>>> >>>>>>>> swift stat doesn't return a StorageUrl for me. It just returns: >>>>>>>> Account: >>>>>>>> Containers: >>>>>>>> Objects: >>>>>>>> Bytes: >>>>>>>> Meta Cdn-Id: >>>>>>>> Meta Nas-Id: >>>>>>>> X-Timestamp: >>>>>>>> X-Trans-Id: >>>>>>>> Accept-Ranges: >>>>>>>> >>>>>>>> I'll see if I can find out from someone at SoftLayer whether they >>>>>>>> have an old version or have large objects turned off unless you can >>>>>>>> think >>>>>>>> of any other ways I might be able to find out by querying the server. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ben >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 12, 2014 at 11:09 PM, Samuel Merritt < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> On 3/12/14 9:14 PM, Ben McCann wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm attempting to download a large file which I have uploaded to >>>>>>>>>> SoftLayer Object Storage using the swift tool. It's downloading >>>>>>>>>> only the >>>>>>>>>> manifest and is not downloading the segments. Any tips on >>>>>>>>>> figuring out >>>>>>>>>> what is going wrong? >>>>>>>>>> >>>>>>>>> >>>>>>>>> It's entirely possible that the cluster in question lacks static >>>>>>>>> large object support. Maybe they're running an old version of Swift >>>>>>>>> (pre-1.8.0/Grizzly), or maybe they haven't enabled the static large >>>>>>>>> object >>>>>>>>> feature. >>>>>>>>> >>>>>>>>> What you'll need to do is probe the cluster to see if static large >>>>>>>>> objects are enabled. >>>>>>>>> >>>>>>>>> The easy way is to politely ask the cluster with a GET request to >>>>>>>>> /info. Run `swift stat -v`, then take the "StorageURL" field and >>>>>>>>> change the >>>>>>>>> path to be just "/info". Strip off any path that's already there. >>>>>>>>> >>>>>>>>> For example, if the output looks like >>>>>>>>> >>>>>>>>> StorageURL: http://192.168.22.2:8080/v1/AUTH_test >>>>>>>>> Auth Token: AUTH_tk89454677b67348e09eefc1d6fe4dfa68 >>>>>>>>> Account: AUTH_test >>>>>>>>> Containers: 1 >>>>>>>>> [...] >>>>>>>>> >>>>>>>>> then you'd simply run >>>>>>>>> >>>>>>>>> curl http://192.168.22.2:8080/info >>>>>>>>> >>>>>>>>> The response will be a JSON hash, so maybe run it through a JSON >>>>>>>>> pretty-printer, e.g. >>>>>>>>> >>>>>>>>> curl http://192.168.22.2:8080/info | python -mjson.tool >>>>>>>>> >>>>>>>>> On my development machine, the response looks something like this >>>>>>>>> (some items removed for brevity): >>>>>>>>> >>>>>>>>> { >>>>>>>>> "container_quotas": {}, >>>>>>>>> "formpost": {}, >>>>>>>>> "slo": { <---- LOOK FOR THIS >>>>>>>>> "max_manifest_segments": 1000, >>>>>>>>> "max_manifest_size": 2097152, >>>>>>>>> "min_segment_size": 71 >>>>>>>>> }, >>>>>>>>> "swift": { >>>>>>>>> "account_listing_limit": 10000, >>>>>>>>> "container_listing_limit": 10000, >>>>>>>>> "max_account_name_length": 256, >>>>>>>>> "max_container_name_length": 256, >>>>>>>>> "max_file_size": 5368709122, >>>>>>>>> "max_meta_count": 90, >>>>>>>>> "max_meta_name_length": 128, >>>>>>>>> "max_meta_value_length": 256, >>>>>>>>> "max_object_name_length": 1024, >>>>>>>>> "version": "1.12.0.106.ga08e813" >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> If you don't see SLO in there, then you're out of luck. Bug >>>>>>>>> Softlayer and see if they'll turn it on. >>>>>>>>> >>>>>>>>> Now, it's also possible that you'll get a 412 response with the >>>>>>>>> body "Bad URL" when you run the curl command above. That will tell >>>>>>>>> you that >>>>>>>>> the cluster is running 1.11.0 or earlier, but it won't tell you >>>>>>>>> anything >>>>>>>>> about SLO support. In that case, try uploading a normal file (cat >>>>>>>>> picture >>>>>>>>> or something) with the header "X-Static-Large-Object: yes". If that >>>>>>>>> succeeds, then SLO is disabled; if SLO is enabled, then the PUT will >>>>>>>>> fail. >>>>>>>>> >>>>>>>>> A more verbose description is available here: >>>>>>>>> https://ask.openstack.org/en/question/7005/static-large-objects/ >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/ >>>>>>>>> openstack >>>>>>>>> Post to : [email protected] >>>>>>>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/ >>>>>>>>> openstack >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> about.me/benmccann >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Mailing list: >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>>> Post to : [email protected] >>>>>>>> Unsubscribe : >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Mailing list: >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> Post to : [email protected] >>>>>>> Unsubscribe : >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> about.me/benmccann >>>>>> >>>>>> _______________________________________________ >>>>>> Mailing list: >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>> Post to : [email protected] >>>>>> Unsubscribe : >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> about.me/benmccann >>>> >>> >>> >> >> >> -- >> about.me/benmccann >> > > > > -- > about.me/benmccann > -- about.me/benmccann
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
