Thanks Casey!
Adding the following to my swiftclient put_object call caused it to start
compressing the data:
headers={'x-object-storage-class': 'STANDARD'}
I appreciate the help!
Bryan
> On Nov 7, 2019, at 9:26 AM, Casey Bodley <[email protected]> wrote:
>
> On 11/7/19 10:35 AM, Bryan Stillwell wrote:
>> Thanks Casey!
>>
>> Hopefully this makes it in before 14.2.5. Is there any way to tell the
>> python boto or swiftclient modules to not send those headers?
>
> It is likely to make 14.2.5.
>
> You'd actually want to force these clients to send the headers as a
> workaround. If you're using boto2, you can inject the header in calls to
> set_contents_from_*(). For boto3, calls like put_object() take a
> StorageClass parameter.
>
>>
>> Bryan
>>
>>> On Nov 7, 2019, at 8:04 AM, Casey Bodley <[email protected]> wrote:
>>>
>>> Hi Bryan,
>>>
>>> This is a bug related to storage classes. Compression does take effect
>>> for requests that specify the storage class via the s3
>>> x-amz-storage-class or swift x-object-storage-class header. But when
>>> this header is absent, we default to the STANDARD storage class without
>>> consulting its compression type. The fix is pending backport to nautilus
>>> in https://tracker.ceph.com/issues/41981.
>>>
>>> On 11/6/19 5:54 PM, Bryan Stillwell wrote:
>>>> Today I tried enabling RGW compression on a Nautilus 14.2.4 test cluster
>>>> and found it wasn't doing any compression at all. I figure I must have
>>>> missed something in the docs, but I haven't been able to find out what
>>>> that is and could use some help.
>>>>
>>>> This is the command I used to enable zlib-based compression:
>>>>
>>>> # radosgw-admin zone placement modify --rgw-zone=default
>>>> --placement-id=default-placement --compression=zlib
>>>>
>>>>
>>>> I then restarted the radosgw process to activate the change (there's only
>>>> 1 RGW in this test cluster):
>>>>
>>>> # systemctl restart ceph-radosgw@radosgw.$(hostname -s)
>>>>
>>>>
>>>> I verified it was enabled correctly with:
>>>>
>>>> # radosgw-admin zone get | jq -r '.placement_pools'
>>>> [
>>>> {
>>>> "key": "default-placement",
>>>> "val": {
>>>> "index_pool": "default.rgw.buckets.index",
>>>> "storage_classes": {
>>>> "STANDARD": {
>>>> "data_pool": "default.rgw.buckets.data",
>>>> "compression_type": "zlib"
>>>> }
>>>> },
>>>> "data_extra_pool": "default.rgw.buckets.non-ec",
>>>> "index_type": 0
>>>> }
>>>> }
>>>> ]
>>>>
>>>>
>>>> Before starting the test I had nothing in the default.rgw.buckets.data
>>>> pool:
>>>>
>>>> # ceph df | grep default.rgw.buckets.data
>>>> default.rgw.buckets.data 16 0 B 0 0 B
>>>> 0 230 TiB
>>>>
>>>>
>>>> I then tried uploading a 1GiB file consisting of all 0s from /dev/zero
>>>> with both S3 (boto) and Swift (swiftclient) and each time they used 1GiB
>>>> of data on the cluster:
>>>>
>>>> # ceph df -f json | jq -r '.' | grep -A9 default.rgw.buckets.data
>>>> "name": "default.rgw.buckets.data",
>>>> "id": 16,
>>>> "stats": {
>>>> "stored": 1073741824,
>>>> "objects": 256,
>>>> "kb_used": 1048576,
>>>> "bytes_used": 1073741824,
>>>> "percent_used": 1.4138463484414387e-06,
>>>> "max_avail": 253148744646656
>>>> }
>>>>
>>>>
>>>> The same thing was reported by bucket stats:
>>>>
>>>> # radosgw-admin bucket stats --bucket=bs-test | jq -r '.usage'
>>>> {
>>>> "rgw.main": {
>>>> "size": 1073741824,
>>>> "size_actual": 1073741824,
>>>> "size_utilized": 1073741824,
>>>> "size_kb": 1048576,
>>>> "size_kb_actual": 1048576,
>>>> "size_kb_utilized": 1048576,
>>>> "num_objects": 1
>>>> }
>>>> }
>>>>
>>>>
>>>> What am I missing?
>>>>
>>>> Thanks,
>>>> Bryan
>>>> _______________________________________________
>>>> ceph-users mailing list -- [email protected]
>>>> To unsubscribe send an email to [email protected]
>>> _______________________________________________
>>> ceph-users mailing list -- [email protected]
>>> To unsubscribe send an email to [email protected]
>
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
