Hi everyone,
Thanks for your input, Chris! So it seems to be indeed a problem with
LocalStorageDriver, not with my understanding :)
What can I do to help getting this fixed in the official distribution?
I'd be happy to provide a patch, test cases, etc.
Regards,
Torf
On 21.07.2015 00:01, Chris Richards wrote:
> Torf,
>
> I stumbled across the same thing and modified my local repository copy. I
> don't know if this was the correct way for libcloud proper, but it moved me
> past the point where I was stuck. (There may be other places that need it,
> but for the APIs I'm using they're sufficient.) You'll also need to do "$
> pip3 install lockfile" to use the LocalStorage driver.
>
> diff -r a61cf0eec1a7 -r ad3deb3173ce libcloud/storage/drivers/local.py
> --- a/libcloud/storage/drivers/local.py Wed Jun 25 18:59:43 2014 -0500
> +++ b/libcloud/storage/drivers/local.py Wed Jun 25 19:03:33 2014 -0500
> @@ -406,7 +406,7 @@
>
> path = self.get_object_cdn_url(obj)
>
> - with open(path) as obj_file:
> + with open(path, 'rb') as obj_file:
> for data in read_in_chunks(obj_file, chunk_size=chunk_size):
> yield data
>
> @@ -490,7 +490,7 @@
> self._make_path(base_path)
>
> with LockLocalStorage(obj_path):
> - obj_file = open(obj_path, 'w')
> + obj_file = open(obj_path, 'wb')
> for data in iterator:
> obj_file.write(data)
>
> As for the upload, here's how I call it:
>
> obj = driver.upload_object_via_stream (
> BytesIO (data), container, objname,
> extra={'content_type': 'application/octet-stream'})
>
> Again, I don't know if this is libcloud proper or not; it works for me!
> (Ship it!)
>
> HTH,
> Chris
>
> On Mon, Jul 20, 2015 at 4:43 PM, Torf <m...@torf.cc> wrote:
>
>> Hi,
>>
>> I'm trying to store binary data in Python 3 to a ``LocalStorageDriver``
>> instance. If I upload data via ``upload_object`` from a file on disk and
>> then use ``download_object_as_stream`` I get a stream of bytes, as
>> expected. However, uploading a stream of bytes manually using
>> ``upload_object_via_stream`` fails:
>>
>>
>>>>> container.upload_object_via_stream(io.BytesIO(b'foobar'), 'key')
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File
>>
>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/base.py",
>> line 157, in upload_object_via_stream
>> iterator, self, object_name, extra=extra, **kwargs)
>> File
>>
>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>> line 497, in upload_object_via_stream
>> obj_file.close()
>> File
>>
>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>> line 69, in __exit__
>> raise value
>> File
>>
>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py",
>> line 495, in upload_object_via_stream
>> obj_file.write(data)
>> TypeError: must be str, not bytes
>>
>>
>> The problem seems to be that ``LocalStorageDriver`` opens the target
>> file for writing in text mode ('w') instead of binary mode ('wb').
>>
>> I couldn't find any information on whether keys and values are supposed
>> to be (Unicode) strings or bytes (or both), so I am not sure whether
>> this is a bug in ``LocalStorageDriver`` or a misunderstanding on my side.
>>
>> This is on Python 3.4 and LibCloud 0.17.0.
>>
>>
>> Regards,
>> Torf
>>
>
