I've filed an issue: https://issues.apache.org/jira/browse/LIBCLOUD-725
On 21.07.2015 13:40, Torf wrote: > Great, will do that. > > > On 21.07.2015 08:21, Tomaz Muraus wrote: >> It would be great if you can submit a pull request with a fix and test >> cases upstream :) >> >> https://libcloud.readthedocs.org/en/latest/development.html#contributing >> >> On Tue, Jul 21, 2015 at 2:11 PM, Torf <m...@torf.cc> wrote: >> >>> Hi everyone, >>> >>> Thanks for your input, Chris! So it seems to be indeed a problem with >>> LocalStorageDriver, not with my understanding :) >>> >>> What can I do to help getting this fixed in the official distribution? >>> I'd be happy to provide a patch, test cases, etc. >>> >>> >>> Regards, >>> Torf >>> >>> >>> On 21.07.2015 00:01, Chris Richards wrote: >>>> Torf, >>>> >>>> I stumbled across the same thing and modified my local repository copy. >>> I >>>> don't know if this was the correct way for libcloud proper, but it moved >>> me >>>> past the point where I was stuck. (There may be other places that need >>> it, >>>> but for the APIs I'm using they're sufficient.) You'll also need to do "$ >>>> pip3 install lockfile" to use the LocalStorage driver. >>>> >>>> diff -r a61cf0eec1a7 -r ad3deb3173ce libcloud/storage/drivers/local.py >>>> --- a/libcloud/storage/drivers/local.py Wed Jun 25 18:59:43 2014 -0500 >>>> +++ b/libcloud/storage/drivers/local.py Wed Jun 25 19:03:33 2014 -0500 >>>> @@ -406,7 +406,7 @@ >>>> >>>> path = self.get_object_cdn_url(obj) >>>> >>>> - with open(path) as obj_file: >>>> + with open(path, 'rb') as obj_file: >>>> for data in read_in_chunks(obj_file, chunk_size=chunk_size): >>>> yield data >>>> >>>> @@ -490,7 +490,7 @@ >>>> self._make_path(base_path) >>>> >>>> with LockLocalStorage(obj_path): >>>> - obj_file = open(obj_path, 'w') >>>> + obj_file = open(obj_path, 'wb') >>>> for data in iterator: >>>> obj_file.write(data) >>>> >>>> As for the upload, here's how I call it: >>>> >>>> obj = driver.upload_object_via_stream ( >>>> BytesIO (data), container, objname, >>>> extra={'content_type': 'application/octet-stream'}) >>>> >>>> Again, I don't know if this is libcloud proper or not; it works for me! >>>> (Ship it!) >>>> >>>> HTH, >>>> Chris >>>> >>>> On Mon, Jul 20, 2015 at 4:43 PM, Torf <m...@torf.cc> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm trying to store binary data in Python 3 to a ``LocalStorageDriver`` >>>>> instance. If I upload data via ``upload_object`` from a file on disk and >>>>> then use ``download_object_as_stream`` I get a stream of bytes, as >>>>> expected. However, uploading a stream of bytes manually using >>>>> ``upload_object_via_stream`` fails: >>>>> >>>>> >>>>>>>> container.upload_object_via_stream(io.BytesIO(b'foobar'), 'key') >>>>> Traceback (most recent call last): >>>>> File "<stdin>", line 1, in <module> >>>>> File >>>>> >>>>> >>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/base.py", >>>>> line 157, in upload_object_via_stream >>>>> iterator, self, object_name, extra=extra, **kwargs) >>>>> File >>>>> >>>>> >>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py", >>>>> line 497, in upload_object_via_stream >>>>> obj_file.close() >>>>> File >>>>> >>>>> >>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py", >>>>> line 69, in __exit__ >>>>> raise value >>>>> File >>>>> >>>>> >>> "/home/torf/projects/coba/venv/lib/python3.4/site-packages/libcloud/storage/drivers/local.py", >>>>> line 495, in upload_object_via_stream >>>>> obj_file.write(data) >>>>> TypeError: must be str, not bytes >>>>> >>>>> >>>>> The problem seems to be that ``LocalStorageDriver`` opens the target >>>>> file for writing in text mode ('w') instead of binary mode ('wb'). >>>>> >>>>> I couldn't find any information on whether keys and values are supposed >>>>> to be (Unicode) strings or bytes (or both), so I am not sure whether >>>>> this is a bug in ``LocalStorageDriver`` or a misunderstanding on my >>> side. >>>>> >>>>> This is on Python 3.4 and LibCloud 0.17.0. >>>>> >>>>> >>>>> Regards, >>>>> Torf >>>>> >>>> >>> >>> >> > > >
