Re: [openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Hi John,
That indeed helped. But I'm stuck in using the python interface to make it
work:

_opts = {'object_uu_threads': 10, 'segment_size': 1048576}

 with SwiftService(options=_opts) as swift:

Seems not working.

Also my another question is about the relationship with uu_threads and
segment, my understanding is that if we upload by segments, we should use
more than 1 threads so that parallelism is possible, right?


Best,

Jialin

On Thu, Oct 5, 2017 at 11:32 AM, John Dickinson  wrote:

> I'm not familiar with that format, but in general, if you're dealing with
> large objects, you can get better performance by splitting the data in the
> client, uploading each separate segment concurrently, and then creating a
> large object manifest to tie the segments together (allowing future access
> by either segments or as a logical whole).
>
> https://docs.openstack.org/swift/latest/overview_large_
> objects.html#module-swift.common.middleware.slo
>
> --John
>
>
>
> On 5 Oct 2017, at 11:24, Jialin Liu wrote:
>
> Thank you John,
>
> I think I figured out the way. My case is a little bit rare as I'm dealing
> with the HDF5 file format, I can not think of any way to retrieve the
> entire in memory file as a single python object and then serialize it.
> What I did is to use the hdf5.get_file_image function to get a memory
> image of the file, and then use, io.ByteIO(image) to make it a file-like
> object.
>
> So far, it works well, but I believe it is not the best way in terms of
> performance.
>
> Best,
> Jialin
>
> On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson  wrote:
>
>> If you've got an arbitrary object in Python, you'll need to serialize it
>> to a file-like object. You could keep it in memory and use a StringIO
>> type, or you could serialize it to disk and open() it like any other
>> file.
>>
>> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
>> are. You, as the Swift client (i.e. API user), need to dump those bytes on
>> the network to send them to Swift. As long as you're transforming your
>> Python object[s] in some regular way that makes sense in your application,
>> it doesn't matter what bytes you send to Swift.
>>
>> --John
>>
>> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>>
>> Hi,
>> It seems to me that openstack swift only supports file upload/download, is
>> it possible to put a python object to swift store?
>> The doc says we could use file-like object, e.g., StringIO, but this is
>> very limited. I'd like to write a numpy array or other python object into
>> the swift store, can anybody tell me the solution?
>>
>> Best,
>> Jialin
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> 
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] how to write python object in openstack swift

[John Dickinson]

I'm not familiar with that format, but in general, if you're dealing with large 
objects, you can get better performance by splitting the data in the client, 
uploading each separate segment concurrently, and then creating a large object 
manifest to tie the segments together (allowing future access by either 
segments or as a logical whole).

https://docs.openstack.org/swift/latest/overview_large_objects.html#module-swift.common.middleware.slo

--John




On 5 Oct 2017, at 11:24, Jialin Liu wrote:

> Thank you John,
>
> I think I figured out the way. My case is a little bit rare as I'm dealing
> with the HDF5 file format, I can not think of any way to retrieve the
> entire in memory file as a single python object and then serialize it.
> What I did is to use the hdf5.get_file_image function to get a memory image
> of the file, and then use, io.ByteIO(image) to make it a file-like object.
>
> So far, it works well, but I believe it is not the best way in terms of
> performance.
>
> Best,
> Jialin
>
> On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson  wrote:
>
>> If you've got an arbitrary object in Python, you'll need to serialize it
>> to a file-like object. You could keep it in memory and use a StringIO
>> type, or you could serialize it to disk and open() it like any other file.
>>
>> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
>> are. You, as the Swift client (i.e. API user), need to dump those bytes on
>> the network to send them to Swift. As long as you're transforming your
>> Python object[s] in some regular way that makes sense in your application,
>> it doesn't matter what bytes you send to Swift.
>>
>> --John
>>
>> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>>
>> Hi,
>> It seems to me that openstack swift only supports file upload/download, is
>> it possible to put a python object to swift store?
>> The doc says we could use file-like object, e.g., StringIO, but this is
>> very limited. I'd like to write a numpy array or other python object into
>> the swift store, can anybody tell me the solution?
>>
>> Best,
>> Jialin
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>


> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Thank you John,

I think I figured out the way. My case is a little bit rare as I'm dealing
with the HDF5 file format, I can not think of any way to retrieve the
entire in memory file as a single python object and then serialize it.
What I did is to use the hdf5.get_file_image function to get a memory image
of the file, and then use, io.ByteIO(image) to make it a file-like object.

So far, it works well, but I believe it is not the best way in terms of
performance.

Best,
Jialin

On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson  wrote:

> If you've got an arbitrary object in Python, you'll need to serialize it
> to a file-like object. You could keep it in memory and use a StringIO
> type, or you could serialize it to disk and open() it like any other file.
>
> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
> are. You, as the Swift client (i.e. API user), need to dump those bytes on
> the network to send them to Swift. As long as you're transforming your
> Python object[s] in some regular way that makes sense in your application,
> it doesn't matter what bytes you send to Swift.
>
> --John
>
> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>
> Hi,
> It seems to me that openstack swift only supports file upload/download, is
> it possible to put a python object to swift store?
> The doc says we could use file-like object, e.g., StringIO, but this is
> very limited. I'd like to write a numpy array or other python object into
> the swift store, can anybody tell me the solution?
>
> Best,
> Jialin
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] how to write python object in openstack swift

[John Dickinson]

If you've got an arbitrary object in Python, you'll need to serialize it to a 
file-like object. You could keep it in memory and use a `StringIO` type, or you 
could serialize it to disk and `open()` it like any other file.

Ultimately, Swift is storing arbitrary bytes and doesn't care what they are. 
You, as the Swift client (i.e. API user), need to dump those bytes on the 
network to send them to Swift. As long as you're transforming your Python 
object[s] in some regular way that makes sense in your application, it doesn't 
matter what bytes you send to Swift.

--John




On 5 Oct 2017, at 8:07, Jialin Liu wrote:

> Hi,
> It seems to me that openstack swift only supports file upload/download, is
> it possible to put a python object to swift store?
> The doc says we could use file-like object, e.g., StringIO, but this is
> very limited. I'd like to write a numpy array or other python object into
> the swift store, can anybody tell me the solution?
>
> Best,
> Jialin
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


signature.asc
Description: OpenPGP digital signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] how to write python object in openstack swift

[Jialin Liu]

Hi,
It seems to me that openstack swift only supports file upload/download, is
it possible to put a python object to swift store?
The doc says we could use file-like object, e.g., StringIO, but this is
very limited. I'd like to write a numpy array or other python object into
the swift store, can anybody tell me the solution?

Best,
Jialin
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev