Re: [openstack-dev] container name in swift
Thanks Iury and John. Best, Jialin On Mon, Apr 2, 2018 at 1:17 PM, Iury Gregory wrote: > According to Swift doc[1] > > Length of container names 256 bytes Cannot contain the / character. > Length of object names 1024 bytes By default, there are no character > restrictions. > > [1] https://docs.openstack.org/swift/latest/api/object_api_ > v1_overview.html > > > > 2018-04-02 17:00 GMT-03:00 Jialin Liu : > >> Hi John, >> What is allowed in container name, but not in object name? >> I need a way to distinguish their name.. >> >> Best, >> Jialin >> >> On Mon, Apr 2, 2018 at 11:56 AM, John Dickinson wrote: >> >>> no >>> >>> On 2 Apr 2018, at 11:46, Jialin Liu wrote: >>> >>> > Hi, >>> > Can a container name in openstack swift contains / ? >>> > e.g., >>> > abc/111/mycontainer >>> > >>> > >>> > Best, >>> > Jialin >>> > >>> __ >>> > OpenStack Development Mailing List (not for usage questions) >>> > Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> __ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.op >>> enstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib >> e >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > > > -- > > > *Att[]'sIury Gregory Melo Ferreira * > *MSc in Computer Science at UFCG* > > *Part of the puppet-manager-core team in OpenStack* > *Social*: https://www.linkedin.com/in/iurygregory > *E-mail: iurygreg...@gmail.com * > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] container name in swift
Hi John, What is allowed in container name, but not in object name? I need a way to distinguish their name.. Best, Jialin On Mon, Apr 2, 2018 at 11:56 AM, John Dickinson wrote: > no > > On 2 Apr 2018, at 11:46, Jialin Liu wrote: > > > Hi, > > Can a container name in openstack swift contains / ? > > e.g., > > abc/111/mycontainer > > > > > > Best, > > Jialin > > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject: > unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] container name in swift
Hi, Can a container name in openstack swift contains / ? e.g., abc/111/mycontainer Best, Jialin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Can swift leverage os page buffer?
Hi, I'm new to openstack swift, I'm a HPC user, by several days of exploration of swift, I have some naive questions: 1. Can swift, e.g., PUT, leverage OS' page buffer?; Is there a Linux kernel module for swift? What are the existing optimizations for caching the write in order to gain better bandwidth on spin disks? 2. Does swift support asynchronous PUT/Get? Also please let me know if the dev list is good for me to ask this kind of questions. Best, Jialin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] leverage page cache in openstack swift
Hi, Is there any existing work that leveraging operating system's page cache for swift? like many other parallel file systems, lustre, the IO is cached in buffer and call is returned immediately to user space. Best, Jialin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to write python object in openstack swift
Hi John,
That indeed helped. But I'm stuck in using the python interface to make it
work:
_opts = {'object_uu_threads': 10, 'segment_size': 1048576}
with SwiftService(options=_opts) as swift:
Seems not working.
Also my another question is about the relationship with uu_threads and
segment, my understanding is that if we upload by segments, we should use
more than 1 threads so that parallelism is possible, right?
Best,
Jialin
On Thu, Oct 5, 2017 at 11:32 AM, John Dickinson wrote:
> I'm not familiar with that format, but in general, if you're dealing with
> large objects, you can get better performance by splitting the data in the
> client, uploading each separate segment concurrently, and then creating a
> large object manifest to tie the segments together (allowing future access
> by either segments or as a logical whole).
>
> https://docs.openstack.org/swift/latest/overview_large_
> objects.html#module-swift.common.middleware.slo
>
> --John
>
>
>
> On 5 Oct 2017, at 11:24, Jialin Liu wrote:
>
> Thank you John,
>
> I think I figured out the way. My case is a little bit rare as I'm dealing
> with the HDF5 file format, I can not think of any way to retrieve the
> entire in memory file as a single python object and then serialize it.
> What I did is to use the hdf5.get_file_image function to get a memory
> image of the file, and then use, io.ByteIO(image) to make it a file-like
> object.
>
> So far, it works well, but I believe it is not the best way in terms of
> performance.
>
> Best,
> Jialin
>
> On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson wrote:
>
>> If you've got an arbitrary object in Python, you'll need to serialize it
>> to a file-like object. You could keep it in memory and use a StringIO
>> type, or you could serialize it to disk and open() it like any other
>> file.
>>
>> Ultimately, Swift is storing arbitrary bytes and doesn't care what they
>> are. You, as the Swift client (i.e. API user), need to dump those bytes on
>> the network to send them to Swift. As long as you're transforming your
>> Python object[s] in some regular way that makes sense in your application,
>> it doesn't matter what bytes you send to Swift.
>>
>> --John
>>
>> On 5 Oct 2017, at 8:07, Jialin Liu wrote:
>>
>> Hi,
>> It seems to me that openstack swift only supports file upload/download, is
>> it possible to put a python object to swift store?
>> The doc says we could use file-like object, e.g., StringIO, but this is
>> very limited. I'd like to write a numpy array or other python object into
>> the swift store, can anybody tell me the solution?
>>
>> Best,
>> Jialin
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] how to write python object in openstack swift
Thank you John, I think I figured out the way. My case is a little bit rare as I'm dealing with the HDF5 file format, I can not think of any way to retrieve the entire in memory file as a single python object and then serialize it. What I did is to use the hdf5.get_file_image function to get a memory image of the file, and then use, io.ByteIO(image) to make it a file-like object. So far, it works well, but I believe it is not the best way in terms of performance. Best, Jialin On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson wrote: > If you've got an arbitrary object in Python, you'll need to serialize it > to a file-like object. You could keep it in memory and use a StringIO > type, or you could serialize it to disk and open() it like any other file. > > Ultimately, Swift is storing arbitrary bytes and doesn't care what they > are. You, as the Swift client (i.e. API user), need to dump those bytes on > the network to send them to Swift. As long as you're transforming your > Python object[s] in some regular way that makes sense in your application, > it doesn't matter what bytes you send to Swift. > > --John > > On 5 Oct 2017, at 8:07, Jialin Liu wrote: > > Hi, > It seems to me that openstack swift only supports file upload/download, is > it possible to put a python object to swift store? > The doc says we could use file-like object, e.g., StringIO, but this is > very limited. I'd like to write a numpy array or other python object into > the swift store, can anybody tell me the solution? > > Best, > Jialin > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] how to write python object in openstack swift
Hi, It seems to me that openstack swift only supports file upload/download, is it possible to put a python object to swift store? The doc says we could use file-like object, e.g., StringIO, but this is very limited. I'd like to write a numpy array or other python object into the swift store, can anybody tell me the solution? Best, Jialin __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
