Hi, Gregory! You are absolutely right! Thanks!
The following sequence solves the problem: rados_striper_set_object_layout_stripe_unit(m_striper, stripe_unit); rados_striper_set_object_layout_stripe_count(m_striper, stripe_count); int stripe_size = stripe_unit * stripe_count; rados_striper_set_object_layout_object_size(m_striper, stripe_size); Now there is very little in the documentation about meaning of above parameters. The only document I found is CERN IT group presentation (page 8). Perhaps it is obvious. Also it seems that optimizing these parameters is meaningful in large scale Ceph installations only. https://indico.cern.ch/event/531810/contributions/2298934/ attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf Now if I have a 6TB disks, then in default installation there would be 6TB/4MB = 1.5M objects per OSD. Does it create any performance hit? Thank you, Alexander On Tue, Oct 10, 2017 at 12:38 AM, Gregory Farnum <[email protected]> wrote: > Well, just from a quick skim, libradosstriper.h has a function > rados_striper_set_object_layout_object_size(rados_striper_t striper, > unsigned int object_size) > and libradosstriper.hpp has one in RadosStriper > set_object_layout_object_size(unsigned int object_size); > > So I imagine you specify it with those the same way you've set the stripe > unit and counts. > > On Sat, Oct 7, 2017 at 12:38 PM Alexander Kushnirenko < > [email protected]> wrote: > >> Hi, Gregory! >> >> It turns out that this error is internal CEPH feature. I wrote standalone >> program to create 132M object in striper mode. It works only for 4M >> stripe. If you set stripe_unit = 2M it still creates 4M stripe_unit. >> Anything bigger than 4M causes crash here >> <https://github.com/ceph/ceph/blob/master/src/osdc/Striper.cc#L64%5C>: >> >> >> __u32 object_size = layout->object_size; >> __u32 su = layout->stripe_unit; >> __u32 stripe_count = layout->stripe_count; >> assert(object_size >= su); <------------ >> >> I'm curious where it gets layout->object_size for object that is just >> been created. >> >> As I understod striper mode was created by CERN guys. In there document >> <https://indico.cern.ch/event/542464/contributions/2202295/attachments/1289543/1919853/HEPUsageOfCeph.pdf> >> they recommend 8M stripe_unit. But it does not work in luminous. >> >> Created I/O context. >> Connected to pool backup with rados_striper_create >> Stripe unit OK 8388608 >> Stripe count OK 1 >> /build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void >> Striper::file_to_extents(CephContext*, const char*, const >> file_layout_t*, uint64_t, uint64_t, uint64_t, std::map<object_t, >> std::vector<ObjectExtent> >&, uint64_t)' thread 7f13bd5c1e00 time >> 2017-10-07 21:44:58.654778 >> /build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >= >> su) >> ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous >> (rc) >> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char >> const*)+0x102) [0x7f13b3f3b332] >> 2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t >> const*, unsigned long, unsigned long, unsigned long, std::map<object_t, >> std::vector<ObjectExtent, std::allocator<ObjectExtent> >, >> std::less<object_t>, std::allocator<std::pair<object_t const, >> std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned >> long)+0x1e1e) [0x7f13bce235ee] >> 3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t >> const*, unsigned long, unsigned long, unsigned long, >> std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned >> long)+0x51) [0x7f13bce23691] >> 4: (libradosstriper::RadosStriperImpl::internal_ >> aio_write(std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&, >> boost::intrusive_ptr<libradosstriper::MultiAioCompletionImpl>, >> ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout >> const&)+0x224) [0x7f13bcda4184] >> 5: (libradosstriper::RadosStriperImpl::write_in_ >> open_object(std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&, ceph_file_layout const&, >> std::__cxx11::basic_string<char, std::char_traits<char>, >> std::allocator<char> > const&, ceph::buffer::list const&, unsigned long, >> unsigned long)+0x13c) [0x7f13bcda476c] >> 6: >> (libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string<char, >> std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::list >> const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5] >> 7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b] >> 8: (()+0x10fb) [0x55dd87b410fb] >> 9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1] >> 10: (()+0xbca) [0x55dd87b40bca] >> >> >> On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum <[email protected]> >> wrote: >> >>> I haven't used the striper, but it appears to make you specify sizes, >>> stripe units, and stripe counts. I would expect you need to make sure that >>> the size is an integer multiple of the stripe unit. And it probably >>> defaults to a 4MB object if you don't specify one? >>> >>> On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup >>>> with libradosstriper1 support. >>>> >>>> Libradosstriber was suggested on this list to solve the problem, that >>>> current CEPH-12 discourages users from using object with very big size >>>> (>128MB). Bareos treat Rados Object as Volume and in CEPH-10 it created >>>> objects with very big size (10G and more). CEPH-10 allowed such behaviour, >>>> put recovery indeed take very long time. So stripping objects seems to be >>>> the right thing to do. >>>> >>>> Bareos supports libradosstriper and the code seems to work. But for >>>> some reason it run only with stripe_unit=4194304, which seems to be typical >>>> value for RadosGW for example. I tried several other values for >>>> stripe_unit, but the code exit with error. >>>> >>>> Is there a particular reason why only 4M size works? Can one use some >>>> CLI to test different stripe sizes? >>>> >>>> Basic flow of creating object in Bareos is the following: >>>> rados_ioctx_create(m_cluster, m_rados_poolname, &m_ctx); >>>> rados_striper_create(m_ctx, &m_striper); >>>> rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit); >>>> rados_striper_set_object_layout_stripe_count(m_striper, >>>> m_stripe_count); >>>> ..... >>>> status = rados_striper_write(m_striper, m_virtual_filename, buffer, >>>> count, offset); >>>> >>>> Alexander >>>> >>>> _______________________________________________ >>>> ceph-users mailing list >>>> [email protected] >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >>>> >>> >>
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
