Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Maged Mokhtar]

>> Would it be 4 objects of 24M and 4 objects of 250KB? Or will the last
4 objects be artificially padded (with 0's) to meet the stripe_unit? 

It will be 4 object of 24M + 1M stored on the 5th object 

If you write 104M :  4 object of 24M + 8M stored on the 5th object 

If you write 105M :  4 object of 24M + 8M stored on the 5th object + 1M
on 6th object 

Maged 

On 2017-10-17 01:59, Christian Wuerdig wrote:

> Maybe an additional example where the numbers don't line up all so
> nicely would be good as well. For example it's not immediately obvious
> to me what would happen with the stripe settings given by your example
> but you write 97M of data
> Would it be 4 objects of 24M and 4 objects of 250KB? Or will the last
> 4 objects be artificially padded (with 0's) to meet the stripe_unit?
> 
> On Tue, Oct 17, 2017 at 12:35 PM, Alexander Kushnirenko
>  wrote: Hi, Gregory, Ian!
> 
> There is very little information on striper mode in Ceph documentation.
> Could this explanation help?
> 
> The logic of striper mode is very much the same as in RAID-0.  There are 3
> parameters that drives it:
> 
> stripe_unit - the stripe size  (default=4M)
> stripe_count - how many objects to write in parallel (default=1)
> object_size  - when to stop increasing object size and create new objects.
> (default =4M)
> 
> For example if you write 132M of data (132 consecutive pieces of data 1M
> each) in striped mode with the following parameters:
> stripe_unit = 8M
> stripe_count = 4
> object_size = 24M
> Then 8 objects will be created - 4 objects with 24M size and 4 objects with
> 8M size.
> 
> Obj1=24MObj2=24MObj3=24MObj4=24M
> 00 .. 07 08 .. 0f 10 .. 17 18 .. 1f  <-- consecutive
> 1M pieces of data
> 20 .. 27 21 .. 2f 30 .. 37 38 .. 3f
> 40 .. 47 48 .. 4f 50 .. 57 58 .. 5f
> 
> Obj5= 8MObj6= 8MObj7= 8MObj8= 8M
> 60 .. 6768 .. 6f70 .. 7778 .. 7f
> 
> Alexander.
> 
> On Wed, Oct 11, 2017 at 3:19 PM, Alexander Kushnirenko
>  wrote: 
> Oh!  I put a wrong link, sorry  The picture which explains stripe_unit and
> stripe count is here:
> 
> https://indico.cern.ch/event/330212/contributions/1718786/attachments/642384/883834/CephPluginForXroot.pdf
> 
> I tried to attach it in the mail, but it was blocked.
> 
> On Wed, Oct 11, 2017 at 3:16 PM, Alexander Kushnirenko
>  wrote: 
> Hi, Ian!
> 
> Thank you for your reference!
> 
> Could you comment on the following rule:
> object_size = stripe_unit * stripe_count
> Or it is not necessarily so?
> 
> I refer to page 8 in this report:
> 
> https://indico.cern.ch/event/531810/contributions/2298934/attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf
> 
> Alexander.
> 
> On Wed, Oct 11, 2017 at 1:11 PM,  wrote: 
> Hi Gregory
> 
> You're right, when setting the object layout in libradosstriper, one
> should set all three parameters (the number of stripes, the size of the
> stripe unit, and the size of the striped object). The Ceph plugin for
> GridFTP has an example of this at
> https://github.com/stfc/gridFTPCephPlugin/blob/master/ceph_posix.cpp#L371
> 
> At RAL, we use the following values:
> 
> $STRIPER_NUM_STRIPES 1
> 
> $STRIPER_STRIPE_UNIT 8388608
> 
> $STRIPER_OBJECT_SIZE 67108864
> 
> Regards,
> 
> Ian Johnson MBCS
> 
> Data Services Group
> 
> Scientific Computing Department
> 
> Rutherford Appleton Laboratory
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

 ___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Christian Wuerdig]

Maybe an additional example where the numbers don't line up all so
nicely would be good as well. For example it's not immediately obvious
to me what would happen with the stripe settings given by your example
but you write 97M of data
Would it be 4 objects of 24M and 4 objects of 250KB? Or will the last
4 objects be artificially padded (with 0's) to meet the stripe_unit?



On Tue, Oct 17, 2017 at 12:35 PM, Alexander Kushnirenko
 wrote:
> Hi, Gregory, Ian!
>
> There is very little information on striper mode in Ceph documentation.
> Could this explanation help?
>
> The logic of striper mode is very much the same as in RAID-0.  There are 3
> parameters that drives it:
>
> stripe_unit - the stripe size  (default=4M)
> stripe_count - how many objects to write in parallel (default=1)
> object_size  - when to stop increasing object size and create new objects.
> (default =4M)
>
> For example if you write 132M of data (132 consecutive pieces of data 1M
> each) in striped mode with the following parameters:
> stripe_unit = 8M
> stripe_count = 4
> object_size = 24M
> Then 8 objects will be created - 4 objects with 24M size and 4 objects with
> 8M size.
>
> Obj1=24MObj2=24MObj3=24MObj4=24M
> 00 .. 07 08 .. 0f 10 .. 17 18 .. 1f  <-- consecutive
> 1M pieces of data
> 20 .. 27 21 .. 2f 30 .. 37 38 .. 3f
> 40 .. 47 48 .. 4f 50 .. 57 58 .. 5f
>
> Obj5= 8MObj6= 8MObj7= 8MObj8= 8M
> 60 .. 6768 .. 6f70 .. 7778 .. 7f
>
> Alexander.
>
>
>
>
> On Wed, Oct 11, 2017 at 3:19 PM, Alexander Kushnirenko
>  wrote:
>>
>> Oh!  I put a wrong link, sorry  The picture which explains stripe_unit and
>> stripe count is here:
>>
>>
>> https://indico.cern.ch/event/330212/contributions/1718786/attachments/642384/883834/CephPluginForXroot.pdf
>>
>> I tried to attach it in the mail, but it was blocked.
>>
>>
>> On Wed, Oct 11, 2017 at 3:16 PM, Alexander Kushnirenko
>>  wrote:
>>>
>>> Hi, Ian!
>>>
>>> Thank you for your reference!
>>>
>>> Could you comment on the following rule:
>>> object_size = stripe_unit * stripe_count
>>> Or it is not necessarily so?
>>>
>>> I refer to page 8 in this report:
>>>
>>>
>>> https://indico.cern.ch/event/531810/contributions/2298934/attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf
>>>
>>>
>>> Alexander.
>>>
>>> On Wed, Oct 11, 2017 at 1:11 PM,  wrote:

 Hi Gregory

 You’re right, when setting the object layout in libradosstriper, one
 should set all three parameters (the number of stripes, the size of the
 stripe unit, and the size of the striped object). The Ceph plugin for
 GridFTP has an example of this at
 https://github.com/stfc/gridFTPCephPlugin/blob/master/ceph_posix.cpp#L371



 At RAL, we use the following values:



 $STRIPER_NUM_STRIPES 1

 $STRIPER_STRIPE_UNIT 8388608

 $STRIPER_OBJECT_SIZE 67108864



 Regards,



 Ian Johnson MBCS

 Data Services Group

 Scientific Computing Department

 Rutherford Appleton Laboratory




 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>>
>>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Gregory Farnum]

That looks right to me... PRs for the Ceph docs are welcome! :)

On Mon, Oct 16, 2017 at 4:35 PM Alexander Kushnirenko 
wrote:

> Hi, Gregory, Ian!
>
> There is very little information on striper mode in Ceph documentation.
> Could this explanation help?
>
> The logic of striper mode is very much the same as in RAID-0.  There are 3
> parameters that drives it:
>
> stripe_unit - the stripe size  (default=4M)
> stripe_count - how many objects to write in parallel (default=1)
> object_size  - when to stop increasing object size and create new objects.
>  (default =4M)
>
> For example if you write 132M of data (132 consecutive pieces of data 1M
> each) in striped mode with the following parameters:
> stripe_unit = 8M
> stripe_count = 4
> object_size = 24M
> Then 8 objects will be created - 4 objects with 24M size and 4 objects
> with 8M size.
>
> Obj1=24MObj2=24MObj3=24MObj4=24M
> 00 .. 07 08 .. 0f 10 .. 17 18 .. 1f  <--
> consecutive 1M pieces of data
> 20 .. 27 21 .. 2f 30 .. 37 38 .. 3f
> 40 .. 47 48 .. 4f 50 .. 57 58 .. 5f
>
> Obj5= 8MObj6= 8MObj7= 8MObj8= 8M
> 60 .. 6768 .. 6f70 .. 7778 .. 7f
>
> Alexander.
>
>
>
>
> On Wed, Oct 11, 2017 at 3:19 PM, Alexander Kushnirenko <
> kushnire...@gmail.com> wrote:
>
>> Oh!  I put a wrong link, sorry  The picture which explains stripe_unit
>> and stripe count is here:
>>
>>
>> https://indico.cern.ch/event/330212/contributions/1718786/attachments/642384/883834/CephPluginForXroot.pdf
>>
>> I tried to attach it in the mail, but it was blocked.
>>
>>
>> On Wed, Oct 11, 2017 at 3:16 PM, Alexander Kushnirenko <
>> kushnire...@gmail.com> wrote:
>>
>>> Hi, Ian!
>>>
>>> Thank you for your reference!
>>>
>>> Could you comment on the following rule:
>>> object_size = stripe_unit * stripe_count
>>> Or it is not necessarily so?
>>>
>>> I refer to page 8 in this report:
>>>
>>>
>>> https://indico.cern.ch/event/531810/contributions/2298934/attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf
>>>
>>>
>>> Alexander.
>>>
>>> On Wed, Oct 11, 2017 at 1:11 PM,  wrote:
>>>
 Hi Gregory

 You’re right, when setting the object layout in libradosstriper, one
 should set all three parameters (the number of stripes, the size of the
 stripe unit, and the size of the striped object). The Ceph plugin for
 GridFTP has an example of this at
 https://github.com/stfc/gridFTPCephPlugin/blob/master/ceph_posix.cpp#L371



 At RAL, we use the following values:



 $STRIPER_NUM_STRIPES 1

 $STRIPER_STRIPE_UNIT 8388608

 $STRIPER_OBJECT_SIZE 67108864



 Regards,



 Ian Johnson MBCS

 Data Services Group

 Scientific Computing Department

 Rutherford Appleton Laboratory



 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


>>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Hi, Gregory, Ian!

There is very little information on striper mode in Ceph documentation.
Could this explanation help?

The logic of striper mode is very much the same as in RAID-0.  There are 3
parameters that drives it:

stripe_unit - the stripe size  (default=4M)
stripe_count - how many objects to write in parallel (default=1)
object_size  - when to stop increasing object size and create new objects.
 (default =4M)

For example if you write 132M of data (132 consecutive pieces of data 1M
each) in striped mode with the following parameters:
stripe_unit = 8M
stripe_count = 4
object_size = 24M
Then 8 objects will be created - 4 objects with 24M size and 4 objects with
8M size.

Obj1=24MObj2=24MObj3=24MObj4=24M
00 .. 07 08 .. 0f 10 .. 17 18 .. 1f  <--
consecutive 1M pieces of data
20 .. 27 21 .. 2f 30 .. 37 38 .. 3f
40 .. 47 48 .. 4f 50 .. 57 58 .. 5f

Obj5= 8MObj6= 8MObj7= 8MObj8= 8M
60 .. 6768 .. 6f70 .. 7778 .. 7f

Alexander.




On Wed, Oct 11, 2017 at 3:19 PM, Alexander Kushnirenko <
kushnire...@gmail.com> wrote:

> Oh!  I put a wrong link, sorry  The picture which explains stripe_unit and
> stripe count is here:
>
> https://indico.cern.ch/event/330212/contributions/1718786/at
> tachments/642384/883834/CephPluginForXroot.pdf
>
> I tried to attach it in the mail, but it was blocked.
>
>
> On Wed, Oct 11, 2017 at 3:16 PM, Alexander Kushnirenko <
> kushnire...@gmail.com> wrote:
>
>> Hi, Ian!
>>
>> Thank you for your reference!
>>
>> Could you comment on the following rule:
>> object_size = stripe_unit * stripe_count
>> Or it is not necessarily so?
>>
>> I refer to page 8 in this report:
>>
>> https://indico.cern.ch/event/531810/contributions/2298934/at
>> tachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf
>>
>>
>> Alexander.
>>
>> On Wed, Oct 11, 2017 at 1:11 PM,  wrote:
>>
>>> Hi Gregory
>>>
>>> You’re right, when setting the object layout in libradosstriper, one
>>> should set all three parameters (the number of stripes, the size of the
>>> stripe unit, and the size of the striped object). The Ceph plugin for
>>> GridFTP has an example of this at https://github.com/stfc/gridFT
>>> PCephPlugin/blob/master/ceph_posix.cpp#L371
>>>
>>>
>>>
>>> At RAL, we use the following values:
>>>
>>>
>>>
>>> $STRIPER_NUM_STRIPES 1
>>>
>>> $STRIPER_STRIPE_UNIT 8388608
>>>
>>> $STRIPER_OBJECT_SIZE 67108864
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Ian Johnson MBCS
>>>
>>> Data Services Group
>>>
>>> Scientific Computing Department
>>>
>>> Rutherford Appleton Laboratory
>>>
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>
>>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Oh!  I put a wrong link, sorry  The picture which explains stripe_unit and
stripe count is here:

https://indico.cern.ch/event/330212/contributions/1718786/attachments/642384/883834/CephPluginForXroot.pdf

I tried to attach it in the mail, but it was blocked.


On Wed, Oct 11, 2017 at 3:16 PM, Alexander Kushnirenko <
kushnire...@gmail.com> wrote:

> Hi, Ian!
>
> Thank you for your reference!
>
> Could you comment on the following rule:
> object_size = stripe_unit * stripe_count
> Or it is not necessarily so?
>
> I refer to page 8 in this report:
>
> https://indico.cern.ch/event/531810/contributions/2298934/at
> tachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf
>
>
> Alexander.
>
> On Wed, Oct 11, 2017 at 1:11 PM,  wrote:
>
>> Hi Gregory
>>
>> You’re right, when setting the object layout in libradosstriper, one
>> should set all three parameters (the number of stripes, the size of the
>> stripe unit, and the size of the striped object). The Ceph plugin for
>> GridFTP has an example of this at https://github.com/stfc/gridFT
>> PCephPlugin/blob/master/ceph_posix.cpp#L371
>>
>>
>>
>> At RAL, we use the following values:
>>
>>
>>
>> $STRIPER_NUM_STRIPES 1
>>
>> $STRIPER_STRIPE_UNIT 8388608
>>
>> $STRIPER_OBJECT_SIZE 67108864
>>
>>
>>
>> Regards,
>>
>>
>>
>> Ian Johnson MBCS
>>
>> Data Services Group
>>
>> Scientific Computing Department
>>
>> Rutherford Appleton Laboratory
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Hi, Ian!

Thank you for your reference!

Could you comment on the following rule:
object_size = stripe_unit * stripe_count
Or it is not necessarily so?

I refer to page 8 in this report:

https://indico.cern.ch/event/531810/contributions/2298934/at
tachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf


Alexander.

On Wed, Oct 11, 2017 at 1:11 PM,  wrote:

> Hi Gregory
>
> You’re right, when setting the object layout in libradosstriper, one
> should set all three parameters (the number of stripes, the size of the
> stripe unit, and the size of the striped object). The Ceph plugin for
> GridFTP has an example of this at https://github.com/stfc/gridFT
> PCephPlugin/blob/master/ceph_posix.cpp#L371
>
>
>
> At RAL, we use the following values:
>
>
>
> $STRIPER_NUM_STRIPES 1
>
> $STRIPER_STRIPE_UNIT 8388608
>
> $STRIPER_OBJECT_SIZE 67108864
>
>
>
> Regards,
>
>
>
> Ian Johnson MBCS
>
> Data Services Group
>
> Scientific Computing Department
>
> Rutherford Appleton Laboratory
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Hi, Gregory!

You are absolutely right! Thanks!

The following sequence solves the problem:
rados_striper_set_object_layout_stripe_unit(m_striper, stripe_unit);
rados_striper_set_object_layout_stripe_count(m_striper, stripe_count);
int stripe_size = stripe_unit * stripe_count;
rados_striper_set_object_layout_object_size(m_striper, stripe_size);

Now there is very little in the documentation about meaning of above
parameters.  The only document I found is CERN IT group presentation (page
8).  Perhaps it is obvious.  Also it seems that optimizing these parameters
is meaningful in large scale Ceph installations only.

https://indico.cern.ch/event/531810/contributions/2298934/
attachments/1358128/2053937/Ceph-Experience-at-RAL-final.pdf

Now if I have a 6TB disks, then in default installation there would be
6TB/4MB = 1.5M objects per OSD.  Does it create any performance hit?

Thank you,
Alexander

On Tue, Oct 10, 2017 at 12:38 AM, Gregory Farnum  wrote:

> Well, just from a quick skim, libradosstriper.h has a function
> rados_striper_set_object_layout_object_size(rados_striper_t striper,
> unsigned int object_size)
> and libradosstriper.hpp has one in RadosStriper
> set_object_layout_object_size(unsigned int object_size);
>
> So I imagine you specify it with those the same way you've set the stripe
> unit and counts.
>
> On Sat, Oct 7, 2017 at 12:38 PM Alexander Kushnirenko <
> kushnire...@gmail.com> wrote:
>
>> Hi, Gregory!
>>
>> It turns out that this error is internal CEPH feature. I wrote standalone
>> program to create 132M object in striper mode. It works only for 4M
>> stripe.  If you set stripe_unit = 2M it still creates 4M stripe_unit.
>> Anything bigger than 4M causes crash here
>> :
>>
>>
>> __u32 object_size = layout->object_size;
>>   __u32 su = layout->stripe_unit;
>>   __u32 stripe_count = layout->stripe_count;
>>   assert(object_size >= su);   <
>>
>> I'm curious where it gets layout->object_size for object that is just
>> been created.
>>
>> As I understod striper mode was created by CERN guys.  In there document
>> 
>> they recommend 8M stripe_unit.  But it does not work in luminous.
>>
>> Created I/O context.
>> Connected to pool backup with rados_striper_create
>> Stripe unit OK 8388608
>> Stripe count OK 1
>> /build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void
>> Striper::file_to_extents(CephContext*, const char*, const
>> file_layout_t*, uint64_t, uint64_t, uint64_t, std::map> std::vector >&, uint64_t)' thread 7f13bd5c1e00 time
>> 2017-10-07 21:44:58.654778
>> /build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >=
>> su)
>>  ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous
>> (rc)
>>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x102) [0x7f13b3f3b332]
>>  2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long, std::map> std::vector,
>> std::less, std::allocator std::vector > > >&, unsigned
>> long)+0x1e1e) [0x7f13bce235ee]
>>  3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
>> const*, unsigned long, unsigned long, unsigned long,
>> std::vector&, unsigned
>> long)+0x51) [0x7f13bce23691]
>>  4: (libradosstriper::RadosStriperImpl::internal_
>> aio_write(std::__cxx11::basic_string> std::allocator > const&, 
>> boost::intrusive_ptr,
>> ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout
>> const&)+0x224) [0x7f13bcda4184]
>>  5: (libradosstriper::RadosStriperImpl::write_in_
>> open_object(std::__cxx11::basic_string> std::allocator > const&, ceph_file_layout const&,
>> std::__cxx11::basic_string> std::allocator > const&, ceph::buffer::list const&, unsigned long,
>> unsigned long)+0x13c) [0x7f13bcda476c]
>>  6: 
>> (libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string> std::char_traits, std::allocator > const&, ceph::buffer::list
>> const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5]
>>  7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b]
>>  8: (()+0x10fb) [0x55dd87b410fb]
>>  9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1]
>>  10: (()+0xbca) [0x55dd87b40bca]
>>
>>
>> On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum 
>> wrote:
>>
>>> I haven't used the striper, but it appears to make you specify sizes,
>>> stripe units, and stripe counts. I would expect you need to make sure that
>>> the size is an integer multiple of the stripe unit. And it probably
>>> defaults to a 4MB object if you don't specify one?
>>>
>>> On Fri, Sep 29, 2017 at 

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[ian.johnson]

Hi Gregory
You're right, when setting the object layout in libradosstriper, one should set 
all three parameters (the number of stripes, the size of the stripe unit, and 
the size of the striped object). The Ceph plugin for GridFTP has an example of 
this at 
https://github.com/stfc/gridFTPCephPlugin/blob/master/ceph_posix.cpp#L371

At RAL, we use the following values:

$STRIPER_NUM_STRIPES 1
$STRIPER_STRIPE_UNIT 8388608
$STRIPER_OBJECT_SIZE 67108864

Regards,

Ian Johnson MBCS
Data Services Group
Scientific Computing Department
Rutherford Appleton Laboratory

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Gregory Farnum]

Well, just from a quick skim, libradosstriper.h has a function
rados_striper_set_object_layout_object_size(rados_striper_t striper,
unsigned int object_size)
and libradosstriper.hpp has one in RadosStriper
set_object_layout_object_size(unsigned int object_size);

So I imagine you specify it with those the same way you've set the stripe
unit and counts.

On Sat, Oct 7, 2017 at 12:38 PM Alexander Kushnirenko 
wrote:

> Hi, Gregory!
>
> It turns out that this error is internal CEPH feature. I wrote standalone
> program to create 132M object in striper mode. It works only for 4M
> stripe.  If you set stripe_unit = 2M it still creates 4M stripe_unit.
> Anything bigger than 4M causes crash here
> :
>
>
> __u32 object_size = layout->object_size;
>   __u32 su = layout->stripe_unit;
>   __u32 stripe_count = layout->stripe_count;
>   assert(object_size >= su);   <
>
> I'm curious where it gets layout->object_size for object that is just been
> created.
>
> As I understod striper mode was created by CERN guys.  In there document
> 
> they recommend 8M stripe_unit.  But it does not work in luminous.
>
> Created I/O context.
> Connected to pool backup with rados_striper_create
> Stripe unit OK 8388608
> Stripe count OK 1
> /build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void
> Striper::file_to_extents(CephContext*, const char*, const file_layout_t*,
> uint64_t, uint64_t, uint64_t, std::map >&, uint64_t)' thread 7f13bd5c1e00 time 2017-10-07 21:44:58.654778
> /build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >=
> su)
>  ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous
> (rc)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x102) [0x7f13b3f3b332]
>  2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
> const*, unsigned long, unsigned long, unsigned long, std::map std::vector,
> std::less, std::allocator > > >&, unsigned
> long)+0x1e1e) [0x7f13bce235ee]
>  3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
> const*, unsigned long, unsigned long, unsigned long,
> std::vector&, unsigned
> long)+0x51) [0x7f13bce23691]
>  4:
> (libradosstriper::RadosStriperImpl::internal_aio_write(std::__cxx11::basic_string std::char_traits, std::allocator > const&,
> boost::intrusive_ptr,
> ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout
> const&)+0x224) [0x7f13bcda4184]
>  5:
> (libradosstriper::RadosStriperImpl::write_in_open_object(std::__cxx11::basic_string std::char_traits, std::allocator > const&, ceph_file_layout
> const&, std::__cxx11::basic_string std::allocator > const&, ceph::buffer::list const&, unsigned long,
> unsigned long)+0x13c) [0x7f13bcda476c]
>  6:
> (libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string std::char_traits, std::allocator > const&, ceph::buffer::list
> const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5]
>  7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b]
>  8: (()+0x10fb) [0x55dd87b410fb]
>  9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1]
>  10: (()+0xbca) [0x55dd87b40bca]
>
>
> On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum 
> wrote:
>
>> I haven't used the striper, but it appears to make you specify sizes,
>> stripe units, and stripe counts. I would expect you need to make sure that
>> the size is an integer multiple of the stripe unit. And it probably
>> defaults to a 4MB object if you don't specify one?
>>
>> On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko <
>> kushnire...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup
>>> with libradosstriper1 support.
>>>
>>> Libradosstriber was suggested on this list to solve the problem, that
>>> current CEPH-12 discourages users from using object with very big size
>>> (>128MB).  Bareos treat Rados Object as Volume and in CEPH-10 it created
>>> objects with very big size (10G and more).  CEPH-10 allowed such behaviour,
>>> put recovery indeed take very long time. So stripping objects seems to be
>>> the right thing to do.
>>>
>>> Bareos supports libradosstriper and the code seems to work. But for some
>>> reason it run only with stripe_unit=4194304, which seems to be typical
>>> value for RadosGW for example.  I tried several other values for
>>> stripe_unit, but the code exit with error.
>>>
>>> Is there a particular reason why only 4M size works?  Can one use some
>>> CLI to test different stripe sizes?
>>>
>>> Basic flow of creating object in Bareos is the following:
>>> 

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Hi, Gregory!

It turns out that this error is internal CEPH feature. I wrote standalone
program to create 132M object in striper mode. It works only for 4M
stripe.  If you set stripe_unit = 2M it still creates 4M stripe_unit.
Anything bigger than 4M causes crash here
:


__u32 object_size = layout->object_size;
  __u32 su = layout->stripe_unit;
  __u32 stripe_count = layout->stripe_count;
  assert(object_size >= su);   <

I'm curious where it gets layout->object_size for object that is just been
created.

As I understod striper mode was created by CERN guys.  In there document

they recommend 8M stripe_unit.  But it does not work in luminous.

Created I/O context.
Connected to pool backup with rados_striper_create
Stripe unit OK 8388608
Stripe count OK 1
/build/ceph-12.2.0/src/osdc/Striper.cc: In function 'static void
Striper::file_to_extents(CephContext*, const char*, const file_layout_t*,
uint64_t, uint64_t, uint64_t, std::map&, uint64_t)' thread 7f13bd5c1e00 time 2017-10-07 21:44:58.654778
/build/ceph-12.2.0/src/osdc/Striper.cc: 64: FAILED assert(object_size >= su)
 ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous
(rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x102) [0x7f13b3f3b332]
 2: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
const*, unsigned long, unsigned long, unsigned long, std::map,
std::less, std::allocator > > >&, unsigned
long)+0x1e1e) [0x7f13bce235ee]
 3: (Striper::file_to_extents(CephContext*, char const*, file_layout_t
const*, unsigned long, unsigned long, unsigned long,
std::vector&, unsigned
long)+0x51) [0x7f13bce23691]
 4:
(libradosstriper::RadosStriperImpl::internal_aio_write(std::__cxx11::basic_string const&,
boost::intrusive_ptr,
ceph::buffer::list const&, unsigned long, unsigned long, ceph_file_layout
const&)+0x224) [0x7f13bcda4184]
 5:
(libradosstriper::RadosStriperImpl::write_in_open_object(std::__cxx11::basic_string const&, ceph_file_layout
const&, std::__cxx11::basic_string const&, ceph::buffer::list const&, unsigned long,
unsigned long)+0x13c) [0x7f13bcda476c]
 6:
(libradosstriper::RadosStriperImpl::write(std::__cxx11::basic_string const&, ceph::buffer::list
const&, unsigned long, unsigned long)+0xd5) [0x7f13bcda4bd5]
 7: (rados_striper_write()+0xdb) [0x7f13bcd9ba0b]
 8: (()+0x10fb) [0x55dd87b410fb]
 9: (__libc_start_main()+0xf1) [0x7f13bc9d72b1]
 10: (()+0xbca) [0x55dd87b40bca]


On Fri, Sep 29, 2017 at 11:46 PM, Gregory Farnum  wrote:

> I haven't used the striper, but it appears to make you specify sizes,
> stripe units, and stripe counts. I would expect you need to make sure that
> the size is an integer multiple of the stripe unit. And it probably
> defaults to a 4MB object if you don't specify one?
>
> On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko <
> kushnire...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup
>> with libradosstriper1 support.
>>
>> Libradosstriber was suggested on this list to solve the problem, that
>> current CEPH-12 discourages users from using object with very big size
>> (>128MB).  Bareos treat Rados Object as Volume and in CEPH-10 it created
>> objects with very big size (10G and more).  CEPH-10 allowed such behaviour,
>> put recovery indeed take very long time. So stripping objects seems to be
>> the right thing to do.
>>
>> Bareos supports libradosstriper and the code seems to work. But for some
>> reason it run only with stripe_unit=4194304, which seems to be typical
>> value for RadosGW for example.  I tried several other values for
>> stripe_unit, but the code exit with error.
>>
>> Is there a particular reason why only 4M size works?  Can one use some
>> CLI to test different stripe sizes?
>>
>> Basic flow of creating object in Bareos is the following:
>> rados_ioctx_create(m_cluster, m_rados_poolname, _ctx);
>> rados_striper_create(m_ctx, _striper);
>> rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit);
>> rados_striper_set_object_layout_stripe_count(m_striper, m_stripe_count);
>> .
>> status = rados_striper_write(m_striper, m_virtual_filename, buffer,
>> count, offset);
>>
>> Alexander
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
#include 
#include 
#include 
#include 
#include 
#define EXIT_FAILURE 1

int main (int argc, const char * argv[])
{


	/* Declare the cluster 

Re: [ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Gregory Farnum]

I haven't used the striper, but it appears to make you specify sizes,
stripe units, and stripe counts. I would expect you need to make sure that
the size is an integer multiple of the stripe unit. And it probably
defaults to a 4MB object if you don't specify one?

On Fri, Sep 29, 2017 at 2:09 AM Alexander Kushnirenko 
wrote:

> Hi,
>
> I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup
> with libradosstriper1 support.
>
> Libradosstriber was suggested on this list to solve the problem, that
> current CEPH-12 discourages users from using object with very big size
> (>128MB).  Bareos treat Rados Object as Volume and in CEPH-10 it created
> objects with very big size (10G and more).  CEPH-10 allowed such behaviour,
> put recovery indeed take very long time. So stripping objects seems to be
> the right thing to do.
>
> Bareos supports libradosstriper and the code seems to work. But for some
> reason it run only with stripe_unit=4194304, which seems to be typical
> value for RadosGW for example.  I tried several other values for
> stripe_unit, but the code exit with error.
>
> Is there a particular reason why only 4M size works?  Can one use some CLI
> to test different stripe sizes?
>
> Basic flow of creating object in Bareos is the following:
> rados_ioctx_create(m_cluster, m_rados_poolname, _ctx);
> rados_striper_create(m_ctx, _striper);
> rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit);
> rados_striper_set_object_layout_stripe_count(m_striper, m_stripe_count);
> .
> status = rados_striper_write(m_striper, m_virtual_filename, buffer, count,
> offset);
>
> Alexander
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Bareos and libradosstriper works only for 4M sripe_unit size

[Alexander Kushnirenko]

Hi,

I'm trying to use CEPH-12.2.0 as storage for with Bareos-16.2.4 backup with
libradosstriper1 support.

Libradosstriber was suggested on this list to solve the problem, that
current CEPH-12 discourages users from using object with very big size
(>128MB).  Bareos treat Rados Object as Volume and in CEPH-10 it created
objects with very big size (10G and more).  CEPH-10 allowed such behaviour,
put recovery indeed take very long time. So stripping objects seems to be
the right thing to do.

Bareos supports libradosstriper and the code seems to work. But for some
reason it run only with stripe_unit=4194304, which seems to be typical
value for RadosGW for example.  I tried several other values for
stripe_unit, but the code exit with error.

Is there a particular reason why only 4M size works?  Can one use some CLI
to test different stripe sizes?

Basic flow of creating object in Bareos is the following:
rados_ioctx_create(m_cluster, m_rados_poolname, _ctx);
rados_striper_create(m_ctx, _striper);
rados_striper_set_object_layout_stripe_unit(m_striper, m_stripe_unit);
rados_striper_set_object_layout_stripe_count(m_striper, m_stripe_count);
.
status = rados_striper_write(m_striper, m_virtual_filename, buffer, count,
offset);

Alexander
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com