Re: [ceph-users] Performance issues with small files

[Bryan Stillwell]

I need to restart the upload process again because all the objects
have a content-type of 'binary/octet-stream' instead of 'image/jpeg',
'image/png', etc.  I plan on enabling monitoring this time so we can
see if there are any signs of what might be going on.  Did you want me
to increase the number of buckets to see if that changes anything?
This is pretty easy for me to do.

Bryan

On Thu, Sep 5, 2013 at 11:07 AM, Mark Nelson  wrote:
> based on your numbers, you were at something like an average of 186 objects
> per bucket at the 20 hour mark?  I wonder how this trend compares to what
> you'd see with a single bucket.
>
> With that many buckets you should have indexes well spread across all of the
> OSDs.  It'd be interesting to know what the iops/throughput is on all of
> your OSDs now (blktrace/seekwatcher can help here, but they are not the
> easiest tools to setup/use).
>
> Mark
>
> On 09/05/2013 11:59 AM, Bryan Stillwell wrote:
>>
>> Mark,
>>
>> Yesterday I blew away all the objects and restarted my test using
>> multiple buckets, and things are definitely better!
>>
>> After ~20 hours I've already uploaded ~3.5 million objects, which much
>> is better then the ~1.5 million I did over ~96 hours this past
>> weekend.  Unfortunately it seems that things have slowed down a bit.
>> The average upload rate over those first 20 hours was ~48
>> objects/second, but now I'm only seeing ~20 objects/second.  This is
>> with 18,836 buckets.
>>
>> Bryan
>>
>> On Wed, Sep 4, 2013 at 12:43 PM, Bryan Stillwell
>>  wrote:
>>>
>>> So far I haven't seen much of a change.  It's still working through
>>> removing
>>> the bucket that reached 1.5 million objects though (my guess is that'll
>>> take
>>> a few more days), so I believe that might have something to do with it.
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson 
>>> wrote:


 Bryan,

 Good explanation.  How's performance now that you've spread the load
 over
 multiple buckets?

 Mark

 On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>
>
> Bill,
>
> I've run into a similar issue with objects averaging ~100KiB.  The
> explanation I received on IRC is that there are scaling issues if
> you're
> uploading them all to the same bucket because the index isn't sharded.
>The recommended solution is to spread the objects out to a lot of
> buckets.  However, that ran me into another issue once I hit 1000
> buckets which is a per user limit.  I switched the limit to be
> unlimited
> with this command:
>
> radosgw-admin user modify --uid=your_username --max-buckets=0
>
> Bryan
>
>
> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer  > wrote:
>
>  I'm testing ceph for storing a very large number of small files.
>I'm seeing some performance issues and would like to see if
> anyone
>  could offer any insight as to what I could do to correct this.
>
>  Some numbers:
>
>  Uploaded 184111 files, with an average file size of 5KB, using
>  10 separate servers to upload the request using Python and the
>  cloudfiles module.  I stopped uploading after 53 minutes, which
>  seems to average 5.7 files per second per node.
>
>
>  My storage cluster consists of 21 OSD's across 7 servers, with
> their
>  journals written to SSD drives.  I've done a default installation,
>  using ceph-deploy with the dumpling release.
>
>  I'm using statsd to monitor the performance, and what's
> interesting
>  is when I start with an empty bucket, performance is amazing, with
>  average response times of 20-50ms.  However as time goes on, the
>  response times go in to the hundreds, and the average number of
>  uploads per second drops.
>
>  I've installed radosgw on all 7 ceph servers.  I've tested using a
>  load balancer to distribute the api calls, as well as pointing the
>  10 worker servers to a single instance.  I've not seen a real
>  different in performance with this either.
>
>
>  Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of
> ram,
>  OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
>  drives for storage.
>
>
>  Any help would be greatly appreciated.
>
>
>  ___
>  ceph-users mailing list
>  ceph-users@lists.ceph.com 
>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Bill Omer]

Sorry, I meant to say the first four characters, for a total of 65539
buckets


On Thu, Sep 5, 2013 at 12:30 PM, Bryan Stillwell  wrote:

> Wouldn't using only the first two characters in the file name result
> in less then 65k buckets being used?
>
> For example if the file names contained 0-9 and a-f, that would only
> be 256 buckets (16*16).  Or if they contained 0-9, a-z, and A-Z, that
> would only be 3,844 buckets (62 * 62).
>
> Bryan
>
>
> On Thu, Sep 5, 2013 at 8:19 AM, Bill Omer  wrote:
> >
> > Thats correct.  We created 65k buckets, using two hex characters as the
> naming convention, then stored the files in each container based on their
> first two characters in the file name.  The end result was 20-50 files per
> bucket.  Once all of the buckets were created and files were being loaded,
> we still observed an increase in latency overtime.
> >
> > Is there a way to disable indexing?  Or are there other settings you can
> suggest to attempt to speed this process up?
> >
> >
> > On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson 
> wrote:
> >>
> >> Just for clarification, distributing objects over lots of buckets isn't
> helping improve small object performance?
> >>
> >> The degradation over time is similar to something I've seen in the
> past, with higher numbers of seeks on the underlying OSD device over time.
>  Is it always (temporarily) resolved writing to a new empty bucket?
> >>
> >> Mark
> >>
> >>
> >> On 09/04/2013 02:45 PM, Bill Omer wrote:
> >>>
> >>> We've actually done the same thing, creating 65k buckets and storing
> >>> 20-50 objects in each.  No change really, not noticeable anyway
> >>>
> >>>
> >>> On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
> >>> mailto:bstillw...@photobucket.com>>
> wrote:
> >>>
> >>> So far I haven't seen much of a change.  It's still working through
> >>> removing the bucket that reached 1.5 million objects though (my
> >>> guess is that'll take a few more days), so I believe that might
> have
> >>> something to do with it.
> >>>
> >>> Bryan
> >>>
> >>>
> >>> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
> >>> mailto:mark.nel...@inktank.com>> wrote:
> >>>
> >>> Bryan,
> >>>
> >>> Good explanation.  How's performance now that you've spread the
> >>> load over multiple buckets?
> >>>
> >>> Mark
> >>>
> >>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
> >>>
> >>> Bill,
> >>>
> >>> I've run into a similar issue with objects averaging
> >>> ~100KiB.  The
> >>> explanation I received on IRC is that there are scaling
> >>> issues if you're
> >>> uploading them all to the same bucket because the index
> >>> isn't sharded.
> >>>The recommended solution is to spread the objects out to
> >>> a lot of
> >>> buckets.  However, that ran me into another issue once I
> hit
> >>> 1000
> >>> buckets which is a per user limit.  I switched the limit to
> >>> be unlimited
> >>> with this command:
> >>>
> >>> radosgw-admin user modify --uid=your_username
> --max-buckets=0
> >>>
> >>> Bryan
> >>>
> >>>
> >>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
> >>> mailto:bill.o...@gmail.com>
> >>> >>
> >>>
> >>> wrote:
> >>>
> >>>  I'm testing ceph for storing a very large number of
> >>> small files.
> >>>I'm seeing some performance issues and would like to
> >>> see if anyone
> >>>  could offer any insight as to what I could do to
> >>> correct this.
> >>>
> >>>  Some numbers:
> >>>
> >>>  Uploaded 184111 files, with an average file size of
> >>> 5KB, using
> >>>  10 separate servers to upload the request using Python
> >>> and the
> >>>  cloudfiles module.  I stopped uploading after 53
> >>> minutes, which
> >>>  seems to average 5.7 files per second per node.
> >>>
> >>>
> >>>  My storage cluster consists of 21 OSD's across 7
> >>> servers, with their
> >>>  journals written to SSD drives.  I've done a default
> >>> installation,
> >>>  using ceph-deploy with the dumpling release.
> >>>
> >>>  I'm using statsd to monitor the performance, and
> what's
> >>> interesting
> >>>  is when I start with an empty bucket, performance is
> >>> amazing, with
> >>>  average response times of 20-50ms.  However as time
> >>> goes on, the
> >>>  response times go in to the hundreds, and the average
> >>> number of
> >>>  uploads per second drops

Re: [ceph-users] Performance issues with small files

[Sage Weil]

On Thu, 5 Sep 2013, Bill Omer wrote:
> Thats correct.  We created 65k buckets, using two hex characters as the
> naming convention, then stored the files in each container based on their
> first two characters in the file name.  The end result was 20-50 files per
> bucket.  Once all of the buckets were created and files were being loaded,
> we still observed an increase in latency overtime.

This might be going too far in the opposite direction.  I would target 
1000's of objects per bucket, not 10's.  The radosgw has to validate 
bucket ACLs on requests.  It caches them, but it probably can't cache 64K 
of them (not by default at least!).  And even if it can, it will take a 
long long time for the cache to warm up.  In any case, the end result is 
that there is probably an extra rados request going on on the backend for 
every request.

Maybe try over ~1000 buckets and see how that goes?  And give the cache a 
bit of time to warm up?

sage



> Is there a way to disable indexing?  Or are there other settings you can
> suggest to attempt to speed this process up?
> 
> 
> On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson  wrote:
>   Just for clarification, distributing objects over lots of
>   buckets isn't helping improve small object performance?
> 
>   The degradation over time is similar to something I've seen in
>   the past, with higher numbers of seeks on the underlying OSD
>   device over time.  Is it always (temporarily) resolved writing
>   to a new empty bucket?
> 
>   Mark
> 
>   On 09/04/2013 02:45 PM, Bill Omer wrote:
>   We've actually done the same thing, creating 65k buckets
>   and storing
>   20-50 objects in each.  No change really, not noticeable
>   anyway
> 
> 
>   On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
> mailto:bstillw...@photobucket.com>>
> wrote:
> 
>     So far I haven't seen much of a change.  It's still working
> through
>     removing the bucket that reached 1.5 million objects though
> (my
>     guess is that'll take a few more days), so I believe that
> might have
>     something to do with it.
> 
>     Bryan
> 
> 
>     On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
>     mailto:mark.nel...@inktank.com>>
> wrote:
> 
>         Bryan,
> 
>         Good explanation.  How's performance now that you've
> spread the
>         load over multiple buckets?
> 
>         Mark
> 
>         On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
> 
>             Bill,
> 
>             I've run into a similar issue with objects averaging
>             ~100KiB.  The
>             explanation I received on IRC is that there are
> scaling
>             issues if you're
>             uploading them all to the same bucket because the
> index
>             isn't sharded.
>                The recommended solution is to spread the objects
> out to
>             a lot of
>             buckets.  However, that ran me into another issue
> once I hit
>             1000
>             buckets which is a per user limit.  I switched the
> limit to
>             be unlimited
>             with this command:
> 
>             radosgw-admin user modify --uid=your_username
> --max-buckets=0
> 
>             Bryan
> 
> 
>             On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
>             mailto:bill.o...@gmail.com>
>              >>
>             wrote:
> 
>                  I'm testing ceph for storing a very large
> number of
>             small files.
>                    I'm seeing some performance issues and would
> like to
>             see if anyone
>                  could offer any insight as to what I could do
> to
>             correct this.
> 
>                  Some numbers:
> 
>                  Uploaded 184111 files, with an average file
> size of
>             5KB, using
>                  10 separate servers to upload the request using
> Python
>             and the
>                  cloudfiles module.  I stopped uploading after
> 53
>             minutes, which
>                  seems to average 5.7 files per second per node.
> 
> 
>                  My storage cluster consists of 21 OSD's across
> 7
>             servers, with their
>                  journals written to SSD drives.  I've done a
> default
>             installation,
>                  using ceph-deploy with the dumpling release.
> 
>                  I'm using statsd to monitor the performance,
> and what's
>             interesting
>                  is when I start with an empty bucket,
> performance is
>             amazing, with
>                  average response times of 20-50ms.  However as
> time
>             goes on, the
>                  response times go in to the hundreds, and the
> average
>             number of
>                  uploads per second drops.
> 
>                  I've installed radosgw on all 7 ceph servers.
>  I've
>             tested using a
>                  load balancer to distribute t

Re: [ceph-users] Performance issues with small files

[Mark Nelson]

based on your numbers, you were at something like an average of 186 
objects per bucket at the 20 hour mark?  I wonder how this trend 
compares to what you'd see with a single bucket.


With that many buckets you should have indexes well spread across all of 
the OSDs.  It'd be interesting to know what the iops/throughput is on 
all of your OSDs now (blktrace/seekwatcher can help here, but they are 
not the easiest tools to setup/use).


Mark

On 09/05/2013 11:59 AM, Bryan Stillwell wrote:

Mark,

Yesterday I blew away all the objects and restarted my test using
multiple buckets, and things are definitely better!

After ~20 hours I've already uploaded ~3.5 million objects, which much
is better then the ~1.5 million I did over ~96 hours this past
weekend.  Unfortunately it seems that things have slowed down a bit.
The average upload rate over those first 20 hours was ~48
objects/second, but now I'm only seeing ~20 objects/second.  This is
with 18,836 buckets.

Bryan

On Wed, Sep 4, 2013 at 12:43 PM, Bryan Stillwell
 wrote:

So far I haven't seen much of a change.  It's still working through removing
the bucket that reached 1.5 million objects though (my guess is that'll take
a few more days), so I believe that might have something to do with it.

Bryan


On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson 
wrote:


Bryan,

Good explanation.  How's performance now that you've spread the load over
multiple buckets?

Mark

On 09/04/2013 12:39 PM, Bryan Stillwell wrote:


Bill,

I've run into a similar issue with objects averaging ~100KiB.  The
explanation I received on IRC is that there are scaling issues if you're
uploading them all to the same bucket because the index isn't sharded.
   The recommended solution is to spread the objects out to a lot of
buckets.  However, that ran me into another issue once I hit 1000
buckets which is a per user limit.  I switched the limit to be unlimited
with this command:

radosgw-admin user modify --uid=your_username --max-buckets=0

Bryan


On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer mailto:bill.o...@gmail.com>> wrote:

 I'm testing ceph for storing a very large number of small files.
   I'm seeing some performance issues and would like to see if anyone
 could offer any insight as to what I could do to correct this.

 Some numbers:

 Uploaded 184111 files, with an average file size of 5KB, using
 10 separate servers to upload the request using Python and the
 cloudfiles module.  I stopped uploading after 53 minutes, which
 seems to average 5.7 files per second per node.


 My storage cluster consists of 21 OSD's across 7 servers, with their
 journals written to SSD drives.  I've done a default installation,
 using ceph-deploy with the dumpling release.

 I'm using statsd to monitor the performance, and what's interesting
 is when I start with an empty bucket, performance is amazing, with
 average response times of 20-50ms.  However as time goes on, the
 response times go in to the hundreds, and the average number of
 uploads per second drops.

 I've installed radosgw on all 7 ceph servers.  I've tested using a
 load balancer to distribute the api calls, as well as pointing the
 10 worker servers to a single instance.  I've not seen a real
 different in performance with this either.


 Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram,
 OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
 drives for storage.


 Any help would be greatly appreciated.


 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com 
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Yehuda Sadeh]

On Thu, Sep 5, 2013 at 9:49 AM, Sage Weil  wrote:
> On Thu, 5 Sep 2013, Bill Omer wrote:
>> Thats correct.  We created 65k buckets, using two hex characters as the
>> naming convention, then stored the files in each container based on their
>> first two characters in the file name.  The end result was 20-50 files per
>> bucket.  Once all of the buckets were created and files were being loaded,
>> we still observed an increase in latency overtime.
>
> This might be going too far in the opposite direction.  I would target
> 1000's of objects per bucket, not 10's.  The radosgw has to validate
> bucket ACLs on requests.  It caches them, but it probably can't cache 64K
> of them (not by default at least!).  And even if it can, it will take a
> long long time for the cache to warm up.  In any case, the end result is
> that there is probably an extra rados request going on on the backend for
> every request.
>
> Maybe try over ~1000 buckets and see how that goes?  And give the cache a
> bit of time to warm up?


There's actually a configurable that can be played with. Try setting
something like this in your ceph.conf:

rgw cache lru size = 10

That is 10 times the default 10k.


Also, I don't remember if the obvious has been stated, but how many
pgs do you have on your data and index pools?

Yehuda

>
> sage
>
>
>
>> Is there a way to disable indexing?  Or are there other settings you can
>> suggest to attempt to speed this process up?
>>
>>
>> On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson  wrote:
>>   Just for clarification, distributing objects over lots of
>>   buckets isn't helping improve small object performance?
>>
>>   The degradation over time is similar to something I've seen in
>>   the past, with higher numbers of seeks on the underlying OSD
>>   device over time.  Is it always (temporarily) resolved writing
>>   to a new empty bucket?
>>
>>   Mark
>>
>>   On 09/04/2013 02:45 PM, Bill Omer wrote:
>>   We've actually done the same thing, creating 65k buckets
>>   and storing
>>   20-50 objects in each.  No change really, not noticeable
>>   anyway
>>
>>
>>   On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
>> mailto:bstillw...@photobucket.com>>
>> wrote:
>>
>> So far I haven't seen much of a change.  It's still working
>> through
>> removing the bucket that reached 1.5 million objects though
>> (my
>> guess is that'll take a few more days), so I believe that
>> might have
>> something to do with it.
>>
>> Bryan
>>
>>
>> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
>> mailto:mark.nel...@inktank.com>>
>> wrote:
>>
>> Bryan,
>>
>> Good explanation.  How's performance now that you've
>> spread the
>> load over multiple buckets?
>>
>> Mark
>>
>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>>
>> Bill,
>>
>> I've run into a similar issue with objects averaging
>> ~100KiB.  The
>> explanation I received on IRC is that there are
>> scaling
>> issues if you're
>> uploading them all to the same bucket because the
>> index
>> isn't sharded.
>>The recommended solution is to spread the objects
>> out to
>> a lot of
>> buckets.  However, that ran me into another issue
>> once I hit
>> 1000
>> buckets which is a per user limit.  I switched the
>> limit to
>> be unlimited
>> with this command:
>>
>> radosgw-admin user modify --uid=your_username
>> --max-buckets=0
>>
>> Bryan
>>
>>
>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
>> mailto:bill.o...@gmail.com>
>> > >>
>> wrote:
>>
>>  I'm testing ceph for storing a very large
>> number of
>> small files.
>>I'm seeing some performance issues and would
>> like to
>> see if anyone
>>  could offer any insight as to what I could do
>> to
>> correct this.
>>
>>  Some numbers:
>>
>>  Uploaded 184111 files, with an average file
>> size of
>> 5KB, using
>>  10 separate servers to upload the request using
>> Python
>> and the
>>  cloudfiles module.  I stopped uploading after
>> 53
>> minutes, which
>>  seems to average 5.7 files per second per node.
>>
>>
>>  My storage cluster consists of 21 OSD's across
>> 7
>> servers, with their
>>  journals written to SSD drives.  I've done a
>> default
>> installation,
>>  using ceph-deploy with the dumpling release.
>>
>>  I'm using statsd to monitor the performance,
>> and what's
>> interesting
>>  is when I st

Re: [ceph-users] Performance issues with small files

[Bill Omer]

I'm using all defaults created with ceph-deploy

I will try the rgw cache setting.  Do you have any other recommendations?


On Thu, Sep 5, 2013 at 1:14 PM, Yehuda Sadeh  wrote:

> On Thu, Sep 5, 2013 at 9:49 AM, Sage Weil  wrote:
> > On Thu, 5 Sep 2013, Bill Omer wrote:
> >> Thats correct.  We created 65k buckets, using two hex characters as the
> >> naming convention, then stored the files in each container based on
> their
> >> first two characters in the file name.  The end result was 20-50 files
> per
> >> bucket.  Once all of the buckets were created and files were being
> loaded,
> >> we still observed an increase in latency overtime.
> >
> > This might be going too far in the opposite direction.  I would target
> > 1000's of objects per bucket, not 10's.  The radosgw has to validate
> > bucket ACLs on requests.  It caches them, but it probably can't cache 64K
> > of them (not by default at least!).  And even if it can, it will take a
> > long long time for the cache to warm up.  In any case, the end result is
> > that there is probably an extra rados request going on on the backend for
> > every request.
> >
> > Maybe try over ~1000 buckets and see how that goes?  And give the cache a
> > bit of time to warm up?
>
>
> There's actually a configurable that can be played with. Try setting
> something like this in your ceph.conf:
>
> rgw cache lru size = 10
>
> That is 10 times the default 10k.
>
>
> Also, I don't remember if the obvious has been stated, but how many
> pgs do you have on your data and index pools?
>
> Yehuda
>
> >
> > sage
> >
> >
> >
> >> Is there a way to disable indexing?  Or are there other settings you can
> >> suggest to attempt to speed this process up?
> >>
> >>
> >> On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson 
> wrote:
> >>   Just for clarification, distributing objects over lots of
> >>   buckets isn't helping improve small object performance?
> >>
> >>   The degradation over time is similar to something I've seen in
> >>   the past, with higher numbers of seeks on the underlying OSD
> >>   device over time.  Is it always (temporarily) resolved writing
> >>   to a new empty bucket?
> >>
> >>   Mark
> >>
> >>   On 09/04/2013 02:45 PM, Bill Omer wrote:
> >>   We've actually done the same thing, creating 65k buckets
> >>   and storing
> >>   20-50 objects in each.  No change really, not noticeable
> >>   anyway
> >>
> >>
> >>   On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
> >> mailto:bstillw...@photobucket.com>>
> >> wrote:
> >>
> >> So far I haven't seen much of a change.  It's still working
> >> through
> >> removing the bucket that reached 1.5 million objects though
> >> (my
> >> guess is that'll take a few more days), so I believe that
> >> might have
> >> something to do with it.
> >>
> >> Bryan
> >>
> >>
> >> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
> >> mailto:mark.nel...@inktank.com>>
> >> wrote:
> >>
> >> Bryan,
> >>
> >> Good explanation.  How's performance now that you've
> >> spread the
> >> load over multiple buckets?
> >>
> >> Mark
> >>
> >> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
> >>
> >> Bill,
> >>
> >> I've run into a similar issue with objects averaging
> >> ~100KiB.  The
> >> explanation I received on IRC is that there are
> >> scaling
> >> issues if you're
> >> uploading them all to the same bucket because the
> >> index
> >> isn't sharded.
> >>The recommended solution is to spread the objects
> >> out to
> >> a lot of
> >> buckets.  However, that ran me into another issue
> >> once I hit
> >> 1000
> >> buckets which is a per user limit.  I switched the
> >> limit to
> >> be unlimited
> >> with this command:
> >>
> >> radosgw-admin user modify --uid=your_username
> >> --max-buckets=0
> >>
> >> Bryan
> >>
> >>
> >> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
> >> mailto:bill.o...@gmail.com>
> >>  >> >>
> >> wrote:
> >>
> >>  I'm testing ceph for storing a very large
> >> number of
> >> small files.
> >>I'm seeing some performance issues and would
> >> like to
> >> see if anyone
> >>  could offer any insight as to what I could do
> >> to
> >> correct this.
> >>
> >>  Some numbers:
> >>
> >>  Uploaded 184111 files, with an average file
> >> size of
> >> 5KB, using
> >>  10 separate servers to upload the request using
> >> Python
> >> and the
> >>  cloudfiles module.  I stopped uploading after
> >> 53
> >> minutes, which
> >>  seems

Re: [ceph-users] Performance issues with small files

[Bryan Stillwell]

Mark,

Yesterday I blew away all the objects and restarted my test using
multiple buckets, and things are definitely better!

After ~20 hours I've already uploaded ~3.5 million objects, which much
is better then the ~1.5 million I did over ~96 hours this past
weekend.  Unfortunately it seems that things have slowed down a bit.
The average upload rate over those first 20 hours was ~48
objects/second, but now I'm only seeing ~20 objects/second.  This is
with 18,836 buckets.

Bryan

On Wed, Sep 4, 2013 at 12:43 PM, Bryan Stillwell
 wrote:
> So far I haven't seen much of a change.  It's still working through removing
> the bucket that reached 1.5 million objects though (my guess is that'll take
> a few more days), so I believe that might have something to do with it.
>
> Bryan
>
>
> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson 
> wrote:
>>
>> Bryan,
>>
>> Good explanation.  How's performance now that you've spread the load over
>> multiple buckets?
>>
>> Mark
>>
>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>>>
>>> Bill,
>>>
>>> I've run into a similar issue with objects averaging ~100KiB.  The
>>> explanation I received on IRC is that there are scaling issues if you're
>>> uploading them all to the same bucket because the index isn't sharded.
>>>   The recommended solution is to spread the objects out to a lot of
>>> buckets.  However, that ran me into another issue once I hit 1000
>>> buckets which is a per user limit.  I switched the limit to be unlimited
>>> with this command:
>>>
>>> radosgw-admin user modify --uid=your_username --max-buckets=0
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer >> > wrote:
>>>
>>> I'm testing ceph for storing a very large number of small files.
>>>   I'm seeing some performance issues and would like to see if anyone
>>> could offer any insight as to what I could do to correct this.
>>>
>>> Some numbers:
>>>
>>> Uploaded 184111 files, with an average file size of 5KB, using
>>> 10 separate servers to upload the request using Python and the
>>> cloudfiles module.  I stopped uploading after 53 minutes, which
>>> seems to average 5.7 files per second per node.
>>>
>>>
>>> My storage cluster consists of 21 OSD's across 7 servers, with their
>>> journals written to SSD drives.  I've done a default installation,
>>> using ceph-deploy with the dumpling release.
>>>
>>> I'm using statsd to monitor the performance, and what's interesting
>>> is when I start with an empty bucket, performance is amazing, with
>>> average response times of 20-50ms.  However as time goes on, the
>>> response times go in to the hundreds, and the average number of
>>> uploads per second drops.
>>>
>>> I've installed radosgw on all 7 ceph servers.  I've tested using a
>>> load balancer to distribute the api calls, as well as pointing the
>>> 10 worker servers to a single instance.  I've not seen a real
>>> different in performance with this either.
>>>
>>>
>>> Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram,
>>> OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
>>> drives for storage.
>>>
>>>
>>> Any help would be greatly appreciated.
>>>
>>>
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Bryan Stillwell]

Wouldn't using only the first two characters in the file name result
in less then 65k buckets being used?

For example if the file names contained 0-9 and a-f, that would only
be 256 buckets (16*16).  Or if they contained 0-9, a-z, and A-Z, that
would only be 3,844 buckets (62 * 62).

Bryan


On Thu, Sep 5, 2013 at 8:19 AM, Bill Omer  wrote:
>
> Thats correct.  We created 65k buckets, using two hex characters as the 
> naming convention, then stored the files in each container based on their 
> first two characters in the file name.  The end result was 20-50 files per 
> bucket.  Once all of the buckets were created and files were being loaded, we 
> still observed an increase in latency overtime.
>
> Is there a way to disable indexing?  Or are there other settings you can 
> suggest to attempt to speed this process up?
>
>
> On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson  wrote:
>>
>> Just for clarification, distributing objects over lots of buckets isn't 
>> helping improve small object performance?
>>
>> The degradation over time is similar to something I've seen in the past, 
>> with higher numbers of seeks on the underlying OSD device over time.  Is it 
>> always (temporarily) resolved writing to a new empty bucket?
>>
>> Mark
>>
>>
>> On 09/04/2013 02:45 PM, Bill Omer wrote:
>>>
>>> We've actually done the same thing, creating 65k buckets and storing
>>> 20-50 objects in each.  No change really, not noticeable anyway
>>>
>>>
>>> On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
>>> mailto:bstillw...@photobucket.com>> wrote:
>>>
>>> So far I haven't seen much of a change.  It's still working through
>>> removing the bucket that reached 1.5 million objects though (my
>>> guess is that'll take a few more days), so I believe that might have
>>> something to do with it.
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
>>> mailto:mark.nel...@inktank.com>> wrote:
>>>
>>> Bryan,
>>>
>>> Good explanation.  How's performance now that you've spread the
>>> load over multiple buckets?
>>>
>>> Mark
>>>
>>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>>>
>>> Bill,
>>>
>>> I've run into a similar issue with objects averaging
>>> ~100KiB.  The
>>> explanation I received on IRC is that there are scaling
>>> issues if you're
>>> uploading them all to the same bucket because the index
>>> isn't sharded.
>>>The recommended solution is to spread the objects out to
>>> a lot of
>>> buckets.  However, that ran me into another issue once I hit
>>> 1000
>>> buckets which is a per user limit.  I switched the limit to
>>> be unlimited
>>> with this command:
>>>
>>> radosgw-admin user modify --uid=your_username --max-buckets=0
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
>>> mailto:bill.o...@gmail.com>
>>> >>
>>>
>>> wrote:
>>>
>>>  I'm testing ceph for storing a very large number of
>>> small files.
>>>I'm seeing some performance issues and would like to
>>> see if anyone
>>>  could offer any insight as to what I could do to
>>> correct this.
>>>
>>>  Some numbers:
>>>
>>>  Uploaded 184111 files, with an average file size of
>>> 5KB, using
>>>  10 separate servers to upload the request using Python
>>> and the
>>>  cloudfiles module.  I stopped uploading after 53
>>> minutes, which
>>>  seems to average 5.7 files per second per node.
>>>
>>>
>>>  My storage cluster consists of 21 OSD's across 7
>>> servers, with their
>>>  journals written to SSD drives.  I've done a default
>>> installation,
>>>  using ceph-deploy with the dumpling release.
>>>
>>>  I'm using statsd to monitor the performance, and what's
>>> interesting
>>>  is when I start with an empty bucket, performance is
>>> amazing, with
>>>  average response times of 20-50ms.  However as time
>>> goes on, the
>>>  response times go in to the hundreds, and the average
>>> number of
>>>  uploads per second drops.
>>>
>>>  I've installed radosgw on all 7 ceph servers.  I've
>>> tested using a
>>>  load balancer to distribute the api calls, as well as
>>> pointing the
>>>  10 worker servers to a single instance.  I've not seen
>>> a real
>>>  different in performance with t

Re: [ceph-users] Performance issues with small files

[Mark Nelson]

On 09/05/2013 09:19 AM, Bill Omer wrote:

Thats correct.  We created 65k buckets, using two hex characters as the
naming convention, then stored the files in each container based on
their first two characters in the file name.  The end result was 20-50
files per bucket.  Once all of the buckets were created and files were
being loaded, we still observed an increase in latency overtime.

Is there a way to disable indexing?  Or are there other settings you can
suggest to attempt to speed this process up?


There's been some talk recently about indexless buckets, but I don't 
think it's possible right now.  Yehuda can probably talk about it.


If you remove objects from the bucket so it is empty does it speed up 
again?  Anything you can tell us about when and how it slows down would 
be very useful!


Mark




On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote:

Just for clarification, distributing objects over lots of buckets
isn't helping improve small object performance?

The degradation over time is similar to something I've seen in the
past, with higher numbers of seeks on the underlying OSD device over
time.  Is it always (temporarily) resolved writing to a new empty
bucket?

Mark


On 09/04/2013 02:45 PM, Bill Omer wrote:

We've actually done the same thing, creating 65k buckets and storing
20-50 objects in each.  No change really, not noticeable anyway


On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
mailto:bstillw...@photobucket.com>
>> wrote:

 So far I haven't seen much of a change.  It's still working
through
 removing the bucket that reached 1.5 million objects though (my
 guess is that'll take a few more days), so I believe that
might have
 something to do with it.

 Bryan


 On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
 mailto:mark.nel...@inktank.com>
>> wrote:

 Bryan,

 Good explanation.  How's performance now that you've
spread the
 load over multiple buckets?

 Mark

 On 09/04/2013 12:39 PM, Bryan Stillwell wrote:

 Bill,

 I've run into a similar issue with objects averaging
 ~100KiB.  The
 explanation I received on IRC is that there are scaling
 issues if you're
 uploading them all to the same bucket because the index
 isn't sharded.
The recommended solution is to spread the
objects out to
 a lot of
 buckets.  However, that ran me into another issue
once I hit
 1000
 buckets which is a per user limit.  I switched the
limit to
 be unlimited
 with this command:

 radosgw-admin user modify --uid=your_username
--max-buckets=0

 Bryan


 On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
 mailto:bill.o...@gmail.com>
>
  

Re: [ceph-users] Performance issues with small files

[Bill Omer]

Thats correct.  We created 65k buckets, using two hex characters as the
naming convention, then stored the files in each container based on their
first two characters in the file name.  The end result was 20-50 files per
bucket.  Once all of the buckets were created and files were being loaded,
we still observed an increase in latency overtime.

Is there a way to disable indexing?  Or are there other settings you can
suggest to attempt to speed this process up?


On Wed, Sep 4, 2013 at 5:21 PM, Mark Nelson  wrote:

> Just for clarification, distributing objects over lots of buckets isn't
> helping improve small object performance?
>
> The degradation over time is similar to something I've seen in the past,
> with higher numbers of seeks on the underlying OSD device over time.  Is it
> always (temporarily) resolved writing to a new empty bucket?
>
> Mark
>
>
> On 09/04/2013 02:45 PM, Bill Omer wrote:
>
>> We've actually done the same thing, creating 65k buckets and storing
>> 20-50 objects in each.  No change really, not noticeable anyway
>>
>>
>> On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
>> > >
>> wrote:
>>
>> So far I haven't seen much of a change.  It's still working through
>> removing the bucket that reached 1.5 million objects though (my
>> guess is that'll take a few more days), so I believe that might have
>> something to do with it.
>>
>> Bryan
>>
>>
>> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
>> > >
>> wrote:
>>
>> Bryan,
>>
>> Good explanation.  How's performance now that you've spread the
>> load over multiple buckets?
>>
>> Mark
>>
>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>>
>> Bill,
>>
>> I've run into a similar issue with objects averaging
>> ~100KiB.  The
>> explanation I received on IRC is that there are scaling
>> issues if you're
>> uploading them all to the same bucket because the index
>> isn't sharded.
>>The recommended solution is to spread the objects out to
>> a lot of
>> buckets.  However, that ran me into another issue once I hit
>> 1000
>> buckets which is a per user limit.  I switched the limit to
>> be unlimited
>> with this command:
>>
>> radosgw-admin user modify --uid=your_username --max-buckets=0
>>
>> Bryan
>>
>>
>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
>> mailto:bill.o...@gmail.com>
>> >>
>>
>> wrote:
>>
>>  I'm testing ceph for storing a very large number of
>> small files.
>>I'm seeing some performance issues and would like to
>> see if anyone
>>  could offer any insight as to what I could do to
>> correct this.
>>
>>  Some numbers:
>>
>>  Uploaded 184111 files, with an average file size of
>> 5KB, using
>>  10 separate servers to upload the request using Python
>> and the
>>  cloudfiles module.  I stopped uploading after 53
>> minutes, which
>>  seems to average 5.7 files per second per node.
>>
>>
>>  My storage cluster consists of 21 OSD's across 7
>> servers, with their
>>  journals written to SSD drives.  I've done a default
>> installation,
>>  using ceph-deploy with the dumpling release.
>>
>>  I'm using statsd to monitor the performance, and what's
>> interesting
>>  is when I start with an empty bucket, performance is
>> amazing, with
>>  average response times of 20-50ms.  However as time
>> goes on, the
>>  response times go in to the hundreds, and the average
>> number of
>>  uploads per second drops.
>>
>>  I've installed radosgw on all 7 ceph servers.  I've
>> tested using a
>>  load balancer to distribute the api calls, as well as
>> pointing the
>>  10 worker servers to a single instance.  I've not seen
>> a real
>>  different in performance with this either.
>>
>>
>>  Each of the ceph servers are 16 core Xeon 2.53GHz with
>> 72GB of ram,
>>  OCZ Vertex4 SSD drives for the journals and Seagate
>> Barracuda ES2
>>  drives for storage.
>>
>>
>>  Any help would be greatly appreciated.
>>
>>
>>  __**___
>>
>>  ceph-users mailing list
>> 

Re: [ceph-users] Performance issues with small files

[Mark Nelson]

Just for clarification, distributing objects over lots of buckets isn't 
helping improve small object performance?


The degradation over time is similar to something I've seen in the past, 
with higher numbers of seeks on the underlying OSD device over time.  Is 
it always (temporarily) resolved writing to a new empty bucket?


Mark

On 09/04/2013 02:45 PM, Bill Omer wrote:

We've actually done the same thing, creating 65k buckets and storing
20-50 objects in each.  No change really, not noticeable anyway


On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
mailto:bstillw...@photobucket.com>> wrote:

So far I haven't seen much of a change.  It's still working through
removing the bucket that reached 1.5 million objects though (my
guess is that'll take a few more days), so I believe that might have
something to do with it.

Bryan


On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson
mailto:mark.nel...@inktank.com>> wrote:

Bryan,

Good explanation.  How's performance now that you've spread the
load over multiple buckets?

Mark

On 09/04/2013 12:39 PM, Bryan Stillwell wrote:

Bill,

I've run into a similar issue with objects averaging
~100KiB.  The
explanation I received on IRC is that there are scaling
issues if you're
uploading them all to the same bucket because the index
isn't sharded.
   The recommended solution is to spread the objects out to
a lot of
buckets.  However, that ran me into another issue once I hit
1000
buckets which is a per user limit.  I switched the limit to
be unlimited
with this command:

radosgw-admin user modify --uid=your_username --max-buckets=0

Bryan


On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer
mailto:bill.o...@gmail.com>
>>
wrote:

 I'm testing ceph for storing a very large number of
small files.
   I'm seeing some performance issues and would like to
see if anyone
 could offer any insight as to what I could do to
correct this.

 Some numbers:

 Uploaded 184111 files, with an average file size of
5KB, using
 10 separate servers to upload the request using Python
and the
 cloudfiles module.  I stopped uploading after 53
minutes, which
 seems to average 5.7 files per second per node.


 My storage cluster consists of 21 OSD's across 7
servers, with their
 journals written to SSD drives.  I've done a default
installation,
 using ceph-deploy with the dumpling release.

 I'm using statsd to monitor the performance, and what's
interesting
 is when I start with an empty bucket, performance is
amazing, with
 average response times of 20-50ms.  However as time
goes on, the
 response times go in to the hundreds, and the average
number of
 uploads per second drops.

 I've installed radosgw on all 7 ceph servers.  I've
tested using a
 load balancer to distribute the api calls, as well as
pointing the
 10 worker servers to a single instance.  I've not seen
a real
 different in performance with this either.


 Each of the ceph servers are 16 core Xeon 2.53GHz with
72GB of ram,
 OCZ Vertex4 SSD drives for the journals and Seagate
Barracuda ES2
 drives for storage.


 Any help would be greatly appreciated.


 _
 ceph-users mailing list
ceph-users@lists.ceph.com 
>
http://lists.ceph.com/__listinfo.cgi/ceph-users-ceph.__com





--
Photobucket 

*Bryan Stillwell*
SENIOR SYSTEM ADMINISTRATOR

E: bstillw...@photobucket.com

>
O: 303.228.5109 
M: 970.310.6085 

Facebook >  Twitter

Re: [ceph-users] Performance issues with small files

[Bill Omer]

We've actually done the same thing, creating 65k buckets and storing 20-50
objects in each.  No change really, not noticeable anyway


On Wed, Sep 4, 2013 at 2:43 PM, Bryan Stillwell
wrote:

> So far I haven't seen much of a change.  It's still working through
> removing the bucket that reached 1.5 million objects though (my guess is
> that'll take a few more days), so I believe that might have something to do
> with it.
>
> Bryan
>
>
> On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson wrote:
>
>> Bryan,
>>
>> Good explanation.  How's performance now that you've spread the load over
>> multiple buckets?
>>
>> Mark
>>
>> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>>
>>> Bill,
>>>
>>> I've run into a similar issue with objects averaging ~100KiB.  The
>>> explanation I received on IRC is that there are scaling issues if you're
>>> uploading them all to the same bucket because the index isn't sharded.
>>>   The recommended solution is to spread the objects out to a lot of
>>> buckets.  However, that ran me into another issue once I hit 1000
>>> buckets which is a per user limit.  I switched the limit to be unlimited
>>> with this command:
>>>
>>> radosgw-admin user modify --uid=your_username --max-buckets=0
>>>
>>> Bryan
>>>
>>>
>>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer >> > wrote:
>>>
>>> I'm testing ceph for storing a very large number of small files.
>>>   I'm seeing some performance issues and would like to see if anyone
>>> could offer any insight as to what I could do to correct this.
>>>
>>> Some numbers:
>>>
>>> Uploaded 184111 files, with an average file size of 5KB, using
>>> 10 separate servers to upload the request using Python and the
>>> cloudfiles module.  I stopped uploading after 53 minutes, which
>>> seems to average 5.7 files per second per node.
>>>
>>>
>>> My storage cluster consists of 21 OSD's across 7 servers, with their
>>> journals written to SSD drives.  I've done a default installation,
>>> using ceph-deploy with the dumpling release.
>>>
>>> I'm using statsd to monitor the performance, and what's interesting
>>> is when I start with an empty bucket, performance is amazing, with
>>> average response times of 20-50ms.  However as time goes on, the
>>> response times go in to the hundreds, and the average number of
>>> uploads per second drops.
>>>
>>> I've installed radosgw on all 7 ceph servers.  I've tested using a
>>> load balancer to distribute the api calls, as well as pointing the
>>> 10 worker servers to a single instance.  I've not seen a real
>>> different in performance with this either.
>>>
>>>
>>> Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram,
>>> OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
>>> drives for storage.
>>>
>>>
>>> Any help would be greatly appreciated.
>>>
>>>
>>> __**_
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> >> >
>>> 
>>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>>
>>>
>>>
>>>
>>> --
>>> Photobucket 
>>>
>>> *Bryan Stillwell*
>>> SENIOR SYSTEM ADMINISTRATOR
>>>
>>> E: bstillw...@photobucket.com 
>>> >> >
>>> O: 303.228.5109
>>> M: 970.310.6085
>>>
>>> Facebook 
>>> >
>>>  Twitter
>>> >
>>>Photobucket
>>> 
>>> >
>>>
>>>
>>>
>>>
>>> __**_
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>>
>>>
>> __**_
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>
>
>
>
> --
> [image: Photobucket] 
>
> *Bryan Stillwell*
>
> SENIOR SYSTEM ADMINISTRATOR
>
> E: bstillw...@photobucket.com
>  O: 303.228.5109
> M: 970.310.6085
>
> [image: Facebook]  [image: 
> Twitter][image:
> Photobucket] 
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Bryan Stillwell]

So far I haven't seen much of a change.  It's still working through
removing the bucket that reached 1.5 million objects though (my guess is
that'll take a few more days), so I believe that might have something to do
with it.

Bryan


On Wed, Sep 4, 2013 at 12:14 PM, Mark Nelson wrote:

> Bryan,
>
> Good explanation.  How's performance now that you've spread the load over
> multiple buckets?
>
> Mark
>
> On 09/04/2013 12:39 PM, Bryan Stillwell wrote:
>
>> Bill,
>>
>> I've run into a similar issue with objects averaging ~100KiB.  The
>> explanation I received on IRC is that there are scaling issues if you're
>> uploading them all to the same bucket because the index isn't sharded.
>>   The recommended solution is to spread the objects out to a lot of
>> buckets.  However, that ran me into another issue once I hit 1000
>> buckets which is a per user limit.  I switched the limit to be unlimited
>> with this command:
>>
>> radosgw-admin user modify --uid=your_username --max-buckets=0
>>
>> Bryan
>>
>>
>> On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer > > wrote:
>>
>> I'm testing ceph for storing a very large number of small files.
>>   I'm seeing some performance issues and would like to see if anyone
>> could offer any insight as to what I could do to correct this.
>>
>> Some numbers:
>>
>> Uploaded 184111 files, with an average file size of 5KB, using
>> 10 separate servers to upload the request using Python and the
>> cloudfiles module.  I stopped uploading after 53 minutes, which
>> seems to average 5.7 files per second per node.
>>
>>
>> My storage cluster consists of 21 OSD's across 7 servers, with their
>> journals written to SSD drives.  I've done a default installation,
>> using ceph-deploy with the dumpling release.
>>
>> I'm using statsd to monitor the performance, and what's interesting
>> is when I start with an empty bucket, performance is amazing, with
>> average response times of 20-50ms.  However as time goes on, the
>> response times go in to the hundreds, and the average number of
>> uploads per second drops.
>>
>> I've installed radosgw on all 7 ceph servers.  I've tested using a
>> load balancer to distribute the api calls, as well as pointing the
>> 10 worker servers to a single instance.  I've not seen a real
>> different in performance with this either.
>>
>>
>> Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram,
>> OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
>> drives for storage.
>>
>>
>> Any help would be greatly appreciated.
>>
>>
>> __**_
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> > >
>> 
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>
>>
>>
>>
>> --
>> Photobucket 
>>
>> *Bryan Stillwell*
>> SENIOR SYSTEM ADMINISTRATOR
>>
>> E: bstillw...@photobucket.com 
>> > >
>> O: 303.228.5109
>> M: 970.310.6085
>>
>> Facebook 
>> >
>>  Twitter
>> >
>>Photobucket
>> 
>> >
>>
>>
>>
>>
>> __**_
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>>
>>
> __**_
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/**listinfo.cgi/ceph-users-ceph.**com
>



-- 
[image: Photobucket] 

*Bryan Stillwell*
SENIOR SYSTEM ADMINISTRATOR

E: bstillw...@photobucket.com
O: 303.228.5109
M: 970.310.6085

[image: Facebook] [image:
Twitter][image:
Photobucket] 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Mark Nelson]

Bryan,

Good explanation.  How's performance now that you've spread the load 
over multiple buckets?


Mark

On 09/04/2013 12:39 PM, Bryan Stillwell wrote:

Bill,

I've run into a similar issue with objects averaging ~100KiB.  The
explanation I received on IRC is that there are scaling issues if you're
uploading them all to the same bucket because the index isn't sharded.
  The recommended solution is to spread the objects out to a lot of
buckets.  However, that ran me into another issue once I hit 1000
buckets which is a per user limit.  I switched the limit to be unlimited
with this command:

radosgw-admin user modify --uid=your_username --max-buckets=0

Bryan


On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer mailto:bill.o...@gmail.com>> wrote:

I'm testing ceph for storing a very large number of small files.
  I'm seeing some performance issues and would like to see if anyone
could offer any insight as to what I could do to correct this.

Some numbers:

Uploaded 184111 files, with an average file size of 5KB, using
10 separate servers to upload the request using Python and the
cloudfiles module.  I stopped uploading after 53 minutes, which
seems to average 5.7 files per second per node.


My storage cluster consists of 21 OSD's across 7 servers, with their
journals written to SSD drives.  I've done a default installation,
using ceph-deploy with the dumpling release.

I'm using statsd to monitor the performance, and what's interesting
is when I start with an empty bucket, performance is amazing, with
average response times of 20-50ms.  However as time goes on, the
response times go in to the hundreds, and the average number of
uploads per second drops.

I've installed radosgw on all 7 ceph servers.  I've tested using a
load balancer to distribute the api calls, as well as pointing the
10 worker servers to a single instance.  I've not seen a real
different in performance with this either.


Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram,
OCZ Vertex4 SSD drives for the journals and Seagate Barracuda ES2
drives for storage.


Any help would be greatly appreciated.


___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




--
Photobucket 

*Bryan Stillwell*
SENIOR SYSTEM ADMINISTRATOR

E: bstillw...@photobucket.com 
O: 303.228.5109
M: 970.310.6085

Facebook Twitter
  Photobucket





___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Performance issues with small files

[Bryan Stillwell]

Bill,

I've run into a similar issue with objects averaging ~100KiB.  The
explanation I received on IRC is that there are scaling issues if you're
uploading them all to the same bucket because the index isn't sharded.  The
recommended solution is to spread the objects out to a lot of buckets.
 However, that ran me into another issue once I hit 1000 buckets which is a
per user limit.  I switched the limit to be unlimited with this command:

radosgw-admin user modify --uid=your_username --max-buckets=0

Bryan


On Wed, Sep 4, 2013 at 11:27 AM, Bill Omer  wrote:

> I'm testing ceph for storing a very large number of small files.  I'm
> seeing some performance issues and would like to see if anyone could offer
> any insight as to what I could do to correct this.
>
> Some numbers:
>
> Uploaded 184111 files, with an average file size of 5KB, using
> 10 separate servers to upload the request using Python and the cloudfiles
> module.  I stopped uploading after 53 minutes, which seems to average 5.7
> files per second per node.
>
>
> My storage cluster consists of 21 OSD's across 7 servers, with their
> journals written to SSD drives.  I've done a default installation, using
> ceph-deploy with the dumpling release.
>
> I'm using statsd to monitor the performance, and what's interesting is
> when I start with an empty bucket, performance is amazing, with average
> response times of 20-50ms.  However as time goes on, the response times go
> in to the hundreds, and the average number of uploads per second drops.
>
> I've installed radosgw on all 7 ceph servers.  I've tested using a load
> balancer to distribute the api calls, as well as pointing the 10 worker
> servers to a single instance.  I've not seen a real different
> in performance with this either.
>
>
> Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram, OCZ
> Vertex4 SSD drives for the journals and Seagate Barracuda ES2 drives for
> storage.
>
>
> Any help would be greatly appreciated.
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>


-- 
[image: Photobucket] 

*Bryan Stillwell*
SENIOR SYSTEM ADMINISTRATOR

E: bstillw...@photobucket.com
O: 303.228.5109
M: 970.310.6085

[image: Facebook] [image:
Twitter][image:
Photobucket] 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Performance issues with small files

[Bill Omer]

I'm testing ceph for storing a very large number of small files.  I'm
seeing some performance issues and would like to see if anyone could offer
any insight as to what I could do to correct this.

Some numbers:

Uploaded 184111 files, with an average file size of 5KB, using
10 separate servers to upload the request using Python and the cloudfiles
module.  I stopped uploading after 53 minutes, which seems to average 5.7
files per second per node.


My storage cluster consists of 21 OSD's across 7 servers, with their
journals written to SSD drives.  I've done a default installation, using
ceph-deploy with the dumpling release.

I'm using statsd to monitor the performance, and what's interesting is when
I start with an empty bucket, performance is amazing, with average response
times of 20-50ms.  However as time goes on, the response times go in to the
hundreds, and the average number of uploads per second drops.

I've installed radosgw on all 7 ceph servers.  I've tested using a load
balancer to distribute the api calls, as well as pointing the 10 worker
servers to a single instance.  I've not seen a real different
in performance with this either.


Each of the ceph servers are 16 core Xeon 2.53GHz with 72GB of ram, OCZ
Vertex4 SSD drives for the journals and Seagate Barracuda ES2 drives for
storage.


Any help would be greatly appreciated.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com