[ceph-users] radosgw X-History-Location

2017-05-19 Thread K K 

Hi all!

jewel 10.2.6 release

I try to setup "X-History-Location: Arhive". But those function not work.
Do anybody know plans to add those option to radosgw?
X-Versions-Location working fine.

Thanks all



-- 
Konstantin___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Intel power tuning - 30% throughput performance increase

2017-05-19 Thread Brad Hubbard 
It just came to my attention that Intel has advised Red Hat never to
lock in C0 as it may affect the life expectancy of server components
such as fans and the CPUs themselves.

FYI, YMMV.

On Fri, May 19, 2017 at 5:53 PM, xiaoguang fan
 wrote:
> I have done a test about close c-stat , but performance didn't increase
>
>
>
> 2017-05-19 15:23 GMT+08:00 Xiaoxi Chen :
>>
>> would be better to document it first on "Known system-wise tuning
>> knobs" in the doc?
>>
>>
>> 2017-05-05 8:28 GMT+08:00 Brad Hubbard :
>> > On Thu, May 4, 2017 at 10:58 AM, Haomai Wang  wrote:
>> >> refer to https://github.com/ceph/ceph/pull/5013
>> >
>> > How about we issue a warning about possible performance implications
>> > if we detect this is not set to 1 *or* 0 at startup?
>> >
>> >>
>> >> On Thu, May 4, 2017 at 7:56 AM, Brad Hubbard 
>> >> wrote:
>> >>> +ceph-devel to get input on whether we want/need to check the value of
>> >>> /dev/cpu_dma_latency (platform dependant) at startup and issue a
>> >>> warning, or whether documenting this would suffice?
>> >>>
>> >>> Any doc contribution would be welcomed.
>> >>>
>> >>> On Wed, May 3, 2017 at 7:18 PM, Blair Bethwaite
>> >>>  wrote:
>>  On 3 May 2017 at 19:07, Dan van der Ster  wrote:
>> > Whether cpu_dma_latency should be 0 or 1, I'm not sure yet. I assume
>> > your 30% boost was when going from throughput-performance to
>> > dma_latency=0, right? I'm trying to understand what is the
>> > incremental
>> > improvement from 1 to 0.
>> 
>>  Probably minimal given that represents a state transition latency
>>  taking only 1us. Presumably the main issue is when the CPU can drop
>>  into the lower states and the compounding impact of that over time. I
>>  will do some simple characterisation of that over the next couple of
>>  weeks and report back...
>> 
>>  --
>>  Cheers,
>>  ~Blairo
>>  ___
>>  ceph-users mailing list
>>  ceph-users@lists.ceph.com
>>  http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Cheers,
>> >>> Brad
>> >>> ___
>> >>> ceph-users mailing list
>> >>> ceph-users@lists.ceph.com
>> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> >
>> >
>> >
>> > --
>> > Cheers,
>> > Brad
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majord...@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majord...@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>



-- 
Cheers,
Brad
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] RGW lifecycle not expiring objects

2017-05-19 Thread Graham Allan 
I've been having a hard time getting the s3 object lifecycle to do 
anything here. I was able to set a lifecycle on a test bucket. As others 
also seem to have found, I do get an EACCES error on setting the 
lifecycle, but it does however get stored:



% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api 
get-bucket-lifecycle-configuration --bucket=testgta
{
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"Expiration": {
"Days": 3
},
"ID": "test"
}
]
}


but many days later I have yet to see any object actually get expired. 
There are some hints in the rgw log that the expiry thread does run 
periodically:



2017-05-19 03:49:03.281347 7f74f1134700  2 
RGWDataChangesLog::ChangesRenewThread: start
2017-05-19 03:49:16.356022 7f74ef931700  2 object expiration: start
2017-05-19 03:49:16.356036 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.00
2017-05-19 03:49:16.359785 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.01
2017-05-19 03:49:16.364667 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.02
2017-05-19 03:49:16.369636 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.03

...

2017-05-19 03:49:16.803270 7f74ef931700 20 proceeding shard = 
obj_delete_at_hint.000126
2017-05-19 03:49:16.806423 7f74ef931700  2 object expiration: stop


"radosgw-admin lc process" gives me no output unless I enable debug, then:


]# radosgw-admin lc process
2017-05-19 15:28:46.383049 7fedb9ffb700  2 
RGWDataChangesLog::ChangesRenewThread: start
2017-05-19 15:28:46.421806 7feddc240c80 10 Cannot find current period zone 
using local zone
2017-05-19 15:28:46.453431 7feddc240c80  2 all 8 watchers are set, enabling 
cache
2017-05-19 15:28:46.614991 7feddc240c80  2 removed watcher, disabling cache


"radosgw-admin lc list" seems to return "empty" output:


# radosgw-admin lc list
[]


Is there anything obvious that I might be missing?

Graham
--
Graham Allan
Minnesota Supercomputing Institute - g...@umn.edu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Recommended API for Ruby on Ceph Object storage.

2017-05-19 Thread Steve Sether 

I'll try this again without HTML.


Hello,

I'm very new to Ceph, and I'm wondering if anyone using Ceph and Ruby 
has any recommendations on which API to use with Ceph and object 
storage.  It looks like there's some attempts at an API for librados on 
Ruby, but it doesn't look terribly complete.



The Swift API looks promising.  The example uses a gem called 
cloudfiles.  That gem looks rather old, and hasn't seen any updates 
since November 2013.  Is this complete, stable and maintained?



I noticed the SWIFT project itself provides some Ruby hooks:

https://github.com/ruby-openstack/ruby-openstack

Is there any reason I couldn't use this to access the Ceph object 
store?  It looks like ruby-openstack is based on cloudfiles.  If 
ruby-openstack is the prefered choice for ceph object store on Ruby, 
it'd be a good thing to mention it on the website.


Any help would be appreciated.  I'd like to make an informed decision 
going in

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Large OSD omap directories (LevelDBs)

2017-05-19 Thread Robin H. Johnson 
On Fri, May 19, 2017 at 01:55:50PM +, george.vasilaka...@stfc.ac.uk wrote:
> Anyone seen this before who can point me in the right direction to start 
> digging?
Your RGW buckets, how many objects in them, and do they have the index
sharded? 

I know we have some very large & old buckets (10M+ RGW objects in a
single bucket), with correspondingly large OMAPs wherever that bucket
index is living (sufficently large that trying to list the entire thing
online is fruitless). ceph's pgmap status says we have 2G RADOS objects
however, and you're only at 61M RADOS objects.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Trustee & Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] [MDS] scrub_path progress

2017-05-19 Thread Andreas Gerstmayr 
Hi,

is there a way to monitor the progress of the 'ceph daemon mds.0
scrub_path / recursive repair' command? It returns immediately
(without any output), but the MDS is scrubbing in the background.

When I start the same command again, I get a JSON response with return_code: -16
What does this return code indicate, and where is it defined in the code?


Thanks,
Andreas
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Changing replica size of a running pool

2017-05-19 Thread Peter Maloney 
On 05/05/17 21:32, Alejandro Comisario wrote:
> Thanks David!
> Any one ? more thoughts ?
>
> On Wed, May 3, 2017 at 3:38 PM, David Turner  > wrote:
>
> Those are both things that people have done and both work. 
> Neither is optimal, but both options work fine.  The best option
> is to definitely just get a third node now as you aren't going to
> be getting it for additional space from it later.  Your usable
> space between a 2 node size 2 cluster and a 3 node size 3 cluster
> is identical.
>
> If getting a third node is not possible, I would recommend a size
> 2 min_size 2 configuration.  You will block writes if either of
> your nodes or any copy of your data is down, but you will not get
> into an inconsistent state that can happen with min_size of 1 (and
> you can always set the min_size of a pool to 1 on the fly to
> perform maintenance).  If you go with the option to use the
> failure domain of OSDs instead of hosts and have size 3, then a
> single node going down will block writes into your cluster.  The
> only you gain from this is having 3 physical copies of the data
> until you get a third node, but a lot of backfilling when you
> change the crush rule.
>
> A more complex option that I think would be a better solution than
> your 2 options would be to create 2 hosts in your crush map for
> each physical host and split the OSDs in each host evenly between
> them.  That way you can have 2 copies of data in a given node, but
> never all 3 copies.  You have your 3 copies of data and guaranteed
> that not all 3 are on the same host.  Assuming min_size of 2, you
> will still block writes if you restart either node.
>
Smart idea.
Or if you have space, size 4 min_size 2 and then you can still lose a
node. And you might think that's more space, but in a way it isn't... if
you count free space reserved for recovery. If your size 3 double nodes
die, then the other has to recover to size 2 and then it'll use the same
space as the size 4 pool. If the size 4 pool loses a node, it won't be
able to recover... it'll stay size 2, which is what your size 3 pool
would have been after recovery. So it's like it's pre-recovered. But you
probably get a bit more write latency in this setup.

> If modifying the hosts in your crush map doesn't sound daunting,
> then I would recommend going that route... For most people that is
> more complex than they'd like to go and I would say size 2
> min_size 2 would be the way to go until you get a third node.
>  #my2cents
>
> On Wed, May 3, 2017 at 12:41 PM Maximiliano Venesio
> mailto:mass...@nubeliu.com>> wrote:
>
> Guys hi.
>
> I have a Jewel Cluster composed by two storage servers which
> are configured on
> the crush map as different buckets to store data.
>
> I've to configure two new pools on this cluster with the certainty
> that i'll have to add more servers in a short term.
>
> Taking into account that the recommended replication size for
> every
> pool is 3, i'm thinking in two possible scenarios.
>
> 1) Set the replica size in 2 now, and in the future change the
> replica
> size to 3 on a running pool.
> Is that possible? Can i have serious issues with the rebalance
> of the
> pgs, changing the pool size on the fly ?
>
> 2) Set the replica size to 3, and change the ruleset to
> replicate by
> OSD instead of HOST now, and in the future change this rule in the
> ruleset to replicate again by host in a running pool.
> Is that possible? Can i have serious issues with the rebalance
> of the
> pgs, changing the ruleset in a running pool ?
>
> Which do you think is the best option ?
>
>
> Thanks in advanced.
>
>
> Maximiliano Venesio
> Chief Cloud Architect | NUBELIU
> E-mail: massimo@nubeliu.comCell: +54 9 11 3770 1853
> 
> _
> www.nubeliu.com 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com 
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
>
>
>
>
> -- 
> *Alejandro Comisario*
> *CTO | NUBELIU*
> E-mail: alejan...@nubeliu.com Cell: +54
> 9 11 3770 1857
> _
> www.nubeliu.com 
>
>
>
> ___
> ceph-users m

[ceph-users] Recommended API for Ruby on Ceph Object storage.

2017-05-19 Thread Steve Sether 

  
  
Hello,
I'm very new to Ceph, and I'm wondering if anyone using Ceph and
  Ruby has any recommendations on which API to use with Ceph and
  object storage.  It looks like there's some attempts at an API for
  librados on Ruby, but it doesn't look terribly complete.



The Swift API looks promising.  The example uses a gem called
  cloudfiles.  That gem looks rather old, and hasn't seen any
  updates since November 2013.  Is this complete, stable and
  maintained?


I noticed the SWIFT project itself provides some Ruby hooks:
https://github.com/ruby-openstack/ruby-openstack
Is there any reason I couldn't use this to access the Ceph object
  store?  It looks like ruby-openstack is based on cloudfiles.  If
  ruby-openstack is the prefered choice for ceph object store on
  Ruby, it'd be a good thing to mention it on the website.

Any help would be appreciated.  I'd like to make an informed
decision going in.
  

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Large OSD omap directories (LevelDBs)

2017-05-19 Thread george.vasilakakos 
Hello Ceph folk,


We have a Ceph cluster (info at bottom) with some odd omap directory sizes in 
our OSDs.
We're looking at 1439 OSDs were the most common omap sizes are 15-40MB.
However a quick sampling reveals some outliers, looking at around the 100 
largest omaps one can see sizes go to a few hundred MB and then up to single 
digit GB sized ones jumping for the last 10 or so:

14G/var/lib/ceph/osd/ceph-769/current/omap
35G/var/lib/ceph/osd/ceph-1278/current/omap
48G/var/lib/ceph/osd/ceph-899/current/omap
49G/var/lib/ceph/osd/ceph-27/current/omap
57G/var/lib/ceph/osd/ceph-230/current/omap
58G/var/lib/ceph/osd/ceph-343/current/omap
58G/var/lib/ceph/osd/ceph-948/current/omap
60G/var/lib/ceph/osd/ceph-470/current/omap
66G/var/lib/ceph/osd/ceph-348/current/omap
67G/var/lib/ceph/osd/ceph-980/current/omap


Any omap that's 500MB when most are 25 is worrying but 67GB is extremely 
worrying, something doesn't seem right. The 67GB omap has 37k .sst files and 
the oldest file in there is from Feb 21st.

Anyone seen this before who can point me in the right direction to start 
digging?

Cluster info:

ceph version 11.2.0

 monmap e8: 5 mons at {...}
election epoch 1850, quorum 0,1,2,3,4 
ceph-mon1,ceph-mon2,ceph-mon3,ceph-mon4,ceph-mon5
mgr active: ceph-mon4 standbys: ceph-mon3, ceph-mon2, ceph-mon1, 
ceph-mon5
 osdmap e27138: 1439 osds: 1439 up, 1439 in
flags sortbitwise,require_jewel_osds,require_kraken_osds
  pgmap v10911626: 5120 pgs, 21 pools, 1834 TB data, 61535 kobjects
2525 TB used, 5312 TB / 7837 TB avail
5087 active+clean
  17 active+clean+scrubbing
  16 active+clean+scrubbing+deep

Most pools are 64 PGs for RGW metadata. There are 3 pools with 1024 and another 
2 with 512 PGs that hold our data. These are all using EC 8+3, the auxiliary 
ones are replicated.

Our data is put into pools via the libradosstriper interface which adds some 
xattrs to be able to read the data back (stripe count, stripe size, stripe unit 
size, original size (pre-striping)) and the client also puts in a couple of 
checksumming related attributes.



Thanks,

George
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com