Re: [ceph-users] ceph-mon fails to start on rasberry pi (raspbian 8.0)

On 12/15/2017 07:03 PM, Andrew Knapp wrote: Has anyone else tried this and had similar problems? Any advice on how to proceed or work around this issue? The daemon's log, somewhere in /var/log/ceph/ceph-mon..log, should have more info. Upload that somewhere and we'll take a look. -Joao

Re: [ceph-users] Multiple independent rgw instances on same cluster

I have 3 realms running in 2 datacenters. 4 realms in total since 2 of them are running with multi-site between the datacenters. We have RGW's for each of the realms that only run for 1 realm each. Be careful on how many PGs you create for everything as there are a lot of pools required for this.

[ceph-users] Multiple independent rgw instances on same cluster

I'm just looking for a sanity check on this... I want two separate rados gateways on the same (luminous) cluster to be completely independent - separate pools, users, data, no sync. After some experimentation it seems like the appropriate thing is to set them up using separate realms. Does

Re: [ceph-users] RGW Logging pool

On Fri, Dec 15, 2017 at 05:21:37PM +, David Turner wrote: > We're trying to build an auditing system for when a user key pair performs > an operation on a bucket (put, delete, creating a bucket, etc) and so far > were only able to find this information in the level 10 debug logging in > the

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

James, You can set these values in ceph.conf. [global] ... osd pool default size = 3 osd pool default min size = 2 ... New pools that are created will use those values. If you run a "ceph -s" and look at the "usage" line, it shows how much space is: 1 used, 2 available, 3 total. ie.

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

In conjunction with increasing the pool size to 3, also increase the pool min_size to 2. `ceph df` and `ceph osd df` will eventually show the full size in use in your cluster. In particular the output of `ceph df` with available size in a pool takes into account the pools replication size.

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

if you have a global setting in ceph.conf it will only affect the creation of new pools. i reccomend using the default size:3 + min_size:2 also check your pools that you have min_size=2 kind regards Ronny Aasen On 15.12.2017 23:00, James Okken wrote: This whole effort went extremely well,

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

This whole effort went extremely well, thanks to Cary, and Im not used to that with CEPH so far. (And openstack ever) Thank you Cary. Ive upped the replication factor and now I see "replicated size 3" in each of my pools. Is this the only place to check replication level? Is there a Global

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

James, Those errors are normal. Ceph creates the missing files. You can check "/var/lib/ceph/osd/ceph-6", before and after you run those commands to see what files are added there. Make sure you get the replication factor set. Cary -Dynamic On Fri, Dec 15, 2017 at 6:11 PM, James Okken

Re: [ceph-users] RGW Logging pool

we never managed to make it work, but I guess the "RGW metadata search" [c|sh]ould have been "the official solution"... - http://ceph.com/geen-categorie/rgw-metadata-search/ - https://marc.info/?l=ceph-devel=149152531005431=2 - http://ceph.com/rgw/new-luminous-rgw-metadata-search/ there was

[ceph-users] ceph-mon fails to start on rasberry pi (raspbian 8.0)

I recently purchased 3 raspberry pi nodes to create a small storage cluster to test with at my home. I found a couple of procedures on setting this up so it appears folks have successfully done this ( https://www.linkedin.com/pulse/ceph-raspberry-pi-rahul-vijayan/). I am running Raspbian

Re: [ceph-users] add hard drives to 3 CEPH servers (3 server cluster)

Thanks again Cary, Yes, once all the backfilling was done I was back to a Healthy cluster. I moved on to the same steps for the next server in the cluster, it is backfilling now. Once that is done I will do the last server in the cluster, and then I think I am done! Just checking on one thing.

[ceph-users] RGW Logging pool

We're trying to build an auditing system for when a user key pair performs an operation on a bucket (put, delete, creating a bucket, etc) and so far were only able to find this information in the level 10 debug logging in the rgw systems logs. We noticed that our rgw log pool has been growing

Re: [ceph-users] Cache tier unexpected behavior: promote on lock

On Thu, Dec 14, 2017 at 9:11 AM, Захаров Алексей wrote: > Hi, Gregory, > Thank you for your answer! > > Is there a way to not promote on "locking", when not using EC pools? > Is it possible to make this configurable? > > We don't use EC pool. So, for us this meachanism is

Re: [ceph-users] Snap trim queue length issues

On Fri, 15 Dec 2017, Piotr Dałek wrote: > On 17-12-14 05:31 PM, David Turner wrote: > > I've tracked this in a much more manual way.  I would grab a random subset > > [..] > > > > This was all on a Hammer cluster.  The changes to the snap trimming queues > > going into the main osd thread made it

Re: [ceph-users] How to raise priority for a pg repair

The method I've used in the past to initiate a repair quickly was to set osd_max_deep_scrubs to 0 across the cluster and then set it to 2 on only the osds that were involved in the pg. Alternatively you could just increase that setting to 3 or more on only those osds involved in the pg to trigger

Re: [ceph-users] S3 objects deleted but storage doesn't free space

You can check to see how backed up your GC is with `radosgw-admin gc list | wc -l`. In one of our clusters, we realized that early testing and re-configuring of the realm completely messed up the GC and that realm had never actually deleted in an object in all the time it had been running in

[ceph-users] How to raise priority for a pg repair

We have some scrub errors on our cluster. A ceph pg repair x.xxx is take in account only after hours. It seems to be linked to deep-scrubs which are running at the same time. It 's look like it has to wait for a slot before launching the repair. I have then two question : is it possible to launch

Re: [ceph-users] cephfs mds millions of caps

So, On Fri, Dec 15, 2017 at 10:58 AM, Yan, Zheng wrote: > > 300k are ready quite a lot. opening them requires long time. does you > mail server really open so many files? Yes, probably. It's a commercial solution. A few thousand domains, dozens of thousands of users and god

Re: [ceph-users] cephfs mds millions of caps

Thanks On Fri, Dec 15, 2017 at 10:46 AM, Yan, Zheng wrote: > recent > version kernel client and ceph-fuse should trim they cache > aggressively when mds recovers. > So the bug (not sure if I can call it a bug) is already fixed in newer kernel? Can I just update the kernel

Re: [ceph-users] cephfs mds millions of caps

On Fri, Dec 15, 2017 at 8:46 PM, Yan, Zheng wrote: > On Fri, Dec 15, 2017 at 6:54 PM, Webert de Souza Lima > wrote: >> Hello, Mr. Yan >> >> On Thu, Dec 14, 2017 at 11:36 PM, Yan, Zheng wrote: >>> >>> >>> The client hold so many

Re: [ceph-users] cephfs mds millions of caps

On Fri, Dec 15, 2017 at 6:54 PM, Webert de Souza Lima wrote: > Hello, Mr. Yan > > On Thu, Dec 14, 2017 at 11:36 PM, Yan, Zheng wrote: >> >> >> The client hold so many capabilities because kernel keeps lots of >> inodes in its cache. Kernel does not trim

Re: [ceph-users] Odd object blocking IO on PG

For those following along at home, already done: http://tracker.ceph.com/issues/22440 On Fri, Dec 15, 2017 at 1:57 AM Brad Hubbard wrote: > On Wed, Dec 13, 2017 at 11:39 PM, Nick Fisk wrote: > > Boom!!

Re: [ceph-users] Any RGW admin frontends?

Hi Dan, On 12/15/2017 10:13 AM, Dan van der Ster wrote: > As we are starting to ramp up our internal rgw service, I am wondering > if someone already developed some "open source" high-level admin tools > for rgw. On the one hand, we're looking for a web UI for users to create > and see their

Re: [ceph-users] Ceph metric exporter HTTP Error 500

Hi, On 12/15/2017 11:53 AM, Falk Mueller-Braun wrote: > since we upgraded to Luminous (12.2.2), we use the internal Ceph > exporter for getting the Ceph metrics to Prometheus. At random times we > get a Internal Server Error from the Ceph exporter, with python having a > key error with some

[ceph-users] Latency metrics for mons, osd applies and commits

Hello, since using the internal ceph exporter (after having upgrade to Ceph Luminous [12.2.2]), I can't find any latency related metrics. Before, i used the the latency of the monitors, the osd apply and commit latency, which were pretty useful for monitoring. Is there any possibility to get

Re: [ceph-users] cephfs mds millions of caps

Hello, Mr. Yan On Thu, Dec 14, 2017 at 11:36 PM, Yan, Zheng wrote: > > The client hold so many capabilities because kernel keeps lots of > inodes in its cache. Kernel does not trim inodes by itself if it has > no memory pressure. It seems you have set mds_cache_size config to

[ceph-users] Ceph metric exporter HTTP Error 500

Hello, since we upgraded to Luminous (12.2.2), we use the internal Ceph exporter for getting the Ceph metrics to Prometheus. At random times we get a Internal Server Error from the Ceph exporter, with python having a key error with some random metric. Often it is "pg_*". Here is an example of

Re: [ceph-users] Problems understanding 'ceph features' output

Thanks for your answer Actually I have the very same configuration on the three "client hosts": on each of them I simply mapped a single rbd volume ... Cheers, Massimo 2017-12-15 11:10 GMT+01:00 Burkhard Linke < burkhard.li...@computational.bio.uni-giessen.de>: > Hi, > > > On 12/15/2017 10:56

Re: [ceph-users] Problems understanding 'ceph features' output

Hi, On 12/15/2017 10:56 AM, Massimo Sgaravatto wrote: Hi I tried the jewel --> luminous update on a small testbed composed by: - 3 mon + mgr nodes - 3 osd nodes (4 OSDs per each of this node) - 3 clients (each client maps a single volume) *snipsnap* [*]     "client": {         "group":

Re: [ceph-users] cephfs miss data for 15s when master mds rebooting

On Fri, Dec 15, 2017 at 1:45 AM, 13605702...@163.com <13605702...@163.com> wrote: > hi > > i used 3 nodes to deploy mds (each node also has mon on it) > > my config: > [mds.ceph-node-10-101-4-17] > mds_standby_replay = true > mds_standby_for_rank = 0 > > [mds.ceph-node-10-101-4-21] >

[ceph-users] Problems understanding 'ceph features' output

Hi I tried the jewel --> luminous update on a small testbed composed by: - 3 mon + mgr nodes - 3 osd nodes (4 OSDs per each of this node) - 3 clients (each client maps a single volume) In short: - I updated the 3 mons - I deployed mgr on the 3 mon hosts - I updated the 3 osd nodes - I updated

[ceph-users] Any RGW admin frontends?

Hi all, As we are starting to ramp up our internal rgw service, I am wondering if someone already developed some "open source" high-level admin tools for rgw. On the one hand, we're looking for a web UI for users to create and see their credentials, quota, usage, and maybe a web bucket browser.

Re: [ceph-users] Snap trim queue length issues

On 17-12-14 05:31 PM, David Turner wrote: I've tracked this in a much more manual way.  I would grab a random subset [..] This was all on a Hammer cluster.  The changes to the snap trimming queues going into the main osd thread made it so that our use case was not viable on Jewel until