[ceph-users] OSD node type/count mixes in the cluster

Wanted to check if anyone has a ceph cluster which has mixed vendor servers both with same disk size i.e. 8TB but different count i.e. Example 10 OSD servers from Dell with 60 Disk per server and other 10 OSD servers from HP with 26 Disk per server. If so does that change any performance

Re: [ceph-users] PG that should not be on undersized+degraded on multi datacenter Ceph cluster

On Thu, Jun 8, 2017 at 11:31 PM, Alejandro Comisario wrote: > Hi Brad. > Taking into consideration the unlikely posibility that someone > realizes what the problem is in this specific case, that would be > higly apreciated. > > I presume that having jewel, if you can

Re: [ceph-users] Cache mode readforward mode will eat your babies?

On Thu, 8 Jun 2017 07:06:04 -0400 Alfredo Deza wrote: > On Thu, Jun 8, 2017 at 3:38 AM, Christian Balzer wrote: > > On Thu, 8 Jun 2017 17:03:15 +1000 Brad Hubbard wrote: > > > >> On Thu, Jun 8, 2017 at 3:47 PM, Christian Balzer wrote: > >> > On Thu, 8 Jun 2017

Re: [ceph-users] rados rm: device or resource busy

I can reproduce this. The key is to look at debug logging on the primary. 2017-06-09 09:30:14.776355 7f9cf26a4700 20 /home/brad/working/src/ceph3/src/cls/lock/cls_lock.cc:247: lock_op 2017-06-09 09:30:14.776359 7f9cf26a4700 20 /home/brad/working/src/ceph3/src/cls/lock/cls_lock.cc:162:

[ceph-users] Living with huge bucket sizes

This has come up quite a few times before, but since I was only working with RBD before I didn't pay too close attention to the conversation. I'm looking for the best way to handle existing clusters that have buckets with a large number of objects (>20 million) in them. The cluster I'm doing

Re: [ceph-users] removing cluster name support

We have an internal management service that works at a higher layer upstream on top of multiple Ceph clusters. It needs a way to differentiate and connect separately to each of those clusters. Presently making that distinction is relatively easy since we create those connections based on

Re: [ceph-users] removing cluster name support

Hi Sage, We do use cluster names, we do not use ceph-deploy or ceph-ansible so in the short term it is not an issue. We have scripts that call cli commands with the --cluster XX parameter, would that still work ? What time frame do you have in mind for removing this ? Cheers /Maged On

Re: [ceph-users] removing cluster name support

Hi Sage, We did at one time run multiple clusters on our OSD nodes and RGW nodes (with Jewel). We accomplished this by putting code in our puppet-ceph module that would create additional systemd units with appropriate CLUSTER=name environment settings for clusters not named ceph. IE, if the

Re: [ceph-users] removing cluster name support

Hi Sage, We need named clusters on the client side. RBD or CephFS clients, or monitoring/admin machines all need to be able to access several clusters. Internally, each cluster is indeed called "ceph", but the clients use distinct names to differentiate their configs/keyrings. Cheers, Dan On

Re: [ceph-users] removing cluster name support

On Thu, 8 Jun 2017, Bassam Tabbara wrote: > Thanks Sage. > > > At CDM yesterday we talked about removing the ability to name your ceph > > clusters. > > Just to be clear, it would still be possible to run multiple ceph > clusters on the same nodes, right? Yes, but you'd need to either (1)

Re: [ceph-users] removing cluster name support

Thanks Sage. > At CDM yesterday we talked about removing the ability to name your ceph > clusters. Just to be clear, it would still be possible to run multiple ceph clusters on the same nodes, right? ___ ceph-users mailing list

[ceph-users] removing cluster name support

At CDM yesterday we talked about removing the ability to name your ceph clusters. There are a number of hurtles that make it difficult to fully get rid of this functionality, not the least of which is that some (many?) deployed clusters make use of it. We decided that the most we can do at

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

Thank jake, can you confirm are you testing this in which ceph version - the out of memory you noticed. There is already a memory leak issue reported in kraken v11.2.0 . which addressed in this tracker .. http://tracker.ceph.com/issues/18924 .. #ceph -v Ok so you are mounting/mapping ceph as a

Re: [ceph-users] RGW lifecycle not expiring objects

Sorry I didn't get to reply until now. The thing is I believe I *do* have a lifecycle configured on at least one bucket. As noted in that issue, I get an error returned when trying to set the lifecycle, but it does appear to get stored: % aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \

Re: [ceph-users] CephFS Snapshot questions

On Thu, Jun 8, 2017 at 3:33 PM, McFarland, Bruce wrote: > John, > > Thanks for your answers. I have a clarification on my questions see below > inline. > > Bruce > > > > From: John Spray > Date: Thursday, June 8, 2017 at 1:45 AM > To: "McFarland,

Re: [ceph-users] rados rm: device or resource busy

Hello, David Turner wrote: : How long have you waited? About a day. : I don't do much with rados objects directly. I usually use RBDs and : cephfs. If you just need to clean things up, you can delete the pool and : recreate it since it looks like it's testing. However this is

Re: [ceph-users] rados rm: device or resource busy

How long have you waited? Watchers of objects in ceph time out after a while and you should be able to delete it. I'm talking around the range of 30 minutes, so it's likely this isn't the problem if you've been wrestling with it long enough to write in about. I don't do much with rados objects

Re: [ceph-users] Changing SSD Landscape

I did stumble across Samsung PM1725/a in both AIC and 2.5” U.2 form factor. AIC starts at 1.6T and goes up to 6.4T, while 2.5” goes from 800G up to 6.4T. The thing that caught my eye with this model is the x8 lanes in AIC, and the 5DWPD over 5 years. No idea on how available it is, or how it

[ceph-users] rados rm: device or resource busy

Hello, I have created a RADOS striped object using $ dd someargs | rados --pool testpool --striper put testfile - and interrupted it in the middle of writing. Now I cannot remove this object: $ rados --pool testpool --striper rm testfile error removing testpool>testfile: (16) Device or

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

Hi Mark / Jayaram, After running the cluster last night, I noticed lots of "Out Of Memory" errors in /var/log/messages, many of these correlate to dead OSD's. If this is the problem, this might now be another case of the high memory use issues reported in Kraken. e.g. my script logs: Thu 8 Jun

Re: [ceph-users] CephFS Snapshot questions

John, Thanks for your answers. I have a clarification on my questions see below inline. Bruce From: John Spray Date: Thursday, June 8, 2017 at 1:45 AM To: "McFarland, Bruce" Cc: "ceph-users@lists.ceph.com" Subject:

Re: [ceph-users] 2x replica with NVMe

Whether or not 2x replica is possible has little to do with the technology and EVERYTHING to do with your use case. How redundant is your hardware for instance? If you have the best drives in the world that will never fail after constant use over 100 years but you don't have redundant power,

Re: [ceph-users] 2x replica with NVMe

Bluestore will make 2x Replica’s “safer” to use in theory. Until Bluestore is in use in the wild, I don’t think anyone can give any guarantees. From: i...@witeq.com [mailto:i...@witeq.com] Sent: 08 June 2017 14:32 To: nick Cc: Vy Nguyen Tan ;

Re: [ceph-users] PG that should not be on undersized+degraded on multi datacenter Ceph cluster

Hi Brad. Taking into consideration the unlikely posibility that someone realizes what the problem is in this specific case, that would be higly apreciated. I presume that having jewel, if you can somehow remediate this, will be something that i will not be able to have on this deploy right?

Re: [ceph-users] 2x replica with NVMe

I'm thinking to delay this project until Luminous release to have Bluestore support. So are you telling me that checksum capability will be present in Bluestore and therefore considering using NVMe with 2x replica for production data will be possibile? From: "nick" To:

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

Hello Mark, As this issue noticed while giving write via librados {C API} only , the same can't be reproduce with rados user space utility. Ref;- http://docs.ceph.com/docs/master/rados/api/librados/ Jack, I guess you also creating load via librados. Thanks Jayaram On Thu, Jun 8, 2017 at 5:46

Re: [ceph-users] 2x replica with NVMe

There are two main concerns with using 2x replicas, recovery speed and coming across inconsistent objects. With spinning disks their size to access speed means recovery can take a long time and increases the chance that additional failures may happen during the recovery process. NVME will

Re: [ceph-users] 2x replica with NVMe

Hi, I think that the replica 2x on HDD/SSD are the same. You should read quote from Wido bellow: ""Hi, As a Ceph consultant I get numerous calls throughout the year to help people with getting their broken Ceph clusters back online. The causes of downtime vary vastly, but one of the biggest

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

Hi Jayaram, Thanks for creating a tracker entry! Any chance you could add a note about how you are generating the 200MB/s client workload? I've not seen this problem in the lab, but any details you could give that would help us reproduce the problem would be much appreciated! Mark On

Re: [ceph-users] Lumionous: bluestore 'tp_osd_tp thread tp_osd_tp' had timed out after 60

Hello Mark, Raised tracker for the issue -- http://tracker.ceph.com/issues/20222 Jake can you share the restart_OSD_and_log-this.sh script Thanks Jayaram On Wed, Jun 7, 2017 at 9:40 PM, Jake Grimmett wrote: > Hi Mark & List, > > Unfortunately, even when using

Re: [ceph-users] Cache mode readforward mode will eat your babies?

On Thu, Jun 8, 2017 at 3:38 AM, Christian Balzer wrote: > On Thu, 8 Jun 2017 17:03:15 +1000 Brad Hubbard wrote: > >> On Thu, Jun 8, 2017 at 3:47 PM, Christian Balzer wrote: >> > On Thu, 8 Jun 2017 15:29:05 +1000 Brad Hubbard wrote: >> > >> >> On Thu, Jun 8, 2017 at

[ceph-users] 2x replica with NVMe

Hi all, i'm going to build an all-flash ceph cluster, looking around the existing documentation i see lots of guides and and use case scenarios from various vendor testing Ceph with replica 2x. Now, i'm an old school Ceph user, I always considered 2x replica really dangerous for production

Re: [ceph-users] CephFS Snapshot questions

On Wed, Jun 7, 2017 at 11:46 PM, McFarland, Bruce wrote: > I have a couple of CephFS snapshot questions > > - Is there any functionality similar to rbd clone/flatten such that > the snapshot can be made writable? Or is that as simple as copying the > .snap/

Re: [ceph-users] Changing SSD Landscape

Looking at that anandtech comparison it seems the Micron usually is worse than the P3700. This week I asked for a few nodes with P3700 400G and got an answer as they're end of sale, and the supplier wouldn't be able to get it anywhere in the world. Has anyone got a good replacement for these?

Re: [ceph-users] Cache mode readforward mode will eat your babies?

On Thu, 8 Jun 2017 17:03:15 +1000 Brad Hubbard wrote: > On Thu, Jun 8, 2017 at 3:47 PM, Christian Balzer wrote: > > On Thu, 8 Jun 2017 15:29:05 +1000 Brad Hubbard wrote: > > > >> On Thu, Jun 8, 2017 at 3:10 PM, Christian Balzer wrote: > >> > On Thu, 8 Jun 2017

Re: [ceph-users] Cache mode readforward mode will eat your babies?

On Thu, Jun 8, 2017 at 3:47 PM, Christian Balzer wrote: > On Thu, 8 Jun 2017 15:29:05 +1000 Brad Hubbard wrote: > >> On Thu, Jun 8, 2017 at 3:10 PM, Christian Balzer wrote: >> > On Thu, 8 Jun 2017 14:21:43 +1000 Brad Hubbard wrote: >> > >> >> On Thu, Jun 8, 2017 at

[ceph-users] Question about the Ceph's performance with spdk

Hi all, I’m new in Ceph, and I wonder to know the performance report exactly about Ceph’s spdk, but I couldn’t find it. The most thing I want to know is the performance improvement before spdk and after. Thanks, Datong Li ___ ceph-users mailing