Re: [ceph-users] mirror OSD configuration

> Is it possible to configure crush map such that it will tolerate "room" > failure? In my case, there is one > network switch per room and one power supply per room, which makes a single > point of (room) failure. Hi, You cannot achieve real room redundancy with just two rooms. At minimum

[ceph-users] mirror OSD configuration

This is my planned OSD configuration: root room1 OSD host1 OSD host2 room2 OSD host3 OSD host4 There are 6 OSDs per host. Is it possible to configure crush map such that it will tolerate "room" failure? In my case, there is one network switch per room

[ceph-users] Ceph Developer Monthly - March 2018

Hey Cephers, This is just a friendly reminder that the next Ceph Developer Monthly meeting is coming up: http://wiki.ceph.com/Planning If you have work that you're doing that it a feature work, significant backports, or anything you would like to discuss with the core team, please add it to

Re: [ceph-users] ceph-deploy won't install luminous (but Jewel instead)

Try using:ceph-deploy --release luminous host1... Mensaje original De: Massimiliano Cuttini Fecha: 28/2/18 12:42 a. m. (GMT+01:00) Para: ceph-users@lists.ceph.com Asunto: [ceph-users] ceph-deploy won't install luminous (but Jewel instead) This is the

Re: [ceph-users] Luminous | PG split causing slow requests

This is super helpful, thanks for sharing, David. I need to a bit more reading into this. On 26 Feb 2018 6:08 p.m., "David Turner" wrote: The slow requests are absolutely expected on filestore subfolder splitting. You can however stop an OSD, split it's subfolders, and

Re: [ceph-users] OSD maintenance (ceph osd set noout)

On Tue, Feb 27, 2018 at 6:37 PM, Andre Goree wrote: > Is it still considered best practice to set 'noout' for OSDs that will be > going under maintenance, e.g., rebooting an OSD ndoe for a kernel update? > > I ask, because I've set this twice now during times which the OSDs

[ceph-users] Kubernetes support for rbd-nbd

Hi all, Heads up for any of you who are using Kubernetes or looking to use Kubernetes to schedule Ceph daemons running within containers (for block volumes). We have been working on adding support for rbd-nbd in the Kubernetes provisioner (as context, the current Kubernetes provisioner offers

[ceph-users] OSD maintenance (ceph osd set noout)

Is it still considered best practice to set 'noout' for OSDs that will be going under maintenance, e.g., rebooting an OSD ndoe for a kernel update? I ask, because I've set this twice now during times which the OSDs would only momentarily be 'out', however each time I've done this, the OSDs

Re: [ceph-users] Cephfs fsal + nfs-ganesha + el7/centos7

Am 19.02.2018 um 17:22 schrieb Daniel Gryniewicz: > To my knowledge, no one has done any work on ganesha + ceph and selinux.   > Fedora (and RHEL) includes config in it's selinux package for ganesha + > gluster, but I'm sure there's missing bits for ceph. Thanks! I was asking here since from the

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

Thanks for making this clear. Dietmar On 02/27/2018 05:29 PM, Alfredo Deza wrote: > On Tue, Feb 27, 2018 at 11:13 AM, Dietmar Rieder > wrote: >> ... however, it would be nice if ceph-volume would also create the >> partitions for the WAL and/or DB if needed. Is there

Re: [ceph-users] Ceph luminous - Erasure code and iSCSI gateway

Thanks - that worked [root@osd01 ~]# rbd --image image_ec1 -p rbd info rbd image 'image_ec1': size 51200 MB in 12800 objects order 22 (4096 kB objects) data_pool: ec_k4_m2 block_name_prefix: rbd_data.1.fe0f643c9869 format: 2 features: layering,

Re: [ceph-users] fast_read in EC pools

Dear Caspar, many thanks for the link! Now I'm pondering - the problem is that we'd certainly want to keep m=2, but reducing k=4 to k=3 already means significant reduction of storage. We'll elaborate on the probability of new OSD hosts being added in the near future and consider this before

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

On Tue, Feb 27, 2018 at 11:13 AM, Dietmar Rieder wrote: > ... however, it would be nice if ceph-volume would also create the > partitions for the WAL and/or DB if needed. Is there a special reason, > why this is not implemented? Yes, the reason is that this was one of

Re: [ceph-users] Ceph luminous - Erasure code and iSCSI gateway

Do your pre-created images have the exclusive-lock feature enabled? That is required to utilize them for iSCSI. On Tue, Feb 27, 2018 at 11:09 AM, Steven Vacaroaia wrote: > Hi Jason, > > Thanks for your prompt response > > I have not been able to find a way to add an existing

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

... however, it would be nice if ceph-volume would also create the partitions for the WAL and/or DB if needed. Is there a special reason, why this is not implemented? Dietmar On 02/27/2018 04:25 PM, David Turner wrote: > Gotcha.  As a side note, that setting is only used by ceph-disk as >

Re: [ceph-users] Ceph luminous - Erasure code and iSCSI gateway

Hi Jason, Thanks for your prompt response I have not been able to find a way to add an existing image ... it looks like I can just create new ones Ill appreciate if you could provide details please For example how would I add the preexisting image named image_ec1 ? rados -p rbd ls | grep

Re: [ceph-users] Ceph luminous - Erasure code and iSCSI gateway

Your image does not live in the EC pool -- instead, only the data portion lives within the EC pool. Therefore, you would need to specify the replicated pool where the image lives when attaching it as a backing store for iSCSI (i.e. pre-create it via the rbd CLI): # gwcli

Re: [ceph-users] Significance of the us-east-1 region when using S3 clients to talk to RGW

s3cmd does have special handling for 'US' and 'us-east-1' that skips the LocationConstraint on bucket creation: https://github.com/s3tools/s3cmd/blob/master/S3/S3.py#L380 On 02/26/2018 05:16 PM, David Turner wrote: I just realized the difference between the internal realm, local realm, and

[ceph-users] Ceph luminous - Erasure code and iSCSI gateway

Hi, I noticed it is possible to use erasure code pool for RBD and CEPHFS https://ceph.com/community/new-luminous-erasure-coding-rbd-cephfs/ This got me thinking that I can deploy iSCSI luns on EC pools However it appears it is not working Anyone able to do that or have I misunderstood ?

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

Gotcha. As a side note, that setting is only used by ceph-disk as ceph-volume does not create partitions for the WAL or DB. You need to create those partitions manually if using anything other than a whole block device when creating OSDs with ceph-volume. On Tue, Feb 27, 2018 at 8:20 AM Caspar

Re: [ceph-users] cannot reboot one of 3 nodes without locking a cluster OSDs stay in...

Can you start giving specifics? Ceph version, were the disks created with ceph-disk or ceph-volume, filestore/bluestore, upgraded from another version, has anything changed recently (an upgrade, migrating some OSDs from filestore to bluestore), etc, etc. Sometimes I've found a node just fails to

Re: [ceph-users] SSD Bluestore Backfills Slow

I have 2 different configurations that are incorrectly showing rotational for the OSDs. The [1]first is a server with disks behind controllers and an NVME riser card. It has 2 different OSD types, one with the block on an HDD and WAL on the NVME as well as a pure NVME OSD. The Hybrid OSD seems

Re: [ceph-users] cannot reboot one of 3 nodes without locking a cluster OSDs stay in...

They are stopped gracefully. i did a reboot 2 days ago. but now it doesnt work. 2018-02-27 14:24 GMT+01:00 David Turner : > `systemctl list-dependencies ceph.target` > > I'm guessing that you might need to enable your osds to be managed by > systemctl so that they can be

Re: [ceph-users] fast_read in EC pools

Oliver, Here's the commit info: https://github.com/ceph/ceph/commit/48e40fcde7b19bab98821ab8d604eab920591284 Caspar 2018-02-27 14:28 GMT+01:00 Oliver Freyermuth : > Am 27.02.2018 um 14:16 schrieb Caspar Smit: > > Oliver, > > > > Be aware that for k=4,m=2 the

Re: [ceph-users] ceph-volume activation

Hi Oliver, No ticket yet... we were distracted. I have the same observations as what you show below... -- dan On Tue, Feb 27, 2018 at 2:33 PM, Oliver Freyermuth wrote: > Am 22.02.2018 um 09:44 schrieb Dan van der Ster: >> On Wed, Feb 21, 2018 at 11:56 PM,

Re: [ceph-users] ceph-volume activation

Am 22.02.2018 um 09:44 schrieb Dan van der Ster: > On Wed, Feb 21, 2018 at 11:56 PM, Oliver Freyermuth > wrote: >> Am 21.02.2018 um 15:58 schrieb Alfredo Deza: >>> On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster >>> wrote: On Wed, Feb

Re: [ceph-users] fast_read in EC pools

Am 27.02.2018 um 14:16 schrieb Caspar Smit: > Oliver, > > Be aware that for k=4,m=2 the min_size will be 5 (k+1), so after a node > failure the min_size is already reached. > Any OSD failure beyond the node failure will probably result in some PG's to > be become incomplete (I/O Freeze) until

Re: [ceph-users] cannot reboot one of 3 nodes without locking a cluster OSDs stay in...

`systemctl list-dependencies ceph.target` I'm guessing that you might need to enable your osds to be managed by systemctl so that they can be stopped when the server goes down. `systemctl enable ceph-osd@{osd number}` On Tue, Feb 27, 2018, 4:13 AM Philip Schroth

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

David, Yes i know, i use 20GB partitions for 2TB disks as journal. It was just to inform other people that Ceph's default of 1GB is pretty low. Now that i read my own sentence it indeed looks as if i was using 1GB partitions, sorry for the confusion. Caspar 2018-02-27 14:11 GMT+01:00 David

Re: [ceph-users] fast_read in EC pools

Oliver, Be aware that for k=4,m=2 the min_size will be 5 (k+1), so after a node failure the min_size is already reached. Any OSD failure beyond the node failure will probably result in some PG's to be become incomplete (I/O Freeze) until the incomplete PG's data is recovered to another OSD in

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

If you're only using a 1GB DB partition, there is a very real possibility it's already 100% full. The safe estimate for DB size seams to be 10GB/1TB so for a 4TB osd a 40GB DB should work for most use cases (except loads and loads of small files). There are a few threads that mention how to check

Re: [ceph-users] Small modifications to Bluestore migration documentation

On Tue, Feb 27, 2018 at 6:55 AM, Alexander Kushnirenko wrote: > Hello, > > Luminous 12.2.2 > > There were several discussions on this list concerning Bluestore migration, > as official documentation does not work quite well yet. In particular this > one >

[ceph-users] Small modifications to Bluestore migration documentation

Hello, Luminous 12.2.2 There were several discussions on this list concerning Bluestore migration, as official documentation does not work quite well yet. In particular this one http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-January/024190.html Is it possible to modify official

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

2018-02-26 23:01 GMT+01:00 Gregory Farnum : > On Mon, Feb 26, 2018 at 3:23 AM Caspar Smit > wrote: > >> 2018-02-24 7:10 GMT+01:00 David Turner : >> >>> Caspar, it looks like your idea should work. Worst case scenario seems >>>

Re: [ceph-users] Proper procedure to replace DB/WAL SSD

2018-02-26 18:02 GMT+01:00 David Turner : > I'm glad that I was able to help out. I wanted to point out that the > reason those steps worked for you as quickly as they did is likely that you > configured your blocks.db to use the /dev/disk/by-partuuid/{guid} instead > of

[ceph-users] cannot reboot one of 3 nodes without locking a cluster OSDs stay in...

I have a 3 node production cluster. All works fine. but i have one failing node. i replaced one disk on sunday. everyting went fine. last night there was another disk broken. Ceph nicely maks it as down. but when i wanted to reboot this node now. all remaining osd's are being kept in and not

[ceph-users] (no subject)

I have a 3 node production cluster. All works fine. but i have one failing node. i replaced one disk on sunday. everyting went fine. last night there was another disk broken. Ceph nicely maks it as down. but when i wanted to reboot this node now. all remaining osd's are being kept in and not