Re: [ceph-users] Min Size equal to Replicated Size Risks

The pool will not actually go read only. All read and write requests will block until both osds are back up. If I were you, I would use min_size=2 and change it to 1 temporarily if needed to do maintenance or troubleshooting where down time is not an option. On Thu, Feb 22, 2018, 5:31 PM Georgios

Re: [ceph-users] Ceph auth caps - make it more user error proof

On Wed, Feb 21, 2018 at 10:54 AM, Enrico Kern wrote: > Hey all, > > i would suggest some changes to the ceph auth caps command. > > Today i almost fucked up half of one of our openstack regions with i/o > errors because of user failure. > > I tried to add osd

Re: [ceph-users] SSD Bluestore Backfills Slow

What's the output of "ceph -s" while this is happening? Is there some identifiable difference between these two states, like you get a lot of throughput on the data pools but then metadata recovery is slower? Are you sure the recovery is actually going slower, or are the individual ops larger or

Re: [ceph-users] PG overdose protection causing PG unavailability

Am 23.02.2018 um 01:05 schrieb Gregory Farnum: > > > On Wed, Feb 21, 2018 at 2:46 PM Oliver Freyermuth > > wrote: > > Dear Cephalopodians, > > in a Luminous 12.2.3 cluster with a pool with: > - 192 Bluestore OSDs

Re: [ceph-users] Ceph Bluestore performance question

Hi Vadim, many thanks for these benchmark results! This indeed looks extremely similar to what we achieve after enabling connected mode. Our 6 OSD-hosts are Supermicro systems with 2 HDDs (Raid 1) for the OS, and 32 HDDs (4 TB) + 2 SSDs for the OSDs. The 2 SSDs have 16 LVM volumes each

Re: [ceph-users] PG overdose protection causing PG unavailability

On Wed, Feb 21, 2018 at 2:46 PM Oliver Freyermuth < freyerm...@physik.uni-bonn.de> wrote: > Dear Cephalopodians, > > in a Luminous 12.2.3 cluster with a pool with: > - 192 Bluestore OSDs total > - 6 hosts (32 OSDs per host) > - 2048 total PGs > - EC profile k=4, m=2 > - CRUSH failure domain =

Re: [ceph-users] PG mapped to OSDs on same host although 'chooseleaf type host'

was the pg-upmap feature used to force a pg to get mapped to a particular osd? mike On Thu, Feb 22, 2018 at 10:28 AM, Wido den Hollander wrote: > Hi, > > I have a situation with a cluster which was recently upgraded to Luminous > and has a PG mapped to OSDs on the same host. > >

Re: [ceph-users] PG mapped to OSDs on same host although 'chooseleaf type host'

On Thu, Feb 22, 2018 at 9:29 AM Wido den Hollander wrote: > Hi, > > I have a situation with a cluster which was recently upgraded to > Luminous and has a PG mapped to OSDs on the same host. > > root@man:~# ceph pg map 1.41 > osdmap e21543 pg 1.41 (1.41) -> up [15,7,4] acting

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

adding ceph-users back on. it sounds like the enterprise samsungs and hitachis have been mentioned on the list as alternatives. i have 2 micron 5200 (pro i think) that i'm beginning testing on and have some micron 9100 nvme drives to use as journals. so the enterprise micron might be good. i did

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

hrm. intel has, until a year ago, been very good with ssds. the description of your experience definitely doesn't inspire confidence. intel also dropping the entire s3xxx and p3xxx series last year before having a viable replacement has been driving me nuts. i don't know that i have the luxury of

Re: [ceph-users] Min Size equal to Replicated Size Risks

All right! Thank you very much Jack! The way I understand this is that it's not necessarily a bad thing. I mean as long as it doesn't harm any data or cannot cause any other issue. Unfortunately my scenario consists of only two OSDs therefore there is a replication factor of 2 and

Re: [ceph-users] Min Size equal to Replicated Size Risks

If min_size == size, a single OSD failure will place your pool read only On 02/22/2018 11:06 PM, Georgios Dimitrakakis wrote: > Dear all, > > I would like to know if there are additional risks when running CEPH > with "Min Size" equal to "Replicated Size" for a given pool. > > What are the

[ceph-users] Min Size equal to Replicated Size Risks

Dear all, I would like to know if there are additional risks when running CEPH with "Min Size" equal to "Replicated Size" for a given pool. What are the drawbacks and what could be go wrong in such a scenario? Best regards, G. ___ ceph-users

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

Hi Mike, I eventually got hold of a customer relations manager at Intel but his attitude was lack luster and Intel never officially responded to any correspondence we sent them. The Intel s4600 drives all passed our standard burn-in tests, they exclusively appear to fail once they handle

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

has anyone tried with the most recent firmwares from intel? i've had a number of s4600 960gb drives that have been waiting for me to get around to adding them to a ceph cluster. this as well as having 2 die almost simultaneously in a different storage box is giving me pause. i noticed that David

Re: [ceph-users] Migrating to new pools

Cumulative followup to various insightful replies. I wrote: No, it's not really possible currently and we have no plans to add >>> such support since it would not be of any long-term value. >> >> The long-term value would be the ability to migrate volumes from, say, a >> replicated pool

Re: [ceph-users] Balanced MDS, all as active and recomended client settings.

On Wed, Feb 21, 2018 at 11:17 PM, Daniel Carrasco wrote: > I want to search also if there is any way to cache file metadata on client, > to lower the MDS load. I suppose that files are cached but the client check > with MDS if there are changes on files. On my server files

[ceph-users] Ceph Tech Talk canceled

Hey Cephers, Sorry for the short notice, but the Ceph Tech Talk schedule for today (Thursday, February 22) has been canceled. Kindest regards, Leo -- Leonardo Vaz Ceph Community Manager Open Source and Standards Team ___ ceph-users mailing list

[ceph-users] PG mapped to OSDs on same host although 'chooseleaf type host'

Hi, I have a situation with a cluster which was recently upgraded to Luminous and has a PG mapped to OSDs on the same host. root@man:~# ceph pg map 1.41 osdmap e21543 pg 1.41 (1.41) -> up [15,7,4] acting [15,7,4] root@man:~# root@man:~# ceph osd find 15|jq -r '.crush_location.host' n02

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

Hi Caspar, Sean and I replaced the problematic DC S4600 disks (after all but one had failed) in our cluster with Samsung SM863a disks. There was an NDA for new Intel firmware (as mentioned earlier in the thread by David) but given the problems we were experiencing we moved all Intel disks to a

Re: [ceph-users] PG overdose protection causing PG unavailability

Am 22.02.2018 um 02:54 schrieb David Turner: > You could set the flag noin to prevent the new osds from being calculated by > crush until you are ready for all of them in the host to be marked in. > You can also set initial crush weight to 0 for new pads so that they won't > receive any PGs

Re: [ceph-users] ceph-volume broken in 12.2.3 for filestore

On Thu, Feb 22, 2018 at 8:03 AM, Alfredo Deza wrote: > For the recent 12.2.3 Luminous release we've found that it is not > possible to mount a device when using filestore [0] > > We are trying to find out how this was able to get pass our functional > testing, Our functional

Re: [ceph-users] Luminous v12.2.3 released

Hi, Debian Packages for stretch have broken dependencies: The following packages have unmet dependencies: ceph-common : Depends: libleveldb1 but it is not installable Depends: libsnappy1 but it is not installable ceph-mon : Depends: libleveldb1 but it is not installable

Re: [ceph-users] PG_DAMAGED Possible data damage: 1 pg inconsistent

Le 22/02/2018 à 05:23, Brad Hubbard a écrit : > On Wed, Feb 21, 2018 at 6:40 PM, Yoann Moulin wrote: >> Hello, >> >> I migrated my cluster from jewel to luminous 3 weeks ago (using ceph-ansible >> playbook), a few days after, ceph status told me "PG_DAMAGED >> Possible data

Re: [ceph-users] Migrating to new pools

On Wed, Feb 21, 2018 at 7:27 PM, Anthony D'Atri wrote: >>> I was thinking we might be able to configure/hack rbd mirroring to mirror to >>> a pool on the same cluster but I gather from the OP and your post that this >>> is not really possible? >> >> No, it's not really

[ceph-users] ceph-volume broken in 12.2.3 for filestore

For the recent 12.2.3 Luminous release we've found that it is not possible to mount a device when using filestore [0] We are trying to find out how this was able to get pass our functional testing, in the meantime the workaround for this problem is to add the mount options in ceph.conf (as

Re: [ceph-users] Many concurrent drive failures - How do I activate pgs?

Hi Sean and David, Do you have any follow ups / news on the Intel DC S4600 case? We are looking into this drives to use as DB/WAL devices for a new to be build cluster. Did Intel provide anything (like new firmware) which should fix the issues you were having or are these drives still

Re: [ceph-users] Ceph Bluestore performance question

Hi Oliver, i also use Infiniband and Cephfs for HPC purposes. My setup: * 4x Dell R730xd and expansion shelf, 24 OSD à 8TB, 128GB Ram, 2x10Core Intel 4th Gen, Mellanox ConnectX-3, no SSD-Cache * 7x Dell R630 Clients * Ceph-Cluster running on Ubuntu Xenial and Ceph Jewel deployed with

Re: [ceph-users] identifying public buckets

On Thu, Feb 22, 2018 at 06:00:12AM +, Robin H. Johnson wrote: > You need to create a RGW user with the system flag set (it might be > possible with the newer admin flag as well). That was the missing piece - thanks very much! I have it working now. Cheers, Dave -- ** Dave Holland ** Systems

Re: [ceph-users] mon service failed to start

Did you remove and recreate the OSDs that used the SSD for their WAL/DB? Or did you try to do something to not have to do that? That is an integral part of the OSD and changing the SSD would destroy the OSDs involved unless you attempted some sort of dd. If you did that, then any corruption for

Re: [ceph-users] ceph-volume activation

On Wed, Feb 21, 2018 at 11:56 PM, Oliver Freyermuth wrote: > Am 21.02.2018 um 15:58 schrieb Alfredo Deza: >> On Wed, Feb 21, 2018 at 9:40 AM, Dan van der Ster >> wrote: >>> On Wed, Feb 21, 2018 at 2:24 PM, Alfredo Deza

Re: [ceph-users] mon service failed to start

Hi Brian, The issue started with failing mon service and after that both OSDs on that node failed to start. Mon service is on SSD disk and WAL/DB of OSDs on that SSD too with lvm. I have changed SSD disk with new one, and changing SATA port and cable but the problem is still remaining. All disk