[ceph-users] Mixed Bluestore and Filestore NVMe OSDs for RGW metadata both running out of space

2018-08-29 Thread David Turner
osd daemon perf dump for a one of my bluestore NVMe OSDs has [1] this excerpt. I grabbed those stats based on Wido's [2] script to determine how much DB overhead you have per object. My [3] calculations for this particular OSD are staggering. 99% of the space used on this OSD is being consumed

Re: [ceph-users] Hammer and a (little) disk/partition shrink...

2018-08-29 Thread Marco Gaiarin
Mandi! David Turner In chel di` si favelave... > Replace the raid controller in the chassis with an HBA before moving into the > new hardware? ;) Eh... some hint on a controller i can buy? > If you do move to the HP controller, make sure you're monitoring the health of > the cache battery in

Re: [ceph-users] Ceph-Deploy error on 15/71 stage

2018-08-29 Thread Jones de Andrade
Hi Eugen. Sorry for the delay in answering. Just looked in the /var/log/ceph/ directory. It only contains the following files (for example on node01): ### # ls -lart total 3864 -rw--- 1 ceph ceph 904 ago 24 13:11 ceph.audit.log-20180829.xz drwxr-xr-x 1 root root 898 ago 28 10:07

Re: [ceph-users] Hammer and a (little) disk/partition shrink...

2018-08-29 Thread David Turner
Replace the raid controller in the chassis with an HBA before moving into the new hardware? ;) If you do move to the HP controller, make sure you're monitoring the health of the cache battery in the controller. We notice a significant increase to await on our OSD nodes behind these when the

Re: [ceph-users] cephfs mount on osd node

2018-08-29 Thread David Turner
The problem with mounting an RBD or CephFS on an OSD node is if you're doing so with the kernel client. In a previous message on the ML John Spray explained this wonderfully. "This is not a Ceph-specific thing -- it can also affect similar systems like Lustre. The classic case is when under

Re: [ceph-users] Error EINVAL: (22) Invalid argument While using ceph osd safe-to-destroy

2018-08-29 Thread Alfredo Deza
I am addressing the doc bug at https://github.com/ceph/ceph/pull/23801 On Mon, Aug 27, 2018 at 2:08 AM, Eugen Block wrote: > Hi, > > could you please paste your osd tree and the exact command you try to > execute? > >> Extra note, the while loop in the instructions look like it's bad. I had >>

[ceph-users] Hammer and a (little) disk/partition shrink...

2018-08-29 Thread Marco Gaiarin
Probably a complex question, with a simple answer: NO. ;-) I need to move disks from a ceph node (still on hammer) from an hardware to another one. The source hardware have a simple SATA/SAS controller, the 'new' server have a RAID controller with no JBOD mode (the infamous HP P410i), so i

[ceph-users] Looking for information on full SSD deployments

2018-08-29 Thread Valmar Kuristik
Hello fellow Ceph users, We have been using a small cluster (6 data nodes with 12 disks each, 3 monitors) with OSDs on spinners and journals on SATA SSD-s for a while now. We still haven't upgraded to Luminous, and are going to test it now, as we also need to switch some projects on a shared

Re: [ceph-users] SSD OSDs crashing after upgrade to 12.2.7

2018-08-29 Thread Alfredo Deza
On Wed, Aug 29, 2018 at 2:06 AM, Wolfgang Lendl wrote: > Hi, > > after upgrading my ceph clusters from 12.2.5 to 12.2.7 I'm experiencing > random crashes from SSD OSDs (bluestore) - it seems that HDD OSDs are not > affected. > I destroyed and recreated some of the SSD OSDs which seemed to

[ceph-users] cephfs mount on osd node

2018-08-29 Thread Jake Grimmett
Hi Marc, We mount cephfs using FUSE on all 10 nodes of our cluster, and provided that we limit bluestore memory use, find it to be reliable*. bluestore_cache_size = 209715200 bluestore_cache_kv_max = 134217728 Without the above tuning, we get OOM errors. As others will confirm, the FUSE client

Re: [ceph-users] New Ceph community manager: Mike Perez

2018-08-29 Thread Sage Weil
Correction: Mike's new email is actually mipe...@redhat.com (sorry, mperez!). sage ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] cephfs mount on osd node

2018-08-29 Thread Marc Roos
I have 3 node test cluster and I would like to expand this with a 4th node that is currently mounting the cephfs and rsync's backups to it. I can remember reading something about that you could create a deadlock situation doing this. What are the risks I would be taking if I would be doing

Re: [ceph-users] prevent unnecessary MON leader re-election

2018-08-29 Thread Joao Eduardo Luis
On 08/29/2018 11:02 AM, William Lawton wrote: > > We have a 5 node Ceph cluster, status output copied below. During our > cluster resiliency tests we have noted that a MON leader election takes > place when we fail one member of the MON quorum, even though the failed > instance is not the current

Re: [ceph-users] Installing ceph 12.2.4 via Ubuntu apt

2018-08-29 Thread Paul Emmerich
The root cause is a restriction in reprepro used to manage the repository: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=570623 Paul 2018-08-29 8:50 GMT+02:00 Thomas Bennett : > Hi David, > > Thanks for your reply. That's how I'm currently handling it. > > Kind regards, > Tom > > On Tue, Aug

[ceph-users] prevent unnecessary MON leader re-election

2018-08-29 Thread William Lawton
Hi. We have a 5 node Ceph cluster, status output copied below. During our cluster resiliency tests we have noted that a MON leader election takes place when we fail one member of the MON quorum, even though the failed instance is not the current MON leader. We speculate that this re-election

Re: [ceph-users] New Ceph community manager: Mike Perez

2018-08-29 Thread Lars Marowsky-Bree
On 2018-08-29T01:13:24, Sage Weil wrote: Most excellent! Welcome, Mike! I look forward to working with you. Regards, Lars -- Architect SDS, Distinguished Engineer SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) "Architects should open

Re: [ceph-users] New Ceph community manager: Mike Perez

2018-08-29 Thread Joao Eduardo Luis
On 08/29/2018 02:13 AM, Sage Weil wrote: > Hi everyone, > > Please help me welcome Mike Perez, the new Ceph community manager! Very happy to have you with us! Let us know if there's anything we can help you with, and don't hesitate to get in touch :) -Joao

Re: [ceph-users] New Ceph community manager: Mike Perez

2018-08-29 Thread Lenz Grimmer
Great news. Welcome Mike! I look forward to working with you, let me know if there is anything I can help you with. Lenz On 08/29/2018 03:13 AM, Sage Weil wrote: > Please help me welcome Mike Perez, the new Ceph community manager! > > Mike has a long history with Ceph: he started at DreamHost

[ceph-users] SSD OSDs crashing after upgrade to 12.2.7

2018-08-29 Thread Wolfgang Lendl
Hi, after upgrading my ceph clusters from 12.2.5 to 12.2.7  I'm experiencing random crashes from SSD OSDs (bluestore) - it seems that HDD OSDs are not affected. I destroyed and recreated some of the SSD OSDs which seemed to help. this happens on centos 7.5 (different kernels tested)

[ceph-users] Ceph cluster "hung" after node failure

2018-08-29 Thread Brett Chancellor
Hi All. I have a ceph cluster that's partially upgraded to Luminous. Last night a host died and since then the cluster is failing to recover. It finished backfilling, but was left with thousands of requests degraded, inactive, or stale. In order to move past the issue, I put the cluster in

Re: [ceph-users] SAN or DAS for Production ceph

2018-08-29 Thread James Watson
Thanks, Tom and John, both of your input really helpful and helped to put things into perspective. Much appreciated. @John, I am based out of Dubai. On Wed, Aug 29, 2018 at 2:06 AM John Hearns wrote: > James, you also use the words enterprise and production ready. > Is Redhat support

Re: [ceph-users] Installing ceph 12.2.4 via Ubuntu apt

2018-08-29 Thread Thomas Bennett
Hi David, Thanks for your reply. That's how I'm currently handling it. Kind regards, Tom On Tue, Aug 28, 2018 at 4:36 PM David Turner wrote: > That is the expected behavior of the ceph repo. In the past when I needed > a specific version I would download the packages for the version to a >