[ceph-users] How many MDS servers

Quick question Ceph guru's. For a 1.1PB raw cephfs system currently storing 191TB of data and 390 million objects (mostly small Python, ML training files etc.) how many MDS servers should I be running? System is Nautilus 14.2.8. I ask because up to know I have run one MDS with one

[ceph-users] Re: Forcibly move PGs from full to empty OSD

Hi Thomas, The two commands you're looking for are: ceph osd pg-upmap-items $pg $source_osd $dest_osd and to remove them ceph osd rm-pg-upmap-items $pg You need to pair this with finding which pg's are on your full osd's. I use ceph pg dump and grep for the pool number and osd. With respect to

[ceph-users] ceph: Can't lookup inode 1 (err: -13)

For testing purposes I changed the kernel 3.10 for a 5.5, now I am getting these messages. I assume the 3.10 was just never displaying these. Could this be a problem with my caps of the fs id user? [Mon Mar 9 23:10:52 2020] ceph: Can't lookup inode 1 (err: -13) [Mon Mar 9 23:12:03 2020]

[ceph-users] Re: Clear health warning

Perfect. Thanks! (adding a standby) peter Peter Eisch Senior Site Reliability Engineer T1.612.445.5135 virginpulse.com |virginpulse.com/global-challenge Australia | Bosnia and Herzegovina | Brazil | Canada | Singapore | Switzerland | United Kingdom | USA Confidentiality Notice: The

[ceph-users] Re: Clear health warning

Right, so you have 3 active MDS's and 0 on standby, which is generating a (correct) health warning. You need to either add more MDS' to be standbys, or reduce the fs to 2 active MDS'. On Mon, Mar 9, 2020 at 3:26 PM Peter Eisch wrote: > > > Three MDS’ total, all serving the primary/original fs:

[ceph-users] Re: Clear health warning

Three MDS’ total, all serving the primary/original fs: [root@cephmon-s03 ]# ceph fs status cephfs1 - 8 clients === +--++-+---+---+---+ | Rank | State | MDS |Activity | dns | inos |

[ceph-users] Re: Clear health warning

How many MDS's do you have total, and how are they assigned? 'ceph fs status'. On Mon, Mar 9, 2020 at 3:14 PM Peter Eisch wrote: > Hi, (nautilus, 14.2.8, whole cluster) > > I doodled with adding a second cephfs and the project got canceled. I > removed the unused cephfs with "ceph fs rm dream

[ceph-users] Clear health warning

Hi, (nautilus, 14.2.8, whole cluster) I doodled with adding a second cephfs and the project got canceled. I removed the unused cephfs with "ceph fs rm dream --yes-i-really-mean-it" and that worked as expected. I have a lingering health warning though which won't clear. The original cephfs1

[ceph-users] Re: Link to Nautilus upgrade

Peter; Or possibly this: https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-mimic-or-luminous Or this: https://docs.ceph.com/docs/master/releases/nautilus/#upgrading-from-pre-luminous-releases-like-jewel Thank you, Dominic L. Hilsbos, MBA Director – Information Technology

[ceph-users] Re: Link to Nautilus upgrade

Peter; Might this be what you're after: https://docs.ceph.com/docs/nautilus/install/upgrading-ceph/# Thank you, Dominic L. Hilsbos, MBA Director – Information Technology Perform Air International Inc. dhils...@performair.com www.PerformAir.com From: Peter Eisch

[ceph-users] Re: Hardware feedback before purchasing for a PoC

Ignacio; Personally, I like to use hardware for a proof of concept that I can roll over into the final system, or repurpose if the project is denied. As such, I would recommend these: Supermicro 5019A-12TN4 Barebones

[ceph-users] Link to Nautilus upgrade

Hi, When upgrading a cluster from Luminous to Nautilus I followed a page on ceph.com.  I need to do another cluster and while I tagged the link, the page no longer exists. https://docs.ceph.com/master/releases/nautilus/#nautilus-old-upgrade Might anyone have either an updated link or point me

[ceph-users] A fast tool to export/copy a pool

I am wondering if there exists a tool, faster than "rados export", that can copy and restore read-only pools (to/from another pool or file system). It looks like "rados export" is very slow because it is single threaded (the best I can tell, --workers doesn't make a difference). Vlad

[ceph-users] Re: Hardware feedback before purchasing for a PoC

Hi Ignacio, El 9/3/20 a las 15:19, Ignacio Ocampo escribió: I was considering 1 GB per TB, but I will switch that to 4 GB per TB to consider blue storage. Much better. :) Regarding the number of devices, Will 2 disks per node help with the cluster speed? Yes, it will help. Generally

[ceph-users] Re: Accidentally removed client.admin caps - fix via mon doesn't work

There's only one mon keyring that's shared by all mons, the mon user therefore doesn't contain the mon name. Try "-n mon." Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89

[ceph-users] Re: ceph df hangs

"ceph df" is handled by the mgr, check if your mgr is up and running and if the user has the necessary permissions for the mgr. -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585

[ceph-users] Re: Hardware feedback before purchasing for a PoC

hello if it's only for a poc you can try to rent devices for some month. for less that 100€ per month you can get this kind of device at ovh.com - 32go RAM - Intel Xeon-E 2274G - 4 c/8 t - 4 GHz/4.9 GHz - 3× 4 To HDD SATA Soft RAID or 2× 960 Go SSD NVMe Soft RAID - 2 x 2Gbps network Personnaly

[ceph-users] Re: Hardware feedback before purchasing for a PoC

Hola Ignacio, El 9/3/20 a las 3:00, Ignacio Ocampo escribió: Hi team, I'm planning to invest in hardware for a PoC and I would like your feedback before the purchase: The goal is to deploy a *16TB* storage cluster, with *3 replicas* thus *3 nodes*. System configuration:

[ceph-users] Re: ceph rbd volumes/images IO details

On Sun, Mar 8, 2020 at 5:13 PM M Ranga Swami Reddy wrote: > > Iam using the Luminous 12.2.11 version with prometheus. > > On Sun, Mar 8, 2020 at 12:28 PM XuYun wrote: > > > You can enable prometheus module of mgr if you are running Nautilus. > > > > > 2020年3月8日 上午2:15，M Ranga Swami Reddy 写道： >

[ceph-users] Accidentally removed client.admin caps - fix via mon doesn't work

Hello Guys, Unfortunately, I''ve deleted some caps from client.admin and tried the following solution so set them back: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015474.html I’ve tried the following: # ssh’d to a mon node and changed dir to the mon directory cd

[ceph-users] Re: Identify slow ops

Hi, there's no issue with full OSDs / pools after setting weight on specific OSDs.   cluster:     id: 6b1b5117-6e08-4843-93d6-2da3cf8a6bae     health: HEALTH_WARN     2 slow ops, oldest one blocked for 345057 sec, mon.ld5505 has slow ops   services:     mon: 3 daemons, quorum