Re: [ceph-users] RBD Snapshot

Hi Have you try the rbd du command? Info simply tells you about the size provisioned. Regards JC While moving. Excuse unintended typos. > On Apr 7, 2019, at 13:21, Spencer Babcock wrote: > > Hello, > > I currently have an issue regarding RBD snapshot. > > The issue: > I can fill a

Re: [ceph-users] Doubts about parameter "osd sleep recovery"

Hi Fabio, have a look here: https://github.com/ceph/ceph/blob/luminous/src/common/options.cc#L2355 It’s designed to relieve the pressure generated by the recovery and backfill on both the drives and the network as it

Re: [ceph-users] Bright new cluster get all pgs stuck in inactive

Hi, I suspect your generated CRUSH rule is incorret because of osd_crush_cooseleaf_type=2 and by default chassis bucket are not created. Changing the type of bucket to host (osd_crush_cooseleaf_type=1 which is the default when using old ceph-deploy or ceph-ansible) for your deployment should

Re: [ceph-users] read performance, separate client CRUSH maps or limit osd read access from each client

Hi Vlad, No need for a specific CRUSH map configuration. I’d suggest you use the primary-affinity setting on the OSD so that only the OSDs that are close to your read point are are selected as primary. See https://ceph.com/geen-categorie/ceph-primary-affinity/ for information Just set the

Re: [ceph-users] network architecture questions

They don’t go though the MONs for IOs but they need access to the MONs over the public network for authentication and to receive the cluster map. JC While moving. Excuse unintended typos. > On Sep 18, 2018, at 17:51, Jean-Charles Lopez wrote: > > Hi > > You deploy 3 MONs

Re: [ceph-users] network architecture questions

> On Sep 17, 2018, at 16:13, solarflow99 wrote: > > Hi, I read through the various documentation and had a few questions: > > - From what I understand cephFS clients reach the OSDs directly, does the > cluster network need to be opened up as a public network? Client traffic only goes over the

Re: [ceph-users] slow requests are blocked

1 > 2018-05-08 13:07:29.726124 7f003fdfa700 4 rocksdb: > [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/r > elease/12.2.4/rpm/el7/BUILD/ceph-12.2.4/src/rocksdb/db/flush_job.cc:264] > [default] [

Re: [ceph-users] slow requests are blocked

Hi, ceph health detail This will tell you which OSDs are experiencing the problem so you can then go and inspect the logs and use the admin socket to find out which requests are at the source. Regards JC > On May 7, 2018, at 03:52, Grigory Murashov wrote: > >

Re: [ceph-users] session lost, hunting for new mon / session established : every 30s until unmount/remount

Hi, if I read you crrectly you have 3 MONs on each data center. This means that when the link goes down you will loose quorum making the cluster unavailable. If my perception is correct, you’d have to start a 7th MON somewhere else accessible from both sites for your cluster to maintain quorum

Re: [ceph-users] Cannot delete a pool

Hi, connect to the ceph-node1 machine and run : ceph daemon mon.ceph-node1 config set mon_allow_pool_delete true You are just using the wrong parameter as an ID JC > On Mar 1, 2018, at 07:41, Max Cuttins wrote: > > I get: > > #ceph daemon mon.0 config set

Re: [ceph-users] Cannot Create MGR

Hi, looks like you haven’t run the ceph-deploy command with the same user name and may be not the same current working directory. This could explain your problem. Make sure the other daemons have a mgr cap authorisation. You can find on this ML details about MGR caps being incorrect for OSDs

Re: [ceph-users] radosgw not listening after installation

ause it can’t create the necessary pools. Only way you can trace that is by bumping up the trace with debug_rgw = 20 in the configuration file and from there you should be able to see where it sits and what it did. > > Let me know if I can send anything else, I’d really like to get this

Re: [ceph-users] radosgw not listening after installation

Hi, first of all just in case, it looks like your script does not deploy any OSDs as you go straight from MON to RGW. then, RGW does listen by default on 7480 and what you see on 6789 is the MON listening. Investigation: - Make sure your ceph-radosgw process is running first. - If not

Re: [ceph-users] Full Ratio

Hi, if you are using an older Ceph version note that the mon_osd_near_full_ration and mon_osd_full_ration must be set in the config file on the MON hosts first and then the MONs restarted one after the other one. If using a recent version there is a command ceph osd set-full-ratio and ceph

Re: [ceph-users] Hadoop on Ceph error

Hi, What’s your Hadoop xml config file like? Have you checked the permissions of the ceph.conf and keyring file? In case all good may be consider setting debug option in ceph.conf.options with Hadoop xml config file JC > On Jan 18, 2018, at 16:55, Bishoy Mikhael

Re: [ceph-users] ceph command hangs

Hi Nathan, I would have place the mon_host parameter and assigned it the IP address for your monitor host in the global section so that the client (ceph -s command). Have you also checked your firewall setup on your MON box? To help diagnose you can also use ceph -s --debug-ms=1 so you can

Re: [ceph-users] ceph status doesnt show available and used disk space after upgrade

OSD 140 > 2017-12-21 02:39:10.622835 7fb40a22b700 0 Cannot get stat of OSD 141 > Not sure whats wrong in my setup > > Regards, > Kevin > > >> On Thu, Dec 21, 2017 at 2:37 AM, Jean-Charles Lopez <jelo...@redhat.com> >> wrote: >> Hi, >>

Re: [ceph-users] ceph status doesnt show available and used disk space after upgrade

Hi, make sure client.admin user has an MGR cap using ceph auth list. At some point there was a glitch with the update process that was not adding the MGR cap to the client.admin user. JC > On Dec 20, 2017, at 10:02, kevin parrikar wrote: > > hi All, > I have

Re: [ceph-users] POOL_NEARFULL

_osd_nearfull_ratio = '0.86' (not observed, > change may require restart) > > > Karun Josy > > On Tue, Dec 19, 2017 at 10:05 PM, Jean-Charles Lopez <jelo...@redhat.com > <mailto:jelo...@redhat.com>> wrote: > OK so it’s telling you that the near full

Re: [ceph-users] POOL_NEARFULL

t; I have recently upgraded to Luminous 12.2.2, haven't seen this flag in 12.2.1 > > > > Karun Josy > > On Tue, Dec 19, 2017 at 9:27 PM, Jean-Charles Lopez <jelo...@redhat.com > <mailto:jelo...@redhat.com>> wrote: > Hi > > did you set quotas on these pool

Re: [ceph-users] POOL_NEARFULL

Hi did you set quotas on these pools? See this page for explanation of most error messages: http://docs.ceph.com/docs/master/rados/operations/health-checks/#pool-near-full JC > On Dec 19, 2017, at 01:48, Karun

Re: [ceph-users] radosgw: Couldn't init storage provider (RADOS)

Hi, try having a look at : - network connectivity issues - firewall configuration issues - missing or inaccessible keyring file for client.rgw.ceph-rgw1 - missing or inaccessible ceph.conf file Regards JC Lopez Senior Technical Instructor, Global Storage Consulting Practice Red Hat, Inc.

Re: [ceph-users] One OSD misbehaving (spinning 100% CPU, delayed ops)

Hi Mathhew, anything special happening on the NIC side that could cause a problem? Packet drops? Incorrect jumbo frame settings causing fragmentation? Have you checked the cstate settings on the box? Have you disabled energy saving settings differently from the other boxes? Any unexpected

Re: [ceph-users] How to set osd_max_backfills in Luminous

Hi, to check a current value use the following command on the machine where the OSD you want to check is running ceph daemon osd.{id} config show | grep {parameter} Or ceph daemon osd.{id} config get {parameter} What you are seeing is actually a known glitch where you are being told it has

Re: [ceph-users] Configuring ceph usage statistics

Hi Richard, you need to grant admin ops capabilities to a specific user to be able to query the usage stats. radosgw-admin caps add --caps "usage=*;buckets=*;metadata=*;users=*;zone=*" --uid=johndoe * can be replace with “read”, “read, write” depending on what you want the user to be able to

Re: [ceph-users] Not able to start OSD

Hi, have you checked the output of "ceph-disk list” on the nodes where the OSDs are not coming back on? This should give you a hint on what’s going one. Also use dmesg to search for any error message And finally inspect /var/log/ceph/ceph-osd.${id}.log to see messages produced by the OSD

Re: [ceph-users] To check RBD cache enabled

>>> [client] >>> admin socket = /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok >>> log file = /var/log/ceph/client.log >>> debug rbd = 20 >>> debug librbd = 20 >>> rbd_non_blocking_aio = false >>> -- >>> >>> >>> Howe

Re: [ceph-users] To check RBD cache enabled

phuser@ceph-las-admin-a1 ceph-cluster]$ sudo ceph --admin-daemon > /var/run/ceph/ceph-mgr.ceph-las-admin-a1.asok config show | grep > "rbd_non_blocking_aio" > "rbd_non_blocking_aio": "true", > --- > > Did I miss something ? > > On

Re: [ceph-users] To check RBD cache enabled

Hi syntax uses the admin socket file : ceph --admin-daemon /var/run/ceph/$cluster-$type.$id.$pid.$cctid.asok config get rbd_cache Should be /var/run/ceph/ceph.client.admin.$pid.$cctid.asok if your connection is using client.admin to connect to the cluster and your cluster name is set to the

Re: [ceph-users] Possible to change the location of run_dir?

Hi use the run_dir parameter in your /etc/ceph/ceph.conf file. It defaults to /var/run/ceph Or use admin_socket = /full/path/to/admin/socket for each daemon or in the global section using environment variable such as $type, $id, $cluster and so on Regards JC > On Sep 20, 2017, at 11:27,

Re: [ceph-users] Ceph fails to recover

Hi, you can play with the following 2 parameters: osd_recovery_max_active osd_max_backfills The higher the number the higher the number of PGs being processed at the same time. Regards Jean-Charles LOPEZ jeanchlo...@mac.com JC Lopez Senior Technical Instructor, Global Storage Consulting

Re: [ceph-users] Rgw install manual install luminous

Hi, see comment in line Regards JC > On Sep 12, 2017, at 13:31, Marc Roos wrote: > > > > I have been trying to setup the rados gateway (without deploy), but I am > missing some commands to enable the service I guess? How do I populate > the

Re: [ceph-users] too few PGs per OSD (16 < min 30) but I set pool_default_pg_num: 300 in Ansible

Hi, see comments below. JC > On Jun 14, 2017, at 07:23, Stéphane Klein wrote: > > Hi, > > I have this parameter in my Ansible configuration: > > pool_default_pg_num: 300 # (100 * 6) / 2 = 300 > > But I have this error: > > # ceph status > cluster

Re: [ceph-users] ONE pg deep-scrub blocks cluster

How Mehmet OK so it does come from a rados put. As you were able to check the VM device objet size is 4 MB. So we'll see after you have removed the object with rados -p rbd rm. I'll wait for an update. JC While moving. Excuse unintended typos. > On Aug 29, 2016, at 14:34, Mehmet

Re: [ceph-users] deep scrubbing causes osd down

Hi andrei There is one parameter, osd_max_scrub I think, that controls the number of scrubs per OSD. But the default is 1 if I'm correct. Can you check on one of your OSDs with the admin socket? Then it remains the option of scheduling the deep scrubs via a cron job after setting

Re: [ceph-users] Directly connect client to OSD using HTTP

Hi There is no limit on the number of gateways you can deploy. So just put a load balancer in front of many of them to avoid SPOF and bottleneck. Rgds JC While moving. Excuse unintended typos. On Mar 28, 2015, at 08:22, c...@jack.fr.eu.org wrote: Hi, I am designing an infrastructure

Re: [ceph-users] Question about primary OSD of a pool

Hi You can verify the exact mapping using the following command: ceph osd map {poolname} {objectname} Check page http://docs.ceph.com/docs/master/man/8/ceph for the ceph command. Cheers JC While moving. Excuse unintended typos. On Feb 1, 2015, at 08:04, Loic Dachary l...@dachary.org wrote:

Re: [ceph-users] Having an issue with: 7 pgs stuck inactive; 7 pgs stuck unclean; 71 requests are blocked 32

Hi Glen Run a ceph pg {id} query on one of your stuck PGs to find out what the PG is waiting for to be completed. Rgds JC On Friday, January 23, 2015, Glen Aidukas gaidu...@behaviormatrix.com wrote: Hello fellow ceph users, I ran into a major issue were two KVM hosts will not start due

Re: [ceph-users] Question about ceph-deploy

http://ceph.com/ceph-deploy/docs So I guess no need for a documentation update ;-) Cheers JC On Nov 26, 2014, at 23:26, mail list louis.hust...@gmail.com wrote: Thanks JC， It works, and i think ceph should modify the manual. On Nov 27, 2014, at 13:59, Jean-Charles LOPEZ jc.lo

Re: [ceph-users] Question about ceph-deploy

Hi Louis, ceph-deploy install —release=giant admin-node Cheers JC On Nov 26, 2014, at 20:38, mail list louis.hust...@gmail.com wrote: ceph-deploy install admin-node ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] evaluating Ceph

Hi, Use ceph mds newfs {metaid} {dataid} instead JC On Nov 25, 2014, at 12:27, Jeripotula, Shashiraj shashiraj.jeripot...@verizon.com wrote: Hi All, I am evaluating Ceph for one of our product requirements. I have gone through the website, http://ceph.com/docs/master/start/

Re: [ceph-users] Ceph fs has error: no valid command found; 10 closest matches: fsid

Hi, Use ceph mds newfs {metaid} {dataid} instead JC On Nov 20, 2014, at 20:43, Huynh Dac Nguyen ndhu...@spsvietnam.vn wrote: Hi All, When i create a new cephfs, this error shows Is it a bug? [root@ho-srv-ceph-02 ceph]# ceph fs new cephfs cephfs_metadata cephfs_data no valid

Re: [ceph-users] slow requests/blocked

Hi Jeff, it would probably wise to first check what these slow requests are: 1) ceph health detail - This will tell you which OSDs are experiencing the slow requests 2) ceph daemon osd.{id} dump_ops_in_flight - To be issued on one of the above OSDs will tell you what theses ops are waiting for.

Re: [ceph-users] bucket cleanup speed

Hi, this is an old thing I remember of and it may not be exactly related but just in case, let’s try it out. Just verify the value of the following parameter rgw_gc_max_objs in the RGW configuration via the admin socket (cep daemon {row.id} config get rgw_gc_max_objs . If the value is 32,

Re: [ceph-users] Cache Tier Statistics

Hi Nick If my brain doesn't fail me you can try ceph daemon osd.{id} perf dump ceph report (not 100% sure if cache stats are in Rgds JC On Saturday, November 8, 2014, Nick Fisk n...@fisk.me.uk wrote: Hi, Does anyone know if there any statistics available specific to the cache tier

Re: [ceph-users] Installing CephFs via puppet

Hi, with ceps-deploy do the following 1) Install ceph-deploy 2) mkdir ~/ceph-deploy 3) cd ~/ceph-deploy 4) ceph-deploy --overwrite-conf config pull {monitorhostname} 5) If version is Giant a) ceph osd pool create cephfsdata b) ceph odd pool create cephfsmeta xxx c) ceph mds newfs

Re: [ceph-users] Best practice about using multiple disks on one single OSD

Hi James, the best practice is to set up 1 OSD daemon per physical disk drive. In your case, each OSD node would hence be 4 OSD daemons using one physical drive per daemon, and deploying a minimum of 3 servers so each object copy resides on a separate physical server. JC On Sep 25, 2014,

Re: [ceph-users] Cache pool stats

Hi ceph daemon osd.x perf dump will show you the stats Andrei JC On Monday, September 15, 2014, Andrei Mikhailovsky and...@arhont.com wrote: Hi Does anyone know how to check the basic cache pool stats for the information like how well the cache layer is working for a recent or historic

Re: [ceph-users] Cache tier unable to auto flush data to storage tier

-pool 16 651 0 5 (Why it is showing 5 objects , while earlier its showing 3 Objects , why it has increased by 2 on adding only 1 object ) [root@ceph-node1 ~]# - Karan - On 14 Sep 2014, at 03:42, Jean-Charles

Re: [ceph-users] Cache tier unable to auto flush data to storage tier

Hi Karan, May be setting the dirty byte ratio (flush) and the full ratio (eviction). Just try to see if it makes any difference - cache_target_dirty_ratio .1 - cache_target_full_ratio .2 Tune the percentage as desired relatively to target_max_bytes and target_max_objects. The first threshold

Re: [ceph-users] CephFS mounting error

Hi Erick, the address to use in the mount syntax is the address of your MON node, not the one of the MDS node. Or may be you have deployed both a MON and an MDS on ceph01? JC On Sep 12, 2014, at 18:41, Erick Ocrospoma zipper1...@gmail.com wrote: On 12 September 2014 20:32, JIten Shah

Re: [ceph-users] radosgw ERROR: can't get key: ret=-2

Hi Benjamin, code extract sync_all_users() erroring is the sync of user stats /* * thread, full sync all users stats periodically * * only sync non idle users or ones that never got synced before, this is needed so that * users that didn't have quota turned on before (or existed before the

Re: [ceph-users] HEALTH_WARN pool has too few pgs

Hi Eric, increase the number of PGs in your pool with Step 1: ceph osd pool set poolname pg_num newvalue Step 2: ceph osd pool set poolname pgp_num newvalue You can check the number of PGs in your pool with ceph osd dump | grep ^pool See documentation:

Re: [ceph-users] Is it still unsafe to map a RBD device on an OSD server?

Hi Sébastien, still the case. Depending on what you do, the OSD process will get to a hang and will suicide. Regards JC On Jun 10, 2014, at 09:46, Sebastien Han sebastien@enovance.com wrote: Hi all, A couple of years ago, I heard that it wasn’t safe to map a krbd block on an OSD

Re: [ceph-users] Mounting CephFS RO?

Hi Shawn, just use the standard option=ro (-o ro) Cheers JC On May 29, 2014, at 20:10, Shawn Edwards lesser.e...@gmail.com wrote: Is it possible to have a CephFS mounted as RW on one machine and RO on another machine? We have a use case where we would have one box writing files to

Re: [ceph-users] CephFS block size

Hi Sherry go to this page in order to look at the description for show_layout and set_layout which will do what you are seeking for: https://ceph.com/docs/master/man/8/cephfs/ Cheers JC On May 24, 2014, at 21:49, Sherry Shahbazi shoo...@yahoo.com wrote: Hi all, I appreciate if anyone

Re: [ceph-users] Performance stats

Hi, fio is probably the tool you are looking that supports both RBD images and other disk devices for testing so that you can bench your Ceph cluster as week as compare it with other devices. Cheers JC On May 14, 2014, at 23:37, yalla.gnan.ku...@accenture.com wrote: Hi All, Is there a

Re: [ceph-users] Ceph not replicating

Hi Do tou have chooseleaf type host or type node in your crush map? How many OSDs do you run on each hosts? Thx JC On Saturday, April 19, 2014, Gonzalo Aguilar Delgado gagui...@aguilardelgado.com wrote: Hi, I'm building a cluster where two nodes replicate objects inside. I found that

Re: [ceph-users] Ceph not replicating

Hi again Looked at your ceph -s. You have only 2 OSDs, one on each node. The default replica count is 2, the default crush map says each replica on a different host, or may be you set it to 2 different OSDs. Anyway, when one of your OSD goes down, Ceph can no longer find another OSDs to host the

Re: [ceph-users] pgs stuck unclean in a pool without name

Hi Cedric, use the radios command to remove the empty pool name if you need to. rados rmpool ‘’ ‘’ —yes-i-really-really-mean-it You won’t be alb to remove it with the ceph command JC On Apr 18, 2014, at 03:51, Cedric Lemarchand ced...@yipikai.org wrote: Hi, I am facing a strange

Re: [ceph-users] radosgw-agent won't run

What is the status of your PGs on the slave zone side. A down or stale PG could definitely cause this. May be a quick ceph -s and ceph health detail could help locate the PG with a problem that could may be then help you get the correct ceph pg {pgid} query command to find out which OSD is

Re: [ceph-users] Ceph: Error librbd to create a clone

Hi Patrick When you call the clone method, add an extra argument specifying the features to be used for the clone: 1 for layering only 3 for layering and striping Adapt the value to your requirements Then it should work. JC On Monday, March 31, 2014, COPINE Patrick patrick.cop...@gmail.com

Re: [ceph-users] Ceph: Error librbd to create a clone

Hi Patrick, when you clone, you clone from a snapshot. This snapshot must be protected and above all, you RBD image must be format 2. Are all of these elements true? JC On Mar 28, 2014, at 09:30, COPINE Patrick patrick.cop...@gmail.com wrote: Hi, I would like create a clone with a

Re: [ceph-users] Remove volume

Hi rbd -p poolname rm imagename e.g. rbd -p volumes rm volume-55abf0d4-01a7-41d0-9e7e-407ad0db213c http://ceph.com/docs JC On Wednesday, March 12, 2014, yalla.gnan.ku...@accenture.com wrote: Hi All, I have ceph installed on Ubuntu nodes. At present I have the following volumes in

Re: [ceph-users] how to configure ceph object gateway

Hi, what commands are “not found”? This page for configuring the RGW works fine as far as I know as I used it no later than a week ago. Can you please give us more details? What is your layout (radosgw installed on a ceph node, mon node, standalone node)? Note: In order to get it running,

Re: [ceph-users] rbd create ... STRIPINGV2 and format 2 or later required

Hi Dieter, you have a problem with your command. You set order = 16 so your RBD objects is going to be 65536 bytes Then you tell RBD that you stripe-unit is going to be 65536 which is the size of your full object. Either decrease the size of —stripe-unit to 8192 for example Or increase order

Re: [ceph-users] rbd create ... STRIPINGV2 and format 2 or later required

Hi Dieter, you have a problem with your command. You set order = 16 so your RBD objects is going to be 65536 bytes Then you tell RBD that you stripe-unit is going to be 65536 which is the size of your full object. Either decrease the size of —stripe-unit to 8192 for example Or increase order

Re: [ceph-users] rbd create ... STRIPINGV2 and format 2 or later required

...@inktank.com wrote: If the stripe size and object size are the same it's just chunking -- that's our default. Should work fine. -Greg Software Engineer #42 @ http://inktank.com | http://ceph.com On Tue, Mar 11, 2014 at 8:23 AM, Jean-Charles LOPEZ jc.lo...@inktank.com wrote: Hi Dieter, you

Re: [ceph-users] how to configure ceph object gateway

Hi, what commands are “not found”? This page for configuring the RGW works fine as far as I know as I used it no later than a week ago. Can you please give us more details? What is your layout (radosgw installed on a ceph node, mon node, standalone node)? Note: In order to get it running,

Re: [ceph-users] how to configure ceph object gateway

rgw.conf sudo a2dissite default These 2 commands was not found in the CentOS. I can start /etc/init.d/ceph-radosgw and create a gateway user.But when I used API, it shows 403 forbidden. I didn't add wildcart to DNS, because I didn't use domain. 在 2014-03-11 10:47:57，Jean-Charles LOPEZ

Re: [ceph-users] osd down

Hi Pavel, It looks like you have deployed your 2 OSDs on the same host. By default, in the CRUSH map, each object is going to be assigned ti 2 OSDs that are on different host. If you want this to work for testing, you’ll have to adapt your CRUSH map so that each copy is dispatch on a bucket

Re: [ceph-users] Use of daily-created/-deleted pools

Hi Lee You could use an Ceph RBD device on a server and export a directory that you would have created on this RBD though NFS. 3 days after the files are uploade, you could snapshot the RBD device, delete the directory containing the files, and a week later, when sure you do not need the