Re: [ceph-users] How to remove a faulty bucket?

Hi! There were originally about 110k objects, but after several bucket check attempts, they "multiplied" and we're at 330k now (I assume it took the original objects, tried to create a new index, crashed, and left both the old an new "entries" or whatever there). Could it relate to bucket

Re: [ceph-users] Calamari ( what a nightmare !!! )

Hi! I think Calamari is more or less deprecated now that ceph luminous is out with Ceph Manager and the dashboard plugin: http://docs.ceph.com/docs/master/mgr/dashboard/ You could also try out: https://www.openattic.org/ or if you want to start a whole new

[ceph-users] Calamari ( what a nightmare !!! )

Hello list, Newbie here, After managing to install ceph, with all possible ways that I could manage  on 4 nodes, 4 osd and 3 monitors , with ceph-deploy and latter with ceph-ansible, I thought to to give a try to install CALAMARI on UBUNTU 14.04 ( another separate server being not a node or

Re: [ceph-users] Recommendations for I/O (blk-mq) scheduler for HDDs and SSDs?

Hi Patrick, Some thoughts about blk-mq: *(virtio-blk)* - it's activated by default on kernels >= 3.13 on driver virtio-blk - *The blk-mq feature is currently implemented, and enabled by default, in the following drivers: virtio-blk, mtip32xx, nvme, and rbd*. (

[ceph-users] Recommendations for I/O (blk-mq) scheduler for HDDs and SSDs?

Hi, after reading a lot about I/O schedulers and performance gains with blk-mq, I switched to a custom 4.14.5 kernel with CONFIG_SCSI_MQ_DEFAULT enabled to have blk-mq for all devices on my cluster. This allows me to use the following schedulers for HDDs and SSDs: mq-deadline, kyber, bfq,

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

On 11/12/2017 15:13, Tobias Prousa wrote: Hi there, I'm running a CEPH cluster for some libvirt VMs and a CephFS providing /home to ~20 desktop machines. There are 4 Hosts running 4 MONs, 4MGRs, 3MDSs (1 active, 2 standby) and 28 OSDs in total. This cluster is up and running since the days

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

You were right. I thought, that mgr is just optional component. I fixed mgr startup and after that my ceph -s responded with correct state and ceph pg dump returns immediately correct output. Thank you for help, everything looks good. For other issues I noticed I will create new thread if

[ceph-users] High Load and High Apply Latency

Hi List, I've got a 5 OSD node cluster running hammer. All of the OSD servers are identical but one has about 3-4x higher load than the others and the OSD's in this node are reporting high apply latency. The cause of the load appears to be the OSD processes. About half of the OSD processes are

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

If you're running Luminous 12.x.x then the mgr daemon is responsible for the output of most commands that query the cluster. If you're having problems with commands not returning when you're querying the cluster, look at seeing up an mgr daemon. On Mon, Dec 11, 2017, 2:07 PM Jan Pekař - Imatic

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

Hi, thank you for response. I started mds manually and accessed cephfs, I'm not running mgr yet, it is not necessary. I just responded to mailing list. It looks, that dump from ceph is incorrect and cluster is "working somehow". So problem is different, that my mgr or mds is not running.

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

After some research it looks, that broken is mainly that dump from ceph -s and ceph pg dump (no response from that command) But I can access data on cephfs (data so far tried). So question is - why is that status stuck, how to fix that? Is there some mon? database to reset and refresh that

Re: [ceph-users] Cluster stuck in failed state after power failure - please help

It honestly just looks like your MDS and MGR daemons are not configured to start automatically. Try starting them manually and then if that fixes the things, go through and enable them to start automatically. Assuming you use systemctl the commands to check and fix this would be something like

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

Hi Zheng, I think I managed to understand what you supposed me to do. The highest inode which was reported to be set erronously to be free was quite exactly identical with the highest inode in output of "cephfs-table tool all list inode". So I used take_ino as you supposed with an max_ino

Re: [ceph-users] public/cluster network

Hi Roman Whilst you can define multiple subnets in the public network directive, the MONs still only bind to a single IP. Your clients need to be able to route to that IP. From what you're saying, 172.x.x.x/24 is an isolated network, so a client on the 10.x.x.x network is not going to be able to

Re: [ceph-users] Sudden omap growth on some OSDs

Hmm, this does all sound odd. Have you tried just restarting the primary OSD yet? That frequently resolves transient oddities like this. If not, I'll go poke at the kraken source and one of the developers more familiar with the recovery processes we're seeing here. -Greg On Fri, Dec 8, 2017 at

Re: [ceph-users] The way to minimize osd memory usage?

I ran an experiment with 1GB memory per OSD using Bluestore. 12.2.2 made a big difference. In addition, you should have a look at your max object size. It looks like you will see a jump in memory usage if a particular OSD happens to be the primary for a number of objects being written in

[ceph-users] Cluster stuck in failed state after power failure - please help

Hi all, hope that somebody can help me. I have home ceph installation. After power failure (it can happen in datacenter also) my ceph booted in non-consistent state. I was backfilling data on one new disk during power failure. First time it booted without some OSDs, but I fixed that. Now I

Re: [ceph-users] Luminous rgw hangs after sighup

There have been other issues related to hangs during realm reconfiguration, ex http://tracker.ceph.com/issues/20937. We decided to revert the use of SIGHUP to trigger realm reconfiguration in https://github.com/ceph/ceph/pull/16807. I just started a backport of that for luminous. On

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

Hi Zheng, I did some more tests with cephfs-table-tool. I realized that disaster recovery implies to possibly reset inode table completely besides doing a session reset using something like cephfs-table-tool all reset inode Would that be close to what you suggested? Is it safe to reset

Re: [ceph-users] Luminous rgw hangs after sighup

That's the issue I remember (#20763)! The hang happened to me once, on this cluster, after upgrade from jewel to 12.2.2; then on Friday I disabled automatic bucket resharding due to some other problems - didn't get any logrotate-related hangs through the weekend. I wonder if these could be

Re: [ceph-users] How to remove a faulty bucket?

On Mon, Dec 11, 2017 at 09:29:11AM +, Martin Emrich wrote: > Hi! > > Am 09.12.17, 00:19 schrieb "Robin H. Johnson" : > > If you use 'radosgw-admin bi list', you can get a listing of the raw > bucket > index. I'll bet that the objects aren't being shown at the S3

Re: [ceph-users] questions about rbd image

Well connecting an rbd to two servers would be like mapping a block device from a storage array onto two different hosts, that's is possible and (was) done. (it would be much more difficult though to connect a single physical harddisk to two computers) The point is that as mentioned above you

Re: [ceph-users] Luminous, RGW bucket resharding

On 11-12-17 16:23, Orit Wasserman wrote: > On Mon, Dec 11, 2017 at 4:58 PM, Sam Wouters wrote: >> Hi Orrit, >> >> >> On 04-12-17 18:57, Orit Wasserman wrote: >>> Hi Andreas, >>> >>> On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder >>> wrote:

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

Hi Zheng, On 12/11/2017 04:28 PM, Yan, Zheng wrote: On Mon, Dec 11, 2017 at 11:17 PM, Tobias Prousa wrote: These are essentially the first commands I did execute, in this exact order. Additionally I did a: ceph fs reset cephfs --yes-i-really-mean-it how many

Re: [ceph-users] luminous 12.2.2 traceback (ceph fs status)

Yes, it include all the available pools on the cluster: *# ceph df* GLOBAL: SIZE AVAIL RAW USED %RAW USED 53650G 42928G 10722G 19.99 POOLS: NAMEID USED %USED MAX AVAIL OBJECTS volumes 13 2979G

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

On Mon, Dec 11, 2017 at 11:17 PM, Tobias Prousa wrote: > > These are essentially the first commands I did execute, in this exact order. > Additionally I did a: > > ceph fs reset cephfs --yes-i-really-mean-it > how many active mds were there before the upgrading. > > Any

Re: [ceph-users] Luminous, RGW bucket resharding

On Mon, Dec 11, 2017 at 4:58 PM, Sam Wouters wrote: > Hi Orrit, > > > On 04-12-17 18:57, Orit Wasserman wrote: >> Hi Andreas, >> >> On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder >> wrote: >>> Hello, >>> With release 12.2.2 dynamic resharding

Re: [ceph-users] luminous 12.2.2 traceback (ceph fs status)

On Mon, Dec 11, 2017 at 3:13 PM, German Anders wrote: > Hi John, > > how are you? no problem :) . Unfortunately the error on the 'ceph fs status' > command is still happening: OK, can you check: - does the "ceph df" output include all the pools? - does restarting ceph-mgr

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

On 12/11/2017 04:05 PM, Yan, Zheng wrote: On Mon, Dec 11, 2017 at 10:13 PM, Tobias Prousa wrote: Hi there, I'm running a CEPH cluster for some libvirt VMs and a CephFS providing /home to ~20 desktop machines. There are 4 Hosts running 4 MONs, 4MGRs, 3MDSs (1 active,

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

On Mon, Dec 11, 2017 at 10:13 PM, Tobias Prousa wrote: > Hi there, > > I'm running a CEPH cluster for some libvirt VMs and a CephFS providing /home > to ~20 desktop machines. There are 4 Hosts running 4 MONs, 4MGRs, 3MDSs (1 > active, 2 standby) and 28 OSDs in total. This

Re: [ceph-users] luminous 12.2.2 traceback (ceph fs status)

Hi John, how are you? no problem :) . Unfortunately the error on the 'ceph fs status' command is still happening: *# ceph fs status* Error EINVAL: Traceback (most recent call last): File "/usr/lib/ceph/mgr/status/module.py", line 301, in handle_command return self.handle_fs_status(cmd)

Re: [ceph-users] luminous 12.2.2 traceback (ceph fs status)

On Mon, Dec 4, 2017 at 6:37 PM, German Anders wrote: > Hi, > > I just upgrade a ceph cluster from version 12.2.0 (rc) to 12.2.2 (stable), > and i'm getting a traceback while trying to run: > > # ceph fs status > > Error EINVAL: Traceback (most recent call last): > File

Re: [ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

On Mon, Dec 11, 2017 at 10:13 PM, Tobias Prousa wrote: > Hi there, > > I'm running a CEPH cluster for some libvirt VMs and a CephFS providing /home > to ~20 desktop machines. There are 4 Hosts running 4 MONs, 4MGRs, 3MDSs (1 > active, 2 standby) and 28 OSDs in total. This

Re: [ceph-users] Luminous, RGW bucket resharding

Hi Orrit, On 04-12-17 18:57, Orit Wasserman wrote: > Hi Andreas, > > On Mon, Dec 4, 2017 at 11:26 AM, Andreas Calminder > wrote: >> Hello, >> With release 12.2.2 dynamic resharding bucket index has been disabled >> when running a multisite environment >>

Re: [ceph-users] Stuck down+peering after host failure.

Hi, I found another possible cause for your problem: http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-pg/#placement-group-down-peering-failure I hope that I helped, Denes. On 12/11/2017 03:43 PM, Denes Dolhay wrote: Hi Aaron! There is an previous post about safely

Re: [ceph-users] Stuck down+peering after host failure.

Hi Aaron! There is an previous post about safely shutting down and restarting a cluster: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-April/017378.html To the problems at hand: What size were you using? Ceph can only obey the failure domain if it knows exactly which osd is

Re: [ceph-users] questions about rbd image

An RBD can only be mapped to a single client host. There is no way around this. An RBD at its core is a block device. Connecting an RBD to 2 servers would be like connecting a harddrive to 2 servers. On Mon, Dec 11, 2017 at 9:13 AM 13605702596 <13605702...@163.com> wrote: > hi Jason > thanks

Re: [ceph-users] Corrupted files on CephFS since Luminous upgrade

Hi, The ceph mds keeps all the capabilities for the files, however the clients modify the the rados data pool objects directly (they do not do the content modification threw the mds). IMHO IF the file (really) gets corrupted because a client write (not some corruption from the mds / osd)

Re: [ceph-users] questions about rbd image

hi Jasonthanks for your answer.there is one more question, that is: can we use rbd image to share data between two clients? one wirtes data, another just reads? thanks At 2017-12-11 21:52:54, "Jason Dillaman" wrote: >On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com

[ceph-users] Upgrade from 12.2.1 to 12.2.2 broke my CephFs

Hi there, I'm running a CEPH cluster for some libvirt VMs and a CephFS providing /home to ~20 desktop machines. There are 4 Hosts running 4 MONs, 4MGRs, 3MDSs (1 active, 2 standby) and 28 OSDs in total. This cluster is up and running since the days of Bobtail (yes, including CephFS). Now

[ceph-users] Stuck down+peering after host failure.

Morning All, I have a large-ish (16 node, 1100 osds) cluster I recent had to move from one DC to another. Before shutting everything down, I set noout, norecover, and nobackfill, thinking this would help everything stand back up again. Upon installation at the new DC, one of the nodes refused

Re: [ceph-users] questions about rbd image

On Mon, Dec 11, 2017 at 7:50 AM, 13605702...@163.com <13605702...@163.com> wrote: > hi > > i'm testing on rbd image. the are TWO questions that confused me. > ceph -v > ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) > uname -r > 3.10.0-514.el7.x86_64 > > (1) does rbd image

[ceph-users] questions about rbd image

hi i'm testing on rbd image. the are TWO questions that confused me. ceph -v ceph version 10.2.10 (5dc1e4c05cb68dbf62ae6fce3f0700e4654fdbbe) uname -r 3.10.0-514.el7.x86_64 (1) does rbd image supports multiple clients to write data simultaneously? if it supports, how can share data between

Re: [ceph-users] The way to minimize osd memory usage?

There’s probably multiple reasons. However I just wanted to chime in that I set my cache size to 1G and I constantly see OSD memory converge to ~2.5GB. In [1] you can see the difference between a node with 4 OSDs, v12.2.2, on the left; and a node with 4 OSDs v12.2.1 on the right. I really

[ceph-users] mgr dashboard and cull Removing data for x

Hi all, I'm playing with the dashboard module in 12.2.2 (and it's very cool!) but I noticed that some OSDs do not have metadata, e.g. this page: http://xxx:7000/osd/perf/74 Has empty metadata. I *am* able to see all the info with `ceph osd metadata 74`. I noticed in the mgr log we have:

Re: [ceph-users] How to remove a faulty bucket?

Hi! Am 09.12.17, 00:19 schrieb "Robin H. Johnson" : If you use 'radosgw-admin bi list', you can get a listing of the raw bucket index. I'll bet that the objects aren't being shown at the S3 layer because something is wrong with them. But since they are in the

[ceph-users] public/cluster network

Hi all, We would like to implement the following setup. Our cloud nodes (CNs) for virtual machines have two 10 Gbps NICs: 10.x.y.z/22 (routed through the backbone) and 172.x.y.z/24 (available only on servers within single rack). CNs and ceph nodes are in the same rack. Ceph nodes have two 10

Re: [ceph-users] Luminous rgw hangs after sighup

Hi! This sounds like http://tracker.ceph.com/issues/20763 (or indeed http://tracker.ceph.com/issues/20866). It is still present in 12.2.2 (just tried it). My workaround is to exclude radosgw from logrotate (remove "radosgw" from /etc/logrotate.d/ceph) from being SIGHUPed, and to rotate the