[ceph-users] ceph-bluestore-tool failed

Hi all, We deployed a testing mimic CEPH cluster using bluestore.We can't run ceph-bluestore-tool on OSD with following error: --- # ceph-bluestore-tool show-label --dev *device* 2018-10-31 09:42:01.712 7f3ac5bb4a00 -1 auth: unable to find a keyring on

Re: [ceph-users] Using FC with LIO targets

On 10/28/2018 03:18 AM, Frédéric Nass wrote: > Hello Mike, Jason, > > Assuming we adapt the current LIO configuration scripts and put QLogic HBAs > in our SCSI targets, could we use FC instead of iSCSI as a SCSI transport > protocol with LIO ? Would this still work with multipathing and ALUA ?

Re: [ceph-users] Removing MDS

That is what I though. I am increasing debug to see where we are getting stuck. I am not sure if it is an issue deactivating or a rdlock issue. Thanks if we discover more we will post a question with details. Rhian Resnick Associate Director Research Computing Enterprise Systems Office of

Re: [ceph-users] Removing MDS

On Tue, Oct 30, 2018 at 4:05 PM Rhian Resnick wrote: > We are running into issues deactivating mds ranks. Is there a way to safely > forcibly remove a rank? No, there's no "safe" way to force the issue. The rank needs to come back, flush its journal, and then complete its deactivation. To get

Re: [ceph-users] Using FC with LIO targets

(CCing Mike since he knows more than me) On Sun, Oct 28, 2018 at 4:19 AM Frédéric Nass wrote: > > Hello Mike, Jason, > > Assuming we adapt the current LIO configuration scripts and put QLogic HBAs > in our SCSI targets, could we use FC instead of iSCSI as a SCSI transport > protocol with LIO ?

Re: [ceph-users] RBD: create imaged with qemu

Your use of "sudo" for the rados CLI tool makes me wonder if perhaps the "nstcc0" user cannot read "/etc/ceph/ceph.conf" or "/etc/ceph/ceph.admin.keyring". If that's not the case, what version of qemu-img are you using? $ rpm -qa | grep qemu-img qemu-img-2.11.2-4.fc28.x86_64 $ qemu-img create -f

[ceph-users] Removing MDS

Evening, We are running into issues deactivating mds ranks. Is there a way to safely forcibly remove a rank? Rhian Resnick Associate Director Research Computing Enterprise Systems Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL

[ceph-users] Filestore to Bluestore migration question

I am relatively new to Ceph and need some advice on Bluestore migration. I tried migrating a few of our test cluster nodes from Filestore to Bluestore by following this ( http://docs.ceph.com/docs/luminous/rados/operations/bluestore-migration/) as the cluster is currently running 12.2.9. The

Re: [ceph-users] reducing min_size on erasure coded pool may allow recovery ?

Thanks for the clarification! Glad to see this feature is being pursued. Chad. On 10/30/2018 12:24 PM, Gregory Farnum wrote: > On Mon, Oct 29, 2018 at 7:43 PM David Turner > wrote: > > min_size should be at least k+1 for EC. There are times to use k for >

Re: [ceph-users] Balancer module not balancing perfectly

I was having a difficult time getting debug logs from the active mgr, but I finally got it. Apparently injecting debug_mgr doesn't work, even when the change is reflected when you query the running config. Modifying the config file and restarting the mgr got it to log for me. Now that I have some

Re: [ceph-users] Packages for debian in Ceph repo

Hello, we provide a public mirror documented on https://croit.io/2018/09/23/2018-09-23-debian-mirror for Ceph Mimic on Debian Stretch. -- Martin Verges Managing director Mobile: +49 174 9335695 E-Mail: martin.ver...@croit.io Chat: https://t.me/MartinVerges croit GmbH, Freseniusstr. 31h, 81247

Re: [ceph-users] ceph-mds failure replaying journal

So a big thank you to @yanzheng for his help getting this back online The quick answer to what we did was downgrade to 13.2.1 as 13.2.2 is broken for cephfs restored the backup of the journal I’d taken as part of following the disaster recovery process documents turned off mds standby replay

Re: [ceph-users] node not using cluster subnet

Thanks for taking the trouble to provide advice I found that the Juniper switch port for the server that did not work did not have the MTU changed to 9200 I am using MTU 9000 for the cluster network Not sure why packet fragmentation created issues but ...all seems fine now Thanks Steven I On

Re: [ceph-users] reducing min_size on erasure coded pool may allow recovery ?

On Mon, Oct 29, 2018 at 7:43 PM David Turner wrote: > min_size should be at least k+1 for EC. There are times to use k for > emergencies like you had. I would suggest seeing it back to 3 once your > back to healthy. > > As far as why you needed to reduce min_size, my guess would be that >

Re: [ceph-users] node not using cluster subnet

The OSDs ping each other on both public and cluster networks. Perhaps the routing isn't working on the public network? Or maybe it's trying to ping from the cluster 192. network into the public 10. network and that isn't getting through? -Greg On Tue, Oct 30, 2018 at 8:34 AM Steven Vacaroaia

Re: [ceph-users] Packages for debian in Ceph repo

Hi! Proxmox has support for rbd as they ship additional packages as well as ceph via their own repo. I ran your command and got this: > qemu-img version 2.8.1(Debian 1:2.8+dfsg-6+deb9u4) > Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers > Supported formats: blkdebug

Re: [ceph-users] Packages for debian in Ceph repo

What version of qemu-img are you using? I found [1] this when poking around on my qemu server when checking for rbd support. This version (note it's proxmox) has rbd listed as a supported format. [1] # qemu-img -V; qemu-img --help|grep rbd qemu-img version 2.11.2pve-qemu-kvm_2.11.2-1 Copyright

Re: [ceph-users] Balancer module not balancing perfectly

I had played with those settings some already, but I just tried again with max_deviation set to 0.0001 and max_iterations set to 1000. Same result. Thanks for the suggestion though. Steve Taylor | Senior Software Engineer | StorageCraft Technology Corporation 380 Data Drive Suite 300 | Draper

Re: [ceph-users] Packages for debian in Ceph repo

Is it possible to use qemu-img with rbd support on Debian Stretch? I am on Luminous and try to connect my image-buildserver to load images into a ceph pool. root@buildserver:~# qemu-img convert -p -O raw /target/test-vm.qcow2 > rbd:rbd_vms_ssd_01/test_vm > qemu-img: Unknown protocol 'rbd' Kevin

Re: [ceph-users] Balancer module not balancing perfectly

>From the balancer module's code for v 12.2.7 I noticed [1] these lines which reference [2] these 2 config options for upmap. You might try using more max iterations or a smaller max deviation to see if you can get a better balance in your cluster. I would try to start with [3] these

[ceph-users] RBD: create imaged with qemu

Hi all, I follow below guide to create images with qemu-rbd: qemu-img create -f raw rbd:quick_rbd_test/own_image 5G; http://docs.ceph.com/docs/master/rbd/qemu-rbd/ However, it always shows "connect error". Does anyone know how to resolve the problem? The info is below:

[ceph-users] node not using cluster subnet

Hi, I am trying to add another node to my cluster which is configured to use a dedicated subnet public_network = 10.10.35.0/24 cluster_network = 192.168.200.0/24 For whatever reason, this node is staring properly and few seconds later is failing and staring to check for connectivity on public

[ceph-users] New us-central mirror request

Hi all, We are GigeNET, a datacenter based in Arlington Heights, IL (close to Chicago). We are starting to mirror ceph and would like to become and official mirror. We meet all of the requirements and have 2x bonded 1Gbps NICs. http://mirrors.gigenet.com/ceph/ Regards, Zachary Muller Systems

[ceph-users] Balancer module not balancing perfectly

I have a Luminous 12.2.7 cluster with 2 EC pools, both using k=8 and m=2. Each pool lives on 20 dedicated OSD hosts with 18 OSDs each. Each pool has 2048 PGs and is distributed across its 360 OSDs with host failure domains. The OSDs are identical (4TB) and are weighted with default weights (3.73).

Re: [ceph-users] Large omap objects - how to fix ?

Hi hijackers, Please read: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-October/030317.html TL;DR: Ceph should reshard big indexes, but after that it leaves them to be removed manually. Starting from some version, deep-scrub reports indexes above some threshold as HALTH_WARN.

Re: [ceph-users] OSD node reinstallation

Basically it's a good idea to backup your /etc/ceph/ folder to reinstall the node. Most everything you need will be in there for your osds. On Tue, Oct 30, 2018, 6:01 AM Luiz Gustavo Tonello < gustavo.tone...@gmail.com> wrote: > Thank you guys, > > It'll save me a bunch of time, because the

Re: [ceph-users] is it right involving cap->session_caps without lock protection in the two functions ?

> On Oct 30, 2018, at 18:10, ? ? wrote: > > Hello: > Recently, we have encountered a kernel dead question, and the reason > we analyses vmcore dmesg is that list_add_tail(>session_caps) in > __ceph_remove_cap has wrong,since >session_cap is NULL! > so we analyses codes with

Re: [ceph-users] Reducing Max_mds

John, Thanks! Rhian Resnick Associate Director Research Computing Enterprise Systems Office of Information Technology Florida Atlantic University 777 Glades Road, CM22, Rm 173B Boca Raton, FL 33431 Phone 561.297.2647 Fax 561.297.0222 [image]

Re: [ceph-users] OSD node reinstallation

Thank you guys, It'll save me a bunch of time, because the process to reallocate OSD files is not so fast. :-) On Tue, Oct 30, 2018 at 6:15 AM Alexandru Cucu wrote: > Don't forget about the cephx keyring if you are using cephx ;) > > Usually sits in: >

Re: [ceph-users] Reducing Max_mds

On Tue, Oct 30, 2018 at 6:36 AM Rhian Resnick wrote: > > Evening, > > > I am looking to decrease our max mds servers as we had a server failure and > need to remove a node. > > > When we attempt to decrease the number of mds servers from 5 to 4 (or any > other number) they never transition to

Re: [ceph-users] OSD node reinstallation

Don't forget about the cephx keyring if you are using cephx ;) Usually sits in: /var/lib/ceph/bootstrap-osd/ceph.keyring --- Alex On Tue, Oct 30, 2018 at 4:48 AM David Turner wrote: > > Set noout, reinstall the OS without going the OSDs (including any journal > partitions and maintaining

Re: [ceph-users] ceph-deploy with a specified osd ID

ceph-deploy doesn't support that. You can use ceph-disk or ceph-volume directly (with basically the same syntax as ceph-deploy), but you can only explicitly re-use an OSD id if you set it to destroyed before. I.e., the proper way to replace an OSD while avoiding unnecessary data movement is: ceph

[ceph-users] Fwd: Ceph Meetup Cape Town

Hi, SARAO is excited to announce that it will be hosting a Ceph Meetup in Cape Town. Date: Wednesday 28'th November Time: 5pm to 8pm Venue: Workshop 17 at the V Waterfront Space is limited, so if you would like to attend, please complete

Re: [ceph-users] slow_used_bytes - SlowDB being used despite lots of space free in BlockDB on SSD?

> > >> On 10/18/2018 7:49 PM, Nick Fisk wrote: > > >>> Hi, > > >>> > > >>> Ceph Version = 12.2.8 > > >>> 8TB spinner with 20G SSD partition > > >>> > > >>> Perf dump shows the following: > > >>> > > >>> "bluefs": { > > >>> "gift_bytes": 0, > > >>> "reclaim_bytes": 0, > > >>>

[ceph-users] Reducing Max_mds

Evening, I am looking to decrease our max mds servers as we had a server failure and need to remove a node. When we attempt to decrease the number of mds servers from 5 to 4 (or any other number) they never transition to standby. They just stay active. ceph fs set cephfs max_mds X