[ceph-users] _setup_block_symlink_or_file failed to create block symlink to spdk:5780A001A5KD: (17) File exists

hi, I am making enable spdk on ceph. I got the error below. Could someone could help me ? Thank you very much. 1. SPDK code will be compiled by default if(CMAKE_SYSTEM_PROCESSOR MATCHES "i386|i686|amd64|x86_64|AMD64|aarch64") option(WITH_SPDK "Enable SPDK" ON) else() option(WITH_SPDK "Enable

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

On Sat, Apr 28, 2018 at 10:25 AM, Oliver Freyermuth wrote: > Am 28.04.2018 um 03:55 schrieb Yan, Zheng: >> On Fri, Apr 27, 2018 at 11:49 PM, Oliver Freyermuth >> wrote: >>> Dear Yan Zheng, >>> >>> Am 27.04.2018 um 15:32 schrieb Yan,

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

Am 28.04.2018 um 03:55 schrieb Yan, Zheng: > On Fri, Apr 27, 2018 at 11:49 PM, Oliver Freyermuth > wrote: >> Dear Yan Zheng, >> >> Am 27.04.2018 um 15:32 schrieb Yan, Zheng: >>> On Fri, Apr 27, 2018 at 7:10 PM, Oliver Freyermuth >>>

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

On Fri, Apr 27, 2018 at 11:49 PM, Oliver Freyermuth wrote: > Dear Yan Zheng, > > Am 27.04.2018 um 15:32 schrieb Yan, Zheng: >> On Fri, Apr 27, 2018 at 7:10 PM, Oliver Freyermuth >> wrote: >>> Dear Yan Zheng, >>> >>> Am 27.04.2018 um

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

On Fri, Apr 27, 2018 at 11:49 PM, Oliver Freyermuth wrote: > Dear Yan Zheng, > > Am 27.04.2018 um 15:32 schrieb Yan, Zheng: >> On Fri, Apr 27, 2018 at 7:10 PM, Oliver Freyermuth >> wrote: >>> Dear Yan Zheng, >>> >>> Am 27.04.2018 um

Re: [ceph-users] Deleting an rbd image hangs

Do you have any reason for why the OSDs crash? Anything the logs? Can you provide an "rbd info noc_tobedeleted"? On Thu, Apr 26, 2018 at 9:24 AM, Jan Marquardt wrote: > Hi, > > I am currently trying to delete an rbd image which is seemingly causing > our OSDs to crash, but it

Re: [ceph-users] Where to place Block-DB?

With filestore, if the NVMe actually died and you were unable to flush the journal to the data part of the OSD, then you lost the full OSD as well. That part hasn't changed at all from filestore to bluestore. There have been some other tickets on the ML here that talk about using `dd` to replace

Re: [ceph-users] Deleting an rbd image hangs

This old [1] blog post about removing super large RBDs is not relevant if you're using object map on the RBDs, however it's method to manually delete an RBD is still valid. You can see if this works for you to manually remove the problem RBD you're having. [1]

Re: [ceph-users] trimming the MON level db

I'm assuming that the "very bad move" means that you have some PGs not in active+clean. Any non-active+clean PG will prevent your mons from being able to compact their db store. This is by design so that if something were to happen where the data on some of the copies of the PG were lost and

Re: [ceph-users] Backup LUKS/Dmcrypt keys

IIRC the dmcrypt keys in Jewel were moved to a partition on the OSD. You should be able to find the keys by mounting those partitions. That is assuming filestore. I don't know where they are for bluestore. On Wed, Apr 25, 2018 at 4:29 PM Kevin Olbrich wrote: > Hi, > > how can I

Re: [ceph-users] The mystery of sync modules

2018-04-27 17:33 GMT+02:00 Sean Purdy : > > Mimic has a new feature, a cloud sync module for radosgw to sync objects > to some other S3-compatible destination. > > This would be a lovely thing to have here, and ties in nicely with object > versioning and DR. But I am put

Re: [ceph-users] Multi-MDS Failover

On Thu, Apr 26, 2018 at 7:04 PM, Scottix wrote: > Ok let me try to explain this better, we are doing this back and forth and > its not going anywhere. I'll just be as genuine as I can and explain the > issue. > > What we are testing is a critical failure scenario and actually

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

Dear Yan Zheng, Am 27.04.2018 um 15:32 schrieb Yan, Zheng: > On Fri, Apr 27, 2018 at 7:10 PM, Oliver Freyermuth > wrote: >> Dear Yan Zheng, >> >> Am 27.04.2018 um 02:58 schrieb Yan, Zheng: >>> On Thu, Apr 26, 2018 at 10:00 PM, Oliver Freyermuth >>>

[ceph-users] The mystery of sync modules

Hi, Mimic has a new feature, a cloud sync module for radosgw to sync objects to some other S3-compatible destination. This would be a lovely thing to have here, and ties in nicely with object versioning and DR. But I am put off by confusion and complexity with the whole multisite/realm/zone

Re: [ceph-users] ceph 12.2.5 - atop DB/WAL SSD usage 0%

Could we infer from this if the usage model is large object sizes rather than small I/Os the benefit of offloading WAL/DB is questionable given that the failure of the SSD (assuming shared amongst HDDs) could take down a number of OSDs and in this case a best practice would be to collocate?

Re: [ceph-users] ceph-mgr not able to modify max_misplaced in 12.2.4

On Fri, Apr 27, 2018 at 7:03 AM, nokia ceph wrote: > Hi Team, > > I was trying to modify the max_misplaced parameter in 12.2.4 as per > documentation , however not able to modify it with following error, > > #ceph config set mgr mgr/balancer/max_misplaced .06 > Invalid

Re: [ceph-users] ceph 12.2.5 - atop DB/WAL SSD usage 0%

rados bench is using 4MB block size for io. Try with with io size 4KB, you will see ssd will be used for write operations. On Fri, Apr 27, 2018 at 4:54 PM, Steven Vacaroaia wrote: > Hi > > During rados bench tests, I noticed that HDD usage goes to 100% but SSD > stays at ( or

[ceph-users] ceph 12.2.5 - atop DB/WAL SSD usage 0%

Hi During rados bench tests, I noticed that HDD usage goes to 100% but SSD stays at ( or very close to 0) Since I created OSD with BLOCK/WAL on SSD, shouldnt I see some "activity' on SSD ? How can I be sure CEPH is actually using SSD for WAL /DB ? Note I only have 2 HDD and one SSD per

Re: [ceph-users] *** SPAM *** Re: Multi-MDS Failover

Hey Dan, Thanks you for the response, the namespace methodology makes more sense and I think that explains what would be up or not. In regards to my original email with number 4 of listing 0 files. I will try to recreate with debug on and submit an issue if that turns out to be a bug. I am

Re: [ceph-users] Inconsistent metadata seen by CephFS-fuse clients

Dear Yan Zheng, Am 27.04.2018 um 02:58 schrieb Yan, Zheng: > On Thu, Apr 26, 2018 at 10:00 PM, Oliver Freyermuth > wrote: >> Dear Cephalopodians, >> >> just now that our Ceph cluster is under high I/O load, we get user reports >> of files not being seen on some

Re: [ceph-users] cluster can't remapped objects after change crush tree

Thanks a lot for your help. Konstantin Shalygin writes: > On 04/27/2018 05:05 PM, Igor Gajsin wrote: >> I have a crush rule like > > > You still can use device classes! > > >> * host0 has a piece of data on osd.0 > Not peace, full object. If we talk about non-EC pools. >> * host1 has pieces of

Re: [ceph-users] cluster can't remapped objects after change crush tree

On 04/27/2018 05:05 PM, Igor Gajsin wrote: I have a crush rule like You still can use device classes! * host0 has a piece of data on osd.0 Not peace, full object. If we talk about non-EC pools. * host1 has pieces of data on osd.1 and osd.2 host1 has copy on osd.1 *or* osd.2 * host2 has

Re: [ceph-users] cluster can't remapped objects after change crush tree

Thanks, man. Thanks a lot. Now I'm understood. So, to be sure If I have 3 hosts, replicating factor is also 3 and I have a crush rule like: { "rule_id": 0, "rule_name": "replicated_rule", "ruleset": 0, "type": 1, "min_size": 1, "max_size": 10, "steps": [ {

Re: [ceph-users] cluster can't remapped objects after change crush tree

On 04/27/2018 04:37 PM, Igor Gajsin wrote: pool 7 'rbd' replicated size 3 min_size 2 crush_rule 0 Your pools have proper size settings - is 3. But you crush have only 2 buckets for this rule (e.g. is your pods). For making this rule work you should have minimum of 3 'pod' buckets. k

Re: [ceph-users] cluster can't remapped objects after change crush tree

# ceph osd pool ls detail pool 1 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 last_change 958 lfor 0/909 flags hashpspool stripe_width 0 application cephfs pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash

Re: [ceph-users] How to deploy ceph with spdk step by step?

Hi Nathan Cutler,Orlando Moreno, Loic Dachary and Sage Weil, I am making spdk enable on ceph. But I failed. My step is listed as below. Could you help check if all the step is right? And

Re: [ceph-users] Collecting BlueStore per Object DB overhead

I've writting a piece of Python code which can be run on a server running OSDs and will print the overhead. https://gist.github.com/wido/b1328dd45aae07c45cb8075a24de9f1f Feedback on this script is welcome, but also the output of what people are observing. For mixed (filestore / bluestore)

Re: [ceph-users] cluster can't remapped objects after change crush tree

On 04/26/2018 11:30 PM, Igor Gajsin wrote: after assigning this rule to a pool it stucks in the same state: `ceph osd pool ls detail` please k ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] ceph osd reweight (doing -1 or actually -0.0001)

Thanks Paul for the explanation, sounds very logical now. -Original Message- From: Paul Emmerich [mailto:paul.emmer...@croit.io] Sent: woensdag 25 april 2018 20:28 To: Marc Roos Cc: ceph-users Subject: Re: [ceph-users] ceph osd reweight (doing -1 or actually -0.0001) Hi, the

Re: [ceph-users] Collecting BlueStore per Object DB overhead

Hi Wido, thanks for the tool. Here are some stats from our cluster: Ceph 12.2.4, 240 OSDs, CephFS only onodes db_used_bytes avg_obj_sizeoverhead_per_obj Mean214871 1574830080 2082298 7607 Max 309855 3018850304 3349799 17753

Re: [ceph-users] *** SPAM *** Re: Multi-MDS Failover

Hi Scott, Multi MDS just assigns different parts of the namespace to different "ranks". Each rank (0, 1, 2, ...) is handled by one of the active MDSs. (You can query which parts of the name space are assigned to each rank using the jq tricks in [1]). If a rank is down and there are no more

[ceph-users] ceph-mgr not able to modify max_misplaced in 12.2.4

Hi Team, I was trying to modify the max_misplaced parameter in 12.2.4 as per documentation , however not able to modify it with following error, #ceph config set mgr mgr/balancer/max_misplaced .06 Invalid command: unused arguments: [u'.06'] config set : Set a configuration option at runtime