Re: [ceph-users] SSD-Cache Tier + RBD-Cache = Filesystem corruption?

On 02/10/2016 06:07 PM, Jason Dillaman wrote: > Can you provide the 'rbd info' dump from one of these corrupt images? > sure, rbd image 'ldap01.root.borked': size 2 MB in 5000 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.18394b3d1b58ba

Re: [ceph-users] Xeon-D 1540 Ceph Nodes

The Supermicro 5018A-AR12L is built for object storage. In our testing, they perform pretty well. You would have to invest in discrete 10G nics to meet all of your requirements. Austin On Thu, Feb 11, 2016 at 8:04 PM, Schlacta, Christ wrote: > If you can swing 2u chassis

Re: [ceph-users] lstat() hangs on single file

On Thu, Feb 11, 2016 at 4:29 PM, Blade Doyle wrote: > > After several months of use without needing any administration at all, I > think I finally found something to debug. > > Attempting to "ls -l" within a directory on CephFS hangs - strace shows its > hanging on lstat():

[ceph-users] ceph 9.2.0 SAMSUNG ssd performance issue?

Hi, ceph VERY SLOW with 24 osd(SAMSUNG ssd). fio /dev/rbd0 iodepth=1 direct=1 IOPS only ~200 fio /dev/rbd0 iodepth=32 direct=1 IOPS only ~3000 But test single ssd deive with fio: fio iodepth=1 direct=1 IOPS ~15000 fio iodepth=32 direct=1 IOPS ~3 Why ceph SO SLOW? Could you give me

Re: [ceph-users] ceph 9.2.0 SAMSUNG ssd performance issue?

Hi. You need to read : https://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ С уважением, Фасихов Ирек Нургаязович Моб.: +79229045757 2016-02-12 10:41 GMT+03:00 Huan Zhang : > Hi, > > ceph VERY SLOW with 24

Re: [ceph-users] Dell Ceph Hardware recommendations

Michael, I wouldn't be concerned with SAS expanders - so long as you've got enough bandwidth to the HBA / RAID controller? The main consideration with the SSDs is the ratio to disk. When you loose an SSD all the OSDs journalling to that will be inconsistent, effectively off-lining them. Too

Re: [ceph-users] Separate hosts for osd and its journal

Thanks for all the help on this! Regards, Mavis On Wed, Feb 10, 2016 at 6:00 PM, Bill Sanders wrote: > Going in a tiny bit more detail to what Michał said, one of the key > reasons for having the journal (in particular, to use SSD's) is to > reduce latency on writes

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

I think I see the problem. It looks like you are performing ops directly against the cache tier instead of the base tier (assuming cache1 is your cache pool). Here are my steps against master where the object is successfully promoted upon 'rbd info': # ceph osd erasure-code-profile set

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

That class method flag to force promotion was to prevent proxying lock/unlock class ops. What was happening was that the cache tier OSD would become the owner of the lock instead of the client (retrieved via cls_get_request_origin). -- Jason Dillaman - Original Message - > From:

Re: [ceph-users] Dell Ceph Hardware recommendations

That PDF specifically calls for P3700 NVMe SSDs, not the consumer 750. You need high endurance drives usually. I’m using 1x400GB Intel P3700 per 9 OSDs (so 4xP3700 per 36 disk chassis). > On 11 Feb 2016, at 17:56, Michael wrote: > > Alex Leake writes: >

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

I'm trying to reproduce this. Jason, I found your commit marks certain cls ops and requiring promotion, but that doesn't include rbd... and I'm not sure why info would need to be promoted. Working on reproducing this under hammer with the appropriate recency settings. sage On Thu, 11 Feb

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

Hi Jason, It's in writeback mode. Nick > -Original Message- > From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel- > ow...@vger.kernel.org] On Behalf Of Jason Dillaman > Sent: 11 February 2016 18:40 > To: Nick Fisk > Cc: Sage Weil ; Samuel Just

Re: [ceph-users] why is there heavy read traffic during object delete?

I saw this go by in the commit log: commit cc2200c5e60caecf7931e546f6522b2ba364227f Merge: f8d5807 12c083e Author: Sage Weil Date: Thu Feb 11 08:44:35 2016 -0500 Merge pull request #7537 from ifed01/wip-no-promote-for-delete-fix osd: fix unnecessary object

Re: [ceph-users] Question: replacing all OSDs of one node in 3node cluster

Hi Mihai, Grüezi Ivan :) Thank both of you for the fast reply. Its appreciated. When I bootstrapped the cluster I used -- osd_pool_default_size = 3 osd_pool_default_min_size = 2 -- in ceph.conf. This is also set for each pool at the moment. I understood from docs this means each object is

[ceph-users] OpenStack Ops Mid-Cycle session on OpenStack/Ceph integration

A "mid-cycle summit" for OpenStack operators will be held in Manchester (England) on Monday/Tuesday next week (15/16 February). The morning session on Tuesday will include a slot on Ceph integration. If there are any Ceph+OpenStack operators, please have a look at the Etherpad with the draft

Re: [ceph-users] Dell Ceph Hardware recommendations

Hello Michael​, I maintain a small Ceph cluster at the University of Bath, our cluster consists of: Monitors: 3 x Dell PowerEdge R630 - 2x Intel(R) Xeon(R) CPU E5-2609 v3 - 64GB RAM - 4x 300GB SAS (RAID 10) OSD Nodes: 6 x Dell PowerEdge R730XD & MD1400 Shelves - 2x Intel(R) Xeon(R) CPU

Re: [ceph-users] SSD-Cache Tier + RBD-Cache = Filesystem corruption?

Assuming the partition table is still zeroed on that image, can you run: # rados -p get rbd_data.18394b3d1b58ba. - | cut -b 512 | hexdump Can you also provide your pool setup: # ceph report --format xml 2>/dev/null | xmlstarlet sel -t -c "//osdmap/pools" -- Jason Dillaman

[ceph-users] Graphing Ceph Latency with Graphite

Hi All, Thought I would just share this as it's taken me a couple of hours to work it out. I'm using Diamond+Carbon+Graphite to collect the perf dump output from the admin sockets. The Diamond collector is really good at capturing everything, but stores the sum+avgcount directly into Carbon. In

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

That’s a relief, I was sensing a major case of face palm occuring when I read Jason's email!!! > -Original Message- > From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel- > ow...@vger.kernel.org] On Behalf Of Sage Weil > Sent: 11 February 2016 21:00 > To: Jason Dillaman

[ceph-users] cancel or remove default pool rbd

I to all, someone have try to cancel rbd default pool? I have the cluster ceph in warning with stale create pg. Is possible cancel default rbd pool and remove all stale pg? Thanks to all for your reply. Andrea. ___ ceph-users mailing list

[ceph-users] Multipath devices with infernalis

All, I have a set of hardware with a few systems connected via IB along with a DDN SFA12K. There are 4 IB/SRP paths to each block device. Those show up as /dev/mapper/mpath[b-d] I am trying to do an initial install/setup of ceph on 3 nodes. Each will be a monitor as well as host a single OSD.

[ceph-users] lstat() hangs on single file

After several months of use without needing any administration at all, I think I finally found something to debug. Attempting to "ls -l" within a directory on CephFS hangs - strace shows its hanging on lstat(): open("/etc/group", O_RDONLY|O_CLOEXEC) = 4 fstat(4, {st_mode=S_IFREG|0644,

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

On Thu, 11 Feb 2016, Nick Fisk wrote: > That’s a relief, I was sensing a major case of face palm occuring when I > read Jason's email!!! https://github.com/ceph/ceph/pull/7617 The tangled logic in maybe_handle_cache wasn't respecting the force promotion bool.

Re: [ceph-users] cancel or remove default pool rbd

Hello Andrea, The question is why won't your PG's go into an active+clean state on the cluster? Are all of your OSD's up/in? Are you satisfying your CRUSH ruleset? Can you provide an output of 'ceph osd tree', 'ceph -s', 'ceph osd crush show-tunables' and your ceph.conf file. Thank you,

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

I was able to reproduce this on master: On Thu, 11 Feb 2016, Jason Dillaman wrote: > I think I see the problem. It looks like you are performing ops directly > against the cache tier instead of the base tier (assuming cache1 is your > cache pool). Here are my steps against master where the

Re: [ceph-users] getting rid of misplaced objects

What does 'ceph osd tree' look like for this cluster? Also have you done anything special to your CRUSH rules? I've usually found this to be caused by modifying OSD weights a little too much. As for the inconsistent PG, you should be able to run 'ceph pg repair' on it:

Re: [ceph-users] why is there heavy read traffic during object delete?

Looks like it to me, yeah. Not sure why it took so long to get noticed though (that is, is that behavior present in the release you're using, or is it a new bug)? -Greg On Thu, Feb 11, 2016 at 12:11 PM, Stephen Lord wrote: > > I saw this go by in the commit log: > >

[ceph-users] Xeon-D 1540 Ceph Nodes

Hi All, Has anyone seen or heard about any Xeon-D based barebones servers more suited for Ceph use. We are looking to roll out a 2nd cluster and would really like something based on these boards that is barebones and dense. There is a lot of fat in most servers that we just don't need and

[ceph-users] R: cancel or remove default pool rbd

Hi Micheal, ceph -s: cluster ea296c34-e9b0-4a53-a740-f0b472f0c81d health HEALTH_WARN 44 pgs degraded 64 pgs stale 44 pgs stuck degraded 64 pgs stuck inactive 64 pgs stuck stale 128 pgs stuck unclean

Re: [ceph-users] Multipath devices with infernalis [solved]

All, Ok, it was indeed me. Firewalld does not seem happy across boots when network manager is involved unless you use something like nm-connection-editor to put the nic in the zone you want... grrr Brian Andrus From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of

[ceph-users] cephx capabilities to forbid rbd creation

Hi! We are on version 9.2.0, 5 mons and 80 OSDS distributed on 10 hosts. How could we twist cephx capabilities so to forbid our KVM+QEMU+libvirt hosts any RBD creation capability ? We currently have an rbd-user key like so : caps: [mon] allow r caps: [osd] allow x

Re: [ceph-users] Xeon-D 1540 Ceph Nodes

If you can swing 2u chassis and 2.5" drives instead, you can trivially get between 15 and 24 drives across the front and rear of a beautiful hot-swap chassis. There are numerous makes and models available from custom builds up/down through used on ebay. Worth a peek. On Thu, Feb 11, 2016 at

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Is this only a problem with EC base tiers or would replicated base tiers see this too? - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Thu, Feb 11, 2016 at 6:09 PM, Sage Weil wrote: > On

Re: [ceph-users] Dell Ceph Hardware recommendations

Hello, As discussed on this very ML and by looking at the moderately well hidden full specifications, using any 7xx Intel SSD for journals is a fools errand. Their endurance is far too low in comparison to their IOPS capacity. Regards, Chibi On Thu, 11 Feb 2016 15:15:12 + Alex Leake

Re: [ceph-users] Dell Ceph Hardware recommendations

Alex Leake writes: > > Hello Michael​, > > I maintain a small Ceph cluster at the University of Bath, our cluster consists of: > > Monitors: > 3 x Dell PowerEdge R630 > > - 2x Intel(R) Xeon(R) CPU E5-2609 v3 > - 64GB RAM > - 4x 300GB SAS (RAID 10) > > OSD Nodes: > 6 x

[ceph-users] getting rid of misplaced objects

Hi, Are there any tips and tricks around getting rid of misplaced objects? I did check the archive but didn’t find anything. Right now my cluster looks like this: pgmap v43288593: 16384 pgs, 4 pools, 45439 GB data, 10383 kobjects 109 TB used, 349 TB / 458 TB avail

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

Hi Sage, Do you think this will get fixed in time for the Jewel release? It still seems to happen in Master and is definitely related to the recency setting. I'm guessing that the info command does some sort of read and then a write. In the old behaviour the read would have always triggered a

Re: [ceph-users] cls_rbd ops on rbd_id.$name objects in EC pool

What's your cache mode? In the master branch, I would expect that class method ops should force a promotion to the cache tier if the base tier is an EC pool [1]. [1] https://github.com/ceph/ceph/blob/master/src/osd/OSD.cc#L8905 -- Jason Dillaman - Original Message - > From: "Nick