Re: [ceph-users] pool distribution quality report script

27;]))) File "./readpgdump.py", line 71, in efficiency_score if weights and weights[osd]: KeyError: 0 On Thu, Mar 5, 2015 at 1:46 PM, Mark Nelson mailto:mnel...@redhat.com>> wrote: Hi Blair, I've updated the script and it now (theoretically) computes optimal

Re: [ceph-users] pool distribution quality report script

k On 03/05/2015 01:52 AM, Blair Bethwaite wrote: Hi Mark, Cool, that looks handy. Though it'd be even better if it could go a step further and recommend re-weighting values to balance things out (or increased PG counts where needed). Cheers, On 5 March 2015 at 15:11, Mark Nelson wrote:

[ceph-users] pool distribution quality report script

Hi All, Recently some folks showed interest in gathering pool distribution statistics and I remembered I wrote a script to do that a while back. It was broken due to a change in the ceph pg dump output format that was committed a while back, so I cleaned the script up, added detection of head

Re: [ceph-users] Persistent Write Back Cache

On 03/04/2015 05:34 AM, John Spray wrote: On 04/03/2015 08:26, Nick Fisk wrote: To illustrate the difference a proper write back cache can make, I put a 1GB (512mb dirty threshold) flashcache in front of my RBD and tweaked the flush parameters to flush dirty blocks at a large queue depth. The

Re: [ceph-users] Ceph Hammer OSD Shard Tuning Test Results

, 20 cores used, numjobs=8, iopdepth=32) this was with more powerfull nodes, but the difference seem to be quite huge - Mail original - De: "aderumier" À: "Mark Nelson" Cc: "ceph-devel" , "ceph-users" Envoyé: Vendredi 27 Février 2015 07:10:42 Ob

Re: [ceph-users] OSD Performance

? That fragmentation percentage looks good. We are considering using just SSD's for OSD's and RAM disk pcie devices for the Journals so this would be ok. Kind regards Kevin Walker +968 9765 1742 On 25 Feb 2015, at 02:35, Mark Nelson wrote: On 02/24/2015 04:21 PM, Kevin Walker wrote: Hi

Re: [ceph-users] OSD Performance

On 02/24/2015 04:21 PM, Kevin Walker wrote: Hi All Just recently joined the list and have been reading/learning about ceph for the past few months. Overall it looks to be well suited to our cloud platform but I have stumbled across a few worrying items that hopefully you guys can clarify the sta

Re: [ceph-users] Erasure Coding CPU Overhead Data

Many thanks, Nick -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Mark Nelson Sent: 21 February 2015 18:23 To: ceph-users@lists.ceph.com Cc: ceph-devel Subject: [ceph-users] Erasure Coding CPU Overhead Data Hi All, Last spring at the tail end of

Re: [ceph-users] ceph-osd pegging CPU on giant, no snapshots involved this time

On 02/23/2015 12:21 PM, Florian Haas wrote: On Wed, Feb 18, 2015 at 9:19 PM, Florian Haas wrote: Hey everyone, I must confess I'm still not fully understanding this problem and don't exactly know where to start digging deeper, but perhaps other users have seen this and/or it rings a bell. S

Re: [ceph-users] ceph-osd pegging CPU on giant, no snapshots involved this time

On 02/19/2015 10:56 AM, Florian Haas wrote: On Wed, Feb 18, 2015 at 10:27 PM, Florian Haas wrote: On Wed, Feb 18, 2015 at 9:32 PM, Mark Nelson wrote: On 02/18/2015 02:19 PM, Florian Haas wrote: Hey everyone, I must confess I'm still not fully understanding this problem and don'

Re: [ceph-users] ceph-osd pegging CPU on giant, no snapshots involved this time

On 02/18/2015 02:19 PM, Florian Haas wrote: Hey everyone, I must confess I'm still not fully understanding this problem and don't exactly know where to start digging deeper, but perhaps other users have seen this and/or it rings a bell. System info: Ceph giant on CentOS 7; approx. 240 OSDs, 6 p

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

d my IOPs rate was around the 3K mark. Something is holding ceph back from performing well with ssds ((( Out of curiosity, did you try the same tests directly on the SSD? Andrei ---- *From: *"Mark Nelson"

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

iginal - De: "Mark Nelson" À: "ceph-devel" Cc: "ceph-users" Envoyé: Mardi 17 Février 2015 18:37:01 Objet: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison Hi All, I wrote up a short document describing some tests I ran recently

Re: [ceph-users] Happy New Chinese Year!

Xīnnián kuàilè! Mark On 02/17/2015 06:23 PM, xmdx...@gmail.com wrote: hi, everyone: Happy New Chinese Year! — 通过 Mailbox 发送 ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/c

Re: [ceph-users] Ceph Dumpling/Firefly/Hammer SSD/Memstore performance comparison

s. The cron job and test suites are here: https://github.com/ceph/ceph-tools/tree/master/regression Mark On 02/17/2015 02:16 PM, Stephen Hindle wrote: I was wondering what the 'CBT' tool is ? Google is useless for that acronym... Thanks! Steve On Tue, Feb 17, 2015 at 10:37 AM, Mark

Re: [ceph-users] Ceph Supermicro hardware recommendation

On 02/08/2015 10:41 PM, Scott Laird wrote: Does anyone have a good recommendation for per-OSD memory for EC? My EC test blew up in my face when my OSDs suddenly spiked to 10+ GB per OSD process as soon as any reconstruction was needed. Which (of course) caused OSDs to OOM, which meant more reco

Re: [ceph-users] btrfs backend with autodefrag mount option

About a year ago I was talking to j On 01/30/2015 07:24 AM, Luke Kao wrote: Dear ceph users, Has anyone tried to add autodefrag and mount option when use btrfs as the osd storage? Sort of. About a year ago I was looking into it, but Josef told me not to use either defrag or autodefrag. (esp

Re: [ceph-users] btrfs backend with autodefrag mount option

oops, mangled the first part of that reply a bit. Need my morning coffee. :) On 01/30/2015 07:56 AM, Mark Nelson wrote: About a year ago I was talking to j On 01/30/2015 07:24 AM, Luke Kao wrote: Dear ceph users, Has anyone tried to add autodefrag and mount option when use btrfs as the osd

Re: [ceph-users] Cache pool tiering & SSD journal

On 01/17/2015 08:17 PM, lidc...@redhat.com wrote: No, if you used cache tiering, It is no need to use ssd journal again. The cache tiering and SSD journals serve a somewhat different purpose. In Ceph, all of the data for every single write is written to both the journal and to the data storag

Re: [ceph-users] NUMA and ceph ... zone_reclaim_mode

On 01/12/2015 07:47 AM, Dan van der Ster wrote: (resending to list) Hi Kyle, I'd like to +10 this old proposal of yours. Let me explain why... A couple months ago we started testing a new use-case with radosgw -- this new user is writing millions of small files and has been causing us some head

Re: [ceph-users] Uniform distribution

I didn't actually calculate the per-OSD object density but yes, I agree that will hurt. On 01/09/2015 12:09 PM, Gregory Farnum wrote: 100GB objects (or ~40 on a hard drive!) are way too large for you to get an effective random distribution. -Greg On Thu, Jan 8, 2015 at 5:25 PM, Mark N

Re: [ceph-users] Uniform distribution

On 01/08/2015 03:35 PM, Michael J Brewer wrote: Hi all, I'm working on filling a cluster to near capacity for testing purposes. Though I'm noticing that it isn't storing the data uniformly between OSDs during the filling process. I currently have the following levels: Node 1: /dev/sdb1

Re: [ceph-users] PG num calculator live on Ceph.com

Hi Michael, Good job! It would be really useful to add in calculations to show the expected distribution and max deviation from the mean. I'm dredging this up from an old email I sent out a year ago, but if we treat this as a "balls into bins" problem ala Raab & Steger: http://www14.in.tum

Re: [ceph-users] rbd read speed only 1/4 of write speed

On 12/16/2014 07:08 PM, Christian Balzer wrote: On Tue, 16 Dec 2014 16:26:17 + VELARTIS Philipp Dürhammer wrote: Hello, Read speed inside our vms (most of them windows) is only ¼ of the write speed. Write speed is about 450MB/s - 500mb/s and Read is only about 100/MB/s Our network is 10

Re: [ceph-users] experimental features

11:15 PM Justin Erenkrantz mailto:jus...@erenkrantz.com>> wrote: On Fri, Dec 5, 2014 at 12:46 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote: > I'm in favor of the "allow experimental features" but instead call it: > > "AL

Re: [ceph-users] experimental features

On 12/05/2014 11:47 AM, David Champion wrote: * On 05 Dec 2014, Sage Weil wrote: adding that fall into this category. Having them in the tree is great because it streamlines QA and testing, but I want to make sure that users are not able to enable the features without being aware of the risks.

Re: [ceph-users] experimental features

On 12/05/2014 11:39 AM, Gregory Farnum wrote: On Fri, Dec 5, 2014 at 9:36 AM, Sage Weil wrote: A while back we merged Haomai's experimental OSD backend KeyValueStore. We named the config option 'keyvaluestore_dev', hoping to make it clear to users that it was still under development, not full

Re: [ceph-users] Quetions abount osd journal configuration

On 11/26/2014 04:05 AM, Yujian Peng wrote: Hi all, I have a ceph cluster in production. Most of the write requests are small. I found that iops is a bottleneck. I want to move all of the journal datas to partitions on SSDs. Here is the procedures: 1.Set noout flag. ceph osd set noout 1.Stop osd

Re: [ceph-users] Tip of the week: don't use Intel 530 SSD's for journals

advice! Michael Kuriger mk7...@yp.com 818-649-7235 MikeKuriger (IM) On 11/25/14, 7:49 AM, "Mark Nelson" wrote: >On 11/25/2014 09:41 AM, Erik Logtenberg wrote: >> If you are like me, you have the journals for your OSD's with rotati

Re: [ceph-users] Tip of the week: don't use Intel 530 SSD's for journals

On 11/25/2014 09:41 AM, Erik Logtenberg wrote: If you are like me, you have the journals for your OSD's with rotating media stored separately on an SSD. If you are even more like me, you happen to use Intel 530 SSD's in some of your hosts. If so, please do check your S.M.A.R.T. statistics regular

Re: [ceph-users] Optimal or recommended threads values

Don't forget number of cores in the node. Basically you want enough threads to keep all of the cores busy while not having so many that you end up with a ton of context switching overhead. Also as you said there's a lot of other factors that may have an affect, like the number of AGs (assumin

Re: [ceph-users] RBD Cache Considered Harmful? (on all-SSD pools, at least)

On 11/21/2014 08:14 AM, Florian Haas wrote: Hi everyone, been trying to get to the bottom of this for a few days; thought I'd take this to the list to see if someone had insight to share. Situation: Ceph 0.87 (Giant) cluster with approx. 250 OSDs. One set of OSD nodes with just spinners put int

Re: [ceph-users] firefly and cache tiers

On 11/20/2014 03:17 PM, Lindsay Mathieson wrote: On Thu, 20 Nov 2014 03:12:44 PM Mark Nelson wrote: Personally I'd suggest a lot of testing first. Not sure if there are any lingering stability issues, but as far as performance goes in firefly you'll only likely see speed ups with v

Re: [ceph-users] firefly and cache tiers

Personally I'd suggest a lot of testing first. Not sure if there are any lingering stability issues, but as far as performance goes in firefly you'll only likely see speed ups with very skewed hot/cold distributions and potentially slow downs in the general case unless you have an extremely fa

Re: [ceph-users] Ceph performance - 10 times slower

Perhaps rados should print a big warning whenever -b parameter does not match object size during a read / random bench run or outright refuse to run? Kind regards René On 20.11.2014 16:55, Mark Nelson wrote: Hi Rene, The easiest way to check is to create a fresh pool and look at the files that are

Re: [ceph-users] Ceph performance - 10 times slower

: 15.53 Gb/s RX So you see, output varies tremendously, network and cluster activity (and ops column) does not. -b does not work. Examples are on a non-productive cluster with 7 servers with SSD osds exlusively, everything 2x10GBit lacp bond network. Kind regards René On 20.11.2014 15:

Re: [ceph-users] Ceph performance - 10 times slower

tency: 0.007074 On Wed, Nov 19, 2014 at 8:55 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote: On 11/19/2014 06:51 PM, Jay Janardhan wrote: Can someone help me what I can tune to improve the performance? The cluster is pushing data at about 13 MB/s with a sin

Re: [ceph-users] Ceph performance - 10 times slower

On 11/19/2014 06:51 PM, Jay Janardhan wrote: Can someone help me what I can tune to improve the performance? The cluster is pushing data at about 13 MB/s with a single copy of data while the underlying disks can push 100+MB/s. Can anyone help me with this? *rados bench results:* Concurrency R

Re: [ceph-users] RBD read performance in Giant ?

On 11/14/2014 05:54 AM, Florent Bautista wrote: On 11/14/2014 12:52 PM, Alexandre DERUMIER wrote: Unfortunately I didn't, do you think hdparm could wrong results ? I really don't known how hdparm is doing his bench (block size ? number of thread ?). BTW, do you have also upgraded librbd on

Re: [ceph-users] Ceph and Compute on same hardware?

Technically there's no reason it shouldn't work, but it does complicate things. Probably the biggest worry would be that if something bad happens on the compute side (say it goes nuts with network or memory transfers) it could slow things down enough that OSDs start failing heartbeat checks ca

Re: [ceph-users] cephfs survey results

On 11/04/2014 03:11 PM, Mark Kirkwood wrote: On 04/11/14 22:02, Sage Weil wrote: On Tue, 4 Nov 2014, Blair Bethwaite wrote: On 4 November 2014 01:50, Sage Weil wrote: In the Ceph session at the OpenStack summit someone asked what the CephFS survey results looked like. Thanks Sage, that was

Re: [ceph-users] Is there an negative relationship between storage utilization and ceph performance?

I'd say it's storage in general, though Ceph can be especially harsh on file systems (RBD can invoke particularly bad fragmentation in btrfs for example due to how COW works). So generally there's a lot of things that can cause slow downs as your disks get full: 1) More objects spread across

Re: [ceph-users] Remote Journal

Hi Dan, I don't know why NBD wouldn't function, but I also don't think it's the way you should go. Putting the journals on the OSD disks isn't a terrible option, but you will suffer the expected double write penalty . If your system disk is an SSD with fast sequential write throughput capab

Re: [ceph-users] Fio rbd stalls during 4M reads

FWIW the specific fio read problem appears to have started after 0.86 and before commit 42bcabf. Mark On 10/24/2014 12:56 PM, Gregory Farnum wrote: There's an issue in master branch temporarily that makes rbd reads greater than the cache size hang (if the cache was on). This might be that. (Ja

Re: [ceph-users] Question/idea about performance problems with a few overloaded OSDs

On 10/21/2014 01:06 PM, Lionel Bouton wrote: Hi Gregory, Le 21/10/2014 19:39, Gregory Farnum a écrit : On Tue, Oct 21, 2014 at 10:15 AM, Lionel Bouton wrote: [...] Any thought? Is it based on wrong assumptions? Would it prove to be a can of worms if someone tried to implement it? Yeah, there

Re: [ceph-users] Ceph counters

On 10/20/2014 08:22 PM, Craig Lewis wrote: I've just started on this myself.. I started with https://ceph.com/docs/v0.80/dev/perf_counters/ I'm currently monitoring the latency, using the (to pick one example) [op_w_latency][sum] and [op_w_latency][avgcount]. Both values are counters, so they

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

On 10/20/2014 09:28 AM, Mark Wu wrote: 2014-10-20 21:04 GMT+08:00 Mark Nelson mailto:mark.nel...@inktank.com>>: On 10/20/2014 06:27 AM, Mark Wu wrote: Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SS

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

On 10/20/2014 06:27 AM, Mark Wu wrote: Test result Update: Number of Hosts Maximum single volume IOPS Maximum aggregated IOPS SSD Disk IOPS SSD Disk Utilization 7 14k 45k 9800+90% 8 21k 50k

Re: [ceph-users] Performance doesn't scale well on a full ssd cluster.

At least historically, high CPU usage and likely context switching and lock contention have been the limiting factor during high IOPS workloads on the test hardware at Inktank (and now RH). I ran benchmarks with a parametric sweep of ceph parameters a while back on SSDs to see if changing any

Re: [ceph-users] Ceph SSD array with Intel DC S3500's

On 10/02/2014 12:48 PM, Adam Boyhan wrote: Hey everyone, loving Ceph so far! Hi! We are looking to role out a Ceph cluster with all SSD's. Our application is around 30% writes and 70% reads random IO. The plan is to start with roughly 8 servers with 8 800GB Intel DC S3500's per server. I

Re: [ceph-users] Why performance of benchmarks with small blocks is extremely small?

On 10/01/2014 11:18 AM, Gregory Farnum wrote: All the stuff I'm aware of is part of the testing we're doing for Giant. There is probably ongoing work in the pipeline, but the fast dispatch, sharded work queues, and sharded internal locking structures that Somnath has discussed all made it. I se

Re: [ceph-users] SSD MTBF

On 09/29/2014 03:58 AM, Dan Van Der Ster wrote: Hi Emmanuel, This is interesting, because we’ve had sales guys telling us that those Samsung drives are definitely the best for a Ceph journal O_o ! Our sales guys or Samsung sales guys? :) If it was ours, let me know. The conventional wisdom

Re: [ceph-users] Bcache / Enhanceio with osds

ert LeBlanc On Sun, Sep 14, 2014 at 6:13 PM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote: On 09/14/2014 05:11 PM, Andrei Mikhailovsky wrote: Hello guys, Was wondering if anyone uses or done some testing with using bcache or enhanceio caching

Re: [ceph-users] IRQ balancing, distribution

On 09/22/2014 01:55 AM, Christian Balzer wrote: Hello, not really specific to Ceph, but since one of the default questions by the Ceph team when people are facing performance problems seems to be "Have you tried turning it off and on again?" ^o^ err, "Are all your interrupts on one CPU?" I'm go

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

every solid state device now. By as much as 50% to handle sustained write operations. Especially important for the journals, as we've found. Maybe not an issue on the short fio run below, but certainly evident on longer runs or lots of historical data on the drives. The max transaction tim

Re: [ceph-users] Bcache / Enhanceio with osds

On 09/15/2014 07:35 AM, Andrei Mikhailovsky wrote: *From: *"Mark Nelson" *To: *ceph-users@lists.ceph.com *Sent: *Monday, 15 September, 2014 1:13:01 AM *Subject: *Re: [ceph-users] Bcache / Enha

Re: [ceph-users] Bcache / Enhanceio with osds

On 09/14/2014 05:11 PM, Andrei Mikhailovsky wrote: Hello guys, Was wondering if anyone uses or done some testing with using bcache or enhanceio caching in front of ceph osds? I've got a small cluster of 2 osd servers, 16 osds in total and 4 ssds for journals. I've recently purchased four additi

Re: [ceph-users] Cache Pool writing too much on ssds, poor performance?

it is missing a considerable amount. Is there a way to check the stats of the cache pool, including hit/miss information and other data? Yes, I am using firefly 0.80.5. Thanks Andrei ---- *From: *"Mark Nelson" *To: *ceph-u

Re: [ceph-users] Cache Pool writing too much on ssds, poor performance?

Something that is very important to keep in mind with the way that the cache tier implementation currently works in Ceph is that cache misses are very expensive. It's really important that your workload have a really big hot/cold data skew otherwise it's not going to work well at all. In your

Re: [ceph-users] Ceph monitor load, low performance

On 09/03/2014 04:34 PM, pawel.orzechow...@budikom.net wrote: Hello Ladies and Gentlemen;-) The reason for the problem was the lack of battery backuped cache. After we had installed it the load is even on all osd's. Glad to hear it was that simple! :) Mark Thanks Pawel --- Paweł Orzechow

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

Excellent, I've been meaning to check into how the TCP transport is going. Are you using a hybrid threadpool/epoll approach? That I suspect would be very effective at reducing context switching, especially compared to what we do now. Mark On 08/28/2014 10:40 PM, Matt W. Benjamin wrote: Hi,

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

On 08/29/2014 06:10 AM, Dan Van Der Ster wrote: Hi Sebastien, Here’s my recipe for max IOPS on a _testing_ instance with SSDs: osd op threads = 2 With SSDs, In the past I've seen increasing the osd op thread count can help random reads. osd disk threads = 2 journal max write bytes

Re: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS

& Regards Somnath -Original Message- From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sebastien Han Sent: Thursday, August 28, 2014 9:12 AM To: ceph-users Cc: Mark Nelson Subject: [ceph-users] [Single OSD performance on SSD] Can't go over 3, 2K IOPS Hey all,

Re: [ceph-users] what does monitor data directory include?

On 08/28/2014 07:48 AM, yuelongguang wrote: hi,all what is in directory, /var/lib/ceph/mon/ceph-cephosd1-mona/store.db/ how to dump? where monmap is stored? That directory is typically a leveldb store, though potentially could be rocksdb or maybe something else after firefly. You can use the

Re: [ceph-users] Best practice K/M-parameters EC pool

On 08/15/2014 06:24 AM, Wido den Hollander wrote: On 08/15/2014 12:23 PM, Loic Dachary wrote: Hi Erik, On 15/08/2014 11:54, Erik Logtenberg wrote: Hi, With EC pools in Ceph you are free to choose any K and M parameters you like. The documentation explains what K and M do, so far so good. Now

Re: [ceph-users] Performance really drops from 700MB/s to 10MB/s

On 08/13/2014 08:19 AM, German Anders wrote: Hi to all, I'm having a particular behavior on a new Ceph cluster. I've map a RBD to a client and issue some performance tests with fio, at this point everything goes just fine (also the results :) ), but then I try to run another new test on a

Re: [ceph-users] slow OSD brings down the cluster

On 08/06/2014 03:43 AM, Luis Periquito wrote: Hi, In the last few days I've had some issues with the radosgw in which all requests would just stop being served. After some investigation I would go for a single slow OSD. I just restarted that OSD and everything would just go back to work. Every

Re: [ceph-users] librbd tuning?

On 08/05/2014 06:19 PM, Mark Kirkwood wrote: On 05/08/14 23:44, Mark Nelson wrote: On 08/05/2014 02:48 AM, Mark Kirkwood wrote: On 05/08/14 03:52, Tregaron Bayly wrote: Does anyone have any insight on how we can tune librbd to perform closer to the level of the rbd kernel module? In our lab

Re: [ceph-users] Ceph writes stall for long perioids with no disk/network activity

On 08/05/2014 08:42 AM, Mariusz Gronczewski wrote: On Mon, 04 Aug 2014 15:32:50 -0500, Mark Nelson wrote: On 08/04/2014 03:28 PM, Chris Kitzmiller wrote: On Aug 1, 2014, at 1:31 PM, Mariusz Gronczewski wrote: I got weird stalling during writes, sometimes I got same write speed for few

Re: [ceph-users] librbd tuning?

On 08/05/2014 02:48 AM, Mark Kirkwood wrote: On 05/08/14 03:52, Tregaron Bayly wrote: Does anyone have any insight on how we can tune librbd to perform closer to the level of the rbd kernel module? In our lab we have a four node cluster with 1GbE public network and 10GbE cluster network. A cli

Re: [ceph-users] librbd tuning?

On 08/04/2014 10:52 AM, Tregaron Bayly wrote: Does anyone have any insight on how we can tune librbd to perform closer to the level of the rbd kernel module? In our lab we have a four node cluster with 1GbE public network and 10GbE cluster network. A client node connects to the public network w

Re: [ceph-users] Ceph writes stall for long perioids with no disk/network activity

On 08/04/2014 03:28 PM, Chris Kitzmiller wrote: On Aug 1, 2014, at 1:31 PM, Mariusz Gronczewski wrote: I got weird stalling during writes, sometimes I got same write speed for few minutes and after some time it starts stalling with 0 MB/s for minutes I'm getting very similar behavior on my clu

Re: [ceph-users] fs as btrfs and ceph journal

Perhaps Cristian is thinking of the clone from journal work that we were talking about last year: http://wiki.ceph.com/Planning/Sideboard/osd%3A_clone_from_journal_on_btrfs I think we never did much beyond Sage's test branch, and it didn't seem to help as much as you would hope. Speaking of wh

Re: [ceph-users] slow read speeds from kernel rbd (Firefly 0.80.4)

On 07/28/2014 11:28 AM, Steve Anthony wrote: While searching for more information I happened across the following post (http://dachary.org/?p=2961) which vaguely resembled the symptoms I've been experiencing. I ran tcpdump and noticed what appeared to be a high number of retransmissions on the ho

Re: [ceph-users] Optimal OSD Configuration for 45 drives?

On 07/25/2014 12:04 PM, Christian Balzer wrote: On Fri, 25 Jul 2014 07:24:26 -0500 Mark Nelson wrote: On 07/25/2014 02:54 AM, Christian Balzer wrote: On Fri, 25 Jul 2014 13:31:34 +1000 Matt Harlum wrote: Hi, I’ve purchased a couple of 45Drives enclosures and would like to figure out the

Re: [ceph-users] Optimal OSD Configuration for 45 drives?

On 07/25/2014 02:54 AM, Christian Balzer wrote: On Fri, 25 Jul 2014 13:31:34 +1000 Matt Harlum wrote: Hi, I’ve purchased a couple of 45Drives enclosures and would like to figure out the best way to configure these for ceph? That's the second time within a month somebody mentions these 45 dri

Re: [ceph-users] Ceph and Infiniband

On 07/23/2014 03:54 AM, Andrei Mikhailovsky wrote: Ricardo, Thought to share my testing results. I've been using IPoIB with ceph for quite some time now. I've got QDR osd/mon/client servers to serve rbd images to kvm hypervisor. I've done some performance testing using both rados and guest vm b

Re: [ceph-users] HW recommendations for OSD journals?

On 07/16/2014 09:58 AM, Riccardo Murri wrote: Hello, I am new to Ceph; the group I'm working in is currently evaluating it for our new large-scale storage. Is there any recommendation for the OSD journals? E.g., does it make sense to keep them on SSDs? Would it make sense to host the journal

Re: [ceph-users] Firefly Upgrade

On 07/14/2014 05:37 AM, Quenten Grasso wrote: Hi All, Just a quick question for the list, has anyone seen a significant increase in ram usage since firefly? I upgraded from 0.72.2 to 80.3 now all of my Ceph servers are using about double the ram they used to. Can you tell me a bit about how yo

Re: [ceph-users] performance tests

On 07/10/2014 03:24 AM, Xabier Elkano wrote: El 10/07/14 09:18, Christian Balzer escribió: On Thu, 10 Jul 2014 08:57:56 +0200 Xabier Elkano wrote: El 09/07/14 16:53, Christian Balzer escribió: On Wed, 09 Jul 2014 07:07:50 -0500 Mark Nelson wrote: On 07/09/2014 06:52 AM, Xabier Elkano wrote

Re: [ceph-users] performance tests

On 07/09/2014 09:53 AM, Christian Balzer wrote: On Wed, 09 Jul 2014 07:07:50 -0500 Mark Nelson wrote: On 07/09/2014 06:52 AM, Xabier Elkano wrote: El 09/07/14 13:10, Mark Nelson escribió: On 07/09/2014 05:57 AM, Xabier Elkano wrote: Hi, I was doing some tests in my cluster with fio tool

Re: [ceph-users] performance tests

On 07/09/2014 06:52 AM, Xabier Elkano wrote: El 09/07/14 13:10, Mark Nelson escribió: On 07/09/2014 05:57 AM, Xabier Elkano wrote: Hi, I was doing some tests in my cluster with fio tool, one fio instance with 70 jobs, each job writing 1GB random with 4K block size. I did this test with 3

Re: [ceph-users] performance tests

On 07/09/2014 05:57 AM, Xabier Elkano wrote: Hi, I was doing some tests in my cluster with fio tool, one fio instance with 70 jobs, each job writing 1GB random with 4K block size. I did this test with 3 variations: 1- Creating 70 images, 60GB each, in the pool. Using rbd kernel module, format

Re: [ceph-users] cephfs and EC

On 07/08/2014 04:28 AM, Stijn De Weirdt wrote: hi all, one of the changes in the 0.82 release (accoridng to the notes) is: mon: prevent EC pools from being used with cephfs can someone clarify this a bit? cephfs with EC pools make no sense? now? ever? or is it just not recommended (i'm also in

Re: [ceph-users] CEPH Cache Tiering

On 07/07/2014 09:17 AM, Lazuardi Nasution wrote: Hi, I'm thinking of using SSD for cache on CEPH where the SSDs are on the same OSD nodes with HDDs. My options are using CEPH Cache Tiering or using another cache software like bCache or FlashCache. On the second option, the SSDs will only cache d

Re: [ceph-users] write performance per disk

On 07/03/2014 08:11 AM, VELARTIS Philipp Dürhammer wrote: Hi, I have a ceph cluster setup (with 45 sata disk journal on disks) and get only 450mb/sec writes seq (maximum playing around with threads in rados bench) with replica of 2 Which is about ~20Mb writes per disk (what y see in atop also)

Re: [ceph-users] Bad Write-Performance on Ceph/Possible bottlenecks?

On 07/04/2014 04:13 AM, Marco Allevato wrote: Hello Ceph-Community, I’m writing here because we have a bad write-performance on our Ceph-Cluster of about _As an overview the technical details of our Cluster:_ 3 x monitoring-Servers; each with 2 x 1 Gbit/s NIC configured as Bond (Link Aggregati

Re: [ceph-users] Poor performance on all SSD cluster

On 06/24/2014 04:46 AM, Mark Kirkwood wrote: On 23/06/14 19:16, Mark Kirkwood wrote: For database types (and yes I'm one of those)...you want to know that your writes (particularly your commit writes) are actually making it to persistent storage (that ACID thing you know). Now I see RBD cache ve

Re: [ceph-users] Poor performance on all SSD cluster

On 06/24/2014 03:45 AM, Mark Kirkwood wrote: On 24/06/14 18:15, Robert van Leeuwen wrote: All of which means that Mysql performance (looking at you binlog) may still suffer due to lots of small block size sync writes. Which begs the question: Anyone running a reasonable busy Mysql server on Ce

Re: [ceph-users] Poor performance on all SSD cluster

On 06/23/2014 12:54 PM, Greg Poirier wrote: On Sun, Jun 22, 2014 at 6:44 AM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote: RBD Cache is definitely going to help in this use case. This test is basically just sequentially writing a single 16k chunk of data out, one at

Re: [ceph-users] Poor performance on all SSD cluster

On 06/22/2014 02:02 AM, Haomai Wang wrote: Hi Mark, Do you enable rbdcache? I test on my ssd cluster(only one ssd), it seemed ok. dd if=/dev/zero of=test bs=16k count=65536 oflag=direct 82.3MB/s RBD Cache is definitely going to help in this use case. This test is basically just sequentia

Re: [ceph-users] Run ceph from source code

I do sometimes compile from source, but I tend to build and start my clusters using the lower level commands. I'm not sure if ceph-deploy is intended to be used with from-source builds (it may very well and I just don't know). Mark On 06/16/2014 05:47 PM, Zhe Zhang wrote: Thanks guys. This

Re: [ceph-users] spiky io wait within VMs running on rbd

72338 7f4db6bdf700 1 heartbeat_map reset_timeout 'OSD::op_tp thread 0x7f4db6bdf700' had timed out after 15 Any ideas why this happens? Thanks. -Simon On Thu, Jun 12, 2014 at 11:14 AM, Mark Nelson mailto:mark.nel...@inktank.com>> wrote: On 06/12/2014 08:47 AM, Xu (Simon) Chen wro

Re: [ceph-users] spiky io wait within VMs running on rbd

On 06/12/2014 08:47 AM, Xu (Simon) Chen wrote: 1) I did check iostat on all OSDs, and iowait seems normal. 2) ceph -w shows no correlation between high io wait and high iops. Sometimes the reverse is true: when io wait is high (since it's a cluster wide thing), the overall ceph iops drops too.

Re: [ceph-users] OSDs

On 06/12/2014 07:27 AM, Christian Kauhaus wrote: Am 12.06.2014 14:09, schrieb Loic Dachary: With the replication factor set to three (which is the default), it can tolerate that two OSD fail at the same time. I've noticed that a replication factor of 3 is the new default in firefly. What rati

Re: [ceph-users] Backfilling, latency and priority

On 06/12/2014 03:44 AM, David wrote: Hi, We have 5 OSD servers, with 10 OSDs each (journals on enterprise SSDs). We lost an OSD and the cluster started to backfill the data to the rest of the OSDs - during which the latency skyrocketed on some OSDs and connected clients experienced massive IO

[ceph-users] Ceph performance profiling with perf

Hi All, For those of you that are interested in performance data, Brendan Gregg wrote a really useful cheat sheet for perf that can give you some idea of the things it can do beyond just profiling and performance counters. The static and dynamic tracing capabilities are especially interesting

Re: [ceph-users] [Annonce]The progress of KeyValueStore in Firely

On 06/03/2014 01:55 PM, Sushma R wrote: Haomai, I'm using the latest ceph master branch. ceph_smalliobench is a Ceph internal benchmarking tool similar to rados bench and the performance is more or less similar to that reported by fio. I tried to use fio with rbd ioengine (http://telekomcloud.

Re: [ceph-users] mellanox SX1012 ethernet|infiniband switch, somebody use it for ceph ?

I don't know very much about using that specific switch with 10GbE and Ceph. Having said that, Mellanox switches are pretty fast and often used in HPC deployments. I'd expect it should be good from a stability/performance point of view, at least on the IB side. If you aren't familiar with IB

Re: [ceph-users] btrfs + cache tier = disaster

Some folks have been interested in running OSDs on different backends so that if one filesystem has some kind of catastrophic bug that will effect lots of OSDs, they can still keep Ceph up and running with data copies on the other file systems. Whether or not that's a better solution than just

<    1   2   3   4   5   6   7   >