Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

On Tue, Oct 20, 2015 at 3:26 AM, Haomai Wang wrote: > The fact is that journal could help a lot for rbd use cases, > especially for small ios. I don' t think it will be bottleneck. If we > just want to reduce double write, it doesn't solve any performance > problem. > One

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

> On 20 Oct 2015, at 10:34, Luis Periquito wrote: > > On Tue, Oct 20, 2015 at 3:26 AM, Haomai Wang wrote: >> The fact is that journal could help a lot for rbd use cases, >> especially for small ios. I don' t think it will be bottleneck. If we >> just

[ceph-users] Write performance issue under rocksdb kvstore

Hi Guys, I am trying latest ceph-9.1.0 with rocksdb 4.1 and ceph-9.0.3 with rocksdb 3.11 as OSD backend. I use rbd to test performance and following is my cluster info. [ceph@xxx ~]$ ceph -s     cluster b74f3944-d77f-4401-a531-fa5282995808      health HEALTH_OK      monmap e1: 1 mons at

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

>> One trick I've been using in my ceph clusters is hiding a slow write >> backend behind a fast journal device. The write performance will be of >> the fast (and small) journal device. This only helps on write, but it >> can make a huge difference. >> > > Do you mean an external filesystem

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

> On 20 Oct 2015, at 11:28, Luis Periquito wrote: > >>> One trick I've been using in my ceph clusters is hiding a slow write >>> backend behind a fast journal device. The write performance will be of >>> the fast (and small) journal device. This only helps on write, but it

[ceph-users] ceph-hammer and debian jessie - missing files on repository

Hi, Thanks guys for supporting the latest debian stable release with latest ceph stable! as version 0.94.4. has been released, i tried to upgrade my debian/jessie cluster with hammer/wheezy packages to hammer/jessie. Unfortunately the download.ceph.com/debian-hammer debian repository is

Re: [ceph-users] Write performance issue under rocksdb kvstore

Haimao, you're right. I add such sync option as configurable for our test purpose. Thanks.Zhi Zhang (David) > Date: Tue, 20 Oct 2015 21:24:49 +0800 > From: haomaiw...@gmail.com > To: zhangz.da...@outlook.com > CC: ceph-users@lists.ceph.com; ceph-de...@vger.kernel.org > Subject: Re:

Re: [ceph-users] Write performance issue under rocksdb kvstore

Got your point. It is not only about the object data itself, but also ceph internal metadata. The best option seems to be your RP and wip-newstore-frags branch. :-) Thanks.Zhi Zhang (David) > Date: Tue, 20 Oct 2015 06:25:43 -0700 > From: s...@newdream.net > To: zhangz.da...@outlook.com > CC:

[ceph-users] Ceph OSDs with bcache experience

Hi, In the "newstore direction" thread on ceph-devel I wrote that I'm using bcache in production and Mark Nelson asked me to share some details. Bcache is running in two clusters now that I manage, but I'll keep this information to one of them (the one at PCextreme behind CloudStack). In this

Re: [ceph-users] [performance] rbd kernel module versus qemu librbd

Hi, I'm able to reach around same performance with qemu-librbd vs qemu-krbd, when I compile qemu with jemalloc (http://git.qemu.org/?p=qemu.git;a=commit;h=7b01cb974f1093885c40bf4d0d3e78e27e531363) on my test, librbd with jemalloc still use 2x more cpu than krbd, so cpu could be bottleneck too.

Re: [ceph-users] Write performance issue under rocksdb kvstore

On Tue, 20 Oct 2015, Z Zhang wrote: > Hi Guys, > > I am trying latest ceph-9.1.0 with rocksdb 4.1 and ceph-9.0.3 with > rocksdb 3.11 as OSD backend. I use rbd to test performance and following > is my cluster info. > > [ceph@xxx ~]$ ceph -s >     cluster b74f3944-d77f-4401-a531-fa5282995808 >  

Re: [ceph-users] Cache Tiering Question

I think what also makes things seem a little disconnected is that the target_max_bytes and relative levels are at the pool level, however I Think the current eviction logic works at a per OSD/PG level and so these values are calculated into estimates per OSD. This means that that depending on

[ceph-users] [performance] rbd kernel module versus qemu librbd

Hi, I have a question about the IOPS performance for real machine and virtual machine. Here is my test situation: 1. ssd pool (9 OSD servers with 2 osds on each server, 10Gb networks for public & cluster networks) 2. volume1: use rbd create a 100G volume from the ssd pool and map to the real

[ceph-users] planet.ceph.com

Hi, I was looking for some ceph resources and saw a reference to planet.ceph.com. However when I opened it I was sent to a dental clinic (?). That doesn't sound right, does it? I was at this page when I saw the reference... thanks ___ ceph-users

Re: [ceph-users] Placement rule not resolved

Hi Robert, Sorry for replying late We finally use a step take at root on the production platform Even if I tested a rule on the sandbox platform with a step take at a non-root level ... and it works. Brgds -Message d'origine- De : Robert LeBlanc [mailto:rob...@leblancnet.us] Envoyé 

[ceph-users] pg incomplete state

Hi folks I've been rebuilding drives in my cluster to add space. This has gone well so far. After the last batch of rebuilds, I'm left with one placement group in an incomplete state. [sudo] password for jpr: HEALTH_WARN 1 pgs incomplete; 1 pgs stuck inactive; 1 pgs stuck unclean pg 3.ea is

[ceph-users] v0.94.4 Hammer released upgrade

trying to upgrade from hammer 0.94.3 to 0.94.4 I'm getting the following error msg while trying to restart the mon daemons ($ sudo restart ceph-mon-all): 2015-10-20 08:56:37.410321 7f59a8c9d8c0 0 ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a), process ceph-mon, pid 6821

Re: [ceph-users] Minimum failure domain

On Mon, Oct 19, 2015 at 7:09 PM, John Wilkins wrote: > The classic case is when you are just trying Ceph out on a laptop (e.g., > using file directories for OSDs, setting the replica size to 2, and setting > osd_crush_chooseleaf_type to 0). Sure, but the text isn’t really

Re: [ceph-users] v0.94.4 Hammer released

Hi Udo, We I've try that and no luck at all. Cheers, *German* 2015-10-20 15:06 GMT-03:00 Udo Lembke : > Hi, > do you have changed the ownership like discribed in Sages mail about > "v9.1.0 Infernalis release candidate released"? > > #. Fix

Re: [ceph-users] Ceph OSDs with bcache experience

On 10/20/2015 09:00 AM, Wido den Hollander wrote: Hi, In the "newstore direction" thread on ceph-devel I wrote that I'm using bcache in production and Mark Nelson asked me to share some details. Bcache is running in two clusters now that I manage, but I'll keep this information to one of them

Re: [ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

Hi Stefan, update the ceph.conf file on your ceph-deploy node (~/ceph-deploy/ceph.conf) and then push the updated config file to other machines in the cluster as well as clients (if your config file is generic between cluster nodes and client nodes). If client config file is different you’ll

Re: [ceph-users] v0.94.4 Hammer released upgrade

On Tue, 20 Oct 2015, German Anders wrote: > Yep also: > > $ ceph-mon -v > ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a) Do you know how you had another version installed? I pushed wip-mon-reset-features which should let you override this.. but I would figure out how it

Re: [ceph-users] v0.94.4 Hammer released

Hi, do you have changed the ownership like discribed in Sages mail about "v9.1.0 Infernalis release candidate released"? #. Fix the ownership:: chown -R ceph:ceph /var/lib/ceph or set ceph.conf to use root instead? When upgrading, administrators have two options: #. Add

Re: [ceph-users] rbd export hangs / does nothing without regular drop_cache

Am 20.10.2015 um 15:03 schrieb Jason Dillaman: Can you provide more details on your setup and how you are running the rbd export? System with Raid 50 and 50TB space. Just running rbd export ... from command line. I'm exporting to a btrfs volume. Stefan > If clearing the pagecache,

[ceph-users] ceph and upgrading OS version

Hello everyone I am planning to upgrade my ceph servers from Ubuntu 12.04 to 14.04 and I am wondering if you have a recommended process of upgrading the OS version without causing any issues to the ceph cluster? Many thanks Andrei ___ ceph-users

Re: [ceph-users] v0.94.4 Hammer released

A change like this below, where we have to change ownership was not add to a point release for hammer right? Den 2015-10-20 kl. 20:06, skrev Udo Lembke: Hi, do you have changed the ownership like discribed in Sages mail about "v9.1.0 Infernalis release candidate released"? #. Fix the

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

> > On 10/20/2015 08:41 AM, Robert LeBlanc wrote: >> >> Given enough load, that fast Jornal will get filled and you will only be >> as fast as the back disk can flush (and at the same time service reads). >> That the the situation we are in right now. We are still seeing better >> performance than

Re: [ceph-users] Poor Read Performance with Ubuntu 14.04 LTS 3.19.0-30 Kernel

I performed this kernel upgrade (to 3.19.30) over the weekend on my cluster, and my before / after benchmarks were very close to each other, about 500MB/s each. On Tue, Oct 6, 2015 at 3:15 PM, Nick Fisk wrote: > I'm wondering if you are hitting the "bug" with the readahead

Re: [ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

And forgot. Yes, update both lines with the new mon node information mon_initial_members and mon_host JC > On Oct 20, 2015, at 07:54, Stefan Eriksson wrote: > > Hi > > I’m using cep-deploy with hammer and recently added a new monitor, I used > this:

Re: [ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

Thanks! I'll do that, should I add a bug report to mention this in the documentation? Den 2015-10-20 kl. 17:25, skrev LOPEZ Jean-Charles: And forgot. Yes, update both lines with the new mon node information mon_initial_members and mon_host JC On Oct 20, 2015, at 07:54, Stefan Eriksson

Re: [ceph-users] v0.94.4 Hammer released

Hi, On 20/10/2015 20:11, Stefan Eriksson wrote: > A change like this below, where we have to change ownership was not add to a > point release for hammer right? Right. ;) I have upgraded my ceph cluster from 0.94.3 to 0.94.4 today without any problem. The daemons used in 0.94.3 and currently

Re: [ceph-users] Ceph OSDs with bcache experience

The thing that worries me with your next-gen design (actually your current design aswell) is SSD wear. If you use Intel SSD at 10 DWPD, that's 12TB/day per 64TB total.  I guess use case dependant,  and perhaps 1:4 write read ratio is quite high in terms of writes as-is. You're also

Re: [ceph-users] Ceph OSDs with bcache experience

OK - seems my android email client (native samsung) messed up "in-reply-to" which confuses some MUA's. Apologies for that () /M On Tue, Oct 20, 2015 at 09:45:25PM +0200, Martin Millnert wrote: > The thing that worries me with your next-gen design (actually your current > design aswell) is SSD

Re: [ceph-users] v0.94.4 Hammer released

Same here, the upgrade went well. So far so good. - Original Message - From: "Francois Lafont" To: "ceph-users" Sent: Tuesday, 20 October, 2015 9:14:43 PM Subject: Re: [ceph-users] v0.94.4 Hammer released Hi, On 20/10/2015 20:11,

Re: [ceph-users] v0.94.4 Hammer released

On 21 October 2015 at 08:09, Andrei Mikhailovsky wrote: > Same here, the upgrade went well. So far so good. > Ditto -- Lindsay ___ ceph-users mailing list ceph-users@lists.ceph.com

Re: [ceph-users] v0.94.4 Hammer released upgrade

On Tue, 20 Oct 2015, German Anders wrote: > trying to upgrade from hammer 0.94.3 to 0.94.4 I'm getting the following > error msg while trying to restart the mon daemons ($ sudo restart > ceph-mon-all): > > 2015-10-20 08:56:37.410321 7f59a8c9d8c0  0 ceph version 0.94.4 >

[ceph-users] add new monitor doesn't update ceph.conf in hammer with ceph-deploy.

Hi I’m using cep-deploy with hammer and recently added a new monitor, I used this: http://docs.ceph.com/docs/hammer/rados/deployment/ceph-deploy-mon/ But it doesn’t say anything about adding conf manually to

Re: [ceph-users] [performance] rbd kernel module versus qemu librbd

Hi, Thanks for you reply. I do more test here and things change more strange, now i only could get about 4k iops in VM: 1. use fio with ioengine rbd to test the volume on the real machine [global] ioengine=rbd clientname=admin pool=vol_ssd rbdname=volume-4f4f9789-4215-4384-8e65-127a2e61a47f

Re: [ceph-users] How ceph client abort IO

I want to abort and retry a IO if taking longer time not completed. Does this make sense in Ceph? How ceph client handle longer timeout IOs? Just wait until it returned, or other error recovery method can be used to handle IO which can not be responsed in time. Thanks. 2015-10-20 21:00 GMT+08:00

Re: [ceph-users] v0.94.4 Hammer released

trying to upgrade from hammer 0.94.3 to 0.94.4 I'm getting the following error msg while trying to restart the mon daemons: 2015-10-20 08:56:37.410321 7f59a8c9d8c0 0 ceph version 0.94.4 (95292699291242794510b39ffde3f4df67898d3a), process ceph-mon, pid 6821 2015-10-20 08:56:37.429036 7f59a8c9d8c0

Re: [ceph-users] rbd export hangs / does nothing without regular drop_cache

Can you provide more details on your setup and how you are running the rbd export? If clearing the pagecache, dentries, and inodes solves the issue, it sounds like it's outside of Ceph (unless you are exporting to a CephFS or krbd mount point). -- Jason Dillaman - Original Message

Re: [ceph-users] How ceph client abort IO

On Tue, 20 Oct 2015, Jason Dillaman wrote: > There is no such interface currently on the librados / OSD side to abort > IO operations. Can you provide some background on your use-case for > aborting in-flight IOs? The internal Objecter has a cancel interface, but it can't yank back buffers,

[ceph-users] too many kworker processes after upgrade to 0.94.3

Hello I've recently upgraded my ceph cluster from 0.94.1 to 0.94.3 and noticed that after about a day i started getting the emails from our network/host monitoring system. The notifications were that there are too many processes on the osd servers. I've not seen this before and I am running

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

The hope is that with some of Somnath's work and perhaps additional future work, we might be able to make the journal a little smarter about how much data to keep and when to flush. While we are still ultimately bound by the backend disk performance, we might be able to absorb writes in a

Re: [ceph-users] How ceph client abort IO

There is no such interface currently on the librados / OSD side to abort IO operations. Can you provide some background on your use-case for aborting in-flight IOs? -- Jason Dillaman - Original Message - > From: "min fang" > To:

Re: [ceph-users] Write performance issue under rocksdb kvstore

On Tue, Oct 20, 2015 at 8:47 PM, Sage Weil wrote: > On Tue, 20 Oct 2015, Z Zhang wrote: >> Hi Guys, >> >> I am trying latest ceph-9.1.0 with rocksdb 4.1 and ceph-9.0.3 with >> rocksdb 3.11 as OSD backend. I use rbd to test performance and following >> is my cluster info. >> >>

Re: [ceph-users] Write performance issue under rocksdb kvstore

Thanks, Sage, for pointing out the PR and ceph branch. I will take a closer look. Yes, I am trying KVStore backend. The reason we are trying it is that few user doesn't have such high requirement on data loss occasionally. It seems KVStore backend without synchronized WAL could achieve better

Re: [ceph-users] Write performance issue under rocksdb kvstore

Actually keyvaluestore would submit transaction with sync flag too(rely to keyvaluedb impl journal/logfile). Yes, if we disable sync flag, keyvaluestore's performance will increase a lot. But we dont provide with this option now On Tue, Oct 20, 2015 at 9:22 PM, Z Zhang

Re: [ceph-users] Write performance issue under rocksdb kvstore

On Tue, 20 Oct 2015, Z Zhang wrote: > Thanks, Sage, for pointing out the PR and ceph branch. I will take a > closer look. > > Yes, I am trying KVStore backend. The reason we are trying it is that > few user doesn't have such high requirement on data loss occasionally. > It seems KVStore

Re: [ceph-users] Ceph journal - isn't it a bit redundant sometimes?

Given enough load, that fast Jornal will get filled and you will only be as fast as the back disk can flush (and at the same time service reads). That the the situation we are in right now. We are still seeing better performance than a raw spindle, but only 150 IOPs, not 15000 IOPS that the SSD