Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2017-02-20 Thread Christian Balzer
Hello, Just a quick update since I didn't have time for this yesterday. I did a similar test as below with only the XFS node active and as expected results are opposite: 3937 IOPS 3.16 3595 IOPS 4.9 As opposed to what I found out yesterday: --- Thus I turned off the XFS node and ran the test

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2017-02-19 Thread Christian Balzer
Hello, On Thu, 16 Feb 2017 17:51:18 +0200 Kostis Fardelas wrote: > Hello, > we are on Debian Jessie and Hammer 0.94.9 and recently we decided to > upgrade our kernel from 3.16 to 4.9 (jessie-backports). We experience > the same regression but with some shiny points Same OS, kernels and Ceph

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2017-02-16 Thread Kostis Fardelas
Hello, we are on Debian Jessie and Hammer 0.94.9 and recently we decided to upgrade our kernel from 3.16 to 4.9 (jessie-backports). We experience the same regression but with some shiny points -- ceph tell osd average across the cluster -- 3.16.39-1: 204MB/s 4.9.0-0: 158MB/s -- 1 rados bench

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-12-19 Thread Yoann Moulin
Hello, Finally, I found time to do some new benchmarks with the latest jewel release (10.2.5) on 4 nodes. Each node has 10 OSDs. I ran 2 times "ceph tell osd.* bench" over 40 OSDs, here the average speed : 4.2.0-42-generic 97.45 MB/s 4.4.0-53-generic 55.73 MB/s 4.8.15-040815-generic

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-26 Thread Lomayani S. Laizer
Hello, do you have journal on disk too ? Yes am having journal on same hard disk. ok and could you do bench with kernel 4.2 ? just to see if you have better throughput. Thanks In ubuntu 14 I was running 4.2 kernel. the throughput was the same around 80-90MB/s per osd. I cant tell the difference

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-26 Thread Yoann Moulin
Hello Mark, > FWIW, on CentOS7 I actually saw a performance increase when upgrading from the > stock 3.10 kernel to 4.4.5 with Intel P3700 NVMe devices. I was encountering > some kind of strange concurrency/locking issues at the driver level that 4.4.5 > resolved. I think your best bet is to

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-26 Thread Yoann Moulin
Hello, > Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are similar. do you have journal on disk too ? > I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have around > 80-90MB/s of OSD speeds in both operating systems ok and could you do bench with kernel 4.2

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-25 Thread Lomayani S. Laizer
Hello, Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are similar. I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have around 80-90MB/s of OSD speeds in both operating systems Only issue am observing now with ubuntu 16 is sometime osd fails on rebooting until

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-25 Thread Mark Nelson
Hi Yoann, FWIW, on CentOS7 I actually saw a performance increase when upgrading from the stock 3.10 kernel to 4.4.5 with Intel P3700 NVMe devices. I was encountering some kind of strange concurrency/locking issues at the driver level that 4.4.5 resolved. I think your best bet is to try

[ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

2016-07-25 Thread Yoann Moulin
Hello, (this is a repost, my previous message seems to be slipping under the radar) Does anyone get a similar behaviour to the one described below ? I found a big performance drop between kernel 3.13.0-88 (default kernel on Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default