Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Christian Balzer]

Hello,


Just a quick update since I didn't have time for this yesterday.

I did a similar test as below with only the XFS node active and as expected
results are opposite:

3937 IOPS 3.16
3595 IOPS 4.9

As opposed to what I found out yesterday:
---
Thus I turned off the XFS node and ran the test again with just the EXT4
node active. And this time 4.9 came out (slightly) ahead:

3645 IOPS 3.16
3970 IOPS 4.9

---

Christian

On Mon, 20 Feb 2017 13:10:38 +0900 Christian Balzer wrote:

> Hello,
> 
> On Thu, 16 Feb 2017 17:51:18 +0200 Kostis Fardelas wrote:
> 
> > Hello,
> > we are on Debian Jessie and Hammer 0.94.9 and recently we decided to
> > upgrade our kernel from 3.16 to 4.9 (jessie-backports). We experience
> > the same regression but with some shiny points  
> 
> Same OS, kernels and Ceph version here, but I can't reproduce this for
> the most part, probably because of other differences.
> 
> 4 nodes, 
> 2 with 4 HDD based and SSD journal OSDs,
> 2 with 4 SSD based OSDs (cache-tier),
> replication 2.
> Half of the nodes/OSDs are using XFS, the other half EXT4.
> 
> > -- ceph tell osd average across the cluster --
> > 3.16.39-1: 204MB/s
> > 4.9.0-0: 158MB/s
> >   
> The "ceph osd tell bench" is really way too imprecise and all over the
> place for me, but the average of the HDD based OSDs doesn't differ
> noticeably.
> 
> > -- 1 rados bench client 4K 2048 threads avg IOPS --
> > 3.16.39-1: 1604
> > 4.9.0-0: 451
> >   
> I'd think 32-64 threads will do nicely.
> As discussed on the ML before, this test is also not particular realistic
> when it comes to actual client performance, but still, a data point is a
> data point. 
> 
> And incidentally this is the only test where I can clearly see something
> similar, with 64 threads and 4K:
> 
> 3400 IOPS 3.16
> 2600 IOPS 4.9
> 
> So where you are seeing a 70% reduction, I'm seeing "only" 25% less.
> 
> Which is of course a perfect match for my XFS vs. EXT4 OSD ratio.
> 
> Thus I turned off the XFS node and ran the test again with just the EXT4
> node active. And this time 4.9 came out (slightly) ahead:
> 
> 3645 IOPS 3.16
> 3970 IOPS 4.9
> 
> So this looks like a regression when it comes to CEPH interacting with XFS.
> Probably aggravated by how the "bench" tests work (lots of object
> creation), as opposed to normal usage with existing objects as tested
> below.
> 
> > -- 1 rados bench client 64K 512 threads avg BW MB/s--
> > 3.16.39-1: 78
> > 4.9.0-0: 31
> >  
> With the default 4MB block size, no relevant difference here again.
> But then again, this creates only a few objects compared to 4KB.
>  
> I've run fio (4M write, 4K write, 4k randwrite) from within a VM against
> the cluster with both kernel versions, no noticeable difference there
> either.
> 
> Just to compare this to the rados bench tests above:
> ---
> root@tvm-01:~# fio --size=18G --ioengine=libaio --invalidate=1 --direct=1 
> --numjobs=1 --rw=write --name=fiojob --blocksize=4M --iodepth=64
> 
> fiojob: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=64
> fio-2.1.11
>   write: io=18432MB, bw=359772KB/s, iops=87, runt= 52462msec
> ---
> OSD processes are at about 35% CPU usage (100% = 1 core), SSDs are at about
> 85% utilization. 
> 
> ---
> root@tvm-01:~# fio --size=4G --ioengine=libaio --invalidate=1 --direct=1 
> --numjobs=1 --rw=write --name=fiojob --blocksize=4K --iodepth=64
> 
> fiojob: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
> fio-2.1.11
>   write: io=4096.0MB, bw=241984KB/s, iops=60495, runt= 17333msec
> ---
> OSD processes are at about 20% CPU  usage, SSDs are at 50%
> utilization.
> 
> ---
> root@tvm-01:~# fio --size=2G --ioengine=libaio --invalidate=1 --direct=1 
> --numjobs=1 --rw=randwrite --name=fiojob --blocksize=4K --iodepth=64
> 
> fiojob: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
> fio-2.1.11
>   write: io=2048.0MB, bw=36086KB/s, iops=9021, runt= 58115msec
> ---
> OSD processes are at 300% (and likely wanting more) CPU usage, SSDs at
> about 25% utilization
> 
> Christian
> 
> > The shiny points are on the following tests:
> > 1 rados bench client 4K 512 threads avg IOPS
> > 1 rados bench client 64K 2048 threads avg BW MB/s
> > 
> > where machines with kernel 4.9 seem to perform slightly better. The
> > overall impression though is that there is a serious regression or
> > something that should be tuned to get the same performance out of the
> > cluster.
> > 
> > Our demo cluster is 4 nodes X 12 OSDs, separate journal on SSD,
> > firefly tunables and everything else default considering our Ceph
> > installation and Debian OS. Each rados bench run 5 times to get an
> > average and caches were dropped before each test.
> > 
> > I wonder if anyone has discovered the culprit so far? Any hints from
> > others to focus our investigation on?
> > 
> > Best regards,
> > Kostis
> > 
> > On 19 December 2016 at 17:17, Yoann Moulin  wrote:  
> > > Hello,
> > >
> > > Finally, I 

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Christian Balzer]

Hello,

On Thu, 16 Feb 2017 17:51:18 +0200 Kostis Fardelas wrote:

> Hello,
> we are on Debian Jessie and Hammer 0.94.9 and recently we decided to
> upgrade our kernel from 3.16 to 4.9 (jessie-backports). We experience
> the same regression but with some shiny points

Same OS, kernels and Ceph version here, but I can't reproduce this for
the most part, probably because of other differences.

4 nodes, 
2 with 4 HDD based and SSD journal OSDs,
2 with 4 SSD based OSDs (cache-tier),
replication 2.
Half of the nodes/OSDs are using XFS, the other half EXT4.

> -- ceph tell osd average across the cluster --
> 3.16.39-1: 204MB/s
> 4.9.0-0: 158MB/s
> 
The "ceph osd tell bench" is really way too imprecise and all over the
place for me, but the average of the HDD based OSDs doesn't differ
noticeably.

> -- 1 rados bench client 4K 2048 threads avg IOPS --
> 3.16.39-1: 1604
> 4.9.0-0: 451
> 
I'd think 32-64 threads will do nicely.
As discussed on the ML before, this test is also not particular realistic
when it comes to actual client performance, but still, a data point is a
data point. 

And incidentally this is the only test where I can clearly see something
similar, with 64 threads and 4K:

3400 IOPS 3.16
2600 IOPS 4.9

So where you are seeing a 70% reduction, I'm seeing "only" 25% less.

Which is of course a perfect match for my XFS vs. EXT4 OSD ratio.

Thus I turned off the XFS node and ran the test again with just the EXT4
node active. And this time 4.9 came out (slightly) ahead:

3645 IOPS 3.16
3970 IOPS 4.9

So this looks like a regression when it comes to CEPH interacting with XFS.
Probably aggravated by how the "bench" tests work (lots of object
creation), as opposed to normal usage with existing objects as tested
below.

> -- 1 rados bench client 64K 512 threads avg BW MB/s--
> 3.16.39-1: 78
> 4.9.0-0: 31
>
With the default 4MB block size, no relevant difference here again.
But then again, this creates only a few objects compared to 4KB.
 
I've run fio (4M write, 4K write, 4k randwrite) from within a VM against
the cluster with both kernel versions, no noticeable difference there
either.

Just to compare this to the rados bench tests above:
---
root@tvm-01:~# fio --size=18G --ioengine=libaio --invalidate=1 --direct=1 
--numjobs=1 --rw=write --name=fiojob --blocksize=4M --iodepth=64

fiojob: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=libaio, iodepth=64
fio-2.1.11
  write: io=18432MB, bw=359772KB/s, iops=87, runt= 52462msec
---
OSD processes are at about 35% CPU usage (100% = 1 core), SSDs are at about
85% utilization. 

---
root@tvm-01:~# fio --size=4G --ioengine=libaio --invalidate=1 --direct=1 
--numjobs=1 --rw=write --name=fiojob --blocksize=4K --iodepth=64

fiojob: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.1.11
  write: io=4096.0MB, bw=241984KB/s, iops=60495, runt= 17333msec
---
OSD processes are at about 20% CPU  usage, SSDs are at 50%
utilization.

---
root@tvm-01:~# fio --size=2G --ioengine=libaio --invalidate=1 --direct=1 
--numjobs=1 --rw=randwrite --name=fiojob --blocksize=4K --iodepth=64

fiojob: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.1.11
  write: io=2048.0MB, bw=36086KB/s, iops=9021, runt= 58115msec
---
OSD processes are at 300% (and likely wanting more) CPU usage, SSDs at
about 25% utilization

Christian

> The shiny points are on the following tests:
> 1 rados bench client 4K 512 threads avg IOPS
> 1 rados bench client 64K 2048 threads avg BW MB/s
> 
> where machines with kernel 4.9 seem to perform slightly better. The
> overall impression though is that there is a serious regression or
> something that should be tuned to get the same performance out of the
> cluster.
> 
> Our demo cluster is 4 nodes X 12 OSDs, separate journal on SSD,
> firefly tunables and everything else default considering our Ceph
> installation and Debian OS. Each rados bench run 5 times to get an
> average and caches were dropped before each test.
> 
> I wonder if anyone has discovered the culprit so far? Any hints from
> others to focus our investigation on?
> 
> Best regards,
> Kostis
> 
> On 19 December 2016 at 17:17, Yoann Moulin  wrote:
> > Hello,
> >
> > Finally, I found time to do some new benchmarks with the latest jewel 
> > release (10.2.5) on 4 nodes. Each node has 10 OSDs.
> >
> > I ran 2 times "ceph tell osd.* bench" over 40 OSDs, here the average speed :
> >
> > 4.2.0-42-generic  97.45 MB/s
> > 4.4.0-53-generic  55.73 MB/s
> > 4.8.15-040815-generic 62.41 MB/s
> > 4.9.0-040900-generic  60.88 MB/s
> >
> > I have the same behaviour with at least 35 to 40% performance drop between 
> > kernel 4.2 and kernel > 4.4
> >
> > I can do further benches if needed.
> >
> > Yoann
> >
> > Le 26/07/2016 à 09:09, Lomayani S. Laizer a écrit :  
> >> Hello,
> >> do you have journal on disk too ?
> >>
> >> Yes am having journal on same hard disk.
> >>
> >> ok and could you do bench with kernel 4.2 ? just to see 

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Kostis Fardelas]

Hello,
we are on Debian Jessie and Hammer 0.94.9 and recently we decided to
upgrade our kernel from 3.16 to 4.9 (jessie-backports). We experience
the same regression but with some shiny points
-- ceph tell osd average across the cluster --
3.16.39-1: 204MB/s
4.9.0-0: 158MB/s

-- 1 rados bench client 4K 2048 threads avg IOPS --
3.16.39-1: 1604
4.9.0-0: 451

-- 1 rados bench client 64K 512 threads avg BW MB/s--
3.16.39-1: 78
4.9.0-0: 31

The shiny points are on the following tests:
1 rados bench client 4K 512 threads avg IOPS
1 rados bench client 64K 2048 threads avg BW MB/s

where machines with kernel 4.9 seem to perform slightly better. The
overall impression though is that there is a serious regression or
something that should be tuned to get the same performance out of the
cluster.

Our demo cluster is 4 nodes X 12 OSDs, separate journal on SSD,
firefly tunables and everything else default considering our Ceph
installation and Debian OS. Each rados bench run 5 times to get an
average and caches were dropped before each test.

I wonder if anyone has discovered the culprit so far? Any hints from
others to focus our investigation on?

Best regards,
Kostis

On 19 December 2016 at 17:17, Yoann Moulin  wrote:
> Hello,
>
> Finally, I found time to do some new benchmarks with the latest jewel release 
> (10.2.5) on 4 nodes. Each node has 10 OSDs.
>
> I ran 2 times "ceph tell osd.* bench" over 40 OSDs, here the average speed :
>
> 4.2.0-42-generic  97.45 MB/s
> 4.4.0-53-generic  55.73 MB/s
> 4.8.15-040815-generic 62.41 MB/s
> 4.9.0-040900-generic  60.88 MB/s
>
> I have the same behaviour with at least 35 to 40% performance drop between 
> kernel 4.2 and kernel > 4.4
>
> I can do further benches if needed.
>
> Yoann
>
> Le 26/07/2016 à 09:09, Lomayani S. Laizer a écrit :
>> Hello,
>> do you have journal on disk too ?
>>
>> Yes am having journal on same hard disk.
>>
>> ok and could you do bench with kernel 4.2 ? just to see if you have better
>> throughput. Thanks
>>
>> In ubuntu 14 I was running 4.2 kernel. the throughput was the same around 
>> 80-90MB/s per osd. I cant tell the difference because each test gives
>> the speeds on same range. I did not test kernel 4.4 in ubuntu 14
>>
>>
>> --
>> Lomayani
>>
>> On Tue, Jul 26, 2016 at 9:39 AM, Yoann Moulin > > wrote:
>>
>> Hello,
>>
>> > Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are 
>> similar.
>>
>> do you have journal on disk too ?
>>
>> > I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I 
>> have around
>> > 80-90MB/s of OSD speeds in both operating systems
>>
>> ok and could you do bench with kernel 4.2 ? just to see if you have 
>> better
>> throughput. Thanks
>>
>> > Only issue am observing now with ubuntu 16 is sometime osd fails on 
>> rebooting
>> > until i start them manually or adding starting commands in rc.local.
>>
>> in my case, it's a test environment, so I don't have notice those 
>> behaviours
>>
>> --
>> Yoann
>>
>> > On Mon, Jul 25, 2016 at 6:45 PM, Yoann Moulin > 
>> > >> wrote:
>> >
>> > Hello,
>> >
>> > (this is a repost, my previous message seems to be slipping under 
>> the radar)
>> >
>> > Does anyone get a similar behaviour to the one described below ?
>> >
>> > I found a big performance drop between kernel 3.13.0-88 (default 
>> kernel on
>> > Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 
>> (default kernel on
>> > Ubuntu Xenial 16.04)
>> >
>> > - ceph version is Jewel (10.2.2).
>> > - All tests have been done under Ubuntu 14.04 on
>> > - Each cluster has 5 nodes strictly identical.
>> > - Each node has 10 OSDs.
>> > - Journals are on the disk.
>> >
>> > Kernel 4.4 has a drop of more than 50% compared to 4.2
>> > Kernel 4.4 has a drop of 40% compared to 3.13
>> >
>> > details below :
>> >
>> > With the 3 kernel I have the same performance on disks :
>> >
>> > Raw benchmark:
>> > dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> 
>> average ~230MB/s
>> > dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => 
>> average ~220MB/s
>> >
>> > Filesystem mounted benchmark:
>> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => 
>> average ~205MB/s
>> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => 
>> average ~214MB/s
>> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => 
>> average ~190MB/s
>> >
>> > Ceph osd Benchmark:
>> > Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  
>> ~81MB/s
>> > Kernel 

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Yoann Moulin]

Hello,

Finally, I found time to do some new benchmarks with the latest jewel release 
(10.2.5) on 4 nodes. Each node has 10 OSDs.

I ran 2 times "ceph tell osd.* bench" over 40 OSDs, here the average speed :

4.2.0-42-generic  97.45 MB/s
4.4.0-53-generic  55.73 MB/s
4.8.15-040815-generic 62.41 MB/s
4.9.0-040900-generic  60.88 MB/s

I have the same behaviour with at least 35 to 40% performance drop between 
kernel 4.2 and kernel > 4.4

I can do further benches if needed.

Yoann

Le 26/07/2016 à 09:09, Lomayani S. Laizer a écrit :
> Hello,
> do you have journal on disk too ?
> 
> Yes am having journal on same hard disk.
> 
> ok and could you do bench with kernel 4.2 ? just to see if you have better
> throughput. Thanks
> 
> In ubuntu 14 I was running 4.2 kernel. the throughput was the same around 
> 80-90MB/s per osd. I cant tell the difference because each test gives
> the speeds on same range. I did not test kernel 4.4 in ubuntu 14
> 
> 
> --
> Lomayani
> 
> On Tue, Jul 26, 2016 at 9:39 AM, Yoann Moulin  > wrote:
> 
> Hello,
> 
> > Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are 
> similar.
> 
> do you have journal on disk too ?
> 
> > I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have 
> around
> > 80-90MB/s of OSD speeds in both operating systems
> 
> ok and could you do bench with kernel 4.2 ? just to see if you have better
> throughput. Thanks
> 
> > Only issue am observing now with ubuntu 16 is sometime osd fails on 
> rebooting
> > until i start them manually or adding starting commands in rc.local.
> 
> in my case, it's a test environment, so I don't have notice those 
> behaviours
> 
> --
> Yoann
> 
> > On Mon, Jul 25, 2016 at 6:45 PM, Yoann Moulin  
> > >> wrote:
> >
> > Hello,
> >
> > (this is a repost, my previous message seems to be slipping under 
> the radar)
> >
> > Does anyone get a similar behaviour to the one described below ?
> >
> > I found a big performance drop between kernel 3.13.0-88 (default 
> kernel on
> > Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 
> (default kernel on
> > Ubuntu Xenial 16.04)
> >
> > - ceph version is Jewel (10.2.2).
> > - All tests have been done under Ubuntu 14.04 on
> > - Each cluster has 5 nodes strictly identical.
> > - Each node has 10 OSDs.
> > - Journals are on the disk.
> >
> > Kernel 4.4 has a drop of more than 50% compared to 4.2
> > Kernel 4.4 has a drop of 40% compared to 3.13
> >
> > details below :
> >
> > With the 3 kernel I have the same performance on disks :
> >
> > Raw benchmark:
> > dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> 
> average ~230MB/s
> > dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => 
> average ~220MB/s
> >
> > Filesystem mounted benchmark:
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => 
> average ~205MB/s
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => 
> average ~214MB/s
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => 
> average ~190MB/s
> >
> > Ceph osd Benchmark:
> > Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  
> ~81MB/s
> > Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average 
> ~109MB/s
> > Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  
> ~50MB/s
> >
> > I did new benchmarks then on 3 new fresh clusters.
> >
> > - Each cluster has 3 nodes strictly identical.
> > - Each node has 10 OSDs.
> > - Journals are on the disk.
> >
> > bench5 : Ubuntu 14.04 / Ceph Infernalis
> > bench6 : Ubuntu 14.04 / Ceph Jewel
> > bench7 : Ubuntu 16.04 / Ceph jewel
> >
> > this is the average of 2 runs of "ceph tell osd.* bench" on each 
> cluster (2 x 30
> > OSDs)
> >
> > bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
> > bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s
> >
> > bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
> > bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
> > bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s
> >
> > bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
> > bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
> > bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s
> >
> > If needed, I have the raw output of "ceph tell osd.* bench"
> >
> > Best regards
> 
> 


-- 
Yoann Moulin
EPFL IC-IT

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Lomayani S. Laizer]

Hello,
do you have journal on disk too ?

Yes am having journal on same hard disk.

ok and could you do bench with kernel 4.2 ? just to see if you have better
throughput. Thanks

In ubuntu 14 I was running 4.2 kernel. the throughput was the same around
80-90MB/s per osd. I cant tell the difference because each test gives the
speeds on same range. I did not test kernel 4.4 in ubuntu 14


--
Lomayani

On Tue, Jul 26, 2016 at 9:39 AM, Yoann Moulin  wrote:

> Hello,
>
> > Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are
> similar.
>
> do you have journal on disk too ?
>
> > I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have
> around
> > 80-90MB/s of OSD speeds in both operating systems
>
> ok and could you do bench with kernel 4.2 ? just to see if you have better
> throughput. Thanks
>
> > Only issue am observing now with ubuntu 16 is sometime osd fails on
> rebooting
> > until i start them manually or adding starting commands in rc.local.
>
> in my case, it's a test environment, so I don't have notice those
> behaviours
>
> --
> Yoann
>
> > On Mon, Jul 25, 2016 at 6:45 PM, Yoann Moulin  > > wrote:
> >
> > Hello,
> >
> > (this is a repost, my previous message seems to be slipping under
> the radar)
> >
> > Does anyone get a similar behaviour to the one described below ?
> >
> > I found a big performance drop between kernel 3.13.0-88 (default
> kernel on
> > Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default
> kernel on
> > Ubuntu Xenial 16.04)
> >
> > - ceph version is Jewel (10.2.2).
> > - All tests have been done under Ubuntu 14.04 on
> > - Each cluster has 5 nodes strictly identical.
> > - Each node has 10 OSDs.
> > - Journals are on the disk.
> >
> > Kernel 4.4 has a drop of more than 50% compared to 4.2
> > Kernel 4.4 has a drop of 40% compared to 3.13
> >
> > details below :
> >
> > With the 3 kernel I have the same performance on disks :
> >
> > Raw benchmark:
> > dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=>
> average ~230MB/s
> > dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   =>
> average ~220MB/s
> >
> > Filesystem mounted benchmark:
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  =>
> average ~205MB/s
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct =>
> average ~214MB/s
> > dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   =>
> average ~190MB/s
> >
> > Ceph osd Benchmark:
> > Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
> > Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
> > Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s
> >
> > I did new benchmarks then on 3 new fresh clusters.
> >
> > - Each cluster has 3 nodes strictly identical.
> > - Each node has 10 OSDs.
> > - Journals are on the disk.
> >
> > bench5 : Ubuntu 14.04 / Ceph Infernalis
> > bench6 : Ubuntu 14.04 / Ceph Jewel
> > bench7 : Ubuntu 16.04 / Ceph jewel
> >
> > this is the average of 2 runs of "ceph tell osd.* bench" on each
> cluster (2 x 30
> > OSDs)
> >
> > bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
> > bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s
> >
> > bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
> > bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
> > bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s
> >
> > bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
> > bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
> > bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s
> >
> > If needed, I have the raw output of "ceph tell osd.* bench"
> >
> > Best regards
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Yoann Moulin]

Hello Mark,

> FWIW, on CentOS7 I actually saw a performance increase when upgrading from the
> stock 3.10 kernel to 4.4.5 with Intel P3700 NVMe devices.  I was encountering
> some kind of strange concurrency/locking issues at the driver level that 4.4.5
> resolved.  I think your best bet is to try different intermediate kernels, 
> track
> it down as much as you can and then look through the kernel changelog.

The point here is I have only installed kernel from linux-image-virtual-lts
package, I expect for my future environment to stay on lts kernel package
maintained by security team.

anyway, I'm still in test, I can test kernels to try to find from which one the
regression start.

> Sorry I can't be of more help!

no problems :)

--
Yoann

> On 07/25/2016 10:45 AM, Yoann Moulin wrote:
>> Hello,
>>
>> (this is a repost, my previous message seems to be slipping under the radar)
>>
>> Does anyone get a similar behaviour to the one described below ?
>>
>> I found a big performance drop between kernel 3.13.0-88 (default kernel on
>> Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default kernel 
>> on
>> Ubuntu Xenial 16.04)
>>
>> - ceph version is Jewel (10.2.2).
>> - All tests have been done under Ubuntu 14.04 on
>> - Each cluster has 5 nodes strictly identical.
>> - Each node has 10 OSDs.
>> - Journals are on the disk.
>>
>> Kernel 4.4 has a drop of more than 50% compared to 4.2
>> Kernel 4.4 has a drop of 40% compared to 3.13
>>
>> details below :
>>
>> With the 3 kernel I have the same performance on disks :
>>
>> Raw benchmark:
>> dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> average 
>> ~230MB/s
>> dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => average 
>> ~220MB/s
>>
>> Filesystem mounted benchmark:
>> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => average 
>> ~205MB/s
>> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => average 
>> ~214MB/s
>> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => average 
>> ~190MB/s
>>
>> Ceph osd Benchmark:
>> Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
>> Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
>> Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s
>>
>> I did new benchmarks then on 3 new fresh clusters.
>>
>> - Each cluster has 3 nodes strictly identical.
>> - Each node has 10 OSDs.
>> - Journals are on the disk.
>>
>> bench5 : Ubuntu 14.04 / Ceph Infernalis
>> bench6 : Ubuntu 14.04 / Ceph Jewel
>> bench7 : Ubuntu 16.04 / Ceph jewel
>>
>> this is the average of 2 runs of "ceph tell osd.* bench" on each cluster (2 
>> x 30
>> OSDs)
>>
>> bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
>> bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s
>>
>> bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
>> bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
>> bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s
>>
>> bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
>> bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
>> bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s
>>
>> If needed, I have the raw output of "ceph tell osd.* bench"
>>
>> Best regards
>>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Yoann Moulin]

Hello,

> Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are similar.

do you have journal on disk too ?

> I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have around
> 80-90MB/s of OSD speeds in both operating systems

ok and could you do bench with kernel 4.2 ? just to see if you have better
throughput. Thanks

> Only issue am observing now with ubuntu 16 is sometime osd fails on rebooting
> until i start them manually or adding starting commands in rc.local.

in my case, it's a test environment, so I don't have notice those behaviours

--
Yoann

> On Mon, Jul 25, 2016 at 6:45 PM, Yoann Moulin  > wrote:
> 
> Hello,
> 
> (this is a repost, my previous message seems to be slipping under the 
> radar)
> 
> Does anyone get a similar behaviour to the one described below ?
> 
> I found a big performance drop between kernel 3.13.0-88 (default kernel on
> Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default 
> kernel on
> Ubuntu Xenial 16.04)
> 
> - ceph version is Jewel (10.2.2).
> - All tests have been done under Ubuntu 14.04 on
> - Each cluster has 5 nodes strictly identical.
> - Each node has 10 OSDs.
> - Journals are on the disk.
> 
> Kernel 4.4 has a drop of more than 50% compared to 4.2
> Kernel 4.4 has a drop of 40% compared to 3.13
> 
> details below :
> 
> With the 3 kernel I have the same performance on disks :
> 
> Raw benchmark:
> dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> average 
> ~230MB/s
> dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => average 
> ~220MB/s
> 
> Filesystem mounted benchmark:
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => average 
> ~205MB/s
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => average 
> ~214MB/s
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => average 
> ~190MB/s
> 
> Ceph osd Benchmark:
> Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
> Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
> Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s
> 
> I did new benchmarks then on 3 new fresh clusters.
> 
> - Each cluster has 3 nodes strictly identical.
> - Each node has 10 OSDs.
> - Journals are on the disk.
> 
> bench5 : Ubuntu 14.04 / Ceph Infernalis
> bench6 : Ubuntu 14.04 / Ceph Jewel
> bench7 : Ubuntu 16.04 / Ceph jewel
> 
> this is the average of 2 runs of "ceph tell osd.* bench" on each cluster 
> (2 x 30
> OSDs)
> 
> bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
> bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s
> 
> bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
> bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
> bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s
> 
> bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
> bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
> bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s
> 
> If needed, I have the raw output of "ceph tell osd.* bench"
> 
> Best regards
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Lomayani S. Laizer]

Hello,
Am running ubuntu 16 with kernel 4.4-0.31-generic and my speed are similar.

I did tests on ubuntu 14 and Ubuntu 16 and the speed is similar. I have
around 80-90MB/s of OSD speeds in both operating systems

Only issue am observing now with ubuntu 16 is sometime osd fails on
rebooting until i start them manually or adding starting commands in
rc.local.

--

Lomayani

On Mon, Jul 25, 2016 at 6:45 PM, Yoann Moulin  wrote:

> Hello,
>
> (this is a repost, my previous message seems to be slipping under the
> radar)
>
> Does anyone get a similar behaviour to the one described below ?
>
> I found a big performance drop between kernel 3.13.0-88 (default kernel on
> Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default
> kernel on
> Ubuntu Xenial 16.04)
>
> - ceph version is Jewel (10.2.2).
> - All tests have been done under Ubuntu 14.04 on
> - Each cluster has 5 nodes strictly identical.
> - Each node has 10 OSDs.
> - Journals are on the disk.
>
> Kernel 4.4 has a drop of more than 50% compared to 4.2
> Kernel 4.4 has a drop of 40% compared to 3.13
>
> details below :
>
> With the 3 kernel I have the same performance on disks :
>
> Raw benchmark:
> dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> average
> ~230MB/s
> dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => average
> ~220MB/s
>
> Filesystem mounted benchmark:
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => average
> ~205MB/s
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => average
> ~214MB/s
> dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => average
> ~190MB/s
>
> Ceph osd Benchmark:
> Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
> Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
> Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s
>
> I did new benchmarks then on 3 new fresh clusters.
>
> - Each cluster has 3 nodes strictly identical.
> - Each node has 10 OSDs.
> - Journals are on the disk.
>
> bench5 : Ubuntu 14.04 / Ceph Infernalis
> bench6 : Ubuntu 14.04 / Ceph Jewel
> bench7 : Ubuntu 16.04 / Ceph jewel
>
> this is the average of 2 runs of "ceph tell osd.* bench" on each cluster
> (2 x 30
> OSDs)
>
> bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
> bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s
>
> bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
> bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
> bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s
>
> bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
> bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
> bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s
>
> If needed, I have the raw output of "ceph tell osd.* bench"
>
> Best regards
>
> --
> Yoann Moulin
> EPFL IC-IT
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Mark Nelson]

Hi Yoann,

FWIW, on CentOS7 I actually saw a performance increase when upgrading 
from the stock 3.10 kernel to 4.4.5 with Intel P3700 NVMe devices.  I 
was encountering some kind of strange concurrency/locking issues at the 
driver level that 4.4.5 resolved.  I think your best bet is to try 
different intermediate kernels, track it down as much as you can and 
then look through the kernel changelog.


Sorry I can't be of more help!

Mark

On 07/25/2016 10:45 AM, Yoann Moulin wrote:

Hello,

(this is a repost, my previous message seems to be slipping under the radar)

Does anyone get a similar behaviour to the one described below ?

I found a big performance drop between kernel 3.13.0-88 (default kernel on
Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default kernel on
Ubuntu Xenial 16.04)

- ceph version is Jewel (10.2.2).
- All tests have been done under Ubuntu 14.04 on
- Each cluster has 5 nodes strictly identical.
- Each node has 10 OSDs.
- Journals are on the disk.

Kernel 4.4 has a drop of more than 50% compared to 4.2
Kernel 4.4 has a drop of 40% compared to 3.13

details below :

With the 3 kernel I have the same performance on disks :

Raw benchmark:
dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> average ~230MB/s
dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => average ~220MB/s

Filesystem mounted benchmark:
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => average ~205MB/s
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => average ~214MB/s
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => average ~190MB/s

Ceph osd Benchmark:
Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s

I did new benchmarks then on 3 new fresh clusters.

- Each cluster has 3 nodes strictly identical.
- Each node has 10 OSDs.
- Journals are on the disk.

bench5 : Ubuntu 14.04 / Ceph Infernalis
bench6 : Ubuntu 14.04 / Ceph Jewel
bench7 : Ubuntu 16.04 / Ceph jewel

this is the average of 2 runs of "ceph tell osd.* bench" on each cluster (2 x 30
OSDs)

bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s

bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s

bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s

If needed, I have the raw output of "ceph tell osd.* bench"

Best regards


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Jewel + kernel 4.4 Massive performance regression (-50%)

[Yoann Moulin]

Hello,

(this is a repost, my previous message seems to be slipping under the radar)

Does anyone get a similar behaviour to the one described below ?

I found a big performance drop between kernel 3.13.0-88 (default kernel on
Ubuntu Trusty 14.04) or kernel 4.2.0 and kernel 4.4.0.24.14 (default kernel on
Ubuntu Xenial 16.04)

- ceph version is Jewel (10.2.2).
- All tests have been done under Ubuntu 14.04 on
- Each cluster has 5 nodes strictly identical.
- Each node has 10 OSDs.
- Journals are on the disk.

Kernel 4.4 has a drop of more than 50% compared to 4.2
Kernel 4.4 has a drop of 40% compared to 3.13

details below :

With the 3 kernel I have the same performance on disks :

Raw benchmark:
dd if=/dev/zero of=/dev/sdX bs=1M count=1024 oflag=direct=> average ~230MB/s
dd if=/dev/zero of=/dev/sdX bs=1G count=1 oflag=direct   => average ~220MB/s

Filesystem mounted benchmark:
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1  => average ~205MB/s
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=direct => average ~214MB/s
dd if=/dev/zero of=/sdX1/test.img bs=1G count=1 oflag=sync   => average ~190MB/s

Ceph osd Benchmark:
Kernel 3.13.0-88-generic : ceph tell osd.ID bench => average  ~81MB/s
Kernel 4.2.0-38-generic  : ceph tell osd.ID bench => average ~109MB/s
Kernel 4.4.0-24-generic  : ceph tell osd.ID bench => average  ~50MB/s

I did new benchmarks then on 3 new fresh clusters.

- Each cluster has 3 nodes strictly identical.
- Each node has 10 OSDs.
- Journals are on the disk.

bench5 : Ubuntu 14.04 / Ceph Infernalis
bench6 : Ubuntu 14.04 / Ceph Jewel
bench7 : Ubuntu 16.04 / Ceph jewel

this is the average of 2 runs of "ceph tell osd.* bench" on each cluster (2 x 30
OSDs)

bench5 / 14.04 / Infernalis / kernel 3.13 :  54.35 MB/s
bench6 / 14.04 / Jewel  / kernel 3.13 :  86.47 MB/s

bench5 / 14.04 / Infernalis / kernel 4.2  :  63.38 MB/s
bench6 / 14.04 / Jewel  / kernel 4.2  : 107.75 MB/s
bench7 / 16.04 / Jewel  / kernel 4.2  : 101.54 MB/s

bench5 / 14.04 / Infernalis / kernel 4.4  :  53.61 MB/s
bench6 / 14.04 / Jewel  / kernel 4.4  :  65.82 MB/s
bench7 / 16.04 / Jewel  / kernel 4.4  :  61.57 MB/s

If needed, I have the raw output of "ceph tell osd.* bench"

Best regards

-- 
Yoann Moulin
EPFL IC-IT
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com