Re: [ceph-users] ceph luminous - performance issue

2018-01-07 Thread Steven Vacaroaia 
Sorry for the delay

Here are the results when using bs=16k and rw=write
( Note: I am running the command directly on a OSD host as root)

fio /home/cephuser/write.fio

write-4M: (g=0): rw=write, bs=16K-16K/16K-16K/16K-16K, ioengine=rbd,
iodepth=32
fio-2.2.8
Starting 1 process
rbd engine: RBD version: 1.12.0
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/172.6MB/0KB /s] [0/11.5K/0 iops]
[eta 00m:00s]


Here are the results when runnnig with bs=4k and rw=randwrite

[root@osd03 ~]# fio /home/cephuser/write.fio
write-4M: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=rbd,
iodepth=32
fio-2.2.8
Starting 1 process
rbd engine: RBD version: 1.12.0
Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/54056KB/0KB /s] [0/13.6K/0 iops]
[eta 00m:00s]


On 3 January 2018 at 15:28, <ceph.nov...@habmalnefrage.de> wrote:

> Hi Steven.
>
> interesting... 'm quite curious after your post now.
>
> I've migrated our prod. CEPH cluster to 12.2.2 and Bluestore just today
> and haven't heard back anything "bad" from the applications/users so far.
> performance tests on our test cluster were good before, but we use S3/RGW
> only anyhow ;)
>
> there are two things I would like to know/learn... could you try/test and
> feed back?!
>
> - change all your tests to use >=16k block size, see also BStore comments
> here (https://www.mail-archive.com/ceph-users@lists.ceph.com/msg43023.html
> )
> - change your "write.fio" file profile from "rw=randwrite" to "rw=write"
> (or something similar :O ) to compare apples with apples ;)
>
> thanks for your efforts and looking forward for those results ;)
>
> best regards
>  Notna
>
> 
> --
>
> Gesendet: Mittwoch, 03. Januar 2018 um 16:20 Uhr
> Von: "Steven Vacaroaia" <ste...@gmail.com>
> An: "Brady Deetz" <bde...@gmail.com>
> Cc: ceph-users <ceph-users@lists.ceph.com>
> Betreff: Re: [ceph-users] ceph luminous - performance issue
>
> Thanks for your willingness to help
>
> DELL R620, 1 CPU, 8 cores, 64 GB RAM
> cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000
>
> SSD drives are Enterprise grade  - 400 GB SSD  Toshiba PX04SHB040
> HDD drives are  - 10k RPM, 600 GB  Toshiba AL13SEB600
>
> Steven
>
>
> On 3 January 2018 at 09:41, Brady Deetz <bde...@gmail.com[mailto:bdeet
> z...@gmail.com]> wrote:
> Can you provide more detail regarding the infrastructure backing this
> environment? What hard drive, ssd, and processor are you using? Also, what
> is providing networking?
>
> I'm seeing 4k blocksize tests here. Latency is going to destroy you.
>
>
> On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" <ste...@gmail.com[mailto:stef9
> 7...@gmail.com]> wrote:
>
> Hi,
>
> I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each
> server), bluestore
>
> I configured the OSD using the following ( /dev/sda is my SSD drive)
> ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde
> --block.wal /dev/sda --block.db /dev/sda
>
> Unfortunately both fio and bench tests show much worse performance for the
> pools than for the individual disks
>
> Example:
> DISKS
> fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k
> --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting
> --name=journal-test
>
> SSD drive
> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0
> iops] [eta 00m:00s]
>
> HD drive
> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0
> iops] [eta 00m:00s]
>
> POOL
>
> fio write.fio
> Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops]
>
>
>  cat write.fio
> [write-4M]
> description="write test with 4k block"
> ioengine=rbd
> clientname=admin
> pool=scbench
> rbdname=image01
> iodepth=32
> runtime=120
> rw=randwrite
> bs=4k
>
>
> rados bench -p scbench 12 write
>
>
> Max bandwidth (MB/sec): 224
> Min bandwidth (MB/sec): 0
> Average IOPS:   26
> Stddev IOPS:24
> Max IOPS:   56
> Min IOPS:   0
> Average Latency(s): 0.59819
> Stddev Latency(s):  1.64017
> Max latency(s): 10.8335
> Min latency(s): 0.00475139
>
>
>
>
> I must be missing something - any help/suggestions will be greatly
> appreciated
>
> Here are some specific info
>
>
> ceph -s
>   cluster:
> id: 91118dde-f231-4e54-a5f0-a1037f3d5142
> health: HEALTH_OK
>
>   services:
> mon: 1 daemons, quorum mon01
> mgr: mon01(active)
> osd: 12 osds: 12 up, 12 in
>
>   d

Re: [ceph-users] ceph luminous - performance issue

2018-01-03 Thread ceph . novice 
Hi Steven.

interesting... 'm quite curious after your post now.

I've migrated our prod. CEPH cluster to 12.2.2 and Bluestore just today and 
haven't heard back anything "bad" from the applications/users so far. 
performance tests on our test cluster were good before, but we use S3/RGW only 
anyhow ;)

there are two things I would like to know/learn... could you try/test and feed 
back?!

- change all your tests to use >=16k block size, see also BStore comments here 
(https://www.mail-archive.com/ceph-users@lists.ceph.com/msg43023.html)
- change your "write.fio" file profile from "rw=randwrite" to "rw=write" (or 
something similar :O ) to compare apples with apples ;)

thanks for your efforts and looking forward for those results ;)

best regards
 Notna

--

Gesendet: Mittwoch, 03. Januar 2018 um 16:20 Uhr
Von: "Steven Vacaroaia" <ste...@gmail.com>
An: "Brady Deetz" <bde...@gmail.com>
Cc: ceph-users <ceph-users@lists.ceph.com>
Betreff: Re: [ceph-users] ceph luminous - performance issue

Thanks for your willingness to help
 
DELL R620, 1 CPU, 8 cores, 64 GB RAM 
cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000
 
SSD drives are Enterprise grade  - 400 GB SSD  Toshiba PX04SHB040
HDD drives are  - 10k RPM, 600 GB  Toshiba AL13SEB600
 
Steven

 
On 3 January 2018 at 09:41, Brady Deetz 
<bde...@gmail.com[mailto:bde...@gmail.com]> wrote:
Can you provide more detail regarding the infrastructure backing this 
environment? What hard drive, ssd, and processor are you using? Also, what is 
providing networking?
 
I'm seeing 4k blocksize tests here. Latency is going to destroy you. 
 

On Jan 3, 2018 8:11 AM, "Steven Vacaroaia" 
<ste...@gmail.com[mailto:ste...@gmail.com]> wrote:

Hi,
 
I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each 
server), bluestore 
 
I configured the OSD using the following ( /dev/sda is my SSD drive)
ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde --block.wal 
/dev/sda --block.db /dev/sda
 
Unfortunately both fio and bench tests show much worse performance for the 
pools than for the individual disks 
 
Example:
DISKS
fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k --numjobs=14 
--iodepth=1 --runtime=60 --time_based --group_reporting --name=journal-test
 
SSD drive
Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0 iops] 
[eta 00m:00s]
 
HD drive
Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0 iops] 
[eta 00m:00s]
 
POOL
 
fio write.fio
Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops] 
 

 cat write.fio
[write-4M]
description="write test with 4k block"
ioengine=rbd
clientname=admin
pool=scbench
rbdname=image01
iodepth=32
runtime=120
rw=randwrite
bs=4k
 
 
rados bench -p scbench 12 write
 

Max bandwidth (MB/sec): 224
Min bandwidth (MB/sec): 0
Average IOPS:           26
Stddev IOPS:            24
Max IOPS:               56
Min IOPS:               0
Average Latency(s):     0.59819
Stddev Latency(s):      1.64017
Max latency(s):         10.8335
Min latency(s):         0.00475139
 
 
 
 
I must be missing something - any help/suggestions will be greatly appreciated 
 
Here are some specific info 
 

ceph -s
  cluster:
    id:     91118dde-f231-4e54-a5f0-a1037f3d5142
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum mon01
    mgr: mon01(active)
    osd: 12 osds: 12 up, 12 in
 
  data:
    pools:   4 pools, 484 pgs
    objects: 70082 objects, 273 GB
    usage:   570 GB used, 6138 GB / 6708 GB avail
    pgs:     484 active+clean
 
  io:
    client:   2558 B/s rd, 2 op/s rd, 0 op/s wr
 

ceph osd pool ls detail
pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0 object_hash 
rjenkins pg_num 128 pgp_num 128 last_change 157 flags hashpspool stripe_width 0 
application rbd
        removed_snaps [1~3]
pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash 
rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool stripe_width 
8192 application rbd
        removed_snaps [1~5]
pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins 
pg_num 128 pgp_num 128 last_change 200 flags hashpspool stripe_width 0 
application rbd
        removed_snaps [1~3]
pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins 
pg_num 100 pgp_num 100 last_change 330 flags hashpspool stripe_width 0
        removed_snaps [1~3]
 
[cephuser@ceph ceph-config]$ ceph osd df tree
ID CLASS WEIGHT  REWEIGHT SIZE  USE    AVAIL %USE  VAR  PGS TYPE NAME
-1       6.55128        - 2237G   198G 2038G     0    0   - root default
-7             0        -     0      0     0     0    0   -     host ods03
-3       2.18475        - 2237G   181G 2055G  8.12 0.96   -     host osd01
 3   hdd 0.54619  1.0  559G 53890M  506G  9.41

Re: [ceph-users] ceph luminous - performance issue

2018-01-03 Thread Steven Vacaroaia 
Thanks for your willingness to help

DELL R620, 1 CPU, 8 cores, 64 GB RAM
cluster network is using 2 bonded 10 GB NICs ( mode=4), MTU=9000

SSD drives are Enterprise grade  - 400 GB SSD  Toshiba PX04SHB040

HDD drives are  - 10k RPM, 600 GB  Toshiba AL13SEB600


Steven

On 3 January 2018 at 09:41, Brady Deetz  wrote:

> Can you provide more detail regarding the infrastructure backing this
> environment? What hard drive, ssd, and processor are you using? Also, what
> is providing networking?
>
> I'm seeing 4k blocksize tests here. Latency is going to destroy you.
>
> On Jan 3, 2018 8:11 AM, "Steven Vacaroaia"  wrote:
>
>> Hi,
>>
>> I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each
>> server), bluestore
>>
>> I configured the OSD using the following ( /dev/sda is my SSD drive)
>> ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde
>> --block.wal /dev/sda --block.db /dev/sda
>>
>> Unfortunately both fio and bench tests show much worse performance for
>> the pools than for the individual disks
>>
>> Example:
>> DISKS
>> fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k
>> --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting
>> --name=journal-test
>>
>> SSD drive
>> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0
>> iops] [eta 00m:00s]
>>
>> HD drive
>> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0
>> iops] [eta 00m:00s]
>>
>> POOL
>>
>> fio write.fio
>> Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops]
>>
>>  cat write.fio
>> [write-4M]
>> description="write test with 4k block"
>> ioengine=rbd
>> clientname=admin
>> pool=scbench
>> rbdname=image01
>> iodepth=32
>> runtime=120
>> rw=randwrite
>> bs=4k
>>
>>
>> rados bench -p scbench 12 write
>>
>> Max bandwidth (MB/sec): 224
>> Min bandwidth (MB/sec): 0
>> Average IOPS:   26
>> Stddev IOPS:24
>> Max IOPS:   56
>> Min IOPS:   0
>> Average Latency(s): 0.59819
>> Stddev Latency(s):  1.64017
>> Max latency(s): 10.8335
>> Min latency(s): 0.00475139
>>
>>
>>
>>
>> I must be missing something - any help/suggestions will be greatly
>> appreciated
>>
>> Here are some specific info
>>
>> ceph -s
>>   cluster:
>> id: 91118dde-f231-4e54-a5f0-a1037f3d5142
>> health: HEALTH_OK
>>
>>   services:
>> mon: 1 daemons, quorum mon01
>> mgr: mon01(active)
>> osd: 12 osds: 12 up, 12 in
>>
>>   data:
>> pools:   4 pools, 484 pgs
>> objects: 70082 objects, 273 GB
>> usage:   570 GB used, 6138 GB / 6708 GB avail
>> pgs: 484 active+clean
>>
>>   io:
>> client:   2558 B/s rd, 2 op/s rd, 0 op/s wr
>>
>> ceph osd pool ls detail
>> pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0
>> object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags
>> hashpspool stripe_width 0 application rbd
>> removed_snaps [1~3]
>> pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash
>> rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool
>> stripe_width 8192 application rbd
>> removed_snaps [1~5]
>> pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash
>> rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool
>> stripe_width 0 application rbd
>> removed_snaps [1~3]
>> pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash
>> rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool
>> stripe_width 0
>> removed_snaps [1~3]
>>
>> [cephuser@ceph ceph-config]$ ceph osd df tree
>> ID CLASS WEIGHT  REWEIGHT SIZE  USEAVAIL %USE  VAR  PGS TYPE NAME
>> -1   6.55128- 2237G   198G 2038G 00   - root default
>> -7 0- 0  0 0 00   - host ods03
>> -3   2.18475- 2237G   181G 2055G  8.12 0.96   - host osd01
>>  3   hdd 0.54619  1.0  559G 53890M  506G  9.41 1.11  90 osd.3
>>  4   hdd 0.54619  1.0  559G 30567M  529G  5.34 0.63  89 osd.4
>>  5   hdd 0.54619  1.0  559G 59385M  501G 10.37 1.22  93 osd.5
>>  6   hdd 0.54619  1.0  559G 42156M  518G  7.36 0.87  93 osd.6
>> -5   2.18178- 2234G   189G 2044G  8.50 1.00   - host osd02
>>  0   hdd 0.54520  1.0  558G 32460M  526G  5.68 0.67  90 osd.0
>>  1   hdd 0.54520  1.0  558G 54578M  504G  9.55 1.12  89 osd.1
>>  2   hdd 0.54520  1.0  558G 47761M  511G  8.35 0.98  93 osd.2
>>  7   hdd 0.54619  1.0  559G 59584M  501G 10.40 1.22  92 osd.7
>> -9   2.18475- 2237G   198G 2038G  8.88 1.04   - host osd03
>>  8   hdd 0.54619  1.0  559G 52462M  508G  9.16 1.08  99 osd.8
>> 10   hdd 0.54619  1.0  559G 35284M  524G  6.16 0.73  88 osd.10
>> 11   hdd 0.54619  1.0  559G 71739M  489G 12.53 1.47  87 osd.11
>> 12   hdd 0.54619  1.0  559G

Re: [ceph-users] ceph luminous - performance issue

2018-01-03 Thread Brady Deetz 
Can you provide more detail regarding the infrastructure backing this
environment? What hard drive, ssd, and processor are you using? Also, what
is providing networking?

I'm seeing 4k blocksize tests here. Latency is going to destroy you.

On Jan 3, 2018 8:11 AM, "Steven Vacaroaia"  wrote:

> Hi,
>
> I am doing a PoC with 3 DELL R620 and 12 OSD , 3 SSD drives ( one on each
> server), bluestore
>
> I configured the OSD using the following ( /dev/sda is my SSD drive)
> ceph-disk prepare --zap-disk --cluster ceph  --bluestore /dev/sde
> --block.wal /dev/sda --block.db /dev/sda
>
> Unfortunately both fio and bench tests show much worse performance for the
> pools than for the individual disks
>
> Example:
> DISKS
> fio --filename=/dev/sda --direct=1 --sync=1 --rw=write --bs=4k
> --numjobs=14 --iodepth=1 --runtime=60 --time_based --group_reporting
> --name=journal-test
>
> SSD drive
> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/465.2MB/0KB /s] [0/119K/0
> iops] [eta 00m:00s]
>
> HD drive
> Jobs: 14 (f=14): [W(14)] [100.0% done] [0KB/179.2MB/0KB /s] [0/45.9K/0
> iops] [eta 00m:00s]
>
> POOL
>
> fio write.fio
> Jobs: 1 (f=0): [w(1)] [100.0% done] [0KB/51428KB/0KB /s] [0/12.9K/0 iops]
>
>  cat write.fio
> [write-4M]
> description="write test with 4k block"
> ioengine=rbd
> clientname=admin
> pool=scbench
> rbdname=image01
> iodepth=32
> runtime=120
> rw=randwrite
> bs=4k
>
>
> rados bench -p scbench 12 write
>
> Max bandwidth (MB/sec): 224
> Min bandwidth (MB/sec): 0
> Average IOPS:   26
> Stddev IOPS:24
> Max IOPS:   56
> Min IOPS:   0
> Average Latency(s): 0.59819
> Stddev Latency(s):  1.64017
> Max latency(s): 10.8335
> Min latency(s): 0.00475139
>
>
>
>
> I must be missing something - any help/suggestions will be greatly
> appreciated
>
> Here are some specific info
>
> ceph -s
>   cluster:
> id: 91118dde-f231-4e54-a5f0-a1037f3d5142
> health: HEALTH_OK
>
>   services:
> mon: 1 daemons, quorum mon01
> mgr: mon01(active)
> osd: 12 osds: 12 up, 12 in
>
>   data:
> pools:   4 pools, 484 pgs
> objects: 70082 objects, 273 GB
> usage:   570 GB used, 6138 GB / 6708 GB avail
> pgs: 484 active+clean
>
>   io:
> client:   2558 B/s rd, 2 op/s rd, 0 op/s wr
>
> ceph osd pool ls detail
> pool 1 'test-replicated' replicated size 2 min_size 1 crush_rule 0
> object_hash rjenkins pg_num 128 pgp_num 128 last_change 157 flags
> hashpspool stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 2 'test-erasure' erasure size 3 min_size 3 crush_rule 1 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 334 flags hashpspool
> stripe_width 8192 application rbd
> removed_snaps [1~5]
> pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 128 pgp_num 128 last_change 200 flags hashpspool
> stripe_width 0 application rbd
> removed_snaps [1~3]
> pool 4 'scbench' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 100 pgp_num 100 last_change 330 flags hashpspool
> stripe_width 0
> removed_snaps [1~3]
>
> [cephuser@ceph ceph-config]$ ceph osd df tree
> ID CLASS WEIGHT  REWEIGHT SIZE  USEAVAIL %USE  VAR  PGS TYPE NAME
> -1   6.55128- 2237G   198G 2038G 00   - root default
> -7 0- 0  0 0 00   - host ods03
> -3   2.18475- 2237G   181G 2055G  8.12 0.96   - host osd01
>  3   hdd 0.54619  1.0  559G 53890M  506G  9.41 1.11  90 osd.3
>  4   hdd 0.54619  1.0  559G 30567M  529G  5.34 0.63  89 osd.4
>  5   hdd 0.54619  1.0  559G 59385M  501G 10.37 1.22  93 osd.5
>  6   hdd 0.54619  1.0  559G 42156M  518G  7.36 0.87  93 osd.6
> -5   2.18178- 2234G   189G 2044G  8.50 1.00   - host osd02
>  0   hdd 0.54520  1.0  558G 32460M  526G  5.68 0.67  90 osd.0
>  1   hdd 0.54520  1.0  558G 54578M  504G  9.55 1.12  89 osd.1
>  2   hdd 0.54520  1.0  558G 47761M  511G  8.35 0.98  93 osd.2
>  7   hdd 0.54619  1.0  559G 59584M  501G 10.40 1.22  92 osd.7
> -9   2.18475- 2237G   198G 2038G  8.88 1.04   - host osd03
>  8   hdd 0.54619  1.0  559G 52462M  508G  9.16 1.08  99 osd.8
> 10   hdd 0.54619  1.0  559G 35284M  524G  6.16 0.73  88 osd.10
> 11   hdd 0.54619  1.0  559G 71739M  489G 12.53 1.47  87 osd.11
> 12   hdd 0.54619  1.0  559G 43832M  516G  7.65 0.90  93 osd.12
> TOTAL 6708G   570G 6138G  8.50
> MIN/MAX VAR: 0.63/1.47  STDDEV: 2.06
>
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com