Dear ceph community,
I just installed ceph luminous in a small NVMe cluster for testing and I tested
2 clients:
Client 1:
VM running centos 7
Ceph client: kernel
# cpus: 4
RAM: 16GB
Fio test
# sudo fio --name=xx --filename=/mnt/mycephfs/test.file3 --filesize=100G
--iodepth=1 --rw=write --bs=4M --numjobs=2 --group_reporting
xx: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T)
4096KiB-4096KiB, ioengine=psync, iodepth=1
...
fio-3.1
Starting 2 processes
xx: Laying out IO file (1 file / 102400MiB)
Jobs: 1 (f=1): [_(1),W(1)][100.0%][r=0KiB/s,w=2325MiB/s][r=0,w=581 IOPS][eta
00m:00s]
xx: (groupid=0, jobs=2): err= 0: pid=24290: Mon Jul 9 17:54:57 2018
write: IOPS=550, BW=2203MiB/s (2310MB/s)(200GiB/92946msec)
clat (usec): min=946, max=464990, avg=3519.59, stdev=7031.97
lat (usec): min=1010, max=465091, avg=3612.53, stdev=7035.85
clat percentiles (usec):
| 1.00th=[ 1188], 5.00th=[ 1631], 10.00th=[ 2245], 20.00th=[ 2409],
| 30.00th=[ 2540], 40.00th=[ 2671], 50.00th=[ 2802], 60.00th=[ 2966],
| 70.00th=[ 3195], 80.00th=[ 3654], 90.00th=[ 5080], 95.00th=[ 6521],
| 99.00th=[ 11469], 99.50th=[ 16450], 99.90th=[100140], 99.95th=[149947],
| 99.99th=[291505]
bw ( MiB/s): min= 224, max= 2064, per=50.01%, avg=1101.97, stdev=205.16,
samples=369
iops : min= 56, max= 516, avg=275.27, stdev=51.29, samples=369
lat (usec) : 1000=0.01%
lat (msec) : 2=7.89%, 4=75.24%, 10=15.42%, 20=1.09%, 50=0.22%
lat (msec) : 100=0.04%, 250=0.08%, 500=0.02%
cpu : usr=2.31%, sys=76.39%, ctx=15743, majf=1, minf=55
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwt: total=0,51200,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: bw=2203MiB/s (2310MB/s), 2203MiB/s-2203MiB/s (2310MB/s-2310MB/s),
io=200GiB (215GB), run=92946-92946msec
Client 2:
Physical machine running Ubuntu xenial
Ceph client: FUSE
# cpus: 56
RAM: 512
Fio test
$ sudo fio --name=xx --filename=/mnt/cephfs/test.file2 --filesize=5G
--iodepth=1 --rw=write --bs=4M --numjobs=1 --group_reporting
xx: (g=0): rw=write, bs=4M-4M/4M-4M/4M-4M, ioengine=sync, iodepth=1
fio-2.2.10
Starting 1 process
xx: Laying out IO file(s) (1 file(s) / 5120MB)
Jobs: 1 (f=1): [W(1)] [91.7% done] [0KB/580.0MB/0KB /s] [0/145/0 iops] [eta
00m:01s]
xx: (groupid=0, jobs=1): err= 0: pid=6065: Mon Jul 9 17:44:02 2018
write: io=5120.0MB, bw=497144KB/s, iops=121, runt= 10546msec
clat (msec): min=3, max=159, avg= 7.94, stdev= 4.81
lat (msec): min=3, max=159, avg= 8.08, stdev= 4.82
clat percentiles (msec):
| 1.00th=[ 4], 5.00th=[ 5], 10.00th=[ 6], 20.00th=[ 7],
| 30.00th=[ 7], 40.00th=[ 8], 50.00th=[ 8], 60.00th=[ 9],
| 70.00th=[ 9], 80.00th=[ 10], 90.00th=[ 11], 95.00th=[ 11],
| 99.00th=[ 12], 99.50th=[ 13], 99.90th=[ 61], 99.95th=[ 159],
| 99.99th=[ 159]
bw (KB /s): min=185448, max=726183, per=97.08%, avg=482611.80,
stdev=118874.09
lat (msec) : 4=1.64%, 10=88.20%, 20=10.00%, 100=0.08%, 250=0.08%
cpu : usr=1.63%, sys=34.44%, ctx=42266, majf=0, minf=1586
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=1280/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=5120.0MB, aggrb=497143KB/s, minb=497143KB/s, maxb=497143KB/s,
mint=10546msec, maxt=10546msec
NOTE: I did an iperf test from Client 2 to ceph nodes and the bandwidth is
~25GBs
QUESTION:
According to the documentation, FUSE is supposed to run slower. I found the
client 2 using FUSE being much slower than client 1. Could someone advice if
this is expected?
Thank you very much
Manuel Sopena Ballesteros | Big data Engineer
Garvan Institute of Medical Research
The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010
T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E:
[email protected]<mailto:[email protected]>
NOTICE
Please consider the environment before printing this email. This message and
any attachments are intended for the addressee named and may contain legally
privileged/confidential/copyright information. If you are not the intended
recipient, you should not read, use, disclose, copy or distribute this
communication. If you have received this message in error please notify us at
once by return email and then delete both messages. We accept no liability for
the distribution of viruses or similar in electronic communications. This
notice should not be removed.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
