Hi David,
This is a long shot, but have you checked the Max queue depth on the
iscsi side. I've got a feeling that lio might be set at 32 as default.
This would definitely have an effect at the high queue depths you are
testing with.
On 8 Dec 2014 16:53, David Moreau Simard <dmsim...@iweb.com> wrote:
Haven't tried other iSCSI implementations (yet). LIO/targetcli makes
it very easy to iQuoting David Moreau Simard <dmsim...@iweb.com>
Haven't tried other iSCSI implementations (yet).
LIO/targetcli makes it very easy to
implement/integrate/wrap/automate around so I'm really trying to get
this right.
PCI-E SSD cache tier in front of spindles-backed erasure coded pool
in 10 Gbps across the board yields results slightly better or very
similar to two spindles in hardware RAID-0 with writeback caching.
With that in mind, the performance is not outright awful by any
means, there's just a lot of overhead we have to be reminded about.
What I'd like to further test but am unable to right now is to see
what happens if you scale up the cluster. Right now I'm testing on
only two nodes.
Does the IOPS scale linearly with increasing amount of OSDs/servers
? Or is it more about a capacity thing ?
Perhaps if someone else can chime in, I'm really curious.
--
David Moreau Simard
On Dec 6, 2014, at 11:18 AM, Nick Fisk <n...@fisk.me.uk> wrote:
Hi David,
Very strange, but I'm glad you managed to finally get the cluster working
normally. Thank you for posting the benchmarks figures, it's interesting to
see the overhead of LIO over pure RBD performance.
I should have the hardware for our cluster up and running early next year, I
will be in a better position to test the iSCSI performance then. I will
report back once I have some numbers.
Just out of interest, have you tried any of the other iSCSI implementations
to see if they show the same performance drop?
Nick
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
David Moreau Simard
Sent: 05 December 2014 16:03
To: Nick Fisk
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Poor RBD performance as LIO iSCSI target
I've flushed everything - data, pools, configs and reconfigured the whole
thing.
I was particularly careful with cache tiering configurations (almost leaving
defaults when possible) and it's not locking anymore.
It looks like the cache tiering configuration I had was causing the problem
? I can't put my finger on exactly what/why and I don't have the luxury of
time to do this lengthy testing again.
Here's what I dumped as far as config goes before wiping:
========
# for var in size min_size pg_num pgp_num crush_ruleset
erasure_code_profile; do ceph osd pool get volumes $var; done
size: 5
min_size: 2
pg_num: 7200
pgp_num: 7200
crush_ruleset: 1
erasure_code_profile: ecvolumes
# for var in size min_size pg_num pgp_num crush_ruleset hit_set_type
hit_set_period hit_set_count target_max_objects target_max_bytes
cache_target_dirty_ratio cache_target_full_ratio cache_min_flush_age
cache_min_evict_age; do ceph osd pool get volumecache $var; done
size: 2
min_size: 1
pg_num: 7200
pgp_num: 7200
crush_ruleset: 4
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects: 0
target_max_bytes: 100000000000
cache_target_dirty_ratio: 0.5
cache_target_full_ratio: 0.8
cache_min_flush_age: 600
cache_min_evict_age: 1800
# ceph osd erasure-code-profile get ecvolumes
directory=/usr/lib/ceph/erasure-code
k=3
m=2
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
========
And now:
========
# for var in size min_size pg_num pgp_num crush_ruleset
erasure_code_profile; do ceph osd pool get volumes $var; done
size: 5
min_size: 3
pg_num: 2048
pgp_num: 2048
crush_ruleset: 1
erasure_code_profile: ecvolumes
# for var in size min_size pg_num pgp_num crush_ruleset hit_set_type
hit_set_period hit_set_count target_max_objects target_max_bytes
cache_target_dirty_ratio cache_target_full_ratio cache_min_flush_age
cache_min_evict_age; do ceph osd pool get volumecache $var; done
size: 2
min_size: 1
pg_num: 2048
pgp_num: 2048
crush_ruleset: 4
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects: 0
target_max_bytes: 150000000000
cache_target_dirty_ratio: 0.5
cache_target_full_ratio: 0.8
cache_min_flush_age: 0
cache_min_evict_age: 1800
# ceph osd erasure-code-profile get ecvolumes
directory=/usr/lib/ceph/erasure-code
k=3
m=2
plugin=jerasure
ruleset-failure-domain=osd
technique=reed_sol_van
========
Crush map hasn't really changed before and after.
FWIW, the benchmarks I pulled out of the setup:
https://gist.github.com/dmsimard/2737832d077cfc5eff34
Definite overhead going from krbd to krbd + LIO...
--
David Moreau Simard
On Nov 20, 2014, at 4:14 PM, Nick Fisk <n...@fisk.me.uk> wrote:
Here you go:-
Erasure Profile
k=2
m=1
plugin=jerasure
ruleset-failure-domain=osd
ruleset-root=hdd
technique=reed_sol_van
Cache Settings
hit_set_type: bloom
hit_set_period: 3600
hit_set_count: 1
target_max_objects
target_max_objects: 0
target_max_bytes: 1000000000
cache_target_dirty_ratio: 0.4
cache_target_full_ratio: 0.8
cache_min_flush_age: 0
cache_min_evict_age: 0
Crush Dump
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3 osd.3
# types
type 0 osd
type 1 host
type 2 chassis
type 3 rack
type 4 row
type 5 pdu
type 6 pod
type 7 room
type 8 datacenter
type 9 region
type 10 root
# buckets
host ceph-test-hdd {
id -5 # do not change unnecessarily
# weight 2.730
alg straw
hash 0 # rjenkins1
item osd.1 weight 0.910
item osd.2 weight 0.910
item osd.0 weight 0.910
}
root hdd {
id -3 # do not change unnecessarily
# weight 2.730
alg straw
hash 0 # rjenkins1
item ceph-test-hdd weight 2.730 } host ceph-test-ssd {
id -6 # do not change unnecessarily
# weight 1.000
alg straw
hash 0 # rjenkins1
item osd.3 weight 1.000
}
root ssd {
id -4 # do not change unnecessarily
# weight 1.000
alg straw
hash 0 # rjenkins1
item ceph-test-ssd weight 1.000 }
# rules
rule hdd {
ruleset 0
type replicated
min_size 0
max_size 10
step take hdd
step chooseleaf firstn 0 type osd
step emit
}
rule ssd {
ruleset 1
type replicated
min_size 0
max_size 4
step take ssd
step chooseleaf firstn 0 type osd
step emit
}
rule ecpool {
ruleset 2
type erasure
min_size 3
max_size 20
step set_chooseleaf_tries 5
step take hdd
step chooseleaf indep 0 type osd
step emit
}
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
Of David Moreau Simard
Sent: 20 November 2014 20:03
To: Nick Fisk
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Poor RBD performance as LIO iSCSI target
Nick,
Can you share more datails on the configuration you are using ? I'll
try and duplicate those configurations in my environment and see what
happens.
I'm mostly interested in:
- Erasure code profile (k, m, plugin, ruleset-failure-domain)
- Cache tiering pool configuration (ex: hit_set_type, hit_set_period,
hit_set_count, target_max_objects, target_max_bytes,
cache_target_dirty_ratio, cache_target_full_ratio,
cache_min_flush_age,
cache_min_evict_age)
The crush rulesets would also be helpful.
Thanks,
--
David Moreau Simard
On Nov 20, 2014, at 12:43 PM, Nick Fisk <n...@fisk.me.uk> wrote:
Hi David,
I've just finished running the 75GB fio test you posted a few days
back on my new test cluster.
The cluster is as follows:-
Single server with 3x hdd and 1 ssd
Ubuntu 14.04 with 3.16.7 kernel
2+1 EC pool on hdds below a 10G ssd cache pool. SSD is also
2+partitioned to
provide journals for hdds.
150G RBD mapped locally
The fio test seemed to run without any problems. I want to run a few
more tests with different settings to see if I can reproduce your
problem. I will let you know if I find anything.
If there is anything you would like me to try, please let me know.
Nick
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf
Of David Moreau Simard
Sent: 19 November 2014 10:48
To: Ramakrishna Nishtala (rnishtal)
Cc: ceph-users@lists.ceph.com; Nick Fisk
Subject: Re: [ceph-users] Poor RBD performance as LIO iSCSI target
Rama,
Thanks for your reply.
My end goal is to use iSCSI (with LIO/targetcli) to export rbd block
devices.
I was encountering issues with iSCSI which are explained in my
previous emails.
I ended up being able to reproduce the problem at will on various
Kernel and OS combinations, even on raw RBD devices - thus ruling out
the hypothesis that it was a problem with iSCSI but rather with Ceph.
I'm even running 0.88 now and the issue is still there.
I haven't isolated the issue just yet.
My next tests involve disabling the cache tiering.
I do have client krbd cache as well, i'll try to disable it too if
cache tiering isn't enough.
--
David Moreau Simard
On Nov 18, 2014, at 8:10 PM, Ramakrishna Nishtala (rnishtal)
<rnish...@cisco.com> wrote:
Hi Dave
Did you say iscsi only? The tracker issue does not say though.
I am on giant, with both client and ceph on RHEL 7 and seems to work
ok,
unless I am missing something here. RBD on baremetal with kmod-rbd
and caching disabled.
[root@compute4 ~]# time fio --name=writefile --size=100G
--filesize=100G --filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1
--sync=0 --randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio,
iodepth=200
fio-2.1.11
Starting 1 process
Jobs: 1 (f=1): [W(1)] [100.0% done] [0KB/853.0MB/0KB /s] [0/853/0
iops] [eta 00m:00s] ...
Disk stats (read/write):
rbd0: ios=184/204800, merge=0/0, ticks=70/16164931,
in_queue=16164942, util=99.98%
real 1m56.175s
user 0m18.115s
sys 0m10.430s
Regards,
Rama
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On
Behalf Of David Moreau Simard
Sent: Tuesday, November 18, 2014 3:49 PM
To: Nick Fisk
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Poor RBD performance as LIO iSCSI target
Testing without the cache tiering is the next test I want to do when
I
have time..
When it's hanging, there is no activity at all on the cluster.
Nothing in "ceph -w", nothing in "ceph osd pool stats".
I'll provide an update when I have a chance to test without tiering.
--
David Moreau Simard
On Nov 18, 2014, at 3:28 PM, Nick Fisk <n...@fisk.me.uk> wrote:
Hi David,
Have you tried on a normal replicated pool with no cache? I've seen
a number of threads recently where caching is causing various
things to
block/hang.
It would be interesting to see if this still happens without the
caching layer, at least it would rule it out.
Also is there any sign that as the test passes ~50GB that the cache
might start flushing to the backing pool causing slow performance?
I am planning a deployment very similar to yours so I am following
this with great interest. I'm hoping to build a single node test
"cluster" shortly, so I might be in a position to work with you on
this issue and hopefully get it resolved.
Nick
-----Original Message-----
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On
Behalf Of David Moreau Simard
Sent: 18 November 2014 19:58
To: Mike Christie
Cc: ceph-users@lists.ceph.com; Christopher Spearman
Subject: Re: [ceph-users] Poor RBD performance as LIO iSCSI target
Thanks guys. I looked at http://tracker.ceph.com/issues/8818 and
chatted with "dis" on #ceph-devel.
I ran a LOT of tests on a LOT of comabination of kernels (sometimes
with tunables legacy). I haven't found a magical combination in
which the following test does not hang:
fio --name=writefile --size=100G --filesize=100G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
Either directly on a mapped rbd device, on a mounted filesystem
(over rbd), exported through iSCSI.. nothing.
I guess that rules out a potential issue with iSCSI overhead.
Now, something I noticed out of pure luck is that I am unable to
reproduce the issue if I drop the size of the test to 50GB. Tests
will complete in under 2 minutes.
75GB will hang right at the end and take more than 10 minutes.
TL;DR of tests:
- 3x fio --name=writefile --size=50G --filesize=50G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
-- 1m44s, 1m49s, 1m40s
- 3x fio --name=writefile --size=75G --filesize=75G
--filename=/dev/rbd0 --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
-- 10m12s, 10m11s, 10m13s
Details of tests here: http://pastebin.com/raw.php?i=3v9wMtYP
Does that ring you guys a bell ?
--
David Moreau Simard
On Nov 13, 2014, at 3:31 PM, Mike Christie <mchri...@redhat.com>
wrote:
On 11/13/2014 10:17 AM, David Moreau Simard wrote:
Running into weird issues here as well in a test environment. I
don't
have a solution either but perhaps we can find some things in common..
Setup in a nutshell:
- Ceph cluster: Ubuntu 14.04, Kernel 3.16.7, Ceph 0.87-1 (OSDs
with separate public/cluster network in 10 Gbps)
- iSCSI Proxy node (targetcli/LIO): Ubuntu 14.04, Kernel 3.16.7,
Ceph
0.87-1 (10 Gbps)
- Client node: Ubuntu 12.04, Kernel 3.11 (10 Gbps)
Relevant cluster config: Writeback cache tiering with NVME PCI-E
cards (2
replica) in front of a erasure coded pool (k=3,m=2) backed by spindles.
I'm following the instructions here:
http://www.hastexo.com/resources/hints-and-kinks/turning-ceph-rbd
- im a ges-san-storage-devices No issues with creating and
mapping a 100GB RBD image and then creating the target.
I'm interested in finding out the overhead/performance impact of
re-exporting through iSCSI so the idea is to run benchmarks.
Here's a fio test I'm trying to run on the client node on the
mounted
iscsi device:
fio --name=writefile --size=100G --filesize=100G
--filename=/dev/sdu --bs=1M --nrfiles=1 --direct=1 --sync=0
--randrepeat=0 --rw=write --refill_buffers --end_fsync=1
--iodepth=200 --ioengine=libaio
The benchmark will eventually hang towards the end of the test
for some
long seconds before completing.
On the proxy node, the kernel complains with iscsi portal login
timeout: http://pastebin.com/Q49UnTPr and I also see irqbalance
errors in syslog: http://pastebin.com/AiRTWDwR
You are hitting a different issue. German Anders is most likely
correct and you hit the rbd hang. That then caused the iscsi/scsi
command to timeout which caused the scsi error handler to run. In
your logs we see the LIO error handler has received a task abort
from the initiator and that timed out which caused the escalation
(iscsi portal login related messages).
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
