Hi, when I am running
rados bench -p benchmark 300 write --run-name bench --no-cleanup
I got weird stalling during writes, sometimes I got same write speed
for few minutes and after some time it starts stalling with 0 MB/s for
minutes
My configuration:
ceph 0.80.5
pool 0 'data' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 200 pgp_num 200 last_change 146 flags hashpspool crash_replay_interval
45 stripe_width 0
pool 1 'metadata' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 200 pgp_num 200 last_change 148 flags hashpspool stripe_width 0
pool 2 'rbd' replicated size 3 min_size 2 crush_ruleset 0 object_hash rjenkins
pg_num 200 pgp_num 200 last_change 144 flags hashpspool stripe_width 0
pool 3 'benchmark' replicated size 3 min_size 2 crush_ruleset 0 object_hash
rjenkins pg_num 400 pgp_num 400 last_change 177 flags hashpspool stripe_width 0
3xMON servers + 5xOSD servers (IBM HS22, 16 GB RAM), 2 disks each (SATA
7,2k with 300GB partition for data) so 10 OSDs.
2 network cards, 2 VLANs on them, 2 active-passive bonds on those,
primary network card set up in a way that cluster traffic uses eth1 and
public traffic uses eth0. I've checked via iperf that I can get 2x1Gbit
on both vlans at the same time.
Usually it have constant performance after restart and then at some
point it just stalls, at this time there is no network traffic or disk
IO going on on any cluster node
After reboot of whole cluster I basically get constant 50MB/s but at
some point some OSDs start either to stall on slow requests:
2014-08-01 19:20:03.522122 7fde7b4de700 0 log [WRN] : 6 slow requests, 1
included below; oldest blocked for > 125.412779 secs
2014-08-01 19:20:03.522138 7fde7b4de700 0 log [WRN] : slow request 120.315923
seconds old, received at 2014-08-01 19:18:03.206101: osd_op(client.16294.0:144
benchmark_data_blade202.non.3dart.com_31312_object143 [write 0~4194304]
3.b31468c1 ack+ondisk+write e303) v4 currently waiting for subops from 2,5
and other (those who stalled) return indecipherable messages in log:
2014-08-01 18:35:09.544098 7fb247773700 0 -- 10.100.226.22:6803/3729
submit_message osd_op_reply(4020
benchmark_data_blade103.non.3dart.com_14996_object4019 [write 0~4194304]
v303'44 uv44 ack = 0) v6 remote, 10.100.226.10:0/1014996, failed lossy con,
dropping message 0x5244100
2014-08-01 18:35:09.544129 7fb247773700 0 -- 10.100.226.22:6803/3729
submit_message osd_op_reply(4020
benchmark_data_blade103.non.3dart.com_14996_object4019 [write 0~4194304]
v303'44 uv44 ondisk = 0) v6 remote, 10.100.226.10:0/1014996, failed lossy con,
dropping message 0x5244100
2014-08-01 18:35:48.679806 7fb250ba0700 0 -- 10.100.226.22:6803/3729
submit_message osd_op_reply(402
benchmark_data_blade103.non.3dart.com_17105_object401 [write 0~4194304] v303'85
uv85 ondisk = 0) v6 remote, 10.100.226.10:0/1017105, failed lossy con, dropping
message 0x5242300
2014-08-01 18:40:40.772300 7fb240158700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.22:6800/3575 pipe(0x3fd0780 sd=168 :52837 s=2 pgs=10 cs=1 l=0
c=0x4658b00).fault with nothing to send, going to standby
2014-08-01 18:54:05.684060 7fb246f72700 0 -- 10.100.226.22:6803/3729
submit_message osd_op_reply(397
benchmark_data_blade202.non.3dart.com_20051_object396 [write 0~4194304] v303'53
uv53 ack = 0) v6 remote, 10.100.226.12:0/1020051, failed lossy con, dropping
message 0x7960a00
2014-08-01 18:54:05.684097 7fb246f72700 0 -- 10.100.226.22:6803/3729
submit_message osd_op_reply(397
benchmark_data_blade202.non.3dart.com_20051_object396 [write 0~4194304] v303'53
uv53 ondisk = 0) v6 remote, 10.100.226.12:0/1020051, failed lossy con, dropping
message 0x7960a00
2014-08-01 19:08:55.388352 7fb241161700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.25:6800/3590 pipe(0x3fd7300 sd=76 :47915 s=2 pgs=3 cs=1 l=0
c=0x465fe80).fault with nothing to send, going to standby
2014-08-01 19:09:01.932211 7fb240259700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.23:6800/3572 pipe(0x3fd2800 sd=103 :6802 s=2 pgs=5 cs=1 l=0
c=0x465a940).fault with nothing to send, going to standby
2014-08-01 19:09:02.021751 7fb23d32b700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.24:6800/3571 pipe(0x3fd4b00 sd=117 :32965 s=2 pgs=6 cs=1 l=0
c=0x4d294a0).fault with nothing to send, going to standby
2014-08-01 19:09:02.021763 7fb23f047700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.23:6802/3726 pipe(0x3fd2580 sd=177 :6802 s=2 pgs=6 cs=1 l=0
c=0x465eb40).fault with nothing to send, going to standby
2014-08-01 19:09:02.664441 7fb241060700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.25:6802/3744 pipe(0x3fd0a00 sd=101 :52387 s=2 pgs=3 cs=1 l=0
c=0x4d28580).fault with nothing to send, going to standby
2014-08-01 19:09:02.718734 7fb23e83f700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.21:6802/3733 pipe(0x3fd5000 sd=111 :44595 s=2 pgs=8 cs=1 l=0
c=0x4d298c0).fault with nothing to send, going to standby
2014-08-01 19:09:03.179360 7fb23c61e700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.24:6802/3725 pipe(0x3fd2300 sd=195 :50640 s=2 pgs=5 cs=1 l=0
c=0x465d800).fault with nothing to send, going to standby
2014-08-01 19:09:05.783864 7fb23db33700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.21:6800/3579 pipe(0x3fd4d80 sd=185 :51815 s=2 pgs=8 cs=1 l=0
c=0x4d29760).fault with nothing to send, going to standby
2014-08-01 19:17:53.543253 7fb23a6cb700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.21:6800/3579 pipe(0x3fd4880 sd=63 :6802 s=0 pgs=0 cs=0 l=0
c=0x4d29e40).accept connect_seq 2 vs existing 1 state standby
2014-08-01 19:17:53.702121 7fb23de36700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.21:6802/3733 pipe(0x7112080 sd=185 :6802 s=0 pgs=0 cs=0 l=0
c=0x7c82100).accept connect_seq 2 vs existing 1 state standby
2014-08-01 19:17:54.200160 7fb23e940700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.23:6802/3726 pipe(0x7111400 sd=68 :6802 s=0 pgs=0 cs=0 l=0
c=0x7c877a0).accept connect_seq 2 vs existing 1 state standby
2014-08-01 19:17:56.549909 7fb23ee45700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.24:6800/3571 pipe(0x3fd1b80 sd=73 :6802 s=0 pgs=0 cs=0 l=0
c=0x7c87900).accept connect_seq 2 vs existing 1 state standby
2014-08-01 19:17:56.677212 7fb23d42c700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.24:6802/3725 pipe(0x3fd4d80 sd=117 :6802 s=0 pgs=0 cs=0 l=0
c=0x7c82260).accept connect_seq 2 vs existing 1 state standby
2014-08-01 19:17:57.381566 7fb23ca22700 0 -- 10.100.245.22:6802/3729 >>
10.100.245.25:6800/3590 pipe(0x3fd7580 sd=19 :6802 s=0 pgs=0 cs=0 l=0
c=0x7c823c0).accept connect_seq 2 vs existing 1 state standby
I've stress-checked network connections and tried on different network
card/switch and its same lockup, the worst part is that even after
stopping bench it does not unlock, just hangs on
HEALTH_WARN 16 requests are blocked > 32 sec; 2 osds have slow requests
16 ops are blocked > 524.288 sec
6 ops are blocked > 524.288 sec on osd.0
10 ops are blocked > 524.288 sec on osd.2
2 osds have slow requests
--
Mariusz Gronczewski, Administrator
Efigence S. A.
ul. WoĊoska 9a, 02-583 Warszawa
T: [+48] 22 380 13 13
F: [+48] 22 380 13 14
E: mariusz.gronczew...@efigence.com
<mailto:mariusz.gronczew...@efigence.com>
signature.asc
Description: PGP signature
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
