20.10.2015 21:52, Joeri Casteels wrote:
On 20 Oct 2015, at 20:47, Lars Ellenberg <[email protected]> wrote:
On Tue, Oct 20, 2015 at 01:07:48PM +0200, Joeri Casteels wrote:
Dear All,
Doing a new set-up of DRBD on Ubuntu LTS 14.04 with stock DRBD
When Setting to protocol A i get double the bandwidth as protocol C anybody an
idea? not changing anything else of the settings except the protocol
PROTOCOL C:
d if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 46.3583 s, 452 MB/s
PROTOCOL A:
dd if=/dev/zero of=/dev/drbd0 bs=1M count=20000 oflag=direct
20000+0 records in
20000+0 records out
20971520000 bytes (21 GB) copied, 23.8356 s, 880 MB/s
dd does:
write one block size # which takes some time
write next bs # which again takes some time
...
Without oflag=direct,
you write into the linux page cache, the write itself takes microseconds
(at most), and you get many such writes per second, resulting usually in
a throughput of several gigabytes per second (until you hit the
boundaries of the page cache and are throttled down to the rate of the
background write-out).
With oflag=direct,
it bypasses the linux page cache,
the single write call takes longer,
and you do no longer get that many writes per second.
With protocol B and C, the single write latency even includes
the network latency and remote disk latency, possibly flush latencies,
and you get even less writes per second.
Throughput is concurrency times request size by request completion latency.
dd has no concurrency, in the given example, request size is 1M,
if you get 450 MB/s, your latency apparently is in the order of 2.3 ms.
If you want more throughput,
you need to decrease latency, or increase concurrency.
Any suggestions on how to improve that then?
There was advise on some of related lists (may be even this) to tune
following parameters:
net/max-buffers=80k
net/max-epoch-size=20000
I additionally use
disk/c-max-rate=1500m
disk/c-fill-target=0
and get >800MB/s writes with protocol C over 10Gbps Ethernet (ConnectX-3
in ethernet or IPoIB mode) with IO depth >=32 and IO size 512k (I use
disktest from LTP project). That is what 6 Intel DC S3610 can provide in
HW RAID-10 with disk cache turned on.
And, of course, Lars is absolutely correct that it is merely impossible
to get high IO speeds with just one IO thread.
Best, Vladislav
_______________________________________________
drbd-user mailing list
[email protected]
http://lists.linbit.com/mailman/listinfo/drbd-user
