On Tue, Nov 8, 2011 at 9:44 PM, MORITA Kazutaka
<[email protected]> wrote:
> At Tue, 8 Nov 2011 10:56:51 -0500,
> Shawn Moore wrote:
>>
>> > Probably, we need to add support for using different NICs for data
>> > I/Os and monitoring.
>>
>> We currently have 4 1Gb nics bonded together using mode 4
>> (LACP/802.3ad). On another note, I am still doing more testing, but
>> it almost looks like the TOTAL cluster speed might be limited to 1Gb
>> instead of up to 4Gb (I know I can't get that much in truth though,
>> but should be higher than 1Gb). Does anyone have any insight into
>> this? We are using enterprise grade switches with two cards (18Gb/s
>> fabric).
>
> How did you measure the total cluster speed? Could your disks be a
> bottleneck?
What I have been doing is using pssh (parallel ssh) to kick off a
script on all nodes at the same time to create a pre-allocated vdi of
the same size. Then obtain the time it takes for the operation to
complete on each node and then do some math. Below is the script and
an example of the data obtained. I did go ahead a re-run my tests
with tmpfs instead of true ext4 disks. This did yield quite a
performance boost, but seems it actually should have been faster given
the usage of ram for the disks.
T_START="$(date +%s)"
collie vdi create test_$(uname -n) ${1}G -P
T_STOP="$(date +%s)"
expr ${T_STOP} - ${T_START}
This is a run using the disks. This created 9 3GB vdi's. With copies
3, that created 27GB worth of vdi data and 81GB worth of total cluster
data. Total cluster speed of 1.46Gb/s.
152 416 sec
153 462 sec
154 394 sec
155 435 sec
156 451 sec
157 483 sec
159 459 sec
160 406 sec
161 486 sec
VDI SIZE 3
COPIES 3
TOT sec 3992
AVG sec 443.5555556
27 GB
27648 MB
62.33266533 MB/s
81 GB
82944 MB
186.997996 MB/s
1495.983968 Mb/s
1.460921844 Gb/s
This is a run using tmpfs. This created 9 3GB vdi's. With copies 3,
that created 27GB worth of vdi data and 81GB worth of total cluster
data. Total cluster speed of 5.73Gb/s.
152 80 sec
153 119 sec
154 127 sec
155 118 sec
156 116 sec
157 142 sec
159 80 sec
160 125 sec
161 110 sec
VDI SIZE 3
COPIES 3
TOT sec 1017
AVG sec 113
27 GB
27648 MB
244.6725664 MB/s
81 GB
82944 MB
734.0176991 MB/s
5872.141593 Mb/s
5.734513274 Gb/s
--
sheepdog mailing list
[email protected]
http://lists.wpkg.org/mailman/listinfo/sheepdog
