On Thu, 23 Aug 2007, Glen Dosey wrote:
What really gets me is that while my NFS reads are around ~50MB/s ,
the writes are basically at wire speed, slowing down to and holding
at about ~90MB/s when we exceed the 4GB file size. That would seem
to indicate to me the server has no problem dealing with a saturated
NIC and reasonably high I/O on the QLA2342 at the same time.
I beg to disagree with your conclusion. At least for networking the Tx
and Rx paths are quite different in what operations they do and
especially how many interrupts they need (due to possible interrupt
mitigation, in either driver or hardware); I think that for the SCSI
stack used to talk to a FC controller the situation is quite similar.
So, I think that a test that would better aproximate the NFS read case
would be:
- run the local reading test on the server (using dd)
- run at the same time a network speed test (using f.e. ttcp) which
sends data from the server to a client
The slowing down that you mention above as happening at 4GB looks to
me like the server's memory used for caching fills up and the writing
to disk is then forced; so it's not the 4GB file-size that is changing
behaviour, but the caching that doesn't help anymore. If you want to
better test the behaviour, try running on a client bonnie++ with a
file size of 8GB or more (although the client's memory size could also
play a role in caching).
50: 69688 963247 1227309 270953 IO-APIC-level qla2xxx
58: 15112 96722 96347 7613 IO-APIC-level qla2xxx
66: 47398161 0 0 0 IO-APIC-level eth0
Seems like the interrupts for the NIC go to only one CPU, while the
ones for the FC controller are spread among CPUs. I think that memory
locality effects come into play here and reduce the speed. Did you
play by any chance with CPU affinity for interrupts ?
atop shows basically the same iostat does, which is that on the initial
read the FC disk is about %85 percent utilized
This seems strange: for 40MB/s the disk is used at 85%, but then you
report getting 160MB/s ? It doesn't add up for me - do you have an
explanation ?
--
Bogdan Costescu
IWR - Interdisziplinaeres Zentrum fuer Wissenschaftliches Rechnen
Universitaet Heidelberg, INF 368, D-69120 Heidelberg, GERMANY
Telephone: +49 6221 54 8869, Telefax: +49 6221 54 8868
E-mail: [EMAIL PROTECTED]
_______________________________________________
Beowulf mailing list, [email protected]
To change your subscription (digest mode or unsubscribe) visit
http://www.beowulf.org/mailman/listinfo/beowulf
