On 2/15/2012 3:13 AM, Dan Serban wrote: > I don't know why I didn't think of that, maybe I spent too much time on > Wikipedia and assumed I'd see 33MHz for all PCI devices.
>> No, "capabilities" tells you what the device can do *and* what's it's >> currently doing. Note the 133MHz is under the sub heading "status". > > Such an elaborate post for this one simple response. How foolish do I feel? Heheh. Note the right hand side of my email address. ;) I can be a bit, engaged, when talking shop. ;) > Yes, I'm familiar with the configuration, though admittedly not familiar > enough with lspci =) You should have never run it. ;) > I was thinking the same thing, but I wanted to test the results with > bonnie++ and simple dd tests to see if I would be gaining much of anything > by putting one card into one bus and the other on the second. Bonnie and dd aren't going to tell you much. They're going to give you bonnie and dd results. Which bear little resemblance to most real world workloads. > While I'm not unfamiliar with the theories, I have been bit by IO problems > in three cases and have the opportunity to test and see for myself on this > install where I'm failing to see the bottlenecks. > > In one case the pci bus was the limiting factor and my file server machine > which was where concurrent 2-7mbit video recordings were being written > would lag out severely due to IO wait. This iowait was caused by the disks running out of seek headroom. The PCI bus was not becoming saturated and thus was not your bottleneck. A 32/33 PCI bus can carry 138 of your 7mbit/s streams assuming -5-10% for bus wide PCI protocol overhead, and assuming there was little/no bus contention with devices other than the capture hardware. That's 69 streams coming through the capture hardware or NIC and 69 streams written to the disk array. The bandwidth required of the disks is only 60 MB/s, but with that many streaming writes the heads can't seek to each track quickly enough while writing each file. You're looking at minimum 69ms of iowait between the first writer thread and the last, assuming 1ms between seeks, and it's probably more on the order of 8ms per seek given the track-track seek time on a 7.2k SATA drive is 5ms. So you're looking at 69 writers * 8ms = 552ms of iowait between the first and last thread. You could have a 4 GB/s PCIe x8 RAID card w/512MB BBWC and 16 7.2K SATA drives attached, and you'd still likely hit this wall with 69 such concurrent 7mb/s streams. Using 15k SAS drives would cut that iowait in half. But 275ms is probably still going to be too high. If you were writing significantly less than 69 streams then you have some other hardware or software problem. > The other is still an issue for me, but I feel NFS is really the culprit in > that case though .. again testing and experience is how I end up feeling > comfortable enough to tell people to go get stuffed when a suggestion is > made that I know to be wrong. Call me a perfectionist or a__l retentive but > that's how I roll when bitten. Just make sure you're testing your actual workloads. Synthetic tests are just that, synthetic. No one has ever been bitten by testing with their actual workload. Countless folks have been bitten by thinking bonnie, iozone, etc results are a substitution for their actual workload. We see this somewhat frequently on the XFS and Linux-RAID lists. > Yes though 66MHz vaguely sounds like half of what I had though didn't > it? :) 64/66 PCI-X yields 528 MB/s of bandwidth. I've seen countless IBM FasTt 600 fiber channel storage arrays deployed with 28 15k FC drives (2 shelves), with a single 4Gb/s FC host connection being used, serving from 4-12 ESX farm nodes and 400-800 users. That's 'only' 400 MB/s full duplex, 800 MB/s total, serving the storage bandwidth needs of an entire organization. The point is, even a lowly 66MHz, 528 MB/s PCI-X bus has a tremendous amount of capability, in the real world. General file serving is a real world workload that doesn't 'need' 800 MB/s of bandwidth, which is what you'll get using two SAT2-MV8 cards in one bus. It doesn't even 'need' 528 MB/s. The problem here is that you're a hobbyist (nothing wrong with that), not an SA, so you're not going to digest or agree with what I'm telling you WRT storage b/w. If you were an SA, you wouldn't be monkeying with upgrading and optimizing 10 year old hardware with PCI-X buses and uber cheap non-RAID SATA HBAs from the same period. So drop the 2 SAT2-MV8 HBAs in slot1 and slot 3 and close J53 so both cards run at 100 MHz (asymmetry is BAD after all). Now you have your 1.6 GB/s of PCI-X b/w which should closely match the average streaming read performance of those 16 drives. Now, you'll never get close to achieving 1.6 GB/s throughput with these drives, but it'll sure be fun to burn hundreds of hours trying. ;) > While a good solution and idea, I refuse to admit that I simply overlooked > that option. =) Or you could just close J53. ;) /me ducks > Correct, I haven't had to use lspci to the extent I needed to here, I But did you really "need" to use it? ;) > simply found the output to be ambiguous. Thanks for straightening it out > for me. You're welcome Dan. Apologies for any jabs/sarcasm that may be offensive. Sometimes it's needed to slap folks into a proper perspective. ;) -- Stan -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4f3bb1ca.9000...@hardwarefreak.com