On Tue, Sep 07, 2010, Jeremy Chadwick wrote: > > I figured there might memory exhaustion of sorts, possibly in the bce(4) > driver itself, that could cause the OP's problem. bce(4) might not be > the problem at all. But the OP's issue seems to only occur when > transmitting data, not receiving: > > http://lists.freebsd.org/pipermail/freebsd-stable/2010-September/058708.html
More information:
Looks like 100M wasn't enough of a test burst to tickle the problem in
my original message... 10G is, though. It's definitely happening in
both directions.
Upgraded to -STABLE on one of the two machines last night, running
GENERIC.
FreeBSD obb 8.1-STABLE FreeBSD 8.1-STABLE #0: Tue Sep 7 19:48:55 PDT 2010
r...@obb:/usr/obj/usr/src/sys/GENERIC amd64
Outgoing:
obb# scp testfile r...@holp:/usr/local/tmp/
testfile 8% 856MB 37.6MB/s 04:09 ETA
Write failed: Cannot allocate memory
lost connection
obb# scp testfile r...@holp:/usr/local/tmp/
testfile 0% 72MB 34.3MB/s 04:56 ETA
Write failed: Cannot allocate memory
lost connection
Incoming:
obb# scp r...@holp:/usr/local/tmp/testfile .
testfile 6% 670MB 31.9MB/s 04:59 ETA
Write failed: Cannot allocate memory
lost connection
obb# scp r...@holp:/usr/local/tmp/testfile .
testfile 1% 118MB 39.3MB/s 04:17 ETA
Write failed: Cannot allocate memory
lost connection
obb# scp r...@holp:/usr/local/tmp/testfile .
testfile 15% 1613MB 29.0MB/s 04:57 ETA
Write failed: Cannot allocate memory
lost connection
> The 2nd-to-last paragraph there is worth noting, specifically how
> limiting maximum addressable memory to 32GB via loader.conf seems to
> work around the issue.
I'd no longer consider this a coincidence, limiting the memory to 16G
eliminates the issue completely. I'll retest with 32G today.
Incoming:
obb# scp r...@holp:/usr/local/tmp/testfile testfile2
testfile 100% 10GB 17.8MB/s 09:35
obb# scp r...@holp:/usr/local/tmp/testfile testfile2
testfile 100% 10GB 17.0MB/s 10:02
Outgoing:
obb# scp testfile r...@holp:/usr/local/tmp/testfile2
testfile 100% 10GB 35.7MB/s 04:47
obb# scp testfile r...@holp:/usr/local/tmp/testfile2
testfile 100% 10GB 35.4MB/s 04:49
> There were other problems with the systems in question back in July, it
> seems. I assume these got hammered out somehow:
>
> http://www.mail-archive.com/[email protected]/msg111408.html
To a degree -- the initial install and cpu count problems are all fixed
up, thanks to help from the list. The Intel 10G panics were stifled
with a newer driver from Intel's site, but I ran out of time to do
any serious testing with it, and just ended up using the broadcoms to
satisfy my time constraint.
--
Mahlon E. Smith
http://www.martini.nu/contact.html
pgpKPFuQxrRC6.pgp
Description: PGP signature
