Tuning GigE network for cluster computing?

2006-06-03 Thread Steve Kargl
First, I'm running FreeBSD 6.1-stable on a small cluster
with 6 nodes that contain Tyan motheriboards.  These broads
have Broadcom GigE NICs that use the bge device.

The cluster will be using MPI to possibly shove large data
sets through a GigE switch, so I'm trying to determine how
best to optimize the transfer.

Would DEVICE_POLLING and/or jumbo frames likely methods for
enhanced speeds?

-- 
Steve
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Tuning GigE network for cluster computing?

2006-06-03 Thread Danial Thom


--- Steve Kargl
[EMAIL PROTECTED] wrote:

 First, I'm running FreeBSD 6.1-stable on a
 small cluster
 with 6 nodes that contain Tyan motheriboards. 
 These broads
 have Broadcom GigE NICs that use the bge
 device.
 
 The cluster will be using MPI to possibly shove
 large data
 sets through a GigE switch, so I'm trying to
 determine how
 best to optimize the transfer.
 
 Would DEVICE_POLLING and/or jumbo frames likely
 methods for
 enhanced speeds?
 
 -- 
 Steve


--- Steve Kargl
[EMAIL PROTECTED] wrote:

 First, I'm running FreeBSD 6.1-stable on a
 small cluster
 with 6 nodes that contain Tyan motheriboards. 
 These broads
 have Broadcom GigE NICs that use the bge
 device.
 
 The cluster will be using MPI to possibly shove
 large data
 sets through a GigE switch, so I'm trying to
 determine how
 best to optimize the transfer.
 
 Would DEVICE_POLLING and/or jumbo frames likely
 methods for
 enhanced speeds?


Firstly we've been discussing how bad and
inefficient the broadcom controllers are. If you
have a tyan MB with onboard controllers they are
incredibly slow (ie inefficient), as well as
being quirky. If you have a PCI-X slot put in an
intel card. They have built-in interrupt
moderation so you don't have dick around with
polling. A 133Mhz intel controller (make sure
they have the GB chips on them and not the EB)
will use half the cpu of the on-board broadcoms.
If you don't believe try it with one box and
measure the cpu usage before and after. Its worth
the $50. investment in the card, believe me.

While polling may marginally decrease the cpu
load (depending on how bad FreeBSD 6.1 is on
interrupt overhead), its also going to add
latency to the processing of packets, which is
the opposite of what you want to do. 

DT

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]


Re: Tuning GigE network for cluster computing?

2006-06-03 Thread Steve Kargl
On Sat, Jun 03, 2006 at 09:28:42AM -0700, Danial Thom wrote:
 
 [EMAIL PROTECTED] wrote:
 
  First, I'm running FreeBSD 6.1-stable on a
  small cluster
  with 6 nodes that contain Tyan motheriboards. 
  These broads
  have Broadcom GigE NICs that use the bge
  device.
  
  The cluster will be using MPI to possibly shove
  large data
  sets through a GigE switch, so I'm trying to
  determine how
  best to optimize the transfer.
  
  Would DEVICE_POLLING and/or jumbo frames likely
  methods for
  enhanced speeds?
 
 
 Firstly we've been discussing how bad and
 inefficient the broadcom controllers are. If you
 have a tyan MB with onboard controllers they are
 incredibly slow (ie inefficient), as well as
 being quirky. If you have a PCI-X slot put in an
 intel card. They have built-in interrupt
 moderation so you don't have dick around with
 polling. A 133Mhz intel controller (make sure
 they have the GB chips on them and not the EB)
 will use half the cpu of the on-board broadcoms.
 If you don't believe try it with one box and
 measure the cpu usage before and after. Its worth
 the $50. investment in the card, believe me.

I'll look into the Intel cards, but unfortunately
I'm probably stuck with the onboard broadcom devices
for the immediate future.  There is one expansion
slot (whether its PCI-X, I don't know).  I was 
actually planning to use the slot for infiniband,
myrinet, or the 10 GiGE cards that Drew Gallatin
has mentioned.

 While polling may marginally decrease the cpu
 load (depending on how bad FreeBSD 6.1 is on
 interrupt overhead), its also going to add
 latency to the processing of packets, which is
 the opposite of what you want to do. 

This is the conclusion that I reached in reading
i386/conf/NOTES.  Thanks for confirming my suspicions.

-- 
Steve
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]