I don't think the NIC-switch cable length makes much difference. I've
used everything from 0.5m to 5m and they all worked fine. These ports
have some form of auto-equalisation which seems to work very well.
Jason
On 25 Nov 2009, at 15:59, John Ford wrote:
Matt and I have had problems with BEE2 - switch with short cables.
The
lanes lose sync, and then take a fixed time to resync. Data is lost
during this process, hence the losses are higher at higher datarates.
With lower rates, it is able to buffer the data and resume without
significant losses.
FWIW, with default PHY settings (max preemph and swing), my BEE2
cable
length preferences are as follows:
BEE2 - IBOB: 1m
BEE2 - switch: 3m
BEE2 - Myricom NIC: anything from 1m to 3m.
I have a variety of copper cables I can try with various lengths.
We'll
see how that goes! It never occurred to me that the cable could be
too
short! We also have a really short cable on the host to switch
ports, so
I'll try adding to that one as well.
On another front, at the SC09 show last week we tried to use a zarlink
fiber-optic cable, and found that the myricom card would not drive
it. It
worked fine between 2 ports on a ROACH. The network guy there said
he's
seen that before, where a NIC couldn't drive enough power to supply
a FO
transceiver over CX-4.
I believe we can improve the range of cable lengths that work, but
you'll have to tweak the PHY's preemph and swing settings.
Unfortunately, on the V2Ps, this must be done at compile time. ROACH
lets you do this at runtime, which aids tuning.
Sounds a lot nicer.
I would be surprised to find that the switch is causing the
problem. I
am currently pumping 8Gbps on each of 8 ports on my XG2000C without a
single dropped packet.
That's good to hear, as we are depending on that to be true...
John
Jason
On 25 Nov 2009, at 02:00, Paul Demorest wrote:
Hey guys,
Jumbo frames are already enabled on the switch. The packet loss
we're seeing is about 0.1%, so most of the data is making it
through. The weird thing is that this number seems independent of
the data rate. Even at <1 MB/s we still lose 0.1% of the data.
As John already said, connecting the bee2 directly to the receiving
computer results in zero packet loss at low data rates. We do lose
some if the data rate is set much higher (>~400 MB/s depending on
the data processing, system load, etc). But that kind of thing is
expected..
-Paul
On Tue, 24 Nov 2009, Peter McMahon wrote:
Page 3 of Jason's switch memo at
http://casper.berkeley.edu/memos/switch_configuration.pdf has the
commands
to enable jumbo frame on some of the Fujitsu switches.
Peter
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Dan
Werthimer
Sent: Tuesday, November 24, 2009 3:34 PM
To: John Ford
Cc: [email protected]
Subject: Re: [casper] 10 GBe Switch weirdness
hi john,
i vaguely remember that there's a command you need to issue
to the fujitsu switches to enable jumbo packets ??
jason will know.
dan
On 11/24/2009 02:01 PM, John Ford wrote:
Hi all. We've been running GUPPI for some time now with a direct
connection from our bee2, "bee2" to our host, "beef". Works fine,
no
dropped packets, etc. Life's fine.
To build our next machine, GUPPI-2, we decided to insert a Fujitsu
XG-2000C switch between them, and now we are losing packets. Same
hardware, same software, except the switch is now between the bee2
and the
host. We have very short cables on all the ports. 1 meter or 0.5
meter.
Changing cables had no effect. Changing switch ports had no
effect. The
ports are set to use jumbo packets. Moving the cable from the
switch back
to the host fixes the problems, and it all magically works again.
Other hosts on the 10 gbe network do not seem to suffer from
packet loss.
The switch's monitoring screens do not show any packet loss in the
switch.
Does the switch do anything to the timing of the packets? Any
ideas about
incompatible NIC's? They are all myricom pcie cards.
Any other ideas?
John
