Matt and I have had problems with BEE2 - switch with short cables. The
lanes lose sync, and then take a fixed time to resync. Data is lost
during this process, hence the losses are higher at higher datarates.
With lower rates, it is able to buffer the data and resume without
significant losses.
FWIW, with default PHY settings (max preemph and swing), my BEE2 cable
length preferences are as follows:
BEE2 - IBOB: 1m
BEE2 - switch: 3m
BEE2 - Myricom NIC: anything from 1m to 3m.
I believe we can improve the range of cable lengths that work, but
you'll have to tweak the PHY's preemph and swing settings.
Unfortunately, on the V2Ps, this must be done at compile time. ROACH
lets you do this at runtime, which aids tuning.
I would be surprised to find that the switch is causing the problem. I
am currently pumping 8Gbps on each of 8 ports on my XG2000C without a
single dropped packet.
Jason
On 25 Nov 2009, at 02:00, Paul Demorest wrote:
Hey guys,
Jumbo frames are already enabled on the switch. The packet loss
we're seeing is about 0.1%, so most of the data is making it
through. The weird thing is that this number seems independent of
the data rate. Even at <1 MB/s we still lose 0.1% of the data.
As John already said, connecting the bee2 directly to the receiving
computer results in zero packet loss at low data rates. We do lose
some if the data rate is set much higher (>~400 MB/s depending on
the data processing, system load, etc). But that kind of thing is
expected..
-Paul
On Tue, 24 Nov 2009, Peter McMahon wrote:
Page 3 of Jason's switch memo at
http://casper.berkeley.edu/memos/switch_configuration.pdf has the
commands
to enable jumbo frame on some of the Fujitsu switches.
Peter
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Dan Werthimer
Sent: Tuesday, November 24, 2009 3:34 PM
To: John Ford
Cc: [email protected]
Subject: Re: [casper] 10 GBe Switch weirdness
hi john,
i vaguely remember that there's a command you need to issue
to the fujitsu switches to enable jumbo packets ??
jason will know.
dan
On 11/24/2009 02:01 PM, John Ford wrote:
Hi all. We've been running GUPPI for some time now with a direct
connection from our bee2, "bee2" to our host, "beef". Works fine,
no
dropped packets, etc. Life's fine.
To build our next machine, GUPPI-2, we decided to insert a Fujitsu
XG-2000C switch between them, and now we are losing packets. Same
hardware, same software, except the switch is now between the bee2
and the
host. We have very short cables on all the ports. 1 meter or 0.5
meter.
Changing cables had no effect. Changing switch ports had no
effect. The
ports are set to use jumbo packets. Moving the cable from the
switch back
to the host fixes the problems, and it all magically works again.
Other hosts on the 10 gbe network do not seem to suffer from
packet loss.
The switch's monitoring screens do not show any packet loss in the
switch.
Does the switch do anything to the timing of the packets? Any
ideas about
incompatible NIC's? They are all myricom pcie cards.
Any other ideas?
John
