Matt,
Since you invoked my name here, I'll summarize my experiences.
The 3m copper links from the iBobs to the BEE2 are completely
unacceptable with even Gore cables. My goal is to entirely eliminate 3m
copper from the ATA beamformer. Your number of 1 failure per hour is on
the good side. I pulled up some notes from my tests at Hat Creek, and
found that in a single beamformer "as-wired" for 24 inputs: Over an
8-hour period, they roughly break down to:
19 With no errors detected over 8 hours
2 With sporadic (roughly once per 90 min) burst linkdowns
2 With ~ 4-8 burst linkdowns per half-hour, every half-hour
1 With more, usually nearly constant burst linkdowns
One of my BEE2-BEE2 links (also over 3m) showed ~ 4-8 burst linkdowns
per half-hour.
This depends very much on the temperature of the devices and also on
which devices are used. For example, the above data was from 4
different BEE2s, and 4 of the 5 links with errors were recorded on
corner chip FPGA#2 of the BEE2 (and the one that was on a different FPGA
was a sporadic error). So something about the receiving hardware
definitely matters here. I also noticed an apparent preference that the
iBobs were more susceptible to creating errors on their bottom/right
XAUI port (X1?) and the top/left xaui port (X0?) tended to produce fewer
errors. Also, for what it's worth, hot-plugging XAUI cables was found
to generally be a BAD idea in an error-prone environment: I found that
simply removing/ reinserting a cable could permanently increase the
error rates by at least an order of magnitude until the power was
hard-reset (the reset button on the iBob was not satisfactory).
Now as to my iBobs: The number above (5 bad, 19 good) is somewhat
incomplete - I did more tests using ~ 1min observation times of the link
quality using a FPGA#2 corner chip as the receiver (the most sensitive
to errors). I found that in that case, only 7 iBobs did not produce
measurable link errors over 3m copper in that 1min time scale. (This was
UNCOOLED)
Since we've cooled the iBobs, these link errors have been mitigated
enough to observe but not entirely eliminated. I've made it a policy to
use XAUI0 for beamformer #1 and XAUI1 for Beamformer #2 (with the
intention of not using XAUI1 when we get our new rack of iBobs for
Beamformer #2). This is because, after cooling, 8 iBobs showed 3m link
errors on XAUI1, but only 2 on XAUI0 (to corner chip #2)
I'd still expect at least 1 link error per day on some links (no data,
just memories). However, replacing the worst offenders with fiber or
1.5m copper (1m was too short) definitely improved this: I have not seen
any errors on fiber/1.5m links during my tests, which spanned many days.
Hope it helps,
Billy
-----Original Message-----
From: Matt Dexter [mailto:[email protected]]
Sent: Wednesday, August 20, 2008 2:18 PM
To: Jonathan Weintroub
Cc: [email protected]; Barott, William Chauncey
Subject: Re: [casper] fast serial communications: orientation to F-O
options
Based on testing with iBob -> F Board I strongly recommend WL Gore over
Fujitsu cables. Fujitsu were completely unacceptable.
Correct, we do NOT trust 3m copper cables for iBob->BEE2.
I don't have a good qualitative number to provide for the failure rate.
As an obscenely crude guess,
1 failure per 1 hour observation with 12 3m cables in use and
~ 7 Gbps continuous user data rate over each 10 Gbps cable.
failure rate may be much higher than this.
Perhaps Billy, the poor fellow who had to find and fight this problem,
will have time to share more valuable data.
BTW, I believe the packetized correlator project uses 1m for
iBob->BEE2. I think there is some thought that 1.5m and 2m cables
might work and perhaps those weren't used only because just 1m and 3m
cables were in stock. Jason Manley and others are the relevant experts.
Correct, the F Board is an RAL board; see attached for a sketch.
Correct, BEE2 -> BEE2 1m and 1.5m copper cables
Again Billy has the direct, and painful, experience but we have
found significant decrease in CX4 error rates as the iBob's FPGAs
were cooled from 80 degC to ~ 45 degC. This is the package temperature
as measured by an IR gun because the iBob was not designed to take
advantage of the FPGA IC's DXP and DXN pins that would tell us the
actual die temperature. See attached sketch of our sampler chassis
showing how we pack the iBob plates into a chassis.
The chassis level fans which blowing cold across iBobs from below isn't
shown. Final attachment shows the cutout we make into the mounting plate
to allow the FPGA's heatsink fan to funtion.
On Wed, 20 Aug 2008, Jonathan Weintroub wrote:
Thanks Matt and John for your input. You have provided needed
guidance for on the performance of copper cables. I may well give the
Gore a try, I have no objection to paying less. I do wonder, though,
if our good experience with 3 m lengths might be related to our use of
Fujitsu (??). Matt, yes I had read the cables & parameters memo. It
did not really really cover fiber optics though (the modifications
needed on an iBOB or BEE2 to retrofit, and sources for cables). Your
response fills this gap---I am sure I'm not alone in wanting to hear
how the PARALIGHT and ZARLINK stuff works out.
In our case, given that we can stick with short cables and have not
yet hit problems with the copper even at 3 m, we'll steer clear of the
cost and complication of F/O for now.
Just to clarify:
At HCRO for ATA we are going to use
iBob -> BEE2 1m copper or 3 to 10m fiber optic
iBob -> F Board 1,2 and 3 m copper (more testing TBD on 3m)
BEE2 -> BEE2 1m and 1.5m
BEE2 -> iBob 3 to 10m fiber optic
It appears you don't trust even 3-m copper on iBOB->BEE2. How bad is
it in your experience?
What is "F Board?" A non-CASPER FFT design . . .?
The third case is 1 m and 1.5m copper, right?
Also, I am probably missing something, but what is the relevance of
the discussion of cooling, pitch and mounting plate modifications in
the context of fast serial?
All of our iBobs will have fans mounted to the FPGA's heatsink
5VDC fan 40x40x6mm fan, digikey GM0504PEV1-8GN
5.9CFM Sunon GM0504PEV1-8GN
and the aluminum mounting plate is hogged out so that they can
still be mounted on 8hp = 1.6" pitch. These fans are in addition
to the chassis level fans (cold air in at bottom front; warm aire
out at top back).
On Aug 19, 2008, at 5:16 PM, John Ford wrote:
(I have re-read a prior thread from early May on this topic
involving
John, Matt, Jouko, and Francois, however the sourcing information
was
from 2005, and the applicability of the information to our
situation
was not entirely clear from the conversation. Also perhaps more
has
been learned since May.)
Our experience here is limited to using the GORE cables for both 10
GbE
and XAUI. The iBOB and BEE2 seem to be particular about needing
short
cables. We use 1 meter cables on our iBOB to BEE links. longer
cables
seem to work with 10 GbE cards (myricom) and commercial switches
(fujitsu). We bought our latest cables along with our latest
myricom 10
GbE cards from bell electronics, a Myricom distributor.
Hope this helps!
John
