Matt, Since you invoked my name here, I'll summarize my experiences. The 3m copper links from the iBobs to the BEE2 are completely unacceptable with even Gore cables. My goal is to entirely eliminate 3m copper from the ATA beamformer. Your number of 1 failure per hour is on the good side. I pulled up some notes from my tests at Hat Creek, and found that in a single beamformer "as-wired" for 24 inputs: Over an 8-hour period, they roughly break down to:
19 With no errors detected over 8 hours 2 With sporadic (roughly once per 90 min) burst linkdowns 2 With ~ 4-8 burst linkdowns per half-hour, every half-hour 1 With more, usually nearly constant burst linkdowns One of my BEE2-BEE2 links (also over 3m) showed ~ 4-8 burst linkdowns per half-hour. This depends very much on the temperature of the devices and also on which devices are used. For example, the above data was from 4 different BEE2s, and 4 of the 5 links with errors were recorded on corner chip FPGA#2 of the BEE2 (and the one that was on a different FPGA was a sporadic error). So something about the receiving hardware definitely matters here. I also noticed an apparent preference that the iBobs were more susceptible to creating errors on their bottom/right XAUI port (X1?) and the top/left xaui port (X0?) tended to produce fewer errors. Also, for what it's worth, hot-plugging XAUI cables was found to generally be a BAD idea in an error-prone environment: I found that simply removing/ reinserting a cable could permanently increase the error rates by at least an order of magnitude until the power was hard-reset (the reset button on the iBob was not satisfactory). Now as to my iBobs: The number above (5 bad, 19 good) is somewhat incomplete - I did more tests using ~ 1min observation times of the link quality using a FPGA#2 corner chip as the receiver (the most sensitive to errors). I found that in that case, only 7 iBobs did not produce measurable link errors over 3m copper in that 1min time scale. (This was UNCOOLED) Since we've cooled the iBobs, these link errors have been mitigated enough to observe but not entirely eliminated. I've made it a policy to use XAUI0 for beamformer #1 and XAUI1 for Beamformer #2 (with the intention of not using XAUI1 when we get our new rack of iBobs for Beamformer #2). This is because, after cooling, 8 iBobs showed 3m link errors on XAUI1, but only 2 on XAUI0 (to corner chip #2) I'd still expect at least 1 link error per day on some links (no data, just memories). However, replacing the worst offenders with fiber or 1.5m copper (1m was too short) definitely improved this: I have not seen any errors on fiber/1.5m links during my tests, which spanned many days. Hope it helps, Billy -----Original Message----- From: Matt Dexter [mailto:[email protected]] Sent: Wednesday, August 20, 2008 2:18 PM To: Jonathan Weintroub Cc: [email protected]; Barott, William Chauncey Subject: Re: [casper] fast serial communications: orientation to F-O options Based on testing with iBob -> F Board I strongly recommend WL Gore over Fujitsu cables. Fujitsu were completely unacceptable. Correct, we do NOT trust 3m copper cables for iBob->BEE2. I don't have a good qualitative number to provide for the failure rate. As an obscenely crude guess, 1 failure per 1 hour observation with 12 3m cables in use and ~ 7 Gbps continuous user data rate over each 10 Gbps cable. failure rate may be much higher than this. Perhaps Billy, the poor fellow who had to find and fight this problem, will have time to share more valuable data. BTW, I believe the packetized correlator project uses 1m for iBob->BEE2. I think there is some thought that 1.5m and 2m cables might work and perhaps those weren't used only because just 1m and 3m cables were in stock. Jason Manley and others are the relevant experts. Correct, the F Board is an RAL board; see attached for a sketch. Correct, BEE2 -> BEE2 1m and 1.5m copper cables Again Billy has the direct, and painful, experience but we have found significant decrease in CX4 error rates as the iBob's FPGAs were cooled from 80 degC to ~ 45 degC. This is the package temperature as measured by an IR gun because the iBob was not designed to take advantage of the FPGA IC's DXP and DXN pins that would tell us the actual die temperature. See attached sketch of our sampler chassis showing how we pack the iBob plates into a chassis. The chassis level fans which blowing cold across iBobs from below isn't shown. Final attachment shows the cutout we make into the mounting plate to allow the FPGA's heatsink fan to funtion. On Wed, 20 Aug 2008, Jonathan Weintroub wrote: > Thanks Matt and John for your input. You have provided needed > guidance for on the performance of copper cables. I may well give the > Gore a try, I have no objection to paying less. I do wonder, though, > if our good experience with 3 m lengths might be related to our use of > Fujitsu (??). Matt, yes I had read the cables & parameters memo. It > did not really really cover fiber optics though (the modifications > needed on an iBOB or BEE2 to retrofit, and sources for cables). Your > response fills this gap---I am sure I'm not alone in wanting to hear > how the PARALIGHT and ZARLINK stuff works out. > > In our case, given that we can stick with short cables and have not > yet hit problems with the copper even at 3 m, we'll steer clear of the > cost and complication of F/O for now. > > Just to clarify: > > > At HCRO for ATA we are going to use > > iBob -> BEE2 1m copper or 3 to 10m fiber optic > > iBob -> F Board 1,2 and 3 m copper (more testing TBD on 3m) > > BEE2 -> BEE2 1m and 1.5m > > BEE2 -> iBob 3 to 10m fiber optic > > > It appears you don't trust even 3-m copper on iBOB->BEE2. How bad is > it in your experience? > > What is "F Board?" A non-CASPER FFT design . . .? > > The third case is 1 m and 1.5m copper, right? > > Also, I am probably missing something, but what is the relevance of > the discussion of cooling, pitch and mounting plate modifications in > the context of fast serial? > > > All of our iBobs will have fans mounted to the FPGA's heatsink > > 5VDC fan 40x40x6mm fan, digikey GM0504PEV1-8GN > > 5.9CFM Sunon GM0504PEV1-8GN > > and the aluminum mounting plate is hogged out so that they can > > still be mounted on 8hp = 1.6" pitch. These fans are in addition > > to the chassis level fans (cold air in at bottom front; warm aire > > out at top back). > > > On Aug 19, 2008, at 5:16 PM, John Ford wrote: > > > > >> (I have re-read a prior thread from early May on this topic involving > >> John, Matt, Jouko, and Francois, however the sourcing information was > >> from 2005, and the applicability of the information to our situation > >> was not entirely clear from the conversation. Also perhaps more has > >> been learned since May.) > > > > Our experience here is limited to using the GORE cables for both 10 > > GbE > > and XAUI. The iBOB and BEE2 seem to be particular about needing short > > cables. We use 1 meter cables on our iBOB to BEE links. longer > > cables > > seem to work with 10 GbE cards (myricom) and commercial switches > > (fujitsu). We bought our latest cables along with our latest > > myricom 10 > > GbE cards from bell electronics, a Myricom distributor. > > > > Hope this helps! > > > > John > > > > > > >
