No problem at all, always glad to help... Knowing how frustrating this can be first-hand, I'd like to do what I can so nobody else has to go through those link problems.
Billy -----Original Message----- From: Matt Dexter [mailto:[email protected]] Sent: Thursday, August 21, 2008 5:08 PM To: Barott, William Chauncey Cc: Jonathan Weintroub; [email protected] Subject: RE: [casper] fast serial communications: orientation to F-O options Thanks Billy. Very helpful as I obviously didn't have the complete status. Hope you don't mind too much I dragged you into this discussion Matt On Thu, 21 Aug 2008, William Chauncey Barott wrote: > Matt, > Since you invoked my name here, I'll summarize my experiences. > > The 3m copper links from the iBobs to the BEE2 are completely > unacceptable with even Gore cables. My goal is to entirely eliminate 3m > copper from the ATA beamformer. Your number of 1 failure per hour is on > the good side. I pulled up some notes from my tests at Hat Creek, and > found that in a single beamformer "as-wired" for 24 inputs: Over an > 8-hour period, they roughly break down to: > > 19 With no errors detected over 8 hours > 2 With sporadic (roughly once per 90 min) burst linkdowns > 2 With ~ 4-8 burst linkdowns per half-hour, every half-hour > 1 With more, usually nearly constant burst linkdowns > > One of my BEE2-BEE2 links (also over 3m) showed ~ 4-8 burst linkdowns > per half-hour. > > This depends very much on the temperature of the devices and also on > which devices are used. For example, the above data was from 4 > different BEE2s, and 4 of the 5 links with errors were recorded on > corner chip FPGA#2 of the BEE2 (and the one that was on a different FPGA > was a sporadic error). So something about the receiving hardware > definitely matters here. I also noticed an apparent preference that the > iBobs were more susceptible to creating errors on their bottom/right > XAUI port (X1?) and the top/left xaui port (X0?) tended to produce fewer > errors. Also, for what it's worth, hot-plugging XAUI cables was found > to generally be a BAD idea in an error-prone environment: I found that > simply removing/ reinserting a cable could permanently increase the > error rates by at least an order of magnitude until the power was > hard-reset (the reset button on the iBob was not satisfactory). > > Now as to my iBobs: The number above (5 bad, 19 good) is somewhat > incomplete - I did more tests using ~ 1min observation times of the link > quality using a FPGA#2 corner chip as the receiver (the most sensitive > to errors). I found that in that case, only 7 iBobs did not produce > measurable link errors over 3m copper in that 1min time scale. (This was > UNCOOLED) > > Since we've cooled the iBobs, these link errors have been mitigated > enough to observe but not entirely eliminated. I've made it a policy to > use XAUI0 for beamformer #1 and XAUI1 for Beamformer #2 (with the > intention of not using XAUI1 when we get our new rack of iBobs for > Beamformer #2). This is because, after cooling, 8 iBobs showed 3m link > errors on XAUI1, but only 2 on XAUI0 (to corner chip #2) > > I'd still expect at least 1 link error per day on some links (no data, > just memories). However, replacing the worst offenders with fiber or > 1.5m copper (1m was too short) definitely improved this: I have not seen > any errors on fiber/1.5m links during my tests, which spanned many days. > > Hope it helps, > > Billy > > > > -----Original Message----- > From: Matt Dexter [mailto:[email protected]] > Sent: Wednesday, August 20, 2008 2:18 PM > To: Jonathan Weintroub > Cc: [email protected]; Barott, William Chauncey > Subject: Re: [casper] fast serial communications: orientation to F-O > options > > Based on testing with iBob -> F Board I strongly recommend WL Gore over > Fujitsu cables. Fujitsu were completely unacceptable. > > Correct, we do NOT trust 3m copper cables for iBob->BEE2. > I don't have a good qualitative number to provide for the failure rate. > As an obscenely crude guess, > 1 failure per 1 hour observation with 12 3m cables in use and > ~ 7 Gbps continuous user data rate over each 10 Gbps cable. > failure rate may be much higher than this. > Perhaps Billy, the poor fellow who had to find and fight this problem, > will have time to share more valuable data. > > BTW, I believe the packetized correlator project uses 1m for > iBob->BEE2. I think there is some thought that 1.5m and 2m cables > might work and perhaps those weren't used only because just 1m and 3m > cables were in stock. Jason Manley and others are the relevant experts. > > Correct, the F Board is an RAL board; see attached for a sketch. > > Correct, BEE2 -> BEE2 1m and 1.5m copper cables > > Again Billy has the direct, and painful, experience but we have > found significant decrease in CX4 error rates as the iBob's FPGAs > were cooled from 80 degC to ~ 45 degC. This is the package temperature > as measured by an IR gun because the iBob was not designed to take > advantage of the FPGA IC's DXP and DXN pins that would tell us the > actual die temperature. See attached sketch of our sampler chassis > showing how we pack the iBob plates into a chassis. > The chassis level fans which blowing cold across iBobs from below isn't > shown. Final attachment shows the cutout we make into the mounting plate > to allow the FPGA's heatsink fan to funtion. > > On Wed, 20 Aug 2008, Jonathan Weintroub wrote: > > > Thanks Matt and John for your input. You have provided needed > > guidance for on the performance of copper cables. I may well give the > > Gore a try, I have no objection to paying less. I do wonder, though, > > if our good experience with 3 m lengths might be related to our use of > > Fujitsu (??). Matt, yes I had read the cables & parameters memo. It > > did not really really cover fiber optics though (the modifications > > needed on an iBOB or BEE2 to retrofit, and sources for cables). Your > > response fills this gap---I am sure I'm not alone in wanting to hear > > how the PARALIGHT and ZARLINK stuff works out. > > > > In our case, given that we can stick with short cables and have not > > yet hit problems with the copper even at 3 m, we'll steer clear of the > > cost and complication of F/O for now. > > > > Just to clarify: > > > > > At HCRO for ATA we are going to use > > > iBob -> BEE2 1m copper or 3 to 10m fiber optic > > > iBob -> F Board 1,2 and 3 m copper (more testing TBD on 3m) > > > BEE2 -> BEE2 1m and 1.5m > > > BEE2 -> iBob 3 to 10m fiber optic > > > > > > It appears you don't trust even 3-m copper on iBOB->BEE2. How bad is > > it in your experience? > > > > What is "F Board?" A non-CASPER FFT design . . .? > > > > The third case is 1 m and 1.5m copper, right? > > > > Also, I am probably missing something, but what is the relevance of > > the discussion of cooling, pitch and mounting plate modifications in > > the context of fast serial? > > > > > All of our iBobs will have fans mounted to the FPGA's heatsink > > > 5VDC fan 40x40x6mm fan, digikey GM0504PEV1-8GN > > > 5.9CFM Sunon GM0504PEV1-8GN > > > and the aluminum mounting plate is hogged out so that they can > > > still be mounted on 8hp = 1.6" pitch. These fans are in addition > > > to the chassis level fans (cold air in at bottom front; warm aire > > > out at top back). > > > > > > On Aug 19, 2008, at 5:16 PM, John Ford wrote: > > > > > > > >> (I have re-read a prior thread from early May on this topic > involving > > >> John, Matt, Jouko, and Francois, however the sourcing information > was > > >> from 2005, and the applicability of the information to our > situation > > >> was not entirely clear from the conversation. Also perhaps more > has > > >> been learned since May.) > > > > > > Our experience here is limited to using the GORE cables for both 10 > > > GbE > > > and XAUI. The iBOB and BEE2 seem to be particular about needing > short > > > cables. We use 1 meter cables on our iBOB to BEE links. longer > > > cables > > > seem to work with 10 GbE cards (myricom) and commercial switches > > > (fujitsu). We bought our latest cables along with our latest > > > myricom 10 > > > GbE cards from bell electronics, a Myricom distributor. > > > > > > Hope this helps! > > > > > > John > > > > > > > > > > > > >
