[casper] ROACH2 10GbE implementation parameters
Hi all, I'm testing a switch for a ROACH2-based packetized FX correlator. I currently have a single ROACH with 8 SFP-ports connected, 4 on a 2m QSFP-4xSFP cable, and 4 on 3m QSFP-4xSFP cable. Both are passive copper (30 AWG). The switch is a Mellanox SX1012, which might be the root of the problem -- Jason, I'm about to email you :) The shorter cable works great, but the longer cable is dropping a few percent of packets even at very low data rates. I'm still investigating, but it looks like sending down the long cable to the switch and receiving on the short cable seems to work better than the other way around. I have left all the implementation settings on the 10gig (v2) block at defaults. Does anyone have any advice on what pre-emph settings etc. might improve things? Thanks Jack
Re: [casper] ROACH2 10GbE implementation parameters
Can you get any sense of whether the packets are being received by the switch and dropped by the roach vs not being received by the switch in the first place? Can you change any transmission parameters on the switch? FWIW, I think you can set preemph and other such params on the ROACH at runtime via software. Dave On Jul 22, 2014, at 11:27 AM, Jack Hickish wrote: Hi all, I'm testing a switch for a ROACH2-based packetized FX correlator. I currently have a single ROACH with 8 SFP-ports connected, 4 on a 2m QSFP-4xSFP cable, and 4 on 3m QSFP-4xSFP cable. Both are passive copper (30 AWG). The switch is a Mellanox SX1012, which might be the root of the problem -- Jason, I'm about to email you :) The shorter cable works great, but the longer cable is dropping a few percent of packets even at very low data rates. I'm still investigating, but it looks like sending down the long cable to the switch and receiving on the short cable seems to work better than the other way around. I have left all the implementation settings on the 10gig (v2) block at defaults. Does anyone have any advice on what pre-emph settings etc. might improve things? Thanks Jack
Re: [casper] ROACH2 10GbE implementation parameters
Hey Dave, Judging by the switch diagnostics, everything is making it through the switch and only being dropped at the ROACH RX end. I'm digging through the switch manual at the moment, but haven't found anything too promising looking so far. It would seem that some of the SFP ports on the ROACH are noticeably worse than others -- the long cable pretty consistently causes problems everywhere, but if I use only the worst ports with the short cable, they get much better, but far from perfect. I've twiddled the SFP+ RX parameters on the roach, but to no avail (though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, I can't see any control ports between the FPGA and the vitesse in system.mhs) Cheers, Jack On 22 July 2014 13:37, David MacMahon dav...@astro.berkeley.edu wrote: Can you get any sense of whether the packets are being received by the switch and dropped by the roach vs not being received by the switch in the first place? Can you change any transmission parameters on the switch? FWIW, I think you can set preemph and other such params on the ROACH at runtime via software. Dave On Jul 22, 2014, at 11:27 AM, Jack Hickish wrote: Hi all, I'm testing a switch for a ROACH2-based packetized FX correlator. I currently have a single ROACH with 8 SFP-ports connected, 4 on a 2m QSFP-4xSFP cable, and 4 on 3m QSFP-4xSFP cable. Both are passive copper (30 AWG). The switch is a Mellanox SX1012, which might be the root of the problem -- Jason, I'm about to email you :) The shorter cable works great, but the longer cable is dropping a few percent of packets even at very low data rates. I'm still investigating, but it looks like sending down the long cable to the switch and receiving on the short cable seems to work better than the other way around. I have left all the implementation settings on the 10gig (v2) block at defaults. Does anyone have any advice on what pre-emph settings etc. might improve things? Thanks Jack
Re: [casper] ROACH2 10GbE implementation parameters
Hi, Jack, On Jul 22, 2014, at 1:46 PM, Jack Hickish wrote: Judging by the switch diagnostics, everything is making it through the switch and only being dropped at the ROACH RX end. I'm digging through the switch manual at the moment, but haven't found anything too promising looking so far. It would seem that some of the SFP ports on the ROACH are noticeably worse than others -- the long cable pretty consistently causes problems everywhere, but if I use only the worst ports with the short cable, they get much better, but far from perfect. You could try going from ROACH2 to switch to 10 GbE NIC is a PC. If that works fine (I suspect it will), then the problem would indeed seem to tbe on the ROACH2 RX side. I've twiddled the SFP+ RX parameters on the roach, but to no avail (though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, I can't see any control ports between the FPGA and the vitesse in system.mhs) Oh, yeah, the pre-emph and other parameters that can be changed in the 10 GbE yellow block are only for the MGT drivers in the ROACH2 FPGA that talk to the Vitesse PHY. I think there is an MDIO interface to the SFP+ PHY that might provide a path for tweaking things in the SFP+ PHY itself. This MDIO interface used to be unreliable, but I think it has been fixed (though I haven't tried it since then). Dave
