Re: [casper] ROACH2 10GbE implementation parameters
though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, Yep, you've got the right of it. You'll need to tweak the Vitesse PHY through the control interface from software; there is an MDIO interface available, which does work. But we don't have a full datasheet for the part and so can't offer a clear solution for you. There are a LOT of parameters to tweak; not just a generic RX equalisation knob like on the Xilinx, for example. I recall a multi-tap filter chain, gain, feedback loops and other trickery on the RX side. So we've just left it on defaults, but it's definitely not optimal. The Mellanox ASICs are the only ones we've had trouble with (all Broadcom based switches seem to work fine with 3m cables, for example) and Mellanox have a software fix for the issue, by tweaking their own PHY TX parameters. I believe this fix will be in their General firmware release soon (though no date has been promised yet). Until then, I suggest you ask for an early-release/testing firmware. Jason
Re: [casper] ROACH2 10GbE implementation parameters
Thanks for the info, I will certainly try and get my hands on some new Mellanox firmware before fighting with the vitesse. Cheers, Jack On 23 July 2014 01:04, Jason Manley jman...@ska.ac.za wrote: though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, Yep, you've got the right of it. You'll need to tweak the Vitesse PHY through the control interface from software; there is an MDIO interface available, which does work. But we don't have a full datasheet for the part and so can't offer a clear solution for you. There are a LOT of parameters to tweak; not just a generic RX equalisation knob like on the Xilinx, for example. I recall a multi-tap filter chain, gain, feedback loops and other trickery on the RX side. So we've just left it on defaults, but it's definitely not optimal. The Mellanox ASICs are the only ones we've had trouble with (all Broadcom based switches seem to work fine with 3m cables, for example) and Mellanox have a software fix for the issue, by tweaking their own PHY TX parameters. I believe this fix will be in their General firmware release soon (though no date has been promised yet). Until then, I suggest you ask for an early-release/testing firmware. Jason
Re: [casper] ROACH2 10GbE implementation parameters
Hi guys, though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, Yep, you've got the right of it. You'll need to tweak the Vitesse PHY through the control interface from software; there is an MDIO interface available, which does work. But we don't have a full datasheet for the part and so can't offer a clear solution for you. Access to Vitesse data sheets requires signing an NDA. Its not actually that much of a pain. Just email a request via the Vitesse web site, and you'll get a response. The downside is that I don't think CASPER as a community can get access to the data sheet and then share it. However, someone like Jack could be the expert and he could aid others :) Cheers, Dave
Re: [casper] ROACH2 10GbE implementation parameters
Can you get any sense of whether the packets are being received by the switch and dropped by the roach vs not being received by the switch in the first place? Can you change any transmission parameters on the switch? FWIW, I think you can set preemph and other such params on the ROACH at runtime via software. Dave On Jul 22, 2014, at 11:27 AM, Jack Hickish wrote: Hi all, I'm testing a switch for a ROACH2-based packetized FX correlator. I currently have a single ROACH with 8 SFP-ports connected, 4 on a 2m QSFP-4xSFP cable, and 4 on 3m QSFP-4xSFP cable. Both are passive copper (30 AWG). The switch is a Mellanox SX1012, which might be the root of the problem -- Jason, I'm about to email you :) The shorter cable works great, but the longer cable is dropping a few percent of packets even at very low data rates. I'm still investigating, but it looks like sending down the long cable to the switch and receiving on the short cable seems to work better than the other way around. I have left all the implementation settings on the 10gig (v2) block at defaults. Does anyone have any advice on what pre-emph settings etc. might improve things? Thanks Jack
Re: [casper] ROACH2 10GbE implementation parameters
Hey Dave, Judging by the switch diagnostics, everything is making it through the switch and only being dropped at the ROACH RX end. I'm digging through the switch manual at the moment, but haven't found anything too promising looking so far. It would seem that some of the SFP ports on the ROACH are noticeably worse than others -- the long cable pretty consistently causes problems everywhere, but if I use only the worst ports with the short cable, they get much better, but far from perfect. I've twiddled the SFP+ RX parameters on the roach, but to no avail (though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, I can't see any control ports between the FPGA and the vitesse in system.mhs) Cheers, Jack On 22 July 2014 13:37, David MacMahon dav...@astro.berkeley.edu wrote: Can you get any sense of whether the packets are being received by the switch and dropped by the roach vs not being received by the switch in the first place? Can you change any transmission parameters on the switch? FWIW, I think you can set preemph and other such params on the ROACH at runtime via software. Dave On Jul 22, 2014, at 11:27 AM, Jack Hickish wrote: Hi all, I'm testing a switch for a ROACH2-based packetized FX correlator. I currently have a single ROACH with 8 SFP-ports connected, 4 on a 2m QSFP-4xSFP cable, and 4 on 3m QSFP-4xSFP cable. Both are passive copper (30 AWG). The switch is a Mellanox SX1012, which might be the root of the problem -- Jason, I'm about to email you :) The shorter cable works great, but the longer cable is dropping a few percent of packets even at very low data rates. I'm still investigating, but it looks like sending down the long cable to the switch and receiving on the short cable seems to work better than the other way around. I have left all the implementation settings on the 10gig (v2) block at defaults. Does anyone have any advice on what pre-emph settings etc. might improve things? Thanks Jack
Re: [casper] ROACH2 10GbE implementation parameters
Hi, Jack, On Jul 22, 2014, at 1:46 PM, Jack Hickish wrote: Judging by the switch diagnostics, everything is making it through the switch and only being dropped at the ROACH RX end. I'm digging through the switch manual at the moment, but haven't found anything too promising looking so far. It would seem that some of the SFP ports on the ROACH are noticeably worse than others -- the long cable pretty consistently causes problems everywhere, but if I use only the worst ports with the short cable, they get much better, but far from perfect. You could try going from ROACH2 to switch to 10 GbE NIC is a PC. If that works fine (I suspect it will), then the problem would indeed seem to tbe on the ROACH2 RX side. I've twiddled the SFP+ RX parameters on the roach, but to no avail (though I think these only affect the short link between the FPGA and the vitesse tranceiver anyway, I can't see any control ports between the FPGA and the vitesse in system.mhs) Oh, yeah, the pre-emph and other parameters that can be changed in the 10 GbE yellow block are only for the MGT drivers in the ROACH2 FPGA that talk to the Vitesse PHY. I think there is an MDIO interface to the SFP+ PHY that might provide a path for tweaking things in the SFP+ PHY itself. This MDIO interface used to be unreliable, but I think it has been fixed (though I haven't tried it since then). Dave
