Re: [casper] Network Switches

2014-09-03 Thread John Ford
Hi all. After looking further into this, I think for our purposes the MSX1024 is the way to go. It has 48 10 gbe ports with SFP+ that we can use with optical transcievers. I don't think there's a 4-1 splitter cable with the ability to use optical SFP+ transcievers, so we'll have to limit our

Re: [casper] Network Switches

2014-09-03 Thread John Ford
Or does someone know of a 40-10 optical cable? Yes, and mellanox have them, including the fibre cable itself in short distances. But it's only multi-mode (40GBASE-SR4 to 4x 10GBASE-SR) so you're limited to ~100m or so. It uses an MTP connector on the 40G side, and an LC connector on the 10G

Re: [casper] Network Switches

2014-09-03 Thread Jason Manley
Just be careful with the 1036 and breaking the 40g ports out to 10g. you can't break out all the ports, and when you break one out, you sometimes lose another 40g port! Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27 21 506 7300 On 03 Sep 2014, at 15:07, John Ford

Re: [casper] Network Switches

2014-09-03 Thread Lincoln Greenhill
John - Mellanox hides this information in the back of their manuals in my experience. Best to download. L. On 9/3/14, 9:11 AM, Jason Manley wrote: Just be careful with the 1036 and breaking the 40g ports out to 10g. you can't break out all the ports, and when you break one out, you

Re: [casper] Network Switches

2014-09-03 Thread John Ford
Just be careful with the 1036 and breaking the 40g ports out to 10g. you can't break out all the ports, and when you break one out, you sometimes lose another 40g port! I just noticed that! It's true also with the 1024. Jason Manley CBF Manager SKA-SA Cell: +27 82 662 7726 Work: +27

[casper] Network Switches

2014-09-02 Thread John Ford
Hi all. Has anyone tested the Mellanox SX-1024 series of switches with ROACH-2 and 10 and/or 40 gb NICs? These switches have 48 10 Gbe ports and 12 40 Gbe ports on them. Thanks! John

Re: [casper] Network Switches

2014-09-02 Thread Dan Werthimer
hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) i think jason has also tested this switch. best wishes, dan

Re: [casper] Network Switches

2014-09-02 Thread John Ford
hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was visiting berkeley. the SX1012 worked beautifully on roach2, after jack upgraded the switch firmware. and it's a great price. ($6K) Yeah, we're looking at a 1024, or a 1036. They all use the

Re: [casper] Network Switches

2014-09-02 Thread Jack Hickish
Hi John, As Dan said I've tested (to some extent) the SX1012. I just used one ROACH2 and corner turned data through 8 x 10GbE ports. It worked well basically up to line rate, with no CRC errors after a few hours of operation, but only after - Jason put me in touch with some Mellanox guys who

Re: [casper] Network Switches

2014-09-02 Thread Lincoln Greenhill
Hi John, Email is passing in the ether. I have eyed the SX1035 and SX1036 w/ upgrades in mind. What is your understanding of the difference? Lincoln On 9/2/14, 11:54 AM, John Ford wrote: hi john, jack hickish tested a mellanox SX1012 (12x40Gbe, or 48x10Gbe, or mixture), when he was

Re: [casper] Network Switches

2014-09-02 Thread Jason Manley
The SX1012 and SX1036 definitely use the same ASIC series and both suffer from the PHY link problem. I've got seven of them here, and tested them with 20 different ROACH-2 boards. As Jack points out, some links are worse than others and swapping cables and ROACH2 mezzanine cards often sorts the

Re: [casper] Network Switches

2014-09-02 Thread Dan Werthimer
hi jason, thanks for all this info. in the leda correlator, the 10/40Gbe links are unidirectional, roach2--switch and switch--gpuserver so i don't think leda would encounter the switch--roach2 bit errors, although it wouldn't hurt to upgrade the firmware. best wishes, dan On Tue, Sep

Re: [casper] Network Switches

2014-09-02 Thread Jack Hickish
Not to steer this conversation too far off track, but does a patch exist to enable the ethernet CRC check within the 10Gb yellow block? When I tested the switch I used Dave MacMahon's CRC generator block, but it would be nice if there was the option to have the 10 gig core use the UDP crc -- or

Re: [casper] Network Switches

2014-09-02 Thread John Ford
Hi John, Email is passing in the ether. I have eyed the SX1035 and SX1036 w/ upgrades in mind. What is your understanding of the difference? I can't see any, really. The sx1035 has slightly lower latency, and is way more expensive... I don't really understand what L2 Lite Switching is,