Re: [casper] Ethernet speed interoperability
Thanks also for this information, Greg, also helps and I very much appreciate it. Best wishes, Jonathan > On Nov 30, 2022, at 7:16 PM, Greg Lindahl wrote: > > Jono, > > The secret is that none of this autonegotiates. You have to set the > link speed and the link width explicitly on the switch. The switch you > own, a N8550-32C, with 100 gig ports, was able to talk to a Google > Transfer Device with a 40 gig port, after explicitly setting the speed > down on the switch. I used a cable with QSFP28 ends. > > I think FEC is the same way, it'll be on by default in 100gig and 25 > gig settings, but you can disable it on the switch if you like. And > Mellanox cards, for example, do have a setting that turns off FEC. > > Why does none of this autonegotiate? I don't understand why, it's > quite similar to Infiniband and PCI Express, both of which > autonegotiate. > > -- greg > > > -- > You received this message because you are subscribed to the Google Groups > "casper@lists.berkeley.edu" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to casper+unsubscr...@lists.berkeley.edu. > To view this discussion on the web visit > https://www.google.com/url?q=https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/20221201001617.GA14629%2540rd.bx9.net=gmail-imap=167045858200=AOvVaw3CZhskrFQaB_1SwVN-YSID. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/1BE767DA-672C-48AB-8E90-870380DDC17A%40cfa.harvard.edu.
Re: [casper] Ethernet speed interoperability
Jono, The secret is that none of this autonegotiates. You have to set the link speed and the link width explicitly on the switch. The switch you own, a N8550-32C, with 100 gig ports, was able to talk to a Google Transfer Device with a 40 gig port, after explicitly setting the speed down on the switch. I used a cable with QSFP28 ends. I think FEC is the same way, it'll be on by default in 100gig and 25 gig settings, but you can disable it on the switch if you like. And Mellanox cards, for example, do have a setting that turns off FEC. Why does none of this autonegotiate? I don't understand why, it's quite similar to Infiniband and PCI Express, both of which autonegotiate. -- greg -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/20221201001617.GA14629%40rd.bx9.net.
Re: [casper] Ethernet speed interoperability
Hi Jack, Thanks for weighing in. Actually your answer doesn’t read as the same one that Mitch gave. Further clarification: > On Nov 30, 2022, at 3:08 PM, Jack Hickish wrote: > > Yes, I think that's probably right. There are old standards which probably > allow you to passively convert a single 10x10G connector which was designed > for 100G into 10 individual SFP-based 10G links, but life is likely easier if > you just throw everything in a switch and let it handle the line rate > differences between the more common 100GbE and 10GbE standards. The 100-10 doesn’t have to be purely passive, a switch which converts 100 G QSFP28 to 10 GigE SFP+, if a current and supported product would be fine. > I would think your best bet is to get a fully QSFP28 switch which supports > both 100GbE and 4x10GbE on its ports, the latter requiring MTP->4xLC fiber > breakouts to connect to 10G SFP interfaces. > I would imagine (though no claims that my imagination aligns with reality) > that any switch with QSFP28 ports which advertise 4x25G breakout mode will > also support 4x10G. The former might be more obviously advertised. So after a bit of googling I find that 25 GigE uses the SFP28 connector, which is pin and electrically compatible (though faster) than SFP+. So I suspect you’re correct here. This could be the solution we seek…. Jonathan -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/4C5CD036-DD4C-4FCB-92E5-18026D3F3FF6%40cfa.harvard.edu.
Re: [casper] Ethernet speed interoperability
> > > I would imagine (though no claims that my imagination aligns with reality) > that any switch with QSFP28 ports which advertise 4x25G breakout mode will > also support 4x10G. The former might be more obviously advertised. > > I think this is correct, the only caveat I'd add is for some switches (at least I found with an Arista 7060X and optic connectors) you need a 40G connector on the switch side rather than a 100G, because it expects RSFEC for 25G+ (and we don't have that in our 10G core) and it will not allow the port to be configured for 10G breakout unless you have a 40G connector. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/CAPU71P-WZGD%2BeED1BFn8YWTmUATag2EBtFybmN4myxW%3DCCwXxQ%40mail.gmail.com.
Re: [casper] Ethernet speed interoperability
On Wed, 30 Nov 2022 at 17:18, 'Jonathan Weintroub' via casper@lists.berkeley.edu wrote: > Hi Mitch, > > Your answer helps a lot, thanks! I think the existence of the old 100 > standard explains conflicting information found on google search. > > To constrain the scenario a bit: we don’t want to use a deprecated > standard. And though it was buried, I did specify that the 100 G standard > we were starting from was QSFP28. So it would seem it is *not* convenient > to transform QSFP28 into 10x SFP+, at least to your understanding. Is that > right? > Yes, I think that's probably right. There are old standards which probably allow you to passively convert a single 10x10G connector which was designed for 100G into 10 individual SFP-based 10G links, but life is likely easier if you just throw everything in a switch and let it handle the line rate differences between the more common 100GbE and 10GbE standards. I would think your best bet is to get a fully QSFP28 switch which supports both 100GbE and 4x10GbE on its ports, the latter requiring MTP->4xLC fiber breakouts to connect to 10G SFP interfaces. I would imagine (though no claims that my imagination aligns with reality) that any switch with QSFP28 ports which advertise 4x25G breakout mode will also support 4x10G. The former might be more obviously advertised. Basically I'm just agreeing with Mitch :) Cheers Jack > If anyone else can comment on what I hope is a clearer framing of the > question, I would appreciate it. > > Thanks, > Jonathan > > > > On Nov 30, 2022, at 11:43 AM, Mitch Burnett > wrote: > > > > Hi Jonathon, > > > > I probably will not be answering your question completely. But perhaps > just a bit of input for some further searching? > > > > My understanding is that the first generation 100G PHY standard was IEEE > 802.3ba-2010 and that used 10 lanes at 10G line rates. However, I believe > the only standard that used a SFP based module was the copper 100GBASE-CR10 > using a QSFP+ transceiver. The rest were the CX style transceiver modules. > > > > And so most 100G switches using QSFP28 transceivers this would not be a > standard implementation they would support because the underlying line rate > is the 28Gbps. Typically what I see for example is that each port of a > 32-port 100G QSFP28 switch can be configured as 32 40G ports that then is > based on the SFP+ standard for 10G and can do 128 10G ports. > > > > So using QSFP28/SFP+ where the underlying line rate in the PHY is 28Gbps > may not be feasible (however I am not an authority here and cannot say it > doesn’t exist). But again, there is an old standard that does use 10x10 and > you may be able to follow that route if interested? But it is old > technology and probably hard to come by? > > > > Best, > > > > Mitch > > > >> On Nov 30, 2022, at 9:09 AM, 'Jonathan Weintroub' via > casper@lists.berkeley.edu wrote: > >> > >> Hi CASPERites, > >> > >> I believe that 40 Gbps Ethernet interoperates rather transparently with > 10 Gbps. Because the 40 Gbps is arranged as four x 10 G lanes, so its > possibly to break a 40 QSFP port into four 10 SFP+ with a suitable breakout > cable. Or with proper packet addressing via a network switch with 40 and > 10 ports. > >> > >> Likewise 100 Gbps interoperates with 50/25 easily. And 400 with 200 and > 200, and so on. > >> > >> The question arose in a meeting today whether it’s a simple matter to > transform a 100 Gbps Ethernet stream into 10 Gbps streams, 10 of them or > whatever? There are various switches on the market with both 100 and 10 > Gbps ports, but not clear whether these re distinct networks, or whether > they transparently interoperate. > >> > >> Can someone offer input as to whether transforming 100 Gbps QSFP28 into > multiple 10 Gbps SFP+ links is easily accomplished in a suitable switch? > Or a complicated endeavor? > >> > >> Hope this is clear. Thanks, > >> > >> Jonathan > >> > >> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "casper@lists.berkeley.edu" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to casper+unsubscr...@lists.berkeley.edu. > >> To view this discussion on the web visit > https://www.google.com/url?q=https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/1C6A1476-CBFD-4498-9DE5-2DE747741556%2540cfa.harvard.edu=gmail-imap=167043141000=AOvVaw0E-4FWmowXFs2Vap0k4io9 > . > > > > -- > > You received this message because you are subscribed to the Google > Groups "casper@lists.berkeley.edu" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to casper+unsubscr...@lists.berkeley.edu. > > To view this discussion on the web visit > https://www.google.com/url?q=https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/0DE96746-94B2-485E-8F76-445EE24A5136%2540byu.edu=gmail-imap=167043141000=AOvVaw2V04Gay2d2NlGlbvrcDLDX > . > > -- > You received this message because you
Re: [casper] Ethernet speed interoperability
Hi Mitch, Your answer helps a lot, thanks! I think the existence of the old 100 standard explains conflicting information found on google search. To constrain the scenario a bit: we don’t want to use a deprecated standard. And though it was buried, I did specify that the 100 G standard we were starting from was QSFP28. So it would seem it is *not* convenient to transform QSFP28 into 10x SFP+, at least to your understanding. Is that right? If anyone else can comment on what I hope is a clearer framing of the question, I would appreciate it. Thanks, Jonathan > On Nov 30, 2022, at 11:43 AM, Mitch Burnett wrote: > > Hi Jonathon, > > I probably will not be answering your question completely. But perhaps just > a bit of input for some further searching? > > My understanding is that the first generation 100G PHY standard was IEEE > 802.3ba-2010 and that used 10 lanes at 10G line rates. However, I believe the > only standard that used a SFP based module was the copper 100GBASE-CR10 using > a QSFP+ transceiver. The rest were the CX style transceiver modules. > > And so most 100G switches using QSFP28 transceivers this would not be a > standard implementation they would support because the underlying line rate > is the 28Gbps. Typically what I see for example is that each port of a > 32-port 100G QSFP28 switch can be configured as 32 40G ports that then is > based on the SFP+ standard for 10G and can do 128 10G ports. > > So using QSFP28/SFP+ where the underlying line rate in the PHY is 28Gbps may > not be feasible (however I am not an authority here and cannot say it doesn’t > exist). But again, there is an old standard that does use 10x10 and you may > be able to follow that route if interested? But it is old technology and > probably hard to come by? > > Best, > > Mitch > >> On Nov 30, 2022, at 9:09 AM, 'Jonathan Weintroub' via >> casper@lists.berkeley.edu wrote: >> >> Hi CASPERites, >> >> I believe that 40 Gbps Ethernet interoperates rather transparently with 10 >> Gbps. Because the 40 Gbps is arranged as four x 10 G lanes, so its >> possibly to break a 40 QSFP port into four 10 SFP+ with a suitable breakout >> cable. Or with proper packet addressing via a network switch with 40 and 10 >> ports. >> >> Likewise 100 Gbps interoperates with 50/25 easily. And 400 with 200 and 200, >> and so on. >> >> The question arose in a meeting today whether it’s a simple matter to >> transform a 100 Gbps Ethernet stream into 10 Gbps streams, 10 of them or >> whatever? There are various switches on the market with both 100 and 10 Gbps >> ports, but not clear whether these re distinct networks, or whether they >> transparently interoperate. >> >> Can someone offer input as to whether transforming 100 Gbps QSFP28 into >> multiple 10 Gbps SFP+ links is easily accomplished in a suitable switch? Or >> a complicated endeavor? >> >> Hope this is clear. Thanks, >> >> Jonathan >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "casper@lists.berkeley.edu" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to casper+unsubscr...@lists.berkeley.edu. >> To view this discussion on the web visit >> https://www.google.com/url?q=https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/1C6A1476-CBFD-4498-9DE5-2DE747741556%2540cfa.harvard.edu=gmail-imap=167043141000=AOvVaw0E-4FWmowXFs2Vap0k4io9. > > -- > You received this message because you are subscribed to the Google Groups > "casper@lists.berkeley.edu" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to casper+unsubscr...@lists.berkeley.edu. > To view this discussion on the web visit > https://www.google.com/url?q=https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/0DE96746-94B2-485E-8F76-445EE24A5136%2540byu.edu=gmail-imap=167043141000=AOvVaw2V04Gay2d2NlGlbvrcDLDX. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/F96BDF14-D29D-4ADC-914A-44FA4915E22D%40cfa.harvard.edu.
Re: [casper] Ethernet speed interoperability interoperability
Hi Jonathon, I probably will not be answering your question completely. But perhaps just a bit of input for some further searching? My understanding is that the first generation 100G PHY standard was IEEE 802.3ba-2010 and that used 10 lanes at 10G line rates. However, I believe the only standard that used a SFP based module was the copper 100GBASE-CR10 using a QSFP+ transceiver. The rest were the CX style transceiver modules. And so most 100G switches using QSFP28 transceivers this would not be a standard implementation they would support because the underlying line rate is the 28Gbps. Typically what I see for example is that each port of a 32-port 100G QSFP28 switch can be configured as 32 40G ports that then is based on the SFP+ standard for 10G and can do 128 10G ports. So using QSFP28/SFP+ where the underlying line rate in the PHY is 28Gbps may not be feasible (however I am not an authority here and cannot say it doesn’t exist). But again, there is an old standard that does use 10x10 and you may be able to follow that route if interested? But it is old technology and probably hard to come by? Best, Mitch > On Nov 30, 2022, at 9:09 AM, 'Jonathan Weintroub' via > casper@lists.berkeley.edu wrote: > > Hi CASPERites, > > I believe that 40 Gbps Ethernet interoperates rather transparently with 10 > Gbps. Because the 40 Gbps is arranged as four x 10 G lanes, so its possibly > to break a 40 QSFP port into four 10 SFP+ with a suitable breakout cable. Or > with proper packet addressing via a network switch with 40 and 10 ports. > > Likewise 100 Gbps interoperates with 50/25 easily. And 400 with 200 and 200, > and so on. > > The question arose in a meeting today whether it’s a simple matter to > transform a 100 Gbps Ethernet stream into 10 Gbps streams, 10 of them or > whatever? There are various switches on the market with both 100 and 10 Gbps > ports, but not clear whether these re distinct networks, or whether they > transparently interoperate. > > Can someone offer input as to whether transforming 100 Gbps QSFP28 into > multiple 10 Gbps SFP+ links is easily accomplished in a suitable switch? Or > a complicated endeavor? > > Hope this is clear. Thanks, > > Jonathan > > > > -- > You received this message because you are subscribed to the Google Groups > "casper@lists.berkeley.edu" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to casper+unsubscr...@lists.berkeley.edu. > To view this discussion on the web visit > https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/1C6A1476-CBFD-4498-9DE5-2DE747741556%40cfa.harvard.edu. -- You received this message because you are subscribed to the Google Groups "casper@lists.berkeley.edu" group. To unsubscribe from this group and stop receiving emails from it, send an email to casper+unsubscr...@lists.berkeley.edu. To view this discussion on the web visit https://groups.google.com/a/lists.berkeley.edu/d/msgid/casper/0DE96746-94B2-485E-8F76-445EE24A5136%40byu.edu.
