Re: Recommended L2 switches for a new IXP
On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote: I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300. We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW support for it. But VXLAN does *not* create a network for you, it relies on some existing underlying IP network, on top of which VXLAN creates stateless tunnels. By using TRILL, we could connect 4 switches into a ring (or any other reasonable topology) and have a fully functional network with shortest-path routing of L2 packets. With VXLAN, we'd need at least two additional IP routers with bunch of 40GE interfaces to perform the functions TRILL supports out of the box. Regards, M.
Re: Recommended L2 switches for a new IXP
For many people eliminating L2 switching and building on top of a L3 network is a good thing, especially if you are using BGP as the control plane. I'm not sure I follow the two routers with 40GE interfaces if you are just building L2 domains to interconnect people. Phil On 1/20/15, 8:04 AM, Marian Ďurkovič m...@bts.sk wrote: On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote: I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300. We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW support for it. But VXLAN does *not* create a network for you, it relies on some existing underlying IP network, on top of which VXLAN creates stateless tunnels. By using TRILL, we could connect 4 switches into a ring (or any other reasonable topology) and have a fully functional network with shortest-path routing of L2 packets. With VXLAN, we'd need at least two additional IP routers with bunch of 40GE interfaces to perform the functions TRILL supports out of the box. Regards, M.
Re: Recommended L2 switches for a new IXP
On 1/17/15, 7:15 PM, Saku Ytti s...@ytti.fi wrote: On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote: Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like routing for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS). I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS. From complexity POV, I don't expect much different development time to write functioning control-plane to either. I'm not against Trill, I think Trill, and especially SPB-M are great, now they just feel too little and 20 years too late. There was no particular reason why SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it didn't, it might have made ethernet 'good enough', that selling MPLS might have been much more difficult. -- ++ytti I think in fairly short order both TRILL and 802.1AQ will be depercated in place of VXLAN and using BGP EVPN as the control plane ala Juniper QFX5100/Nexus 9300. Phil
Re: Recommended L2 switches for a new IXP
On 19/01/2015 10:12, Marian Ďurkovič wrote: Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support per-flow loadbalancing on IXP participants' LAGs. not completely true. Extreme XOS has an interesting hack to work around this. Nick
Re: Recommended L2 switches for a new IXP
On Sat, Jan 17, 2015 at 09:15:04PM +0200, Saku Ytti wrote: On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote: Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like routing for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS). I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS. Well, it can, but as usual the devil is in the detail. For example, loadbalancing on outgoing LAGs depends on *inbound* packet encapsulation as follows: - native ethernet, TRILL, L3 MPLS : hash based on L3 and L4 headers - L2 MPLS, MACinMAC : hash based on L2 headers only. Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support per-flow loadbalancing on IXP participants' LAGs. In any case, we preferred TRILL over SPB-M not just because of that, but mainly due to a fact that TRILL provides real routing using IS-IS as we know it from IP world, while SPB still builds on top of MST and just cleverly uses multiple trees. Yes, compatibility with existing ASICs was one of the main design goals of SPB, but that's irrelevant once you have Trident HW. Regards, M.
Re: Recommended L2 switches for a new IXP
On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote: Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like routing for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS). I'm not sure what the burden refers to, but cost implications to me seem same, trident HW can do VPLS. From complexity POV, I don't expect much different development time to write functioning control-plane to either. I'm not against Trill, I think Trill, and especially SPB-M are great, now they just feel too little and 20 years too late. There was no particular reason why SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it didn't, it might have made ethernet 'good enough', that selling MPLS might have been much more difficult. -- ++ytti
Re: Recommended L2 switches for a new IXP
Last year we installed four 1RU TRILL switches in SIX - see http://www.six.sk/images/trill_ring.png Our experience after 100 days of production is only the best - TRILL setup is pretty straightforward and thanks to IS-IS it provides shortest-path IP-like routing for L2 ethernet packets over any reasonable topology out of the box (without the burden and cost implications of VPLS). Trident ASICs perform deep packet inspection so ECMP loadbalancing based on L3 and L4 headers inside TRILL-encapsulated packets works for both IPv4 and IPv6. Port-security is supported on physical ports as well as on LAGs - and L4 access-lists could be applied at the same time. As most 1RU switches are based on Trident ASICs, you just need to pick a vendor which implements TRILL properly and of course thoroughly test before deployment. We selected Huawei Cloud Engine 6850 boxes. Regards, M. Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards
Re: Recommended L2 switches for a new IXP
We always adhere to JTAC: http://kb.juniper.net/InfoCenter/index?page=contentid=KB21476actp=SUBSCRI PTION unless otherwise required by their support to change. Currently it is Junos 13.2X51-D26. My advice to you is to not use 14.1 unless you have a reason, as that is more of a dev branch in terms of stability than anything. We use VRRP, OSPF, MC-LAG, and so forth. Nothing super fancy. Stephen Carter | IT Systems Administrator | Gun Lake Tribal Gaming Commission 1123 129th Avenue, Wayland, MI 49348 Phone 269.792.1773 On 1/15/15, 4:17 AM, Richard Hartmann richih.mailingl...@gmail.com wrote: On Tue, Jan 13, 2015 at 4:45 PM, Stephen R. Carter stephen.car...@gltgc.org wrote: We love our 5100s here. Out of interest: Are you running 13.2 or 14.1? What features are you using? Our own experiences with a bunch of 48 96 port machines running 14.1 is painful to say the least. Richard brhrfont face='Arial' color='Gray' size='1'The information contained in this electronic transmission (email) is confidential information and may be subject to attorney/client privilege. It is intended only for the use of the individual or entity named above. ANY DISTRIBUTION OR COPYING OF THIS MESSAGE IS PROHIBITED, except by the intended recipient. Attempts to intercept this message are in violation of 18 U.S.C. 2511(1) of the Electronic Communications Privacy Act (ECPA), which subjects the interceptor to fines, imprisonment and/or civil damages./font
Re: Recommended L2 switches for a new IXP
On Tue, Jan 13, 2015 at 4:45 PM, Stephen R. Carter stephen.car...@gltgc.org wrote: We love our 5100s here. Out of interest: Are you running 13.2 or 14.1? What features are you using? Our own experiences with a bunch of 48 96 port machines running 14.1 is painful to say the least. Richard
Re: Recommended L2 switches for a new IXP
Software Defined Networking (SDN) features that QFX5100 supports: Automatic configuration of OVSDB-managed VXLANs with trunk interfaces 14.1X53-D15 OVSDB support 14.1X53-D10 OpenFlow v1.0 14.1X53-D10 OpenFlow v1.3.1 14.1X53-D10 VXLAN Gateway 14.1X53-D10 http://pathfinder.juniper.net/feature-explorer/select-software.html?swName=Junos+OStyp=1#family=platform=QFX5100rel=14.1X53-D15swName=Junos+OS On Tue, Jan 13, 2015 at 10:10:56PM +, Jeff Tantsura wrote: What does it mean - to be SDN ready? Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 3:25 AM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
On Wednesday, January 14, 2015 12:25:30 AM Jeff Tantsura wrote: AhhhŠ vertically integrated horizontal API¹s Green, vertically integrated horizontal API's :-). Mark. signature.asc Description: This is a digitally signed message part.
Re: Recommended L2 switches for a new IXP
On Wednesday, January 14, 2015 12:47:09 AM Jeff Tantsura wrote: Got you - artificially disabling 90% of the features otherwise supported by the OS and using half baked HAL makes product SDN ready! Sorry for the sarcasm, couldn¹t resist :) I once tested a Junos release with the X blah blah D blah blah letters in there on an EX4550. Couldn't even get LACP going, until I realized it was some kind of QFX'y release for the non-QFX EX boxes. Promptly got ride of that. Mark. signature.asc Description: This is a digitally signed message part.
Re: Recommended L2 switches for a new IXP
You can see what we have at the SIX here - http://www.seattleix.net/topology.html Mike -- Michael K. Smith mksm...@mac.com On Jan 11, 2015, at 10:37 PM, Manuel Marín m...@transtelco.net wrote: Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards
Re: Recommended L2 switches for a new IXP
Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark.
Re: Recommended L2 switches for a new IXP
QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
We love our 5100s here. I have 4 48S, and 2 24q¹s. Super fast, TISSU when it works is awesome as well... like, really awesome. Stephen Carter | IT Systems Administrator | Gun Lake Tribal Gaming Commission 1123 129th Avenue, Wayland, MI 49348 Phone 269.792.1773 On 1/13/15, 3:29 AM, Stepan Kucherenko t...@megagroup.ru wrote: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. brhrfont face='Arial' color='Gray' size='1'The information contained in this electronic transmission (email) is confidential information and may be subject to attorney/client privilege. It is intended only for the use of the individual or entity named above. ANY DISTRIBUTION OR COPYING OF THIS MESSAGE IS PROHIBITED, except by the intended recipient. Attempts to intercept this message are in violation of 18 U.S.C. 2511(1) of the Electronic Communications Privacy Act (ECPA), which subjects the interceptor to fines, imprisonment and/or civil damages./font
Re: Recommended L2 switches for a new IXP
AhhhŠ vertically integrated horizontal API¹s Cheers, Jeff -Original Message- From: Nick Hilliard n...@foobar.org Date: Tuesday, January 13, 2015 at 2:23 PM To: Jeff Tantsura jeff.tants...@ericsson.com, Eduardo Schoedler lis...@esds.com.br, nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP On 13/01/2015 22:10, Jeff Tantsura wrote: What does it mean - to be SDN ready? it means fully buzzword compliant. Nick
Re: Recommended L2 switches for a new IXP
My mistake, it's the OCX1100. http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-hardware-software.html 2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com: What does it mean - to be SDN ready? Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 3:25 AM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
Either way, you can do SDN and automation with most Juniper kit. On purchase of JCare you get free access to Junos Space - great for provisioning and management of an IXP. Regards, Tim Raphael On 14 Jan 2015, at 6:28 am, Eduardo Schoedler lis...@esds.com.br wrote: My mistake, it's the OCX1100. http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-hardware-software.html 2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com: What does it mean - to be SDN ready? Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 3:25 AM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
Got you - artificially disabling 90% of the features otherwise supported by the OS and using half baked HAL makes product SDN ready! Sorry for the sarcasm, couldn¹t resist :) Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 2:28 PM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP My mistake, it's the OCX1100. http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-h ardware-software.html 2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com: What does it mean - to be SDN ready? Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 3:25 AM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
Manuel Marín writes: Dear Nanog community [...] There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Stupid thought from someone who has never built an IXP, but has been looking at recent trends in data center networks: There are these white-box switches mostly designed for top-of-rack or spine (as in leaf-spine/fat-tree datacenter networks) applications. They have all the necessary port speeds - well 100G seems to be a few months off. I'm thinking of brands such as Edge-Core, Quanta etc. You can get them as bare-metal versions with no switch OS on them, just a bootloader according to the ONIE standard. Equipment cost seems to be on the order of $100 per SFP+ port w/o optics for a second-to-last generation (Trident-based) 48*10GE+4*40GE ToR switch. Now, for the limited and somewhat special L2 needs of an IXP, couldn't someone hack together a suitable switch OS based on Open Network Linux (ONL) or something like that? You wouldn't even need MAC address learning or most types of flooding, because at an IXP this often hurts rather than helps. For building larger fabrics you might be using something other (waves hands) than TRILL; maybe you could get away without slightly complex multi-chassis multi-channel mechanisms, and so on. Flow support sounds somewhat tough, but full netflow support that would get Roland Dobbins' usable telemetry seal of approval is probably out of reach anyway - it's a high-end feature with classical gear. With white-box switches, you could try to use the given 5-tuple flow hardware capabilities - which might not scale that well -, or use packet sampling, or try to use the built-in flow and counter mechanisms in an application-specific way. (Except *that's* a lot of work on the software side, and a usably efficient implementation requires slightly sophisticated hardware/software interfaces.) Instead of a Linux-based switch OS, one could also build an IXP application using OpenFlow and some kind of central controller. (Not to be confused with SDX: Software Defined Internet Exchange.) Has anybody looked into the feasibility of this? The software could be done as an open-source community project to make setting up regional IXPs easier/cheaper. Large IXPs could sponsor this so they get better scalability - although I'm not sure how well something like the leaf-spine/fat-tree design maps to these IXPs, which are typically distributed over several locations. Maybe they could use something like Facebook's new design, treating each IXP location as a pod. -- Simon. [1] https://code.facebook.com/posts/360346274145943
Re: Recommended L2 switches for a new IXP
What does it mean - to be SDN ready? Cheers, Jeff -Original Message- From: Eduardo Schoedler lis...@esds.com.br Date: Tuesday, January 13, 2015 at 3:25 AM To: nanog@nanog.org nanog@nanog.org Subject: Re: Recommended L2 switches for a new IXP QFX5100 is SDN ready. -- Eduardo Schoedler 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru: Is there any particular reason you prefer EX4600 over QFX5100 ? Not counting obvious differences like ports and upgrade options. It's the same chipset after all, and with all upgrades they have the same 10G density (with breakouts). Is that because you can have more 40G ports with EX4600 ? I'm still trying to find out if there are any noticeable software or feature differences. On 13.01.2015 09:01, Mark Tinka wrote: On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. -- Eduardo Schoedler
Re: Recommended L2 switches for a new IXP
On 13/01/2015 22:10, Jeff Tantsura wrote: What does it mean - to be SDN ready? it means fully buzzword compliant. Nick
Re: Recommended L2 switches for a new IXP
I look forward to this thread. I think one important thing is who is your addressable market size? I'm working with a startup IXP and there's only 20 carriers in the building. A chassis based switch would be silly as there would never be that many people present. 2x 1U switches would be more than plenty in their environment. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Manuel Marín m...@transtelco.net To: nanog@nanog.org Sent: Monday, January 12, 2015 12:35:15 AM Subject: Recommended L2 switches for a new IXP Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards
Re: Recommended L2 switches for a new IXP
On 12/01/2015 06:35, Manuel Marín wrote: We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. For a startup IXP, it would probably not be sensible to use chassis based kit due to cost / real estate issues. Some personal opinions: - I have a strong preference for using only open bridging protocols. This excludes out vendor proprietary fabrics (VDX, OTV, etc). This is important for when you do fabric upgrades on multi-site IXPs. - You will probably want a product which supports sflow, as peer-to-peer traffic graphs are massively useful. Most vendors support sflow on most of their products with the notable exception of Cisco where only the Nexus 3K team were enlightened enough to shim it in. I haven't yet come across a L2 netflow implementation which works well enough to be an adequate substitute, but ymmv. - VPLS based fabrics may be important if you have an interesting topology. If it is important to you, then you will need a VPLS implementation which will do proper load balancing over multiple links. Most don't and this is a very hard problem to handle on smaller kit. - There is no excuse for vendor transceiver locking or transceiver crippling (e.g. refusing to show DDM values) and vendors who do this need to be made aware that it's not an acceptable business proposition. - you need kit which will support Layer 2 ACLs and Layer 3 ACLs on layer 2 interfaces. - you should get in with the open-ix crowd and chat to people over pizza or peanuts. You will learn a lot from in an afternoon of immersion with peers. Nick
Re: Recommended L2 switches for a new IXP
We used to use Brocade FastIrons until we needed more 10G port density. We moved to Brocade SX's. Originally, when it was 2 or 3 peers, we used an old Netgear switch. :) Aaron On 1/12/2015 7:07 AM, Mike Hammett wrote: I look forward to this thread. I think one important thing is who is your addressable market size? I'm working with a startup IXP and there's only 20 carriers in the building. A chassis based switch would be silly as there would never be that many people present. 2x 1U switches would be more than plenty in their environment. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Manuel Marín m...@transtelco.net To: nanog@nanog.org Sent: Monday, January 12, 2015 12:35:15 AM Subject: Recommended L2 switches for a new IXP Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards -- Aaron Wendel Chief Technical Officer Wholesale Internet, Inc. (AS 32097) (816)550-9030 http://www.wholesaleinternet.com
Re: Recommended L2 switches for a new IXP
Substantial amounts of hive mind went into this topic in the formation of Open-IX and particularly around optimizing costs and maximizing traffic. See http://bit.ly/N-OIX1 for a reference. Best, -M On Mon, Jan 12, 2015 at 10:34 AM, Justin Wilson - MTIN li...@mtin.net wrote: Like Mike says, it depends on your market. Are these markets where there are existing exchanges? Cost per port is what we always look at. If we are going into a market where there won't be much growth we look at Cisco and Force 10. Their cost per port is usually cheaper for smaller 10 Gig switches. You need something that is fairly robust. Reliability in an exchange is a key component. If you go with a non-chassis switch make sure you have redundancy in your design. We like Chassis based switches because they tend to be more robust. But thats just my take on it. Justin --- Justin Wilson j...@mtin.net http://www.mtin.net Managed Services - xISP Solutions - Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering - Transit - Internet Exchange On Jan 12, 2015, at 10:24 AM, Aaron aa...@wholesaleinternet.net wrote: We used to use Brocade FastIrons until we needed more 10G port density. We moved to Brocade SX's. Originally, when it was 2 or 3 peers, we used an old Netgear switch. :) Aaron On 1/12/2015 7:07 AM, Mike Hammett wrote: I look forward to this thread. I think one important thing is who is your addressable market size? I'm working with a startup IXP and there's only 20 carriers in the building. A chassis based switch would be silly as there would never be that many people present. 2x 1U switches would be more than plenty in their environment. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Manuel Marín m...@transtelco.net To: nanog@nanog.org Sent: Monday, January 12, 2015 12:35:15 AM Subject: Recommended L2 switches for a new IXP Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards -- Aaron Wendel Chief Technical Officer Wholesale Internet, Inc. (AS 32097) (816)550-9030 http://www.wholesaleinternet.com
Re: Recommended L2 switches for a new IXP
Like Mike says, it depends on your market. Are these markets where there are existing exchanges? Cost per port is what we always look at. If we are going into a market where there won’t be much growth we look at Cisco and Force 10. Their cost per port is usually cheaper for smaller 10 Gig switches. You need something that is fairly robust. Reliability in an exchange is a key component. If you go with a non-chassis switch make sure you have redundancy in your design. We like Chassis based switches because they tend to be more robust. But thats just my take on it. Justin --- Justin Wilson j...@mtin.net http://www.mtin.net Managed Services – xISP Solutions – Data Centers http://www.thebrotherswisp.com Podcast about xISP topics http://www.midwest-ix.com Peering – Transit – Internet Exchange On Jan 12, 2015, at 10:24 AM, Aaron aa...@wholesaleinternet.net wrote: We used to use Brocade FastIrons until we needed more 10G port density. We moved to Brocade SX's. Originally, when it was 2 or 3 peers, we used an old Netgear switch. :) Aaron On 1/12/2015 7:07 AM, Mike Hammett wrote: I look forward to this thread. I think one important thing is who is your addressable market size? I'm working with a startup IXP and there's only 20 carriers in the building. A chassis based switch would be silly as there would never be that many people present. 2x 1U switches would be more than plenty in their environment. - Mike Hammett Intelligent Computing Solutions http://www.ics-il.com - Original Message - From: Manuel Marín m...@transtelco.net To: nanog@nanog.org Sent: Monday, January 12, 2015 12:35:15 AM Subject: Recommended L2 switches for a new IXP Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards -- Aaron Wendel Chief Technical Officer Wholesale Internet, Inc. (AS 32097) (816)550-9030 http://www.wholesaleinternet.com
Re: Recommended L2 switches for a new IXP
On Mon, Jan 12, 2015 at 10:43 AM, Nick Hilliard n...@foobar.org wrote: [ clip, good stuff ] - you should get in with the open-ix crowd and chat to people over pizza or peanuts. You will learn a lot from in an afternoon of immersion with peers. And you can find that crowd here http://mailman.open-ix.org/mailman/listinfo/public if interested. Best, -M
Re: Recommended L2 switches for a new IXP
On Monday, January 12, 2015 05:54:38 PM Bill Woodcock wrote: We see a lot of IXPs being formed or upgrading with Cisco Nexus 3524 switches, which have 48 1G-10G SFP/SFP+ physical ports, license-limited to 24 active, upgradeable to 48 active. FWIW, 83% of IXPs have 48 or fewer participants, and 70% of IXPs have 24 or fewer participants. And the failure rate of chassis-based switches is _way_ higher than that of stand-alone switches. So we never recommend that an IXP buy a switch larger than necessary to accommodate 18 months reasonably-projectable growth. Would tend to agree with this approach, and the above. Multi-rate (i.e., 1Gbps/10Gbps SFP/SFP+) standalone 1U switches are reasonable these days. The issue you'll probably run into with them is limited support for features you find being implemented by larger exchange points (VPLS, Sflow, e.t.c.), and quirks with the hardware that could impact things like Layer 2 or Layer 3 filtering (especially if they are using off-the-self silicon), e.t.c. Test before you buy, in as far as you can anticipate your (growth) needs. Mark. signature.asc Description: This is a digitally signed message part.
RE: Recommended L2 switches for a new IXP
People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. cheers
Re: Recommended L2 switches for a new IXP
That's what I had recommended him directly ;) Mehmet On Jan 12, 2015, at 1:41 PM, Tony Wicks t...@wicks.co.nz wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. cheers
Re: Recommended L2 switches for a new IXP
On Mon, Jan 12, 2015 at 4:41 PM, Tony Wicks t...@wicks.co.nz wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. (and you can't do anything worthwhile for acls to protect that device from the world/ix-users)
Re: Recommended L2 switches for a new IXP
On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote: People seem to be avoiding recommending actual devices, well I would recommend the Juniper EX4600 - http://www.juniper.net/us/en/products-services/switching/ ex-series/ex4600/ They are affordable, highly scalable, stackable and run JunOS. We've been quite happy with the EX4550, but the EX4600 is good too, particularly if you're coming from its younger brother. Mark. signature.asc Description: This is a digitally signed message part.
Re: Recommended L2 switches for a new IXP
On Jan 12, 2015, at 10:34 AM, Justin Wilson - MTIN li...@mtin.net wrote: Cost per port is what we always look at. If we are going into a market where there won’t be much growth we look at Cisco and Force 10. Their cost per port is usually cheaper for smaller 10 Gig switches. You need something that is fairly robust. We see a lot of IXPs being formed or upgrading with Cisco Nexus 3524 switches, which have 48 1G-10G SFP/SFP+ physical ports, license-limited to 24 active, upgradeable to 48 active. FWIW, 83% of IXPs have 48 or fewer participants, and 70% of IXPs have 24 or fewer participants. And the failure rate of chassis-based switches is _way_ higher than that of stand-alone switches. So we never recommend that an IXP buy a switch larger than necessary to accommodate 18 months reasonably-projectable growth. -Bill signature.asc Description: Message signed with OpenPGP using GPGMail
Recommended L2 switches for a new IXP
Dear Nanog community We are trying to build a new IXP in some US Metro areas where we have multiple POPs and I was wondering what do you recommend for L2 switches. I know that some IXPs use Nexus, Brocade, Force10 but I don't personally have experience with these switches. It would be great if you can share your experience and recommendations. There are so many options that I don't know if it makes sense to start with a modular switch (usually expensive because the backplane, dual dc, dual CPU, etc) or start with a 1RU high density switch that support new protocols like Trill and that supposedly allow you to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G ports for exchange participants, 40G/100G for uplinks between switches and flow support for statistics and traffic analysis. Thank you and have a great day. Regards
