Re: starlink ixp peering progress

[Bill Woodcock]

> On Feb 27, 2024, at 08:54, Dave Taht  wrote:
> One of the things I learned today was that starlink has published an 
> extensive guide as to how existing BGP AS holders can peer with them to get 
> better service.

Yes, essentially every AS does this.  The ones that follow best-practices tend 
to be pretty uniform:

https://pch.net/peering
https://aws.amazon.com/peering/policy/
https://www.bytedance.com/en/peering
https://peering.google.com/#/options/peering
https://openconnect.netflix.com/en/peering/
https://www.lumen.com/en-us/about/legal/peering-policy.html
http://peering.ovh.net/

Starlink’s peering policy is straight-forward and follows best practices.  Then 
there are ones that get a little further afield, some of which can get kinda 
unusual in their fine-print:

https://learn.microsoft.com/en-us/azure/internet-peering/policy
https://www.verizon.com/business/terms/peering/
https://www.zayo.com/wp-content/uploads/2022/07/Zayo-Global-IP-Interconnection-Policy-Final-1.pdf
https://wholesale.orange.com/international/en/peering-policy.html

> I am curious if there is a way to see how many have peered already?

Well, if Starlink operates a looking-glass, sure.  Or you can derive an idea, 
albeit an incomplete one, from public data.  If any national communications 
regulators are paying attention, they may have placed a regulatory requirement 
on Starlink to make this public data, though it might have to be dug out of 
regulatory compliance filings, and might not be up-to-date.

> how many they could actually peer with?

That you’d need to script, though many people have.  We have an internal tool 
that tells us that about our own network, and I suspect pretty much every 
network large enough to have a dedicated peering team does likewise.  If you 
were to write such a tool for nonspecific use, we have public datasets that 
would show you who potential peers were at each IXP, and what routes / how many 
addresses they were advertising at each IXP…  Obviously if you’re learning 
Deutche Telekom’s routes in Frankfurt and Munich, it matters somewhat less 
whether you also peer with them in Karlsruhe, assuming they’re advertising the 
same routes everywhere, though it’s still good.  If they’re doing regional 
announcement, you might need to peer with them in different location to 
“collect the whole set” of their routes.  That’s not so common now, though it 
was a fad for a while, maybe fifteen years ago.  I haven’t tried to quantify 
the degree of regional announcement lately… that’s a good small project for a 
student who wants to learn about routing and interconnection.

> And progress over time since inception is there a tool for that?

I think you’d have to throw together your own tools for that, or derive it from 
public data such as the routing archives that we, RIPE, and Route-Views 
maintain.

> Is there a better email list to discuss ixp stuff?

The two I know of are ixp-disc...@pch.net and ixp-disc...@itu.int.  Both are 
pretty quiet, though both have very helpful people on them.

   -Bill




signature.asc
Description: Message signed with OpenPGP

Re: starlink ixp peering progress

[Bill Woodcock]

> On Feb 27, 2024, at 08:54, Dave Taht  wrote:
> One of the things I learned today was that starlink has published an 
> extensive guide as to how existing BGP AS holders can peer with them to get 
> better service.

Yes, essentially every AS does this.  The ones that follow best-practices tend 
to be pretty uniform:

https://pch.net/peering
https://aws.amazon.com/peering/policy/
https://www.bytedance.com/en/peering
https://peering.google.com/#/options/peering
https://openconnect.netflix.com/en/peering/
https://www.lumen.com/en-us/about/legal/peering-policy.html
http://peering.ovh.net/

Starlink’s peering policy is straight-forward and follows best practices.  Then 
there are ones that get a little further afield, some of which can get kinda 
unusual in their fine-print:

https://learn.microsoft.com/en-us/azure/internet-peering/policy
https://www.verizon.com/business/terms/peering/
https://www.zayo.com/wp-content/uploads/2022/07/Zayo-Global-IP-Interconnection-Policy-Final-1.pdf
https://wholesale.orange.com/international/en/peering-policy.html

> I am curious if there is a way to see how many have peered already?

Well, if Starlink operates a looking-glass, sure.  Or you can derive an idea, 
albeit an incomplete one, from public data.  If any national communications 
regulators are paying attention, they may have placed a regulatory requirement 
on Starlink to make this public data, though it might have to be dug out of 
regulatory compliance filings, and might not be up-to-date.

> how many they could actually peer with?

That you’d need to script, though many people have.  We have an internal tool 
that tells us that about our own network, and I suspect pretty much every 
network large enough to have a dedicated peering team does likewise.  If you 
were to write such a tool for nonspecific use, we have public datasets that 
would show you who potential peers were at each IXP, and what routes / how many 
addresses they were advertising at each IXP…  Obviously if you’re learning 
Deutche Telekom’s routes in Frankfurt and Munich, it matters somewhat less 
whether you also peer with them in Karlsruhe, assuming they’re advertising the 
same routes everywhere, though it’s still good.  If they’re doing regional 
announcement, you might need to peer with them in different location to 
“collect the whole set” of their routes.  That’s not so common now, though it 
was a fad for a while, maybe fifteen years ago.  I haven’t tried to quantify 
the degree of regional announcement lately… that’s a good small project for a 
student who wants to learn about routing and interconnection.

> And progress over time since inception is there a tool for that?

I think you’d have to throw together your own tools for that, or derive it from 
public data such as the routing archives that we, RIPE, and Route-Views 
maintain.

> Is there a better email list to discuss ixp stuff?

The two I know of are ixp-disc...@pch.net and ixp-disc...@itu.int.  Both are 
pretty quiet, though both have very helpful people on them.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: starlink ixp peering progress

[Warren Kumari]

Or this: https://bgp.tools/as/14593#peers


Personally I find bgp.tools to be the friendliest…

I realize that this thread is turning into an Me too! type
thread, but it does seem useful to share which tools work best for each of
us…

W



On Tue, Feb 27, 2024 at 7:33 AM, Marco Davids  wrote:

> Or this?
>
> https://bgp.he.net/AS14593#_peers6
>
> Op 27/02/2024 om 13:17 schreef b...@uu3.net:
>
> Well, for some basic overview you can use CAIDA AS rank.
>
> You can use it directly, or you may try my (more user friendly) frontend
> for it: http://as-rank.uu3.net/?as=14593
>
> -- Original message --
>
> From: Dave Taht 
> To: NANOG 
> Subject: starlink ixp peering progress
> Date: Tue, 27 Feb 2024 02:54:44 -0500
>
> One of the things I learned today was that starlink has published an
> extensive guide as to how existing BGP AS holders can peer with them to get
> better service.
>
> https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink
>
> I am curious if there is a way to see how many have peered already, how
> many they could actually peer with?, and progress over time since
> inception what would be the right tools for that? This is pretty
> impressive for peering so far:
>
> https://www.peeringdb.com/net/18747
>
> Is there a better email list to discuss ixp stuff?
>
> --
> Marco
>

Re: starlink ixp peering progress

[Mike Hammett]

The best way I've found (and it is indeed rather incomplete) is to have a BGP 
feed going to something like QRator from that AS (or a downstream AS) that then 
performs analytics on the BGP feed. Starlink is unlikely to have BGP customers, 
so that makes it a bit more difficult. 


https://radar.qrator.net/as/14593/ipv4/neighbors/peerings 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "NANOG"  
Sent: Tuesday, February 27, 2024 1:54:44 AM 
Subject: starlink ixp peering progress 

One of the things I learned today was that starlink has published an 
extensive guide as to how existing BGP AS holders can peer with them 
to get better service. 

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink 

I am curious if there is a way to see how many have peered already, 
how many they could actually peer with?, and progress over time since 
inception what would be the right tools for that? This is pretty 
impressive for peering so far: 

https://www.peeringdb.com/net/18747 

Is there a better email list to discuss ixp stuff? 


-- 
https://blog.cerowrt.org/post/2024_predictions/ 
Dave Täht CSO, LibreQos 

Re: starlink ixp peering progress

[Marco Davids (Private) via NANOG]

Or this?

https://bgp.he.net/AS14593#_peers6

Op 27/02/2024 om 13:17 schreef b...@uu3.net:

Well, for some basic overview you can use CAIDA AS rank.

You can use it directly, or you may try my (more user friendly)
frontend for it: http://as-rank.uu3.net/?as=14593


-- Original message --

From: Dave Taht 
To: NANOG 
Subject: starlink ixp peering progress
Date: Tue, 27 Feb 2024 02:54:44 -0500

One of the things I learned today was that starlink has published an
extensive guide as to how existing BGP AS holders can peer with them
to get better service.

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink

I am curious if there is a way to see how many have peered already,
how many they could actually peer with?, and progress over time since
inception what would be the right tools for that? This is pretty
impressive for peering so far:

https://www.peeringdb.com/net/18747

Is there a better email list to discuss ixp stuff?




--
Marco

Re: starlink ixp peering progress

[borg]

Well, for some basic overview you can use CAIDA AS rank.

You can use it directly, or you may try my (more user friendly)
frontend for it: http://as-rank.uu3.net/?as=14593


-- Original message --

From: Dave Taht 
To: NANOG 
Subject: starlink ixp peering progress
Date: Tue, 27 Feb 2024 02:54:44 -0500

One of the things I learned today was that starlink has published an
extensive guide as to how existing BGP AS holders can peer with them
to get better service.

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink

I am curious if there is a way to see how many have peered already,
how many they could actually peer with?, and progress over time since
inception what would be the right tools for that? This is pretty
impressive for peering so far:

https://www.peeringdb.com/net/18747

Is there a better email list to discuss ixp stuff?


-- 
https://blog.cerowrt.org/post/2024_predictions/
Dave Täht CSO, LibreQos

starlink ixp peering progress

[Dave Taht]

One of the things I learned today was that starlink has published an
extensive guide as to how existing BGP AS holders can peer with them
to get better service.

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink

I am curious if there is a way to see how many have peered already,
how many they could actually peer with?, and progress over time since
inception what would be the right tools for that? This is pretty
impressive for peering so far:

https://www.peeringdb.com/net/18747

Is there a better email list to discuss ixp stuff?


-- 
https://blog.cerowrt.org/post/2024_predictions/
Dave Täht CSO, LibreQos

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Nick,

Thanks for your remarks. It's actually an ongoing discussion.

Le 18/01/2024 à 18:24, Nick Hilliard a écrit :
two issues here: the smaller issue is that CDNs sometimes want their own 
routable IP address blocks, especially if they're connecting directly to 
the IXP, which usually means /24 in practice. It doesn't always happen, 
and sometimes the CDN is happy to use provider address space (i.e. IXP), 
or smaller address blocks. But it's something to note.


I'd rather have CDN use some of their anycast /24 to peer with the IX, 
with a back-end connectivity for their control-plane and back-feeding.


The bigger issue is: who pays the transit costs for the CDN's cache-fill 
requirements? CDNs typically won't pay for cache-fill for installations 
like this, and if one local ISP is pulling disproportionate quantities 
of data compared to other ISPs at the IXP, then this can cause problems 
unless there's an shared billing mechanism built in.


We're willing to provide a dedicated LAN, with routed access, to fill 
caches and administer the machines. It would be fully dissociated from 
the IXP though, unless we could find a way to make it work and as to 
meet extra requirements upon redundancy.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Nick Hilliard]

Jérôme Nicolle wrote on 18/01/2024 14:38:
Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs.


two issues here: the smaller issue is that CDNs sometimes want their own 
routable IP address blocks, especially if they're connecting directly to 
the IXP, which usually means /24 in practice. It doesn't always happen, 
and sometimes the CDN is happy to use provider address space (i.e. IXP), 
or smaller address blocks. But it's something to note.


The bigger issue is: who pays the transit costs for the CDN's cache-fill 
requirements? CDNs typically won't pay for cache-fill for installations 
like this, and if one local ISP is pulling disproportionate quantities 
of data compared to other ISPs at the IXP, then this can cause problems 
unless there's an shared billing mechanism built in.


Nick

Re: Shared cache servers on an island's IXP

[Gael Hernandez]

Hosting authoritative and recursive dns servers at the IXP would
drastically improve the experience of users most of the time.

Of course, Stephane considerations are correct and there’s no solution for
when global connectivity is lost and responses will stop being sent.

Gaël


On Thu 18 Jan 2024 at 14:42, Jérôme Nicolle  wrote:

> Hi Gael,
>
> Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
> > Friends from PCH (www.pch.net <http://www.pch.net>) operate backend
> > services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They
> > would be very happy to help.
>
> I'm sure they would, I'm a big fan of their work BTW. Though hosting
> them in a densely connected area isn't the same as it will in remote
> locations, I guess there could be some work to be done to get it running
> properly, as Stephane wrote.
>
> How would you think we could work on that ? I mean, disconnected or
> extremely high latency scenarii should be on a research roadmap by
> SpaceX' standards, right ? ;-)
>
> Best regards,
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Tom,

Le 18/01/2024 à 15:20, Tom Beecher a écrit :
Many CDNs have hardware options for self hosted caches. I think there 
would likely be concerns about <20G of connectivity to those caches 
though. With an incorrect setup, you could end up maxing out those links 
just with cache fill traffic itself.


In a case where these servers are on a dedicated network peering with 
the ISPs, I think it would be safe to throttle the sync feeds to not 
saturate actual uplinks.


At least, that we can do, but throttling uncached content to customers 
is forbidden (net neutrality).


Though Netflix is supposedly sending pre-loaded servers, and I think 
that - in this location - it's gonna mean a lot already. The quastion is 
: how would the servers peer with local ISPs.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Mike Hammett]

Some will work directly on the IX via BGP. Others have to go behind a member of 
the IX. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Jérôme Nicolle"  
To: "Mehmet"  
Cc: nanog@nanog.org 
Sent: Thursday, January 18, 2024 8:38:31 AM 
Subject: Re: Shared cache servers on an island's IXP 

Hello Mehmet, 

Le 18/01/2024 à 12:58, Mehmet a écrit : 
> VMs are no go for big content companies except Microsoft. You can run 
> Microsoft CDN on VM but rest of the content will need to be cached. You 
> can actually install this yourself 

I've already read most docs for caching servers provided by major 
actors. What I'm mostly concerned about is their ability to peer with 
multiple AS on the local IXP, as to not over-replicate them. 

Should I establish a dedicated network peering on the IXP ? Or will they 
come with their own ASNs ? The peering case is quite not documented on 
publicly available specs, if even possible. 

> Depending on how much traffic do you have , you may be able to get 
> facebook, youtube, netflix caches i think minimum bw requirement changes 
> per region 

Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs. 

Best regards, 

-- 
Jérôme Nicolle 
+33 6 19 31 27 14 

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hi Gael,

Le 18/01/2024 à 13:48, Gael Hernandez a écrit :
Friends from PCH (www.pch.net ) operate backend 
services for DNS authoritative ccTLDs and the Quad9 DNS resolver. They 
would be very happy to help.


I'm sure they would, I'm a big fan of their work BTW. Though hosting 
them in a densely connected area isn't the same as it will in remote 
locations, I guess there could be some work to be done to get it running 
properly, as Stephane wrote.


How would you think we could work on that ? I mean, disconnected or 
extremely high latency scenarii should be on a research roadmap by 
SpaceX' standards, right ? ;-)


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hello Mehmet,

Le 18/01/2024 à 12:58, Mehmet a écrit :
VMs are no go for big content companies except Microsoft. You can run 
Microsoft CDN on VM but rest of the content will need to be cached. You 
can actually install this yourself


I've already read most docs for caching servers provided by major 
actors. What I'm mostly concerned about is their ability to peer with 
multiple AS on the local IXP, as to not over-replicate them.


Should I establish a dedicated network peering on the IXP ? Or will they 
come with their own ASNs ? The peering case is quite not documented on 
publicly available specs, if even possible.


Depending on how much traffic do you have , you may be able to get 
facebook, youtube, netflix caches i think minimum bw requirement changes 
per region


Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs.


Best regards,

--
Jérôme Nicolle
+33 6 19 31 27 14

Re: Shared cache servers on an island's IXP

[Tom Beecher]

Many CDNs have hardware options for self hosted caches. I think there would
likely be concerns about <20G of connectivity to those caches though. With
an incorrect setup, you could end up maxing out those links just with cache
fill traffic itself.


On Thu, Jan 18, 2024 at 6:54 AM Jérôme Nicolle  wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Re: Shared cache servers on an island's IXP

[Stephane Bortzmeyer]

On Thu, Jan 18, 2024 at 12:53:19PM +0100,
 Jérôme Nicolle  wrote 
 a message of 36 lines which said:

> - Low redundancy of old cables (2)
> - Total service loss when both cables are down because of congestion on
> satelite backups

A problem which is not often mentioned is that most (all?) "local
caches" (CDN, DNS resolvers, etc) do not have an "offline mode" (or
"disconnected-from-master mode"). During an outage, they continue to
work for some time then break suddenly, in a not-friendly way, serving
various error messages instead of old data and/or useful
messages. (For instance, the DNS resolver may not be able to serve
stale answers.)

The time during which they can continue to work when they are
disconnected from their master is typically undocumented (except for
the DNS), and discovered only when there is a long outage.

Making the Internet work better with sometimes-broken connectivity is
still an area of research.

Re: Shared cache servers on an island's IXP

[Mehmet]

Hi Jérôme

VMs are no go for big content companies except Microsoft. You can run
Microsoft CDN on VM but rest of the content will need to be cached. You can
actually install this yourself

Depending on how much traffic do you have , you may be able to get
facebook, youtube, netflix caches i think minimum bw requirement changes
per region

Good luck

On Thu, Jan 18, 2024 at 06:53 Jérôme Nicolle  wrote:

> Hello,
>
> I'm trying to find out the best way to consolidate connectivity on an
> island.
>
> The current issues are :
> - Low redundancy of old cables (2)
> - Low system capacity of said cables (<=20Gbps)
> - Total service loss when both cables are down because of congestion on
> satelite backups
> - Sheer price of bandwidth
>
> On the plus side, there are over 5 AS on the island, an IXP and
> small-ish collocation capacity (approx 10kW available, could be
> upgraded, second site available later this year).
>
> We'd like to host cache servers and/or VMs on the IXP, with an option to
> anycast many services - without hijacking them, that goes without saying
> - such as quad-whatever DNS resolvers, NTP servers and whatever else
> could be useful for weather-induced disaster-recovery and/or offload
> cables.
>
> Do you think most CDNs, stream services and CSPs could accommodate a
> scenario where we'd host their gear or provide VMs for them to announce
> on the local route-servers ? If not, what could be a reasonable
> technical arrangement ?
>
> Thanks !
>
> --
> Jérôme Nicolle
> +33 6 19 31 27 14
>

Shared cache servers on an island's IXP

[Jérôme Nicolle]

Hello,

I'm trying to find out the best way to consolidate connectivity on an 
island.


The current issues are :
- Low redundancy of old cables (2)
- Low system capacity of said cables (<=20Gbps)
- Total service loss when both cables are down because of congestion on 
satelite backups

- Sheer price of bandwidth

On the plus side, there are over 5 AS on the island, an IXP and 
small-ish collocation capacity (approx 10kW available, could be 
upgraded, second site available later this year).


We'd like to host cache servers and/or VMs on the IXP, with an option to 
anycast many services - without hijacking them, that goes without saying 
- such as quad-whatever DNS resolvers, NTP servers and whatever else 
could be useful for weather-induced disaster-recovery and/or offload cables.


Do you think most CDNs, stream services and CSPs could accommodate a 
scenario where we'd host their gear or provide VMs for them to announce 
on the local route-servers ? If not, what could be a reasonable 
technical arrangement ?


Thanks !

--
Jérôme Nicolle
+33 6 19 31 27 14

RE: Any experiences using SIIT-DC in an IXP setting ?

[Vasilenko Eduard via NANOG]

As I understand the initial question: the client has no IPv4.
Initial “4” in 464XLAT means IPv4 client.

DNS64 could mislead the client that the server (on the internet) is available 
on IPv6.
Then NAT64 would convert IPv6 to IPv4.
But it is not stateless by any means (requested below).

Ed/
From: Ca By [mailto:cb.li...@gmail.com]
Sent: Monday, October 10, 2022 7:27 PM
To: Vasilenko Eduard 
Cc: Carlos Martinez-Cagnazzo ; NANOG 
Subject: Re: Any experiences using SIIT-DC in an IXP setting ?



On Mon, Oct 10, 2022 at 9:17 AM Vasilenko Eduard via NANOG 
mailto:nanog@nanog.org>> wrote:
The technology for IPv6 client to connect IPv4 web server on Internet is just 
not specified in IETF.
Ed/
Ed, you seem to be not so familiar with the this ietf body of work

RFC6877

“ 464XLAT is a simple and scalable

   technique to quickly deploy limited IPv4 access service to IPv6-only

   edge networks without encapsulation.

”

To the OP, you can google jpix has done this.



From: NANOG 
[mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei@nanog.org<mailto:huawei@nanog.org>]
 On Behalf Of Carlos Martinez-Cagnazzo
Sent: Monday, October 10, 2022 6:57 PM
To: NANOG mailto:nanog@nanog.org>>
Subject: Any experiences using SIIT-DC in an IXP setting ?

Hi all,

I'm looking at a use case for stateless 6-4 mappings in the context of an IXP.

The problem we are looking to solve is allowing IXP members who have no IPv4 of 
their own and in most cases they have a /26 or /27 issued by their transit 
provider and rely on CGN to provide service to their customers. They do have 
their own AS numbers and IPv6 prefixes though.

Any comments are appreciated. PM is fine too.

Thanks!

/Carlos

Re: Any experiences using SIIT-DC in an IXP setting ?

[Ca By]

On Mon, Oct 10, 2022 at 9:17 AM Vasilenko Eduard via NANOG 
wrote:

> The technology for IPv6 client to connect IPv4 web server on Internet is
> just not specified in IETF.
>
> Ed/
>
Ed, you seem to be not so familiar with the this ietf body of work

RFC6877

“ 464XLAT is a simple and scalable

   technique to quickly deploy limited IPv4 access service to IPv6-only
   edge networks without encapsulation.


”

To the OP, you can google jpix has done this.



*From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] *On
Behalf Of *Carlos Martinez-Cagnazzo

> *Sent:* Monday, October 10, 2022 6:57 PM
> *To:* NANOG 
> *Subject:* Any experiences using SIIT-DC in an IXP setting ?
>
>
>
> Hi all,
>
>
>
> I'm looking at a use case for stateless 6-4 mappings in the context of an
> IXP.
>
>
>
> The problem we are looking to solve is allowing IXP members who have no
> IPv4 of their own and in most cases they have a /26 or /27 issued by their
> transit provider and rely on CGN to provide service to their customers.
> They do have their own AS numbers and IPv6 prefixes though.
>
>
>
> Any comments are appreciated. PM is fine too.
>
>
>
> Thanks!
>
>
>
> /Carlos
>
>
>

RE: Any experiences using SIIT-DC in an IXP setting ?

[Vasilenko Eduard via NANOG]

The technology for IPv6 client to connect IPv4 web server on Internet is just 
not specified in IETF.
Ed/
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Carlos Martinez-Cagnazzo
Sent: Monday, October 10, 2022 6:57 PM
To: NANOG 
Subject: Any experiences using SIIT-DC in an IXP setting ?

Hi all,

I'm looking at a use case for stateless 6-4 mappings in the context of an IXP.

The problem we are looking to solve is allowing IXP members who have no IPv4 of 
their own and in most cases they have a /26 or /27 issued by their transit 
provider and rely on CGN to provide service to their customers. They do have 
their own AS numbers and IPv6 prefixes though.

Any comments are appreciated. PM is fine too.

Thanks!

/Carlos

Any experiences using SIIT-DC in an IXP setting ?

[Carlos Martinez-Cagnazzo]

Hi all,

I'm looking at a use case for stateless 6-4 mappings in the context of an
IXP.

The problem we are looking to solve is allowing IXP members who have no
IPv4 of their own and in most cases they have a /26 or /27 issued by their
transit provider and rely on CGN to provide service to their customers.
They do have their own AS numbers and IPv6 prefixes though.

Any comments are appreciated. PM is fine too.

Thanks!

/Carlos

Re: Equinix Dallas IXP ? Down ?

[Faisal Imtiaz]

Thank you ..looks like a localized issue only affecting out port.


Regards,

Faisal Imtiaz
fai...@snappytelecom.net
Tel: 305-663-5518 ext 232

(sent from mobile device)


From: Kaiser, Erich 
Sent: Thursday, January 23, 2020 9:59:21 AM
To: Tom Beecher 
Cc: Faisal Imtiaz ; nanog@nanog.org 
Subject: Re: Equinix Dallas IXP ? Down ?

No issues here.

Erich Kaiser
The Fusion Network



On Thu, Jan 23, 2020 at 8:48 AM Tom Beecher  wrote:
I see no issues on 2 separate Equinix Dallas connections.

On Thu, Jan 23, 2020 at 9:16 AM Faisal Imtiaz 
mailto:fai...@snappytelecom.net>> wrote:
Hello,
Quick question, is there known issue with Equinix Dallas IXP ?
(Or it is just our connection ?  Seeing all peers down).

Thanks.
Regards.

Faisal Imtiaz
Snappy Internet & Telecom
http://www.snappytelecom.net

Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: 
supp...@snappytelecom.net<mailto:supp...@snappytelecom.net<mailto:supp...@snappytelecom.net>>

Re: Equinix Dallas IXP ? Down ?

[Kaiser, Erich]

No issues here.

Erich Kaiser
The Fusion Network



On Thu, Jan 23, 2020 at 8:48 AM Tom Beecher  wrote:

> I see no issues on 2 separate Equinix Dallas connections.
>
> On Thu, Jan 23, 2020 at 9:16 AM Faisal Imtiaz 
> wrote:
>
>> Hello,
>> Quick question, is there known issue with Equinix Dallas IXP ?
>> (Or it is just our connection ?  Seeing all peers down).
>>
>> Thanks.
>> Regards.
>>
>> Faisal Imtiaz
>> Snappy Internet & Telecom
>> http://www.snappytelecom.net
>>
>> Tel: 305 663 5518 x 232
>>
>> Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net
>> <mailto:supp...@snappytelecom.net>
>>
>>

Re: Equinix Dallas IXP ? Down ?

[Tom Beecher]

I see no issues on 2 separate Equinix Dallas connections.

On Thu, Jan 23, 2020 at 9:16 AM Faisal Imtiaz 
wrote:

> Hello,
> Quick question, is there known issue with Equinix Dallas IXP ?
> (Or it is just our connection ?  Seeing all peers down).
>
> Thanks.
> Regards.
>
> Faisal Imtiaz
> Snappy Internet & Telecom
> http://www.snappytelecom.net
>
> Tel: 305 663 5518 x 232
>
> Help-desk: (305)663-5518 Option 2 or Email: supp...@snappytelecom.net
> <mailto:supp...@snappytelecom.net>
>
>

Equinix Dallas IXP ? Down ?

[Faisal Imtiaz]

Hello,
Quick question, is there known issue with Equinix Dallas IXP ?
(Or it is just our connection ?  Seeing all peers down).

Thanks.
Regards.

Faisal Imtiaz
Snappy Internet & Telecom
http://www.snappytelecom.net

Tel: 305 663 5518 x 232

Help-desk: (305)663-5518 Option 2 or Email: 
supp...@snappytelecom.net<mailto:supp...@snappytelecom.net>

IXP Renumbering: Equinix Miami (formerly known as NOTA)

[Fredy Künzler]

It seems that many participants of the Equinix Miami Exchange (formerly
Nap of the Americas NOTA) are unaware of the required IP renumbering.
Init7 AS13030 sees at least 50% of the new BGP sessions still down.

While the usefulness of the replacement of the peering mesh /23 with
another /23 could be discussed, the process itself hasn't been
announced, at least we didn't became aware of it until recently, and
only because some peers asked for new sessions.

Equinix provided a website with the necessary information, but it seems
that they failed to announce it. Interestingly the schedule is already
overdue.

https://ix.equinix.com/home/ip-migration/mi/

I would propose that some IXP operators should write a BCOP to avoid
future renumbering pain.

-- 
Fredy Künzler

Init7 (Switzerland) Ltd.
Technoparkstrasse 5
CH-8406 Winterthur
https://www.init7.net/

USA to Mexico IXP Equipment Recommendations

[NJ]

Advice and/or suggestions wanted. Any input greatly appreciated.

*Scenario:*
We have run fiber from the USA to Mexico and have an exchange point in
place. Our fiber connections are dark and therefore we can use any
configuration we want on as many pairs as we want. Currently we have decided
to light two 100gig connections and they are piping across the border
already. We have 200gig going to Los Angeles (Wilshire Adjacent) and
are looking to the mexico interested community of network engineers to
identify how our peering point in Tijuana can be made to best service
the networks in the region (and abroad if they build in). Our IXP is
good, but we want to update/upgrade and put in a future-compatible
robust solution instead of what we have in there now. We are an open
IXP and are wondering how you would prefer to peer and what your
equipment recommendations are and why.  We're not afraid to spend
(some) money but we do want it to not be complete overkill however
should last at least 5 to 10 years before any upgrade would be
required. Also to note, we are running our own bandwidth over the
pipes as well so we would like to keep them separated.


*Norm*
A.A.S. R.A. Lifetime Member
*Archaeology, Astronautics and SETI Research Association*
SETI Project Lifetime Contributor

MANRS IXP Webinar: Tuesday, 13 March

[Chris Grundemann]

Hail NANOGers!

If you operate an IX in North America, this message is for you.

(I'm passing it along on behalf of my former colleagues at the Internet
Society.)

Hope to "see" you on the webinar this Tuesday!


———
Hi,

The MANRS IXP Partnership program is designed to invite and encourage
participation of IXPs in MANRS (Mutually Agreed Norms for Routing
Security). In accordance with the principles of the initiative, the
membership depends on the visible commitment to improve the routing
security of the peering fabric and Internet infrastructure in general and
demonstrated by the implementation of defined Actions.

The Internet Society is looking for leading IXPs around the world that have
a track record in contributing to routing security, already implement the
majority of Actions and are willing to participate in the launch of this
program, planned in early 2018. The set of Actions was developed by the
development team consisting of IXP representatives around the world. It has
been reviewed in by various IXP communities, such as EURO-IX, LAC-IX, Af-IX.

I’d like to invite you to participate in a webinar on March 13 at 14:00 EDT
(18:00 UTC) where we present the IXP Partnership Program and the Actions to
the North-American IXP community and invite your feedback. Call details are
below. We would also ask you to disseminate the information about the
webinar to the NA IX community.

MANRS (https://www.manrs.org) is a collaborative initiative, coordinated by
the Internet Society. It was launched in November 2014 and has received
much encouragement from all sections of the Internet industry. The key
objective of MANRS is to gain industry-wide agreement on a minimum set of
practices for secure routing across the Internet, through coordinated
action by many parties.

MANRS was designed for network operators, but other parties can play an
important role in facilitating improvements in routing security such as
Internet Exchange Points (IXPs). Many of them represent active communities
with common operational objectives and already contribute to a more
resilient and secure Internet infrastructure.


Thanks!

Mark Buell
Regional Bureau Director, North America
Internet Society





Details:
Topic: MANRS and IXPs
Time: Mar 13, 2018 7:00 PM Amsterdam, Berlin, Rome, Stockholm, Vienna

Join from PC, Mac, Linux, iOS or Android: https://isoc.zoom.us/j/288357813

Or iPhone one-tap :
US: +16699006833,,288357813#  or +16465588656,,288357813#
Or Telephone:
Dial(for higher quality, dial a number based on your current location):
US: +1 669 900 6833  or +1 646 558 8656
Meeting ID: 288 357 813
International numbers available: https://isoc.zoom.
us/zoomconference?m=WXysPTqEpKm5ELZq6evhxxHUX43prdSf




-- 
@ChrisGrundemann
http://chrisgrundemann.com

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Nick W]

1.9.7 definitely applies to Infinity:

ER-8-XG:
https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e1000.v1.9.7+hotfix.1.5005858.tar
(SHA256:b1a16900e3fbe1eef3876548ac7eda12a95ef849d4328f22b478459e2a506b92)



On Tue, Aug 8, 2017 at 9:07 PM, Josh Reynolds <j...@kyneticwifi.com> wrote:

> Forgot reply all...
>
> That does not apply to the infinity. Those shipped with 1.9.8dev.
>
>
> On Aug 8, 2017 8:03 PM, "Mike Hammett" <na...@ics-il.net> wrote:
>
> > 1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released on
> > May 1st.
> >
> > https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-
> > EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
> >
> >
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> > - Original Message -
> >
> > From: "Nick W" <nickdwh...@gmail.com>
> > To: nanog@nanog.org
> > Sent: Thursday, July 20, 2017 10:55:28 PM
> > Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
> >
> > Tried the Infinity, unsuccessfully. Several of them. Ended up pulling
> them
> > all, sitting in my homelab for now. Multiple full tables, nothing fancy
> for
> > firewall or QOS, but ran into issues with random ribd/bgpd crashes and
> > kernel panics. I've submitted a lot of logs and core dumps to UBNT. I
> would
> > personally stay away from them until they are out of beta, and possibly
> > even another 6-12 months after that.
> >
> > The current stable EdgeMax version (1.9.1.1) is relatively stable, but
> > using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF,
> BGP)
> > - nothing too major, but can be annoying. Probably okay for what you
> > described. Depending on how much throughput you need, an ERPro, or
> Mikrotik
> > would probably be fine. If you need 10G, load up VyOS on some cheap
> servers
> > with an Intel or Solarflare card... probably cheaper than a beta Infinity
> > or Mikrotik.
> >
> > On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders <j...@instituut.net> wrote:
> >
> > > Dear NANOG,
> > >
> > > Some friends of mine are operating a nonprofit (on shoe string) and
> > looking
> > > to connect some CDN caches to an IX fabric. A BGP speaking device is
> > needed
> > > between the caches and the BGP peers connected to the fabric. The BGP
> > > speaker is needed to present the peers on the IX with a unified view of
> > the
> > > assemblage of CDN nodes.
> > >
> > > I was wondering whether anyone was experience with the "EdgeRouter
> > Infinity
> > > XG" device, specifically in the role of a simple peering router for a
> > > couple of tens of thousands of routes. (I'd point default to the left
> and
> > > take just the on-net routes on the right to reduce the table size
> > > requirement).
> > >
> > > I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> > > automatable (supports idempotency), can forward IMIX at line-rate,
> *flow,
> > > and exposes some telemetry via SNMP.
> > >
> > > Any note sharing would be appreciated!
> > >
> > > Kind regards,
> > >
> > > Job
> > >
> >
> >
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Brielle Bruns]

On 8/11/2017 10:30 PM, Josh Reynolds wrote:

I'm dumb, Brielle is right.

1.9.0, 1.9.5, 1.9.7h1

1.9.8dev and 1.9.8b1 are for two other newer products.



Ubiquiti has been pretty active in developing improvements lately.

I do recommend anyone who does use the Edge* series in production that 
they join the beta program on the Ubnt forums to keep current on whats 
being worked on.  It pays off in the long run, and gives you a chance to 
test out new features and fixes before deploying them in real world 
scenarios.


The Infinity isn't a perfect fit for everyone, but it is a 'disruptive' 
device and worth a look if you have > 1G needs.  Now that its 'out 
there' in production, hopefully those that can put it to use will be 
able to give out real world numbers.


(Note:  I don't work for Ubnt, but I've used many of their products in 
the Edge and Unifi lines.  I only speak for myself.)



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

I'm dumb, Brielle is right.

1.9.0, 1.9.5, 1.9.7h1

1.9.8dev and 1.9.8b1 are for two other newer products.

On Aug 11, 2017 11:16 PM, "Brielle Bruns"  wrote:

> On 8/11/2017 9:34 AM, Josh Reynolds wrote:
>
>> As an additional note, sometimes drivers get backported, this is how
>> 1.9.7hotfix1 works on Infinity. There are multiple trees in various stages
>> of dev at any given time.
>>
>
>
>
> The Infinity started out on 1.9.0 (which is what my Infinity alpha
> hardware had) and went up from there as the various in between releases
> came up, up until the current 1.9.7 release.
>
> The 1.9.8 dev releases are not currently for the Infinity or any previous
> ER hardware.  That's about all I can say.
>
>
> --
> Brielle Bruns
> The Summit Open Source Development Group
> http://www.sosdg.org/ http://www.ahbl.org
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Brielle Bruns]

On 8/11/2017 9:34 AM, Josh Reynolds wrote:

As an additional note, sometimes drivers get backported, this is how
1.9.7hotfix1 works on Infinity. There are multiple trees in various stages
of dev at any given time.




The Infinity started out on 1.9.0 (which is what my Infinity alpha 
hardware had) and went up from there as the various in between releases 
came up, up until the current 1.9.7 release.


The 1.9.8 dev releases are not currently for the Infinity or any 
previous ER hardware.  That's about all I can say.



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

As an additional note, sometimes drivers get backported, this is how
1.9.7hotfix1 works on Infinity. There are multiple trees in various stages
of dev at any given time.

On Aug 11, 2017 10:29 AM, "Josh Reynolds" <j...@kyneticwifi.com> wrote:

> Since it's been announced now...
>
> I have an alpha unit. It came with 1.9.8dev straight from $manufacturer.
> My box still has markings from customs all over it.
>
> Expect a new version with some minor fixes soon. A lot of firmware work
> going on at the moment on various edgeOS product lines.
>
> On Aug 11, 2017 10:03 AM, "Nick W" <nickdwh...@gmail.com> wrote:
>
>> 1.9.7 definitely applies to Infinity:
>>
>> ER-8-XG:
>> https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e1000.v1.9.7
>> +hotfix.1.5005858.tar
>> (SHA256:b1a16900e3fbe1eef3876548ac7eda12a95ef849d4328f22b478459e2a506b92)
>>
>>
>>
>> On Tue, Aug 8, 2017 at 9:07 PM, Josh Reynolds <j...@kyneticwifi.com>
>> wrote:
>>
>>> Forgot reply all...
>>>
>>> That does not apply to the infinity. Those shipped with 1.9.8dev.
>>>
>>>
>>> On Aug 8, 2017 8:03 PM, "Mike Hammett" <na...@ics-il.net> wrote:
>>>
>>> > 1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released
>>> on
>>> > May 1st.
>>> >
>>> > https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-
>>> > EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
>>> >
>>> >
>>> >
>>> >
>>> > -
>>> > Mike Hammett
>>> > Intelligent Computing Solutions
>>> > http://www.ics-il.com
>>> >
>>> > Midwest-IX
>>> > http://www.midwest-ix.com
>>> >
>>> > - Original Message -
>>> >
>>> > From: "Nick W" <nickdwh...@gmail.com>
>>> > To: nanog@nanog.org
>>> > Sent: Thursday, July 20, 2017 10:55:28 PM
>>> > Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
>>> >
>>> > Tried the Infinity, unsuccessfully. Several of them. Ended up pulling
>>> them
>>> > all, sitting in my homelab for now. Multiple full tables, nothing
>>> fancy for
>>> > firewall or QOS, but ran into issues with random ribd/bgpd crashes and
>>> > kernel panics. I've submitted a lot of logs and core dumps to UBNT. I
>>> would
>>> > personally stay away from them until they are out of beta, and possibly
>>> > even another 6-12 months after that.
>>> >
>>> > The current stable EdgeMax version (1.9.1.1) is relatively stable, but
>>> > using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF,
>>> BGP)
>>> > - nothing too major, but can be annoying. Probably okay for what you
>>> > described. Depending on how much throughput you need, an ERPro, or
>>> Mikrotik
>>> > would probably be fine. If you need 10G, load up VyOS on some cheap
>>> servers
>>> > with an Intel or Solarflare card... probably cheaper than a beta
>>> Infinity
>>> > or Mikrotik.
>>> >
>>> > On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders <j...@instituut.net>
>>> wrote:
>>> >
>>> > > Dear NANOG,
>>> > >
>>> > > Some friends of mine are operating a nonprofit (on shoe string) and
>>> > looking
>>> > > to connect some CDN caches to an IX fabric. A BGP speaking device is
>>> > needed
>>> > > between the caches and the BGP peers connected to the fabric. The BGP
>>> > > speaker is needed to present the peers on the IX with a unified view
>>> of
>>> > the
>>> > > assemblage of CDN nodes.
>>> > >
>>> > > I was wondering whether anyone was experience with the "EdgeRouter
>>> > Infinity
>>> > > XG" device, specifically in the role of a simple peering router for a
>>> > > couple of tens of thousands of routes. (I'd point default to the
>>> left and
>>> > > take just the on-net routes on the right to reduce the table size
>>> > > requirement).
>>> > >
>>> > > I hope the device can do at least 2xLACP trunks, has a sizable FIB,
>>> is
>>> > > automatable (supports idempotency), can forward IMIX at line-rate,
>>> *flow,
>>> > > and exposes some telemetry via SNMP.
>>> > >
>>> > > Any note sharing would be appreciated!
>>> > >
>>> > > Kind regards,
>>> > >
>>> > > Job
>>> > >
>>> >
>>> >
>>>
>>
>>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

Since it's been announced now...

I have an alpha unit. It came with 1.9.8dev straight from $manufacturer. My
box still has markings from customs all over it.

Expect a new version with some minor fixes soon. A lot of firmware work
going on at the moment on various edgeOS product lines.

On Aug 11, 2017 10:03 AM, "Nick W" <nickdwh...@gmail.com> wrote:

> 1.9.7 definitely applies to Infinity:
>
> ER-8-XG:
> https://dl.ubnt.com/firmwares/edgemax/v1.9.7/ER-e1000.v1.9.
> 7+hotfix.1.5005858.tar
> (SHA256:b1a16900e3fbe1eef3876548ac7eda12a95ef849d4328f22b478459e2a506b92)
>
>
>
> On Tue, Aug 8, 2017 at 9:07 PM, Josh Reynolds <j...@kyneticwifi.com>
> wrote:
>
>> Forgot reply all...
>>
>> That does not apply to the infinity. Those shipped with 1.9.8dev.
>>
>>
>> On Aug 8, 2017 8:03 PM, "Mike Hammett" <na...@ics-il.net> wrote:
>>
>> > 1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released
>> on
>> > May 1st.
>> >
>> > https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-
>> > EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
>> >
>> >
>> >
>> >
>> > -
>> > Mike Hammett
>> > Intelligent Computing Solutions
>> > http://www.ics-il.com
>> >
>> > Midwest-IX
>> > http://www.midwest-ix.com
>> >
>> > - Original Message -
>> >
>> > From: "Nick W" <nickdwh...@gmail.com>
>> > To: nanog@nanog.org
>> > Sent: Thursday, July 20, 2017 10:55:28 PM
>> > Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
>> >
>> > Tried the Infinity, unsuccessfully. Several of them. Ended up pulling
>> them
>> > all, sitting in my homelab for now. Multiple full tables, nothing fancy
>> for
>> > firewall or QOS, but ran into issues with random ribd/bgpd crashes and
>> > kernel panics. I've submitted a lot of logs and core dumps to UBNT. I
>> would
>> > personally stay away from them until they are out of beta, and possibly
>> > even another 6-12 months after that.
>> >
>> > The current stable EdgeMax version (1.9.1.1) is relatively stable, but
>> > using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF,
>> BGP)
>> > - nothing too major, but can be annoying. Probably okay for what you
>> > described. Depending on how much throughput you need, an ERPro, or
>> Mikrotik
>> > would probably be fine. If you need 10G, load up VyOS on some cheap
>> servers
>> > with an Intel or Solarflare card... probably cheaper than a beta
>> Infinity
>> > or Mikrotik.
>> >
>> > On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders <j...@instituut.net> wrote:
>> >
>> > > Dear NANOG,
>> > >
>> > > Some friends of mine are operating a nonprofit (on shoe string) and
>> > looking
>> > > to connect some CDN caches to an IX fabric. A BGP speaking device is
>> > needed
>> > > between the caches and the BGP peers connected to the fabric. The BGP
>> > > speaker is needed to present the peers on the IX with a unified view
>> of
>> > the
>> > > assemblage of CDN nodes.
>> > >
>> > > I was wondering whether anyone was experience with the "EdgeRouter
>> > Infinity
>> > > XG" device, specifically in the role of a simple peering router for a
>> > > couple of tens of thousands of routes. (I'd point default to the left
>> and
>> > > take just the on-net routes on the right to reduce the table size
>> > > requirement).
>> > >
>> > > I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
>> > > automatable (supports idempotency), can forward IMIX at line-rate,
>> *flow,
>> > > and exposes some telemetry via SNMP.
>> > >
>> > > Any note sharing would be appreciated!
>> > >
>> > > Kind regards,
>> > >
>> > > Job
>> > >
>> >
>> >
>>
>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Jeff Waddell]

When I lasted checked in with Ubiquiti on these issues for that and the
ER-Pros - they told me that everything was to be resolved in 2.0

We shall see...

On Tue, Aug 8, 2017 at 9:20 PM, Mike Hammett <na...@ics-il.net> wrote:

> Ah, okay. I haven't used one yet.
>
> Also, I don't talk about beta outside of beta. ;-)
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> - Original Message -
>
> From: "Josh Reynolds" <j...@kyneticwifi.com>
> To: "Mike Hammett" <na...@ics-il.net>
> Cc: "NANOG" <nanog@nanog.org>
> Sent: Tuesday, August 8, 2017 8:07:36 PM
> Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
>
>
> Forgot reply all...
>
>
> That does not apply to the infinity. Those shipped with 1.9.8dev.
>
>
>
>
>
> On Aug 8, 2017 8:03 PM, "Mike Hammett" < na...@ics-il.net > wrote:
>
>
> 1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released on
> May 1st.
>
> https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-
> EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> - Original Message -
>
> From: "Nick W" < nickdwh...@gmail.com >
> To: nanog@nanog.org
> Sent: Thursday, July 20, 2017 10:55:28 PM
> Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
>
> Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them
> all, sitting in my homelab for now. Multiple full tables, nothing fancy for
> firewall or QOS, but ran into issues with random ribd/bgpd crashes and
> kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would
> personally stay away from them until they are out of beta, and possibly
> even another 6-12 months after that.
>
> The current stable EdgeMax version (1.9.1.1) is relatively stable, but
> using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP)
> - nothing too major, but can be annoying. Probably okay for what you
> described. Depending on how much throughput you need, an ERPro, or Mikrotik
> would probably be fine. If you need 10G, load up VyOS on some cheap servers
> with an Intel or Solarflare card... probably cheaper than a beta Infinity
> or Mikrotik.
>
> On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders < j...@instituut.net > wrote:
>
> > Dear NANOG,
> >
> > Some friends of mine are operating a nonprofit (on shoe string) and
> looking
> > to connect some CDN caches to an IX fabric. A BGP speaking device is
> needed
> > between the caches and the BGP peers connected to the fabric. The BGP
> > speaker is needed to present the peers on the IX with a unified view of
> the
> > assemblage of CDN nodes.
> >
> > I was wondering whether anyone was experience with the "EdgeRouter
> Infinity
> > XG" device, specifically in the role of a simple peering router for a
> > couple of tens of thousands of routes. (I'd point default to the left and
> > take just the on-net routes on the right to reduce the table size
> > requirement).
> >
> > I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> > automatable (supports idempotency), can forward IMIX at line-rate, *flow,
> > and exposes some telemetry via SNMP.
> >
> > Any note sharing would be appreciated!
> >
> > Kind regards,
> >
> > Job
> >
>
>
>
>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Mike Hammett]

Ah, okay. I haven't used one yet. 

Also, I don't talk about beta outside of beta. ;-) 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Josh Reynolds" <j...@kyneticwifi.com> 
To: "Mike Hammett" <na...@ics-il.net> 
Cc: "NANOG" <nanog@nanog.org> 
Sent: Tuesday, August 8, 2017 8:07:36 PM 
Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"? 


Forgot reply all... 


That does not apply to the infinity. Those shipped with 1.9.8dev. 





On Aug 8, 2017 8:03 PM, "Mike Hammett" < na...@ics-il.net > wrote: 


1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released on May 
1st. 

https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message - 

From: "Nick W" < nickdwh...@gmail.com > 
To: nanog@nanog.org 
Sent: Thursday, July 20, 2017 10:55:28 PM 
Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"? 

Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them 
all, sitting in my homelab for now. Multiple full tables, nothing fancy for 
firewall or QOS, but ran into issues with random ribd/bgpd crashes and 
kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would 
personally stay away from them until they are out of beta, and possibly 
even another 6-12 months after that. 

The current stable EdgeMax version (1.9.1.1) is relatively stable, but 
using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP) 
- nothing too major, but can be annoying. Probably okay for what you 
described. Depending on how much throughput you need, an ERPro, or Mikrotik 
would probably be fine. If you need 10G, load up VyOS on some cheap servers 
with an Intel or Solarflare card... probably cheaper than a beta Infinity 
or Mikrotik. 

On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders < j...@instituut.net > wrote: 

> Dear NANOG, 
> 
> Some friends of mine are operating a nonprofit (on shoe string) and looking 
> to connect some CDN caches to an IX fabric. A BGP speaking device is needed 
> between the caches and the BGP peers connected to the fabric. The BGP 
> speaker is needed to present the peers on the IX with a unified view of the 
> assemblage of CDN nodes. 
> 
> I was wondering whether anyone was experience with the "EdgeRouter Infinity 
> XG" device, specifically in the role of a simple peering router for a 
> couple of tens of thousands of routes. (I'd point default to the left and 
> take just the on-net routes on the right to reduce the table size 
> requirement). 
> 
> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is 
> automatable (supports idempotency), can forward IMIX at line-rate, *flow, 
> and exposes some telemetry via SNMP. 
> 
> Any note sharing would be appreciated! 
> 
> Kind regards, 
> 
> Job 
> 

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

Forgot reply all...

That does not apply to the infinity. Those shipped with 1.9.8dev.


On Aug 8, 2017 8:03 PM, "Mike Hammett" <na...@ics-il.net> wrote:

> 1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released on
> May 1st.
>
> https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-
> EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
>
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> - Original Message -
>
> From: "Nick W" <nickdwh...@gmail.com>
> To: nanog@nanog.org
> Sent: Thursday, July 20, 2017 10:55:28 PM
> Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?
>
> Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them
> all, sitting in my homelab for now. Multiple full tables, nothing fancy for
> firewall or QOS, but ran into issues with random ribd/bgpd crashes and
> kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would
> personally stay away from them until they are out of beta, and possibly
> even another 6-12 months after that.
>
> The current stable EdgeMax version (1.9.1.1) is relatively stable, but
> using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP)
> - nothing too major, but can be annoying. Probably okay for what you
> described. Depending on how much throughput you need, an ERPro, or Mikrotik
> would probably be fine. If you need 10G, load up VyOS on some cheap servers
> with an Intel or Solarflare card... probably cheaper than a beta Infinity
> or Mikrotik.
>
> On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders <j...@instituut.net> wrote:
>
> > Dear NANOG,
> >
> > Some friends of mine are operating a nonprofit (on shoe string) and
> looking
> > to connect some CDN caches to an IX fabric. A BGP speaking device is
> needed
> > between the caches and the BGP peers connected to the fabric. The BGP
> > speaker is needed to present the peers on the IX with a unified view of
> the
> > assemblage of CDN nodes.
> >
> > I was wondering whether anyone was experience with the "EdgeRouter
> Infinity
> > XG" device, specifically in the role of a simple peering router for a
> > couple of tens of thousands of routes. (I'd point default to the left and
> > take just the on-net routes on the right to reduce the table size
> > requirement).
> >
> > I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> > automatable (supports idempotency), can forward IMIX at line-rate, *flow,
> > and exposes some telemetry via SNMP.
> >
> > Any note sharing would be appreciated!
> >
> > Kind regards,
> >
> > Job
> >
>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Mike Hammett]

1.9.7+hotfix.1 is the currently available stable. 1.9.1.1 was released on May 
1st. 

https://community.ubnt.com/t5/EdgeMAX-Updates-Blog/EdgeMAX-EdgeRouter-software-security-release-v1-9-7-hotfix-1/ba-p/2019161
 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Nick W" <nickdwh...@gmail.com> 
To: nanog@nanog.org 
Sent: Thursday, July 20, 2017 10:55:28 PM 
Subject: Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"? 

Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them 
all, sitting in my homelab for now. Multiple full tables, nothing fancy for 
firewall or QOS, but ran into issues with random ribd/bgpd crashes and 
kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would 
personally stay away from them until they are out of beta, and possibly 
even another 6-12 months after that. 

The current stable EdgeMax version (1.9.1.1) is relatively stable, but 
using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP) 
- nothing too major, but can be annoying. Probably okay for what you 
described. Depending on how much throughput you need, an ERPro, or Mikrotik 
would probably be fine. If you need 10G, load up VyOS on some cheap servers 
with an Intel or Solarflare card... probably cheaper than a beta Infinity 
or Mikrotik. 

On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders <j...@instituut.net> wrote: 

> Dear NANOG, 
> 
> Some friends of mine are operating a nonprofit (on shoe string) and looking 
> to connect some CDN caches to an IX fabric. A BGP speaking device is needed 
> between the caches and the BGP peers connected to the fabric. The BGP 
> speaker is needed to present the peers on the IX with a unified view of the 
> assemblage of CDN nodes. 
> 
> I was wondering whether anyone was experience with the "EdgeRouter Infinity 
> XG" device, specifically in the role of a simple peering router for a 
> couple of tens of thousands of routes. (I'd point default to the left and 
> take just the on-net routes on the right to reduce the table size 
> requirement). 
> 
> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is 
> automatable (supports idempotency), can forward IMIX at line-rate, *flow, 
> and exposes some telemetry via SNMP. 
> 
> Any note sharing would be appreciated! 
> 
> Kind regards, 
> 
> Job 
> 

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Nick W]

Tried the Infinity, unsuccessfully. Several of them. Ended up pulling them
all, sitting in my homelab for now. Multiple full tables, nothing fancy for
firewall or QOS, but ran into issues with random ribd/bgpd crashes and
kernel panics. I've submitted a lot of logs and core dumps to UBNT. I would
personally stay away from them until they are out of beta, and possibly
even another 6-12 months after that.

The current stable EdgeMax version (1.9.1.1) is relatively stable, but
using an outdated ZebOS (1.2.0?) with a number of issues (MPLS, OSPF, BGP)
- nothing too major, but can be annoying. Probably okay for what you
described. Depending on how much throughput you need, an ERPro, or Mikrotik
would probably be fine. If you need 10G, load up VyOS on some cheap servers
with an Intel or Solarflare card... probably cheaper than a beta Infinity
or Mikrotik.

On Mon, Jul 3, 2017 at 3:07 PM, Job Snijders  wrote:

> Dear NANOG,
>
> Some friends of mine are operating a nonprofit (on shoe string) and looking
> to connect some CDN caches to an IX fabric. A BGP speaking device is needed
> between the caches and the BGP peers connected to the fabric. The BGP
> speaker is needed to present the peers on the IX with a unified view of the
> assemblage of CDN nodes.
>
> I was wondering whether anyone was experience with the "EdgeRouter Infinity
> XG" device, specifically in the role of a simple peering router for a
> couple of tens of thousands of routes. (I'd point default to the left and
> take just the on-net routes on the right to reduce the table size
> requirement).
>
> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> automatable (supports idempotency), can forward IMIX at line-rate, *flow,
> and exposes some telemetry via SNMP.
>
> Any note sharing would be appreciated!
>
> Kind regards,
>
> Job
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Jérôme Nicolle]

Hello Jeremy,

Le 04/07/2017 à 01:10, Jeremy Austin a écrit :
> can certainly handle a few tens of thousands of
> routes fine (single core BGP though), 

It can take multiple full views. It's also faster than an MX104.

> but I can't vouch for its ability to
> do IMIX or *flow at line rate

I wouldn't load one to 80g, but at 10-20G, it creates no bottleneck.

The entire packet-pipeline is in software. IPFIX is not sampled, it's
1:1 only AFAIK. It's also lacking some features, meaning you'd need to
filter through pmacct to add BGP informations.

Best regards,

-- 
Jérôme Nicolle

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Tim Pozar]

BTW...  At Fandor (before I left) we got one of the last /24s that ARIN
had.   Our transit providers at the office were Monkey Brains (wireless)
and Zayo (fiber).  We purchased a ER Pro, upgraded the memory and were
peering v4 with both on this box.  MB didn't have V6 at that point.  We
did nail up our V6 announcement with Zayo and got it that way.

If folks need config examples.

Tim

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

6.3 ;)

- Josh

On Jul 5, 2017 2:10 PM, "Paul Gear"  wrote:

> On 04/07/17 12:28, Hugo Slabbert wrote:
> >
> > ...
> >>>
> >>> As far as automation, it's a JunOS-like CLI originally based on vyatta,
> >>> which AT now owns - and one of the main reasons is it's
> scriptability,
> >>> use of Ansible and other tools right on the device, python, etc.
> >
> > Technically I believe it's based on VyOS rather than Vyatta.  Same base,
> > but just delineating that VyOS is open source and I don't believe AT
> > wields any control over it.
>
> EdgeOS was forked from Vyatta well before (around Vyatta Core 6.2?) VyOS
> took up the last public Vyatta release.  It has therefore diverged
> somewhat from current VyOS releases, but the two are still
> mostly-compatible.
>
> Paul
>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Paul Gear]

On 04/07/17 12:28, Hugo Slabbert wrote:
> 
> ...
>>>
>>> As far as automation, it's a JunOS-like CLI originally based on vyatta,
>>> which AT now owns - and one of the main reasons is it's scriptability,
>>> use of Ansible and other tools right on the device, python, etc.
> 
> Technically I believe it's based on VyOS rather than Vyatta.  Same base,
> but just delineating that VyOS is open source and I don't believe AT
> wields any control over it.

EdgeOS was forked from Vyatta well before (around Vyatta Core 6.2?) VyOS
took up the last public Vyatta release.  It has therefore diverged
somewhat from current VyOS releases, but the two are still
mostly-compatible.

Paul

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Jared Geiger]

The RAM is upgradeable but it can support quite a few full tables out of
the box. The routing software under the hood got upgraded by Ubiquiti to
ZebOS https://www.ipinfusion.com/products/zebos/ from the VyOS code.

There is a Cavium bug regarding UDP packets though that can be nasty if you
hit it.
https://community.ubnt.com/t5/EdgeMAX/UDP-packet-loss-with-EdgeRouter-Lite/m-p/1343012

Even though the thread starts by talking about the Lite, all of the Cavium
EdgeRouters  currently have the problem. The beta work around is to
restrict packet forwarding to only use one of the CPU cores. This is with
or without hardware offloading enabled. Hopefully Cavium will have a real
fix soon. I have two of these I'm itching to put into production once the
bugs are worked out.

On Mon, Jul 3, 2017 at 8:28 PM, Josh Reynolds  wrote:

> I kinda feel the same way.  I wish FRR was a big more mature at this
> point though.
>
> On Mon, Jul 3, 2017 at 10:21 PM, Baldur Norddahl
>  wrote:
> > Why not use a Linux or BSD computer for this? It is cheap and you know
> > exactly what you are getting. It will forward 10 gig at line rate at
> least
> > for normal traffic.
> >
> > Regards
> >
> > Baldur
> >
> > Den 3. jul. 2017 21.08 skrev "Job Snijders" :
> >
> >> Dear NANOG,
> >>
> >> Some friends of mine are operating a nonprofit (on shoe string) and
> looking
> >> to connect some CDN caches to an IX fabric. A BGP speaking device is
> needed
> >> between the caches and the BGP peers connected to the fabric. The BGP
> >> speaker is needed to present the peers on the IX with a unified view of
> the
> >> assemblage of CDN nodes.
> >>
> >> I was wondering whether anyone was experience with the "EdgeRouter
> Infinity
> >> XG" device, specifically in the role of a simple peering router for a
> >> couple of tens of thousands of routes. (I'd point default to the left
> and
> >> take just the on-net routes on the right to reduce the table size
> >> requirement).
> >>
> >> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> >> automatable (supports idempotency), can forward IMIX at line-rate,
> *flow,
> >> and exposes some telemetry via SNMP.
> >>
> >> Any note sharing would be appreciated!
> >>
> >> Kind regards,
> >>
> >> Job
> >>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

I kinda feel the same way.  I wish FRR was a big more mature at this
point though.

On Mon, Jul 3, 2017 at 10:21 PM, Baldur Norddahl
 wrote:
> Why not use a Linux or BSD computer for this? It is cheap and you know
> exactly what you are getting. It will forward 10 gig at line rate at least
> for normal traffic.
>
> Regards
>
> Baldur
>
> Den 3. jul. 2017 21.08 skrev "Job Snijders" :
>
>> Dear NANOG,
>>
>> Some friends of mine are operating a nonprofit (on shoe string) and looking
>> to connect some CDN caches to an IX fabric. A BGP speaking device is needed
>> between the caches and the BGP peers connected to the fabric. The BGP
>> speaker is needed to present the peers on the IX with a unified view of the
>> assemblage of CDN nodes.
>>
>> I was wondering whether anyone was experience with the "EdgeRouter Infinity
>> XG" device, specifically in the role of a simple peering router for a
>> couple of tens of thousands of routes. (I'd point default to the left and
>> take just the on-net routes on the right to reduce the table size
>> requirement).
>>
>> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
>> automatable (supports idempotency), can forward IMIX at line-rate, *flow,
>> and exposes some telemetry via SNMP.
>>
>> Any note sharing would be appreciated!
>>
>> Kind regards,
>>
>> Job
>>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Baldur Norddahl]

Why not use a Linux or BSD computer for this? It is cheap and you know
exactly what you are getting. It will forward 10 gig at line rate at least
for normal traffic.

Regards

Baldur

Den 3. jul. 2017 21.08 skrev "Job Snijders" :

> Dear NANOG,
>
> Some friends of mine are operating a nonprofit (on shoe string) and looking
> to connect some CDN caches to an IX fabric. A BGP speaking device is needed
> between the caches and the BGP peers connected to the fabric. The BGP
> speaker is needed to present the peers on the IX with a unified view of the
> assemblage of CDN nodes.
>
> I was wondering whether anyone was experience with the "EdgeRouter Infinity
> XG" device, specifically in the role of a simple peering router for a
> couple of tens of thousands of routes. (I'd point default to the left and
> take just the on-net routes on the right to reduce the table size
> requirement).
>
> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
> automatable (supports idempotency), can forward IMIX at line-rate, *flow,
> and exposes some telemetry via SNMP.
>
> Any note sharing would be appreciated!
>
> Kind regards,
>
> Job
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

EdgeOS was forked and employees were poached from Vyatta before it was
purchased by Broadcom, when it was open source.  I think a few things
came down from VyOS after that, but not many.

On Mon, Jul 3, 2017 at 9:28 PM, Hugo Slabbert  wrote:
>
> On Mon 2017-Jul-03 19:26:17 -0500, Josh Reynolds 
> wrote:
>
>> On Jul 3, 2017 7:23 PM, "Josh Reynolds"  wrote:
>>
>>> Specs...
>>>
>>>
>>>- MIPS64 16 Core 1.8 GHz
>>>- 16 GB DDR4 RAM
>>>- 8 MB NOR Flash 4 GB eMMC NAND Flash
>>>- Data Ports: (1) RJ45 Serial Port, (8) SFP+ Ports (1) RJ45 Gigabit
>>>Ethernet Port
>>>- 2 hotswap power supplies
>>>
>>>
>>> No LACP. ECMP is currently broken. MPLS/VPLS is currently broken and not
>>> done in hardware - this may eventually change. As far as the other stuff,
>>> "telemetry" etc - no.
>>>
>>> As far as BGP crunching, plenty of routes, etc - it would easily and
>>> happily be fine with that.
>>>
>>> As far as automation, it's a JunOS-like CLI originally based on vyatta,
>>> which AT now owns - and one of the main reasons is it's scriptability,
>>> use of Ansible and other tools right on the device, python, etc.
>
>
> Technically I believe it's based on VyOS rather than Vyatta.  Same base, but
> just delineating that VyOS is open source and I don't believe AT wields
> any control over it.
>
>>>
>>> - Josh
>>>
>
> --
> Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
> pgp key: B178313E   | also on Signal
>
>
>>> On Jul 3, 2017 2:09 PM, "Job Snijders"  wrote:
>>>
 Dear NANOG,

 Some friends of mine are operating a nonprofit (on shoe string) and
 looking
 to connect some CDN caches to an IX fabric. A BGP speaking device is
 needed
 between the caches and the BGP peers connected to the fabric. The BGP
 speaker is needed to present the peers on the IX with a unified view of
 the
 assemblage of CDN nodes.

 I was wondering whether anyone was experience with the "EdgeRouter
 Infinity
 XG" device, specifically in the role of a simple peering router for a
 couple of tens of thousands of routes. (I'd point default to the left
 and
 take just the on-net routes on the right to reduce the table size
 requirement).

 I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
 automatable (supports idempotency), can forward IMIX at line-rate,
 *flow,
 and exposes some telemetry via SNMP.

 Any note sharing would be appreciated!

 Kind regards,

 Job

>>>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Hugo Slabbert]

On Mon 2017-Jul-03 19:26:17 -0500, Josh Reynolds  wrote:


On Jul 3, 2017 7:23 PM, "Josh Reynolds"  wrote:


Specs...


   - MIPS64 16 Core 1.8 GHz
   - 16 GB DDR4 RAM
   - 8 MB NOR Flash 4 GB eMMC NAND Flash
   - Data Ports: (1) RJ45 Serial Port, (8) SFP+ Ports (1) RJ45 Gigabit
   Ethernet Port
   - 2 hotswap power supplies


No LACP. ECMP is currently broken. MPLS/VPLS is currently broken and not
done in hardware - this may eventually change. As far as the other stuff,
"telemetry" etc - no.

As far as BGP crunching, plenty of routes, etc - it would easily and
happily be fine with that.

As far as automation, it's a JunOS-like CLI originally based on vyatta,
which AT now owns - and one of the main reasons is it's scriptability,
use of Ansible and other tools right on the device, python, etc.


Technically I believe it's based on VyOS rather than Vyatta.  Same base, 
but just delineating that VyOS is open source and I don't believe AT 
wields any control over it.




- Josh



--
Hugo Slabbert   | email, xmpp/jabber: h...@slabnet.com
pgp key: B178313E   | also on Signal


On Jul 3, 2017 2:09 PM, "Job Snijders"  wrote:


Dear NANOG,

Some friends of mine are operating a nonprofit (on shoe string) and
looking
to connect some CDN caches to an IX fabric. A BGP speaking device is
needed
between the caches and the BGP peers connected to the fabric. The BGP
speaker is needed to present the peers on the IX with a unified view of
the
assemblage of CDN nodes.

I was wondering whether anyone was experience with the "EdgeRouter
Infinity
XG" device, specifically in the role of a simple peering router for a
couple of tens of thousands of routes. (I'd point default to the left and
take just the on-net routes on the right to reduce the table size
requirement).

I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
automatable (supports idempotency), can forward IMIX at line-rate, *flow,
and exposes some telemetry via SNMP.

Any note sharing would be appreciated!

Kind regards,

Job





signature.asc
Description: Digital signature

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Josh Reynolds]

- Josh

On Jul 3, 2017 7:23 PM, "Josh Reynolds"  wrote:

> Specs...
>
>
>- MIPS64 16 Core 1.8 GHz
>- 16 GB DDR4 RAM
>- 8 MB NOR Flash 4 GB eMMC NAND Flash
>- Data Ports: (1) RJ45 Serial Port, (8) SFP+ Ports (1) RJ45 Gigabit
>Ethernet Port
>- 2 hotswap power supplies
>
>
> No LACP. ECMP is currently broken. MPLS/VPLS is currently broken and not
> done in hardware - this may eventually change. As far as the other stuff,
> "telemetry" etc - no.
>
> As far as BGP crunching, plenty of routes, etc - it would easily and
> happily be fine with that.
>
> As far as automation, it's a JunOS-like CLI originally based on vyatta,
> which AT now owns - and one of the main reasons is it's scriptability,
> use of Ansible and other tools right on the device, python, etc.
>
> - Josh
>
> On Jul 3, 2017 2:09 PM, "Job Snijders"  wrote:
>
>> Dear NANOG,
>>
>> Some friends of mine are operating a nonprofit (on shoe string) and
>> looking
>> to connect some CDN caches to an IX fabric. A BGP speaking device is
>> needed
>> between the caches and the BGP peers connected to the fabric. The BGP
>> speaker is needed to present the peers on the IX with a unified view of
>> the
>> assemblage of CDN nodes.
>>
>> I was wondering whether anyone was experience with the "EdgeRouter
>> Infinity
>> XG" device, specifically in the role of a simple peering router for a
>> couple of tens of thousands of routes. (I'd point default to the left and
>> take just the on-net routes on the right to reduce the table size
>> requirement).
>>
>> I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
>> automatable (supports idempotency), can forward IMIX at line-rate, *flow,
>> and exposes some telemetry via SNMP.
>>
>> Any note sharing would be appreciated!
>>
>> Kind regards,
>>
>> Job
>>
>

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Jeremy Austin]

On Mon, Jul 3, 2017 at 2:44 PM, Seth Mattinen  wrote:

>
> EdgeRouter is... meh. If I was looking at that class of gear I'd go with a
> Mikrotik.


Job,

There is a bit of a price differential here, depending on whether you need
SFP+; the Infinity is "dead cheap", and has fairly opaque BGP
daemon+debugging tools. Also still technically a beta product. Not sure if
it meets your automation requirements. I wouldn't want to be deploying them
in a redundant pair, myself, but just when you say something can't be done…

Mikrotik's CCR1072: 10-gig router (shipping, not anything that's just been
announced) has an API, can certainly handle a few tens of thousands of
routes fine (single core BGP though), but I can't vouch for its ability to
do IMIX or *flow at line rate. This has probably been stress tested by
somebody. I doubt the sampling is in hardware.

If you don't need 10G ports then your options expand considerably. Do you
have a target throughput?

-- 
Jeremy Austin

(907) 895-2311 office
(907) 803-5422 cell
jhaus...@gmail.com

Heritage NetWorks
Whitestone Power & Communications
Vertical Broadband, LLC

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Seth Mattinen]

On 7/3/17 12:07 PM, Job Snijders wrote:

I was wondering whether anyone was experience with the "EdgeRouter Infinity
XG" device, specifically in the role of a simple peering router for a
couple of tens of thousands of routes. (I'd point default to the left and
take just the on-net routes on the right to reduce the table size
requirement).



EdgeRouter is... meh. If I was looking at that class of gear I'd go with 
a Mikrotik.


~Seth

Re: EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Mike Hammett]

I'm Ubiquiti's biggest critic. I'll check with my colleagues. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Job Snijders" <j...@instituut.net> 
To: nanog@nanog.org 
Sent: Monday, July 3, 2017 2:07:28 PM 
Subject: EdgeRouter Infinity as medium-sized "IXP Peering Router"? 

Dear NANOG, 

Some friends of mine are operating a nonprofit (on shoe string) and looking 
to connect some CDN caches to an IX fabric. A BGP speaking device is needed 
between the caches and the BGP peers connected to the fabric. The BGP 
speaker is needed to present the peers on the IX with a unified view of the 
assemblage of CDN nodes. 

I was wondering whether anyone was experience with the "EdgeRouter Infinity 
XG" device, specifically in the role of a simple peering router for a 
couple of tens of thousands of routes. (I'd point default to the left and 
take just the on-net routes on the right to reduce the table size 
requirement). 

I hope the device can do at least 2xLACP trunks, has a sizable FIB, is 
automatable (supports idempotency), can forward IMIX at line-rate, *flow, 
and exposes some telemetry via SNMP. 

Any note sharing would be appreciated! 

Kind regards, 

Job 

EdgeRouter Infinity as medium-sized "IXP Peering Router"?

[Job Snijders]

Dear NANOG,

Some friends of mine are operating a nonprofit (on shoe string) and looking
to connect some CDN caches to an IX fabric. A BGP speaking device is needed
between the caches and the BGP peers connected to the fabric. The BGP
speaker is needed to present the peers on the IX with a unified view of the
assemblage of CDN nodes.

I was wondering whether anyone was experience with the "EdgeRouter Infinity
XG" device, specifically in the role of a simple peering router for a
couple of tens of thousands of routes. (I'd point default to the left and
take just the on-net routes on the right to reduce the table size
requirement).

I hope the device can do at least 2xLACP trunks, has a sizable FIB, is
automatable (supports idempotency), can forward IMIX at line-rate, *flow,
and exposes some telemetry via SNMP.

Any note sharing would be appreciated!

Kind regards,

Job

IXP economics Was: Re: NANOG67 - Tipping point of community and sponsor bashing?

[Martin Hannigan]

Well. Its complicated. I think this is far more political than about COGS.
But hey. Why not?

I agree with Dave. Shocking. I know. At least the context. He's right.
Thanks for reminding us. We know these things. We'll have to see how IXP
communities react now. Perhaps espresso service will be defined as good
outreach? If it is, thats certainly up to them.

Btw, have you thanked your local US branded euro IXP? They created market
pressures that saved many of us a Brinks truck full of cash by increasing
competitiveness. A lot of us owe them thanks and support for doing
good.  If you do, grab Job, John, Henk, Pauline, Arnold or Andreas and give
them a big COGS crushing hug. Go easy on Pauline, less crush please.

Best,

Marty

On Thursday, June 16, 2016, Zbyněk Pospíchal <zby...@dialtelecom.cz> wrote:

> Dne 16.06.16 v 17:17 Niels Bakker napsal(a):
> > * zby...@dialtelecom.cz <javascript:;> (Zbyněk Pospíchal) [Thu 16 Jun
> 2016, 14:23 CEST]:
>
> >> Are you sure they still want them if they have to pay for these
> >> features separately?
> >>
> >> Currently, such luxury functions are increasing costs also for
> >> networks who don't need/want it.
> >
> > sFlow statistics isn't a luxury function.
>
> Anything more than plain L2 in an IXP is a kind of luxury. An IXP member
> with it's own flow collection (or at least mac accounting) can feel they
> don't need sFlow statistics in an exchange. It's also proven it's
> possible to run an IXP, including a big one, without sFlow stats.
>
> We can say the same about route servers, SLA, customer portals etc. (ok,
> remote peering is a different case).
>
> If IXP members think they have to pay such functionality in their port
> fees, ok, it's their own decision, but member's opinion "we don't need
> it and we don't want to pay for it" is rational and plausible.
>
> Best Regards,
> Zbynek
>

Re: Global/distributed IXP operators?

[Marty Strong via NANOG]

I think he means IXes like NetIX: http://netix.net/map, a single distributed 
fabric.

Regards,
Marty Strong
--
CloudFlare - AS13335
Network Engineer
ma...@cloudflare.com
+44 7584 906 055
smartflare (Skype)

http://www.peeringdb.com/view.php?asn=13335

> On 29 May 2016, at 13:26, Mike Hammett <na...@ics-il.net> wrote:
> 
> Could you define what you mean by a distributed\global IXP? There are plenty 
> of IXPs, but there aren't really global IXPs, those just become networks. 
> 
> 
> 
> 
> - 
> Mike Hammett 
> Intelligent Computing Solutions 
> http://www.ics-il.com 
> 
> 
> 
> Midwest Internet Exchange 
> http://www.midwest-ix.com 
> 
> 
> - Original Message -
> 
> From: "Daniel Rohan" <dro...@gmail.com> 
> To: "NANOG" <nanog@nanog.org> 
> Sent: Friday, May 27, 2016 12:47:07 PM 
> Subject: Global/distributed IXP operators? 
> 
> If there are any operators working at distributed/global IXPs and wouldn't 
> mind having their brains picked regarding design questions, would you make 
> yourselves known to me (on or off-list is fine). 
> 
> Thanks, 
> 
> Dan 
> 

Re: Global/distributed IXP operators?

[Mike Hammett]

Could you define what you mean by a distributed\global IXP? There are plenty of 
IXPs, but there aren't really global IXPs, those just become networks. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



Midwest Internet Exchange 
http://www.midwest-ix.com 


- Original Message -

From: "Daniel Rohan" <dro...@gmail.com> 
To: "NANOG" <nanog@nanog.org> 
Sent: Friday, May 27, 2016 12:47:07 PM 
Subject: Global/distributed IXP operators? 

If there are any operators working at distributed/global IXPs and wouldn't 
mind having their brains picked regarding design questions, would you make 
yourselves known to me (on or off-list is fine). 

Thanks, 

Dan 

Global/distributed IXP operators?

[Daniel Rohan]

If there are any operators working at distributed/global IXPs and wouldn't
mind having their brains picked regarding design questions, would you make
yourselves known to me (on or off-list is fine).

Thanks,

Dan

Re: Recommended L2 switches for a new IXP

[Marian Ďurkovič]

On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote:
 I think in fairly short order both TRILL and 802.1AQ will be depercated in 
 place of VXLAN and using BGP EVPN as the control plane ala Juniper 
 QFX5100/Nexus 9300. 

We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW
support for it. But VXLAN does *not* create a network for you, it relies on
some existing underlying IP network, on top of which VXLAN creates stateless
tunnels.

By using TRILL, we could connect 4 switches into a ring (or any other
reasonable topology) and have a fully functional network with shortest-path
routing of L2 packets.

With VXLAN, we'd need at least two additional IP routers with bunch of
40GE interfaces to perform the functions TRILL supports out of the box.

Regards,

   M.

  

Re: Recommended L2 switches for a new IXP

[Phil Bedard]

For many people eliminating L2 switching and building on top of a L3 
network is a good thing, especially if you are using BGP as the control 
plane. 

I'm not sure I follow the two routers with 40GE interfaces if you are just 
building L2 domains to interconnect people.  

Phil 



On 1/20/15, 8:04 AM, Marian Ďurkovič m...@bts.sk wrote:

On Mon, Jan 19, 2015 at 09:37:35PM -0500, Phil Bedard wrote:
 I think in fairly short order both TRILL and 802.1AQ will be depercated 
in 
 place of VXLAN and using BGP EVPN as the control plane ala Juniper 
 QFX5100/Nexus 9300. 

We also evaluated VXLAN for IXP deployment, since Trident-2 introduced HW
support for it. But VXLAN does *not* create a network for you, it relies 
on
some existing underlying IP network, on top of which VXLAN creates 
stateless
tunnels.

By using TRILL, we could connect 4 switches into a ring (or any other
reasonable topology) and have a fully functional network with 
shortest-path
routing of L2 packets.

With VXLAN, we'd need at least two additional IP routers with bunch of
40GE interfaces to perform the functions TRILL supports out of the box.

Regards,

   M.

Re: Recommended L2 switches for a new IXP

[Phil Bedard]

On 1/17/15, 7:15 PM, Saku Ytti s...@ytti.fi wrote:


On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:

 Our experience after 100 days of production is only the best -  TRILL 
setup
 is pretty straightforward and thanks to IS-IS it provides shortest-path 
 IP-like routing for L2 ethernet packets over any reasonable topology 
 out of the box (without the burden and cost implications of VPLS).

I'm not sure what the burden refers to, but cost implications to me seem 
same,
trident HW can do VPLS.
From complexity POV, I don't expect much different development time to 
write
functioning control-plane to either.

I'm not against Trill, I think Trill, and especially SPB-M are great, now 
they
just feel too little and 20 years too late. There was no particular 
reason why
SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it
didn't, it might have made ethernet 'good enough', that selling MPLS might
have been much more difficult.

-- 
  ++ytti


I think in fairly short order both TRILL and 802.1AQ will be depercated in 
place of VXLAN and using BGP EVPN as the control plane ala Juniper 
QFX5100/Nexus 9300. 

Phil

Re: Recommended L2 switches for a new IXP

[Nick Hilliard]

On 19/01/2015 10:12, Marian Ďurkovič wrote:
 Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support
 per-flow loadbalancing on IXP participants' LAGs.

not completely true.  Extreme XOS has an interesting hack to work around this.

Nick

Re: Recommended L2 switches for a new IXP

[Marian Ďurkovič]

On Sat, Jan 17, 2015 at 09:15:04PM +0200, Saku Ytti wrote:
 On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:
 
  Our experience after 100 days of production is only the best -  TRILL setup
  is pretty straightforward and thanks to IS-IS it provides shortest-path 
  IP-like routing for L2 ethernet packets over any reasonable topology 
  out of the box (without the burden and cost implications of VPLS).
 
 I'm not sure what the burden refers to, but cost implications to me seem same,
 trident HW can do VPLS.

Well, it can, but as usual the devil is in the detail.

For example, loadbalancing on outgoing LAGs depends on *inbound* packet 
encapsulation as follows:

- native ethernet, TRILL, L3 MPLS : hash based on L3 and L4 headers
- L2 MPLS, MACinMAC : hash based on L2 headers only.

Thus if you use VPLS or SPB-M on Trident HW, the egress PE doesn't support
per-flow loadbalancing on IXP participants' LAGs.

In any case, we preferred TRILL over SPB-M not just because of that, but 
mainly due to a fact that TRILL provides real routing using IS-IS as we 
know it from IP world, while SPB still builds on top of MST and just cleverly
uses multiple trees. Yes, compatibility with existing ASICs was one of the 
main design goals of SPB, but that's irrelevant once you have Trident HW. 

Regards,

   M. 

Re: Recommended L2 switches for a new IXP

[Saku Ytti]

On (2015-01-17 12:02 +0100), Marian Ďurkovič wrote:

 Our experience after 100 days of production is only the best -  TRILL setup
 is pretty straightforward and thanks to IS-IS it provides shortest-path 
 IP-like routing for L2 ethernet packets over any reasonable topology 
 out of the box (without the burden and cost implications of VPLS).

I'm not sure what the burden refers to, but cost implications to me seem same,
trident HW can do VPLS.
From complexity POV, I don't expect much different development time to write
functioning control-plane to either.

I'm not against Trill, I think Trill, and especially SPB-M are great, now they
just feel too little and 20 years too late. There was no particular reason why
SPB-M couldn't have existed 20 years ago in HW. But perhaps it's good it
didn't, it might have made ethernet 'good enough', that selling MPLS might
have been much more difficult.

-- 
  ++ytti

Re: Recommended L2 switches for a new IXP

[Marian Ďurkovič]

Last year we installed four 1RU TRILL switches in SIX - see
  http://www.six.sk/images/trill_ring.png

Our experience after 100 days of production is only the best -  TRILL setup
is pretty straightforward and thanks to IS-IS it provides shortest-path 
IP-like routing for L2 ethernet packets over any reasonable topology 
out of the box (without the burden and cost implications of VPLS).
Trident ASICs perform deep packet inspection so ECMP loadbalancing based
on L3 and L4 headers inside TRILL-encapsulated packets works for both IPv4
and IPv6. Port-security is supported on physical ports as well as on LAGs
- and L4 access-lists could be applied at the same time. 

As most 1RU switches are based on Trident ASICs, you just need to pick
a vendor which implements TRILL properly and of course thoroughly test
before deployment. We selected Huawei Cloud Engine 6850 boxes.

Regards,

   M.
 
 Dear Nanog community
 
 We are trying to build a new IXP in some US Metro areas where we have
 multiple POPs and I was wondering what do you recommend for L2 switches. I
 know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
 experience with these switches. It would be great if you can share your
 experience and recommendations. There are so many options that I don't know
 if it makes sense to start with a modular switch (usually expensive because
 the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
 switch that support new protocols like Trill and that supposedly allow you
 to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
 ports for exchange participants, 40G/100G for uplinks between switches and
 flow support for statistics and traffic analysis.
 
 Thank you and have a great day.
 
 Regards

Re: Recommended L2 switches for a new IXP

[Stephen R. Carter]

We always adhere to JTAC:
http://kb.juniper.net/InfoCenter/index?page=contentid=KB21476actp=SUBSCRI
PTION unless otherwise required by their support to change.

Currently it is Junos 13.2X51-D26.

My advice to you is to not use 14.1 unless you have a reason, as that is
more of a dev branch in terms of stability than anything.

We use VRRP, OSPF, MC-LAG, and so forth. Nothing super fancy.

Stephen Carter | IT Systems Administrator  | Gun Lake Tribal Gaming
Commission
1123 129th Avenue, Wayland, MI 49348
Phone 269.792.1773 







On 1/15/15, 4:17 AM, Richard Hartmann richih.mailingl...@gmail.com
wrote:

On Tue, Jan 13, 2015 at 4:45 PM, Stephen R. Carter
stephen.car...@gltgc.org wrote:
 We love our 5100s here.

Out of interest: Are you running 13.2 or 14.1?

What features are you using?


Our own experiences with a bunch of 48  96 port machines running 14.1
is painful to say the least.


Richard


brhrfont face='Arial' color='Gray' size='1'The information contained in 
this electronic transmission (email) is confidential information and may be 
subject to attorney/client privilege. It is intended only for the use of the 
individual or entity named above. ANY DISTRIBUTION OR COPYING OF THIS MESSAGE 
IS PROHIBITED, except by the intended recipient. Attempts to intercept this 
message are in violation of 18 U.S.C. 2511(1) of the Electronic Communications 
Privacy Act (ECPA), which subjects the interceptor to fines, imprisonment 
and/or civil damages./font

Re: Recommended L2 switches for a new IXP

[Richard Hartmann]

On Tue, Jan 13, 2015 at 4:45 PM, Stephen R. Carter
stephen.car...@gltgc.org wrote:
 We love our 5100s here.

Out of interest: Are you running 13.2 or 14.1?

What features are you using?


Our own experiences with a bunch of 48  96 port machines running 14.1
is painful to say the least.


Richard

Re: Recommended L2 switches for a new IXP

[Chuck Anderson]

Software Defined Networking (SDN) features that QFX5100 supports:

Automatic configuration of OVSDB-managed VXLANs with trunk interfaces 
14.1X53-D15
OVSDB support 14.1X53-D10
OpenFlow v1.0 14.1X53-D10
OpenFlow v1.3.1 14.1X53-D10
VXLAN Gateway 14.1X53-D10

http://pathfinder.juniper.net/feature-explorer/select-software.html?swName=Junos+OStyp=1#family=platform=QFX5100rel=14.1X53-D15swName=Junos+OS

On Tue, Jan 13, 2015 at 10:10:56PM +, Jeff Tantsura wrote:
 What does it mean -  to be SDN ready?
 
 Cheers,
 Jeff
 
 
 
 
 -Original Message-
 From: Eduardo Schoedler lis...@esds.com.br
 Date: Tuesday, January 13, 2015 at 3:25 AM
 To: nanog@nanog.org nanog@nanog.org
 Subject: Re: Recommended L2 switches for a new IXP
 
 QFX5100 is SDN ready.
 
 --
 Eduardo Schoedler
 
 
 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:
 
  Is there any particular reason you prefer EX4600 over QFX5100 ? Not
  counting obvious differences like ports and upgrade options.
 
  It's the same chipset after all, and with all upgrades they have the
  same 10G density (with breakouts). Is that because you can have more 40G
  ports with EX4600 ?
 
  I'm still trying to find out if there are any noticeable software or
  feature differences.
 
  On 13.01.2015 09:01, Mark Tinka wrote:
   On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
  
   People seem to be avoiding recommending actual devices,
   well I would recommend the Juniper EX4600 -
  
   http://www.juniper.net/us/en/products-services/switching/
   ex-series/ex4600/
  
   They are affordable, highly scalable, stackable and run
   JunOS.
  
   We've been quite happy with the EX4550, but the EX4600 is
   good too, particularly if you're coming from its younger
   brother.
  
   Mark.
  
 
 
 
 
 -- 
 Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Mark Tinka]

On Wednesday, January 14, 2015 12:25:30 AM Jeff Tantsura 
wrote:

 AhhhŠ vertically integrated horizontal API¹s

Green, vertically integrated horizontal API's :-).

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Recommended L2 switches for a new IXP

[Mark Tinka]

On Wednesday, January 14, 2015 12:47:09 AM Jeff Tantsura 
wrote:

 Got you - artificially disabling 90% of the features
 otherwise supported by the OS and using half baked HAL
 makes product SDN ready! Sorry for the sarcasm, couldn¹t
 resist :)

I once tested a Junos release with the X blah blah D blah 
blah letters in there on an EX4550. Couldn't even get LACP 
going, until I realized it was some kind of QFX'y release 
for the non-QFX EX boxes.

Promptly got ride of that.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Recommended L2 switches for a new IXP

[Michael Smith]

You can see what we have at the SIX here - 
http://www.seattleix.net/topology.html

Mike
--
Michael K. Smith
mksm...@mac.com

On Jan 11, 2015, at 10:37 PM, Manuel Marín m...@transtelco.net wrote:

Dear Nanog community

We are trying to build a new IXP in some US Metro areas where we have
multiple POPs and I was wondering what do you recommend for L2 switches. I
know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
experience with these switches. It would be great if you can share your
experience and recommendations. There are so many options that I don't know
if it makes sense to start with a modular switch (usually expensive because
the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
switch that support new protocols like Trill and that supposedly allow you
to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
ports for exchange participants, 40G/100G for uplinks between switches and
flow support for statistics and traffic analysis.

Thank you and have a great day.

Regards

Re: Recommended L2 switches for a new IXP

[Stepan Kucherenko]

Is there any particular reason you prefer EX4600 over QFX5100 ? Not
counting obvious differences like ports and upgrade options.

It's the same chipset after all, and with all upgrades they have the
same 10G density (with breakouts). Is that because you can have more 40G
ports with EX4600 ?

I'm still trying to find out if there are any noticeable software or
feature differences.

On 13.01.2015 09:01, Mark Tinka wrote:
 On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
 
 People seem to be avoiding recommending actual devices,
 well I would recommend the Juniper EX4600 -

 http://www.juniper.net/us/en/products-services/switching/
 ex-series/ex4600/

 They are affordable, highly scalable, stackable and run
 JunOS.
 
 We've been quite happy with the EX4550, but the EX4600 is 
 good too, particularly if you're coming from its younger 
 brother.
 
 Mark.
 

Re: Recommended L2 switches for a new IXP

[Eduardo Schoedler]

QFX5100 is SDN ready.

--
Eduardo Schoedler


2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:

 Is there any particular reason you prefer EX4600 over QFX5100 ? Not
 counting obvious differences like ports and upgrade options.

 It's the same chipset after all, and with all upgrades they have the
 same 10G density (with breakouts). Is that because you can have more 40G
 ports with EX4600 ?

 I'm still trying to find out if there are any noticeable software or
 feature differences.

 On 13.01.2015 09:01, Mark Tinka wrote:
  On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
 
  People seem to be avoiding recommending actual devices,
  well I would recommend the Juniper EX4600 -
 
  http://www.juniper.net/us/en/products-services/switching/
  ex-series/ex4600/
 
  They are affordable, highly scalable, stackable and run
  JunOS.
 
  We've been quite happy with the EX4550, but the EX4600 is
  good too, particularly if you're coming from its younger
  brother.
 
  Mark.
 




-- 
Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Stephen R. Carter]

We love our 5100s here.

I have 4 48S, and 2 24q¹s.

Super fast, TISSU when it works is awesome as well... like, really awesome.

Stephen Carter | IT Systems Administrator  | Gun Lake Tribal Gaming
Commission
1123 129th Avenue, Wayland, MI 49348
Phone 269.792.1773 

On 1/13/15, 3:29 AM, Stepan Kucherenko t...@megagroup.ru wrote:


Is there any particular reason you prefer EX4600 over QFX5100 ? Not
counting obvious differences like ports and upgrade options.

It's the same chipset after all, and with all upgrades they have the
same 10G density (with breakouts). Is that because you can have more 40G
ports with EX4600 ?

I'm still trying to find out if there are any noticeable software or
feature differences.

On 13.01.2015 09:01, Mark Tinka wrote:
 On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
 
 People seem to be avoiding recommending actual devices,
 well I would recommend the Juniper EX4600 -

 http://www.juniper.net/us/en/products-services/switching/
 ex-series/ex4600/

 They are affordable, highly scalable, stackable and run
 JunOS.
 
 We've been quite happy with the EX4550, but the EX4600 is
 good too, particularly if you're coming from its younger
 brother.
 
 Mark.
 


brhrfont face='Arial' color='Gray' size='1'The information contained in 
this electronic transmission (email) is confidential information and may be 
subject to attorney/client privilege. It is intended only for the use of the 
individual or entity named above. ANY DISTRIBUTION OR COPYING OF THIS MESSAGE 
IS PROHIBITED, except by the intended recipient. Attempts to intercept this 
message are in violation of 18 U.S.C. 2511(1) of the Electronic Communications 
Privacy Act (ECPA), which subjects the interceptor to fines, imprisonment 
and/or civil damages./font

Re: Recommended L2 switches for a new IXP

[Jeff Tantsura]

AhhhŠ vertically integrated horizontal API¹s

Cheers,
Jeff




-Original Message-
From: Nick Hilliard n...@foobar.org
Date: Tuesday, January 13, 2015 at 2:23 PM
To: Jeff Tantsura jeff.tants...@ericsson.com, Eduardo Schoedler
lis...@esds.com.br, nanog@nanog.org nanog@nanog.org
Subject: Re: Recommended L2 switches for a new IXP

On 13/01/2015 22:10, Jeff Tantsura wrote:
 What does it mean -  to be SDN ready?

it means fully buzzword compliant.

Nick

Re: Recommended L2 switches for a new IXP

[Eduardo Schoedler]

My mistake, it's the OCX1100.
http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-hardware-software.html

2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com:

 What does it mean -  to be SDN ready?

 Cheers,
 Jeff




 -Original Message-
 From: Eduardo Schoedler lis...@esds.com.br
 Date: Tuesday, January 13, 2015 at 3:25 AM
 To: nanog@nanog.org nanog@nanog.org
 Subject: Re: Recommended L2 switches for a new IXP

 QFX5100 is SDN ready.
 
 --
 Eduardo Schoedler
 
 
 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:
 
  Is there any particular reason you prefer EX4600 over QFX5100 ? Not
  counting obvious differences like ports and upgrade options.
 
  It's the same chipset after all, and with all upgrades they have the
  same 10G density (with breakouts). Is that because you can have more 40G
  ports with EX4600 ?
 
  I'm still trying to find out if there are any noticeable software or
  feature differences.
 
  On 13.01.2015 09:01, Mark Tinka wrote:
   On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
  
   People seem to be avoiding recommending actual devices,
   well I would recommend the Juniper EX4600 -
  
   http://www.juniper.net/us/en/products-services/switching/
   ex-series/ex4600/
  
   They are affordable, highly scalable, stackable and run
   JunOS.
  
   We've been quite happy with the EX4550, but the EX4600 is
   good too, particularly if you're coming from its younger
   brother.
  
   Mark.
  
 
 
 
 
 --
 Eduardo Schoedler




-- 
Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Tim Raphael]

Either way, you can do SDN and automation with most Juniper kit. On purchase 
of JCare you get free access to Junos Space - great for provisioning and 
management of an IXP.

Regards,

Tim Raphael

 On 14 Jan 2015, at 6:28 am, Eduardo Schoedler lis...@esds.com.br wrote:
 
 My mistake, it's the OCX1100.
 http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-hardware-software.html
 
 2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com:
 
 What does it mean -  to be SDN ready?
 
 Cheers,
 Jeff
 
 
 
 
 -Original Message-
 From: Eduardo Schoedler lis...@esds.com.br
 Date: Tuesday, January 13, 2015 at 3:25 AM
 To: nanog@nanog.org nanog@nanog.org
 Subject: Re: Recommended L2 switches for a new IXP
 
 QFX5100 is SDN ready.
 
 --
 Eduardo Schoedler
 
 
 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:
 
 Is there any particular reason you prefer EX4600 over QFX5100 ? Not
 counting obvious differences like ports and upgrade options.
 
 It's the same chipset after all, and with all upgrades they have the
 same 10G density (with breakouts). Is that because you can have more 40G
 ports with EX4600 ?
 
 I'm still trying to find out if there are any noticeable software or
 feature differences.
 
 On 13.01.2015 09:01, Mark Tinka wrote:
 On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
 
 People seem to be avoiding recommending actual devices,
 well I would recommend the Juniper EX4600 -
 
 http://www.juniper.net/us/en/products-services/switching/
 ex-series/ex4600/
 
 They are affordable, highly scalable, stackable and run
 JunOS.
 
 We've been quite happy with the EX4550, but the EX4600 is
 good too, particularly if you're coming from its younger
 brother.
 
 Mark.
 
 
 
 --
 Eduardo Schoedler
 
 
 -- 
 Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Jeff Tantsura]

Got you - artificially disabling 90% of the features otherwise supported
by the OS and using half baked HAL makes product SDN ready!
Sorry for the sarcasm, couldn¹t resist :)





Cheers,
Jeff



-Original Message-
From: Eduardo Schoedler lis...@esds.com.br
Date: Tuesday, January 13, 2015 at 2:28 PM
To: nanog@nanog.org nanog@nanog.org
Subject: Re: Recommended L2 switches for a new IXP

My mistake, it's the OCX1100.
http://www.networkworld.com/article/2855056/sdn/juniper-unbundles-switch-h
ardware-software.html

2015-01-13 20:10 GMT-02:00 Jeff Tantsura jeff.tants...@ericsson.com:

 What does it mean -  to be SDN ready?

 Cheers,
 Jeff




 -Original Message-
 From: Eduardo Schoedler lis...@esds.com.br
 Date: Tuesday, January 13, 2015 at 3:25 AM
 To: nanog@nanog.org nanog@nanog.org
 Subject: Re: Recommended L2 switches for a new IXP

 QFX5100 is SDN ready.
 
 --
 Eduardo Schoedler
 
 
 2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:
 
  Is there any particular reason you prefer EX4600 over QFX5100 ? Not
  counting obvious differences like ports and upgrade options.
 
  It's the same chipset after all, and with all upgrades they have the
  same 10G density (with breakouts). Is that because you can have more
40G
  ports with EX4600 ?
 
  I'm still trying to find out if there are any noticeable software or
  feature differences.
 
  On 13.01.2015 09:01, Mark Tinka wrote:
   On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
  
   People seem to be avoiding recommending actual devices,
   well I would recommend the Juniper EX4600 -
  
   http://www.juniper.net/us/en/products-services/switching/
   ex-series/ex4600/
  
   They are affordable, highly scalable, stackable and run
   JunOS.
  
   We've been quite happy with the EX4550, but the EX4600 is
   good too, particularly if you're coming from its younger
   brother.
  
   Mark.
  
 
 
 
 
 --
 Eduardo Schoedler




-- 
Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Simon Leinen]

Manuel Marín writes:
 Dear Nanog community
 [...] There are so many options that I don't know if it makes sense to
 start with a modular switch (usually expensive because the backplane,
 dual dc, dual CPU, etc) or start with a 1RU high density switch that
 support new protocols like Trill and that supposedly allow you to
 create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
 ports for exchange participants, 40G/100G for uplinks between switches
 and flow support for statistics and traffic analysis.

Stupid thought from someone who has never built an IXP,
but has been looking at recent trends in data center networks:

There are these white-box switches mostly designed for top-of-rack or
spine (as in leaf-spine/fat-tree datacenter networks) applications.
They have all the necessary port speeds - well 100G seems to be a few
months off.  I'm thinking of brands such as Edge-Core, Quanta etc.

You can get them as bare-metal versions with no switch OS on them,
just a bootloader according to the ONIE standard.  Equipment cost
seems to be on the order of $100 per SFP+ port w/o optics for a
second-to-last generation (Trident-based) 48*10GE+4*40GE ToR switch.

Now, for the limited and somewhat special L2 needs of an IXP, couldn't
someone hack together a suitable switch OS based on Open Network Linux
(ONL) or something like that?

You wouldn't even need MAC address learning or most types of flooding,
because at an IXP this often hurts rather than helps.  For building
larger fabrics you might be using something other (waves hands) than
TRILL; maybe you could get away without slightly complex multi-chassis
multi-channel mechanisms, and so on.

Flow support sounds somewhat tough, but full netflow support that
would get Roland Dobbins' usable telemetry seal of approval is
probably out of reach anyway - it's a high-end feature with classical
gear.  With white-box switches, you could try to use the given 5-tuple
flow hardware capabilities - which might not scale that well -, or use
packet sampling, or try to use the built-in flow and counter mechanisms
in an application-specific way.  (Except *that's* a lot of work on the
software side, and a usably efficient implementation requires slightly
sophisticated hardware/software interfaces.)

Instead of a Linux-based switch OS, one could also build an IXP
application using OpenFlow and some kind of central controller.
(Not to be confused with SDX: Software Defined Internet Exchange.)

Has anybody looked into the feasibility of this?

The software could be done as an open-source community project to make
setting up regional IXPs easier/cheaper.

Large IXPs could sponsor this so they get better scalability - although
I'm not sure how well something like the leaf-spine/fat-tree design maps
to these IXPs, which are typically distributed over several locations.
Maybe they could use something like Facebook's new design, treating each
IXP location as a pod.
-- 
Simon.
[1] https://code.facebook.com/posts/360346274145943

Re: Recommended L2 switches for a new IXP

[Jeff Tantsura]

What does it mean -  to be SDN ready?

Cheers,
Jeff




-Original Message-
From: Eduardo Schoedler lis...@esds.com.br
Date: Tuesday, January 13, 2015 at 3:25 AM
To: nanog@nanog.org nanog@nanog.org
Subject: Re: Recommended L2 switches for a new IXP

QFX5100 is SDN ready.

--
Eduardo Schoedler


2015-01-13 6:29 GMT-02:00 Stepan Kucherenko t...@megagroup.ru:

 Is there any particular reason you prefer EX4600 over QFX5100 ? Not
 counting obvious differences like ports and upgrade options.

 It's the same chipset after all, and with all upgrades they have the
 same 10G density (with breakouts). Is that because you can have more 40G
 ports with EX4600 ?

 I'm still trying to find out if there are any noticeable software or
 feature differences.

 On 13.01.2015 09:01, Mark Tinka wrote:
  On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:
 
  People seem to be avoiding recommending actual devices,
  well I would recommend the Juniper EX4600 -
 
  http://www.juniper.net/us/en/products-services/switching/
  ex-series/ex4600/
 
  They are affordable, highly scalable, stackable and run
  JunOS.
 
  We've been quite happy with the EX4550, but the EX4600 is
  good too, particularly if you're coming from its younger
  brother.
 
  Mark.
 




-- 
Eduardo Schoedler

Re: Recommended L2 switches for a new IXP

[Nick Hilliard]

On 13/01/2015 22:10, Jeff Tantsura wrote:
 What does it mean -  to be SDN ready?

it means fully buzzword compliant.

Nick

Re: Recommended L2 switches for a new IXP

[Mike Hammett]

I look forward to this thread. 

I think one important thing is who is your addressable market size? I'm working 
with a startup IXP and there's only 20 carriers in the building. A chassis 
based switch would be silly as there would never be that many people present. 
2x 1U switches would be more than plenty in their environment. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 



- Original Message -

From: Manuel Marín m...@transtelco.net 
To: nanog@nanog.org 
Sent: Monday, January 12, 2015 12:35:15 AM 
Subject: Recommended L2 switches for a new IXP 

Dear Nanog community 

We are trying to build a new IXP in some US Metro areas where we have 
multiple POPs and I was wondering what do you recommend for L2 switches. I 
know that some IXPs use Nexus, Brocade, Force10 but I don't personally have 
experience with these switches. It would be great if you can share your 
experience and recommendations. There are so many options that I don't know 
if it makes sense to start with a modular switch (usually expensive because 
the backplane, dual dc, dual CPU, etc) or start with a 1RU high density 
switch that support new protocols like Trill and that supposedly allow you 
to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G 
ports for exchange participants, 40G/100G for uplinks between switches and 
flow support for statistics and traffic analysis. 

Thank you and have a great day. 

Regards 

Re: Recommended L2 switches for a new IXP

[Nick Hilliard]

On 12/01/2015 06:35, Manuel Marín wrote:
 We are trying to build a new IXP in some US Metro areas where we have
 multiple POPs and I was wondering what do you recommend for L2 switches. I
 know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
 experience with these switches. It would be great if you can share your
 experience and recommendations.

For a startup IXP, it would probably not be sensible to use chassis based
kit due to cost / real estate issues.

Some personal opinions:

- I have a strong preference for using only open bridging protocols.  This
excludes out vendor proprietary fabrics (VDX, OTV, etc).  This is important
for when you do fabric upgrades on multi-site IXPs.

- You will probably want a product which supports sflow, as peer-to-peer
traffic graphs are massively useful.  Most vendors support sflow on most of
their products with the notable exception of Cisco where only the Nexus 3K
team were enlightened enough to shim it in.  I haven't yet come across a L2
netflow implementation which works well enough to be an adequate
substitute, but ymmv.

- VPLS based fabrics may be important if you have an interesting topology.
 If it is important to you, then you will need a VPLS implementation which
will do proper load balancing over multiple links.  Most don't and this is
a very hard problem to handle on smaller kit.

- There is no excuse for vendor transceiver locking or transceiver
crippling (e.g. refusing to show DDM values) and vendors who do this need
to be made aware that it's not an acceptable business proposition.

- you need kit which will support Layer 2 ACLs and Layer 3 ACLs on layer 2
interfaces.

- you should get in with the open-ix crowd and chat to people over pizza or
peanuts.  You will learn a lot from in an afternoon of immersion with peers.

Nick

Re: Recommended L2 switches for a new IXP

[Aaron]

We used to use Brocade FastIrons until we needed more 10G port density.  
We moved to Brocade SX's.


Originally, when it was 2 or 3 peers, we used an old Netgear switch. :)

Aaron

On 1/12/2015 7:07 AM, Mike Hammett wrote:

I look forward to this thread.

I think one important thing is who is your addressable market size? I'm working 
with a startup IXP and there's only 20 carriers in the building. A chassis 
based switch would be silly as there would never be that many people present. 
2x 1U switches would be more than plenty in their environment.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com



- Original Message -

From: Manuel Marín m...@transtelco.net
To: nanog@nanog.org
Sent: Monday, January 12, 2015 12:35:15 AM
Subject: Recommended L2 switches for a new IXP

Dear Nanog community

We are trying to build a new IXP in some US Metro areas where we have
multiple POPs and I was wondering what do you recommend for L2 switches. I
know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
experience with these switches. It would be great if you can share your
experience and recommendations. There are so many options that I don't know
if it makes sense to start with a modular switch (usually expensive because
the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
switch that support new protocols like Trill and that supposedly allow you
to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
ports for exchange participants, 40G/100G for uplinks between switches and
flow support for statistics and traffic analysis.

Thank you and have a great day.

Regards




--

Aaron Wendel
Chief Technical Officer
Wholesale Internet, Inc. (AS 32097)
(816)550-9030
http://www.wholesaleinternet.com

Re: Recommended L2 switches for a new IXP

[Martin Hannigan]

Substantial amounts of hive mind went into this topic in the formation of
Open-IX and particularly around optimizing costs and maximizing traffic.
See http://bit.ly/N-OIX1 for a reference.

Best,

-M




On Mon, Jan 12, 2015 at 10:34 AM, Justin Wilson - MTIN li...@mtin.net
wrote:

 Like Mike says, it depends on your market.   Are these markets where there
 are existing exchanges?

 Cost per port is what we always look at.  If we are going into a market
 where there won't be much growth we look at Cisco and Force 10.  Their cost
 per port is usually cheaper for smaller 10 Gig switches. You need something
 that is fairly robust.

 Reliability in an exchange is a key component.  If you go with a
 non-chassis switch make sure you have redundancy in your design.  We like
 Chassis based switches because they tend to be more robust.  But thats just
 my take on it.

 Justin

 ---
 Justin Wilson j...@mtin.net
 http://www.mtin.net
 Managed Services - xISP Solutions - Data Centers
 http://www.thebrotherswisp.com
 Podcast about xISP topics
 http://www.midwest-ix.com
 Peering - Transit - Internet Exchange

  On Jan 12, 2015, at 10:24 AM, Aaron aa...@wholesaleinternet.net wrote:
 
  We used to use Brocade FastIrons until we needed more 10G port density.
 We moved to Brocade SX's.
 
  Originally, when it was 2 or 3 peers, we used an old Netgear switch. :)
 
  Aaron
 
  On 1/12/2015 7:07 AM, Mike Hammett wrote:
  I look forward to this thread.
 
  I think one important thing is who is your addressable market size? I'm
 working with a startup IXP and there's only 20 carriers in the building. A
 chassis based switch would be silly as there would never be that many
 people present. 2x 1U switches would be more than plenty in their
 environment.
 
 
 
 
  -
  Mike Hammett
  Intelligent Computing Solutions
  http://www.ics-il.com
 
 
 
  - Original Message -
 
  From: Manuel Marín m...@transtelco.net
  To: nanog@nanog.org
  Sent: Monday, January 12, 2015 12:35:15 AM
  Subject: Recommended L2 switches for a new IXP
 
  Dear Nanog community
 
  We are trying to build a new IXP in some US Metro areas where we have
  multiple POPs and I was wondering what do you recommend for L2
 switches. I
  know that some IXPs use Nexus, Brocade, Force10 but I don't personally
 have
  experience with these switches. It would be great if you can share your
  experience and recommendations. There are so many options that I don't
 know
  if it makes sense to start with a modular switch (usually expensive
 because
  the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
  switch that support new protocols like Trill and that supposedly allow
 you
  to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
  ports for exchange participants, 40G/100G for uplinks between switches
 and
  flow support for statistics and traffic analysis.
 
  Thank you and have a great day.
 
  Regards
 
 
 
  --
  
  Aaron Wendel
  Chief Technical Officer
  Wholesale Internet, Inc. (AS 32097)
  (816)550-9030
  http://www.wholesaleinternet.com
  
 

Re: Recommended L2 switches for a new IXP

[Justin Wilson - MTIN]

Like Mike says, it depends on your market.   Are these markets where there are 
existing exchanges? 

Cost per port is what we always look at.  If we are going into a market where 
there won’t be much growth we look at Cisco and Force 10.  Their cost per port 
is usually cheaper for smaller 10 Gig switches. You need something that is 
fairly robust.

Reliability in an exchange is a key component.  If you go with a non-chassis 
switch make sure you have redundancy in your design.  We like Chassis based 
switches because they tend to be more robust.  But thats just my take on it.

Justin

---
Justin Wilson j...@mtin.net
http://www.mtin.net
Managed Services – xISP Solutions – Data Centers
http://www.thebrotherswisp.com 
Podcast about xISP topics
http://www.midwest-ix.com
Peering – Transit – Internet Exchange 

 On Jan 12, 2015, at 10:24 AM, Aaron aa...@wholesaleinternet.net wrote:
 
 We used to use Brocade FastIrons until we needed more 10G port density.  We 
 moved to Brocade SX's.
 
 Originally, when it was 2 or 3 peers, we used an old Netgear switch. :)
 
 Aaron
 
 On 1/12/2015 7:07 AM, Mike Hammett wrote:
 I look forward to this thread.
 
 I think one important thing is who is your addressable market size? I'm 
 working with a startup IXP and there's only 20 carriers in the building. A 
 chassis based switch would be silly as there would never be that many people 
 present. 2x 1U switches would be more than plenty in their environment.
 
 
 
 
 -
 Mike Hammett
 Intelligent Computing Solutions
 http://www.ics-il.com
 
 
 
 - Original Message -
 
 From: Manuel Marín m...@transtelco.net
 To: nanog@nanog.org
 Sent: Monday, January 12, 2015 12:35:15 AM
 Subject: Recommended L2 switches for a new IXP
 
 Dear Nanog community
 
 We are trying to build a new IXP in some US Metro areas where we have
 multiple POPs and I was wondering what do you recommend for L2 switches. I
 know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
 experience with these switches. It would be great if you can share your
 experience and recommendations. There are so many options that I don't know
 if it makes sense to start with a modular switch (usually expensive because
 the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
 switch that support new protocols like Trill and that supposedly allow you
 to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
 ports for exchange participants, 40G/100G for uplinks between switches and
 flow support for statistics and traffic analysis.
 
 Thank you and have a great day.
 
 Regards
 
 
 
 -- 
 
 Aaron Wendel
 Chief Technical Officer
 Wholesale Internet, Inc. (AS 32097)
 (816)550-9030
 http://www.wholesaleinternet.com
 
 

Re: Recommended L2 switches for a new IXP

[Martin Hannigan]

On Mon, Jan 12, 2015 at 10:43 AM, Nick Hilliard n...@foobar.org wrote:


[ clip, good stuff ]


- you should get in with the open-ix crowd and chat to people over pizza or
 peanuts.  You will learn a lot from in an afternoon of immersion with
 peers.



And you can find that crowd here
http://mailman.open-ix.org/mailman/listinfo/public if interested.

Best,

-M

Re: Recommended L2 switches for a new IXP

[Mark Tinka]

On Monday, January 12, 2015 05:54:38 PM Bill Woodcock wrote:

 We see a lot of IXPs being formed or upgrading with Cisco
 Nexus 3524 switches, which have 48 1G-10G SFP/SFP+
 physical ports, license-limited to 24 active,
 upgradeable to 48 active.
 
 FWIW, 83% of IXPs have 48 or fewer participants, and 70%
 of IXPs have 24 or fewer participants.  And the failure
 rate of chassis-based switches is _way_ higher than that
 of stand-alone switches.  So we never recommend that an
 IXP buy a switch larger than necessary to accommodate 18
 months reasonably-projectable growth.

Would tend to agree with this approach, and the above.

Multi-rate (i.e., 1Gbps/10Gbps SFP/SFP+) standalone 1U 
switches are reasonable these days. The issue you'll 
probably run into with them is limited support for features 
you find being implemented by larger exchange points (VPLS, 
Sflow, e.t.c.), and quirks with the hardware that could 
impact things like Layer 2 or Layer 3 filtering (especially 
if they are using off-the-self silicon), e.t.c.

Test before you buy, in as far as you can anticipate your 
(growth) needs.

Mark.


signature.asc
Description: This is a digitally signed message part.

RE: Recommended L2 switches for a new IXP

[Tony Wicks]

People seem to be avoiding recommending actual devices, well I would
recommend the Juniper EX4600 -

http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/

They are affordable, highly scalable, stackable and run JunOS.

cheers

Re: Recommended L2 switches for a new IXP

[Mehmet Akcin]

That's what I had recommended him directly ;)

Mehmet 

 On Jan 12, 2015, at 1:41 PM, Tony Wicks t...@wicks.co.nz wrote:
 
 People seem to be avoiding recommending actual devices, well I would
 recommend the Juniper EX4600 -
 
 http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/
 
 They are affordable, highly scalable, stackable and run JunOS.
 
 cheers
 
 
 

Re: Recommended L2 switches for a new IXP

[Christopher Morrow]

On Mon, Jan 12, 2015 at 4:41 PM, Tony Wicks t...@wicks.co.nz wrote:
 People seem to be avoiding recommending actual devices, well I would
 recommend the Juniper EX4600 -

 http://www.juniper.net/us/en/products-services/switching/ex-series/ex4600/

 They are affordable, highly scalable, stackable and run JunOS.

(and you can't do anything worthwhile for acls to protect that device
from the world/ix-users)

Re: Recommended L2 switches for a new IXP

[Mark Tinka]

On Monday, January 12, 2015 11:41:20 PM Tony Wicks wrote:

 People seem to be avoiding recommending actual devices,
 well I would recommend the Juniper EX4600 -
 
 http://www.juniper.net/us/en/products-services/switching/
 ex-series/ex4600/
 
 They are affordable, highly scalable, stackable and run
 JunOS.

We've been quite happy with the EX4550, but the EX4600 is 
good too, particularly if you're coming from its younger 
brother.

Mark.


signature.asc
Description: This is a digitally signed message part.

Re: Recommended L2 switches for a new IXP

[Bill Woodcock]

 On Jan 12, 2015, at 10:34 AM, Justin Wilson - MTIN li...@mtin.net wrote:
 Cost per port is what we always look at.  If we are going into a market where 
 there won’t be much growth we look at Cisco and Force 10.  Their cost per 
 port is usually cheaper for smaller 10 Gig switches. You need something that 
 is fairly robust.

We see a lot of IXPs being formed or upgrading with Cisco Nexus 3524 switches, 
which have 48 1G-10G SFP/SFP+ physical ports, license-limited to 24 active, 
upgradeable to 48 active.

FWIW, 83% of IXPs have 48 or fewer participants, and 70% of IXPs have 24 or 
fewer participants.  And the failure rate of chassis-based switches is _way_ 
higher than that of stand-alone switches.  So we never recommend that an IXP 
buy a switch larger than necessary to accommodate 18 months 
reasonably-projectable growth.

-Bill






signature.asc
Description: Message signed with OpenPGP using GPGMail

Recommended L2 switches for a new IXP

[Manuel Marín]

Dear Nanog community

We are trying to build a new IXP in some US Metro areas where we have
multiple POPs and I was wondering what do you recommend for L2 switches. I
know that some IXPs use Nexus, Brocade, Force10 but I don't personally have
experience with these switches. It would be great if you can share your
experience and recommendations. There are so many options that I don't know
if it makes sense to start with a modular switch (usually expensive because
the backplane, dual dc, dual CPU, etc) or start with a 1RU high density
switch that support new protocols like Trill and that supposedly allow you
to create Ethernet Fabric/Clusters. The requirements are simple, 1G/10G
ports for exchange participants, 40G/100G for uplinks between switches and
flow support for statistics and traffic analysis.

Thank you and have a great day.

Regards

Is there list of IXPs (containing the information of the AS# of the IXP)

[Song Li]

Hi everyone,

I'm searching for a list of IXPS which contains the information of the 
ASN of the IXP. Some resources are good:


https://prefix.pch.net/applications/ixpdir/?show_active_only=0sort=trafficorder=desc
https://www.telegeography.com/products/internet-exchange-directory/profiles-by-name/index.html

but they do not contain the AS# of the IXP. Can anybody help me?

Thanks!

Best!

--
Song Li
Room 4-204, FIT Building,
Network Security,
Department of Electronic Engineering,
Tsinghua University, Beijing 100084, China
Tel:( +86) 010-62446440
E-mail: refresh.ls...@gmail.com

Re: Is there list of IXPs (containing the information of the AS# of the IXP)

[Jeroen Massar]

On 2014-12-22 14:30, Song Li wrote:
 Hi everyone,
 
 I'm searching for a list of IXPS which contains the information of the
 ASN of the IXP. Some resources are good:
 
 https://prefix.pch.net/applications/ixpdir/?show_active_only=0sort=trafficorder=desc
 
 https://www.telegeography.com/products/internet-exchange-directory/profiles-by-name/index.html
 
 
 but they do not contain the AS# of the IXP. Can anybody help me?

IXs themselves do not have ASNs, as they are Layer 2 providers.

The prefixes used for the peering fabric might be part of some ASN
though (eg AMS-IX uses 1200).

Check http://www.peeringdb.com for most likely the info you are really
looking for.

Greets,
 Jeroen

Re: Is there list of IXPs (containing the information of the AS# of the IXP)

[Randy Bush]

 I'm searching for a list of IXPS which contains the information of the 
 ASN of the IXP.

the best source is https://www.peeringdb.com/

[ i was amused to find CIX (http://www.cix.org/, the one which used to be
  in the bay) still in my ix bookmarks. ]

randy

Re: Is there list of IXPs (containing the information of the AS# of the IXP)

[Niels Bakker]

I'm searching for a list of IXPS which contains the information of 
the ASN of the IXP.


* ra...@psg.com (Randy Bush) [Mon 22 Dec 2014, 14:54 CET]:

the best source is https://www.peeringdb.com/


It's not.  Let's take an example, AMS-IX: 
https://www.peeringdb.com/private/exchange_view.php?id=26
That record doesn't say AS1200 anywhere.  You'll have to search for 
Amsterdam Internet Exchange to find AS1200; a search for AMS-IX 
will lead you only to its route server ASN.  There is no way to filter 
participants by IXP as Network Type doesn't offer that option.


Euro-IX will give you most serious IXPs globally.  For example, 
https://www.euro-ix.net/ixps/2-AMS-IX#network is AMS-IX's entry 
and it lists all pertinent information.



-- Niels.

Re: Is there list of IXPs (containing the information of the AS# of the IXP)

[Nick Hilliard]

On 22/12/2014 13:50, Jeroen Massar wrote:
 IXs themselves do not have ASNs, as they are Layer 2 providers.

most modern IXPs will have an ASN for their route server, and possibly a
separate asn for their mgmt infrastructure.

Not sure how useful the mgmt ASN is, although most IXPs will paradoxically
include this on their list of members.

Nick

Re: Is there list of IXPs (containing the information of the AS# of the IXP)

[Song Li]

在 2014/12/22 22:26, Nick Hilliard 写道:

On 22/12/2014 13:50, Jeroen Massar wrote:

IXs themselves do not have ASNs, as they are Layer 2 providers.


most modern IXPs will have an ASN for their route server, and possibly a
separate asn for their mgmt infrastructure.

Not sure how useful the mgmt ASN is, although most IXPs will paradoxically
include this on their list of members.

Nick



Thanks for your help!

I studied all the AS-Path in the routing table (from routeviews and 
RIPE), and found that some ASN of IXPs were included in some AS-Path. I 
think that under normal circumstances they should not appear in the 
AS-Path, hence i want to filter out them.


Best!
--
Song Li
Room 4-204, FIT Building,
Network Security,
Department of Electronic Engineering,
Tsinghua University, Beijing 100084, China
Tel:( +86) 010-62446440
E-mail: refresh.ls...@gmail.com