Need recommendations for high-feature, high-density L3 Switch
We have some aging infrastructure and need to start budgeting next-gen. * The network has several small routers as individual edges to peers, WAN, SIP services. * It has a couple 6509s as Internet edge (full tables, 2 carriers, no transit, simple policies) * It has some Nexus 7K as an aggregation layer for all the server pods * It has some 6509s as a backbone to interconnect the aggregation layers and inter-site links. * We do run VRFs/MPLS across our backbone with L3, L2 and L1 services. Nothing super fancy, but it's a requirement. Two approaches: 1 - Look at ASR9010 (or something similar) to replace all of the above. Pros: It has the density, it has features, port buffers, seems to have good granular virtualization, seems to have a good reputation amongst heavy users. Cons: It is very expensive fully populated and there is some oversubscription on the higher-density cards. 2 - Look at one solution to consolidate all edge routers and a separate solution to consolidate backbone/aggregation. Pros: Less density required at the edge layers, so a cheaper solution is possible; Not requiring full BGP tables and port buffers at the backbone/agg layer widens the selection a LOT considering the number of vendors with high-feature/high-density L3 switches. Cons: Now we are looking at 4 boxes per data center rather than 2. So... Is there something in the same class as the ASR9000s that also have a good reputation? Will need at least 48 ports of 10G, 24x1Gb, limited oversubscription, good feature sets, not astronomically priced. If we can't find the perfect fit, we will just look at two separate solutions. Also... has anyone used a CSR1000v or Vyatta VM-based solution on something like a Pluribus? I know you can run them on any server, but there are vendors like Pluribus who integrate the server hardware with a full-feature physical switch. (Their E68 is the one we are considering) I'm assuming you aren't going to get anywhere near the features and performance of an ASR9010, but... can you get close? Thanks. CWB
Question for service providers regarding tenant use of public IPv4 on your infrastructure
(accidentally sent this to nanog-request earlier, sorry if there is a double post) We are an enterprise and we do not yet have a sophisticated service-provider model yet for billing, capacity-management, or infrastructure consumption. We have a few vBlocks that we consume internally for IT/business needs. Recently, the decision was made to start offering our infrastructure to partner businesses to deploy their apps on, which will then be made available to their customers. The ingress/egress, the virtualization and even the orchestration part are essentially covered. We've tackled the security part as well. However, we have some tenants that want to egress to the internet locally rather than backhaul the traffic to their premise. Naturally, we could ask each tenant to provide their own internet for this, but the business wants to explore a dedicated, customer-only internet and chargeback/showback. My question is: how are cloud providers handling the use of their IP space when they don't have full control over what their tenants are doing? More specifically, if you own a large block of IPs, how do you prevent business impact (or other tenant impact) if one tenant does something that causes an upstream ISP to blacklist/block? We don't want to put more controls in path between the tenant and the internet, we just want to know how to manage upstream relations. I've heard that some ISPs don't block a specific IP when they see malicious behavior; they do a WHOIS and block the whole range. That would, of course, impact multiple tenants. I'm guessing Amazon and other similar providers have some arrangements with peering ISPs and law-enforcement to ensure that there is consultation before action is taken? Or do ISPs put some level of security between their tenants and the internet to prevent this? I've been told that the majority do not have any intelligent filtering beyond bogon-lists. I'd imagine that would cause huge operational overhead and frustrate the tenants. If you've tackled this issue as part of your business, I'd appreciate any feedback. Thanks in advance. CWB This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
IPv6 /48 advertisements
I accidentally sent this to nanog-request yesterday. I could use some feedback from anyone that can help, please. Question: will carriers accept IPv6 advertisements smaller than /48? Our org was approved a /36 based on number of locations. The bulk of those IPs will be in the data centers. As we were chopping up the address space, it was determined that the remote campus locations would be fine with a /60 per site. (16 networks of /64). There are usually less than 50 people at the majority of these locations and only about 10 different functional VLANs (Voice, Data, Local Services, Wireless, Guest Wireless, etc...). Now, there has been talk about putting an internet link in every campus rather than back hauling it all to the data centers via MPLS. However, if we do this, then would we need a /48 per campus? That is massively wasteful, at 65,536 networks per location. Is the /48 requirement set in stone? Will any carriers consider longer prefixes? I know some people are always saying that the old mentality of conserving space needs to go away, but I was bitten by that IPv4 issue back in the day and have done a few VLSM network overhauls. I'd rather not massively allocate unless it's a requirement. Thanks in advance. CWB This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
RE: IPv6 /48 advertisements
I had a feeling... thanks for the feedback. CWB From: Blake Dunlap [mailto:iki...@gmail.com] Sent: Wednesday, December 18, 2013 9:32 AM To: Edward Dore Cc: Cliff Bowles; nanog@nanog.org Subject: Re: IPv6 /48 advertisements Regardless of the carriers, you'll find most ASs on the internet only listen to /48 or larger. So even if you get your prefixes accepted by your provider, don't assume you can get anywhere, or have your packets not fall in to uRPF blackholes randomly without a larger aggregate announcement. -Blake On Wed, Dec 18, 2013 at 10:22 AM, Edward Dore edward.d...@freethought-internet.co.ukmailto:edward.d...@freethought-internet.co.uk wrote: If you're talking about announcing each location separately, then RIPE have a couple of useful articles about prefix visibility on Ripe Labs: https://labs.ripe.net/Members/emileaben/ripe-atlas-a-case-study-of-ipv6-48-filtering https://labs.ripe.net/Members/dbayer/visibility-of-prefix-lengths Otherwise I guess you'll need to talk to your chosen carrier(s) about aggregating your space for you, which will come down to their policies on what routes they will carry internally. Edward Dore Freethought Internet On 18 Dec 2013, at 16:11, Cliff Bowles cliff.bow...@apollogrp.edumailto:cliff.bow...@apollogrp.edu wrote: I accidentally sent this to nanog-request yesterday. I could use some feedback from anyone that can help, please. Question: will carriers accept IPv6 advertisements smaller than /48? Our org was approved a /36 based on number of locations. The bulk of those IPs will be in the data centers. As we were chopping up the address space, it was determined that the remote campus locations would be fine with a /60 per site. (16 networks of /64). There are usually less than 50 people at the majority of these locations and only about 10 different functional VLANs (Voice, Data, Local Services, Wireless, Guest Wireless, etc...). Now, there has been talk about putting an internet link in every campus rather than back hauling it all to the data centers via MPLS. However, if we do this, then would we need a /48 per campus? That is massively wasteful, at 65,536 networks per location. Is the /48 requirement set in stone? Will any carriers consider longer prefixes? I know some people are always saying that the old mentality of conserving space needs to go away, but I was bitten by that IPv4 issue back in the day and have done a few VLSM network overhauls. I'd rather not massively allocate unless it's a requirement. Thanks in advance. CWB This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system. This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
Dynamic routing through firewall
Request for feedback... We have a need for an external partner to dynamically advertise their network to us in two separate data centers. The hitch is that, touching external partners, our edge routers for B2B partners reside in DMZs. Now, to ensure failover from one data center to another when there is a link/device problem, we need to pass dynamic routing updates through each DMZ firewall to core routing at each data center. (yes, the data centers are in sync via backbone routing) We have multiple B2B peers, so we have multiple VRFs on the edge routers that all need to talk to our core routing, but not necessarily with each other... so we are on the hook for both the routing of these tenants as well as the security inspection. The 4 common options we've considered: 1. Routing through the firewall across a tunnel: Pro - relatively simple, Con - occasional MTU issues and the firewalls may not be able to disassemble/reassemble the tunnel packets for policy inpsection. 2. Transparent firewalling: Pro - extremely easy, Con - some vendors cannot support stateful failover in HA pairs, some won't do stateful at all, so you need to open up rules bidirectional (i.e. reply ports GT 1024), plus all the whizbang IDS/IPS goes out the window along with NAT and other stuff, so it will be a very point solution 3. BGP on the firewall: Pro - moderately easy unless the policies get very sophisticated, and firewalls automatically learn where prefixes are automatically; Con - it's BGP on the firewall... neither the network or the security teams are thrilled about it, support calls tend to loop in both teams, a security guy can cause a lot of problems with human error, we've seen some firewalls with memory leak vulnerabilities and experienced issues in the past 4. BGP through the firewall (multihop): Pro - easy to configure, doesn't require routing on the firewall; Con - for every prefix our upstream edge router advertises to our core, we will need statics in the firewall to make sure that it knows where to forward. We can do a default pointing to the edge router with some large summaries pointing back inside (or wherever), but you get the point - the more prefixes that aren't covered by the default, the less scalable it becomes 5. Firewall on a stick: This is untested, but the idea was floated that we could peer the edge router with our core router, but have Policy-routes on every customer VRF setting the next-hop as the firewall. The firewall will apply policy, and (like option 4) use statics to forward to the core. Reverse path traffic would pretty much do the same from Core-to firewall-to edge. It sounds ugly, and we haven't tested it, but we didn't want to toss it. A lot of you work in multi-tenant environments, and some of you are responsible for the security between tenants. I'd like to hear alternatives, if you know of any. And if you use one of the options I mentioned, please say why you chose it and how it works. Finally, if you tried one of the options and it was terrible, please explain. Thanks in advance! CWB This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
RE: NANOG Digest, Vol 66, Issue 60
+1 on the Cyclades (Avocent). We have about 8 of them spread across 3 data centers. We've used them for roughly 4-5 years. Won't say they are intuitive to set up, but once they are, they work well. Don't think I've seen one go down, but then again, they aren't subject to much change. Cost was pretty reasonable, but that's relative to your business/budget. -CWB -Original Message- From: nanog-requ...@nanog.org [mailto:nanog-requ...@nanog.org] Sent: Wednesday, July 24, 2013 8:59 AM To: nanog@nanog.org Subject: NANOG Digest, Vol 66, Issue 60 Send NANOG mailing list submissions to nanog@nanog.org To subscribe or unsubscribe via the World Wide Web, visit http://mailman.nanog.org/mailman/listinfo/nanog or, via email, send a message with subject or body 'help' to nanog-requ...@nanog.org You can reach the person managing the list at nanog-ow...@nanog.org When replying, please edit your Subject line so it is more specific than Re: Contents of NANOG digest... Today's Topics: 1. 48V DC Terminal server recommendations (Jeremy Bresley) 2. Re: 48V DC Terminal server recommendations (William Herrin) 3. RE: 48V DC Terminal server recommendations (Warren Bailey) 4. Re: 48V DC Terminal server recommendations (Michael Brown) 5. Re: 48V DC Terminal server recommendations (Joe Hamelin) 6. Re: 48V DC Terminal server recommendations (Warren Bailey) 7. Re: 48V DC Terminal server recommendations (Ian Goodall) 8. Re: 48V DC Terminal server recommendations (david peahi) 9. Re: 48V DC Terminal server recommendations (Tim Jackson) 10. Re: 48V DC Terminal server recommendations (Jay Ashworth) -- Message: 1 Date: Wed, 24 Jul 2013 09:59:41 -0500 From: Jeremy Bresley b...@brezworks.com To: nanog@nanog.org Subject: 48V DC Terminal server recommendations Message-ID: 51efebdd.4010...@brezworks.com Content-Type: text/plain; charset=ISO-8859-1; format=flowed Looking for recommendations on a good terminal server to put into a telco colocate facility. Requirements: 8-16 ports for Cisco console access (RJ-45s preferred, DB9s if we have to) -48V DC power USB/internal modem for OOB access NEBS Level 1 (or better) compliance. So far I've found Perle has several models that meet 3 out of 4, but none that meet all the requirements. The only OpenGear boxes we're seeing with DC power is a little 4 port unit and they don't mention NEBS compliance. Lantronix mentions DC power for their SLC line, but doesn't mention anything about NEBS compliance either. Anybody have any recommendations for one they've used that meets all 4 of those requirements? Thanks! Jeremy TheBrez Bresley b...@brezworks.com -- Message: 2 Date: Wed, 24 Jul 2013 11:08:46 -0400 From: William Herrin b...@herrin.us To: Jeremy Bresley b...@brezworks.com Cc: nanog@nanog.org Subject: Re: 48V DC Terminal server recommendations Message-ID: CAP-guGX2xHR-z4W=SbfjCa-jEgm+te7P=MoEv5sVmg68fL=f...@mail.gmail.com Content-Type: text/plain; charset=ISO-8859-1 On Wed, Jul 24, 2013 at 10:59 AM, Jeremy Bresley b...@brezworks.com wrote: Looking for recommendations on a good terminal server to put into a telco colocate facility. Requirements: 8-16 ports for Cisco console access (RJ-45s preferred, DB9s if we have to) -48V DC power USB/internal modem for OOB access NEBS Level 1 (or better) compliance. Hi Jeremy, Have you considered the obvious: a Cisco 2610/2811 or equivalent with a 16 port async NM card, an analog modem WIC card and a DC power supply? I didn't see it on the list of things you'd considered. Regards, Bill Herrin -- William D. Herrin her...@dirtside.com b...@herrin.us 3005 Crane Dr. .. Web: http://bill.herrin.us/ Falls Church, VA 22042-3004 -- Message: 3 Date: Wed, 24 Jul 2013 15:14:53 + From: Warren Bailey wbai...@satelliteintelligencegroup.com To: Jeremy Bresley b...@brezworks.com, nanog@nanog.org nanog@nanog.org Subject: RE: 48V DC Terminal server recommendations Message-ID: f5wg5o0me19ke4h9frdxww4w.137467...@email.android.com Content-Type: text/plain; charset=iso-8859-1 Uplogix. Sent from my Mobile Device. Original message From: Jeremy Bresley b...@brezworks.com Date: 07/24/2013 8:02 AM (GMT-08:00) To: nanog@nanog.org Subject: 48V DC Terminal server recommendations Looking for recommendations on a good terminal server to put into a telco colocate facility. Requirements: 8-16 ports for Cisco console access (RJ-45s preferred, DB9s if we have to) -48V DC power USB/internal modem for OOB access NEBS Level 1 (or better) compliance. So far I've found Perle has several models that meet 3 out of 4, but none that meet all the requirements. The only OpenGear boxes we're seeing with DC power is a little 4 port unit and they don't mention NEBS compliance. Lantronix
RE: NANOG Digest, Vol 65, Issue 74
As stated, every vendor has its merits. If you really put some time into developing a list of requirements and then structure a bakeoff that tests those, you will learn a lot. Some things to think about: * don't let JUNOS or any other CLI deter you. You just need to factor in training and hiring efforts/costs. We switched to Juniper for 50+ campus routers (haven't used their switches yet) because they had way better bang for the buck. The engineers that whined about it not being Cisco were not the ones I cared to keep. The engineers that went out and learned JUNOS then slapped it on their resume were, by far, the more reliable and skilled engineers. Also, when you are hiring, I bet that you will find that engineers with substantial experience in other platforms will also perform very well on the technical interviews. They will probably know advanced BGP, MPLS, tunneling, multicast, QOS and other stuff that your average interviewee does not. It's a mindset. *politics: we replaced a large section of our network with Foundry (a price-per-port) decision. They worked as well as any vendor out there, but their support was... not polished as Cisco or Juniper. But the real problem came from the low level support engineers who had a CCNA and were Cisco-oriented. Now, when we had Cisco blade/power/code failures, it was a network failure. When the Foundry had a problem, it was a Foundry failure. I watched a huge outage due to a poor spanning tree design get branded as a Foundry issue. Management hears this enough and eventually we are told to replace the Foundry switches. I pulled ticket logs and proved that the support team had nearly twice the amount of open tickets and logged failures with Cisco as they did with Foundry, but it didn't matter. *politics again: If you are a big cisco shop and you decide to use another vendor somewhere, I GUARANTEE that a regional sales VP and some ducklings in suits will soon walk directly into the CIO's office. They will argue that the bakeoff was skewed, that price-per-port value doesn't factor in a lot of other value that cisco brings, they will even question the skillset of your engineers who performed the bakeoff, etc... they will instill Fear-Uncertainty-Doubt. They will offer another 2 or 3 % discount, they will throw in free professional services, and so on. Hell, they may put a Cisco employee on your board of directors. Short story - if there's a lot of money involved, you may wind up back with Cisco. I've seen it more than once That being said, I don't dislike Cisco at all. Their support is top notch and their training is pretty good. They take good care of their clients. A LOT of their products are good... some are not. But I did want to prepare you for the fun if you seriously consider another vendor. We have selected Mellanox for a small data warehouse, but that was a point solution due to the Infiniband requirements. We have selected Arista for a large Hadoop deployment. So far, they are a great product and a great value. Support seems good, but we haven't called them much yet. That's a good thing. One other thing to consider is future state and emerging technologies. If you are an architect or if you work with architecture to obtain design direction, ask about future needs for multi-tenancy, SDN, automation and such. I think you'll find that not only is Arista way out ahead of some vendors with this, they are using Open source code, more or less. Cisco has onePK, but their automation and API integration is not only proprietary, it's misleading. I haven't seen the other vendor solutions yet, so I can't say who is BEST at automation, orchestration, and SDN... So... determine what's important to your network today and in 3-5 years, then look at what's being offered. cwb -Original Message- From: nanog-requ...@nanog.org [mailto:nanog-requ...@nanog.org] Sent: Tuesday, June 18, 2013 8:18 PM To: nanog@nanog.org Subject: NANOG Digest, Vol 65, Issue 74 Send NANOG mailing list submissions to nanog@nanog.org To subscribe or unsubscribe via the World Wide Web, visit http://mailman.nanog.org/mailman/listinfo/nanog or, via email, send a message with subject or body 'help' to nanog-requ...@nanog.org You can reach the person managing the list at nanog-ow...@nanog.org When replying, please edit your Subject line so it is more specific than Re: Contents of NANOG digest... Today's Topics: 1. Re: Network Vendor suggestions/reviews, Arista Networks, Dell Force10, Juniper, Extreme Networks etc... (Phil Fagan) 2. Re: Network Vendor suggestions/reviews, Arista Networks, Dell Force10, Juniper, Extreme Networks etc... (Mike Hale) 3. Re: Network Vendor suggestions/reviews, Arista Networks, Dell Force10, Juniper, Extreme Networks etc... (Phil Fagan) 4. Re: Network Vendor suggestions/reviews, Arista Networks, Dell Force10, Juniper, Extreme Networks etc... (Brent
BGP AS question
Greetings. We have a few facilities within a 30 mile radius, and each has an ISP link. We use P2P links at the edge to make certain traffic sourcing from one facility, and destined to the Public IPs at another, stay on the dirty links rather than punting out to the ISP. All sites use the same BGP AS. Recently, we were required to turn up an additional facility in a short time frame. It also uses the same BGP AS. However, it does not have a dirty cross-connect link. So, even though this facility has unique /24 public IP blocks, it still has the same AS. One thing we are noticing is that some ISPs don't seem to have a problem allowing this traffic, and some do. I suspect some don't like traffic with the same source and destination BGP AS, even though the prefixes are different at each location. But other ISPs seem to permit this with no problem. My question is: is normal BGP default behavior to permit or to allow this type of traffic? Also, would it be easier to ask the ISP to make an exception, or to buy another AS for the rogue facility? Thanks. Clifford W Bowles, Technical Director Apollo Group | IT Services | Network Engineering 4025 S. Riverpoint Parkway | CF-C201 | Phoenix, AZ 85040 phone: 602-557-6762 | fax: 602-557-6607 | email: cliff.bow...@apollogrp.edumailto:cliff.bow...@apollogrp.edu This message is private and confidential. If you have received it in error, please notify the sender and remove it from your system.
