Re: BGPMON Alert Questions
So we're somewhat safe until the fast food burger grills and fries cookers advance to level-3 routing? Or Daquiri blenders get their own ASNs? that happened in the late '90s Bad enough that professional folks can goof to this extent luckily, you, valdis, and i never make mistakes :) the point it to engineer the network so we are not affected by the inevitable mistakes as chris and i were noting privately, this seems not to have damaged a lot of traffic, more than compensated for by the traffic on nanog :) randy
Re: BGPMON Alert Questions
On Thu, 03 Apr 2014 15:00:41 +0900, Randy Bush said: Bad enough that professional folks can goof to this extent luckily, you, valdis, and i never make mistakes :) You must have me confused with somebody else. I wouldn't have a world-wide reputation for getting myself out of holes I've dug if I wasn't incredible at hole digging in the first place. :) pgpEI9muX7POb.pgp Description: PGP signature
Re: BGPMON Alert Questions
On 3 April 2014 04:43, Randy Bush ra...@psg.com wrote: i very much doubt this is a 7007, where bgp was redistributed into rip, which sliced it into a jillion /24s, and then redistributed from rip back into bgp. I could be wrong, but I thought AS7007 was nothing to do with RIP? http://www.merit.edu/mail.archives/nanog/1997-04/msg00444.html M
Re: BGPMON Alert Questions
On Wednesday, April 02, 2014 08:59:58 PM Justin M. Streiner wrote: It's pretty clear that both parties have dropped the ball in a big way, in terms of sane BGP filtering practices. It's amazing, isn't it? I have a customer of one my upstreams (Upstream A), at the moment, who are leaking my routes to another one of their upstreams (Upstream B). The problem is that Upstream B is re-announcing my leaked routes from their customer to the rest of the Internet. So both Upstream B's customer as well as Upstream B are at fault. That Upstream B is simply accepting everything their customer is sending to them without applying proper filters, or checking to confirm that what their customer needs to send them should come from them is absolutely and unacceptably shocking! A lot of people seem to have forgotten 2008. Mark. signature.asc Description: This is a digitally signed message part.
Re: BGPMON Alert Questions
On 4/2/2014 11:30 PM, Barry Greene wrote: Hi Team, Confirmation from my team talking directly to Indosat - self inflected with a bad update during a maintenance window. Nothing malicious or intentional. Barry Did you get any details on what specifically went wrong? I don't recall any switch in my routing gear to re-originate every prefix on the planet as my own.
Re: BGPMON Alert Questions
On 03/04/2014 13:09, ML wrote: Did you get any details on what specifically went wrong? I don't recall any switch in my routing gear to re-originate every prefix on the planet as my own. Easy enough to do by e.g. redistributing your ebgp into your IGP and then back again, or by a variety of other means. It happened between 05:00 and 06:00 local time, so it's reasonable to assume that it was maintenance gone wrong. Horribly wrong. Nick
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 02:22:44 AM Randy Bush wrote: and, btw, how many of those whose prefixes were mis-originated had registered those prefixes in the rpki? It is probably a bit of a hammer at this stage, but we are in limited deployment of dropping all Invalids using RPKI. We shall be rolling out, network-wide, in 2014, where all Invalids are dropped. At this stage, short of a mis- origination, it's mostly longer prefixes of an aggregate that are not ROA'd. I was asleep when Indosat was mis-originating, but it'd have been nice to see what our test-bed was doing to any Indosat- injected prefixes that have ROA's. Mark. signature.asc Description: This is a digitally signed message part.
Re: BGPMON Alert Questions
It is probably a bit of a hammer at this stage, but we are in limited deployment of dropping all Invalids using RPKI. We shall be rolling out, network-wide, in 2014, where all Invalids are dropped. At this stage, short of a mis- origination, it's mostly longer prefixes of an aggregate that are not ROA'd. sadly, my (legacy) address space is in the arin region. and arin does not see allowing me to protect my prefixes from mis-origination as a serious goal. randy
Re: BGPMON Alert Questions
I wonder who we should be going after here? Indosat or their upstream? Probably both, since if this happened with an ISP deeper in the Internet core, chances are they don't have what our concept of an upstream is. you want revenge or to prevent the effects of recurrence? one nice thing about origin validation is that anyone who validates anywhere on the internet can reject the mis-origination(s). randy
Re: BGPMON Alert Questions
Was a specific Upstream at fault or several upstream providers? It appears they have 9 upstream links -- http://www.cidr-report.org/cgi-bin/as-report?as=4761 On 4/3/2014 8:41 AM, Mark Tinka wrote: I wonder who we should be going after here? Indosat or their upstream?
Re: BGPMON Alert Questions
On 03/04/2014 13:41, Mark Tinka wrote: max-prefix could have come in handy here. But this is an old song (let alone prefix filtering or RPKI). I'm currently seeing ~100 prefixes originating from 4761, and an additional 725 transited through 4761. This would not be difficult to handle with prefix lists, assuming some level of automation. Nick
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 02:51:20 PM Randy Bush wrote: you want revenge or to prevent the effects of recurrence? I'd like to consider targeted suggestions for fixes that address the specific challenges affecting seasoned upstreams vs. their downstream customers. I can understand how an ISP with relatively little experience can mess this up (and glad to help here to educate wherever possible). But if an established provider is still struggling with this, why is that? one nice thing about origin validation is that anyone who validates anywhere on the internet can reject the mis-origination(s). +1. Mark. signature.asc Description: This is a digitally signed message part.
RE: BGPMON Alert Questions
We have a registered prefix that was affected. The RPKI may have helped though; only one BGPMON peer saw the mis-originated route. Much better than being on the 10+ list. -Original Message- From: Randy Bush [mailto:ra...@psg.com] Sent: Wednesday, April 02, 2014 7:23 PM To: North American Network Operators' Group Subject: Re: BGPMON Alert Questions note joels careful use of 'injected'. imiho, 'hijacked' is perjorative implying evil intent. i very much doubt that is the case here. it looks much more like an accident. could we try to be less accusatory with our language. 'injected', 'mis-originated', ... would seem to descrive the situation. and, btw, how many of those whose prefixes were mis-originated had registered those prefixes in the rpki? randy This message and any attachments should be treated as confidential information of Griffin Technology, Inc.
Re: BGPMON Alert Questions
On Thu, Apr 3, 2014 at 9:15 AM, Mark Tinka mark.ti...@seacom.mu wrote: On Thursday, April 03, 2014 02:51:20 PM Randy Bush wrote: you want revenge or to prevent the effects of recurrence? I'd like to consider targeted suggestions for fixes that address the specific challenges affecting seasoned upstreams vs. their downstream customers. at this point it's hard to come up with a suggestion aside from: stop being negligent :( if after so many incidents and so many years, and seeing so many of your friends trip on the stairs and break an arm, you'd think providers would route-filter their customers just to avoid going to the hospital. I can understand how an ISP with relatively little experience can mess this up (and glad to help here to educate wherever possible). But if an established provider is still struggling with this, why is that? I'm going to guess: 1) who's going to pay for the filtering setup work? 2) we have always done it this way... why change? 3) adrenaline rush? one nice thing about origin validation is that anyone who validates anywhere on the internet can reject the mis-origination(s). +1. Mark.
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 02:57:31 PM Nick Hilliard wrote: I'm currently seeing ~100 prefixes originating from 4761, and an additional 725 transited through 4761. This would not be difficult to handle with prefix lists, assuming some level of automation. Indeed. I, for example, have an upstream that filters only on AS_PATH. Naturally, we are quite aggressive and insistent about filtering both on AS_PATH and prefix list across interconnects to our downstreams, but if things were to blow up on our side, the upstream in question would not be protected (unless, of course, they are relying on max- prefix as well). Mark. signature.asc Description: This is a digitally signed message part.
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 02:52:16 PM Anthony Williams wrote: Was a specific Upstream at fault or several upstream providers? It appears they have 9 upstream links -- http://www.cidr-report.org/cgi-bin/as-report?as=4761 There probably won't be only one provider at fault. It could be all an ISP's providers are at fault, or it could be that two providers along a single AS_PATH are simultaneously at fault. It's a big weakness of our Internet, but we still need to figure out the best way to fix it, until, at least, RPKI is more widely adopted. At this stage, it appears education, and implementation of that education, is our only recourse. But how do we do this at scale? Mark. signature.asc Description: This is a digitally signed message part.
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 03:55:11 PM Christopher Morrow wrote: I'm going to guess: 1) who's going to pay for the filtering setup work? Well, your customers are paying you to ensure they don't get cut off due to your negligence. You also don't want to become a watch-out-for-that-one peer within the community. But, perhaps those two ideals are not significant motivation for change :-\. 2) we have always done it this way... why change? This is probably a more endemic issue of our industry, where operators find it hard to keep up with the times (there is no shortage of -bis or BCP documents) through actual useful implementation (BCP-38) vs. talk (SDN hype). In the case of nailing routing filters for customers, one thought that comes to mind is if your organization is large enough, throw a warm body at the issue. There are lots of interns or folk you can hire on a temporary basis to focus on cleaning all this up, and getting the NOC trained and clued up on the new strategy. The new strategy is not just shiny, it could actually save you loss of customers and community respect. But that's just me... Mark. signature.asc Description: This is a digitally signed message part.
Re: BGPMON Alert Questions
On Thu, Apr 3, 2014 at 11:05 AM, Mark Tinka mark.ti...@seacom.mu wrote: On Thursday, April 03, 2014 03:55:11 PM Christopher Morrow wrote: I'm going to guess: 1) who's going to pay for the filtering setup work? Well, your customers are paying you to ensure they don't get cut off due to your negligence. I think you mean they are paying me to carry their bits across the network... and they are paying me to do it with minimal hassle to THEM... telling me prefixes to add to their list is hassle. You also don't want to become a watch-out-for-that-one peer within the community. sure... not sure how much that matters to higher-ups? there's no such thing as bad PR, right? But, perhaps those two ideals are not significant motivation for change :-\. apparently they are not. 2) we have always done it this way... why change? This is probably a more endemic issue of our industry, where operators find it hard to keep up with the times (there is no shortage of -bis or BCP documents) through actual useful implementation (BCP-38) vs. talk (SDN hype). In the case of nailing routing filters for customers, one thought that comes to mind is if your organization is large enough, throw a warm body at the issue. There are lots of interns or folk you can hire on a temporary basis to focus on cleaning all this up, and getting the NOC trained and there's a salient point about training time and internal systems complexity to keep in mind here as well :( clued up on the new strategy. The new strategy is not just shiny, it could actually save you loss of customers and community respect. agreed. But that's just me... it's not just you.
Re: BGPMON Alert Questions
On Thursday, April 03, 2014 05:13:40 PM Christopher Morrow wrote: I think you mean they are paying me to carry their bits across the network... and they are paying me to do it with minimal hassle to THEM... telling me prefixes to add to their list is hassle. Agree - but, as an operator, that is my problem. Not my customer's problem. there's a salient point about training time and internal systems complexity to keep in mind here as well :( The ground is littered with pot holes. They are everywhere you turn. Mark. signature.asc Description: This is a digitally signed message part.
Cisco warranty
Hi, I bought a C3750G-12S which is now end of sale on cisco website. This device is now defective. Since I bought it from a reseller and not directly from cisco, cisco is refusing to take it under warranty and tells me to have the reseller take care of it. The reseller doesnt wan't to hear about this device since it is end of sale. According to cisco website, end of sale means the device is still covered for 5 years. My question is: Is it normal for my supplier to refuse to take it under warranty ? Is there (from your experience) a chance I might get cisco to deal with it ? Thanks Laurent
Re: Cisco warranty
On 14-04-03 12:44 PM, Laurent CARON wrote: I bought a C3750G-12S which is now end of sale on cisco website. This device is now defective. Since I bought it from a reseller and not directly from cisco, cisco is refusing to take it under warranty and tells me to have the reseller take care of it. The reseller doesnt wan't to hear about this device since it is end of sale. Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. According to cisco website, end of sale means the device is still covered for 5 years. This is not base warranty - this is potential coverage. Base warranty is 90 days: http://www.cisco.com/go/warranty My question is: Is it normal for my supplier to refuse to take it under warranty? See above. Is there (from your experience) a chance I might get cisco to deal with it ? Not likely. Specific information for this product's EOL is here: http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/eol_c51-696372.html You'll need to have a service contract associated with the device (SMARTnet). Unfortunately for you, from that page: End of New Service Attachment Date: January 30, 2014 For equipment and software that is not covered by a service-and-support contract, this is the last date to order a new service-and-support contract or add the equipment and/or software to an existing service-and-support contract. So if you don't already have SMARTnet, you're probably out of luck. M. -- Michael Brown| The true sysadmin does not adjust his behaviour Systems Administrator| to fit the machine. He adjusts the machine mich...@supermathie.net | until it behaves properly. With a hammer, | if necessary. - Brian
Re: Cisco warranty
Another point to consider when purchasing Cisco gear - there is a difference between a reseller, and an Authorized Reseller. Without going into specifics, I had experience with a previous customer that purchased Cisco gear from a reseller - all On Thu, Apr 3, 2014 at 1:26 PM, Michael Brown mich...@supermathie.netwrote: On 14-04-03 12:44 PM, Laurent CARON wrote: I bought a C3750G-12S which is now end of sale on cisco website. This device is now defective. Since I bought it from a reseller and not directly from cisco, cisco is refusing to take it under warranty and tells me to have the reseller take care of it. The reseller doesnt wan't to hear about this device since it is end of sale. Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. According to cisco website, end of sale means the device is still covered for 5 years. This is not base warranty - this is potential coverage. Base warranty is 90 days: http://www.cisco.com/go/warranty My question is: Is it normal for my supplier to refuse to take it under warranty? See above. Is there (from your experience) a chance I might get cisco to deal with it ? Not likely. Specific information for this product's EOL is here: http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/eol_c51-696372.html You'll need to have a service contract associated with the device (SMARTnet). Unfortunately for you, from that page: End of New Service Attachment Date: January 30, 2014 For equipment and software that is not covered by a service-and-support contract, this is the last date to order a new service-and-support contract or add the equipment and/or software to an existing service-and-support contract. So if you don't already have SMARTnet, you're probably out of luck. M. -- Michael Brown| The true sysadmin does not adjust his behaviour Systems Administrator| to fit the machine. He adjusts the machine mich...@supermathie.net | until it behaves properly. With a hammer, | if necessary. - Brian -- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
Re: Cisco warranty
Another point to consider when purchasing Cisco gear - there is a HUGE difference between a reseller, and an Authorized Reseller. Without going into specifics, I had experience with a previous customer that purchased Cisco gear from a reseller - all refurbished/grey-market that Cisco had as located over in Europe. When the time came, Cisco didn't honor the warranty on the gear. A good authorized reseller should help you - not necessarily exchange the unit if you haven't purchased SmartNet, but give you something more than go call Cisco. Cisco generally takes a dim view of resellers treating it's customers that way. Again, first hand experience... On Thu, Apr 3, 2014 at 1:46 PM, Steven Fischer sfischer1...@gmail.comwrote: Another point to consider when purchasing Cisco gear - there is a difference between a reseller, and an Authorized Reseller. Without going into specifics, I had experience with a previous customer that purchased Cisco gear from a reseller - all On Thu, Apr 3, 2014 at 1:26 PM, Michael Brown mich...@supermathie.netwrote: On 14-04-03 12:44 PM, Laurent CARON wrote: I bought a C3750G-12S which is now end of sale on cisco website. This device is now defective. Since I bought it from a reseller and not directly from cisco, cisco is refusing to take it under warranty and tells me to have the reseller take care of it. The reseller doesnt wan't to hear about this device since it is end of sale. Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. According to cisco website, end of sale means the device is still covered for 5 years. This is not base warranty - this is potential coverage. Base warranty is 90 days: http://www.cisco.com/go/warranty My question is: Is it normal for my supplier to refuse to take it under warranty? See above. Is there (from your experience) a chance I might get cisco to deal with it ? Not likely. Specific information for this product's EOL is here: http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/eol_c51-696372.html You'll need to have a service contract associated with the device (SMARTnet). Unfortunately for you, from that page: End of New Service Attachment Date: January 30, 2014 For equipment and software that is not covered by a service-and-support contract, this is the last date to order a new service-and-support contract or add the equipment and/or software to an existing service-and-support contract. So if you don't already have SMARTnet, you're probably out of luck. M. -- Michael Brown| The true sysadmin does not adjust his behaviour Systems Administrator| to fit the machine. He adjusts the machine mich...@supermathie.net | until it behaves properly. With a hammer, | if necessary. - Brian -- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy -- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
Re: Cisco warranty
Back about ten years and three companies ago when I was a baby System Administrator, I made that mistake. Suffice it to say that I HIGHLY recommend looking for the authorized reseller (and getting SMARTNet up-front--if nothing else, that process will generally reliably inform you as to Cisco's opinion of your reseller and/or the gear they're selling) -- Josh Sholes On 4/3/14, 1:46 PM, Steven Fischer sfischer1...@gmail.com wrote: Another point to consider when purchasing Cisco gear - there is a difference between a reseller, and an Authorized Reseller. Without going into specifics, I had experience with a previous customer that purchased Cisco gear from a reseller - all On Thu, Apr 3, 2014 at 1:26 PM, Michael Brown mich...@supermathie.netwrote: On 14-04-03 12:44 PM, Laurent CARON wrote: I bought a C3750G-12S which is now end of sale on cisco website. This device is now defective. Since I bought it from a reseller and not directly from cisco, cisco is refusing to take it under warranty and tells me to have the reseller take care of it. The reseller doesnt wan't to hear about this device since it is end of sale. Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. According to cisco website, end of sale means the device is still covered for 5 years. This is not base warranty - this is potential coverage. Base warranty is 90 days: http://www.cisco.com/go/warranty My question is: Is it normal for my supplier to refuse to take it under warranty? See above. Is there (from your experience) a chance I might get cisco to deal with it ? Not likely. Specific information for this product's EOL is here: http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-s eries-switches/eol_c51-696372.html You'll need to have a service contract associated with the device (SMARTnet). Unfortunately for you, from that page: End of New Service Attachment Date: January 30, 2014 For equipment and software that is not covered by a service-and-support contract, this is the last date to order a new service-and-support contract or add the equipment and/or software to an existing service-and-support contract. So if you don't already have SMARTnet, you're probably out of luck. M. -- Michael Brown| The true sysadmin does not adjust his behaviour Systems Administrator| to fit the machine. He adjusts the machine mich...@supermathie.net | until it behaves properly. With a hammer, | if necessary. - Brian -- To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy
Re: BGPMON Alert Questions
On Thu, Apr 3, 2014 at 11:13 AM, Christopher Morrow morrowc.li...@gmail.com wrote: On Thu, Apr 3, 2014 at 11:05 AM, Mark Tinka mark.ti...@seacom.mu wrote: On Thursday, April 03, 2014 03:55:11 PM Christopher Morrow wrote: I'm going to guess: 1) who's going to pay for the filtering setup work? Well, your customers are paying you to ensure they don't get cut off due to your negligence. I think you mean they are paying me to carry their bits across the network... and they are paying me to do it with minimal hassle to THEM... telling me prefixes to add to their list is hassle. I know this old saw and sales people will apply pressure to Ops if their customers balk at the extra overhead. The time is now to push back, hard, against that practice. I realize you know this, Chris but are trying to characterize the mindset. The new strategy is not just shiny, it could actually save you loss of customers and community respect. agreed. But that's just me... it's not just you Yes, let's seize the bull by the horns. Tony
Re: Cisco warranty
On Thu, Apr 03, 2014 at 01:26:58PM -0400, Michael Brown wrote: Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. This is not true. Cisco provides a limited lifetime warranty on hardware purchased from them or an authorized reseller, with our without SmartNet. For an example, browse to http://www.cisco-servicefinder.com/warrantyfinder.aspx and look up specific products. I looked up an WS-C3750G-48TS-S, and while the warranty does not cover support (TAC contracts do), there is a lifetime warranty good for 5 years from EoS with a 10 business day turnaround. -- Brandon Ewing(nicot...@warningg.com) pgpDB77tdAZ6Z.pgp Description: PGP signature
Re: BGPMON Alert Questions
On Thu, Apr 3, 2014 at 2:31 PM, Tony Tauber ttau...@1-4-5.net wrote: On Thu, Apr 3, 2014 at 11:13 AM, Christopher Morrow morrowc.li...@gmail.com wrote: I know this old saw and sales people will apply pressure to Ops if their customers balk at the extra overhead. The time is now to push back, hard, against that practice. I realize you know this, Chris but are trying to characterize the mindset. I agree with you (both tony and mark)... the mindset was the point, and getting over that is certainly something we all should do. -chris
Starting a greenfield carrier backbone network that can scale to national and international service. What would you do?
Hello everyone, It's been some time since I've been subscribed/replied/posted here (or on WISPA for that matter). I've been pretty busy running a non profit startup (protip: don't do that. It's really really terrible) :) I'm cofounder and CTO of the Free Networking Foundation. Our goal is to bring broadband (5 mbps symmetric to start) bandwidth to the 2/3 of Americans who currently can't get it (rural, urban core, undeserved, $ILEC stops on otherside of street etc). Efforts so far primarily have consisted of WiFI last (square) mile delivery using Ubiquiti hardware and the qmp.cat firmware (also meraki access points that were donated, for some reason this seems to happen quite a bit). We've helped numerous networks get started, grow and (soon we hope) become self sustaining in Austin, Kansas City, Los Angeles, Detroit, New York and a few other places throughout the US. The networks are in various stages of maturity of course, but a number of them are fully operational and passing real traffic. Especially the one in Kansas City (it spans both states). These are (point to point, routed) access/distribution networks which connect into colocation providers blended networks. So that's the background and current state of affairs. Not really NANOG material. The next step is to secure our v6 space and AS number. Now that's not horribly difficult or really worthy of NANOG (though I do greatly appreciate folks on the list who helped me through the theory/practice of that process sometime ago). It appears to be fairly straightforward if you are not an LIR. Simply go through the paperwork (LOA, submit to ARIN, get out the credit card, textbook BGP config and done). And if FNF was operating the networks (we don't, we just help with organizing/consulting/software guidance/hardware spend optimization/logistics etc) and if there was just one POP (and associated administrative body), then again it wouldn't be that interesting or worth cluttering up NANOG. FNF goal is to serve as an LIR, SWIPing out /48 chunks to neighborhood level operators. They would then peer with whatever upstream ISPs are regionally close and announce out the space. This of course would be associated with a training program, registration in an IPAM tool etc. Regarding the above? What do the operators on this list wish they could of been trained in starting out? I mean obviously they should have good mastery and working experience of CCNA level material, along with exposure to higher level concepts of WAN networking. What are the tricks, the gotchas, the man that would of saved my company a million bucks in transit costs. Yes I realize these sort of things are usually closely held. I also am striving to create an entirely new breed of operators running BGP enabled sites with ipv6. The more I can do to help ease those folks integration into the internet, the better. In short, the often debated issue on this list of v6 endpoint explosion is going to be very very very real. What IPAM tools out there can scale to a multi hundred million node, distributed, eventual consistency national level? (I've been working closely with guifi.net, and we are attempting to relaunch that as a very slick Apple like experience with a libremap (couchdb based) system. I'd love to hear from folks across the spectrum of experience and network size. From folks who have been dual homed for ~1 year at a single site, to tier1 operators who were there when it all started. So what would you like to see done in a greenfield, open source, open governance carrier backbone network? What would a dream TIER1 (and I use that in the default free zone sense of the word) look like to you? Also how the heck would one get this bootstrapped at a sustainable pace? Would one create numerous tier2 regional carriers, and they would feed into an over arching tier1? I'm thinking something like a 501c8 type structure ( http://www.irs.gov/Charities--Non-Profits/Other-Non-Profits/Fraternal-Societies[1] ) As far as I know, this is the first time that an intentional community type approach is taken and a tier1 is the end goal. Not evolving into one, buying ones way into it, but a manifest destiny type approach to building a backbone. Please feel free to reach out to me directly (char...@thefnf.org[2] ) if you wish to have a one on one discussion. In particular I'm interested in legal expertise in these sort of areas (law/compliance/contracting/negotiations for right of way etc etc etc). Thanks for reading. I look forward to the discussion! PS: Yes, I'm young and idealistic. I'm also grounded/practical/focused. I'm currently working on making the access portion of the network as smooth and turnkey as possible. (That basically means packaging up zeroshell/observium/powerdns/libremap/trigger and other bits/bobs into a nice livecd/ova/openvz package). I also like to think about the next wave of issues while working on
Recommendation on NTP appliances/devices
Anyone have recommendations on NTP appliances; i.e. make, model, gps vs cell, etc.? Roof/outdoor/window access not available. Would ideally need to be able to handle bursts of up to a few thousand simultaneous queries. Needs IPv6 support. Thanks!
Anyone from AS577 and AS852 in the house?
Bell and Telus, if you're listening - I need to inquire about BGP community support on your respective networks that cannot be addressed by info published in RADB, by our assigned AM, SE, your NOC or any support documentation on your respective websites on the subject. Please hit me up off-list. Thanks in advance!
Re: Recommendation on NTP appliances/devices
On Thu, Apr 03, 2014 at 06:55:02PM -0400, David Hubbard wrote: Anyone have recommendations on NTP appliances; i.e. make, model, gps vs cell, etc.? Roof/outdoor/window access not available. Would ideally need to be able to handle bursts of up to a few thousand simultaneous queries. Needs IPv6 support. Without roof access I'd suggest CDMA instead of GPS: http://www.endruntechnologies.com/ntp-server.htm Appears to fit your requirements. --msa
Re: BGPMON Alert Questions
one nice thing about origin validation is that anyone who validates anywhere on the internet can reject the mis-origination(s). +1. a non-op sec person who follows nanog in read-only mode pointed out in private email that this is a subtle difference from prefix filtering. in general, i can not prefix filter N hops away. randy
Re: Cisco warranty
On Thu, Apr 3, 2014 at 1:46 PM, Brandon Ewing nicot...@warningg.com wrote: On Thu, Apr 03, 2014 at 01:26:58PM -0400, Michael Brown wrote: Did you purchase SMARTnet when you bought the device? If you didn't, you're probably SOL. This is not true. Cisco provides a limited lifetime warranty on hardware purchased from them or an authorized reseller, with our without SmartNet. On some: not all their hardware, they offer limited lifetime warranty. Lifetime is the exception to the rule: many of their components are 90 days or 1 year. The limited bit is also important --- they have restrictions in fine print. It's strongly recommended you buy their SmartNet, if you want their reps to treat you reasonably and make efforts to fulfill your paper warranty. Getting the manufacturer rep to actually honor the paper warranty and allow you an RMA, when there is no paid support is another thing altogether. May require a great deal of persistence on your part, As in continuing to contact Cisco and refusing to take NO as an acceptable answer to your RMA request. Or it may just not happen For an example, browse to http://www.cisco-servicefinder.com/warrantyfinder.aspx and look up specific products. I looked up an WS-C3750G-48TS-S, and while the warranty does not cover support (TAC contracts do), there is a lifetime warranty good for 5 years from EoS with a 10 business day turnaround. -- Brandon Ewing( nicot...@warningg.com) -- -JH
Re: Recommendation on NTP appliances/devices
We have symmetricom (now microsemi) and are very happy with them, but we use the roof mounted gps antennas. They will peer with public ntp severs if that would work for you. David Hubbard dhubb...@dino.hostasaurus.com wrote: Anyone have recommendations on NTP appliances; i.e. make, model, gps vs cell, etc.? Roof/outdoor/window access not available. Would ideally need to be able to handle bursts of up to a few thousand simultaneous queries. Needs IPv6 support. Thanks!
Re: BGPMON Alert Questions
Good point, which makes me ask: So which 5 to 10 networks, implementing source validation, could result in the greatest coverage or protection for the largest part of the Internet to the best of my knowledge, no one has looked at this for origin validation. sharon goldberg and co-conspirators have done a lot of work in the area, see her pubs at https://www.cs.bu.edu/~goldbe/. but the concentration seems to be on bgpsec which deploys quite differently randy
Re: Starting a greenfield carrier backbone network that can scale to national and international service. What would you do?
Let's start with your basic assumption here. Why would you build a backbone at all if your goal is to solve last mile problems? It seems to me that the expense and distraction of building a large backbone network doesn't contribute to your goals at all, given that there are many high quality, nationwide backbone networks in North America today available at reasonable cost. On Thu, 3 Apr 2014, char...@thefnf.org wrote: Hello everyone, It's been some time since I've been subscribed/replied/posted here (or on WISPA for that matter). I've been pretty busy running a non profit startup (protip: don't do that. It's really really terrible) :) I'm cofounder and CTO of the Free Networking Foundation. Our goal is to bring broadband (5 mbps symmetric to start) bandwidth to the 2/3 of Americans who currently can't get it (rural, urban core, undeserved, $ILEC stops on otherside of street etc). Efforts so far primarily have consisted of WiFI last (square) mile delivery using Ubiquiti hardware and the qmp.cat firmware (also meraki access points that were donated, for some reason this seems to happen quite a bit). We've helped numerous networks get started, grow and (soon we hope) become self sustaining in Austin, Kansas City, Los Angeles, Detroit, New York and a few other places throughout the US. The networks are in various stages of maturity of course, but a number of them are fully operational and passing real traffic. Especially the one in Kansas City (it spans both states). These are (point to point, routed) access/distribution networks which connect into colocation providers blended networks. So that's the background and current state of affairs. Not really NANOG material. The next step is to secure our v6 space and AS number. Now that's not horribly difficult or really worthy of NANOG (though I do greatly appreciate folks on the list who helped me through the theory/practice of that process sometime ago). It appears to be fairly straightforward if you are not an LIR. Simply go through the paperwork (LOA, submit to ARIN, get out the credit card, textbook BGP config and done). And if FNF was operating the networks (we don't, we just help with organizing/consulting/software guidance/hardware spend optimization/logistics etc) and if there was just one POP (and associated administrative body), then again it wouldn't be that interesting or worth cluttering up NANOG. FNF goal is to serve as an LIR, SWIPing out /48 chunks to neighborhood level operators. They would then peer with whatever upstream ISPs are regionally close and announce out the space. This of course would be associated with a training program, registration in an IPAM tool etc. Regarding the above? What do the operators on this list wish they could of been trained in starting out? I mean obviously they should have good mastery and working experience of CCNA level material, along with exposure to higher level concepts of WAN networking. What are the tricks, the gotchas, the man that would of saved my company a million bucks in transit costs. Yes I realize these sort of things are usually closely held. I also am striving to create an entirely new breed of operators running BGP enabled sites with ipv6. The more I can do to help ease those folks integration into the internet, the better. In short, the often debated issue on this list of v6 endpoint explosion is going to be very very very real. What IPAM tools out there can scale to a multi hundred million node, distributed, eventual consistency national level? (I've been working closely with guifi.net, and we are attempting to relaunch that as a very slick Apple like experience with a libremap (couchdb based) system. I'd love to hear from folks across the spectrum of experience and network size. From folks who have been dual homed for ~1 year at a single site, to tier1 operators who were there when it all started. So what would you like to see done in a greenfield, open source, open governance carrier backbone network? What would a dream TIER1 (and I use that in the default free zone sense of the word) look like to you? Also how the heck would one get this bootstrapped at a sustainable pace? Would one create numerous tier2 regional carriers, and they would feed into an over arching tier1? I'm thinking something like a 501c8 type structure ( http://www.irs.gov/Charities--Non-Profits/Other-Non-Profits/Fraternal-Societies[1] ) As far as I know, this is the first time that an intentional community type approach is taken and a tier1 is the end goal. Not evolving into one, buying ones way into it, but a manifest destiny type approach to building a backbone. Please feel free to reach out to me directly (char...@thefnf.org[2] ) if you wish to have a one on one discussion. In particular I'm interested in legal expertise in these sort of areas (law/compliance/contracting/negotiations for right of way etc etc etc). Thanks for
Re: Recommendation on NTP appliances/devices
On a tangential note, it's all very nice to say We have brand X and like them, but I'd be curious to hear from folks who have deployed at least four divergent brands with non-overlapping GPS chip sets and software [*] to keep a conspiracy of errors from causing the time to suddenly be massively incorrect. Not that this has ever happened in the past in a single vendor configuration [cough]. Along the same lines I'm troubled by the lack of divergent sources these days - everything seems slaved to GPS either directly or indirectly (might be nice to have stuff out there that got its time exclusively via Galileo or Glonass). The sole exception that I can think of offhand is that I have an office within ground wave of WWVB, which would be a tasty ingredient. GOES is gone. LORAN is defunded. And so it goes; all our eggs are in one basket. I've thought about posting this request to the NTP developers list, but maybe someone who's an operator and actually cares about keeping the byzantine generals sequestered from each other has solved this problem recently. Clues? -r [*] to the extent possible; I'm sure that there's a lot of reference implementation DNA floating around out there) Berry Mobley be...@gadsdenst.org writes: We have symmetricom (now microsemi) and are very happy with them, but we use the roof mounted gps antennas. They will peer with public ntp severs if that would work for you. David Hubbard dhubb...@dino.hostasaurus.com wrote: Anyone have recommendations on NTP appliances; i.e. make, model, gps vs cell, etc.? Roof/outdoor/window access not available. Would ideally need to be able to handle bursts of up to a few thousand simultaneous queries. Needs IPv6 support. Thanks!
Re: Recommendation on NTP appliances/devices
Once upon a time, Rob Seastrom r...@seastrom.com said: Along the same lines I'm troubled by the lack of divergent sources these days - everything seems slaved to GPS either directly or indirectly (might be nice to have stuff out there that got its time exclusively via Galileo or Glonass). Since you mentioned GLONASS: it had a 10+ hour outage yesterday, apparently due to a bad ephemeris upload. Did anybody have a GLONASS-using NTP server experience problems? -- Chris Adams c...@cmadams.net
Re: BGPMON Alert Questions
On Thu, Apr 3, 2014 at 8:50 PM, Randy Bush ra...@psg.com wrote: Good point, which makes me ask: So which 5 to 10 networks, implementing source validation, could result in the greatest coverage or protection for the largest part of the Internet to the best of my knowledge, no one has looked at this for origin validation. sharon goldberg and co-conspirators have done a lot of work in the area, see her pubs at https://www.cs.bu.edu/~goldbe/. but the concentration seems to be on bgpsec which deploys quite differently Right, we (and others) have not looked at the efficacy of a partial deployment of origin validation (using the RPKI) yet. But, we did look at partial deployments of BGPSEC. We found that a large number of networks (around 50% of ASes) need to deploy BGPSEC before its security benefits really kick in. The reasons for this include (1) routing policies during partial deployment might not prioritize the BGPSEC validity over its AS path or local pref, (2) you need every node on an AS path to deploy BGPSEC before it works. Full paper here: https://www.cs.bu.edu/~goldbe/papers/partialSec.pdf We also looked at prefix filtering and found that it has better partial deployment characteristics. Our analysis assumed that ISPs only filter routes from their *stub* customers. (We defined a stub an AS that does not have its own customers.) Then we looked at the fraction of attacks that would be eliminated, if the X largest ISPs correctly implemented prefix filtering. (Large was measured in terms of the number of customers ASes the ISP had.) See Figure 18 on pg 15 of this paper, and the text explaining it in the middle of the right column on pg 15: http://research.microsoft.com/pubs/120428/BGPAttack-full.pdf Finally, like Randy says, RPKI deploys quite different from BGPSEC. My intuition says that (1) once the RPKI is fully populated with ROAs for all originated prefixes, then (2) a partial deployment of origin validation at a few large ISPs should be fairly effective. But I would have to validate this with experiments before I can be sure, or say exactly how many ISPs, etc. Sharon -- Sharon Goldberg Computer Science, Boston University http://www.cs.bu.edu/~goldbe
Re: Recommendation on NTP appliances/devices
Chris Adams c...@cmadams.net writes: Once upon a time, Rob Seastrom r...@seastrom.com said: Along the same lines I'm troubled by the lack of divergent sources these days - everything seems slaved to GPS either directly or indirectly (might be nice to have stuff out there that got its time exclusively via Galileo or Glonass). Since you mentioned GLONASS: it had a 10+ hour outage yesterday, apparently due to a bad ephemeris upload. Did anybody have a GLONASS-using NTP server experience problems? It would be the height of arrogance to think that this couldn't happen to GPS. I want redundancy. -r
Re: Recommendation on NTP appliances/devices
On Thu, Apr 3, 2014 at 8:46 PM, Rob Seastrom r...@seastrom.com wrote: Chris Adams c...@cmadams.net writes: Once upon a time, Rob Seastrom r...@seastrom.com said: Along the same lines I'm troubled by the lack of divergent sources these days - everything seems slaved to GPS either directly or indirectly (might be nice to have stuff out there that got its time exclusively via Galileo or Glonass). Since you mentioned GLONASS: it had a 10+ hour outage yesterday, apparently due to a bad ephemeris upload. Did anybody have a GLONASS-using NTP server experience problems? It would be the height of arrogance to think that this couldn't happen to GPS. I want redundancy. Sadly, right now that either means your own real clock, or WWV. The cellphone time is (as far as I know, for the networks I saw data on) all coming off GPS. Fortunately real clocks are coming way down in cost. So the question is, if you want redundancy, what do your failure modes look like. Is some low level drift if GPS goes away and stays away for an extended period OK? In that case, redundancy probably would be a single local high grade clock. Do you want multi-vendor-common-mode-failure-resistant low drift if GPS goes away? In that case, you probably need 3 local clocks. Possibly 4, if you want to be able to down one for maintenance and still have 3 operating when the fit hits the shan, so that if one of the remaining ones drifts you know which of the 3 is out of whack and to exclude from the live source. Just two operating and you're SOL on figuring out which one is off. This is why spacecraft and aircraft often have 3 or 4 of each critical thing; 3 gets you only fly with all 3 working and the ability to detect the bad instrument; 4 lets you fly with one down for maintenance and still have safe redundant operation, increasing dispatch reliability. -- -george william herbert george.herb...@gmail.com
Re: Recommendation on NTP appliances/devices
Loves my old Heathkit WWVB unit. Keeps drift in check most days. Pairs nicely with the Spectracom 9383. Looking at the Microsemi TP-5000 w/ rubidium oscillator. /bill On Thu, Apr 03, 2014 at 10:25:07PM -0400, Rob Seastrom wrote: On a tangential note, it's all very nice to say We have brand X and like them, but I'd be curious to hear from folks who have deployed at least four divergent brands with non-overlapping GPS chip sets and software [*] to keep a conspiracy of errors from causing the time to suddenly be massively incorrect. Not that this has ever happened in the past in a single vendor configuration [cough]. Along the same lines I'm troubled by the lack of divergent sources these days - everything seems slaved to GPS either directly or indirectly (might be nice to have stuff out there that got its time exclusively via Galileo or Glonass). The sole exception that I can think of offhand is that I have an office within ground wave of WWVB, which would be a tasty ingredient. GOES is gone. LORAN is defunded. And so it goes; all our eggs are in one basket. I've thought about posting this request to the NTP developers list, but maybe someone who's an operator and actually cares about keeping the byzantine generals sequestered from each other has solved this problem recently. Clues? -r [*] to the extent possible; I'm sure that there's a lot of reference implementation DNA floating around out there) Berry Mobley be...@gadsdenst.org writes: We have symmetricom (now microsemi) and are very happy with them, but we use the roof mounted gps antennas. They will peer with public ntp severs if that would work for you. David Hubbard dhubb...@dino.hostasaurus.com wrote: Anyone have recommendations on NTP appliances; i.e. make, model, gps vs cell, etc.? Roof/outdoor/window access not available. Would ideally need to be able to handle bursts of up to a few thousand simultaneous queries. Needs IPv6 support. Thanks!
Re: Recommendation on NTP appliances/devices
On Thu, Apr 03, 2014 at 09:06:57PM -0700, George Herbert wrote: Sadly, right now that either means your own real clock, or WWV. The cellphone time is (as far as I know, for the networks I saw data on) all coming off GPS. Fortunately real clocks are coming way down in cost. There are commercially available NTP servers with GPS + Rb oscillators... for NTP use you could basically let it sync up a couple days, disconnect the GPS and let it freerun. You'd still be within a millisecond of GPS even after a couple years most likely. Reconnect it to GPS for a couple days every 1-2 years to resync it. More fun and cheaper to build your own I'd bet, if you had the time. With clocks/oscillators designed to provide hold-over for synchronous networks and microwave RF systems (parts per million or billion) the demands of NTP for general use in an IP network are pretty modest. You lose more accuracy in NTP stratum 1-2 across a (relaively) jittery WAN link than a cheap atomic clock does in a long time. -Will
Re: BGPMON Alert Questions
On Friday, April 04, 2014 05:06:22 AM Sharon Goldberg wrote: We also looked at prefix filtering and found that it has better partial deployment characteristics. Our analysis assumed that ISPs only filter routes from their *stub* customers. (We defined a stub an AS that does not have its own customers.) Just curious; in your considerations, how would/did you treat cases where ISP's filter their downstreams, to include their downstream's downstreams? Mark. signature.asc Description: This is a digitally signed message part.
