Re: Layer3 down?
From my IT department: It seems that the internet is having issues at a Layer3 Communications router, this normally would not be a problem but L3 runs some routing for the internet backbone. The techs at L3 are working on the problem and we do not have an eta as to when it will be back up, this situation is completely out of our control. You may experience short network outages or severe latency. Thanks for posting my morning smile. :-) --Michael Dillon
2006.06.05 NANOG welcome notes
(getting my notes from today's talks out, finally. ^_^ ; --Matt) 2006.06.05 Welcome notes Program chair, Steve Feldman Thanks to Rodney Joffe, Neustar/Ultra services People who were instrumental in getting connectivity into the room here deserve a big round of applause NANOG program committee, Joe Abley ...wow, slide went fast. ^_^; Agenda changes--none so far. Remote participation: streaming options: http://www.nanog.org/streaming.html Realmedia multicast MPEG2, Reminders: network security don't use cleartext passwords! do use end-to-end encryption (ssh, vpn) PGP key signing see link off www.nanog.org for details Beer and Gear tonight. Interpreting badges Blue badge: steering committee yellow badge: program committee red badge: mailing list committee green = blue + yellow Green dot == peering coordinator Black dot == network security red dot == PGP signing Lightning talks six 10 minute slots available on-topic for mailing list signups http://www.nanogpc.org/lightning deadling is 12:30pm tuesday talks will be wednesday morning. Over to Rodney Joffe now. Welcome to NANOG 37 Almost the NANOG that wasn't. significant effort, thanks to Merit and others; an uphill battle, both from time and location. Encourage other people to host! Not expensive, just takes time. Benefits of Hosting: choose the location (sort of): much easier to do at home. if you wait too long, you don't have a choice of venues. Tee-shirt: you get to pick designs for it! Wonderful engineering opportunties! NANOG Community--you get to give something back to the community. He's hosted through three companies now over time. Exposure--touched on by Randy and Bill yesterday. Generate favourable goodwill for your company; it's not a marketing event, but still gets name out. Network Architecture Plan A: Existing SBC/ATT OC12 at demarc cool! we should be able to get connectibity real cheap Nope: SJCC exclusive licensee says $10K charge to access it, even if gear is donated. Plan B: AboveNet GigE at the DEMARC, hot but unused, and owned by AboveNet from an earlier conf. Nope: SJCC exclusive licensee still has mandatory $10K access fee Plan C: Hilton has fiber run to SJCC infrastructure OK. Let's use microwave to connect Hilton to MAE-WEST 55 S. Market st. Plan C: Microwave from the Hilton to Joe Pucik's 13th floor office at mae-west fiber from 12th floor to 2nd floor meet me room cross connect to SD across SD fiber to PAIX Palo Alto to Jared Mauch/NTT But... 55 S Market == Faraday cage. :( No signal. FC quality glass. Plan D: Tango 54mb microwave from hilton to Dave 'Bungie' Rand's 18th floor penthouse roof at 50 west san fernando fiber from bungi to switch and data PAIX, to Jared Mauch at NTT. Hotel wasn't the biggest challenge, connectivity was. First thing you need to do after picking a hotel that can handle 500 people, 200 rooms. ONLY pick hotels that have fiber with someone you recognize from NANOG. Second thing is to make sure there aren't any access or use fees that have nothing to do with the equipment or bringing gear in. Otherwise, you find there are interesting charges that can be levied against you as you progress! Shout outs: Jared Mauch Christopher Queseda Joe Pucik? Ralph Whitmore Dave Rand UltraDNS and Neustar Volunteers! $10,000 colo space--picture is evil. Other end is the penthouse of the Knight Ridder building. Nice shot!
2006.06.05 NANOG-NOTES TCP authentication with Ron Bonica
2006.06.05 Ron Bonica slides are at http://www.nanog.org/mtg-0606/pdf/ron-bonica-joint%20presenters.pdf Authentication for TCP-based routing and management protocols, from Juniper. A joint presentation, Alcatel, Cisco, Juniper. Starts at NANOG at Washington 2 years ago, security BOF; someone said they would MD5 auth if they could update keys without bouncing their sessions. Suprisingly small number of people actually using MD5 authentication. Motivation many ops don't authenticate TCP based routing protocols RFC 2385 doesn't meet operator needs. Concerns: CPU utilization not so much of an issue for Juniper, [Cisco, Alcatel] Juniper architecture separates forwarding and control plane Key management hard to change keys requires bouncing sessions Weak cryptography easy attacks against MD5 Alternative approaches Application: in the protocols (BGP, LDP, etc) TLS --too much of a headache transport TCP Network IKE/IPsec Chosen Approach: better TCP authentication enhanced TCP auth option Hitless key rollover key chains configured on peer systems time based key rollover key identifier Stronger cryptography HMAC-SHA-1-96 CMAC-AES-128-96 Enhanced Authentication Option Kind - Length T/K Alg ID Res Key ID KEY Key chain contains a tolerance parameter up to 64 keys each key contains id [0..63] auth algorithm shared secret start and end time, both for trans and receive Sending system procedure identify active key candidates start-time = system-time end-time system-time if there are no candidates, log event and discard outbound packet If there are multiple candidates, select key with most recent start-time for sending Calculate MAC using active key calculate over TCP pseudo header, TCP header, and TCP payload by default, include TCP options (if you set the T bit, ignore TCP options) Format enhanced auth option active key ID ... Receiving system proc. lookup key specified by TCP option determine whether that key is eligible startime = system - tolerance end time endtime + tolerance [not sure if that shouldn't be end time system time + tolerance, actually. --MNP] Calculate MAC if calculated MAC matches received MAC, accept the packet auth error procedure discard datagram log do NOT send indication to originator (doesn't adjust TCP counters) Config example: see examples on slide deck, they went past too quickly. Q: how many of us are authenticating IBGP sessions? A: majority in the room are Q: how many of us are interested in a better way of handling key changes? A: lots of people! Q: Russ Bundy?: are you planning on taking this work into IETF to publish through that path? A: Yes, went to RPsec, RPsec2, and SAG working group mailing lists. Q: Randy Bush, IIJ, were there any simpler proposals? Clearly this was designed for the IVTF. A: None that weren't already rejected by the team themselves. Q: Steve Bellovin, Columbia U. No longer security IAD. Why reject IKE and IPsec? It does all this, plus more (which isn't so good) Why not tie algorithm to the key, get it out of the header, get more bits for other uses? A: actually, alg. could be taken from the key; that's the type of comment they're looking for in the IETF; one arg for putting it in option is that is a quick way to check without calculating the MAC, second Q: is more interesting; why not use IKE with just auth. -- no need for confidentiality in this case. It was discussed, one idea was to just use IPsec with preshared keys, but then you have same key rollover system, and key negotiation. Those are all probably good ideas. Would like to do this as a first phase, allow for manual key rollovers, and in a second phase, you can negotiate a key for one-time use. Q: but in IKE/IPsec, you can use preshared key mode in IKE; A: but in this case, you'd still need a system like this to roll over the keys since you want to be able to change keys on each end asynchronously Christopher Ranch: Made the right choices, thank you! Q: Eric ? from cisco: why is this more complicated? A: being able to have multiple keys and roll them over. There are networks that have used the same key for 10 years since they don't want to bounce their sessions. You just can't do that with IKE. This is an operations driven requirement, that it be hitless. Q: Jared Mauch, NTT america: how does he go in and take his iBGP session, roll to this system without making the NANOG mailing list? A: [no answer provided] Q: Bora Kilf?, Broadcom: about IKE not being able to roll keys without a hit; if you use IKE v1, you can lose the IKE SA, have the IPsec SA, rekey your IKE SA, and then rekey your IPsec SA. He agrees with steve, looks like they're re-iventing large parts of IKE/IPsec all over again. Q: Gary? want to avoid colo meets; you want to be able to re-set keys without having to coordinate people in different timezones. Encourage people to participate in SAG to discuss this and provide feedback. Research forum speakers up next.
2006.06.05 NANOG-NOTES AS-PATH prepending measurements
2006.06.05 Active measurement of the AS path
prepending method.
[ slides are at
http://www.nanog.org/mtg-0606/pdf/samantha-lo.pdf
This is the research forum part of the meeting,
people doing real research on real networks.
Samantha Lo and Rocky KC Chang
department of computing
{cssmlo,[EMAIL PROTECTED]
Kowloon, Hong Kong
Dr. Rocky Chang is her supervisor.
Motivations:
Apply AS-path prepending on a trial-and-error
basis to control the inbound traffic.
How effective can the AS-path prepending method be?
what would happen to the routes after prepending on
a link?
The measurement setup; dual-homed stub AS.
connected to 9304 and 4528
Two upstream links, L1 and L2.
Announce a beacon prefix to both links with
prepending on L1.
graph of prepending length on the X axis.
from 0 to 5, then back down. Wait 6 hours
between each change to stabilize.
goes from 102:29 at 0 on L1, to 14:91 at 5
on L1.
Greatest change is between prepending length
of 2 and 3.
When decreasing, see an unbalanced phenomenon.
Who was responsive to prepending?
Incoming link to beacon prefix changes,
next-hop of routes also changes in remote AS
Passive-responsive are those where the next-hop
for the route didn't change, but the subsequent
path is different.
Active-responsive, next-hop actually changes.
Non-responsive ASes, see no change.
43 ASes
no change in either incoming link or next-hop
On L1: 14 ASes
use one next-hop only
Passive-responsive ASes
26 ASes
incoming link change
no change in next-hop
Active responsive ASes
47 ASes:
both incoming link and next-hop changes
possible reasons:
apply shortest-path policy
no localpref override.
Active responsive ASes:
UUnet,
Teleglobe,
bunch of others, slide went pretty quickly
Most of them are located 4 AS-hops away from L1;
after prepending, they are 5 AS-hops away from L2.
Routes to L1 at 4, via L2 at 6 when starting.
What if both ASpaths via L1 and L2 have the
same length?
equal to or greater policy:
AS1239
located 4 AS hops via L1, and 5 AS hops via L2.
AS3662 has prepended once.
So prepending once on L1, 5 6, no change.
prepending twice on L1, 6 = 6, route changes to L2
even though they're equal.
AS3257, located same as 1239
when increasing prepending to 2,
L1 is 6 (4+2), L2 is (5+1), but still uses L1.
When increased to 3, 76, it finally changes to L2.
When decreasing to 2 again, it's equal again, but
it doesn't flip back to L1 until the prepending is
down to 1, at which point 5 6, then it finally
shifts to L1.
This is the greater than policy. Same prepending
length, uses different routes. 'sticks' to previously
used path.
BGP update graph.
After prepending, update messages continue for several
hours.
Conclusions and future work
Route changes are introduced by active-responsive
ASes
shortest path policies
topology - when they will change
possible applications
predict amount of traffic shifts
discover the upstream ASes policies
Thankss to Michael Lo and Lorenzo Collitti
Q: Randy Bush, IIJ, notes that from her slides,
Tim Griffen describes her delayed reaction
as 'BGP wedgies'. It comes from the BGP tie
breakers, it's not something you'll be able
to predict.
A: She notes it's a policy choice of tiebreakers.
Q: Randy insists it's not a matter of policy
choice; it's built into BGP, and not something
they have control over.
Moving on to next speaker
2006.06.05 NANOG-NOTES Pretty Good BGP Josh Karlin
2006.06.05 Pretty Good BGP Josh Karlin, Stephanie Forrest, Jennifer Rexford slides are at: http://www.nanog.org/mtg-0606/pdf/josh-karlin.pdf Main idea: delay suspicious routes lower the preference of suspicious routes for 24 hrs Benefits: network has a chance to stop the attack before it spreads accidental short-term routes do no harm no loss of reachability adaptive simple Algorithm Detection: monitor BGP update messages treat origin AS for prefix seen in past few days as normal new origin AS treated with suspicion for 24 hours. treat new sub-prefixes as suspicious for 24 hours. Response: suspicious prefixes given low localpref, not used or propagated suspicious sub-prefixes are temporarily ignored Example prefix hijack (without PGBGP) same specificity Example sub-prefix hijack (without PGBGP) two /9's cut from a /8 In these examples, AS 5 acted in its own self interest, but it helped protect the rest of the net beyond it. Simulations of two deployment strategies Random, and core+random. Random, with 0 deployed, half the network will be affected, better solution as higher fraction of ASes deploying it. If core of network deploys (core ASes have at least 15 peer-to-peer links) only 62 out of the 20,000 ASes. All but 2% of network protected with that. Sub-prefix hijack suppression a bit tougher, but still good results as core implements it. hijacks in the wild 1997, AS 7007 sub-prefix hijacked most of the internet for over 2 hours Dec 2005 26-95 hijackings during month jan 2006, panix's /16 stolen by conEd Feb 26, 2006, sprint and verio carried TTNET as origin AS for 4/8, 8/8, and 12/8 IAR: internet alert registry IAR verifies hijack attempts a near realtime database of suspicious routes email alerts are sent to those who opt-in for the ASes they choose to recieve alerts for operators recieve alerts only when their AS has caused the hijack or is the victim Tier1 ASs receive one hijack alert per day typically working prototype Solutions with guarantees (and lots of overhead) sBGP soBGP psBGP Anomaly dectors Whisper MOAS lists Geographic based Good Practice proper route filters Route filters protect the internet from you and your customers, not vice versa. Why pretty good BGP? maintains autonomy incrementally deployable no flag day no change to the BGP protocol Effective with a small deployment only requires a software upgrade or change in config generation. Most important, requires minimum operator intervention http://cs.unm.edu/~karlinjf/pgbgp/ Q: (someone)? from UCLA--if you delay the route for 24 hours, if the original AS withdraws it, what happens? A: you'll still end up using the new route, as it just has a lower localpref, so moves will still work. Q: Danny McPherson -- what if origin AS is spoofed to match the origin AS by the hijacker--does this stop it? A: No, that's a man-in-the-middle, or at least it looks like it, and this can't handle that, so it's only pretty good; that would be a later phase. Q: He also notes if your prefix is hijacked, your email alert is likely to get jacked as well. A: True--subscribe from multiple prefixes/domains to be safe! Q: Phil Rosenthal, ISPrime. What happens when a small ISP in south america leaks the internet to an upstream that doesn't filter them? A: Yes, those leaks suck up a lot of memory; this doesn't help because the origin AS is still correct, but the intervening paths are bogus. If the route for a sub-prefix is seen with the origin AS along the path, not seen as a hijack. Q: Jared Mauch, NTT america; follow-on point, you just have a strange AS along the path, but the rest of the origin is correct. A: No, they don't look at the whole path yet; maybe in the future Q: Sandy Murphy, Sparta--thinking of statement at the end, it handles backup routes ok. it works best where operational changes of the origin happen at a human-paced interval. There are some prefixes which seem to oscillate at a much more rapid pace. What about studying prefix behaviour over a longer period of time? Is it locked into 24 hours, or can be adjusted to match better frequency? A: Not locked at 24 hours, could be adjusted to different 'sensitivity' as needed. Q: Randy Bush, IIJ: The internet is not static, those things which relay on viewing it as static like route flap dampening can bite us. We need to enable more and more dynamic behaviour, not less, and Randy thinks this is going the wrong direction. A: That's nice, but presenter disagrees and thinks this is a helpful step in the right direction.
2006.06.05 NANOG-NOTES interdomain routng via Wiser, Ratul Mahjan
2006.06.05 A simple coordination mechasims for interdomain routing [slides are at: http://www.nanog.org/mtg-0606/pdf/ratul-mahajan.pdf Ratul Mahjan David Wetherall Tom Anderson University of Washington now @ Microsoft Research the nature of internet routing within a contractual framework, ISPs select routes that are best for themselves. Potential downsides higher BW provisioning requires manual tweaking to fix egregious problems inefficient end-to-end paths An alternative approach: coordinated routing ISPs have joint control path selection is based on the preferences of both ISPs Potential benefits lower BW provisioning no egregious cases that need manual tweaking efficient end-to-end paths basis for interdomain QoS Existing mechanisms cannot implement coordinated routing route optimization boxes help (stub) ISPs pick better routs from those available MEDS implement receiver's preferences. Cannot create better paths that don't already show up in the routing table. What makes for a good coordination mechanism? MEDS have some nice properties ISPs can express their own metrics ISPs are not required to disclose sensitive info lightweight requires only pairwise contracts Provides joint control and benefits all ISPs. Our solution: Wiser operates in a lowest-cost routing framework downstream ISPs advertise their cost upstream ISPs select paths based on both their own and received costs. Problems with vanilla lowest-cost routing ISP costs are incomparable Can be easily gamed When you bring incomparable costs together, the ISPs that use higher costs win out. Cost normalization Normalize costs such that both ISPs have equal say Normalize such that sum of costs is the same. Makes the system harder to game. Bounds on cost usage Downstreams log cost usage of the upstream ISPs Compute the ratio of avg. cost of paths used and announced Contractually stipulate a bound on the ratio. Similar to existing ratio requirements. Wiser in action Announce costs in routing messages. normalization occurs between ISP pairs. Example results using major ISP topologies for experiments Wiser provides better control under link failure. Wiser produces shorter path lengths Implementation XORP prototype Simple, backward-compatible extensions to BGP embed costs in non-transitive BGP communities border routers jointly compute normalization factors and log cost usage a slightly modified BGP decision process Benefits even the first two ISPs that deploy it. Summary Wiser is a simple mechanism to coordinate interdomain routing may lower provisioning, reduce manual tweaking, produce more efficient paths and help with interdomain QoS Feedback: [EMAIL PROTECTED] http://www.cs.washington.edu/research/networking/negotiation/ Danny McPhereson: Q: how do you normalize across multiple ISPs? A: routing advertisements happen on the sum of the costs announced from me to you, and from you to me. He derived it from different values in his experimentation; utilization and latency in general. Q: Randy Bush: Whatever metrics are, you just normalize by summing them up. But Danny notes if you have multiple ISPs, how do you normalize them together? Q: Danny: where was the 20ms of control plane savings seen that he claimed in slide 11? A: That was based on default ISP policy, prefer customers over peers, etc. So the delay was control plane plus data plane; it wasn't control plane alone. He based it on the old rockefuel equation. Randy Bush: vendors--this is cool stuff, open your ears. Break time now.
2006.06.05 NANOG-NOTES IPv6 deployment at Comcast
Randy Bush, moderator of the next section He begged to do the introduction for a specific reason; deployment of IPv6 that is beneficial to this companies' PL; possibly the only one in existence thus far. He did a very studied and purposeful view of using IPv6 to benefit his company! IPv6 @ comcast Managing 100+ million IP Addresses [slides are at: http://www.nanog.org/mtg-0606/pdf/alain-durand.pdf Alain Durand Office of the CTO Director IPv6 Architecture [EMAIL PROTECTED] Agenda Comcast needs for IPv6 Comcast plans for IPv6 Challenges simplistic view of comcast IP problem 20 million subscribers in video 2.5 set-top boxes per subscriber 2 IP per set top-box per DOCSIS std. total 100 millions IP addresses needed that's not including high speed data, nor comcast digital voice, nor mergers/acquisition Used to use RFC1918 for cable modems. that space was exhausted in 2005 Comcast recently was allocated the largest part of net 73 and has renumbered cable modems in that space. In the control plane, all devices need to be remotely managed, so NAT isn't going to help us IPv6 is the clear solution for us However, even we are starting now, the move to IPv6 isn't going to happen overnight. Triple play effect on the use of IP addresses 2005 HSD only 2006 T+ Cable Modem 1 1 Home computer/router1 1 eMTA (voice adapter)0 1-2 Set top box (STB) 0 2 total num of IP addresss 1-2 8-9 (assume 2.5 STB per household IP Addresses: Natural Growth vs New Services nice graph--based on trends, not real data Contingency plans: use public address space use dark space (pre-RFC1918 space) federalization (split into separate domains) (trying to avoid that) IPv6 strategy start early deployment plans started back in 2005 deploy v6 initially on the control plane for the management and operation of the edge devices they manage DOCSIS CM, set top boxes, packetCable MTA (voice) be ready to offer customers new services that use IPv6 LATER, not now--first step is to just be able to manage their own gear. migration to v6 must be minimally disruptive. deploying v6 must be in roadmap for all vendors ops, infrastructure, systems must be ready to support v6 devices. over time, IPv6 will penetrate Comcast DNA Deploy v6 for IP addrs of the CM and STB architecture: dual-stack at the core, v6 only at the edges deployment approach: from the core to the edges backbone-regional networks-CMTS-devices this is an incremental deloyment; existing deployments will be untouched in the beginning Follow same operational model as with IPv4; lots of DHCP! News Flash: All routers on Comcast IP backbone are IPv6 eanbled first ping on 10GE production backbone TTLs aren't quite working properly, still checking on that. [so, even mainstream vendors still don't have v6 working quite properly yet] New CM will be v6 ready (dual-stack capable) On an IPv4 only CMTS, CM will have v4 address only On v6 enabled CMTS, CM will only have v6 address No CM boxes will have both; if they could support v4 on all, wouldn't have this issue to start with! Provisioning, Monitoring, Back-Office mostly software upgrade problem not unlike the Y2K issue fields need to be bigger in database and web scripts Should system X be upgraded for v6? does it communicate with devices that are v6 only? payload Q: does sstem x manipulate IP data that could be v6 (store, input, display) Comcast inventory analysis About 100 systems 10 need major upgrades for transport 30 need minor upgrades just for display/storage Back office management of cable modems. network transport will still be v4 however, back office systems may need to be modified to display/input/store v6 related data (CM v6 addr) Payload can be v6 while transport is v4. IPv6 certification Basic IPv4 compliance taken for granted today IP level component testing is limited IPv6 is still new technology maturity level of vendor implementations vary greatly some have v6 for 10 years even those have features not fully baked others have nothing, will rush to buy 3rd party stack. Bar for v6 product acceptance has to be higher than what we typically accept now for IPv4 Formal v6 requirement list before purchasing v6 conformance testing/certification to accept product v6 training most engineers have heard about it, don't know much fear factor can expect new hires to have 2-4 years of v4, but 0 v6 initial and continuous training process is critical! v6 vendors CM (cable modems) (DOCSIS 3.0/2.0b) CMTS Router Provisioning system OSS Video/Voice back-end systems Retail Market (Consumer electronics) Home Gateways Video (eg TV with embedded cable modem) Right now, provisioning system is most challenging. v6 protocols MIBS: some OSS vendors implement RFC2465 (deprecated) some router vendor implement partial RFC4293 (new combined v4+v6 MIB, but only v6 part)
2006.06.05 NANOG-NOTES Network Neutrality panel notes
(since there's no slides for these online anywhere, and the slides were going past pretty quickly, I have to apologize for the gaps in the notes ahead of time. --MNP) 2006.06.05 Network Neutrality Panel [slides are not yet online] next up is the controversial subject of network neutrality; Bill Woodcock will be chairing the panel, so Randy Bush can go be a member of the audience again. Bill Woodcock: network neutrality has been in front of press and legislatures for the past several months, and has been in the works behind the scenes for almost a year. Brokaw Price, peering at Yahoo! Sean Doran, free agent, rooting for Sprint back in the day Sean Donelan, now at cisco, Gene Lew, at neustar now, has done cable operations before. Sean can pretend to be Vint Cerf for this panel. :D Network Neutrality--what does it mean to operations people? History: Michael Powell, Feb, 2004, defined four internet consumer rights (chairman FCC) freedom to access content freedom to use applications freedom to connect personal devices freedom to obtain service plan information History of net neutrality concept: feb 2005, madison river telephone company consent decree Madison River shall not block ports used for VoIP... August 2005, FCC policy statement access lawful internet content run applications and services subject to the needs of law enforcement connect legal devices that do no harm to the networks competition amongst... All of these principles are subject to reasonable network management. That last bullet is what gives telcos ability to quote QoS as a mandatory network management requirement. March 2006, internet non-discrimination act senate bill 2360 only 2% of americans have a choice in last mile. shall not interfere with block, degrade, alter, modify, impair, or change bits or content May take measures to protect customers from attacks may protect their own network infrastructu May 2006 Internet freedom and non-discrimination modifies clayton anti-trust act. Passed based on party lines. Turns over to Sean to talk about his thoughts. Sean Donelan Doesn't represent anyone right now. The Huck Finn approach. And no, you can't configure this on your router. What are we talking about? Rep John Conyers (D-MI) internet as we know it is at risk Same guy noted in 2003 about cable operators being smart enough to not poison their customer pool Not really a new issue: ANS CO+RE in 1991 unapproved networks filtered from RE gateways ANS and CIX, June 1992 ANS agreed to provisionally interconnect CIX proposed filtering resellers (194) NSF NAP/NREN solicitation (1993) required NSPs connect to all priority Network Access Points (NAPs) uncertainty created opportunities for new service providers pizzahut.com debate--couldn't reach it from university networks, but you could reach it from UUnet. Two-tiered internet even back then. Regulations chasing change TitleI -- General FCC Authority (pre 1984) TitleII common carrier voice/phone calls/later data TitleIII spectrum licensing broadcast TV and radio TitleVI Cable Television (post 1984) FCC moved DSL (but not UNE-L and cable modems) back into TitleI again. VoIP is still unknown. Telecom act of 1996 was another biggie. before 1996, enhanced services transmitted over a common carrier. After 1996, info services were defined by the telecom act; they run over telecommunications provisioned by anyone, not just common carriers. That's why cable companies can offer telecommunications even though they're not common carriers. Also why Radio and TV can put IP in subcarrier without being common carriers. So, we have a really odd blend, they're not mutually exclusive. Now you have the potential for new entrants from any direction. Most home services now come over cable companies; multiple resellers of same product wasn't enough to satisfy customers. Customers are very fickle Interfering with a customer's use of the Internet would hurt the provider's business. No-one can predict what the next Killer App will be. and *everyone* can complain Both sides need each other to succeed. Predicting in advance what customers want and will consider improvement vs interference is hard to the point of being nearly impossible. And what customers consider an improvement one year may become interference, or what was once interference will now be considered improvement. If you start writing regulations, you imply investigative and reporting requirements along with it. Who would enforce these regulations? And regulations seldom prevent people from being evil, the government simply sets the price for being evil. Broadcast fairness doctrine--equal time; nobody can buy public advocacy on national networks, except for politicians, who get the best rates. Kingsbury committment--ATT had to interconnect with everyone--but that meant you didn't need a second long distance company, so ended up supporting monopoly expansion. Universal service backfired by giving a monopoly to those giving
2006.06.05 NANOG-NOTES Peering BOF notes
(This time around I opted to go to the peering BOF and take some notes. It's the one downside to parallel tracks--wish I could be in two places at once. ^_^;; --MNP) 2006.06.05 Peering BOF Bill Norton introduces the Agenda; unfortunately, my laptop took so long to boot, I missed the Agenda slide. Doug Toy?, Transit Cost Survey, data collected at NANOG 36; he's just here to present the collected info, not really representing anyone. Recap: At NANOG 36, people indicated their cost per Mb and commit level. length of contract was usually 1-2 years. 42 data samples collected avg $25/Mb $95/$10 were the extremes. Avg commit level 1440 Mbps Other observations as expected, cost per Mbs tends to decrease as the commit level increases. Tier1's are more expensive Cost tends to vary more with Tier 1 providers than with others. between 0-500Mb commit level, prices are all over; at higher commits, prices level out at the bottom. Question: Mbps, is that the cap, the usage, inbound plus outbound? A: That's the general 95th percentile higher of the two inbound and outbound. Committed amount. Graphs tend to approach a hard bottom; as commit increases, doesn't change all that much. Bottom is around $10/Mb, even though commit levels increase. Of samples collected, 2/3 were from Tier1 providers. 90% of contracts are 1-2 year in length, so didn't cause much variance. Tier 1 definition is based on Wikipedia definition. Questions from audience? Q: Data looked pretty clean; were there samples pulled out to make it look cleaner? A: No, other than people who left fields blank on the survey. Q: was there a timestamp of when contract started? A: much of it wasn't complete. Mostly within the 1-2 year range for length as well as start date, so nothing really ancient in there. BillNorton; people had some concerns about violating NDA or contract details when filling out the survey. Where do we draw the line in doing these types of surveys? SteveGibbard; NDA is agreement between transit provider and customer, and this was anonymized and voluntary. Data is interesting, both for purchasers and sellers of transit. Q: 42 samples graphed, there were 80-100 people in the room at the time; so the real comments from the rest of people weren't counted? A: No, there were less than 50 submissions total, of which 42 were complete. Q: Patrick; many people put more than one transit provider on their form; how were those other transit providers handled? A: no clue, he just got a spreadsheet with data. Back to Bill Norton Peering Lists Issue -- make available to customer prospects? 15 mins. Peering disclosure dilemma: customers often asked for peering list, sometimes peerings restricted under NDAs Metric for determining connectedness, capacity, resiliancy. Is there a better metric for customers? IX capacity in/out Peering pipe size? ISPs are getting commonly asked about this, based on hands raised in the room. How many people lose business because the customer doesn't get an answer? Sylvie, VSNL notes they provide the info when they're under an RFP; they won't give capacity, they give an aggregate, they won't go peer by peer, that would be a violation of NDA. BillN: are the NDAs written to allow total numbers like that? Sylvie: you should not disclose capacity per location or per circuit, but they don't forbid aggregate total numbers. BillN: is there something else that could be given to the customer that would satisfy their question without revealing what Chuck: A lot of ISPs lie about their peerings; he runs AMSIX, people claim to have multiple gig to the peering exchange, he knows they don't really have that much. Patrick: but he can look at the peering stats on AMSIX--Chuck notes only members can. Patrick: customers ask how many gigs they can send to a provider; it's available headroom, so they ask their upstreams how much available headroom is left. Most providers are having a lot of trouble getting the right capacities to the right networks. The reason many don't answer is they don't like the answer they have to give. Ted Seely, Sprint, how do you solve the problem? There's lots of traffic that needs to be exchanged, how do you fix it? Patrick: how about everyone upgrade to 10gigE in many places? If you can't afford it, stop selling bandwidth. But most people can't go to all the different providers, they have to buy from a small subset of providers. RAS: No technology problem with doing it; it's the money. Not charging enough to cover the costs of the technology you need to install to cover the bandwidth. Ted notes you can't just link at one spot, you have to connect at six places, and you need to have links in and out of the site to support the volume, etc. Patrick: can you tell us how much they exchange? 40Gb times 6 providers in six locations is probably more traffic that Sprint has in total. Ted Seeley; it's a time scale issue--yes, it can be solved, but in what timeframe? BillN feels it's reasonable information
2006.06.05 NANOG-NOTES BGP tools BOF notes
(ok, last set of notes for tonight, and then it's off to bed for 90 minutes of sleep before heading back to the convention center. ^_^; --MNP) 2006.06.05 Welcome to the 4th BGP Tools BOF! [slides are at http://www.nanog.org/mtg-0606/pdf/lixia-zhang.pdf Nick Feamster GeorgeTech Dan Massey CUS Mohit Lad and Lixia Zhang, UCLA The Goal sharing some tools develop from our research efforts. hopefully will be useful for operations community. Also to collect input on new tools we would like to see so they can develop them. Routing Configuration Checker Nick Feamster O-BGP data organization tool Dan Massey [slides are at http://www.nanog.org/mtg-0606/pdf/dan-massey.pdf The Datapository by Nick Feamster [I'm sorry, that just sounds *far* too much like something you do *NOT* want your bedside nurse administering...--MNP] Visualizing BGP dynamics using Link-Rank by Mohit Lad Open discussions and demos Nick Feamster Network Troubleshooting: rcc and beyond rcc: router configuration checker proactive routing configuration analysis idea: analyze configs before deployment many faults can be detected with static analysis. rcc implementation. http://nms.csail.mit.edu/rcc/ preprocessor - parser - relational database (mySQL), constraints - verifier - faults verifier is a template checker and set of constraints your configs are checked against. He's looking for GUI developers. very bare-bones command line right now. Parsing configurations--shows some output. He shows examples of the abilene configs, which are non anonymized. show all routers peering with a given AS, can look at route maps in each direction, etc. After running rcc on it, you get a web output which shows relationships--oh, pictures don't matter, with some more grease could be a reasonable representation of your network. Q: Randy Bush asks if it could show which peering sessions are missing? A: Not yet, but it could be added, thank you! Shows processing and errors; you get a page that summarizes the things RCC thinks are errors. Signalling partition? that's a missing iBGP session; he needs some better lingo in places. Also shows anomalous imports, could be intended for traffic engineering; that's inconsistent policy in ISP speak. Some of the names will get fixed to make Randy Bush happy. Yes, but surprises happen! link failures node failures traffic volumes shift network devices wedged ... two problems detection localization Need to marry static config analysis with dynamic information (route is configured but isn't in the dynamic table) he skips a closer look, just some jargon. Detection: analyze routing dynamics; drill down on interesting operational issues. idea: routers exhibit correlated behaviour blips across signals may be more operationally interesting than any spike in one signalling system. How do you spot things in the churn? Detection three types of events single-router bursts correlated bursts multi-router bursts ---common; and commonly missed using simple thresholds Localization: joint dynamic/static which routers are border routers for that burst topological properties of routers in the burst. proactive analysis - deployment - dynamic - reactive detection - diagnosis/correction - static - By going back to the configs, lets you see if it's something happening inside the network, or on the edge. Specific Focus: firewall configuration difficult to understand and audit configs subject to continual modifications roughly 1-2 touches per day federated policy, distributed dependencies each department has independent policies local changes may affect global behaviour (These are pulled from Georgia Tech; 130 firewall configs. Builds static connectivity matrix.) Reactive monitoring...use probes from subnets to verify reachability/connectivity. (immediate) open issues reachability and reliability of controller service-level probes diagnostic tools != service-level happiness policy conformance. Q: can it give suggested remediation, or provide config templates for new routers being added? A: Good idea! OK, over to next presenter. Helps with understanding BGP data. BGP data collection and organization (OBGP) Tool Colorado state university/university of Arizona/UCLA BGP data collection takes lots of BGP data, from RIPE RIS, etc. ISP BGP peer router - update oreg - rib+update - feeds into gigabytes of data, different formats, potential errors enter in, and severe lack of metadata. Other tools can use it, LinkRank, BGP-Inspect, and a bunch of people cite it in reports and research. OBGP motivation Large Volume of Data data from many sources (RIPE, RV, private data) Long time scales and very recent (real-time?) data Slightly different formats RIPE/RV use different naming conventions different dump intervals different timezones for older data Lack of MetaData would like to only see desired peers and desired update types Possible errors in the data are updates missing due to log errors?
Re: 2006.06.05 NANOG-NOTES Peering BOF notes
Matthew Petach wrote: Thank you Matt, these notes are almost like being there. Excellent work. Also Ted Seely at the peering bof? Shocked there wasn't a riot. They're getting into the peering fray, and only a year old. This is gigs and gigs, has potential to dwarf current peering traffic. Current issues could be tiny compared to the flood of potential issues when hundreds of Gbs comes flooding towards customers. Problem is extrapolating far into the future from rates seen at the very start. I remember some time ago numbers being thrown around of how quick imode was being adopted, which,if those rates had continued, would have meant most of the world being on imode today. Swedish police, 100Gb of peer-to-peer traffic at peak, AMSIX lost 10gig, LINX lost 5gigs, probably lost about 40Gb weds/thurs last week when the swedish police shut it down. Which site? Pirate Bay torrent tracker. Do we have weekly and monthly stats? This looks interesting. Comment from audience is that live events are still going to be the challenge; HDTV is getting gb/sec from cameras, needs to feed it out, no chance to cache, so multicast ends up being the only viable option for it. Multicast is caching with zero retention time -avg. Robert Seastrom--should you do v6 at all? Should you be a pioneer, and make the v6 people happy, sure, do it; if you want to make money, no. I think Alain from comcast had a different take on it. DanGolding notes that the tier1's are stuck on a treadmill; they can't peer with you even if they desire it, for fear of messing with their own ratios. I don't think sprint or 701 care too much about their own ratios any more. Dan Golding, network neutrality on the peering front. One year old company, video content, already doing 20gig/sec. This is a buttload of traffic, and they're already getting into the peering mode; will this traffic ultimately dwarf the rest of our traffic? Depends on if this is a sustainable business model. During the dotcom days, lots of companies were going to dwarf the rest of our traffic, but things tend to return to mean. It gets harder to sustain 20% growth rates month over month, the further up and to the right you go. Others are tempted to jump on. MSOs see non-neutrality as a way to keep other's VoIP off their networks On the other hand, the Bells will squeeze them; the cable companies lack peering. I am not convinced lack of peering is such a huge impediment, esp. at transit rates going on now a days (see matt's earlier notes). Patrick Gilmore: Tier 0, how does that work when VZ and ATT don't make up the bulk of the internet anymore. What about the rest of the world? Would this statement be true for bulk of the internet in the US? How is this determination made? Patrick Gilmore asks for non-US peering coordinators; who would care if ATT/VZ depeered you? Not many respond It is possible we could have learned more if the question were posed in the following fashion 1) How many non-US peering coordinators (have I mentioned how much I dislike this term, I prefer SFI Secretaries) have current, active peering with VZ/ATT? 2) Of those that answered yes to #1, how many would care if they depeered you? Easy not to care when you don't have SFI in the first place. Dan Golding notes that if we had many different ways of getting local loop to your house, it would be less of an issue. Incent development of alternate methods; wifi, 3G/4G/5G networks, etc. Hah, wireless is never going to compete on a purely bandwidth perspective with fiber/copper (regardless of the chorus of people sounding off of how wifi is used to get the majority of bits on cable/dsl - true, but thats a very limited scope deployment, we are talking about replacement of cable/dsl here, not how to get from your couch to the wall-jack). The way to get wireless working is to emphasize the mobile aspect of wireless, but with youtube et al pushing huge bits, wireless as a replacement cable/dsl, not so likely. Mikhail Abramson, with high speed cellular, mobile is making network neutrality less of an issue, since you do have more options. The GSM providers are happy with the internet bandwidth usage on mobile data, it's making them really good money. Details and breakdown of revenue? Is it ringtone and SMS bandwidth, or is it gprs/hspda type bandwidth. If we all went to a common $10 provider, we could create a new tier0 and bypass the bellheads. The last mile _consumers_ aren't likely to be able to go to this $10 carrier. /vijay
Re: 2006.06.05 NANOG-NOTES Peering BOF notes
Nice notes - thanks. * [EMAIL PROTECTED] (Matthew Petach) [Tue 06 Jun 2006, 12:52 CEST]: [..] BillN: is there something else that could be given to the customer that would satisfy their question without revealing what Chuck: A lot of ISPs lie about their peerings; he runs AMSIX, people claim to have multiple gig to the peering exchange, he knows they don't really have that much. Patrick: but he can look at the peering stats on AMSIX--Chuck notes only members can. Don't know a Chuck at AMS-IX, and he's wrong about port details being available only to existing members. Click on any name at http://www.ams-ix.net/connected/ and get the full Monty. Patrick: customers ask how many gigs they can send to a provider; it's available headroom, so they ask their upstreams how much available headroom is left. Most providers are having a lot of trouble getting the right capacities to the right networks. The reason many don't answer is they don't like the answer they have to give. As a transit provider you're doing something if for the majority of your time you are dealing with customer complaints about packet loss. -- Niels.
Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss.
Hallo colleagues, Maybe someone of you can help me to understand the phenomenon of pack loss when using asynchronous routing? I have customers who are complaining about packet loss and they are providing me with MTRs and pathpings (that's some sort of traceroute that pings every hop it sees several times - comes with windows xp) that show the loss starting at my routers and ending at their server (=the last hop). All users are coming from a (dialup-)network where the way from them to our servers are going via a carrier different than the carrier we are using to route the traffic back to the dial user. The interesting thing is that there is no loss at all when the users either use a ping instead of this pathping/mtr-stuff or when I perform a ping or even an mtr on my server in direction of the dialup customer. The nasty thing is that there is de facto NO LOSS on the line but the users is seeing some sort of phantom loss. The problem immediately disappears when I change to way back to the same carrier as the way to us so that we have synchronous routing again. My assumption is that pathping and mtr somehow get irritated by the icmp messages due to a wrong timing or something like that. Any ideas? Thanks, Gunther
Re: 2006.06.05 NANOG-NOTES Peering BOF notes
On Tue, 6 Jun 2006, vijay gill wrote: Swedish police, 100Gb of peer-to-peer traffic at peak, AMSIX lost 10gig, LINX lost 5gigs, probably lost about 40Gb weds/thurs last week when the swedish police shut it down. Which site? Pirate Bay torrent tracker. Do we have weekly and monthly stats? This looks interesting. http://www.ams-ix.net/technical/stats/ http://stats.autonomica.se/mrtg/sums_max/All.html https://stats.linx.net/cgi-pub/exchange?log=combined.bits The thepiratebay.org tracker was taken down a week ago, came back saturday but traffic still hasn't caught up. All the above numbers are estimations, I'd be very interested in US based data. -- Mikael Abrahamssonemail: [EMAIL PROTECTED]
AW: Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss.
Hello Alan, Thanks for your reply. I know that icmp is being handled by the router's cpu and that the forwarding is being done in hardware. The interesting part here is that the loss starts at the router where the asynchronous routing begins and is even being seen at the final destination host - but this only happens with pathping and mtr. An ordinary ping doesn't show this problem. The problems in mtr/pathping disappear when a synchronous routing is being established. What's the case for this? What are the programmers of mtr and pathping doing wrong or is there another problem mayb in the implementation of icmp? Gunther -Ursprüngliche Nachricht- Von: Alan Clegg [mailto:[EMAIL PROTECTED] Gesendet: Dienstag, 6. Juni 2006 17:27 An: Gunther Stammwitz Betreff: Re: Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss. [..] I'd guess that the routing devices are just droping/ignoring ICMP echo request packets. They are the least interesting packet, and are quite often, responses to them are given extremely low priority. Just my guess, AlanC
Re: Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss.
On 6-Jun-2006, at 08:19, Gunther Stammwitz wrote: I have customers who are complaining about packet loss and they are providing me with MTRs and pathpings (that's some sort of traceroute that pings every hop it sees several times - comes with windows xp) (if it comes with win xp, then that sounds interesting-yet-surprising -- it's more usually found at http://www.bitwizard.nl/mtr/). [...] The nasty thing is that there is de facto NO LOSS on the line but the users is seeing some sort of phantom loss. The starting point for any investigation like this is to compare the traceroute that apparently shows loss or other problems with traceroutes from strategic points in the path back to the source. If there's a congestion problem which is the cause of the concern then comparing traceroutes in both directions will usually help find it. If there's no congestion problem, or the apparent problem is unusual latency or loss in the numbers mtr displays for particular routers in the path, then mtr's ICMP echo requests towards the control elements of particular routers are probably being deliberately rate-limited by the operators of those routers. Joe
Re: Phantom packet loss is being shown when using pathping in connection with asynchronous routing - although there is no real loss.
Gunther Stammwitz [EMAIL PROTECTED] wrote: I have customers who are complaining about packet loss and they are providing me with MTRs and pathpings (that's some sort of traceroute that pings every hop it sees several times - comes with windows xp) that show the loss starting at my routers and ending at their server (=the last hop). All users are coming from a (dialup-)network where the way from them to our servers are going via a carrier different than the carrier we are using to route the traffic back to the dial user. The interesting thing is that there is no loss at all when the users either use a ping instead of this pathping/mtr-stuff or when I perform a ping or even an mtr on my server in direction of the dialup customer. The nasty thing is that there is de facto NO LOSS on the line but the users is seeing some sort of phantom loss. The problem immediately disappears when I change to way back to the same carrier as the way to us so that we have synchronous routing again. My assumption is that pathping and mtr somehow get irritated by the icmp messages due to a wrong timing or something like that. Any ideas? Try varying the mtr interval, such as -i .1 (must be root for 1). Does the packetloss significantly increase with this faster mtr? Try slower -i 10. Does the packetloss significantly decrease or go away? If the answer to both above questions is yes, then I would suspect ICMP rate limiting. You could also try varying the speed of ping. Windows is pretty limited, but on unix you can do things like .1 second intervals (-i .1 as root). Does a faster ping trigger this apparent loss? If so, ICMP rate limiting. The only part that I don't get is that you can mtr to him without packetloss. Although the path in-between may be different, the final hop packetloss should exactly equal what he sees when mtring you. A round-trip is a round-trip, and results should be identical regardless of who originates. I can't think of any way this would be different unless echo and echo-reply were being rate limited independently. My home ISP (apartment ethernet t1 service, which is actually multiple T3s) has a Packeteer or something along that line. If I use ping, everything is fine since it goes so slow. If I use MTR, it works fine for the first few seconds then sees 90% packetloss on all hops from then on once the rate limiter burst bucket runs dry. Of course, TCP still sees no packetloss even when mtr is seeing this heavy rate limited loss...
Re: 2006.06.05 NANOG-NOTES Peering BOF notes
vijay gill [EMAIL PROTECTED] writes: Matthew Petach wrote: Robert Seastrom--should you do v6 at all? Should you be a pioneer, and make the v6 people happy, sure, do it; if you want to make money, no. I think Alain from comcast had a different take on it. The specific context was should YouTube (the presenter) do v6. When one is speaking about intra-enterprise VOD or IPTV (ie, if you are Comcast), the answer may be (and probably is) completely different from the i am a video dump for streaming joe and jane luddite's home videos and teenagers drinking a mentos/pepsi cocktail over the public internet scenario. ---Rob
Re: 2006.06.05 NANOG-NOTES Peering BOF notes
On Tue, Jun 06, 2006 at 09:10:21AM -0400, vijay gill wrote: If we all went to a common $10 provider, we could create a new tier0 and bypass the bellheads. The last mile _consumers_ aren't likely to be able to go to this $10 carrier. The example wasn't fully documented in the notes (though very much appreciated!). The idea was a content provider (say YouTube) and a non-bell broadband provider (say Covad) would both interconnect on the $10/meg carrier. -- Matt Peterson 38B4 B706 3BA7 97B7 F638 1198 6AB4 CDF2 552A 0DC9 --
ipv6 @ sprint, somebody home?
[EMAIL PROTECTED]: host kay.sprintlink.net[199.0.233.8] said: 553 5.3.0 [EMAIL PROTECTED]... User Unknown (in reply to RCPT TO command) It's must be 6/6/6 that it ain't working. I guess they are scared that IPv6 might scare their fisherprice routers ;) Anybody a *working* contact so that they can also be nicely reminded of the fact that the 6bone has come to an end and that they should nicely ask their paying customers to stop announcing 6bone space? http://www.sixxs.net/tools/grh/lg/?=prefixfind=3ffe::/16 And http://www.sprintv6.net/ doesn't contain any contact info before you say google it. Then again the following url clearly shows their 'interrest' http://www.sprintv6.net/aspath/bgp-page-complete.html Last change on the tree detected on Sun DEC 11 2005, h.22:50 Fisherprice, fisherprice /sarcastic mode Greets, Jeroen -- Just in case, yes [EMAIL PROTECTED] is whois 3ffe:2900::/24 : ipv6-site:SPRINT origin: AS6175 descr:SprintLink 6bone Reston, VA, US country: US prefix: 3FFE:2900::/24 ... contact: RJR2-6BONE remarks: This object is automatically converted from the RIPE181 registry notify: [EMAIL PROTECTED] changed: [EMAIL PROTECTED] 20010117 changed: [EMAIL PROTECTED] 20010423 changed: [EMAIL PROTECTED] 20020925 source: 6BONE And no I am not mailing their ipv6-support addy which doesn't respond ;) PS: Happy Slayer Day: http://www.nationaldayofslayer.org !!! signature.asc Description: This is a digitally signed message part
Zebra/linux device production networking?
Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_tc024.htm and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
(ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..) Unless you are absolutely certain of how routers need to work for your environment, and am willing to engineer your way out of problems, using this platform to achieve 99.x% uptime is quite not practical. Overall, this is a bad business decision, and if you quite had the clues to engineer most of the problems, you wouldn't be asking this question anyway ;) It's really a matter of lacking commercial support to route your traffic. If you can support yourself, then great, by all means go for it, and there are several operators running stable on cheap gears. If you can't support yourself, then you are opening up a can of worms. With that said, if you are looking to do one-router network for BGP, you may want to take a look at OpenBGPd, which is stable but currently lacks IGP support (though, openospfd is under works). Zebra is only stable when it's doing nothing or next to nothing. james
Re: Zebra/linux device production networking?
Linux routers are great for redundantly routing between your cable-modem and DSL at home. Using a linux router in production is a very very bad idea, although it may seem appealing to suits with no networking knowledge. I'm sure that other posters will provide you with many pages of reasons why linux routers suck, but I'll keep it short. 1. Mean Time Between Failures 2. OS exploits 3. Service/support Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment?
Re: Zebra/linux device production networking?
IMHO, it's a bad idea. A less intrusive alternative might be a FreeBSD based platform running Xorp/Quagga. Tiffany.On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: Greetings fellow nanogers,Long time lurker, first time poster (please, be gentle!).After looking at the archives, I didn't see this particular discussion,so here we go.First, a little background.. My CTO made my stomach curdle today when he announced that he wanted todo away with all our cisco [routers] and instead use Linux/zebra boxen.We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He hasinformed me that he has found many people who do this for their coredevices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device(core and/or regional, I'd be interested in both) in a production (read:99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did youfind out about after implementation?Has there been any discussion (or musings) of moving towards such asolution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in.Here's the article that started it all (this was featured on /., solikely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_tc024.htmand another:http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will ofcourse summarize to list or to individuals.(ps, particulars are deliberately not included.. I'm not looking foradvice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Albert Meyer wrote: 2. OS exploits One might argue that is an issue with any device. Cisco have their fair share of IOS updates due to security related bugs. Linux appears to have many, mostly due to the number of services that you can run. It's not like a Linux router is going to run Sendmail or Apache. David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgQHTIgPQWnLowkRAt4bAKDWOP4MOu3tnxTGxZDqPY+nlmS9DgCfZ1qi M8eUX6BsNNePrtEfT88Z/Aw= =pGLM -END PGP SIGNATURE-
RE: Zebra/linux device production networking?
Hi, I am also newbie poster so likewise plz be kind. I tend to agree with the comments made so far, however depending upon the business, budgets are not always available that might match the requirements and hence I can to some degree understand the use of such boxes for small organisations. I would be interested to know how many software (for want of a better description) routers are in live production in this kind of environment i.e. the 99.% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. Also does anyone have any peering policies which would exclude peers with software routers specifically, most have a requirement for the ability to support stable BGP peering but I have not seen any specific exclusions for such devices? Mark From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Tiffany Snyder Sent: 06 June 2006 23:29 To: Nick Burke Cc: nanog@merit.edu Subject: Re: Zebra/linux device production networking? IMHO, it's a bad idea. A less intrusive alternative might be a FreeBSD based platform running Xorp/Quagga. Tiffany. On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: Greetings fellow nanogers, Long time lurker, first time poster (please, be gentle!). After looking at the archives, I didn't see this particular discussion, so here we go. First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Has there been any discussion (or musings) of moving towards such a solution? I've seen a lot of articles talking about it, but I've not actually seen many network operators chiming in. Here's the article that started it all (this was featured on /., so likely you've read it already). http://www.businessweek.com/technology/content/nov2004/tc20041129_5206_t c024.htm and another: http://www.networkworld.com/community/?q=node/5693 Feel free to respond off list. If anyone else is interested, I will of course summarize to list or to individuals. (ps, particulars are deliberately not included.. I'm not looking for advice, just if anyone has any solid experience with this..)
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? Sure - I've done this before. We ran 7200s on the border (DS-3 interfaces for Linux didn't make sense at the time) and Linux boxes running all these features (plus some others) on the core. Worked flawlessly and the only downtime encountered over the two years it was running was during failover which took 5sec. Of course, the time invested in building it totally offset any savings, but that particular employer considered your time to be 'free', even though you could be billing instead, but that's a whole other argument. However, if I've got a Cisco router, in my city I can easily find 20 people in half an hour who I'd trust to get into my gear and work on it. I'd find another 50 if I went out 200miles. Linux on the other hand - Maybe three, including me. State wide, probably not even 20. I'm not talking RHCE people - I'm talking about people who can really troubleshoot kernel networking issues, device driver problems and so forth. Not easily accessible (or cheap) resources. Right now I've got a pair of Linux boxes (Debian based, 2.6 kernels) running Quagga (Zebra fork - I'd recommend it over Zebra) for BGP and OSPF, pulling two full loads. HSRP is provided with LinuxVirtualServer (aka heartbeat) and I'm doing dot1q with STP. No PVST support on Linux though. It all just works. Had a memory problem on one box, which killed it, but I've had that on plenty of Cisco gear too. None of the problems have really been 'Linux' related. 99% of them are user related, in that, I set an IP wrong, or I screw up a netmask - Usual kind of junk. Basically, if you're not comfortable with the idea of it, you're not comfortable supporting it. It'll cost leaps and bounds more supporting the environment compared to Cisco hardware. I have specific Linux expertise and experience which makes me go I can do that on Linux and have it work without problems, but also coming from a Cisco background I know where the line between being able to prove a point and making something that is manageable comes into play. Right now we're looking at building out a small POP in another building. I'm seriously considering a pair of Linux boxes running Quagga rather than 7200s that we'd normally go with. I can easily dump 3+ full loads on them, plus I can get gig connections on PCIe without having to fork out 10 grand on a NPE-G1. Am I going to do it? No idea. Technically, there is no issue. If I drop dead the day after it's built and someone new has to maintain it, then that's a potential problem. David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgdATIgPQWnLowkRAjPvAKDSoK/9kAZNjjQrix5aoMhM0v5fvACg7ilj 0fJYz8JLrH7iTjP49+XgmvE= =RAkO -END PGP SIGNATURE-
Re: Zebra/linux device production networking?
On Jun 6, 2006, at 4:42 PM, Nick Burke wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? We started out on a FreeBSD/Zebra routing solution for our company (content provider). While it did work acceptably for many years, it wasn't what I'd call robust. The router was a single P4 2.4GHz server. We had 4 GigE ports to 4 uplinks, each giving us a full BGP feed. Then two more GigE ports to our switches. We could route over 750mbps easily, without any packet loss or latency. The biggest issue we'd have was Zebra's single-threadedness. After a restart of bgpd, it would spend so much CPU time handling the BGP updates that it would get very very behind in processing BGP keepalives, and our sessions would time out before it had finished handling the initial burst. I'd have to shut down all sessions, then bring them up one at a time. It wasn't so much bgpd taking that much CPU, but bgpd not having very much left after the server was handling a few hundred mbps of traffic. Perhaps a dual CPU server would have worked better, but we never tried. There were also issues where you could get two zebra routers deadlocked - they'd both have many megabytes of BGP updates to send each other, and both would want to send a full update until completion before accepting any data in. Mucking with the kernel to allow TCP sockets to have a 16MB receive buffer helped, but still wasn't a cure. You're also giving up things like RIBs, fancy queuing/rate limiting, and any kind of hardware acceleration. Doing hundreds of megabits is easy, but software based routers seem to have trouble under DoS situations (lots of tiny packets) quicker. However, it was about as close to free as you could get. We re-used an old server, and only had to buy some 2 port ethernet cards. Support for Zebra is pretty iffy though. More often than not, I'd post a message to the Zebra mailing list to report a bug, and would get a Yeah, known bug! reply. The original author has all but abandoned development, leading to a fork called Quagga. Quagga is better (we still use it in a few places), but is still mostly a polished up Zebra. In the end, we needed to start pushing more traffic than we were able get our Zebra box to do. A couple 20+ minute outages during peak usage because of deadlocked bgpd processes helped my case that we needed to buy some Junipers instead. I know you're not giving specifics, but any kind of description of just how much traffic you're intending to push and how many ports you need would help in giving relevant advice. If you're talking about 1 BGP feed for 10mbps, I'd say go for it. If you're talking about a dozen sessions, and 2gbps of traffic... no way. Where you are between those is what really matters.
Re: Zebra/linux device production networking?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Mark D. Kaye wrote: I would be interested to know how many software (for want of a better description) routers are in live production in this kind of environment i.e. the 99.% Uptime variety, from speaking to people albeit randomly in data centres it would seem to be more common than one might expect. With the prevalence of Metro Ethernet, I'd think it's probably a pretty common thing. People run firewalls as routers (stuff like CheckPoint), which is basically Linux or FreeBSD, although not with EGP/IGP. Also does anyone have any peering policies which would exclude peers with software routers specifically, most have a requirement for the ability to support stable BGP peering but I have not seen any specific exclusions for such devices? MD5 authed BGP sessions might be an issue - At least with Linux it requires a kernel patch (works for me). I'd peered with plenty of big carriers with Linux stuff and they don't care. I probably have more issues with a carrier I peer with who uses Juniper and feeds me my prefixes at a rate of about 50/sec, rather than 2000/sec that I get from others using Cisco (My gear is Cisco in this instance) David -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEhgiuTIgPQWnLowkRAo8eAJ9ZLANIku/rvRbRn5z5/kwbNnOspwCg5HfJ nUnzCg1xmcRc/4v3uiq1/eY= =bVnW -END PGP SIGNATURE-
Re: Zebra/linux device production networking?
On Tue, 6 Jun 2006, Nick Burke wrote: First, a little background.. My CTO made my stomach curdle today when he announced that he wanted to do away with all our cisco [routers] and instead use Linux/zebra boxen. We are a small company, so naturally penny pinching is the primary motivation. That, and the sheer joy of watching me squirm. He has informed me that he has found many people who do this for their core devices. I'm not so certain about this whole situation, so I humbly ask: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? And, if you care to spend this much time, what pitfalls/benefits did you find out about after implementation? Having done exactly that previously, I wouldn't recommend it. While it will work, most of the time, reaching 99.999% will be a challenge. Amount of engineering time you will spend in order to reach that point (and to maintain your setup) will dwarf the cost of leasing proper equipment. Issues encountered: *) Performance under ddos: Linux routing stack is route-cache-based. That means, performance is a function of flows per second, and even small random src/dst ddos will kill you. Even when this is fixed, performance will be limited by pps - and the worst case performance of PC router is not as impressive as omg i can route 1gbit with p3/1ghz. In the end, worst case performance is what really matters, and it isn't all that awesome. *) Management: It takes certain amount of sysadmin time to manage each PC router (tools/etc). *) Integration: As it is not designed as a complete system, you will have little wierdnesses, such as, quagga not seeing kernel-installed routes, or netlink not being able to keep up with route updates, etc. All of those are fairly small things, but there are more than enough of them. *) Troubleshooting/continuity of operations: It takes two orders of magnitude more clue to troubleshoot zebra network - there are simply *lots* more things that can possibly go wrong - you don't worry just about your links breaking, you have to worry about your software being buggy. While any CCIE will most likely be able to troubleshoot and run a cisco-based network, pool of engineers sufficiently clued in a myriad of things that relate to troubleshooting of a PC router (ie. both network engineer, system admin, protocol engineer, kernel hacker, and at times, zebra-source-code-hacker) is far smaller. *) Maturity: While it has been improving, things like Quagga have still have stability issues and wierd issues that are resolved by killing ospfd. Because of a greater state of flux in such environment, you are likely to encounter things like oh, this bug is fixed in latest release - and then having to retest the new release which has completely different bugs. Yes, I know, you get that with proprietary vendors - but at least you get a benefit of *them* doing at least some amount of testing prior to release. *) Redundancy: Adding more redundancy to such a system is not likely to increase availability - in fact, it is likely to decrease availability because of added complexity and more things to break. Your problems are not likely to be the PC losing power (complete failure). Your problem will be things like zebra's idea of routing table being different from kernel's idea, zebra being unhappy after a transit flaps sucking up CPU time, leading to other things timing out, etc. Redundancy will excarcerbate these issues, making troubleshooting *harder*. So, in conclusion, if you have a large number of clued linux hackers who have nothing better to do, it may be a good idea. Otherwise, you'll realize you are spending far more on sysadmin time than you are saving on equipment cost. -- Alex Pilosov| DSL, Colocation, Hosting Services President | [EMAIL PROTECTED]877-PILOSOFT x601 Pilosoft, Inc. | http://www.pilosoft.com
Re: ipv6 @ sprint, somebody home?
On Tue, Jun 06, 2006 at 09:45:18PM +0200, Jeroen Massar wrote: And http://www.sprintv6.net/ doesn't contain any contact info before you say google it. Then again the following url clearly shows their 'interrest' http://www.sprintv6.net/aspath/bgp-page-complete.html Last change on the tree detected on Sun DEC 11 2005, h.22:50 those people at PAIX Palo Alto i think are still waiting for the nap lan to number out of 3ffe space. It's the same as the IPv4 lan (vlan6) you just set up the v6 ips there.. I suspect in another few days all these routes will go away and will start to be filtered more effectively. - jared -- Jared Mauch | pgp key available via finger from [EMAIL PROTECTED] clue++; | http://puck.nether.net/~jared/ My statements are only mine.
Re: Zebra/linux device production networking?
(resent after getting on nanog-post) On 6/6/06, Nick Burke [EMAIL PROTECTED] wrote: How many of you have actually use(d) Zebra/Linux as a routing device (core and/or regional, I'd be interested in both) in a production (read: 99.999% required, hsrp, bgp, dot1q, other goodies) environment? I work for a company putting together an open source router platform. (Vyatta.com) We have a linux distro that is built off of XORP, but has plenty of enhancements that make it more friendly for a typical router jockey. It has dot1q support, bgp, ospf, rip, vrrp and many other goodies. We're currently going through UNH testing of protocol conformance. We are always looking for folks to test the software out and see how it suits their needs. (or not) Caveats: 1. Keep in mind that current sever hardware won't push line rate GigE at 64-bytes, but I find it quite reasonable as a candidate for the access layer. (t1/t3 and possibly oc3 termination) So don't expect it to perform to the same level as dedicated hardware solutions. A few hundred Mbps of inet traffic (not 64 byte frames) is reasonable. 2. Keep in mind that cheap PC hardware will result in bad MTBF. Your PC router hardware should be quality gear with redundancy if you can't tolerate any downtime. We believe there's a place for open source routing platforms, but it'll take some testing from the router community to solidify and verify the stacks. Want to help? --joel
Lightning talk selections
The Program Committee has selected these lightning talks for presentation during the general session tomorrow morning: Analysis of DNS Root Server Location, Martin Hannigan Metro WDM in provider networks, Alex Pilosov Fashonably Late - What Your Networks RTT Says About Itself, Anton Kapela Thepiratebay busted - network impact, Mikael Abrahamsson Alerting prefix owners of hijacks in near real time, Mohit Lad Reigning in the botnets operating on your network, Rick Wesson Thanks to everyone who submitted a proposal, unfortunately we couldn't fit them all into the available time. Steve Feldman PC chair
Fwd: At Dig BIG, Nothing Runs Like a Deere (Really Backhoes and Fiber together)
All, Just in case any of you want to see how the other half lives... and destroy some infrastructure after learning more about how to build it this week. :) Work with JOHN DEERE Equipment to build REAL utilities in a REAL work setting. Dig BIG takes place at the best place on the planet for underground working and learning - The Planet Underground. For two BIG days, The Planet Underground becomes a huge work site with acres of REAL underground utilities. You won't want to miss this REAL BIG opportunity to use the John Deere equipment at Dig BIG. Work side by side with colleagues with the best equipment available! Free registration available for a limited time.http://underspace.com/mailer/redir.php?id=239st_id=164 I subscribe to Underground Focus magazine and I just received this in my inbox. Register at http://www.underspace.com/TPU/register_digbig.php in case you really do want to attend. Now back to the regularly scheduled noise with some content due to NANOG 37. -Robert Tellurian Networks - The Ultimate Internet Connection http://www.tellurian.com | 888-TELLURIAN | 973-300-9211 Well done is better than well said. - Benjamin Franklin
Re: ipv6 @ sprint, somebody home?
On Tue, 2006-06-06 at 21:45 +0200, Jeroen Massar wrote: Hello, [EMAIL PROTECTED]: host kay.sprintlink.net[199.0.233.8] said: 553 5.3.0 [EMAIL PROTECTED]... User Unknown (in reply to RCPT TO command) The correct email address is [EMAIL PROTECTED] -- Nicolas DEFFAYET NDSoftware http://www.ndsoftware.com/
