RE: Charter ARP Leak
We'll I would for one be very interested if the 8 ARP packets a second count against the caps. Given len of 46 or 60 is not much, but that's about a gig of traffic almost assuming 8 of those a second happen(and my cold medicine addled mind is working). I'm sure it's not just that when it comes to garbage received on your interface either. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of David Coulson Sent: Monday, December 29, 2014 12:57 PM To: nanog@nanog.org Subject: Re: Charter ARP Leak Not sure I understand what all the excitement is about?
Google IP Engineering Contact
Can someone at Google that has some control over how Google’s system that says an IP is suddenly in Iran please contact me off list ASAP. We can not wait a month for your tool to remove this. Also in the future that tool needs to ping the ip and check the RTT time and compare to well established GEOIP databases before flagging ip’s as not belonging to the country they’re actually in. We had to change our office firewall ip because suddenly Google thought it was in China instead of Georgia a few months back. Corey Touchet Manager Enterprise Infrastructure Total Server Solutions
Re: 10Gb iPerf kit?
You really want one of these http://www.jdsu.com/en-us/Test-and-Measurement/products/a-z-product-list/Pa ges/tb-6000.aspx#.VGFcetZ65PI Or it¹s larger 9000 series that can scale to 100Gb. On 11/10/14, 7:26 PM, Daniel Rohan dro...@gmail.com wrote: We're looking for a semi-portable solution to validate 10Gb customer circuits and hitting walls surrounding PCI lanes and the amount of data laptops can push via their busses. We'd prefer to not have techs lugging around server equipment for these tests. Anyone out there testing 10gbE with iPerf? If so, what are you using? Thanks, Dan
Re: Multicast Internet Route table.
14 years at Verizon Wireless and I despised the crop of multicast products that seemed to pop up from time to time. Even in a fully controlled network multicast remains at best black magic. There are ways to make it more reliable and prevent people from ruining the setups especially for PIM type setups, but I would agree with others, unicast has better advantages though you have to keep up with the bandwidth curve. Content delivery systems moving the content closer to edge customers makes this less of a problem as well. Torrent style distribution appears to be particularly effective as long as you can maintain a pool of users to distribute the content. Blizzard has made good progress using this for game updates will a fallback to http if you can¹t get the content via torrents. On 9/2/14, 6:46 AM, John Kristoff j...@cymru.com wrote: On Tue, 2 Sep 2014 04:47:37 + S, Somasundaram (Somasundaram) somasundara...@alcatel-lucent.com wrote: 1: Does all the ISP's provide Multicast Routing by default? No not all and even those that do often do not do so on the same gear, links and peers as their unicast forwarding. 2: Is there any placeholder where one can get to know the Multicast Internet Route table (usage, stability etc) just like Unicast Route table (http://bgpupdates.potaroo.net)? Marshall Eubanks at one time probably maintained the most comprehensive IP multicast status pages at http://www.multicasttech.com/status (no longer available). I've not seen nor heard from Marshall in a long time so I wouldn't expect this to come back any time soon. Sadly I don't know of any suitable replacement, but you might find some of that by searching here, if nothing else using the router proxies to examine status by hand: http://noc.net.internet2.edu/ CAIDA used to do some, but I'm not sure they have anything active any longer, browsing their tools and data may turn up some hints to other work. The once NLANR inspired and run Beacon project hasn't completely died out, there is this I found at ja.net for instance: http://www.beacon.ja.net/ Interdomain IP multicast has practically since the beginning been a notoriously niche and limited service compared to unicast service. There are a handful of reasons for that, but I think you will find it becoming decreasingly available rather than more so on interdomain basis. John
Re: Recommendations, Colo Reno, Albuquerque, Phoenix, Las Vegas
See response off list. On 9/2/14, 5:35 PM, Eric A Louie elo...@yahoo.com wrote: Does anyone have recommendations for Colocation space in any of those 4 cities? thanks Eric
Re: RTT of ICMP TTL exceeded messages in Level3 network remains the same throughout the network
Depends on the setup. With MPLS and traffic engineering tunnels you can make a 30 hop path look like one hop easily. On 8/13/14, 9:25 AM, Martin T m4rtn...@gmail.com wrote: Hi, if I make a traceroute to a host in San Jose in Level3 network from DigitalOcean server in Amsterdam, then in Level3 network(hop 6 in example below) the RTT remains the same: # traceroute -q 1 -I ZYNGA-INC.edge1.SanJose3.Level3.net traceroute to ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114), 30 hops max, 60 byte packets 1 5.101.103.253 (5.101.103.253) 0.265 ms 2 95.85.0.229 (95.85.0.229) 0.236 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.275 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.630 ms 5 4.68.63.41 (4.68.63.41) 0.635 ms 6 vl-3603-ve-227.csw2.Amsterdam1.Level3.net (4.69.162.153) 155.309 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 155.627 ms 8 ae-46-46.ebr2.London1.Level3.net (4.69.143.74) 153.470 ms 9 * 10 ae-61-61.csw1.NewYork1.Level3.net (4.69.134.66) 148.972 ms 11 * 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 147.881 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 149.632 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 151.107 ms 15 ZYNGA-INC.edge1.SanJose3.Level3.net (4.53.208.114) 154.431 ms # In other words, one sees the RTT of the end-host as a RTT for all the hops in Level3 netwotk. If I make the traceroute to penultimate hop ae-4-90.edge1.SanJose3.Level3.net, then RTT is as expected: root@vserver:~# traceroute -q 1 -I ae-4-90.edge1.SanJose3.Level3.net traceroute to ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208), 30 hops max, 60 byte packets 1 5.101.103.254 (5.101.103.254) 0.228 ms 2 95.85.0.237 (95.85.0.237) 0.217 ms 3 ix-4-2-0-0.tcore1.AV2-Amsterdam.as6453.net (195.219.194.25) 0.276 ms 4 if-7-2.tcore1.AD1-Amsterdam.as6453.net (195.219.194.46) 0.656 ms 5 4.68.63.41 (4.68.63.41) 0.607 ms 6 vl-3604-ve-228.csw2.Amsterdam1.Level3.net (4.69.162.157) 0.696 ms 7 ae-56-221.ebr2.Amsterdam1.Level3.net (4.69.153.201) 0.677 ms 8 ae-45-45.ebr2.London1.Level3.net (4.69.143.70) 7.059 ms 9 ae-44-44.ebr1.NewYork1.Level3.net (4.69.137.78) 76.311 ms 10 ae-81-81.csw3.NewYork1.Level3.net (4.69.134.74) 76.265 ms 11 ae-82-82.ebr2.NewYork1.Level3.net (4.69.148.41) 76.820 ms 12 ae-2-2.ebr1.SanJose1.Level3.net (4.69.135.185) 149.101 ms 13 ae-91-91.csw4.SanJose1.Level3.net (4.69.153.14) 150.557 ms 14 ae-4-90.edge1.SanJose3.Level3.net (4.69.152.208) 162.022 ms root@vserver:~# All the ICMP TTL exceeded messages except the first and the penultimate one in Level3 network have MPLS extensions header(s24.postimg.org/4z9at9z45/ICMP_echo_reply_MPLS_extensions.png) which is always the same except the tag value changes. How does this technically work? What are the advantages of such setup? thanks, Martin
Re: AutoTask as a ticketing system in a MNS NOC
It¹s not taboo to say Remedy is it? On 8/7/14, 2:49 PM, Ameen Pishdadi apishd...@gmail.com wrote: Well what do u recommend Sent from my iPhone On Aug 7, 2014, at 3:08 PM, Chris Adams c...@cmadams.net wrote: Once upon a time, Chris Garrett ch...@aperturefiber.com said: Does anyone on list have any firsthand experience with this software as a primary ticketing platform in a high volume NOC? A small ISP I used to work for switched to Autotask a couple of years ago, and I was not impressed. The web UI was slow, the API was slower, and their standard mail gateway was broken. For example: they used AT for CRM as well, and the mail gateway tried to auto-associate tickets with contacts based on email address. That would be great, but we had some people that were contacts for multiple customers (using the same email address), and emails from them to the ticket system would just go into a black hole (no ticket, no bounce, no notification). There are various third-party tools available to handle the email gateway as well; I don't know how well they may work, but it seemed to me that a ticket system that needed third-party tools to handle email was broken. -- Chris Adams c...@cmadams.net
Re: IPv6 route annoucement
Pretty strong reaction for a single prefix. Now if you said you wanted to advertise all your /64¹s that would be a different conversation. On 8/7/14, 2:58 PM, John York jo...@griffintechnology.com wrote: Hoping to not start a war... We (a multi-homed end-user site) are finally getting IPv6-enabled Internet connectivity from one of our ISPs. In conversations regarding our BGP config, the ISP has balked at allowing us to advertise our ARIN-assigned /44, saying things like, do you know how many addresses that is!!?? Am I way off base in thinking this network size is not out of the norm? I know it's a lot of addresses (19 octillion-something?), but that assignment was based on the same criteria that got us a /22 in v4 space. Should accepting a /44 in v6 not be equivalent, policy-wise, to accepting a /22 in v4? Thanks, John -- John York Information Technology | Network Administrator Phone: 615-399-7000 x:333 Griffin Technology 2030 Lindell Avenue Nashville, TN 37203 USA
Re: ASR9K xml agent vs netconf
I always preferred the displays where you have commands without all the bracket garbage and just indented text for sub items. On the MX the show configuration | display set is about as close as you can get, but it¹s workable. Kudo¹s is that you can just dump it in as well and get what you want. I think the only time I really get annoyed at the JunOS configurations is when I¹m staring down any of their switches. On 8/5/14, 7:32 AM, Dale W. Carder dwcar...@wisc.edu wrote: Thus spake Jeremy (jba...@gmail.com) on Fri, Aug 01, 2014 at 03:07:19PM -0700: I'm currently working on writing some automation around the ASR9K platform and I've been looking at both the netconf and xml interfaces. Anyone have experience with either? It looks like the XML interface is much more feature rich, supporting both config and operational state objects where netconf is limited to config only. Currently I'm leaning towards the xml interface, I wasted a week of my life trying to get xml interface on n9k to work correctly. I would never use it again, as it obviously gets no QA. There is likely a fundamental design flaw in that the cli is not itself an xml client like you see on other platforms. The XML interface, and CLI (presumably netconf) may all be distinct clients to sysdb. I did get (3) ddts' assigned, related to missing data compared to cli, endian issues, etc. My recommendation is DO NOT USE IT. I went back to screen scraping for ios-xr. Related to this and other issues, all of our subsequent purchases have been MX. Dale AS{59,2381,3128}
Re: The Cidr Report
I wonder if it would be possible to run a bogon style BGP server that tells you about various subnets that have a valid higher aggregation so we can easily filter out the good de-aggregation vs just brute forcing via prefix filters. Sent from my iPhone On Aug 2, 2014, at 9:31 PM, keith kouzmanoff ke...@kouzmanoff.com wrote: link didn't work for me, I think http://www.cidr-report.org/as2.0/ is the proper link On 8/1/2014 5:00 PM, cidr-rep...@potaroo.net wrote: This report has been generated at Fri Aug 1 21:13:59 2014 AEST. The report analyses the BGP Routing Table of AS2.0 router and generates a report on aggregation potential within the table. Check http://www.cidr-report.org/2.0 for a current version of this report. Recent Table History Date PrefixesCIDR Agg 25-07-14508935 285928 26-07-14508775 286040 27-07-14508959 286213 28-07-14509275 286189 29-07-14509477 286110 30-07-14509841 286214 31-07-14510150 286361 01-08-14510519 286381 AS Summary 47759 Number of ASes in routing system 19365 Number of ASes announcing only one prefix 3794 Largest number of prefixes announced by an AS AS28573: NET Serviços de Comunicação S.A.,BR 120495616 Largest address span announced by an AS (/32s) AS4134 : CHINANET-BACKBONE No.31,Jin-rong Street,CN Aggregation Summary The algorithm used in this report proposes aggregation only when there is a precise match using the AS path, so as to preserve traffic transit policies. Aggregation is also proposed across non-advertised address space ('holes'). --- 01Aug14 --- ASnumNetsNow NetsAggr NetGain % Gain Description Table 510651 286295 22435643.9% All ASes AS28573 3794 214 358094.4% NET Serviços de Comunicação S.A.,BR AS6389 2943 80 286397.3% BELLSOUTH-NET-BLK - BellSouth.net Inc.,US AS17974 2801 190 261193.2% TELKOMNET-AS2-AP PT Telekomunikasi Indonesia,ID AS7029 2887 485 240283.2% WINDSTREAM - Windstream Communications Inc,US AS4766 2949 928 202168.5% KIXS-AS-KR Korea Telecom,KR AS18881 2062 43 201997.9% Global Village Telecom,BR AS7545 2347 677 167071.2% TPG-INTERNET-AP TPG Telecom Limited,AU AS18566 2047 565 148272.4% MEGAPATH5-US - MegaPath Corporation,US AS10620 2939 1463 147650.2% Telmex Colombia S.A.,CO AS7303 1775 438 133775.3% Telecom Argentina S.A.,AR AS22773 2725 1401 132448.6% ASN-CXA-ALL-CCI-22773-RDC - Cox Communications Inc.,US AS4755 1866 594 127268.2% TATACOMM-AS TATA Communications formerly VSNL is Leading ISP,IN AS4323 1642 424 121874.2% TWTC - tw telecom holdings, inc.,US AS6983 1390 314 107677.4% ITCDELTA - Earthlink, Inc.,US AS22561 1305 242 106381.5% AS22561 - CenturyTel Internet Holdings, Inc.,US AS7552 1261 237 102481.2% VIETEL-AS-AP Viettel Corporation,VN AS9829 1653 738 91555.4% BSNL-NIB National Internet Backbone,IN AS6147 1043 147 89685.9% Telefonica del Peru S.A.A.,PE AS38285 956 112 84488.3% M2TELECOMMUNICATIONS-AU M2 Telecommunications Group Ltd,AU AS24560 1153 345 80870.1% AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services,IN AS4808 1207 416 79165.5% CHINA169-BJ CNCGROUP IP network China169 Beijing Province Network,CN AS7738 977 190 78780.6% Telemar Norte Leste S.A.,BR AS4788 1023 261 76274.5% TMNET-AS-AP TM Net, Internet Service Provider,MY AS8151 1450 691 75952.3% Uninet S.A. de C.V.,MX AS18101
Re: Muni Fiber and Politics
But in the cases of small rural communities it¹s perfectly reasonable to just setup wifi to cover the town and backhaul a DS3 back to a more connected location. There¹s plenty of small wireless companies out there trying to serve these folks. On 8/2/14, 3:15 PM, Leo Bicknell bickn...@ufp.org wrote: There are plenty of cities with zero ISP's interested in serving them today, I can't argue that point. However I believe the single largest reason why that is true is that the ISP today has to bear the capital cost of building out the physical plant to serve the customers. 15-20 year ROI's don't work for small businesses or wall street. But if those cities were to build a municipal fiber network like we've described, and pay for it with 15-20 year municipal bonds the ISP's wouldn't have to bear those costs. They could come in drop one box in a central location and start offering service. Which is why I said, if municipalities did this, I am very skeptical there would be more than a handful without a L3 operator. You can imagine a city of 50 people in North Dakota or the Northern Territories might have this issue because the long haul cost to reach the town is so high, but it's going to be a rare case. I firmly believe the municipal fiber networks presence would bring L3 operators to 90-95% of cities. On Aug 2, 2014, at 2:04 PM, Scott Helms khe...@zcorum.com wrote: Happens all the time, which is why I asked Leo about that scenario. There are large swarths of the US and even more in Canada where that's the norm. On Aug 2, 2014 1:29 PM, Owen DeLong o...@delong.com wrote: Such a case is unlikely. On Aug 1, 2014, at 13:32, Scott Helms khe...@zcorum.com wrote: I can never see a case where letting them play at Layer 3 or above helps. That¹s bad news, stay away. But I think some well crafted L2 services could actually _expand_ consumer choice. I mean running a dark fiber GigE to supply voice only makes no sense, but a 10M channel on a GPON serving a VoIP box mayŠ Even in those cases where there isn't a layer 3 operator nor a chance for a viable resale of layer 1/2 services. -- Leo Bicknell - bickn...@ufp.org - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/
Re: Muni Fiber and Politics
Not really, the law can say must provide standards compliant access for interconnections with a agreed upon base set of features it must support. Any provider that wants something extra can negotiate the reasonable costs of implementation. On 8/1/14, 8:44 AM, Owen DeLong o...@delong.com wrote: On Aug 1, 2014, at 12:08 AM, Mark Tinka mark.ti...@seacom.mu wrote: On Friday, August 01, 2014 08:54:07 AM mcfbbqroast . wrote: This would be my humble suggestion: - lines provider runs fibre pair from each home to co. By default the lines provider installs a simple consumer terminal, with gigabit Ethernet outputs and POTS. The problem with this is it allows the lines provider to dictate the technology to be used by all higher-layer service providers. IMHO, this is undesirable, because it blocks innovation and service differentiation on this basis. Ideally, the lines provider is simply a lines provider and provides a number of dark fiber pairs between the serving wire center (what you called a CO) and each premise served by the SWC. Termination at the customer end should be a box in which a customer terminal can be installed and the fibers should all be terminated on some standard form of patch panel (ST or LC probably preferred, but others may be acceptable). It would then be up to the service provider(s) to provide the terminals and decide between customer self-install and truck-rolls for service turn-up. - lines provider provides a reasonably oversubscribed service to soft hand over to ISPs (think 96 Gbps lines to 2 10gbps ports). Perhaps upgrading so such a ratio never becomes congested could be a requirement? Putting the lines provider into this part of the equation preserves many of the problems with the existing model. - lines provider also rents individual lines to ISPs which they can use directly. Rent should be lower than soft handover. Now you¹ve got competition operating at a disadvantage to the incumbent lines provider, preserving this aspect of the problems with the current system. IMHO, this should be the only service the lines provider is allowed to sell. In that way, the lines provider is not in competition with its wholesale customers. If you want examples of how well the model you propose tends to work, look no further than the incredible problematic nature of MCI¹s attempt to offer local phone service over Pacific Bell/SBC/ATT circuits. This way ISPs can easily offer services. POTS over VoIP can be setup on installation of the terminal (so handover to the ISP is seamless). Finally business and residential services can also be provided over the fibre directly (this will be attractive to ISPs with many ports, to reduce costs, and premium/business ISPs to add control). This is also true of dark fiber pairs, with the added advantage that the service providers can differentiate themselves on chosen technology, can offer innovative services and can leverage existing infrastructure to deploy newer technologies as they develop. - ideally the lines provider would aid in providing cheap backhaul from the co (while still allowing 3rd party users to bring fibre in). I don¹t think this is so ideal. Again, it creates an opportunity for the lines provider to leverage their infrastructure in a way that it can become a barrier to competition. This is, IMHO, the opposite of good. Wholesale mode. Doable. Works best if the lines provider is not a service provider; or regulation in your market ensures a service provider who is also a lines provider is mandated to unbundle at reasonable cost. Even when mandated to unbundle at a reasonable cost, often other games are played (trouble ticket for service billed by lines provider resolved in a day, trouble ticket for service on unbundled element resolved in 14 days, etc.). IMHO, experience has taught us that the lines provider (or as I prefer to call them, the Layer 1 infrastructure provider) must be prohibited from playing at the higher layers. Owen
Re: Carrier Grade NAT
There¹s still a lot of websites that are not with the times. No ipv6 on CNN, FOX, or NBC news websites. Slashdot.org shame on you! Comcast and ATT work, but not Verizon. No surprise there. Power company nope. I think CGN is fine for 99% of customers out there. Until the iPhone came out Verizon Wireless had natted all their blackberry customers and saved million¹s of IP¹s. Then Apple and Google blew a hole into that plan. Then again I¹m for IPv4 just running out and finally pushing people to adopt. The US Govt has done a better job of moving to IPv6 than private industry which frankly is amazing all things considered. Comcast is pushing over 1TBPS of IPv6 traffic, but I¹m sure that¹s mainly video from Youtube and Netflix. On 7/30/14, 9:45 AM, Owen DeLong o...@delong.com wrote: The only actual residential data I can offer is my own. I am fully dual stack and about 40% of my traffic is IPv6. I am a netflix subscriber, but also an amazon prime member. I will say that if amazon would get off the dime and support IPv6, it would make a significant difference. Other than amazon and my financial institutions and Kaiser, living without IPv4 wouldn't actually pose a hardship as near as I can tell from my day without v4 experiment on June 6. I know Kaiser is working on it. Amazon apparently recently hired Yuri Rich to work on their issues. So that would leave my financial institutions. I think we are probably less than 5 years from residential IPv4 becoming a service that carries a surcharge, if available. Owen On Jul 29, 2014, at 22:42, Julien Goodwin na...@studio442.com.au wrote: On 29/07/14 22:22, Owen DeLong wrote: On Jul 29, 2014, at 4:13 PM, Mark Andrews ma...@isc.org wrote: In message 20140729225352.go7...@hezmatt.org, Matt Palmer writes: On Wed, Jul 30, 2014 at 09:28:53AM +1200, Tony Wicks wrote: 2. IPv6 is nice (dual stack) but the internet without IPv4 is not a viable thing, perhaps one day, but certainly not today (I really hate clueless people who shout to the hills that IPv6 is the solution for today's internet access) Do you have IPv6 deployed and available to your entire customer base, so that those who want to use it can do so? To my way of thinking, CGNAT is probably going to be the number one driver of IPv6 adoption amongst the broad customer base, *as long as their ISP provides it*. Add to that over half your traffic will switch to IPv6 as long as the customer has a IPv6 capable CPE. That's a lot less logging you need to do from day 1. That would be nice, but I¹m not 100% convinced that it is true. Though it will be an increasing percentage over time. Definitely a good way of reducing the load on your CGN, with the additional benefit that your network is part of the solution rather than part of the problem. Being on the content provider side I don't know the actual percentages in practice, but in the NANOG region you've got Google/Youtube, NetFlix, Akamai Facebook all having a significant amount of their services v6 native. I'd be very surprised if these four together weren't a majority of any consumer-facing network's traffic in peak times.
Upgrade Path Options from 6500 SUP720-3BXL for Edge Routing
I’m curious what other providers have gone with when moving away from SUP720-3BXL 6500 platforms. I’m platform agnostic and just as comfortable with Juniper as with Cisco. It’s a conversation were having since the 3BXL’s are running into limits with the large number of prefixes, long eBGP convergence times, and 10G port density. Basically were looking to carry multiple full routing tables from several 4+ carriers plus internet exchange traffic so the ability to handle 1-2M IPV4 and 500K+ IPv6 and decent 10G port density and/or 40G options as well. Also should have decent CPU capabilities so it can crunch routes in a reasonable amount of time. Right now my thinking are MX480 or ASR9k platforms. Opinions on those are equally welcome as alternatives, but I’d love to hear from those with personal experiences today vs sales people trying to tell me it would route the world :) Thanks, Corey T
Re: Many players make up application performance (was Re: Richard Bennett, NANOG posting, and Integrity)
What I would like to see is someone who sets up a VPN that has an endpoint path that¹s the same as NetFlix. If their streaming performance improves that would be very telling. Heck you could use 2 machines and do a side by side. However I doubt Level3 is going to sit there and lie about their connection to Verizon being overloaded, and for Verizon to do any kind of meaningful QOS it would require an effort on the Level3 side of the connection as well. On 7/29/14, 8:33 AM, McElearney, Kevin kevin_mcelear...@cable.comcast.com wrote: On 7/28/14, 5:35 PM, Jim Richardson weaselkee...@gmail.com wrote: I pay for (x) bits/sec up/down. From/to any eyecandysource. If said eyecandy origination can't handle the traffic, then I see a slowdown, that's life. But if $IP_PROVIDER throttles it specifically, rather than throttling me to (x),I consider that fraud. I didn't pay for (x) bits/sec from some whitelist of sources only. Along with paying $IP_PROVIDER for (x) bits/sec up/down, you are also paying (or the product of advertising) eyecandysource to deliver a service (w/ a level of quality). $IP_PROVIDER plays a big role in delivering your *overall* Internet experience, but eyecandysource plays an even bigger role delivering your *specific* eyecandy experience. If eyecandystore has internal challenges, business negotiation/policy objectives, or uses poor adaptive routing path decisions, this has a direct and material impact to your *specific* eyecandy experience (and some have found fixable by hiding your source IP with a VPN). While ISPs do play a big role in this, people tend to miss eyecandystore decisions (and business drivers) as a potential factors in isolated application performance issues.