Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

It looks like Google is experimenting with a change in course on this issue.

Here’s a look at the IPv6 routing table tonight on my router bordering Cogent.

*>i 2607:f8b0:4013::/48
2620:121:a000:f0::2(fe80::618:d6ff:fef1:c540)
  0150  0   15169 i
*2001:550:2:22::1d:1(fe80::12f3:11ff:fe29:2c24)
  090   0   174 
6461 15169 i
*>i 2607:f8b0:4014::/48
2620:121:a000:f0::2(fe80::618:d6ff:fef1:c540)
  0110  0   6939 
6461 15169 i
*2001:550:2:22::1d:1(fe80::12f3:11ff:fe29:2c24)
  090   0   174 
6461 15169 i
*>i 2607:f8b0:4016::/48
2620:121:a000:f0::2(fe80::618:d6ff:fef1:c540)
  0150  0   15169 i
*2001:550:2:22::1d:1(fe80::12f3:11ff:fe29:2c24)
  090   0   174 
6461 15169 i


This is only 3 IPv6 prefixes (out of 47 prefixes seen in my IPv6 routing 
table).  Two of these prefixes I see via direct peering with Google and, 
alternatively, via Cogent through Zayo transit.  One of these prefixes doesn’t 
advertise in Google’s direct peering session (at least not in mine, but HE 
picks it up via Zayo and Cogent picks it up via Zayo).

All of these are /48 subnets of their greater 2620:f8b0::/32 prefix, which does 
show up in both their direct session and in HE via Zayo.


> On Mar 13, 2016, at 9:31 AM, Dennis Burgess  wrote:
> 
> In the end, google has made a choice. I think these kinds of choices will 
> delay IPv6 adoption.  
> 
> -Original Message-
> From: Damien Burke [mailto:dam...@supremebytes.com] 
> Sent: Friday, March 11, 2016 2:51 PM
> To: Mark Tinka ; Owen DeLong ; Dennis 
> Burgess 
> Cc: North American Network Operators' Group 
> Subject: RE: Cogent - Google - HE Fun
> 
> Just received an updated statement from cogent support:
> 
> "We appreciate your concerns. This is a known issue that originates with 
> Google as it is up to their discretion as to how they announce routes to us 
> v4 or v6. 
> 
> Once again, apologies for any inconvenience."
> 
> And:
> 
> "The SLA does not cover route transit beyond our network. We cannot route to 
> IPs that are not announced to us by the IP owner, directly or through a 
> network peer."
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

I understand.  I tend to take a more market by market view of each network 
rather than a global perspective.  Clearly, for the enterprise use case with a 
diversity of users spread across the globe, or even nationally, the use case is 
a bit different.

Having said that, I am rather terribly curious...  I’ve not really seen any of 
the major national non-eyeballs who didn’t have congestion at some peering 
points to major eyeball networks for not insignificant periods. 

Which transit have you found to be the very best for minimizing those concerns 
in the general case?


> On Mar 14, 2016, at 1:23 PM, Matthew Huff <mh...@ox.com> wrote:
> 
> We don't serve a market. We are a private business. We are multi-homed with 
> multiple providers, none of which is an eyeball network. Even if we wanted to 
> peer, most of them are not available in our area, but our the only choice for 
> some of our employees.
> 
> Cogent still has congestion issues at various peering points as has been 
> reported in this and other mailing lists recently. Like I said, if VOIP and 
> VPN aren't an issue, go ahead and use cogent. But if packet loss makes your 
> access useless, then avoid them if it all possible. YMMV.
> 
> 
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC   | Phone: 914-460-4039
> aim: matthewbhuff| Fax:   914-694-5669
> 
> 
>> -Original Message-
>> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
>> Sent: Monday, March 14, 2016 1:41 PM
>> To: Matthew Huff <mh...@ox.com>
>> Cc: William Herrin <b...@herrin.us>; James Milko <jmi...@gmail.com>;
>> nanog@nanog.org
>> Subject: Re: Cogent - Google - HE Fun
>> 
>> I would have concurred on this not so very long ago, but Cogent has made
>> serious strides in improving this.
>> 
>> In particular, I think Cogent is fairly trustworthy to at least AT and
>> Verizon at this point.
>> 
>> As for Charter, Comcast, Cox, and the like, I’ve come to believe that
>> there’s really no substitute for direct interconnection to those guys if
>> they’re part of the market you serve.
>> 
>> My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs, if
>> they’re serving clients whose broadband access is one of the major cable
>> providers, I always encourage the client to establish a BGP session
>> directly into that provider (whether purchasing enterprise transit from
>> them, but just accepting customer routes and advertising with a no-
>> export prefix or formal paid peering, etc.)
>> 
>> The impact that it has on service quality is measurable and it’s a
>> significant impact in many cases.
>> 
>>> On Mar 14, 2016, at 9:58 AM, Matthew Huff <mh...@ox.com> wrote:
>>> 
>>> One caveat about Cogent even as a third or extra provider.
>>> 
>>> Because of disputes with eyeball networks, there is significant
>> congestion at peering points with Cogent. We saw consistent 5-10% packet
>> loss over many months traversing Cogent through to Charger, Cox and
>> Verizon as well as others. For web access and even streaming video, with
>> buffers, this might not be an issue. But for corporate use with VOIP
>> and/or VPNs, it was a killer. We had to cancel our Cogent service and
>> work with our remaining providers to de-preference Cogent completely.
>>> 
>>> 
>>> 
>>> 
>>> Matthew Huff | 1 Manhattanville Rd
>>> Director of Operations   | Purchase, NY 10577
>>> OTA Management LLC   | Phone: 914-460-4039
>>> aim: matthewbhuff| Fax:   914-694-5669
>>> 
>>> 
>>>> -Original Message-
>>>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William
>> Herrin
>>>> Sent: Monday, March 14, 2016 10:47 AM
>>>> To: James Milko <jmi...@gmail.com>
>>>> Cc: nanog@nanog.org
>>>> Subject: Re: Cogent - Google - HE Fun
>>>> 
>>>> On Mon, Mar 14, 2016 at 9:14 AM, James Milko <jmi...@gmail.com>
>> wrote:
>>>>> On Sun, Mar 13, 2016 at 8:32 PM, William Herrin <b...@herrin.us>
>>>> wrote:
>>>>>> At the very least, no one who is clueful about "The Internet" is
>>>>>> single-homed to Cogent with any protocol.
>>>>> 
>>>>> s/single-homed/dual-homed/
>>>>> 
>>>>> It's not like losing Google/HE because your other transit dropped is
>>>>> acceptable.
>>>> 
>>>> Hi James,
>>>> 
>>>> Cogent is effective at reducing cost as the third or subsequent
>> provider
>>>> in one's mix.
>>>> 
>>>> Regards,
>>>> Bill Herrin
>>>> 
>>>> 
>>>> --
>>>> William Herrin  her...@dirtside.com  b...@herrin.us
>>>> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

I would have concurred on this not so very long ago, but Cogent has made 
serious strides in improving this.

In particular, I think Cogent is fairly trustworthy to at least AT and 
Verizon at this point.

As for Charter, Comcast, Cox, and the like, I’ve come to believe that there’s 
really no substitute for direct interconnection to those guys if they’re part 
of the market you serve.

My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs, if they’re 
serving clients whose broadband access is one of the major cable providers, I 
always encourage the client to establish a BGP session directly into that 
provider (whether purchasing enterprise transit from them, but just accepting 
customer routes and advertising with a no-export prefix or formal paid peering, 
etc.)

The impact that it has on service quality is measurable and it’s a significant 
impact in many cases.

> On Mar 14, 2016, at 9:58 AM, Matthew Huff  wrote:
> 
> One caveat about Cogent even as a third or extra provider.
> 
> Because of disputes with eyeball networks, there is significant congestion at 
> peering points with Cogent. We saw consistent 5-10% packet loss over many 
> months traversing Cogent through to Charger, Cox and Verizon as well as 
> others. For web access and even streaming video, with buffers, this might not 
> be an issue. But for corporate use with VOIP and/or VPNs, it was a killer. We 
> had to cancel our Cogent service and work with our remaining providers to 
> de-preference Cogent completely.
> 
> 
> 
> 
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC   | Phone: 914-460-4039
> aim: matthewbhuff| Fax:   914-694-5669
> 
> 
>> -Original Message-
>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin
>> Sent: Monday, March 14, 2016 10:47 AM
>> To: James Milko 
>> Cc: nanog@nanog.org
>> Subject: Re: Cogent - Google - HE Fun
>> 
>> On Mon, Mar 14, 2016 at 9:14 AM, James Milko  wrote:
>>> On Sun, Mar 13, 2016 at 8:32 PM, William Herrin 
>> wrote:
 At the very least, no one who is clueful about "The Internet" is
 single-homed to Cogent with any protocol.
>>> 
>>> s/single-homed/dual-homed/
>>> 
>>> It's not like losing Google/HE because your other transit dropped is
>>> acceptable.
>> 
>> Hi James,
>> 
>> Cogent is effective at reducing cost as the third or subsequent provider
>> in one's mix.
>> 
>> Regards,
>> Bill Herrin
>> 
>> 
>> --
>> William Herrin  her...@dirtside.com  b...@herrin.us
>> Owner, Dirtside Systems . Web: 



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

Freddy,

As there is no IPv6 transit between HE and Cogent, this would have the effect 
of isolating ones network services from the single-homed customers of Cogent.  
I’m not sure that many of us could get away with that.  Further, I’m not sure 
that it’s appropriate to punish the single-homed Cogent customers.  I’ll grant, 
this is just what Google has done, but they’re well positioned to weather that 
storm and have a level of visibility and brand loyalty that will allow them to 
have a chance of success at it.

I think the softer approach of reducing the relevancy of Cogent’s IPv6 transit 
service and indeed the relevancy of peering with Cogent for IPv6 is a way 
forward that more of us could get behind.

Thanks,

Matt Hardeman

> On Mar 10, 2016, at 4:42 PM, Fredy Kuenzler <kuenz...@init7.net> wrote:
> 
> This would work for those which are using IPv6 transit from Cogent.
> 
> For anyone else which is using IPv6 transit from Hurricane Electric and some 
> other suppliers such as L3 or NTT: to set the community 'do not announce to 
> Cogent' only on every other transit but HE would help to isolate Cogent 
> without much collateral damage. It would support Google/HE's position. And 
> maybe help to bring back Cogent onto a cooperative track, after all.
> 
> --
> Fredy Kuenzler
> Init7 (Switzerland) Ltd.
> St.-Georgen-Strasse 70
> CH-8400 Winterthur
> Switzerland
> 
> http://www.init7.net/
> 
> 
>> Am 10.03.2016 um 23:19 schrieb Matthew D. Hardeman <mharde...@ipifony.com>:
>> 
>> I have contemplated whether such mechanisms matter to Cogent, etc.
>> 
>> I’m inclined to think that if Google is willing to pull the routes and they 
>> still don’t blink, then certainly us smaller shops aren’t going to impact 
>> them…
>> 
>> However…  If enough prefixes disappear from the _apparent_ Cogent table as 
>> viewed by outsiders, this may ultimately impact their sales of new 
>> interconnection….
>> 
>> For those of us multihomed with Cogent and other transit providers on IPv6 
>> there is a less drastic way to impact the perceived value of Cogent’s IPv6 
>> routing table to outsiders and especially to Cogent’s peers — and one that 
>> still doesn’t negatively impact the single-home customers of Cogent:
>> 
>> "set community 174:3000" on your IPv6 advertisement to Cogent.  This will 
>> constrain the advertisement to Cogent and Cogent’s customers only.  For good 
>> measure, prepend your own AS to this advertisement at least a couple of 
>> times, potentially discouraging even Cogent customers who see the route from 
>> using it if they have other transit.  It will prevent the path via Cogent 
>> being selected by Cogent IPv6 peers versus your other transit providers.
>> 
>> 
>>> On Mar 10, 2016, at 3:47 PM, Fredy Kuenzler <kuenz...@init7.net> wrote:
>>> 
>>> Am 10.03.2016 um 22:25 schrieb Damien Burke <dam...@supremebytes.com>:
>>>> Anyone who is multihomed with cogent ipv6 in their mix should shutdown 
>>>> their IPv6 bgp session. Let’s see if we can make their graph freefall.
>>> 
>>> 
>>> Alternative:
>>> 
>>> set community [do not announce to Cogent]
>>> 
>>> *SCNR*
>> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

Mark,

I certainly agree that intentional harm of a purely malicious nature is to be 
discouraged.

What I proposed, as an alternative to some of the more extreme mechanisms being 
discussed, is a mechanism whereby IPv6 _customers_ of Cogent transit 
services--and who also receive IPv6 transit from at least one other 
relationship--can modify their IPv6 advertisements to Cogent such that they 
utilize that transit link with Cogent for the one thing you can reliably count 
on it for in the IPv6 world: reaching other Cogent IPv6 customers, especially 
the single-homed ones.

In essence, adding BGP community “174:3000” to your IPv6 advertisements to 
Cogent instructs Cogent that this route should only be advertised internal to 
Cogent and to Cogent’s customers.  It should not be announced to peers.  
Combining that with prepends of your own AS in the IPv6 advertisements to 
Cogent also reduces traffic from other multi-homed Cogent IPv6 customers.  In 
any event, if enough Cogent customers do this, it does greatly reduce the 
amount of traffic that Cogent gets to transit from their various IPv6 peers 
while still avoiding harm to innocent end-users (or for that matter, to guilty 
end users).

The mechanism I proposed has numerous benefits:

1.  It utilizes only a mechanism created by Cogent and documented for use by 
Cogent transit customers to achieve routing policy that benefits IPv6 customers 
of Cogent.
2.  It causes no harm to single-homed Cogent customers.
3.  It causes no direct harm to Cogent.  The sole indirect harm that it might 
bring upon Cogent if adopted en-masse would be to significantly drop the amount 
of traffic crossing Cogent’s SFI peerings on IPv6, which I acknowledge may have 
consequences for Cogent.  If it did have such consequences, it’s Cogent’s game 
and Cogent’s rules.  They could change it any time.  If it indirectly harms 
Cogent by lowering overall traffic volume on paid multi-homed customer transit 
connections, Cogent could easily remedy that by becoming an IPv6 network that 
one would want to exchange IPv6 transit traffic with rather than being an IPv6 
network that one begrudgingly pays because one does business with others who 
are Cogent single-homed.

I do reiterate, however, that I would strongly discourage any kind of routing 
tricks that leave the innocent Cogent customers out in the cold.  That hurts 
those Cogent customers as well as you and/or your own customers and users.  
Please, someone, think of the end-users here.

My advice to Cogent would be to remember something real simple:  When Big Boss 
#1 at RandomCorp has no trouble reaching Google services all night every night 
at home and then he comes to work and his office Internet does everything but 
Google….  What he’ll remember is “Charter works with Google, whoever we’re 
using at the office doesn’t.  Let’s switch.”  It’s shocking to me that an ISP 
with a retail segment thinks you can survive if Google doesn’t work, no matter 
what Google did to ensure it played out that way.  Frankly, Google could scream 
that they cut Cogent off because they didn’t like Cogent’s face and no one at 
retail would care.  They just want their Gmail back.  If Google wanted to force 
the issue faster, they could just stop the IPv4 transit routes to Cogent.  I 
think they’re taking a more balanced and conservative approach though.

Thanks,

Matt Hardeman

> On Mar 10, 2016, at 4:29 PM, Mark Andrews  wrote:
> 
> 
> I don't think anyone should be colluding to hurt Cogent or anyone
> else for that matter and this thread appears to be heading in this
> direction.
> 
> Mark
> -- 
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742 INTERNET: ma...@isc.org



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent - Google - HE Fun

[Matthew D. Hardeman]

I have contemplated whether such mechanisms matter to Cogent, etc.

I’m inclined to think that if Google is willing to pull the routes and they 
still don’t blink, then certainly us smaller shops aren’t going to impact them…

However…  If enough prefixes disappear from the _apparent_ Cogent table as 
viewed by outsiders, this may ultimately impact their sales of new 
interconnection….

For those of us multihomed with Cogent and other transit providers on IPv6 
there is a less drastic way to impact the perceived value of Cogent’s IPv6 
routing table to outsiders and especially to Cogent’s peers — and one that 
still doesn’t negatively impact the single-home customers of Cogent:

"set community 174:3000" on your IPv6 advertisement to Cogent.  This will 
constrain the advertisement to Cogent and Cogent’s customers only.  For good 
measure, prepend your own AS to this advertisement at least a couple of times, 
potentially discouraging even Cogent customers who see the route from using it 
if they have other transit.  It will prevent the path via Cogent being selected 
by Cogent IPv6 peers versus your other transit providers.


> On Mar 10, 2016, at 3:47 PM, Fredy Kuenzler  wrote:
> 
> Am 10.03.2016 um 22:25 schrieb Damien Burke :
>> Anyone who is multihomed with cogent ipv6 in their mix should shutdown their 
>> IPv6 bgp session. Let’s see if we can make their graph freefall.
> 
> 
> Alternative:
> 
> set community [do not announce to Cogent]
> 
> *SCNR*



smime.p7s
Description: S/MIME cryptographic signature

Re: Cogent & Google IPv6

[Matthew D. Hardeman]

What’s truly amazing to me about this is that only Cogent seems to be engaging 
in this kind of behavior on IPv6.  Furthermore, the only people Cogent is 
hurting with their willful ignorance of the changing peering landscape in IPv6 
is THEIR OWN PAYING CUSTOMERS.  Which is really bizarre when you think about 
it.  I’m trying to understand this from Cogent’s perspective and failing.  They 
are creating a problem that impacts only their customers while others do not 
create this same problem.  How can they imagine this is benefiting them?


> On Feb 24, 2016, at 1:53 PM, Max Tulyev  wrote:
> 
> If you connected to Internet ONLY through Cogent - there is no other
> way. If you have another upstreams - Google should be reachable.
> 
> On 24.02.16 21:46, Matt Hoppes wrote:
>> Correct me if I'm wrong, but if Cogent isn't peering with Google IPv6,
>> shouldn't the traffic flow out to one of their peer points where another
>> peer DOES peer with Google IPv6 and get you in?
>> 
>> Isn't that how the Internet is suppose to work?
>> 
>> 
>> On 2/24/16 2:43 PM, Damien Burke wrote:
>>> Not sure. I got the same thing today as well.
>>> 
>>> Is this some kind of ipv6 war?
>>> 
>>> -Original Message-
>>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ian Clark
>>> Sent: Wednesday, February 24, 2016 10:25 AM
>>> To: NANOG
>>> Subject: Cogent & Google IPv6
>>> 
>>> Anyone know what's actually going on here?  We received the following
>>> information from the two of them, and this just started a week or so ago.
>>> 
>>> 
>>> *From Cogent, the transit provider for a branch office of ours:*
>>> 
>>> Dear Cogent Customer,
>>> 
>>> Thank you for contacting Cogent Customer Support for information about
>>> the Google IPv6 addresses you are unable to reach.
>>> 
>>> Google uses transit providers to announce their IPv4 routes to Cogent.
>>> 
>>> At this time however, Google has chosen not to announce their IPv6
>>> routes to Cogent through transit providers.
>>> 
>>> We apologize for any inconvenience this may cause you and will notify
>>> you if there is an update to the situation.
>>> 
>>> 
>>> 
>>> *From Google (re: Cogent):*
>>> 
>>> Unfortunately it seems that your transit provider does not have IPv6
>>> connectivity with Google. We suggest you ask your transit provider to
>>> look for alternatives to interconnect with us.
>>> 
>>> Google maintains an open interconnect policy for IPv6 and welcomes any
>>> network to peer with us for access via IPv6 (and IPv4). For those
>>> networks that aren't able, or chose not to peer with Google via IPv6,
>>> they are able to reach us through any of a large number of transit
>>> providers.
>>> 
>>> For more information in how to peer directly with Google please visit
>>> https://peering.google.com
>>> 
>>> 
>>> -- 
>>> Ian Clark
>>> Lead Network Engineer
>>> DreamHost
>>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Dear Windstream engineers

[Matthew D. Hardeman]

You offer this service to your customers, don’t you?   ;-)

Seriously, it’s a good question.  Most IP transit providers offering BGP 
services do offer RTBH.

> On Jan 29, 2016, at 10:51 PM, George Skorup  wrote:
> 
> Why doesn't Windstream have RTBH for their BGP customers? It cannot be 
> impossible to implement.



smime.p7s
Description: S/MIME cryptographic signature

Re: The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

While I agree it’s still going to be a while before it becomes a critical 
issue, more and more environments are going IPv6 first with IPv4 as a NAT’ed 
service…

I think the mobile carriers are going to be the ones to really push adoption.

> On Jan 22, 2016, at 7:53 PM, Constantine A. Murenin <muren...@gmail.com> 
> wrote:
> 
> On 21 January 2016 at 19:42, Matthew D. Hardeman <mharde...@ipifony.com> 
> wrote:
>> An excellent point.  Nobody would tolerate this in IPv4 land.  Those 
>> disputes tended to end in days and weeks (sometimes months), but not years.
>> 
>> That said, as IPv6 is finally gaining traction, I suspect we’ll be seeing 
>> less tolerance for this behavior.
> 
> Nope.  Most user-facing apps are in support of Happy Eyeballs.
> 
> When Facebook's FB.ME was down on IPv6 just a short while ago in 2013,
> it took DAYS for anyone to notice.
> 
>  http://puck.nether.net/pipermail/outages/2013-May/005571.html
> 
> Lots of popular sites publish  with non-reachable services all the
> time, and still noone notices to this day.
> 
> The old school command line tools are the only ones affected.  One may
> also notice it with `ssh -D` SOCKS5 proxying, but only if one's
> browser doesn't decide to leak out hostname resolution and operate
> directly with IPv4-addresses to start with, like Chrome does.
> 
> Cheers,
> Constantine.SU.

Re: The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

Bill,

I find that I agree with much of what you’ve said.

If we further constrain the arguments that you set forth so as to cover only 
that traffic which the customers of the two networks would be able to exchange 
in any event, by way of transit services purchased by one or the other of the 
two networks, then I agree wholeheartedly, at least on a purely logical basis.

In that instance, the traffic is exchanged regardless (though often over links 
that saturate at peaks) and furthermore at additional expense to one or both of 
the networks involved.

From a logical perspective, if two networks will permit their subscribers to 
exchange data, why would those two networks not elect the least cost, highest 
quality mechanism for exchanging that traffic?

I can only think of economic reasons, and specifically the hope for potential 
revenue from the other networks’ customer, because the parties have been unable 
to exchange data reliably over congested transit links.

Look, for example, to what was quite obviously the intentional peak-period 
congestion on various Comcast transit and peering links.

I’ve personally acted in a technical and administrative capacity in helping 
clients of mine (voice service providers) add private paid peering / paid 
customer links into Comcast just to overcome voice quality issues during peak 
periods resulting from clearly congested transit and peering links.  It was 
obvious during those arrangements that Comcast had chosen to allow those links 
to congest as a policy matter in order to extract additional revenue by 
charging desperate “new customers” a premium toll for access to their 
subscribers behind the wall-of-congestion.

What’s fundamentally different in this IPv6 only Hurricane Electric <-> Cogent 
matter is that rather than have the traffic flow via transit (whether congested 
or not), there is quite simply no path between those two IPv6 networks.  
Hurricane Electric, clearly the IPv6 leader refuses to engage in the purchase 
of transit services for IPv6, and Cogent refuses to peer with HE on either 
protocol no matter what.  Thus, no flow of traffic between the two networks on 
IPv6.

Presumably Cogent’s policy is mostly about denying Hurricane Electric to the 
“Tier 1” club, on IPv6 that ship has sailed.  Let’s face it: when the really 
tough Tier 1s are peering with you (like Sprint, Level 3, AT), you’re in.  
Even Sprint peers with HE on IPv6 (though they do not on IPv4).  Honestly, I 
think Cogent is the only hold-out.  At least the only one that matters.

In as far as HE maintains an open peering policy both for IPv4 and IPv6, it’s 
clear that Cogent is the bad actor, denying their customers a path to Hurricane 
Electric customers.  I think the only reason this has been tolerated so far is 
that IPv6 has been a fringe matter until now.  Even today it’s a minority of 
network traffic, but it’s gaining fast.

If I were Cogent, I’d be more worried about denying my customers access to HE’s 
IPv6 network than the other way around.

Matt Hardeman


> On Jan 22, 2016, at 7:03 PM, William Herrin  wrote:
> 
> On Thu, Jan 21, 2016 at 1:52 PM, Brandon Butterworth
>  wrote:
>> I'd like to peer with all tier 1's, they are thus all bad as
>> they won't.
> 
> Correct.
> 
> I've said it before and I'll say it again: an ISP's refusal to
> maintain a settlement-free open peering policy is directly linked with
> said company's fraudulent double-billing for services.
> 
> In case you don't see it, I'll explain: whatever fictions you may tell
> yourselves, your customers pay you to connect them to the entire
> Internet. So do the other guy's customers. Settlement free peering
> means that at no _additional_ charge to anyone, you accept the packets
> your customers have paid you to accept from the other guy's customers.
> And vice versa. Peering does not trade packets you haven't been paid
> for. That's another fiction. Peering only trades packets one of your
> customers has paid you for.
> 
> I get from there to double-billing because the alternative to
> settlement free peering is a paid relationship. The other guy has to
> buy from you directly (becoming the second payer for each packet) or
> he has to buy from one of the peers you've accepted But the peers
> you've accepted are constrained by ratios an related technical
> requirements which functionally prevent them from adding a sizable
> amount of traffic from that other guy, so unless he's doing a trifling
> business he pretty much has to buy service from you. Even though
> another customer has already paid you to perform that activity, you
> refuse to do the job unless the second party also becomes your
> customer and pays you. Fraud. Hidden behind a wall of technical
> minutiae but fraud all the same.
> 
> 
> Don't get me wrong. You can cure this fraud without going to extremes.
> An open peering policy doesn't require you to buy hardware for the
> other guy's convenience. Let 

The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

Hi everyone,

I know the long and storied history of Cogent and HE failing to peer for IPv6 
and failing to provide (from either side) for IPv6 transit between their two 
networks has been mentioned and covered on this list before, but I am rather 
surprised it has not garnered much attention.

Until recently, that is.  I notice an increasing number of people tweeting at 
both HE and Cogent about the problem.

From HE’s public statements on the matter, it’s pretty clear that they would 
gladly peer with Cogent for IPv6 but that Cogent declines to do this.  I simply 
cannot understand Cogent’s logic on this.  Cogent is the one loosing out here, 
to my way of thinking.  They have far less IPv6 coverage than HE.

I myself, on behalf of my employers, am a direct customer of IP transit 
services from both Cogent and HE.

I don’t know about others similarly positioned, but my Cogent rep tries to call 
me at least twice a month.  I’m going to start taking (more of) his calls and 
letting him know his account with us is in jeopardy come renewal time if Cogent 
can’t get a full IPv6 route table to happen.

Today, with Cogent & HE as peers, I am world reachable via IPv6.  If either 
peer went down however, part of the internet couldn’t reach me via IPv6 because 
either HE wouldn’t have a route or Cogent wouldn’t have a route.  That’s 
ridiculous.

Since Cogent is clearly the bad actor here (the burden being Cogent's to prove 
otherwise because HE is publicly on record as saying that they’d love to peer 
with Cogent), I’m giving serious consideration to dropping Cogent come renewal 
time and utilizing NTT or Zayo instead.

While that would not immediately solve the problem that if the NTT or Zayo link 
went down, single-homed Cogent customers would loose access to me via IPv6, I’m 
actually ok with that.  It at least lets ensures that when there is a problem, 
the problem affects only single-home Cogent clients.  Thus, the problem is 
borne exclusively by the people who pay the bad actor who is causing this 
problem.  That tends to get uncomfortable for the payee (i.e. Cogent).

I intend to email my Cogent sales guy regarding this matter and make this a 
sticking point in every phone conversation I have with him.  I call on others 
similarly situated to consider whether you may like to follow suit in this 
approach.  I’ve come to believe that it’s best for my interests and I also 
believe that it’s best for the internet community at large, as ubiquitous 
worldwide routing of IPv6 becomes more essential with each passing day.

In closing, I further add that it’s a mystery to me why Cogent wouldn’t desire 
an IPv6 peering with HE.  Let’s face it, if any of us had to choose a 
single-home IPv6 internet experience, between HE or Cogent, we’d all choose HE. 
 If those were the two options, HE is the “real” IPv6 internet and Cogent is a 
tiny sliver of the IPv6 internet.  I have actually wondered if HE is holding 
IPv6 peering with Cogent hostage, contingent on peering all protocols (IPv4 and 
IPv6) with Cogent.  There, I could see why Cogent might hesitate.  To my 
knowledge, however, this is not the case and I have heard no public accusation 
that HE is imposing such a constraint.  I would love to hear anyone from HE 
tell as much of the story as they are able.

PS - As an aside, has anyone noticed HE’s been growing their network by leaps 
and bounds this past year?  Direct peerings with AT and CenturyLink, more 
domestic US and Canadian POPs, and I believe the number of pathways across the 
North American continent has been improved substantially, too.

Thanks,

Matt Hardeman
IPiFony Systems, Inc.
AS6082

Re: The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

I’m inclined to agree with you, subject to some caveats:

1.  I think more Cogent customers need to be more vocal about it.  There hasn’t 
been an impetus to do so until recently.  Now real people (not network engineer 
sorts) are starting to use IPv6 for real.

2.  I agree with you in principle.  In an idea world, take HE and two others.  
I would however still say that if you could only take two, take HE and take 
something other than Cogent.  It’s a win-win if the experience of single-home 
Cogent customers gets to be worse as a result.  Perhaps having things 
occasionally break — only for single-home Cogent customers — is a benefit.


> On Jan 21, 2016, at 12:40 PM, Daniel Corbe <dco...@hammerfiber.com> wrote:
> 
> 
>> On Jan 21, 2016, at 1:07 PM, Matthew D. Hardeman <mharde...@ipifony.com> 
>> wrote:
>> 
>> Since Cogent is clearly the bad actor here (the burden being Cogent's to 
>> prove otherwise because HE is publicly on record as saying that they’d love 
>> to peer with Cogent), I’m giving serious consideration to dropping Cogent 
>> come renewal time and utilizing NTT or Zayo instead.
>> 
>> While that would not immediately solve the problem that if the NTT or Zayo 
>> link went down, single-homed Cogent customers would loose access to me via 
>> IPv6, I’m actually ok with that.  It at least lets ensures that when there 
>> is a problem, the problem affects only single-home Cogent clients.  Thus, 
>> the problem is borne exclusively by the people who pay the bad actor who is 
>> causing this problem.  That tends to get uncomfortable for the payee (i.e. 
>> Cogent).
>> 
>> 
> 
> Take two transit providers that aren’t in the group of (HE, Cogent).  Cogent 
> is probably banking on this being the response; figuring that they have the 
> financial resources to outlast HE if they’re both shedding customers.  
> 
> If you really wanted to stick it to Cogent, take 3 transit providers: HE and 
> two of any other providers besides Cogent.  
> 
> Cogent clearly aren’t going to cave to their own customers asking them to 
> peer with HE.  Otherwise it would have happened by now.  
> 
> Cogent sucks for lots of reasons and this one isn’t even in the top 5 IMHO.
> 
> 

New peerings between Hurricane Electric and Level3?

[Matthew D. Hardeman]

Yesterday I was looking at some of the IPv4 and IPv6 session summaries on 
http://lg.he.net and saw that both the Equinix Los Angeles and Equinix Ashburn 
site routers have new IPv4 and IPv6 sessions (not yet running, but 
administratively up for about 6 days now) configured for AS3356.

I know they already peer IPv6, though not at those sites.  Is this the first 
hint that HE and Level3 are coming around on an IPv4 and IPv6 peering agreement?

Re: The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

An excellent point.  Nobody would tolerate this in IPv4 land.  Those disputes 
tended to end in days and weeks (sometimes months), but not years.

That said, as IPv6 is finally gaining traction, I suspect we’ll be seeing less 
tolerance for this behavior.


> On Jan 21, 2016, at 8:30 PM, Matthew Kaufman <matt...@matthew.at> wrote:
> 
> 
> 
>> On Jan 21, 2016, at 1:05 PM, Ca By <cb.li...@gmail.com> wrote:
>> 
>> On Thu, Jan 21, 2016 at 10:52 AM, Brandon Butterworth <bran...@rd.bbc.co.uk>
>> wrote:
>> 
>>>>> On Jan 21, 2016, at 1:07 PM, Matthew D. Hardeman <
>>> mharde...@ipifony.com> wrote:
>>>>> Since Cogent is clearly the bad actor here (the burden being
>>>>> Cogent's to prove otherwise because HE is publicly on record as saying
>>>>> that theyd love to peer with Cogent)
>>> 
>>> I'd like to peer with all tier 1's, they are thus all bad as
>>> they won't.
>>> 
>>> HE decided they want to be transit free for v6 and set out on
>>> a campaign of providing free tunnels/transit/peering to establish
>>> this. Cogent, for all their faults, are free to not accept the
>>> offer.
>>> 
>>> Can the Cogent bashing stop now, save it for when they do something
>>> properly bad.
>>> 
>>> brandon
>> 
>> Selling a service that is considered internet but does not deliver full
>> internet access is generally considered properly bad.
>> 
>> I would not do business with either company, since neither of them provide
>> a full view.
>> 
>> CB
> 
> I note that if IPv6 was actually important, neither one could have gotten 
> away with it for so long.
> 
> Matthew Kaufman
> 
> (Sent from my iPhone)



smime.p7s
Description: S/MIME cryptographic signature

Re: The IPv6 Travesty that is Cogent's refusal to peer Hurricane Electric - and how to solve it

[Matthew D. Hardeman]

I hear you.

Taken to extremes, I can see how the argument sounds like that.

However…  I have some thoughts on what you’ve said.

Most of us would never get peerings to all the Tier 1s.

But…

Hurricane Electric already has IPv6 peering to every network that matters, save 
for Cogent’s.  Every other accepted Tier 1 peers with HE on IPv6.  Even SPRINT. 
 If we got back historically, they (Sprint) were among the most coveted and 
hardest to get IP peerings.  Even they recognized HE’s dominance of the IPv6 
space early on.

I’m not bashing Cogent.  I’m a customer of theirs and they’ve generally served 
me well.  The trouble I have in accepting Cogent’s behavior in this matter is 
that it just seems irrational.  If a typical, public forum peering dispute 
arose between HE & Cogent regarding IPv6, frankly and pretty objectively, you’d 
expect it to be Hurricane Electric questioning the value of peering Cogent IPv6 
rather than Cogent questioning HE.

I don’t question these parties’ rights not to peer, but I do question the logic 
behind it.  I think Cogent is hurting themselves on this more than HE is 
getting hurt by it.


> On Jan 21, 2016, at 12:52 PM, Brandon Butterworth <bran...@rd.bbc.co.uk> 
> wrote:
> 
>>> On Jan 21, 2016, at 1:07 PM, Matthew D. Hardeman <mharde...@ipifony.com> 
>>> wrote:
>>> Since Cogent is clearly the bad actor here (the burden being
>>> Cogent's to prove otherwise because HE is publicly on record as saying
>>> that theyd love to peer with Cogent)
> 
> I'd like to peer with all tier 1's, they are thus all bad as
> they won't.
> 
> HE decided they want to be transit free for v6 and set out on
> a campaign of providing free tunnels/transit/peering to establish
> this. Cogent, for all their faults, are free to not accept the
> offer.
> 
> Can the Cogent bashing stop now, save it for when they do something
> properly bad.
> 
> brandon

Re: New peerings between Hurricane Electric and Level3?

[Matthew D. Hardeman]

Intriguing.  If it were only that though, wouldn’t they just still pick it up 
via TeliaSonera IC?

I did notice that in the past few months, TeliaSonera has been dropping AS3549 
from spots where they had session with both AS3549 and with AS3356 and now 
reaches AS3549 via AS3356.


> On Jan 21, 2016, at 1:08 PM, Marty Strong <ma...@cloudflare.com> wrote:
> 
> I’ve heard from the grape vine that this is due to the GBLX to Level3 
> transition, and it’s in fact paid IP transit.
> 
> Regards,
> Marty Strong
> --
> CloudFlare - AS13335
> Network Engineer
> ma...@cloudflare.com
> +44 7584 906 055
> smartflare (Skype)
> 
> http://www.peeringdb.com/view.php?asn=13335
> 
>> On 21 Jan 2016, at 18:37, Matthew D. Hardeman <mharde...@ipifony.com> wrote:
>> 
>> Yesterday I was looking at some of the IPv4 and IPv6 session summaries on 
>> http://lg.he.net and saw that both the Equinix Los Angeles and Equinix 
>> Ashburn site routers have new IPv4 and IPv6 sessions (not yet running, but 
>> administratively up for about 6 days now) configured for AS3356.
>> 
>> I know they already peer IPv6, though not at those sites.  Is this the first 
>> hint that HE and Level3 are coming around on an IPv4 and IPv6 peering 
>> agreement?
> 

Re: New peerings between Hurricane Electric and Level3?

[Matthew D. Hardeman]

That’s an excellent point, actually.

> On Jan 21, 2016, at 1:45 PM, Patrick W. Gilmore <patr...@ianai.net> wrote:
> 
> Make the AS path longer, losing traffic, and therefore revenue?
> 
> Why would they do that?
> 
> The twtelecom customers cannot multi-home (most of them anyway). Most of 
> 3549’s traffic has other paths to the Internet.
> 
> -- 
> TTFN,
> patrick
> 
>> On Jan 21, 2016, at 2:22 PM, Matthew D. Hardeman <mharde...@ipifony.com> 
>> wrote:
>> 
>> I was actually surprised they didn’t just leave GBLX customers on AS3549, 
>> kill all external AS3549 peerings, and treat AS3549 downline as a Level3 
>> customer, accepting L3 and GBLX communities from GBLX customers.
>> 
>> That seems more along the lines of what they’re doing with the AS4323 TW 
>> Telecom customers.  (Though, in fairness, AS3356 has always carried AS4323 
>> as a customer as far as I recall.)  It will be interesting to see if whether 
>> they kill off AS4323 peerings.
>> 
>>> On Jan 21, 2016, at 1:13 PM, Marty Strong <ma...@cloudflare.com> wrote:
>>> 
>>> Depends on the market and how far along their migration is going. In 
>>> experience with GTT (AS4436) they’re still not finished migrating 
>>> everything to AS3257.
>>> 
>>> Regards,
>>> Marty Strong
>>> --
>>> CloudFlare - AS13335
>>> Network Engineer
>>> ma...@cloudflare.com
>>> +44 7584 906 055
>>> smartflare (Skype)
>>> 
>>> http://www.peeringdb.com/view.php?asn=13335
>>> 
>>>> On 21 Jan 2016, at 19:12, Matthew D. Hardeman <mharde...@ipifony.com> 
>>>> wrote:
>>>> 
>>>> Intriguing.  If it were only that though, wouldn’t they just still pick it 
>>>> up via TeliaSonera IC?
>>>> 
>>>> I did notice that in the past few months, TeliaSonera has been dropping 
>>>> AS3549 from spots where they had session with both AS3549 and with AS3356 
>>>> and now reaches AS3549 via AS3356.
>>>> 
>>>> 
>>>>> On Jan 21, 2016, at 1:08 PM, Marty Strong <ma...@cloudflare.com> wrote:
>>>>> 
>>>>> I’ve heard from the grape vine that this is due to the GBLX to Level3 
>>>>> transition, and it’s in fact paid IP transit.
>>>>> 
>>>>> Regards,
>>>>> Marty Strong
>>>>> --
>>>>> CloudFlare - AS13335
>>>>> Network Engineer
>>>>> ma...@cloudflare.com
>>>>> +44 7584 906 055
>>>>> smartflare (Skype)
>>>>> 
>>>>> http://www.peeringdb.com/view.php?asn=13335
>>>>> 
>>>>>> On 21 Jan 2016, at 18:37, Matthew D. Hardeman <mharde...@ipifony.com> 
>>>>>> wrote:
>>>>>> 
>>>>>> Yesterday I was looking at some of the IPv4 and IPv6 session summaries 
>>>>>> on http://lg.he.net and saw that both the Equinix Los Angeles and 
>>>>>> Equinix Ashburn site routers have new IPv4 and IPv6 sessions (not yet 
>>>>>> running, but administratively up for about 6 days now) configured for 
>>>>>> AS3356.
>>>>>> 
>>>>>> I know they already peer IPv6, though not at those sites.  Is this the 
>>>>>> first hint that HE and Level3 are coming around on an IPv4 and IPv6 
>>>>>> peering agreement?
>>>>> 
>>>> 
>>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Best Source for ARIN Region /24

[Matthew D. Hardeman]

I’m aware of the /24 block for facilitation concept, but my client’s use case 
can qualify as an end-user rather than as an ISP, thus their annual operating 
cost is smaller than even the X-SMALL ISP category, which they’d land in — if 
they opted for the smaller /36 initial IPv6 direct allocation, rather than the 
default /32 direct allocation.

That seems to balance toward buying an existing /24.

> On Jan 11, 2016, at 8:00 PM, Rafael Possamai <rafaelpo...@gmail.com> wrote:
> 
> If you apply for an IPv6 block, as an ISP, and you have the intention of 
> truly utilizing it, then you can apply for a /24 to facilitate that 
> transition. 
> 
> It will cost you about $1500 or so, which is about half of what a /24 is 
> going for in the transfer market.
> 
> Thing is, if you take the IPv6 block just to use the /24 they give you, then 
> one could argue you are cheating the system.
> 
> 
> 
> On Mon, Jan 11, 2016 at 1:19 PM, Matthew D. Hardeman <mharde...@ipifony.com 
> <mailto:mharde...@ipifony.com>> wrote:
> I’m looking to buy a /24 of space for a new multi-homed network in the ARIN 
> region.  Can anyone out there speak to going rates for a /24 and best places 
> to shop?
> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Best Source for ARIN Region /24

[Matthew D. Hardeman]

I’m looking to buy a /24 of space for a new multi-homed network in the ARIN 
region.  Can anyone out there speak to going rates for a /24 and best places to 
shop?



smime.p7s
Description: S/MIME cryptographic signature

Re: Best Source for ARIN Region /24

[Matthew D. Hardeman]

So far, some of the off-list responses that I’ve seen from my inquiry are 
beating out the pricing that shows on Hilco Streambank’s site.


> On Jan 11, 2016, at 2:01 PM, Christopher Dye <chris@paragon.net> wrote:
> 
> I just paid way too much from Hilco Streambank on Auction. I think I ended up 
> spending about $2500 + ARIN fees (but I really needed it). 
> www.ipv4auctions.com
> 
> Christopher Dye
> Chief Technology Officer
> Paragon Solutions Group, Inc. 
> 
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ray Orsini
> Sent: Monday, January 11, 2016 1:22 PM
> To: Matthew D. Hardeman <mharde...@ipifony.com>; nanog@nanog.org
> Subject: RE: Best Source for ARIN Region /24
> 
> Ditto here. Seems like Matthew beat me to the question
> 
> Regards,
> Ray Orsini – CEO
> Orsini IT, LLC – Technology Consultants
> VOICE DATA  BANDWIDTH  SECURITY  SUPPORT
> P: 305.967.6756 x1009   E: r...@orsiniit.com   TF: 844.OIT.VOIP
> 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016 
> http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices | View 
> Your Tickets
> 
> 
> 
> 
> -----Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Matthew D.
> Hardeman
> Sent: Monday, January 11, 2016 2:19 PM
> To: nanog@nanog.org
> Subject: Best Source for ARIN Region /24
> 
> I’m looking to buy a /24 of space for a new multi-homed network in the ARIN 
> region.  Can anyone out there speak to going rates for a /24 and best places 
> to shop?



smime.p7s
Description: S/MIME cryptographic signature

Re: Ashburn

[Matthew D. Hardeman]

On that topic...

Has anyone noticed at 56 Marietta (in the main entrance lobby which is 
currently being remodeled), ground floor by the elevators is an ominous 
"Meet-Me Room #3" behind a closed door?  A door that has an L-3 Communications 
biometric access reader.  Which is different from all the other HID readers 
facility wide.  One day I noticed the hum of heavy equipment through that door. 
 Most suspect for a "meet-me room" where most of the time it's just passive 
fiber-optic patches.  It's also suspect to have a Meet-Me room on a floor that 
has no customers, but, whatever...

Makes you wonder.

- Original Message -
From: "Keith Stokes" 
To: "Christopher Morrow" , "Matt Hoppes" 

Cc: "North American Network Operators' Group" 
Sent: Wednesday, September 16, 2015 10:39:56 AM
Subject: Re: Ashburn

Or router bugs.

Or even inserting new NSA taps since some of the rest have been caught.

---

Keith Stokes


From: NANOG  on behalf of Christopher Morrow 

Sent: Wednesday, September 16, 2015 10:34 AM
To: Matt Hoppes
Cc: North American Network Operators' Group
Subject: Re: Ashburn

removal of nsa taps

On Wed, Sep 16, 2015 at 10:34 AM, Matt Hoppes
 wrote:
> What the world is going on in Ashburn?  Over the last two days I've seen
> multiple flaps from multiple carriers going through there.  They generally
> last about two to three minutes and then everything restores.