Frontier fiber outage in Tampa Bay?

[David Hubbard]

Curious if anyone else is having issues with Frontier fiber service in Tampa 
Bay region?  I've got a couple locations down.  Their support could not 
possibly be worse, so figured I'd start asking around while I consider giving 
up lol

Thanks

Arelion/Telia AS1299 issues?

[David Hubbard]

Hey all, anyone aware of issues with Arelion this morning?  We have a bunch of 
end users on at least Cox and Cogeco who are having serious issues with service 
access, and the problem appears to be on the return path where it traverses 
Arelion.  Source net is Lumen/L3 3356 but loss/latency doesn’t appear to creep 
up until already within Arelion’s network.

Imperva mentioned outage/degradation due to a national ISP issue too, so I 
suspect it’s affecting more than just certain peerings.

Thanks,

David

Re: Lossy cogent p2p experiences?

[David Hubbard]

Some interesting new developments on this, independent of the divergent network 
equipment discussion. 

Cogent had a field engineer at the east coast location where my local loop 
(10gig wave) meets their equipment, i.e. (me – patch cable to loop provider’s 
wave equipment – wave – patch cable to Cogent equipment).  On the other end, 
the geographically distant west coast direction, it’s Cogent equipment to my 
equipment in the same facility with just patch cable.  They connected some 
model of EXFO’s NetBlazer FTBx 8880-series testing device to a port on their 
east coast network device, not disconnecting my circuit.  Originally, they were 
planning to have someone physically loop at their equipment at the other end, 
but I volunteered that my Arista gear supports a provider-facing loop at the 
transceiver level if they wanted to try that, so my loop, cabling, and 
transceiver could be part of the testing.

One direction at a time, they interrupted the point to point config to create a 
point to point between one direction of my gear, set to loopback mode, and the 
NetBlazer device.  The device was set to use five parallel streams.  In the 
close direction, where the third-party wave is involved, they ran at full 5 x 
2gbps for thirty minutes, had zero packets lost, no issues.  My monitoring 
confirmed this rate of port input was occurring, although oddly not output, but 
perhaps Arista doesn’t “see”/count the retransmitted packets in phy loopback 
mode.

In the distant direction across their backbone, their equipment at the remote 
end, and the fiber patch cable to me, they tested at 9.5 Gbit for thirty 
minutes through my device in loopback mode.  The result was, of 2.6B packets 
sent, only 334 packets lost.  They configured for 9.5 gbps rate of testing, so 
five 1.9gbps streams.  Across the five streams, the report has a “frame loss” 
and out of sequence section.  Zero out of sequence, but among the five streams, 
loss seconds / count were 3 / 26, 3 / 48, 1 / 5, 13 / 221, 1 / 34.  I’m not 
familiar with this testing device, but to me that suggests it’s stating how 
many of the total seconds experienced loss, and the counted packet loss.  So 
really the only one that stands out is the one with thirteen seconds where loss 
occurred, but the packet counts we’re talking about are miniscule.  Again, my 
monitoring at the interface level showed this 9.5gbps of testing occurring for 
the thirty minutes the report says.

So, now I’m just completely confused.  How is this device, traversing the same 
equipment, ports, cables, able to achieve far greater average throughput, and 
almost no loss, across a very long duration?  There are times I’ll be able to 
achieve nearly the same, but never for a test longer than ten seconds as it 
just falls off from there.  For example, I did a five parallel stream TCP test 
with iperf just now and did achieve a net throughput of 8.16 Gbps with about 
1200 retransmits.  Same five stream test run for half hour like theirs, I got 
no better than 2.64 Gbps and 183,000 retransmits.

iperf and UDP allow me to see loss at any rate of transmit exceeding ~140mbps, 
in just seconds, not a half hour.  To rule out my gear, I’m also able to 
perform the same tests from the same systems (both VM and physical) using 
public addresses and traversing the internet, as these are publicly connected 
systems.  I get far lower loss and much greater throughput on the internet 
path.  For example, simple ten second test of a single stream at 400 Mbit UDP; 
5 packets lost across internet, 491 across P2P.  Single stream TCP across the 
internet for ten seconds; 3.47 Gbps, 162 retransmits.  Across the P2P, this 
time at least, 637 Mbps, 3633 retransmits.

David



From: David Hubbard 
Date: Friday, September 1, 2023 at 10:19 AM
To: Nanog@nanog.org 
Subject: Re: Lossy cogent p2p experiences?
The initial and recurring packet loss occurs on any flow of more than ~140 
Mbit.  The fact that it’s loss-free under that rate is what furthers my opinion 
it’s config-based somewhere, even though they say it isn’t.

From: NANOG  on behalf 
of Mark Tinka 
Date: Friday, September 1, 2023 at 10:13 AM
To: Mike Hammett , Saku Ytti 
Cc: nanog@nanog.org 
Subject: Re: Lossy cogent p2p experiences?

On 9/1/23 15:44, Mike Hammett wrote:
and I would say the OP wasn't even about elephant flows, just about a network 
that can't deliver anything acceptable.

Unless Cogent are not trying to accept (and by extension, may not be able to 
guarantee) large Ethernet flows because they can't balance them across their 
various core links, end-to-end...

Pure conjecture...

Mark.

Re: Lossy cogent p2p experiences?

[David Hubbard]

The initial and recurring packet loss occurs on any flow of more than ~140 
Mbit.  The fact that it’s loss-free under that rate is what furthers my opinion 
it’s config-based somewhere, even though they say it isn’t.

From: NANOG  on behalf 
of Mark Tinka 
Date: Friday, September 1, 2023 at 10:13 AM
To: Mike Hammett , Saku Ytti 
Cc: nanog@nanog.org 
Subject: Re: Lossy cogent p2p experiences?

On 9/1/23 15:44, Mike Hammett wrote:
and I would say the OP wasn't even about elephant flows, just about a network 
that can't deliver anything acceptable.

Unless Cogent are not trying to accept (and by extension, may not be able to 
guarantee) large Ethernet flows because they can't balance them across their 
various core links, end-to-end...

Pure conjecture...

Mark.

Re: Lossy cogent p2p experiences?

[David Hubbard]

That’s not what I’m trying to do, that’s just what I’m using during testing to 
demonstrate the loss to them.  It’s intended to bridge a number of networks 
with hundreds of flows, including inbound internet sources, but any new TCP 
flow is subject to numerous dropped packets at establishment and then ongoing 
loss every five to ten seconds.  The initial loss and ongoing bursts of loss 
cause the TCP window to shrink so much that any single flow, between systems 
that can’t be optimized, ends up varying from 50 Mbit/sec to something far 
short of a gigabit.  It was also fine for six months before this miserable 
behavior began in late June.


From: Eric Kuhnke 
Date: Thursday, August 31, 2023 at 4:51 PM
To: David Hubbard 
Cc: Nanog@nanog.org 
Subject: Re: Lossy cogent p2p experiences?
Cogent has asked many people NOT to purchase their ethernet private circuit 
point to point service unless they can guarantee that you won't move any single 
flow of greater than 2 Gbps. This works fine as long as the service is used 
mostly for mixed IP traffic like a bunch of randomly mixed customers together.

What you are trying to do is probably against the guidelines their engineering 
group has given them for what they can sell now.

This is a known weird limitation with Cogent's private circuit service.

The best working theory that several people I know in the neteng community have 
come up with is because Cogent does not want to adversely impact all other 
customers on their router in some sites, where the site's upstreams and links 
to neighboring POPs are implemented as something like 4 x 10 Gbps. In places 
where they have not upgraded that specific router to a full 100 Gbps upstream. 
Moving large flows >2Gbps could result in flat topping a traffic chart on just 
1 of those 10Gbps circuits.



On Thu, Aug 31, 2023 at 10:04 AM David Hubbard 
mailto:dhubb...@dino.hostasaurus.com>> wrote:
Hi all, curious if anyone who has used Cogent as a point to point provider has 
gone through packet loss issues with them and were able to successfully 
resolve?  I’ve got a non-rate-limited 10gig circuit between two geographic 
locations that have about 52ms of latency.  Mine is set up to support both 
jumbo frames and vlan tagging.  I do know Cogent packetizes these circuits, so 
they’re not like waves, and that the expected single session TCP performance 
may be limited to a few gbit/sec, but I should otherwise be able to fully 
utilize the circuit given enough flows.

Circuit went live earlier this year, had zero issues with it.  Testing with 
common tools like iperf would allow several gbit/sec of TCP traffic using 
single flows, even without an optimized TCP stack.  Using parallel flows or UDP 
we could easily get close to wire speed.  Starting about ten weeks ago we had a 
significant slowdown, to even complete failure, of bursty data replication 
tasks between equipment that was using this circuit.  Rounds of testing 
demonstrate that new flows often experience significant initial packet loss of 
several thousand packets, and will then have ongoing lesser packet loss every 
five to ten seconds after that.  There are times we can’t do better than 50 
Mbit/sec, but it’s rare to achieve gigabit most of the time unless we do a 
bunch of streams with a lot of tuning.  UDP we also see the loss, but can still 
push many gigabits through with one sender, or wire speed with several nodes.

For equipment which doesn’t use a tunable TCP stack, such as storage arrays or 
vmware, the retransmits completely ruin performance or may result in ongoing 
failure we can’t overcome.

Cogent support has been about as bad as you can get.  Everything is great, 
clean your fiber, iperf isn’t a good test, install a physical loop oh wait we 
don’t want that so go pull it back off, new updates come at three to seven day 
intervals, etc.  If the performance had never been good to begin with I’d have 
just attributed this to their circuits, but since it worked until late June, I 
know something has changed.  I’m hoping someone else has run into this and 
maybe knows of some hints I could give them to investigate.  To me it sounds 
like there’s a rate limiter / policer defined somewhere in the circuit, or an 
overloaded interface/device we’re forced to traverse, but they assure me this 
is not the case and claim to have destroyed and rebuilt the logical circuit.

Thanks!

Lossy cogent p2p experiences?

[David Hubbard]

Hi all, curious if anyone who has used Cogent as a point to point provider has 
gone through packet loss issues with them and were able to successfully 
resolve?  I’ve got a non-rate-limited 10gig circuit between two geographic 
locations that have about 52ms of latency.  Mine is set up to support both 
jumbo frames and vlan tagging.  I do know Cogent packetizes these circuits, so 
they’re not like waves, and that the expected single session TCP performance 
may be limited to a few gbit/sec, but I should otherwise be able to fully 
utilize the circuit given enough flows.

Circuit went live earlier this year, had zero issues with it.  Testing with 
common tools like iperf would allow several gbit/sec of TCP traffic using 
single flows, even without an optimized TCP stack.  Using parallel flows or UDP 
we could easily get close to wire speed.  Starting about ten weeks ago we had a 
significant slowdown, to even complete failure, of bursty data replication 
tasks between equipment that was using this circuit.  Rounds of testing 
demonstrate that new flows often experience significant initial packet loss of 
several thousand packets, and will then have ongoing lesser packet loss every 
five to ten seconds after that.  There are times we can’t do better than 50 
Mbit/sec, but it’s rare to achieve gigabit most of the time unless we do a 
bunch of streams with a lot of tuning.  UDP we also see the loss, but can still 
push many gigabits through with one sender, or wire speed with several nodes.

For equipment which doesn’t use a tunable TCP stack, such as storage arrays or 
vmware, the retransmits completely ruin performance or may result in ongoing 
failure we can’t overcome.

Cogent support has been about as bad as you can get.  Everything is great, 
clean your fiber, iperf isn’t a good test, install a physical loop oh wait we 
don’t want that so go pull it back off, new updates come at three to seven day 
intervals, etc.  If the performance had never been good to begin with I’d have 
just attributed this to their circuits, but since it worked until late June, I 
know something has changed.  I’m hoping someone else has run into this and 
maybe knows of some hints I could give them to investigate.  To me it sounds 
like there’s a rate limiter / policer defined somewhere in the circuit, or an 
overloaded interface/device we’re forced to traverse, but they assure me this 
is not the case and claim to have destroyed and rebuilt the logical circuit.

Thanks!

Re: Cogent Abuse - Bogus Propagation of ASN 36471

[David Hubbard]

Heck, I can’t even get Cogent to keep my paid services functional; going on 
four weeks with an unusable 10gig point to point.


From: NANOG  on behalf 
of Mike Hammett 
Date: Thursday, July 20, 2023 at 1:03 PM
To: Tom Beecher 
Cc: nanog@nanog.org 
Subject: Re: Cogent Abuse - Bogus Propagation of ASN 36471
If they (or anyone else) want to give me free service to use as I see fit 
(well, legally), I'll gladly accept their offer.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Tom Beecher" 
To: "Matthew Petach" 
Cc: nanog@nanog.org
Sent: Thursday, July 20, 2023 11:38:50 AM
Subject: Re: Cogent Abuse - Bogus Propagation of ASN 36471
In short--I'm having a hard time understanding how a non-paying entity still 
has working connectivity and BGP sessions, which makes me suspect there's a 
different side to this story we're not hearing yet.   ^_^;

I know Cogent has long offered very cheap transit prices, but this seems very 
aggressive! :)

On Thu, Jul 20, 2023 at 12:28 PM Matthew Petach 
mailto:mpet...@netflight.com>> wrote:


On Thu, Jul 20, 2023 at 8:09 AM Pete Rohrman 
mailto:prohr...@stage2networks.com>> wrote:

Ben,

Compromised as in a nefarious entity went into the router and changed passwords 
and did whatever.  Everything advertised by that comprised router is bogus.  
The compromised router is owned by OrgID: S2NL (now defunct).  AS 36471 belongs 
to KDSS-23.  
The compromised router does not belong to Kratos 
KDSS-23, and is 
causing routing problems.  The compromised router needs to be shut down.  The 
owner of the compromised router ceased business, and there isn't anyone around 
to address this at S2NL.  The only people that can resolve this is Cogent.   
Cogent's defunct customer's router was compromised, and is spewing out bogus 
advertisements.

Pete


Hi Pete,

This seems a bit confusing.

So, S2NL was a bill-paying customer of Cogent with a BGP speaking router.
They went out of business, and stopped paying their Cogent bills.
Cogent, out of the goodness of their hearts, continued to let a non-paying 
customer keep their connectivity up and active, and continued to freely import 
prefixes across BGP neighbors from this non-paying defunct customer.
Now, someone else has gained access to this non-paying, defunct customer's 
router (which Cogent is still providing free connectivity to, out of the 
goodness of their hearts), and is generating RPKI-valid announcements from it, 
which have somehow not caused a flurry of messages on the outages list about 
prefix hijackings.

The elements to your claim don't really seem to add up.
1) ISPs aren't famous for letting non-bill-paying customers stay connected for 
very long past the grace period on their billing cycle, let alone long after 
the company has gone belly-up.
2) It's not impossible to generate RPKI-valid announcements from a hijacked 
network, but it's very difficult to generate *bogus* RPKI-valid announcements 
from a compromised router--that's the whole point of RPKI, to be able to 
validate that the prefixes being announced from an origin are indeed the ones 
that are owned by that origin.

Can you provide specific prefix and AS_PATH combinations being originated by 
that router that are "bogus" and don't belong to the router's ASN?

If, however, what you meant is that the router used to be ASN X, and is now 
suddenly showing up as ASN 36471, and Cogent happily changed their BGP neighbor 
statements to match the new ASN, even though the entity no longer exists and 
hasn't been paying their bills for some time, then that would imply a level of 
complicity on Cogent's part that would make them unlikely to respond to your 
abuse reports.  That would be a very strong allegation to make, and the 
necessary level of documented proof of that level of malfeasance would be 
substantial.

In short--I'm having a hard time understanding how a non-paying entity still 
has working connectivity and BGP sessions, which makes me suspect there's a 
different side to this story we're not hearing yet.   ^_^;

Thanks!

Matt

Anyone from Akamai available?

[David Hubbard]

Trying to determine an outage issue between Akamai and 33398; any Akamai folks 
on here?

Thanks,

David

Any Frontier AS 5650 folks on here?

[David Hubbard]

Have spent 90 minutes with tech support trying to get a peering issue a few 
hundred miles away in front of the right department, and all I have to show for 
it is broken local equipment lol.

Thanks,

David

Cogent & Google reachability stable?

[David Hubbard]

It seemed like a decade in the making but has the IPv6 transit between Cogent 
and Google (via that showed up last fall remained stable?  I’d ruled them out 
on a number of projects for this reason but may reconsider if it has been 
reliable.  Appears HE (ASN6939) is still unreachable though…  I feel like less 
entities are single homed to HE, but it would still be a calculated risk.

Thanks,
David

Re: What's a "normal" ratio of web sites to IP addresses...

[David Hubbard]

I don't know that there is a normal as it likely depends heavily on the revenue 
per customer and the service's tolerance for giving out IP addresses.  It also 
depends heavily on the back end infrastructhre and what kind of service is 
being provided.  There's probably massive scale behind Cloudflare IP addresses. 
 There are middleware-style ecommerce and blog platforms where there is the 
same, i.e. lots of sites behind any given IP because every customer receives 
the same service from the same software; likely thousands or more per IP in 
that case.  As you get more custom, probably far less per IP as that's when 
sites tend to start being mapped to dedicated virtual machines / servers, 
shared hosting, etc. where it goes anywhere from a few hundred to one site on a 
dedicated server.

Sorry to go off on a tangent but this got me wanting to rant. __

Still, to this day, SEO "experts" continue to guide clients towards service 
platforms (hosting, ecommerce, blogs, etc.) where they know it remains possible 
to get an exclusive IP address because they are "sure" that will produce 
meaningful search positioning gains.  I started a thread on this topic on nanog 
about this back in what I think was 2003 because every business entity had an 
SEO expert insisting their various websites receive IP addresses on subnets 
that differed enough to be "distant" from one another because Google would 
otherwise penalize them.  I expressed frustration at that because it ensured 
sites that had no technical need for an exclusive IP address would get one 
anyway, wasting a rapidly depleting resource, and costing the provider in the 
process while they could still get address space.

A Google Director, Craig Silverstein, said this wasn't the case, but just 
casually in a slashdot interview.

Matt Cutts later refuted it directly in 2006:  
https://www.mattcutts.com/blog/myth-busting-virtual-hosts-vs-dedicated-ip-addresses/

And he made the point once more in a 2013 Youtube video.

Three semi-official statements on the subject, the most recent nine years ago.  
So, it hasn't done much to dissuade the SEO experts of continuing to steer 
their clients towards places they think an exclusive IP will be issued.  
Fortunately the huge rise of CDN's seems to be getting things back on track, 
because those can produce more meaningful SEO benefit from the faster transit 
to eyeballs, putting exclusive IP recommendations on the back burner.

David



On 3/31/22, 6:19 PM, "NANOG on behalf of Bill Woodcock" 
 wrote:

…in a run-of-the-mill web hoster?

This is really a question specifically for folks with web-site-hosting 
businesses.

If you had, say, ten million web site customers, each with their own unique 
domain name, how many IPv4 addresses would you think was a reasonable number to 
host those on?  HTTP name-based virtual-hosting means that you could, 
hypothetically, pile all ten million into a single IP address.  At the other 
end of the spectrum, you could chew up ten million IPv4 addresses, giving a 
unique one to each customer.  Presumably the actual practice lies somewhere 
in-between.  But what ratio do people in that business think is reasonable?  
10:1?  100:1?  1,000:1?

I’m happy to take private replies and summarize/anonymize back to the list, 
if people prefer.

Thanks!

-Bill

Re: Cogent ...

[David Hubbard]

I recently cancelled a circuit with them that began life as transit and 
converted to P2P, where the BGP fee never disappeared, and had been fighting 
them on it for eight months.  Now that the circuit is gone they've switched to 
completely ignore mode.  So, not likely I'll use them again.  I did the initial 
conversion because I got tired of customers with Google IPv6 issues and 
fortunately had a P2P need it could satisfy for a bit of time.  

On 3/31/22, 11:40 AM, "NANOG on behalf of Laura Smith via NANOG" 
 wrote:

Hmmm

Spring has sprung and the waft of drivel from a new season Cogent 
salesdroid filled my telephone earpiece today.

I've never liked the Cogent way of business and my understanding of their 
IP transit is that it falls into the "cheap for a reason" category.

However, perhaps someone would care to elaborate (either on or off-list) 
what the deal is with the requirement to sign NDAs with Cogent before they'll 
discuss things like why they still charge for BGP, or indeed any other 
technical or pricing matters. Seems weird ?!?

Laura

Opinions on Arista for BGP?

[David Hubbard]

Hi all, would love to get any current opinions (on or off list) on the 
stability of Arista’s BGP implementation these days.  Been many years since I 
last looked into it and wasn’t ready for a change yet.  Past many years have 
been IOS XR on NCS5500 platform and Arista everywhere but the edge.  I’ve been 
really happy with them in the other roles, so am thinking about edge now.  I do 
like and use XR’s RPL, and prefix/as/community/object sets, but we can live 
without via our own config management if there aren’t easy equivalents.  No 
fancy needs at all, just small web server networks, so just need reliable eBGP 
and internal OSPF/OSPFv3.

Thanks,

David

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[David Hubbard]

After sending them abuse reports for years with only an increase in malicious 
traffic, I have no expectation of anything they do getting better or being for 
the benefit of the internet as a whole.  Only reason this is probably getting 
any attention from them is in hopes they don’t irreparably damage their IPO; 
they seem to have no issues with their customers' compromised servers damaging 
the businesses of others on a continuous basis.  



On 3/12/21, 7:25 AM, "NANOG on behalf of Daniel Karrenberg" < > wrote:



On 11 Mar 2021, at 21:43, Randy Bush wrote:

> ...  but in a week or two
> i hope he can tell us results of more analysis. …

Actually just *the way* in which OVH communicates about this gives hope 
that we will indeed hear a useful analysis. It may be fortunate that 
this happened before they went public and thus corporate communications 
were ‘professionalised’ and vetted by the legal department. And yes 
he looked tired! Still did it in two languages. Good man.

Daniel

Re: OVH datacenter SBG2 in Strasbourg on fire 

[David Hubbard]

Was thinking the exact same thing; they and Digital Ocean seem to compete for 
the number two spot behind China as a malicious traffic source on all my public 
facing networks.

From the pics, the place looked like it has had quite a few semi-trailer 
containers added around it, perhaps they were running a little too much 
equipment in the building and it required power / hvac augmenting but fire 
suppression was left as it was...



On 3/10/21, 8:55 AM, "NANOG on behalf of JORDI PALET MARTINEZ via NANOG" 
 wrote:

In addition to that, even if this is not good for many "honest" people that 
was using the DC, we need to take it in the positive side. In my own case, OVH 
is probably the cause of 80% of the abuse cases I report, and they never react. 
I'm convinced I'm not the only one, as I read in other ops mailing lists ...

So, the positive side is a) during some days, we can see an interesting 
decrease in abuse cases, b) because the so many abuse cases, many OVH "honest" 
customers are often being filtered because they share addresses with the "bad 
guys", so it is an opportunity for them to move to alternative DCs that 
probably are more careful about "bad guys".

A good topic for researchers :-)

Regards,
Jordi
@jordipalet



El 10/3/21 16:44, "NANOG en nombre de Andy Ringsmuth" 
 escribió:


> On Mar 10, 2021, at 3:23 AM, Fredy Kuenzler  
wrote:
> 
> Very sad day for our colleagues at OVH AS16276 as they lost their 
datacenter SBG-2 in Strasbourg/France completly („everything is destroyed“) in 
a fire  and the neighboring SBG1/SBG3/SBG4 at least temporary.
> 
> 
https://www.dna.fr/amp/faits-divers-justice/2021/03/10/strasbourg-important-incendie-dans-une-entreprise-situee-sur-un-site-seveso-au-port-du-rhin

Sad to see of course, but also a little surprising that fire 
suppression systems didn’t, well, suppress the fire.

Unless they didn’t exist?



Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas, 1863




**
IPv4 is over
Are you ready for the new Internet ?
http://www.theipv6company.com
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the exclusive use of the 
individual(s) named above and further non-explicilty authorized disclosure, 
copying, distribution or use of the contents of this information, even if 
partially, including attached files, is strictly prohibited and will be 
considered a criminal offense. If you are not the intended recipient be aware 
that any disclosure, copying, distribution or use of the contents of this 
information, even if partially, including attached files, is strictly 
prohibited, will be considered a criminal offense, so you must reply to the 
original sender to inform about this communication and delete it.

Re: Viable Third Option?

[David Hubbard]

Yep, unlike 3356 who could care less if you have an outage, NTT never fails to 
have a ticket opened and email to all the contact points within minutes of a 
BGP session going down, asking if we need any assistance.  I’ve been really 
happy with their noc on debugging issues, and just proactive contact in 
general.  The peering seems good, as does the pricing.

Cogent honestly hasn’t been bad, but the v6 thing, to Google of all places, 
just makes their CEO look like a sh*t head; he needs to check his ego and just 
pay for the peering since Google doesn’t appear to be the one who will blink 
first.  I mean how can you seriously sell a circuit to anyone in a data center 
and have a caveat that massive, that no other provider has.



From: Dovid Bender 
Date: Wednesday, February 17, 2021 at 4:50 PM
To: David Hubbard 
Cc: "nanog@nanog.org" 
Subject: Re: Viable Third Option?

Second for NTT. We have found that their pricing wasn’t to far off from HE. I 
can count on one hand in 10 years how many times we had issues and needed to 
contact them.

On Wed, Feb 17, 2021 at 14:06 David Hubbard 
mailto:dhubb...@dino.hostasaurus.com>> wrote:
I’ve been pretty happy with NTT but their POPs can be limited; I’ve had to pick 
up waves to them, which sometimes still comes out ahead.  I’m slowly dropping 
Cogent due to the v6 issues.  I haven’t been able to try HE because they and a 
frequent colo provider I use (Switch) don’t seem to get along.

From: NANOG 
mailto:dino.hostasaurus@nanog.org>>
 on behalf of Mike Hammett mailto:na...@ics-il.net>>
Date: Wednesday, February 17, 2021 at 11:52 AM
To: NANOG list mailto:nanog@nanog.org>>
Subject: Viable Third Option?

This is from the perspective of an eyeball network. I understand that content 
networks would have different objectives and reasons. For instance, I have 
little to no reason as an eyeball network to exchange traffic with any other 
eyeball network (aside from P2P games). For a content network, getting into the 
eyeball networks is their objective.

My crystal ball tells me this thread will spiral out of control because people 
won't be able to keep it on topic, but it is a question that I hear VERY often. 
I also expect a lot of purely bad or outdated information to get thrown out.

Please try to keep it on topic and not being pedantic over relatively 
unimportant details.

There are two major low-cost providers, Cogent and HE.

Cogent

  *   Refuses to peer IPv6 with HE
  *   Refuses to peer IPv6 with Google
  *   Aggressive sales tactics
Hurricane

  *   Doesn't have Cogent IPv6 because of Cogent's refusal
  *   Lack of communities for anything other than blackholes

I know there are a variety of other providers such as Fusion Network that 
operate at similar price points, but are available in way fewer locations.

What else is out there? Anyone else that isn't 5x, 10x the cost?

Cogent and HE get looked down upon (and sometimes deservedly so), but when I 
talk to someone trying to sell me a port in 350 Cermak for 8x the cost of 
Cogent and HE, you better have a very good argument for why you're worth it...  
and they never do. "We're not Cogent." "and?" Many times I'm quoted transit 
that costs more than Cogent + IX + HE and they don't really have a good 
argument for it.

As an eyeball, I join an IX and there goes 50% - 85% of my traffic and almost 
all of my traffic that anyone is going to notice or complain about if there are 
issues (video streaming).

I do understand that enterprise eyeballs may have different requirements.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
Error! Filename not specified.<https://www.facebook.com/ICSIL>Error! Filename 
not 
specified.<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>Error! 
Filename not 
specified.<https://www.linkedin.com/company/intelligent-computing-solutions>Error!
 Filename not specified.<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
Error! Filename not specified.<https://www.facebook.com/mdwestix>Error! 
Filename not 
specified.<https://www.linkedin.com/company/midwest-internet-exchange>Error! 
Filename not specified.<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
Error! Filename not specified.<https://www.facebook.com/thebrotherswisp>Error! 
Filename not 
specified.<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Re: Viable Third Option?

[David Hubbard]

I’ve been pretty happy with NTT but their POPs can be limited; I’ve had to pick 
up waves to them, which sometimes still comes out ahead.  I’m slowly dropping 
Cogent due to the v6 issues.  I haven’t been able to try HE because they and a 
frequent colo provider I use (Switch) don’t seem to get along.

From: NANOG  on behalf 
of Mike Hammett 
Date: Wednesday, February 17, 2021 at 11:52 AM
To: NANOG list 
Subject: Viable Third Option?

This is from the perspective of an eyeball network. I understand that content 
networks would have different objectives and reasons. For instance, I have 
little to no reason as an eyeball network to exchange traffic with any other 
eyeball network (aside from P2P games). For a content network, getting into the 
eyeball networks is their objective.

My crystal ball tells me this thread will spiral out of control because people 
won't be able to keep it on topic, but it is a question that I hear VERY often. 
I also expect a lot of purely bad or outdated information to get thrown out.

Please try to keep it on topic and not being pedantic over relatively 
unimportant details.

There are two major low-cost providers, Cogent and HE.

Cogent

  *   Refuses to peer IPv6 with HE
  *   Refuses to peer IPv6 with Google
  *   Aggressive sales tactics
Hurricane

  *   Doesn't have Cogent IPv6 because of Cogent's refusal
  *   Lack of communities for anything other than blackholes

I know there are a variety of other providers such as Fusion Network that 
operate at similar price points, but are available in way fewer locations.

What else is out there? Anyone else that isn't 5x, 10x the cost?

Cogent and HE get looked down upon (and sometimes deservedly so), but when I 
talk to someone trying to sell me a port in 350 Cermak for 8x the cost of 
Cogent and HE, you better have a very good argument for why you're worth it...  
and they never do. "We're not Cogent." "and?" Many times I'm quoted transit 
that costs more than Cogent + IX + HE and they don't really have a good 
argument for it.

As an eyeball, I join an IX and there goes 50% - 85% of my traffic and almost 
all of my traffic that anyone is going to notice or complain about if there are 
issues (video streaming).

I do understand that enterprise eyeballs may have different requirements.



-
Mike Hammett
Intelligent Computing Solutions
[Image removed by sender.][Image removed by 
sender.][Image 
removed by 
sender.][Image
 removed by sender.]
Midwest Internet Exchange
[Image removed by sender.][Image removed by 
sender.][Image 
removed by sender.]
The Brothers WISP
[Image removed by sender.][Image 
removed by sender.]

Cross country latency on 3356?

[David Hubbard]

Curious if anyone is seeing issues with 3356 cross country, particularly 
Orlando-LA?  I have to assume they’re having issues in Texas, so perhaps too 
much capacity has been lost and it’s overloading what is functioning?

David

Re: Frontier Tampa issues

[David Hubbard]

Yes, exactly same issue for us, and it has happened in the past a few years ago 
fortunately.  Any chance the route takes a Level 3 (3356) path?  I’m just 
theorizing here, but my belief is they have some kind of link aggregation in 
the path from TB to 3356 (or maybe just internal near some edge) and some 
traffic is getting hashed onto a problematic link/interface/linecard, etc. 
where IPSec gets dropped.  One of our locations lost IPSec ability to some 
normal VPN endpoints but not others.  And here’s why I think this is the 
issue….  if you change the source and/or destination IP address by one, you may 
find some or all of your sessions magically work again.

In our case, one of our office locations has a static assignment of 
(fortunately) five IP’s.  We only have one external exposed, four site to site 
VPN’s.  Two began failing Saturday morning.  I moved the office firewall’s 
external IP minus 1 and that fixed both, but broke one that had been fine.  On 
the remote end fortunately I have equipment that’s able to override the local 
IP for VPN traffic, so without impacting other things it talks to, I was able 
to add a new IP one off from the previous, and use that for traffic just to 
this office location; that fixed the remaining issue.

If I’d not seen this previously several years ago, and wasted who knows how 
many hours trying to figure it out, it would have once again taken forever to 
resolve.  Trying to get through their support layer to someone who can really 
help is impossible.  The support is really complete garbage at this point after 
the Verizon dump; I was going to say service, but that’s been stable outside of 
these random weird issues that are impossible to resolve with support.

I tried to be a nice guy and raise this through the support channels, but could 
not make it past the layer where they want me to take our office down to have 
someone plug a laptop in with our normal WAN IP and “prove” ipsec isn’t working 
with different equipment.  I was like dude I just told you what I did to get it 
working again, offered packet captures, just escalate it, but ultimately gave 
up and hung up.

David

From: NANOG  on behalf 
of Nick Olsen 
Date: Sunday, January 24, 2021 at 8:42 PM
To: "nanog@nanog.org" 
Subject: Frontier Tampa issues

Anyone else seeing weird things on Tampa/Bradenton FIOS connections?

I've got three unrelated customers that cant establishes IPsec back to me.

And a third that can't process credit cards out to their third party merchant.

Customers are in 47.196.0.0/14.

In All instances, I see the traffic leave the CPE behind the FIOS circuit. The 
IPSEC traffic never makes it to my DC. And no clue on the credit card traffic. 
But it goes un-ack'd

And just now a fifth has appeared that can't query DNS against 8.8.8.8. 
Responses go out and never come back.

The first four all started around noon today.

Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!

[David Hubbard]

The leaking past the VPN thing is pretty obnoxious.  There are people who may 
be subject to policy and/or regulatory requirements that don’t permit split 
tunnels (even if supposedly not in userspace), so it will be interesting to see 
what burdens the use of an OS that intentionally leaks data will place on 
certain companies.  In contrast, it’s pretty funny that while they let their 
own data collection apps leak past a tunnel to call home, they do not let the 
link local ipv6 traffic that Sidecar uses leak past a non-split VPN; i.e. if 
I’m on corporate VPN, I can no longer connect my tablet as a Sidecar monitor to 
my Macbook because that traffic is blocked.


From: NANOG  on behalf 
of Mark Tinka 
Organization: SEACOM
Date: Tuesday, November 17, 2020 at 2:37 AM
To: Saku Ytti 
Cc: North American Network Operators Group 
Subject: Re: Apple Catalina Appears to Introduce Massive Jitter - SOLVED!


On 11/17/20 09:26, Saku Ytti wrote:
https://support.apple.com/en-us/HT202491

I am not trying to make any argument, just wanted to add context.

Yes, saw that too, and that post by Apple is also highlighted (and explained) 
in the same report.

The Gatekeeper OCSP checks remain unencrypted.

It still leave two glaring issues:

  *   Apple are still not saying anything about their OS apps bypassing local 
firewalls and leaking our IP address and location past any VPN's we may be 
running on Big Sur.

  *   The backdoor in iMessage's encryption that allows Apple and other 
"interested parties" to view our iMessage texts.
Mark.

bgp dampening and anycast networks (particularly cloudflare)

[David Hubbard]

Hi all, was curious if anyone has found it necessary to alter their route 
dampening rules related to anycast networks, and Cloudflare especially?  I’ve 
got a customer whose target web server has been going intermittently 
inaccessible from a very geographically distant Cloudflare location (AU), while 
no reports of issues from anywhere closer to the US.  I’m seeing a bunch of 
their /24’s dampened on my side in several locations, and they appear to be 
networks that favor or are specific to AU, so I’m thinking that’s the issue.  
I’m going to whitelist their ASN, but perhaps I need to work on the policy to 
be more tolerant of flaps compared to years past with the increase in anycast 
use?

Thanks,

David

Re: Cogent Layer 2

[David Hubbard]

I had a discussion with them about a point to point circuit last year and ran 
into some weirdness around how burstable it would be for specific IP to IP 
streams as our use case was cheap circuit / high speed data replication between 
given endpoints.  The sales rep was suggesting to me that I’d see specific 
source/destination IP pairs capped at 2gbps regardless of circuit speed, which 
suggested to me it was not actually a point to point wave but some type of 
encapsulated service.  We didn’t get into whether it was usable for non-IP, etc.



From: NANOG  on behalf 
of Mike Hammett 
Date: Wednesday, October 14, 2020 at 1:38 PM
To: "nanog@nanog.org" 
Subject: Cogent Layer 2

Are any legitimate beefs with Cogent limited to their IP policies, BGP session 
charges, and peering disputes? Meaning, would using them for layer 2 be 
reasonable?



-
Mike Hammett
Intelligent Computing Solutions
[Image removed by sender.][Image removed by 
sender.][Image 
removed by 
sender.][Image
 removed by sender.]
Midwest Internet Exchange
[Image removed by sender.][Image removed by 
sender.][Image 
removed by sender.]
The Brothers WISP
[Image removed by sender.][Image 
removed by sender.]

Re: CenturyLink -> Lumen

[David Hubbard]

I think this is just someone trying to pull the stock price out of the dumps by 
branding themselves a “tech company”.  There are still things from the 
TWTelecom days they haven’t finished integrating into the control panel, this 
should be fun watching them try to change the name at the same time.

From: NANOG  on behalf 
of "R. Leigh Hennig" 
Reply-To: "R. Leigh Hennig" 
Date: Wednesday, September 16, 2020 at 12:51 AM
To: "nanog@nanog.org" 
Subject: CenturyLink -> Lumen

https://www.fiercetelecom.com/telecom/centurylink-rebrands-re-defines-enterprise-sector-as-lumen-technology

Curious. Any thoughts on how this changes their business approach, if any? 
Obviously something like this has to be planned far in advance, but I can’t 
help but wonder what impact the recent outage and bad press might have had on 
their plans here, possibly accelerating them? Probably not. But it’s an 
interesting move regardless.


. | R. Leigh Hennig, Principal Network Architect
..| Markley Group https://markleygroup.com


Sent from ProtonMail Mobile

Re: curious spam...

[David Hubbard]

Here in Florida the self-preservation interests of the two party system have 
resulted in all voter registrations being made public, including email, d/o/b, 
phone, home address (since you can't legally register any other), party 
affiliation.  If you used your private email for any state government 
registrations, they may have leaked it as soon as you moved.  Alternatively, if 
your previous state had already leaked it, and you have declared a party 
affiliation, the state level entity likely shared it with the national entity, 
who then shared it with the new state entity where you moved so they can spam 
you all over again.  I made the mistake of donating to a party backed candidate 
about a decade ago, and the cesspool of political entities associated with that 
party continue to email and text me every single cycle.  Unless I start suing, 
or change all my contact info, there's no likely any way I'll ever get it to 
stop.

I believe several states' DMV's have been found to be selling license 
registration info as a revenue source too.

Florida does have a way to not have your personal info released; it's 
conveniently only available to people you'd expect, first responders, judges, 
and of course, members of congress.



On 9/14/20, 2:32 PM, "NANOG on behalf of William Herrin" 
 wrote:

Howdy,

I've noticed something odd. When I lived in Virginia, I started
receiving email directly to my gmail box from my U.S. Representative.
Unsolicited spam from Congressmen is nothing new but it was a little
odd that they found my gmail box (which I don't give out) and not one
of the hundreds of aliases at herrin.us or dirtside.com which I do
give out. The gmail box exists only in mail headers; "From" is always
a different address.

I moved to Seattle. Today I found my grmail box subscribed to a
congressman's list from a nearby Washington jurisdiction. Not some
random congressman. And not any of the addresses I give out; my gmail
box's address which I don't.

Anyone else have a similar experience? Any idea how a hidden address
is making it on to relevant congressmens' lists but not any others?
That's weird right?

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Does anyone actually like CenturyLink?

[David Hubbard]

It’s just due to network size.  Horrid service and reliability aside, if you 
have enough eyeballs, application providers will want to directly peer, and if 
you have enough app providers on net then access providers will want to peer.  
With all the acquisitions, they have a ton of fiber in the ground and they keep 
rolling the networks in, causing people to continue to buy their circuits.

The service has been horrible compared to the twtelecom days.  Fortunately 
their layer two stuff seems to be managed independently of their IP networks, 
so those remained up.

From: NANOG  on behalf 
of Ross Tajvar 
Date: Sunday, August 30, 2020 at 11:05 AM
To: North American Network Operators' Group 
Subject: Does anyone actually like CenturyLink?

I've never heard a single positive word about them, and I've had my fair share 
of issues myself (as an indirect customer). But it seems that lots of people 
put them in their transit blend. Other than lack of options, why would anyone 
use them? To me, it just seems like asking for trouble...but maybe I'm missing 
something?

Re: Centurylink having a bad morning?

[David Hubbard]

I just brought one of my sessions back up to attempt to avoid the blackholing, 
should be full feed, getting all of 850 v4 routes and 106 v6.

From: NANOG  on behalf 
of Tomas Lynch 
Date: Sunday, August 30, 2020 at 9:41 AM
To: Drew Weaver 
Cc: "nanog@nanog.org" 
Subject: Re: Centurylink having a bad morning?

Flapping in Miami, Dallas, Atlanta, Los Angeles, Seattle and San Jose. It is 
also affecting some data centers in Europe too. but haven't seen flaps there, 
just suboptimal routing.

On Sun, Aug 30, 2020 at 8:53 AM Drew Weaver 
mailto:drew.wea...@thenap.com>> wrote:
Saw the flapping in Cleveland but not in Cincinnatti or Ashburn…

From: Tomas Lynch mailto:tomas.ly...@gmail.com>>
Sent: Sunday, August 30, 2020 8:45 AM
To: Mel Beckman mailto:m...@beckman.org>>
Cc: Drew Weaver mailto:drew.wea...@thenap.com>>; 
nanog@nanog.org
Subject: Re: Centurylink having a bad morning?

BGP sessions randomly flapping or having routing issues in different cities 
since ~5AM EST

On Sun, Aug 30, 2020 at 8:42 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
The CL portal loads for me, and I can log in, but it is slower than usual. Not 
seeing traffic issues on our CL circuits.
-mel via cell

On Aug 30, 2020, at 5:23 AM, Drew Weaver via NANOG 
mailto:nanog@nanog.org>> wrote:
Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Re: Centurylink having a bad morning?

[David Hubbard]

Same.  Also, as reported on outages list, what’s even worse is that they appear 
to be continuing to propagate advertisements from circuits whose sessions have 
been turned down.  I validated ours still were via a couple looking glass 
portals.  Down Detector shows nearly every major service provider impacted.

They’re not reachable so who knows if they’re even working on it.  I feel like 
they’ve been cutting heavily on the network ops side in recent years…

From: NANOG  on behalf 
of Drew Weaver via NANOG 
Reply-To: Drew Weaver 
Date: Sunday, August 30, 2020 at 8:23 AM
To: "nanog@nanog.org" 
Subject: Centurylink having a bad morning?

Hello,

Woke up this morning to a bunch of reports of issues with connectivity had to 
shut down some Level3/CTL connections to get it to return to normal.

As of right now their support portal won’t load: 
https://www.centurylink.com/business/login/

Just wondering what others are seeing.

Product for heat containment per rack unit?

[David Hubbard]

Curious if anyone has knowledge of a vendor / product designed to make it 
possible to use back-to-front cooled equipment in racks that need to be 
‘sealed’ for heat containment reasons?  I’d envision this looking like some 
kind of adjustable depth sleeve, to get the cold air to the equipment, and 
perhaps a brush strip opening to allow power cables in?

Thanks!

Re: Is there *currently* a shortage of IPv4 addresses?

[David Hubbard]

Agreeing with the other replies about scarcity.  Also wanted to comment that 
address exhaustion affects web hosts particularly hard because "SEO experts" 
continue to believe that if a site they work on does not have an exclusive IP, 
they're being  penalized by Google.  They'll convince clients to migrate around 
hosts until they find one that will allocate an address, so the choice is buy 
address space or suffer if your platform is not otherwise unique.



On 8/4/20, 3:36 PM, "NANOG on behalf of Anne P. Mitchell, Esq." 
 wrote:

I know that a shortage of IPv4 addresses has been anticipated for quite 
some time (literally decades), however, is there a shortage *right now*?

I ask, because Liquid Web is using it as an excuse to raise their prices:

"We're contacting you today to inform you of a change to your account. As 
you may know, the global shortage of IPv4 addresses 
(https://www.ripe.net/manage-ips-and-asns/ipv4/ipv4-run-out) continues to 
impact web hosting companies around the world. ... Effective August 31st, we 
will be updating our per IPv4 address price to $2.00 per IP."

Anne

--
Anne P. Mitchell,  Attorney at Law
Dean of Cyberlaw & Cybersecurity, Lincoln Law School
CEO, SuretyMail Email Reputation Certification
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law)
Board of Directors, Denver Internet Exchange
Chair Emeritus, Asilomar Microcomputer Workshop
Former Counsel: Mail Abuse Prevention System (MAPS)

Re: IPv6 over vxlan+evpn Arista?

[David Hubbard]

Nope the underlay can be v4-v4, just need to be able to carry the v4+v6 overlay 
to allow for migration of addresses.

From: Tyler Conrad 
Date: Friday, July 10, 2020 at 12:39 PM
To: David Hubbard 
Cc: "nanog@nanog.org" 
Subject: Re: IPv6 over vxlan+evpn Arista?

Do you need to carry the v6 af in the underlay? I’ve used 6pe/6vPE to carry v6 
over v4 next-hops in the overlay without issue, but can’t say I’ve tested a 
dual-stack vtep.

On Fri, Jul 10, 2020 at 08:07 David Hubbard 
mailto:dhubb...@dino.hostasaurus.com>> wrote:
Hi all, was curious if anyone is doing dual stack v4/v6 over Arista’s 
implementation of vxlan / evpn (the inter-data center transport would be v4)?  
They have plenty of references for v4 deployments but had to check on v6 
support, which can make one nervous; they did confirm it’s supported.  Looking 
to leverage it as a way to migrate a dual-stack data center without 
re-addressing.

Thanks,

David

IPv6 over vxlan+evpn Arista?

[David Hubbard]

Hi all, was curious if anyone is doing dual stack v4/v6 over Arista’s 
implementation of vxlan / evpn (the inter-data center transport would be v4)?  
They have plenty of references for v4 deployments but had to check on v6 
support, which can make one nervous; they did confirm it’s supported.  Looking 
to leverage it as a way to migrate a dual-stack data center without 
re-addressing.

Thanks,

David

Re: RIPE NCC Executive Board election

[David Hubbard]

I suspect he’d want to slow adoption and push his frankestein IPv4 because any 
extension of IPv4 use makes the netblocks’s he’s obtained questionable 
‘ownership’ of more valuable, in theory.

From: NANOG  on behalf 
of Baldur Norddahl 
Date: Wednesday, May 13, 2020 at 5:02 PM
To: "nanog@nanog.org" 
Subject: Re: RIPE NCC Executive Board election

Akamai already has 15% peak IPv6 traffic:

https://blogs.akamai.com/2020/02/at-21-tbps-reaching-new-levels-of-ipv6-traffic.html

Some internet service providers may have more than half of their traffic as 
IPv6.

Some countries are now crossing more than 50% IPv6 availability:

https://www.google.com/intl/en/ipv6/statistics.html

Why do you think you can overtake the IPv6 train? Why would we want to abandon 
the work already done?

Re: RIPE NCC Executive Board election

[David Hubbard]

It just keeps getting dumber by the minute.  My home ISP hasn’t even updated 
firmware to one that supports v6, but yeah, they’re surely going to update to 
your Frankenstein ipv4 because you’re going to give them a taste of addresses 
from the nightmare pool that will reach even less of the internet than v6.

From: NANOG  on behalf of Elad Cohen 
Date: Wednesday, May 13, 2020 at 3:41 PM
To: Mikael Abrahamsson 
Cc: NANOG list 
Subject: Re: RIPE NCC Executive Board election


Do you realise that this means you're requiring changing *every*
socket-speaking application in the world?

Every internet host that will want to speak IPv4+ , will have an update (for 
example through the operating systems automatic updates mechanisms)



It's taken us decades to get applications to use the new struct to support
IPv6+IPv4, resetting the timer back to 0 and starting over does not help
deployment. It just kicks it another 20 years down the line.

I wrote about the usage of a roundtable in order to implement everything fast 
(the roundtable will include one representative from each of the operating 
system vendors, one representative from each of the routing equipment 
manufacturers and one representative from each of the 5 RIR's), if I will be 
elected to RIPE board I will do everything in my power so this roundtable will 
be formed fast and that the needed updates will be created fast. Each party in 
the roundtable will receive an amount of free IPv4 addresses from the new IPv4+ 
pool, and each ASN will also receive for example a /21 , home-routers and 
home-modems will not be needed to be updated and they will support IPv4+.



You're just inventing yet another incompatible standard and you have to
touch everything, DHCP, DNS all applications etc.

There is an adjustment to IPv4+ that the format of addresses will not be 
[0-655365].[0-655365]v4 - but it will be [256-511].[0-255].[0-255].[0-255]
So IPv4+ addresses will be in the format of IPv4 addresses - it will end-user 
adoption of IPv4+ easier and also integration in the applicative layer easier 
(as application developers will only need to set that the first number can be 
higher instead to support a new format of [0-655365].[0-655365]v4 )



From: Mikael Abrahamsson 
Sent: Wednesday, May 13, 2020 10:22 PM
To: Elad Cohen 
Cc: Brielle ; NANOG list 
Subject: Re: RIPE NCC Executive Board election

On Wed, 13 May 2020, Elad Cohen wrote:

> LOL funny seeing you changing your mind by 180 degrees when someone you
> know in the community writing to you the exact same thing.

"In addition, the sockets API should be extended to support IPxl with a
new socket domain PF_IPXL which is identical to PF_INET in every respect
save that the IP addresses are 8 bytes long instead of 4."

Do you realise that this means you're requiring changing *every*
socket-speaking application in the world?

It's taken us decades to get applications to use the new struct to support
IPv6+IPv4, resetting the timer back to 0 and starting over does not help
deployment. It just kicks it another 20 years down the line.

You're just inventing yet another incompatible standard and you have to
touch everything, DHCP, DNS all applications etc.

--
Mikael Abrahamssonemail: swm...@swm.pp.se

Re: RIPE NCC Executive Board election

[David Hubbard]

LOL the IPv4+ thing was a pretty entertaining read.  You clearly don’t have 
even a basic understanding of the v4 packet structure, or that the octet 
display concept is simply for human benefit.  IPv6 can be implemented with 
‘software updates’ too…

From: NANOG  on behalf of Elad Cohen 
Date: Wednesday, May 13, 2020 at 9:47 AM
To: "Ronald F. Guilmette" , "nanog@nanog.org" 

Subject: Re: RIPE NCC Executive Board election

Hello Everyone,

My apology for not providing an official response to the first "The Ronald 
Show" that took place here many months ago, I was out of hospital after full 
anesthesia and it took me months to get back to myself.

What was done towards me and what being done to me by Ronald is an intentional 
personal attack against me and I will explain.

I didn't agree to provide private business documents to the illegal anonymous 
organization "The Spamhaus Project" that they tried to extort from me - then I 
found out who are the real people behind the illegal anonymous organization 
"The Spamhaus Project" - and then they started to attack me (including here on 
Nanog, with false information, when I was out of hospital and wasn't in the 
condition to respond to them).

"The Spamhaus Project" is an illegal anonymous organization according to their 
own words in their own following presentation:

https://www.scribd.com/document/445894312/Spamhaus-Illegal-Private-Data-Violation

They wrote in it that they receive on a regular basis - massive amount of 
illegaly-obtained privacy data from their contacts in many internet companies 
and internet organizations - and then they share it in illegal way (without any 
warrant) with Law Enforcement Agencies - this is the reason that Law 
Enforcement Agencies are doing nothing regarding the illegal anonymous 
organization "The Spamhaus Project".

Ronald Guilmette is the front person of the illegal anonymous organization "The 
Spamhaus Project".



and that said legal counsel has then
proceeded to threaten various officials of the City of Cape Town,
South Africa with possible legal action if they do not relinquish
to him their rights in and title to the 165.25.0.0/16 block

This is a complete lie, in order to shame CoCT I will not share their internal 
correspondences. Anyone interested to know more information can email me 
directly.



"I am assured that at no time did the City of Cape Town ever sell, trade, or 
barter away their rights to this valuable IPv4 block"

Ronald is "assured" exactly as he is "assured" that all Dutch people are 
criminals and all Colombian network oeprators are criminals and all Chicago 
citizens are criminals according to his statements in the following links:
https://imgur.com/AcmgwEX
https://imgur.com/WUZvdNJ
https://imgur.com/a/Rzrbxkz

Ronald was called an antisemitic and a racist person here on Nanog in the 
following two links, by people which are not related to me:
https://imgur.com/AQCmZlk
https://imgur.com/a/Rzrbxkz


In the first "The Ronald Show" many months ago - Ronald called me in two 
antisemitic names that are being used to relate to Jews in Shakespeare 
literature (just like Ronald is relating to Shakespeare in 
https://imgur.com/AcmgwEX ) - as part of his defamation campaign towards me - 
none of you raised a voice and not only that but some Nanog subscribers enjoyed 
his racism and antisemitism (without a single proof against me) as was written 
here by them (with a popcorn).

The "source" in "The Spamhaus Project" that supported and pumped Ronald 
Imagination is the criminal of the anonymous twitter account: 
https://twitter.com/underthebreach , that person according to his own words in 
his own criminal anonymous twitter account - is a master of cyber influence 
operations (meaning to influence people without a single proof) - that person 
is also an employee of the Israeli-based company GeoEdge and they are a direct 
competitor of a company that used the netblocks that Ronald attacked - not only 
that but Ronald also attacked another Israeli-based company called Divineworks 
(here in Nanog) and they are also a direct competitor of the Israeli-based 
company GeoEdge. What was done here is a cyber influence operation without a 
single proof because of a business competition.

That criminal https://twitter.com/underthebreach which is a member of "The 
Spamhaus Project" and pumped Ronald Imagination - is not the only person which 
is abusing the power of "The Spamhaus Project" for commericial goals, that are 
many many more people and companies behind the illegal anonymous organization 
"The Spamhaus Project" which are making profit from it, for example:

- Vincent Schonau from Open-Xchange (AKA "Vincenet Hanna" of "The Spamhaus 
Project") that "The Spamhaus Project" is attacking the competitors of 
Open-Xchange.
- Laura Atkins from WordToTheWise which is selling a Spamhaus listing removal 
service (https://wordtothewise.com)

Real identities behind "The Spamhaus Project":

"Mike 

Re: An appeal for more bandwidth to the Internet Archive

[David Hubbard]

Could the operation be moved out of California to achieve dramatically reduced 
operating costs and perhaps solve some problems via cost savings vs increased 
donation?  I have to imagine with the storage and processing requirements that 
the footprint and power usage in SFO is quite costly.  I have equipment in a 
few California colo's and it's easily 3x what I pay for similar in Nevada, 
before even getting into tax abatement advantages.



On 5/12/20, 1:33 PM, "NANOG on behalf of colin johnston" 
 wrote:

Is the increased usage due to more users or more existing users having 
higher bandwidth at home to request faster ?
Would be interested if IPS configured firewall used to block out invalid 
traffic/spam traffic and if such traffic increased when back end network 
capacity increased ?
What countries are requesting the most data and does this analysis throw up 
questions as to why ?
Are there high network usage hitters which raise question as to why asking 
for so much data time and time again and is this valid traffic use ?

Colin


> On 12 May 2020, at 17:33, Tim Požár  wrote:
> 
> Jared...
> 
> Thanks for sharing this.  I was the first Director of Operations from '96 
to '98, at was was then Internet Archive/Alex.  I was the network architect 
back then got them their ASN and original address space. Folks may also know, I 
help start SFMIX with Matt Peterson.
> 
> A bit more detail in this...  Some of this I got from Jonah Edwards who 
is the current Network Architect at IA.  Yes, the bottle neck was the line 
cards.  They have upgraded and that has certainly helped the bandwidth of late.
> 
> Peering would be a big help for IA. At this point they have two 10Gb LAG 
interfaces that show up on SFMIX that was turned up last February. Looking at 
the last couple of weeks the 95th percentile on this 20Gb LAG is 3 Gb.  As they 
just turned up on SFMIX, they are just starting to get peers turned up there. 
Eyeball networks that show up on SFMIX are highly encouraged to start peering 
with them.  Alas, they are v4 only at this point.
> 
> Additionally, if folks do have some fat pipes that can donate bandwidth 
at 200 Paul, I am sure Jonah won't turn it down.
> 
> Tim
> 
> On 5/12/20 4:45 AM, Jared Brown wrote:
>> Hello all!
>> Last week the Internet Archive upgraded their bandwidth 30% from 47 Gbps 
to 62 Gbps. It was all gobbled up immediately. There's a lovely solid green 
graph showing how usage grows vertically as each interface comes online until 
it too is 100% saturated. Looking at the graph legend you can see that their 
usage for the past 24 hours averages 49.76G on their 50G of transport.
>> To see the pretty pictures follow the below link:
>> 
https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/
>> Relevant parts from the blog post:
>> "A year ago, usage was 30Gbits/sec. At the beginning of this year, we 
were at 40Gbits/sec, and we were handling it. ...
>> Then Covid-19 hit and demand rocketed to 50Gbits/sec and overran our 
network infrastructure’s ability to handle it.  So much so, our network 
statistics probes had difficulty collecting data (hence the white spots in the 
graphs).
>> We bought a second router with new line cards, and got it installed and 
running (and none of this is easy during a pandemic), and increased our 
capacity from 47Gbits/sec peak to 62Gbits/sec peak.   And we are handling it 
better, but it is still consumed."
>> It is obvious that the Internet Archive needs more bandwidth to power 
the Wayback machine and to fulfill its mission of being the Internet library 
and the historic archive of our times.
>> The Internet Archive is present at Digital Realty SFO (200 Paul) and a 
member of the San Francisco Metropolitan Internet Exchange (SFMIX).
>> I appeal to all list members present or capable of getting to these 
facilities to peer with and/or donate bandwidth to the Internet Archive.
>> I appeal to all vendors and others with equipment that they can donate 
to the Internet Archive to contact them so that they can scale their services 
and sustain their growth.
>> The Internet Archive is currently running 10G equipment. If you can help 
them gain 100G connectivity, 100G routing, 100G switching and/or 100G DWDM 
capabilities, please reach out to them. They have the infrastructure and dark 
fiber to transition to 100G, but lack the equipment. You can find the Internet 
Archive's contact information below or you can contact Jonah at the Archive Org 
directly either by email or via the contact information available on his 
Twitter profile @jonahedwards.
>> You can also donate at https://archive.org/donate/
>> The Internet Archive is a 501(c)(3) non-profit. Donations are  
tax-deductible.
>> Contact information:
>> https://archive.org/about/contact.php
>> Volunteering:
>> 

Alternative for Google Safe Browsing for Network Administrators?

[David Hubbard]

Just received notice that Google is eliminating the Safe Browsing for Network 
Administrators service…. in favor of a new paid alternative; imagine that.

Are there recommended similar services out there that will send netblock owners 
alerts related to suspected compromised websites, malware distribution, C, 
etc. activity?

Thanks

Re: FYI - Suspension of Cogent access to ARIN Whois

[David Hubbard]

When they spam me I typically just ask if they have IPv6 to Google and never 
hear back…

From: NANOG  on behalf of David Guo via NANOG 

Reply-To: David Guo 
Date: Monday, January 6, 2020 at 11:06 AM
To: John Curran , "nanog@nanog.org" 
Subject: RE: FYI - Suspension of Cogent access to ARIN Whois

Good News! But we still received several spams from Cogent for our RIPE and 
APNIC ASNs.

From: NANOG  On Behalf Of John Curran
Sent: Monday, January 6, 2020 11:43 PM
To: nanog@nanog.org
Subject: FYI - Suspension of Cogent access to ARIN Whois
Importance: High

On 22 Sep 2019, at 8:52 AM, Tim Burke mailto:t...@tburke.us>> 
wrote:

That is just The Cogent Way™, unfortunately. I just had (yet another) Cogent 
rep spam me using an email address that is _only_ used as an ARIN contact, 
trying to sell me bandwidth. When I called him out on it, with 
complia...@arin.net CCed, he backpedaled and 
claimed to obtain my information from Google.

ARIN has repeatedly informed Cogent that their use of the ARIN Whois for 
solicitation is contrary to the terms of use and that they must stop.  Despite 
ARIN’s multiple written demands to Cogent to cease these prohibited activities, 
ARIN has continued to receive complaints from registrants that Cogent continues 
to engage in these prohibited solicitation activities.

For this reason, ARIN has suspended Cogent Communications’ use of ARIN’s Whois 
database effective today and continuing for a period of six months.  For 
additional details please refer to 
https://www.arin.net/vault/about_us/corp_docs/20200106_whois_tos_violation.pdf  
  ARIN will restore Cogent’s access to the Whois database at an earlier time if 
Cogent meets certain conditions, including instructing its sales personnel not 
to engage in the prohibited solicitation activities.

Given the otherwise general availability of ARIN Whois, it is quite possible 
that Cogent personnel may evade the suspension via various means and continue 
their solicitation.  If that does occur, please inform us (via 
complia...@arin.net), as ARIN is prepared to extend 
the suspension and/or bring appropriate legal action.

FYI,
/John

John Curran
President and CEO
American Registry for Internet Numbers

Re: all major US carriers received text messages overnight that appear to have been sent around Valentine's Day 2019

[David Hubbard]

Playing devil’s advocate, perhaps they were under emergency court order to not 
deliver texts for a certain duration, market, who knows what, and that order 
just ended, but some type of non-disclosure / secrecy directive continues to 
exist… may have just had to come up with something to say because their other 
agreements would not have permitted discarding the texts… 

David

From: NANOG  on behalf 
of Mark Stevens 
Date: Friday, November 8, 2019 at 1:45 PM
To: "nanog@nanog.org" 
Subject: Re: all major US carriers received text messages overnight that appear 
to have been sent around Valentine's Day 2019

Reading Syniverse's cause of trouble (lame excuse) tells me their data handling 
processes are poor and seemingly shady since I do not buy reason for the 
trouble.

On 11/8/2019 1:34 PM, Kain, Becki (.) wrote:
Esp on Valentine’s day.  Of all the days that clear communication is important. 
 I’d be very interested in their reasoning for why these messages were not sent 
and held.

From: NANOG  On Behalf 
Of Oliver O'Boyle
Sent: Friday, November 08, 2019 1:31 PM
To: Matt Hoppes 

Cc: North American Network Operators' Group 

Subject: Re: all major US carriers received text messages overnight that appear 
to have been sent around Valentine's Day 2019

We apologize for finally getting around to our job and doing what we were paid 
to do...

On Fri, Nov 8, 2019 at 1:27 PM Matt Hoppes 
mailto:mattli...@rivervalleyinternet.net>> 
wrote:
“During an internal maintenance cycle last night, 168,149 previously 
undelivered text messages were inadvertently sent to multiple mobile operators’ 
subscribers," Syniverse said in a statement.


how do you inadvertently send messages that were supposed to be sent but worked 
and sent? Isn’t that the desired outcome?

On Nov 8, 2019, at 12:54 PM, Brandon Svec 
mailto:bs...@teamonesolutions.com>> wrote:
From: 
https://www.usatoday.com/story/tech/2019/11/08/thousands-people-just-got-text-messages-sent-valentines-day/2527660001/

It seems there is a company that has everyone's text messages..

"Some mobile carriers rely on a third-party text platform called Syniverse to 
relay messages. The vendor said in a statement that its IT staff unknowingly 
caused the texts to be delivered this week."
-Brandon




On Fri, Nov 8, 2019 at 9:47 AM Brian J. Murrell 
mailto:br...@interlinx.bc.ca>> wrote:
On Thu, 2019-11-07 at 22:42 +, Chris Kimball via NANOG wrote:
> Does anyone have any more information on this?

Yeah, like who (in the private sector -- we all knew the NSA already
are doing this) has access to and is archiving *everyone*s text
messages?  And why?

Cheers,
b.


--
:o@>

Security alert aggregator?

[David Hubbard]

Curious if anyone knows of a security alert aggregation service?  For example, 
go and plug in all the various vendors hardware and software packages your 
enterprise uses, and then the service subscribes to all the random RSS feeds, 
CVE lists, vendor mailing lists, etc. to feed you the data instead of needing 
staff to write something custom, and then have checks to ensure the custom 
thing is still pulling from the right location, etc.

Thanks

Re: Cogent sales reps who actually respond

[David Hubbard]

Our sales rep has been great, but unfortunately, for him, every time he calls 
and I ask if Cogent is going to get me IPv6 transit to Google, he has to say 
no, and then I tell him I can’t purchase any more circuits.

From: NANOG  on behalf of Owen DeLong 
Date: Monday, September 16, 2019 at 9:20 AM
To: "n...@as37662.com n...@as37662.com" 
Cc: "nanog@nanog.org" 
Subject: Re: Cogent sales reps who actually respond

Given their practice of harvesting whois updates in order to spam newly 
acquired AS contacts, any time it is my decision, Cogent is ineligible as a 
vendor.

I’ve had no trouble getting their reps to respond when the decision has come 
from above, but I prefer to avoid doing business with them.

Owen



On Sep 15, 2019, at 13:13 , n...@as37662.com 
n...@as37662.com 
mailto:n...@as37662.com>> wrote:

Hi fellow network operators,
Do any orgs here have experience with a good Cogent rep? The rep we got via 
Cogent's website is unresponsive to even basic questions. It feels like we are 
dealing with a bot and copy-pasted replies.
Thanks
Ruldu

Re: IPAM recommendations

[David Hubbard]

I wish Digital Ocean would put as much effort into policing their network; at 
least two thirds of the malicious traffic hitting our customers comes from an 
even split between them and OVH.

From: NANOG  on behalf of Mel Beckman 

Date: Thursday, September 5, 2019 at 10:48 AM
To: Phillip Carroll 
Cc: nanog 
Subject: Re: IPAM recommendations

I agree with Phil, Netbox is a great opens source IPAM project. We currently 
use ManageEngine, but I plan to switch to Netbox when our current license is up 
for renewal. NetBox. The project is supported by Digital Ocean, which is the 
kind of corporate sponsorship that keeps open source project from dying out.

It’s one of the few IPAM products that recognizes that IP addresses can be 
assigned to interfaces on a device, not necessarily the device itself. It also 
supports interfaces having multiple IP addresses. Netbox uses Postgres under 
the covers, which has IP addresses as a native data type. That means you can 
also build your own SQL queries to interface with other systems.

The tool is not frilly, but has all the features an IPAM should have for 
accurate and timely resource management. Plus the code looks clean.
 -mel

On Sep 5, 2019, at 6:48 AM, Phillip Carroll 
mailto:phill...@phmgmt.com>> wrote:


https://github.com/netbox-community/netbox


From: NANOG 
mailto:nanog-bounces+phillipc=phmgmt@nanog.org>>
 On Behalf Of Andrew Latham
Sent: Thursday, September 5, 2019 8:20 AM
Cc: nanog mailto:nanog@nanog.org>>
Subject: Re: IPAM recommendations

 [EXTERNAL EMAIL]

Please check the mailing list archives as a resource. I made a short list last 
time https://lathama.net/DCIM which looks to be June 20th 2018

On Thu, Sep 5, 2019 at 3:37 AM Mehmet Akcin 
mailto:meh...@akcin.net>> wrote:
Looking for IPAM recommendations, preferably open source, API is a plus (almost 
must, almost..). 40-50K IPs to be managed.

thanks in advance.


--
- Andrew "lathama" Latham -

Re: CenturyLink/Level 3 combined AS

[David Hubbard]

Cogent is great, or worthless, depending on whether you like talking to Google 
via IPv6.

From: NANOG  on behalf of Darin Steffl 

Date: Saturday, June 8, 2019 at 9:10 AM
To: Brielle Bruns 
Cc: North American Network Operators' Group 
Subject: Re: CenturyLink/Level 3 combined AS

Ok just so simplify things.

Is Cogent or CenturyLink/L3 better for transit?

On Fri, Jun 7, 2019, 3:00 PM Brielle Bruns 
mailto:br...@2mbit.com>> wrote:
On 6/7/2019 11:03 AM, Romeo Czumbil wrote:
> All new CL Internet get's provisioned on AS3356
> You would need a strong case for them to put you on AS209


Got provisioned last year on AS209 when they turned up my ent Fiber with
BGP.

Could depend heavily on what services and where.

--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: SFP supplier in Europe?

[David Hubbard]

Flexoptix may be an option; they're Germany.  Even US shipping is typically two 
day.

On 4/4/19, 4:10 PM, "NANOG on behalf of nanog-...@mail.com" 
 wrote:

Hello NANOG,

Could somebody recommend an SFP supplier in Europe with a warehouse in the 
EU and fast shipping? I need to pick up some 80km Bidi SFPs and I'd prefer to 
use a supplier has and will keep stock locally.

Jared

Re: Frontier rural FIOS & IPv6

[David Hubbard]

Things are no better in Spectrum land; gotta love the innovation in monopoly 
markets….  I ask every year and expect it in perhaps thirty.

From: NANOG  on behalf of "Aaron C. de Bruyn via 
NANOG" 
Reply-To: "Aaron C. de Bruyn" 
Date: Sunday, March 31, 2019 at 4:26 PM
To: "C. A. Fillekes" 
Cc: NANOG mailing list 
Subject: Re: Frontier rural FIOS & IPv6

You're not alone.

I talked with my local provider about 4 years ago and they said "We will 
probably start looking into IPv6 next year".
I talked with them last month and they said "Yeah, everyone seems to be 
offering it.  I guess I'll have to start reading how to implement it".

I'm sure 2045 will finally be the year of IPv6 everywhere.

-A

On Sat, Mar 30, 2019 at 7:36 AM C. A. Fillekes 
mailto:cfille...@gmail.com>> wrote:

So by COB yesterday we now officially have FIOS at our farm.

Went from 3Mbps to around 30 measured average.  Yay.

It's a business account, Frontier.  But...still no IPv6.

The new router's capable of it.  What's the hold up?

Customer service's response is "We don't offer that".

Was wrong Re: Did IPv6 between HE and Google ever get resolved?

[David Hubbard]

Oops, I was corrected that HE doesn’t have IPv6 issues with Google, not sure 
why I had that in my head.  Cogent certainly does but something had me thinking 
there’s another big name that has the same problem.

David

From: NANOG  on behalf of David Hubbard 

Date: Thursday, March 28, 2019 at 12:40 PM
To: NANOG List 
Subject: Did IPv6 between HE and Google ever get resolved?

Hey all, I’ve been having bad luck searching around, but did IPv6 transit 
between HE and google ever get resolved?  Ironically, I can now get to them 
cheaply from a location we currently have equipment that has been Cogent-only, 
so if it fixes the IPv6 issue I’d like to make the move.  Anyone peer with HE 
in general and want to share their experience offlist?  With the price, if 
they’re a good option, I’d consider rolling them in to other locations where we 
have redundancy already, so the v6 isn’t as big a deal there.

Thanks

Did IPv6 between HE and Google ever get resolved?

[David Hubbard]

Hey all, I’ve been having bad luck searching around, but did IPv6 transit 
between HE and google ever get resolved?  Ironically, I can now get to them 
cheaply from a location we currently have equipment that has been Cogent-only, 
so if it fixes the IPv6 issue I’d like to make the move.  Anyone peer with HE 
in general and want to share their experience offlist?  With the price, if 
they’re a good option, I’d consider rolling them in to other locations where we 
have redundancy already, so the v6 isn’t as big a deal there.

Thanks

Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)

[David Hubbard]

On 3/19/19, 8:23 PM, "NANOG on behalf of Ronald F. Guilmette" 
 wrote:


In message 
, 
Tom Beecher  wrote:

>Calling everyone an idiot in the midst of Endless Pontification isn't
>really a recipe for success.

I did not call "everyone" an idiot.  I'm quite completely sure that there
are innumerable people in all of the referenced companies who are consumate
and hardworking professionals who excel at ther jobs.  I do believe however,
based on considerable experience and much hard evidence, that the abuse
handling departnments at OVH and DigitalOcean, and indeed at essentially
-every- sizable hosting company are less than entirely well staffed, less
than entirely well trained, less than entirely well funded, and often
inadequately effective, either due to their limited willingness or their
limited authority, as circumscribed by management, when it comes to the
execution of their assigned duties.  The abuse handling function at *every*
Internet company is the ugly stepchild, ignored whenever possible, and
typically starved of resources by management whose overriding consideration
is this quarter's P statement, and by extension, the nearest upcoming
executive bonus period.


Regards,
rfg


Why not just drop any prefixes from the respective ASN's?  We had to do that 
with OVH after the endless attacks coming from their networks, and lack of 
abuse response.  OVH really loves to shift the abuse around to new prefixes; I 
got tired of spending time staying ahead of it.

Re: Arista Layer3

[David Hubbard]

On 3/5/19, 2:28 PM, "NANOG on behalf of Saku Ytti"  wrote:

Hey Dmitry,

> What do you think about Arista 7280SR (DCS-7280SR-48C6-M-R) as a BGP 
peering router with 3 x upstream with full route view in RIB (ipv4 + ipv6) and 
another IXP feed?
> Considering switching from ASR9001 which is doing perfect work but has no 
more ports left.
> The price is very competitive comparing to MX or ASR and this 
router-switch have 48x10Gig + 6x100GigE ports.

You should compare 7280SR against NCS5500 and PTX1k, not ASR and MX.
ANET is great company, with great people, but they are like 2 years
old in SP market and this is quite visible. It is impressive though
what they've done in so little time.

-- 
  ++ytti

I love the NCS5501, but once Arista gets the 2M-route capacity down into the 
48x10g format, I'd jump ship in a heartbeat; currently you have to do a much 
larger chassis-based device or their 100gig 7280 to have that route scale.  My 
big gripes with the 5501 are that, due to its architecture, if you want to do 
uRPF, you chop your route scale in half, even on the 5501-SE.  5501 also has no 
supported configuration where you have both first hop redundancy and physical 
path redundancy, because you can't do both VRRP (its only redundant first hop 
option) and BVI's, can't do MC-LAG, can't do vPC, so you need switches in 
addition to the 5501's if that's the goal..

David

Any detail on 3356 outage this morning?

[David Hubbard]

Curious if anyone has detail on the cause of the CenturyLink/L3 outage this 
morning?  Their master ticket response is not exactly confidence inspiring; 
hey, routers nationwide decided to reboot, but don’t worry, service was 
restored with no manual intervention….


*** CASCADED EXTERNAL NOTES 05-Feb-2019 14:06:31 GMT From CASE: 15846504 - Event
Event Conclusion Summary

Outage Start: February 05, 2019 11:00 GMT
Outage Stop: February 05, 2019 12:37 GMT

Root Cause: Multiple devices rebooted impacting IP services in multiple markets.
Fix Action: Services restored with no CenturyLink intervention.

Reason for Outage (RFO) Summary: On February 05, 2018 at 11:00 GMT, CenturyLink 
identified a service impact in all markets network wide. The IP NOC reported 
multiple devices rebooted impacting IP services in multiple markets. Services 
restored on their with no CenturyLink intervention. The IP NOC engaged the 
equipment vendor, Tier III Technical Support, and Operations Engineering to 
conduct a post analysis review of the incident.

This service impact has concluded; if additional issues are experienced, please 
contact the CenturyLink Repair Center.

Re: Cellular backup connections

[David Hubbard]

I’ve found the antenna choice and placement can make a huge difference in a 
data center environment.  In some cases it required going to a directional high 
gain antenna pointed towards a desirable tower, which we found by having 
someone monitor / reload the Opengear web interface while another person moved 
the antenna around, to figure out where the best signal strength was produced.

Ours are all Verizon units, but in data centers near some VZ towers, the little 
omnidirectional paddle antennas that come with the Opengear boxes have been 
sufficient, even if the unit is mounted in a rack.  Even with ping times being 
in the 150-300ms range, normally SSH isn’t too bad, but it’s certainly not 
snappy.  I’d say it’s not quite as bad as trying to use SSH via Wifi on a 
Southwest flight, but not as good as a serial console connection.




From: NANOG  on behalf of Dovid Bender 

Date: Friday, December 28, 2018 at 7:08 AM
To: NANOG 
Subject: Cellular backup connections

Hi All,

I finally got around to setting up a cellular backup device in our new POP. I 
am currently testing with T-Mobile where the cell signal strength is at 80%. 
The connection is 4G. When SSH'ing in remotely the connection seems rather 
slow. Ping times seem to be all over the place (for instance now I am seeing: 
rtt min/avg/max/mdev = 174.142/336.792/555.574/99.599 ms) . Is that just 
cellular or is that more related to the provider and the location where I am? I 
could in theory test with VZ and ATT as well. With Verizon they charge $500.00 
just to get a public IP and I want to avoid that if possible.

Thanks and sorry in advance if this is off topic.

Re: Facebook doesn't have a route to my ISP's (Cogeco) IPv6 space?

[David Hubbard]

Yikes, they should change their name rather than be mistaken for Cogent lol

On 12/20/18, 2:04 PM, "Clayton Zekelman"  wrote:


Cogent != Cogeco

Cogent - US Backbone Provider
Cogeco - Canadian Cable TV & Internet provider

At 01:00 PM 20/12/2018, David Hubbard wrote:
>Google and HE don't have IPv6 connectivity with 
>Cogent because Cogent's CEO has been in some 
>decades long pissing match with them about free 
>settlement free peering.  That's the unfortunate 
>reality of the situation; nothing you can do 
>other than have another route to HE/Google IPv6 
>targets.  We have some Cogent circuits that are 
>effectively useless for IPv6 as our customer 
>base has heavy traffic to/from Google cloud 
>services, so they can't be used for a backup / 
>DR scenario; their only real value is an optimal 
>route to other Cogent customers.  I'm slowly 
>replacing our Cogent circuits when feasible 
>because the reality is our customers reaching 
>Google over IPv6 via all our upstreams is more 
>valuable than Cogent's cost savings.
>
>
>
>On 12/20/18, 12:37 PM, "NANOG on behalf of 
>Brian J. Murrell" behalf of br...@interlinx.bc.ca> wrote:
>
> I've been trying to figure out why I can reach an IPv6 address at
> Facebook (2a03:2880:f012:3:face:b00c:0:1) through (only) one of my two
> Internet connections as well as via an HE IPv6 tunnel but not the 
other
> of my two ISP connections
>
> At one point in time a traceroute was dying inside of he.net:
>
>  Host 
> Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. 
> 2001:1970:5261:d600::1  0.0% 
>   72.1   1.3   0.7   2.9   0.8
>  2. 
> 2001:1970:4000:82::10.0% 
>   7   10.0  14.0   8.3  37.9  10.6
>  3. 
> 2001:1970:0:1a6::1 16.7% 
>   7   13.2 215.5  10.8 1031. 455.9
>  4. 
> he.ip6.torontointernetxchange.net   0.0% 
>   7   12.3  12.9  11.2  15.3   1.6
>  5. 
> 100ge9-2.core2.chi1.he.net  0.0% 
>   7   23.6  23.0  21.3  27.6   2.2
>  6. 
> 100ge15-2.core1.chi1.he.net 0.0% 
>   7   21.7  22.5  21.6  24.9   1.2
>  7. 
> 100ge12-1.core1.atl1.he.net 0.0% 
>   7   34.2  35.1  34.1  36.1   0.7
>  8. 
> 100ge5-1.core1.tpa1.he.net  0.0% 
>   7   49.1  46.6  44.8  49.1   1.5
>  9. 
> 100ge12-1.core1.mia1.he.net 0.0% 
>   7   51.6  54.5  50.5  73.3   8.3
> 10. ???
>
> But I think it getting that far time was an anomaly and frankly it
> usually dies even before exiting my ISP's (Cogeco) network like this:
>
>  Host 
> Loss%   Snt   Last   Avg  Best  Wrst StDev
>  1. 
> 2001:1970:5261:d600::1   0.0% 
>   330.6   0.7   0.6   1.0   0.1
>  2. 
> 2001:1970:4000:82::1 0.0% 
>   338.2  10.8   8.1  40.5   5.6
>  3. 
> 2001:1970:0:1a7::1  15.2% 
>   33   23.4  20.1  16.5  23.4   1.5
>  4. 
> 2001:1970:0:61::1   33.3% 
>   33   16.8  17.6  14.5  25.9   2.5
>  5. 
> 2001:1978:1300::10.0% 
>   33   16.0  17.5  14.2  29.6   3.1
>  6. 
> 2001:1978:203::450.0% 
>   33   30.7  30.7  28.4  35.1   1.7
>  7. ???
>
> When I asked the kind folks at he.net for some advice about the 
problem
> (i.e. in the first traceroute above) their diagnosis was that
> Facebook's IPv6 router(s) likely didn't have a route back to my Cogeco
> IPv6 address.
>
> Trying to talk to my ISP (again, Cogeco) has been impossible.  One
> simply cannot reach the people who know more than how to reset your
> router and configure your e-mail.
>
> I wonder how I could go any further with this to confirm the diagnosis
> that Facebook doesn't have a route to the Cogeco network's IPv6 
address
> space given that I only have access to my end of the path.
>
> Cheers,
> b.
>
>

-- 

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409

Re: Facebook doesn't have a route to my ISP's (Cogeco) IPv6 space?

[David Hubbard]

Google and HE don't have IPv6 connectivity with Cogent because Cogent's CEO has 
been in some decades long pissing match with them about free settlement free 
peering.  That's the unfortunate reality of the situation; nothing you can do 
other than have another route to HE/Google IPv6 targets.  We have some Cogent 
circuits that are effectively useless for IPv6 as our customer base has heavy 
traffic to/from Google cloud services, so they can't be used for a backup / DR 
scenario; their only real value is an optimal route to other Cogent customers.  
I'm slowly replacing our Cogent circuits when feasible because the reality is 
our customers reaching Google over IPv6 via all our upstreams is more valuable 
than Cogent's cost savings.

 

On 12/20/18, 12:37 PM, "NANOG on behalf of Brian J. Murrell" 
 wrote:

I've been trying to figure out why I can reach an IPv6 address at
Facebook (2a03:2880:f012:3:face:b00c:0:1) through (only) one of my two
Internet connections as well as via an HE IPv6 tunnel but not the other
of my two ISP connections

At one point in time a traceroute was dying inside of he.net:

 Host  Loss%   Snt   Last   Avg  Best  
Wrst StDev
 1. 2001:1970:5261:d600::1  0.0% 72.1   1.3   0.7   
2.9   0.8
 2. 2001:1970:4000:82::10.0% 7   10.0  14.0   8.3  
37.9  10.6
 3. 2001:1970:0:1a6::1 16.7% 7   13.2 215.5  10.8 
1031. 455.9
 4. he.ip6.torontointernetxchange.net   0.0% 7   12.3  12.9  11.2  
15.3   1.6
 5. 100ge9-2.core2.chi1.he.net  0.0% 7   23.6  23.0  21.3  
27.6   2.2
 6. 100ge15-2.core1.chi1.he.net 0.0% 7   21.7  22.5  21.6  
24.9   1.2
 7. 100ge12-1.core1.atl1.he.net 0.0% 7   34.2  35.1  34.1  
36.1   0.7
 8. 100ge5-1.core1.tpa1.he.net  0.0% 7   49.1  46.6  44.8  
49.1   1.5
 9. 100ge12-1.core1.mia1.he.net 0.0% 7   51.6  54.5  50.5  
73.3   8.3
10. ???

But I think it getting that far time was an anomaly and frankly it
usually dies even before exiting my ISP's (Cogeco) network like this:

 Host   Loss%   Snt   Last   Avg  Best  
Wrst StDev
 1. 2001:1970:5261:d600::1   0.0%330.6   0.7   0.6  
 1.0   0.1
 2. 2001:1970:4000:82::1 0.0%338.2  10.8   8.1  
40.5   5.6
 3. 2001:1970:0:1a7::1  15.2%33   23.4  20.1  16.5  
23.4   1.5
 4. 2001:1970:0:61::1   33.3%33   16.8  17.6  14.5  
25.9   2.5
 5. 2001:1978:1300::10.0%33   16.0  17.5  14.2  
29.6   3.1
 6. 2001:1978:203::450.0%33   30.7  30.7  28.4  
35.1   1.7
 7. ???

When I asked the kind folks at he.net for some advice about the problem
(i.e. in the first traceroute above) their diagnosis was that
Facebook's IPv6 router(s) likely didn't have a route back to my Cogeco
IPv6 address.

Trying to talk to my ISP (again, Cogeco) has been impossible.  One
simply cannot reach the people who know more than how to reset your
router and configure your e-mail.

I wonder how I could go any further with this to confirm the diagnosis
that Facebook doesn't have a route to the Cogeco network's IPv6 address
space given that I only have access to my end of the path.

Cheers,
b.

Re: Cogent charging 50/mo for BGP (not IPs, the service)

[David Hubbard]

They charge it even if you’re using your own address space.  It’s a fee simply 
for establishing BGP with them on a given circuit.  I believe if you used 
static routes and their space, you would not have to pay it.

From: NANOG  on behalf of Josh Luthman 

Date: Wednesday, October 17, 2018 at 12:10 PM
To: Brielle Bruns 
Cc: NANOG list 
Subject: Re: Cogent charging 50/mo for BGP (not IPs, the service)

I view Cogent IP space as a way to lock customers to their service, ie make 
them sticky.
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Wed, Oct 17, 2018, 12:03 PM Brielle Bruns 
mailto:br...@2mbit.com>> wrote:
On 10/17/2018 9:47 AM, Josh Luthman wrote:
> Has anyone else dealt with this mess?  Even my Cogent rep admits it's
> unique to their business.

That sounds like the BS the first company I worked for tried to pull.

One would think they'd welcome customers bringing their own IP space
since it saves them money by not using up precious Cogent IPv4 address
space.

Hell, I even have BGP for v4 and v6 over my CenturyLink biz fiber, and
its available as part of the enhanced package they offer with no extra fees.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Cogent charging 50/mo for BGP (not IPs, the service)

[David Hubbard]

Yep we pay it on our circuits, begrudgingly.  Wouldn’t mind it as much if it 
actually delivered me every BGP prefix in the global routing table…

From: NANOG  on behalf of Josh Luthman 

Date: Wednesday, October 17, 2018 at 11:49 AM
To: NANOG list 
Subject: Cogent charging 50/mo for BGP (not IPs, the service)

Has anyone else dealt with this mess?  Even my Cogent rep admits it's unique to 
their business.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

Re: Whats going on at Cogent

[David Hubbard]

Yeah google is the issue for us.  We provide web services and a LOT of our 
customers have software that is making calls of various types to Google 
services, or even just email delivery to Google hosted email; if all but a 
Cogent transit link to a given data center were down, all of those customers’ 
sites would begin failing at some level because the servers generally try v6 if 
the application level wasn’t explicit.  Cogent doesn’t seem to care since their 
CEO is in some pissing match with Google.  They must be deriving enough revenue 
from last mile v4-only turn ups that they don’t really care about dual stack 
customers.

That being said, can’t say I’ve been impressed with their MPLS / metroE 
offerings either.  When doing the pricing/sizing routine on a project, I 
learned that they have an internal concept of src-dst flows on those types of 
circuits, and if they can’t see your labels, or otherwise hash the traffic, or 
it all truly is point to point, you may not get the full bandwidth, or may need 
to buy a capacity larger than what the flow will be.

From: NANOG  on behalf of DaKnOb 
Date: Tuesday, October 16, 2018 at 10:06 AM
To: Dovid Bender 
Cc: NANOG 
Subject: Re: Whats going on at Cogent

When I call and mention it I’m told that it’s HE’s fault (despite the lovely 
cake), but when I also bring Google, then they tell me to get a different 
provider just for this traffic, or meet them at an IX and send my traffic from 
there.

About the staff rotation I’ve seen it too, and I’ve also seen an increase in 
salespeople calling, for example when an AS is registered etc. in addition to 
the normal calls..

On 16 Oct 2018, at 16:54, Dovid Bender 
mailto:do...@telecurve.com>> wrote:
They call me every few months. the last time they emailed me I said I wasn't 
interested because of the HE issue. I have yet to get another email...


On Tue, Oct 16, 2018 at 9:29 AM, Ca By 
mailto:cb.li...@gmail.com>> wrote:


On Tue, Oct 16, 2018 at 5:16 AM David Hubbard 
mailto:dhubb...@dino.hostasaurus.com>> wrote:
Have had the same sales rep for several years now; unfortunately he has no 
ability to fix their IPv6 peering issue so we’re slowly removing circuits, but 
otherwise for a handful of 10gig DIA circuits it’s been stable.


Yep, this.  Whenever Cogent calls, this is what i tell them. Black-holing HE 
and Google ipv6 traffic, which is what they do if i use a default route from 
them, is dead on arrival.  Shows they make bad decisions and dont put the 
customer first, or even create such an illusion.


From: NANOG mailto:nanog-boun...@nanog.org>> on behalf 
of Ryan Gelobter mailto:rya...@atwgpc.net>>
Date: Tuesday, October 16, 2018 at 6:04 AM
To: NANOG mailto:nanog@nanog.org>>
Subject: Whats going on at Cogent

Anyone else seen terrible support and high turnover of sales/account people at 
Cogent the last few months? Is there something going on over there internally? 
I'm sure some people will say Cogent has always been crap but in the past their 
account reps and support were pretty good. It seems to have gone downhill the 
last 12 months really bad.

Regards,
Ryan

Re: Whats going on at Cogent

[David Hubbard]

Have had the same sales rep for several years now; unfortunately he has no 
ability to fix their IPv6 peering issue so we’re slowly removing circuits, but 
otherwise for a handful of 10gig DIA circuits it’s been stable.

From: NANOG  on behalf of Ryan Gelobter 

Date: Tuesday, October 16, 2018 at 6:04 AM
To: NANOG 
Subject: Whats going on at Cogent

Anyone else seen terrible support and high turnover of sales/account people at 
Cogent the last few months? Is there something going on over there internally? 
I'm sure some people will say Cogent has always been crap but in the past their 
account reps and support were pretty good. It seems to have gone downhill the 
last 12 months really bad.

Regards,
Ryan

Re: ifIndex

[David Hubbard]

I do that too, but I’m referring to XR when you use different speed optics in a 
multi-speed port; if you have a SFP+ port and 10gig SFP, you’ll get one 
ifindex.  New use case requires swapping to a gigE SFP and you’ll get a new 
ifindex.  Take the port out of service, remove the GigE SFP and the related 
config, yet both ifindexes remain; until the device is reloaded.  At that the 
gigE ifindex goes away leaving just the native-speed ifindex.

It’s a pain for management because we’re forced to make exclusions in our NMS 
for ifindex’s that may disappear at some point, because they show as down with 
no way to make that not the case.  Worse, if that port is put to use again at 
the non-native speed, and has such an exclusion in place, we don’t auto learn 
the new usage because of the exclusion.

I tried to argue with TAC that if the gigE SFP has been removed from the SFP+ 
port, and its config has been deleted, the corresponding ifindex and related 
counters should be gone; it no longer exists in any form.  If you reload, it 
will disappear, but that’s the only way.

From: Mel Beckman 
Date: Saturday, October 13, 2018 at 4:46 PM
To: David Hubbard 
Cc: "nanog@nanog.org" 
Subject: Re: ifIndex

David,

All you have to do is turn on IFindex persistence:

https://www.cisco.com/c/en/us/td/docs/routers/crs/software/crs_r4-2/system_management/command/reference/b_sysman_cr42crs/b_sysman_cr42crs_chapter_01101.html#wp2192797756

We do this on our XRs and it works perfectly.

-mel via cell

On Oct 13, 2018, at 9:20 AM, David Hubbard 
mailto:dhubb...@dino.hostasaurus.com>> wrote:
Cisco tries very hard to make such useless data occur in XR.  If you have a 
gigE SFP in an SFP+ port, a new ifindex will appear for the resulting 
GigabitEthernetX port, then it remains even if both the config and SFP have 
been removed.  Automated systems will keep querying it as if it were a downed 
port, but wait, reboot, and suddenly it vanishes.  I went back and forth with 
TAC for weeks explaining that SNMP interfaces should not disappear as a result 
of a reboot, I should either be able to remove it, or it's stuck there forever, 
but a reboot should not cause a change.  They didn't care; it is 'by design'.

On 10/13/18, 8:47 AM, "NANOG on behalf of Mel Beckman" 
mailto:nanog-boun...@nanog.org> on behalf of 
m...@beckman.org<mailto:m...@beckman.org>> wrote:

   Saku,

   The issue isn't that ifindexes change during operation. That would truly 
make SNMP useless. The issue is that they change across reboots. That's where 
features such as Cisco's Interface Index Persistence helps out.

   -mel via cell


On Oct 13, 2018, at 2:59 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 12 Oct 2018 at 21:40, Chris Adams 
mailto:c...@cmadams.net>> wrote:

Is there any good excuse that SNMP client software can't handle a basic
design of SNMP - indexed tables?  ifIndex is far from the only index in
SNMP, and many of them still change today at various times.

It isn't that hard to fetch the indexed field in a bulk get, rewalking
the table if you don't get what you expected.  Cricket did this in 1999.

It's never going to be provably correct, depending on what stability means.

You fetch relation at t0, then at t1 you fetch data. Was the relation
same at t0 and t1? You can gain some confidence by fetching relation
again at t2 and disregard data if t0 != t2. But this becomes polling
expensive quite fast, and still not provably correct. This may be
nitpicking, but I've always felt uneasy about the lack of guarantee.

I wonder if those who have stable indeces, have them for all cases,
all logical interfaces and virtual interfaces?

--
++ytti

Re: ifIndex

[David Hubbard]

Cisco tries very hard to make such useless data occur in XR.  If you have a 
gigE SFP in an SFP+ port, a new ifindex will appear for the resulting 
GigabitEthernetX port, then it remains even if both the config and SFP have 
been removed.  Automated systems will keep querying it as if it were a downed 
port, but wait, reboot, and suddenly it vanishes.  I went back and forth with 
TAC for weeks explaining that SNMP interfaces should not disappear as a result 
of a reboot, I should either be able to remove it, or it's stuck there forever, 
but a reboot should not cause a change.  They didn't care; it is 'by design'. 

On 10/13/18, 8:47 AM, "NANOG on behalf of Mel Beckman" 
 wrote:

Saku,

The issue isn't that ifindexes change during operation. That would truly 
make SNMP useless. The issue is that they change across reboots. That's where 
features such as Cisco's Interface Index Persistence helps out. 

-mel via cell

> On Oct 13, 2018, at 2:59 AM, Saku Ytti  wrote:
> 
>> On Fri, 12 Oct 2018 at 21:40, Chris Adams  wrote:
>> 
>> Is there any good excuse that SNMP client software can't handle a basic
>> design of SNMP - indexed tables?  ifIndex is far from the only index in
>> SNMP, and many of them still change today at various times.
>> 
>> It isn't that hard to fetch the indexed field in a bulk get, rewalking
>> the table if you don't get what you expected.  Cricket did this in 1999.
> 
> It's never going to be provably correct, depending on what stability 
means.
> 
> You fetch relation at t0, then at t1 you fetch data. Was the relation
> same at t0 and t1? You can gain some confidence by fetching relation
> again at t2 and disregard data if t0 != t2. But this becomes polling
> expensive quite fast, and still not provably correct. This may be
> nitpicking, but I've always felt uneasy about the lack of guarantee.
> 
> I wonder if those who have stable indeces, have them for all cases,
> all logical interfaces and virtual interfaces?
> 
> -- 
>  ++ytti

Re: bloomberg on supermicro: sky is falling

[David Hubbard]

They actually profit from fraud; and my theory is that that's why issuers have 
mostly ceased allowing consumers to generate one time use card numbers via 
portal or app, even though they claim it's simply because "you're not 
responsible for fraud."  When a stolen credit card is used, the consumer 
disputes the resulting fraudulent charges.  The dispute makes it to the 
merchant account issuer, who then takes back the money their merchant had 
collected, and generally adds insult to injury by charging the merchant a 
chargeback fee for having to deal with the issue (Amex is notable for not doing 
this).  The fee is often as high as $20, so the merchant loses whatever 
merchandise or service they sold, loses the money, and pays the merchant 
account bank a fee on top of that.

Regarding CVV; PCI permits it being stored 'temporarily', but with specific 
conditions on how that are far more restrictive than the card number.  Suffice 
it to say, it should not be possible for an intrusion to obtain it, and we know 
how that goes

These days javascript being inserted on the payment page of a compromised site, 
to steal the card in real time, is becoming a more common occurrence than 
actually breaching an application or database.  Websites have so much third 
party garbage loaded into them now, analytics, social media, PPC ads, etc. that 
it's nearly impossible to know what should or shouldn't be present, or if a 
given block of JS is sending the submitted card in parallel to some other 
entity.  There's technologies like subresource integrity to ensure the correct 
code is served by a given page, but that doesn't stop someone from replacing 
the page, etc.



On 10/10/18, 10:41 AM, "NANOG on behalf of Naslund, Steve" 
 wrote:

Yet this data gets compromised again and again, and I know for a fact that 
the CVV was compromised in at least four cases I personally am aware of.  As 
long as the processors are getting the money, do you really think they are 
going to kick out someone like Macy's or Home Depot?  After all, it is really 
only an inconvenience to you and neither of them care much about that.

Steve



>It's been a while since I've had to professionally worry about this,
>but as I recall, compliance with PCI [Payment Card Industry] Data
>Security Standards prohibit EVER storing the CVV.  Companies which
>do may find themselves banned from being able to process card
>payments if they're found out (which is unlikely).
>   - Brian

Re: Application or Software to detect or Block unmanaged swicthes

[David Hubbard]

This thread has piqued my curiosity on whether there'd be a way to detect a 
rogue access point, or proxy server with an inside and outside interface?  
Let's just say 802.1x is in place too to make it more interesting.  For 
example, could employee X, who doesn't want their department to be back billed 
for more switch ports, go and get some reasonable wifi router, throw DD-WRT on 
it, and set up 802.1x client auth to the physical network using their 
credentials?  They then let their staff wifi into it and the traffic is NAT'd.  
I'm sure anyone in a university setting has encountered this.  Obviously policy 
can forbid, but any way to detect it other than seeing traffic patterns on a 
port not match historical once the other users have been combined onto it, or 
those other users' ports go down?

David
 

On 6/7/18, 10:18 AM, "NANOG on behalf of Mel Beckman"  wrote:

When we do NIST-CSF audits, we run an SNMP NMS called Intermapper, which 
has a Layer-2 collection feature that identifies the number and MACs of devices 
on any given switch port. We export this list and cull out all the known 
managed switch links. Anything remaining that has more than one MAC per port is 
a potential violation that we can readily inspect. It’s not perfect, because an 
unmanaged switch might only have one device connected, in which case it wont be 
detected. You can also get false positives from hosts running virtualization, 
if the v-kernel generates synthetic MAC addresses. But it’s amazing how many 
times we find unmanaged switches squirreled away under desks or in ceilings.

 -mel 

> On Jun 7, 2018, at 4:54 AM, Jason Hellenthal  
wrote:
> 
> As someone already stated the obvious answers, the slightly more 
difficult route to be getting a count of allowed devices and MAC addresses, 
then moving forward with something like ansible to poll the count of MAC’s on 
any given port ... of number higher than what’s allowed, suspend the port and 
send a notification to the appropriate parties.
> 
> 
> All in all though sounds like a really brash thing to do to your network 
team and will generally know and have a very good reason for doing so... but 
not all situations are created equally so good luck.
> 
> 
> -- 
> 
> The fact that there's a highway to Hell but only a stairway to Heaven 
says a lot about anticipated traffic volume.
> 
>> On Jun 7, 2018, at 03:57, segs  wrote:
>> 
>> Hello All,
>> 
>> Please I have a very interesting scenario that I am on the lookout for a
>> solution for, We have instances where the network team of my company 
bypass
>> controls and processes when adding new switches to the network.
>> 
>> The right parameters that are required to be configured on the switches
>> inorder for the NAC solution deployed to have full visibility into end
>> points that connects to such switches are not usually configured.
>> 
>> This poses a problem for the security team as they dont have visibility
>> into such devices that connect to such switches on the NAC solution, the
>> network guys usually connect the new switches to the trunk port and they
>> have access to all VLANs.
>> 
>> Is there a solution that can detect new or unmanaged switches on the
>> network, and block such devices or if there is a solution that block 
users
>> that connect to unmanaged switches on the network even if those users 
have
>> domain PCs.
>> 
>> Anticipating your speedy response.
>> 
>> Thank You!

Re: Curiosity about AS3356 L3/CenturyLink network resiliency (in general)

[David Hubbard]

Yes, I do, as stated in my initial email.  My inquiry is about whether this 
level of downtime, and lack of redundancy for a given region, is normal for 
3356.  There are some markets where diverse paths are not so easy to acquire.

From: Robert DeVita <radev...@mejeticks.com>
Sent: Saturday, May 19, 2018 5:36:23 PM
To: David Hubbard; nanog@nanog.org
Subject: Re: Curiosity about AS3356 L3/CenturyLink network resiliency (in 
general)

If this is a know issue and has happened before and point to point circuits 
aren’t effected you always have the opportunity to diversify your own network 
and get private lines back to Miami, Jax, Atlanta or Dallas to create your own 
diversity don’t you?

Robert DeVita
Managing Director
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com
_
From: David Hubbard <dhubb...@dino.hostasaurus.com>
Sent: Wednesday, May 16, 2018 12:03 PM
Subject: Curiosity about AS3356 L3/CenturyLink network resiliency (in general)
To: <nanog@nanog.org>


I’m curious if anyone who’s used 3356 for transit has found shortcomings in how 
their peering and redundancy is configured, or what a normal expectation to 
have is. The Tampa Bay market has been completely down for 3356 IP services 
twice so far this year, each for what I’d consider an unacceptable period of 
time (many hours). I’m learning that the entire market is served by just two 
fiber routes, through cities hundreds of miles away in either direction. So, 
basically two fiber cuts, potentially 1000+ miles apart, takes the entire 
region down. The most recent occurrence was a week or so ago when a Miami-area 
cut and an Orange, Texas cut (1287 driving miles apart) took IP services down 
for hours. It did not take point to point circuits to out of market locations 
down, so that suggests they even have the ability to be more redundant and 
simply choose not to.

I feel like it’s not unreasonable to expect more redundancy, or a much smaller 
attack surface given a disgruntled lineman who knows the routes could take an 
entire region down with a planned cut four states apart. Maybe other regions 
are better designed? Or are my expectations unreasonable? I carry three peers 
in that market, so it hasn’t been outage-causing, but I use 3356 in other 
markets too, and have plans for more, but it makes me wonder if I just haven't 
had the pleasure of similar outages elsewhere yet and I should factor that 
expectation into the design. It creates a problem for me in one location where 
I can only get them and Cogent, since Cogent can't be relied on for IPv6 
service, which I need.

Thanks

Curiosity about AS3356 L3/CenturyLink network resiliency (in general)

[David Hubbard]

I’m curious if anyone who’s used 3356 for transit has found shortcomings in how 
their peering and redundancy is configured, or what a normal expectation to 
have is.  The Tampa Bay market has been completely down for 3356 IP services 
twice so far this year, each for what I’d consider an unacceptable period of 
time (many hours).  I’m learning that the entire market is served by just two 
fiber routes, through cities hundreds of miles away in either direction.  So, 
basically two fiber cuts, potentially 1000+ miles apart, takes the entire 
region down.  The most recent occurrence was a week or so ago when a Miami-area 
cut and an Orange, Texas cut (1287 driving miles apart) took IP services down 
for hours.  It did not take point to point circuits to out of market locations 
down, so that suggests they even have the ability to be more redundant and 
simply choose not to.

I feel like it’s not unreasonable to expect more redundancy, or a much smaller 
attack surface given a disgruntled lineman who knows the routes could take an 
entire region down with a planned cut four states apart.  Maybe other regions 
are better designed?  Or are my expectations unreasonable?  I carry three peers 
in that market, so it hasn’t been outage-causing, but I use 3356 in other 
markets too, and have plans for more, but it makes me wonder if I just haven't 
had the pleasure of similar outages elsewhere yet and I should factor that 
expectation into the design.  It creates a problem for me in one location where 
I can only get them and Cogent, since Cogent can't be relied on for IPv6 
service, which I need.

Thanks

Re: IPv4 and IPv6 hijacking by AS 6

[David Hubbard]

Unfortunately, that's how it's done in route policy on XR, so people bouncing 
between flavors can easily make that mistake.


On 4/13/18, 4:15 AM, "NANOG on behalf of Bjørn Mork"  wrote:

Anurag Bhatia  writes:

> Similar for AS2.

I believe we've seen bogus low AS number announcements a few times
before, and they've usually been caused by attemts to configure
AS path prepending without understanding and/or reading the docs.

Someone might have wrongly assumed that

   set as-path prepend 133711 133711

could be written shorter like

   set as-path prepend 133711 2

and there you go...




Bjørn

Re: Cloudflare 1.1.1.1 public DNS different as path info for 1.0.0.1 and 1.1.1.1 london

[David Hubbard]

I'm finding it unreachable from at least one Level 3 router.  I'm seeing 
behavior which makes me suspect 1.1.1.1/32 has been incorrectly defined an 
interface IP on that device; one of our locations gets an immediate ping 
response for 1.1.1.1, and a traceroute of one hop, which is that first upstream 
hop.  1.0.0.1 is reachable like normal across several hops.

On 4/3/18, 1:36 PM, "NANOG on behalf of George Skorup" 
 wrote:

1.1.1.1 not usable via Windstream peering in Chicago.

# traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
...
  3  be4.agr01.chcg02-il.us.windstream.net (40.136.99.22)  5.158 ms 
5.116 ms  7.565 ms
  4  ae13-0.cr01.chcg01-il.us.windstream.net (40.136.99.44)  4.673 ms  
4.644 ms  4.600 ms
  5  et8-0-0-0.cr02.dlls01-tx.us.windstream.net (40.128.10.135) 27.136 
ms  27.099 ms  27.053 ms
  6  xe0-2-3-0.cr02.dnvt01-co.us.windstream.net (40.136.97.125) 29.075 
ms  28.381 ms  28.336 ms
  7  xe3-3-1-0.pe03.dums01-tx.us.windstream.net (173.189.57.195) 46.121 
ms  46.193 ms  46.148 ms
  8  * * *
  9  * * *
10  * * *
11  * * *
12  * * *
13  *^C

# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=248 time=43.2 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=248 time=43.9 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=248 time=42.8 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 42.892/43.344/43.915/0.489 ms

# nslookup
 > server 1.1.1.1
Default server: 1.1.1.1
Address: 1.1.1.1#53
 > google.com
;; connection timed out; trying next origin
;; connection timed out; no servers could be reached

Re: NG Firewalls & IPv6

[David Hubbard]

I’ve been doing dual stack through Fortinet products for many years without 
issue.  Well, no issue from a technical perspective.  Sometimes you have to dig 
for a bit to find the equivalent v6 CLI commands, and occasionally there’s GUI 
stuff missing that requires CLI where the v4 equivalent didn’t, but not a bad 
experience overall.  Does v6 vpn’s great too.  Haven’t delved into dynamic 
routing protocols on them so can’t speak to that.  Happy to answer questions.

David

From: NANOG  on behalf of Joe Klein 
Sent: Monday, April 2, 2018 6:58:14 PM
To: NANOG list
Subject: NG Firewalls & IPv6

All,

At security and network tradeshows over the last 15 years, I have asked
companies if their products supported "IPv6". They all claimed they did,
but were unable to verify any successful installations. Later they told me
it was on their "Roadmap" but were unable to provide an estimated year,
because it was a trade secret.

Starting this last year at BlackHat US, I again visited every product
booth, asking if their products supported dual-stack or IPv6 only
operations. Receiving only the same unsupported answers, I decided to focus
on one product category.

To the gurus of the NANOG community, What are your experiences with
installing and managing Next Generations firewalls? Do they support IPv6
only environments? Details? Stories?

If you prefer not to disparage those poor product companies, please contact
me off the list.

Thanks,

Joe Klein

"inveniet viam, aut faciet" --- Seneca's Hercules Furens (Act II, Scene 1)
PGP Fingerprint: 295E 2691 F377 C87D 2841 00C1 4174 FEDF 8ECF 0CC8

Re: Console Servers & Cellular Providers

[David Hubbard]

We get static IP's to facilitate monitoring that the OOB remains online (easier 
to hit a non-changing IP than getting false positives for outage between an IP 
change and DDnS or whatever other type of update needs to happen), and it also 
makes IPSec VPN easy if your roving sysadmins know what IP to VPN into for a 
given site, when DNS may or may not be working.


On 2/7/18, 12:49 PM, "NANOG on behalf of Chris Marget" 
 wrote:

Lots of references to static IPs from cellular providers for OoB access in
this thread. Why? It seems like a dial-home scheme is an obvious solution
here, whether it's Opengear's Lighthouse product, openvpn, or whatever...

Do you all have a security directive that demands whitelisted IP addresses?

I've got a handful of OoB systems that dial home via cellular, but only
after they've been poked by SMS. Opengear's auto-response facilitates that,
and I've done it with EEM (to start DMVPN) on Cisco ISRs.

The main headache I've run into is that it's tough to get a SIM card from
ATT that does data and SMS: ATT's M2M plans don't allow SMS, and moving the
SIM from an iPhone to "a computer" causes the SMS capability to vanish. My
ATT OoB boxes (used only where Verizon is reported to not work) are online
all the time.

Re: Console Servers & Cellular Providers

[David Hubbard]

Going to depend entirely on the data center.  I've got OpenGear boxes deployed 
in a variety of places, using Verizon LTE with static IP.  One Level 3 colo I'm 
in I had to buy a high gain directional antenna to get the signal strength up 
above -80, where below that you're lucky to get a reasonable SSH experience, 
but then I'm in a Switch colo in Vegas that has dramatically more customers and 
equipment,  and I get almost double that signal strength, inside a rack, inside 
a metal heat chamber, with the built-in antennas.  Just depends on the 
structure and proximity to a tower I'm guessing.

On 2/7/18, 11:39 AM, "NANOG on behalf of James Milko"  wrote:

 How is cell reception in multi-story data centers/carrier hotels?  Good
enough for remote management?


JM

Re: Temp at Level 3 data centers

[David Hubbard]

Thanks for all the opinions and experiences with this on this on and off list.  
The facility in question is not one that has a cold/hot row or containment 
concept so ambient temp plays a greater role than in other facilities.  Some 
folks from Level 3 reached out and are working to help me with the situation, 
so hopefully things are headed in the right direction now.

David

From: Martin Hannigan <hanni...@gmail.com>
Date: Friday, October 13, 2017 at 4:05 PM
To: David Hubbard <dhubb...@dino.hostasaurus.com>
Cc: "nanog@nanog.org" <nanog@nanog.org>
Subject: Re: Temp at Level 3 data centers



Hi David,

80F seems ~reasonable to me. What is the inlet temp, the temperature air is 
going in at? What kind of gear are operating? Routers and switches? Servers? 
Disk? Is the cabinet top fan working? Most modern equipment should be able to 
handle those temps. As another poster noted, are these triggers modifiable (or 
have they been)? I would refer to the manufactures guidelines. You haven't 
given us enough information to help. You can refer (them) to ASHRAE standards 
in your conversation. I'd be surprised if they weren't already well aware of it 
and practicing most of what it preaches. They may operate safely outside of 
some norms.

15F-20F cooler? You might be paying too much for colo if that's true.

Best,

-M<



On Wed, Oct 11, 2017 at 8:31 AM, David Hubbard 
<dhubb...@dino.hostasaurus.com<mailto:dhubb...@dino.hostasaurus.com>> wrote:
Curious if anyone on here colo’s equipment at a Level 3 facility and has found 
the temperature unacceptably warm?  I’m having that experience currently, where 
ambient temp is in the 80’s, but they tell me that’s perfectly fine because 
vented tiles have been placed in front of all equipment racks.  My equipment is 
alarming for high temps, so obviously not fine.  Trying to find my way up to 
whomever I can complain to that’s in a position to do something about it but it 
seems the support staff have been told to brush questions about temp off as 
much as possible.  Was wondering if this is a country-wide thing for them or 
unique to the data center I have equipment in.  I have equipment in several 
others from different companies and most are probably 15-20 degrees cooler.

Thanks,

David

Temp at Level 3 data centers

[David Hubbard]

Curious if anyone on here colo’s equipment at a Level 3 facility and has found 
the temperature unacceptably warm?  I’m having that experience currently, where 
ambient temp is in the 80’s, but they tell me that’s perfectly fine because 
vented tiles have been placed in front of all equipment racks.  My equipment is 
alarming for high temps, so obviously not fine.  Trying to find my way up to 
whomever I can complain to that’s in a position to do something about it but it 
seems the support staff have been told to brush questions about temp off as 
much as possible.  Was wondering if this is a country-wide thing for them or 
unique to the data center I have equipment in.  I have equipment in several 
others from different companies and most are probably 15-20 degrees cooler.

Thanks,

David

Re: Has Level3 done away with traceroute??

[David Hubbard]

I’m seeing the same thing across our Level 3 circuits, even if the traffic 
never leaves their network.  This was not the case as recently as two days ago 
when I had a reachability ticket open with them.  My SE says they’re having an 
internal issue that’s being worked on; didn’t provide detail.

On 9/21/17, 1:13 PM, "NANOG on behalf of Van Dyk, Donovan via NANOG" 
 wrote:

Hello All,

Recently I was troubleshooting a network event for a client of our who 
resides on the Level3 network. While trying to verify the path, I noticed I am 
no longer able to traceroute through the Level3 network.
The funny thing is this is not just isolated to the /32. It appears to be 
that the entire 4.0.0.0/9 network is no longer able to traceroute through. 
Everything dies on their edge network.

This appears to be isolated to traceroute. I have check this in NA and EU.

My carrier contacted Level3 who pretty much stated that they can’t provide 
anything.

I have checked multiple looking glasses and other online tools and none of 
them make it. Even Level3 looking glass drops the packets.

Does anyone know anything about this? I’m pretty sure this is the first 
time we are seeing this.


Random 4.0.0.0/9 address.

NTT looking glass
Tracing the route to 4.35.230.7

1   *
ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec  1 msec
 2   *  *  *
 3   *  *  *
 4   *  *  *
 5   *  *  *
 6   *  *  *


TATA looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Telia looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Level3 looking glass
Traceroute results from Atlanta, GA to 
4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)

  1  0.0.0.0  * * *
  2  0.0.0.0  * * *
  3  0.0.0.0  * * *
  4  0.0.0.0  * * *
  5  0.0.0.0  * * *
  6  0.0.0.0  * * *
  7  0.0.0.0  * * *
  8  0.0.0.0  * * *
  9  0.0.0.0  * * *
 10  0.0.0.0  * * *
 11  0.0.0.0  * * *

--
Donovan Van Dyk
SOC Network Engineer
Fort Lauderdale, FL USA

[cid:image001.png@01D332DA.F4DD00A0]
The information contained in this electronic mail transmission and its 
attachments may be privileged and confidential and protected from disclosure. 
If the reader of this message is not the intended recipient (or an individual 
responsible for delivery of the message to such person), you are strictly 
prohibited from copying, disseminating or distributing this communication. If 
you have received this communication in error, please notify the sender 
immediately and destroy all electronic, paper or other versions.

Anyone using Arista 7280R as edge router?

[David Hubbard]

Hey all, have some Brocade MLXe’s that can no longer handle a full v4 and v6 
route table while also having VRF support (dumb CAM profile limitations in the 
software).  Mine don’t do anything fancy; just BGP to a few upstream peers and 
OSPF/OSPFv3 to the inside, management VRF, some ACL’s.  I’m looking at the 
ASR9001 with add-on ports since I need (10) 10gig.  However, I’ve also been 
running some Arista 7280SE’s for the past 18 months with no issues, and they 
want me to consider their 7280R since it would give me more ports, in addition 
to some higher speed ports, which would be nice if I ever want to upgrade some 
of our peering to 40 or 100gig.

Arista’s specs say the 7500R / 7280R can handle 1M ipv4+ipv6 routes in hardware 
(FIB):

https://www.arista.com/assets/data/pdf/Whitepapers/FlexRoute-WP.pdf

In theory, it would last at least a few years if the v4 table doesn’t get too 
crazy between now and then.

Curious if anyone has deployed a 7500R or 7280R in this role and what the 
feedback has been?

The 9001’s 4M ‘credits’ for the combo of v4 +(2)v6 routes obviously goes much 
further, but I think either one would make it to their expected end of life, or 
if not on the Arista side, I’d probably have spent half as much.

Thanks,

David

Re: 10G MetroE 1-2U Switch

[David Hubbard]

Would Arista 7280R work?  Gets you 48+ 10gig and a couple QSFP ports even in 
the cheapest model.  I believe it has the features you want.  Haven’t done MPLS 
with it, but I’ve got some running OSPF/OSPFv3 with no issues.

On 4/13/17, 5:37 PM, "NANOG on behalf of Erik Sundberg" 
 wrote:

Hey Nanog,

Looking for a new metroE Edge switch that has more that 10x 10G ports. I am 
having a hard time finding anything worthwhile without buying a full blown 
ASR9K Chassis or another vendor's chassis.

Requirements
MEF compliant
1-2U small foot print
10G Ports will be used for ENNI's and UNI Ports
Prefer MPLS support for L2VPN's (EoMPLS and VPLS)
QOS per Sub interface\vlan on a ENNI
Cost effect 10G Ports
100G Not required


Looking at the
ASR920's - Great box for 1G but not enough 10G Ports Only 4
NCS5001/NCS5501 - New\unproven\probably buggy, Lacking some features & QOS 
issues :/
ASR900 - Looks good, but was hoping for a smaller foot print. If I remember 
right the 8x10G Cards can't go in every slot.

Any other platforms I should be looking at?

Ciena, Brocade, Juniper?



Thanks in advance!

-Erik



CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files 
or previous e-mail messages attached to it may contain confidential information 
that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby 
notified that any disclosure, copying, distribution or use of any of the 
information contained in or attached to this transmission is STRICTLY 
PROHIBITED. If you have received this transmission in error please notify the 
sender immediately by replying to this e-mail. You must destroy the original 
transmission and its attachments without reading or saving in any manner. Thank 
you.

Alternatives to bgpmon?

[David Hubbard]

Anyone have recommendations for an alternative service that works like bgpmon 
(external reachability/peer monitoring, route hijack alerts, etc)?  Since their 
OpenDNS acquisition, I’ve found the service not working reliably, as in I 
receive no alerts even when I’m intentionally taking one of our peers offline, 
and after two attempts to find out why this is, I receive no response, so it 
seems support is now broken as well.

Thanks,

David

Verizon wireless to stop issuing static IPv4

[David Hubbard]

Thought the list would find this interesting.  Just received an email from VZ 
wireless that they’re going to stop selling static IPv4 for wireless 
subscribers in June.  That should make for some interesting support calls on 
the broadband/fios side; one half of the company is forcing ipv6, the other 
can’t provide it.  At least now we have a big name forcing the issue though.

David

Here’s complete text:

On June 30, 2017, Verizon will stop issuing new Public Static IPv4 addresses 
due to a shortage of available addresses. Customers that currently have active 
Public Static IPv4 addresses will retain those addresses, and Verizon will 
continue to fully support existing Public Static IPv4 addresses. In order to 
reserve new IP addresses, your company will need to convert to the Persistent 
Prefix IPv6 requirements and implement new Verizon-certified IPv6 devices.





Why should you make the move to Persistent Prefix IPv6?





•

Unlike IPv4, which is limited to a 32-bit prefix, Persistent Prefix IPv6 has 
128-bit addressing scheme, which aligns to current international agreements and 
standards.



•

Persistent Prefix IPv6 will provide the device with an IP address unique to 
that device that will remain with that device until the address is relinquished 
by the user (i.e., when the user moves the device off the Verizon Wireless 
network).



•

IPv4-only devices are not compatible with Persistent Prefix IPv6 addresses.

Re: Common Reliable Out Of Band Management Options at Carrier Hotels

[David Hubbard]

Provided you can get a cell signal, we’ve been very happy with Opengear boxes.  
We’d been using their ACM5508 which is eight serial ports, two Ethernet, cell.  
It runs linux, you can ssh into it, do fancy things like keep the cell side 
down and use text messages to bring it up if you need to get in, does VPN, 
PPTP, monitors environmental things if needed, etc.  They replaced that model 
with the 7004 and 7008 (4 or 8 serial).  They have console servers if you need 
more ports; we have a 32-port daisy chained to a 5508 in a location we had 
serial growth, but their 7200-series is cell plus high density serial in one.

In a data center with particularly bad cell reception, Opengear recommended 
getting a high gain antenna from wpsantennas.com.  I contacted them and the 
recommendation for my specific use case was a Panorama WMMG-7-27.  We had it 
mounted above the overhead infrastructure on top of our cage and it 
dramatically improved the signal to make it a non-issue.

David

On 1/17/17, 4:59 PM, "NANOG on behalf of Darin Herteen" 
 wrote:

Greetings list,


We are exploring standardizing our Out Of Band options across our network 
and various off-net locations and the question was brought up "What about 
carrier hotels? What constraints might present themselves at those locations?"


Assuming each hotel we are located in can provide either Ethernet or DSL 
I'm guessing that is going to come a cost (cross-connects, rack space etc..) 
that might end up being cost prohibitive.


So my inquiry is... What does the list find to be a reasonably priced yet 
reliable solution in carrier hotels for OOB? Or is that contradictory :)


Thoughts on Cellular?


Any experience/insight would be appreciated.


Thanks,


Darin

Re: SoCal FIOS outage(?) / static IP readdressing

[David Hubbard]

Last 18 hour outage I experienced got me a fantastic half month credit.  It 
cost us more to pay me for the time I spent on hold than the credit was worth, 
so I no longer call them if we’re down and downdetector shows others in the 
area are too.  We’re in the process of moving the circuit to a backup role, but 
it’s proving to be a long process getting fiber run to an alternative.

David 

On 1/4/17, 9:48 AM, "nanog-boun...@nanog.org on behalf of 
valdis.kletni...@vt.edu"  wrote:

On Wed, 04 Jan 2017 00:28:57 -0800, "Paul B. Henson" said:

> I'm about at the point where next time it goes down and it appears to be
> a remote issue I'm not going to bother to call it in; I'll just cross my
> fingers and hope it fixes itself within a day or so and only report it
> if it doesn't. I don't think my calls today did anything but waste my
> time.

Even if nothing else happens, calling in and reporting the problem *does*
(or at least it *should*) set the clock running for any SLA-related 
compensation.

Re: OSPFv3 with IPSec between Cisco and Juniper gears

[David Hubbard]

Wouldn’t you want to use hexadecimal instead of ascii-text, since that would 
match what the Cisco is asking for?  I’m just throwing this out there, I’m not 
familiar with Juniper but their docs seem to suggest that using hex will cause 
it to ask for 40 hex chars.

David

On 11/10/16, 3:14 PM, "NANOG on behalf of Philippe Bonvin via NANOG" 
 wrote:

Hello folks,


Quick question about incompatibility between Cisco and Juniper gears.


Without IPSec, OSPFv3 is working as expected.

I'm trying to configure IPSec authentification of OSPFv3 between a Juniper 
SRX and a Cisco router but it seems that they didn't agree to a common key 
length.


Can you confirm that this is a well-known problem or give me the right 
configuration that I should use ?


The error message on the juniper:

[edit security ipsec security-association ospfv3 manual direction 
bidirectional authentication key ascii-text]
  'ascii-text "..."'
Authentication key size must be 20 bytes

On the cisco side:

cisco(config-if)#ipv6 ospf authentication ipsec spi 256 sha1 0 ?
  Hex-string  SHA-1 key (40 chars)?



Here is an output of the config I'm using on the SRX side:



ipsec {
security-association ospfv3 {
mode transport;
manual {
direction bidirectional {
protocol ah;
spi 256;
authentication {
algorithm hmac-sha1-96;
key ascii-text "..."; ## SECRET-DATA
}
}
}
}
}

interface ge-0/0/0.0 {
ipsec-sa ospfv3;
}


Thanks for your help,
Philippe


[EDSI-Tech Sarl]
Philippe Bonvin, Directeur
EDSI-Tech Sàrl
EPFL Innovation Park, Batiment C, 1015 Lausanne, Suisse | Téléphone: +41 
(0) 21 566 14 15, ext. 99
Savoie Technolac, 17 Avenue du Lac Léman, 73375 Le Bourget-du-Lac, France | 
Téléphone: +33 (0)4 86 15 44 78, ext. 99

Disclaimer:
This email is confidential and intended solely for the use of the 
individual to whom it is addressed. If you are not the intended recipient of 
this information, be advised that you have received this email in error and 
that any usage, disclosure, distribution, copying of the information or any 
part of it in any form whatsoever is strictly prohibited.
If you have received this email in error please notify the EDSI-Tech 
helpdesk by phone on +41 21 566 14 15 and then delete this e-mail.

Re: Dyn DDoS this AM?

[David Hubbard]

Do we know the attack destinations so we can watch transit traffic destined for 
it to help sources that may be unaware?

David

Level 3 voice outage?

[David Hubbard]

Curious if anyone else is having issues with Level 3 (legacy Twtelecom 
specifically) enterprise SIP?  I’m at minute 45 of being on hold with them, so 
I suspect they are having known issues.  Our sales rep mentioned a toll free 
outage being tracked under master ticket 11377637 but I don’t have the details 
of that yet.

We’re seeing our toll free numbers completely down, since what I believe to be 
4a EST time frame.  Most of our toll numbers have an unusual 25 second delay 
before we get any SIP traffic from their equipment, but the call does 
ultimately connect.

David

Re: Arista unqualified SFP

[David Hubbard]

We’ve done this as well, and Arista support hasn’t hassled us about anything 
yet so I’ve been pleased.  I’ve been very happy using Flexoptics transceivers 
in all kinds of equipment too, if anyone’s looking for something they know 
works, and you get a programmer that will let you code optics to certain 
vendors switches that don’t have unlock keys.  It won’t work on all though, so 
investigate before investing if that’s a concern.

David

On 8/17/16, 4:33 PM, "NANOG on behalf of Ryan DiRocco" 
 wrote:

Exactly this, get your unlock key that is tied to your company and you are 
off to the races, bake it into your standard config.  Your SE or support team 
should be able to get this to you :)

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ryan, Spencer
Sent: Wednesday, August 17, 2016 3:53 PM
To: Stanislaw; nanog@nanog.org
Subject: Re: Arista unqualified SFP

Yes, email support and ask for the unlock code, they will make you agree 
that you know that 3rd party optics may explode the switch and it's not their 
fault.


The command they give you will have a key/hash built into it (but will work 
on any switch) that ties the "unlock" to your org.


Ours looks like this:


service unsupported-transceiver DescriptionOfKeyFromAristaGoesHere 
00 (hex key)


Spencer Ryan | Senior Systems Administrator | 
sr...@arbor.net
Arbor Networks
+1.734.794.5033 (d) | +1.734.846.2053 (m)
www.arbornetworks.com



From: NANOG  on behalf of Stanislaw 
Sent: Wednesday, August 17, 2016 3:50:12 PM
To: nanog@nanog.org
Subject: Arista unqualified SFP

Hi all,
Is there a way for unlocking off-brand transceivers usage on Arista 
switches?

I've got an Arista 7050QX switch with 4.14 EOS version. Then it has been 
found out that Arista switches seem to not have possibility to unlock off-brand 
xcievers usage (by some service command or so).

I've patched /usr/lib/python2.7/site-packages/XcvrAgent.py, made the 
checking function bypass the actual check and it helped: ports are not in 
errdisable state anymore. But despite of xceivers are detected correctly, links 
aren't coming up (they are in notconnect state).

If anyone possibly have does have a sacred knowledge of bringing 
off-branded transceivers to life on Arista switches, your help'd be very 
appreciated. Thanks.

Re: Email to text - vtext.com blacklisting ip

[David Hubbard]

We’d experienced similar, plus, email to text doesn’t work if the path between 
alerting system and email gateway is broken.

We bought a few of these cellular gateways:  http://www.smseagle.eu/

Then I went into a t-mobile store and bought a few $25/mo SIM cards, put credit 
card on file to auto renew each month, slapped them in, and pointed our NMS’s 
at them.  Now we can send SMS alerts from each facility and have had no 
reliability issues.  There’s an easy to write for http interface, and many 
common things, like Zabbix or Nagios, already have modules written.

David


On 8/16/16, 7:33 PM, "NANOG on behalf of Sam Norris"  wrote:

Same boat...  We are sending messages to phonenum...@vtext.com and getting
bouncebacks or lost items.  I assume its because some limits are now being 
put
into place.  We are a Verizon subscriber so I am paying, it is not a free
service.  But  I am totally up for paid services if you can recommend 
some
that will reliably get us texts to our verizon phones.

Sam


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ryan, Spencer
> Sent: Tuesday, August 16, 2016 4:17 PM
> To: Josh Luthman; Mike
> Cc: NANOG list
> Subject: RE: Email to text - vtext.com blacklisting ip
> 
> I agree. Pay Pager duty or a SMS gateway with a SLA. Relying on  the free
service
> for anything critical is asking for trouble.
> 
> 
> 
> Sent from my Verizon, Samsung Galaxy smartphone
> 
> 
>  Original message 
> From: Josh Luthman 
> Date: 8/16/16 6:09 PM (GMT-05:00)
> To: Mike 
> Cc: NANOG list 
> Subject: Re: Email to text - vtext.com blacklisting ip
> 
> If it's critical I'd suggest a service than can depended on...
> 
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Aug 16, 2016 5:45 PM, "Mike"  wrote:
> 
> > Hi,
> >
> >
> > I have a server that monitors my network and issues text messages if
> > there are events of note that require human intervention. There is some
> > process filtering that ensures it also is not able to issue more than 1
> > alert maximum per 5 minutes, to ensure it doesn't flood pagers with
> > messages all screaming the sky is falling when things are not going 
well.
> > Recently however, this server is no longer able to deliver messages to
> > vtext.com - it gets nothing but 554 errors:
> >
> >
> > telnet 69.78.67.53 25
> > Trying 69.78.67.53...
> > Connected to 69.78.67.53.
> > Escape character is '^]'.
> > 554 txslspamp10.vtext.com
> > Connection closed by foreign host.
> >
> > Granted on some days during challenging times it can send 30 or 40
> > messages before we get to it and get it squelched / silenced, but it's
> > otherwise reasonably well behaved IMHO and I don't think we are any 
heavy
> > volume sender. So I am trying to figure out why it's blacklisted then 
and
> > am rolling snake eyes.  If anyone who is an admin for verizon or who has
> > any insight to share I'd certainly appreciate it. Email to text is a
> > critical function we depend on.
> >
> >
> > Thank you.
> >
> >
> >

Level3 (3356) to outlook.office365.com via v6?

[David Hubbard]

Curious if anyone else is having issues reaching outlook.office365.com via ipv6 
over Level 3?  Our customers have begun reporting failures checking email, and 
in the ones who have had this issue, are using the mail server name 
outlook.office365.com and are on v6.  Traceroute6 shows the traffic dying 
shortly into Level 3 land at 2001:1900:4:1::3d1 which is likely a Tampa-area 
router.

Thanks,

David

Re: Operations task management software?

[David Hubbard]

Full automation is planned but does not eliminate the need for the software.  
Zero human auditing of fully automated processes and data collection are not 
acceptable to various certifying entities, the relevant auditors, the 
inevitably involved lawyers, and won’t pick up on bad data, like a bad 
thermometer or snmp counter that says a CRAC is 65 degrees when it’s really 90. 
 So I’m still going to need a management solution to the issue whether it’s to 
tell someone to do the work or to tell someone to check the automated work.

David

On 7/27/16, 7:19 PM, "Lee" <ler...@gmail.com> wrote:

On 7/27/16, David Hubbard <dhubb...@dino.hostasaurus.com> wrote:
> Hi all, curious if anyone has recommendations on software that helps 
manage
> routine duties assigned to operations staff?

Have computers do the routine scut work - not people.

> For example, let’s say we have a P that says someone from the netops 
group
> must check that Rancid is successfully backing up all router configs
> bi-weekly.

You've got the source code for rancid, so change rancid-run to do something 
like
  LOGFILE=$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S`; export LOGFILE
change the
  ) >$LOGDIR/$GROUP.`date +%Y%m%d.%H%M%S` 2>&1
to
  ) >$LOGFILE 2>&1

and then in control_rancid do something like
  grep "clogin error:" $LOGFILE | sort | uniq -c >$TMP.fail
  if [ -s $TMP.fail ]; then
 # got some output, mail the report
 ...

Do the same type thing for checking on
> backup failures, backup internet circuit status, out of band interfaces, 
etc.

Automate the checks, put the scripts in crontab & mail out an
"OhNoes!" or "all clear" msg at the end.   At which point you're left
with the problem of making sure the managers are looking at the emails
& making sure whatever problems are found actually get fixed :)

Regards,
Lee
 

Operations task management software?

[David Hubbard]

Hi all, curious if anyone has recommendations on software that helps manage 
routine duties assigned to operations staff?

For example, let’s say we have a P that says someone from the netops group 
must check that Rancid is successfully backing up all router configs bi-weekly. 
 Ideally, it would send an email reminder to this pre-defined group of people 
saying hey, it’s Monday, someone needs to check this and come acknowledge the 
task as having been completed.  If that doesn’t occur, pre-defined manager X is 
notified on Tuesday.  If manager X doesn’t get someone to complete the task, 
director Y is notified, so on and so forth.  Then, perhaps periodically it 
emails manager X anyway and says hey, it’s been three months, you need to audit 
netops to ensure they’re actually doing the Rancid audit and not just checking 
that it was done.  This could be applied to the staff who check on backup 
failures, backup internet circuit status, out of band interfaces, etc.

A data center I looked at recently had QR code stickers on all of their 
infrastructure stuff and there were staff assigned to check and log certain 
displayed values each day.  The software would at least ensure they actually 
visited the equipment by requiring they scan the relevant QR code when in front 
of it.  So I figure something that does what I’m looking for properly already 
exists.

Thanks,

David

Re: syslog server

[David Hubbard]

https://www.graylog.org/

On 6/6/16, 4:59 PM, "NANOG on behalf of Maximino Velazquez" 
 wrote:

>Hi nanog community
>
>I need help !!
>
>What is the best syslog server  (opensource)?
>
>Thanks for your help
>
>Regards.
>
>-- 
>
>
>
>Max Velazquez |

Re: Level 3 issues?

[David Hubbard]

I just heard from someone there is suspicion that a fiber cut occurred in FL, 
possibly Miami area, and it has revealed a capacity issue on the L3 network.  
Haven’t received official word on that yet, but I know our legacy TWTC 
connection is nearly as useless as our L3 connection thanks to the network 
merging activities.

David




On 5/16/16, 4:10 PM, "Jordan Medlen" <jordan-med...@bisk.com> wrote:

>Have been seeing issues since just after 3P. Had to swing my traffic over to 
>another provider. Level3 says issues seen from Costa Rica on up to WDC.
>
>
>Thank you,
>
>Jordan Medlen
>Enterprise Communications Manager
>Bisk Education
>(813) 612-6207
> 
> <http://www.bisk.com/>
>
>On 5/16/16, 3:49 PM, "NANOG on behalf of David Hubbard" 
><nanog-boun...@nanog.org on behalf of dhubb...@dino.hostasaurus.com> wrote:
>
>>Anyone seeing issues with Level 3 networking right now?  We’re seeing huge 
>>latency and loss on traffic coming inbound (to us, AS33260) but it seems to 
>>be at the peering points with other major ISP’s and Level 3.  Comcast for 
>>example:
>>
>>  333 ms21 ms70 ms  te-3-5-ur01.hershey.pa.pitt.comcast.net 
>> [68.85.42.29]
>>  4 *   33 ms   106 ms  162.151.48.173
>>  5   214 ms54 ms41 ms  162.151.21.229
>>  6   561 ms   764 ms   459 ms  4.68.71.133
>>
>>Thanks,
>>
>>David
>

Level 3 issues?

[David Hubbard]

Anyone seeing issues with Level 3 networking right now?  We’re seeing huge 
latency and loss on traffic coming inbound (to us, AS33260) but it seems to be 
at the peering points with other major ISP’s and Level 3.  Comcast for example:

  333 ms21 ms70 ms  te-3-5-ur01.hershey.pa.pitt.comcast.net 
[68.85.42.29]
  4 *   33 ms   106 ms  162.151.48.173
  5   214 ms54 ms41 ms  162.151.21.229
  6   561 ms   764 ms   459 ms  4.68.71.133

Thanks,

David

Re: NIST NTP servers

[David Hubbard]

Ed, and anyone else reading this thread, I’m curious if you’ve looked at their 
authenticated NTP offering which uses different servers:

http://www.nist.gov/pml/div688/grp40/auth-ntp.cfm


We’re considering that but haven’t tried yet.

David




On 5/9/16, 11:01 PM, "NANOG on behalf of b f"  wrote:

>Hello List,
>
>
>In search of stable, disparate stratum 1 NTP sources.
>
>Looking for anyone’s advice/experiences (good/bad/ugly/weird) using NIST’s
>NTP servers per: http://tf.nist.gov/tf-cgi/servers.cgi
>
>We tried using “time.nist.gov” which returns varying round-robin addresses
>(as the link says), but Cisco IOS resolved the FQDN and embedded the
>numeric address in the “ntp server” config statement.
>
>
>
>After letting the new server config go through a few days of update cycles,
>the drift, offset and reachability stats are not anywhere as good as what
>the stats for the Navy time server are - 192.5.41.41 / tock.usno.navy.mil.
>
>
>I would greatly appreciate and feedback / advice, etc.
>
>
>Thanks!!!
>
>
>Ed

Re: Fri AM AT outage

[David Hubbard]

Hopefully the job posting includes replacing this guy:



http://www.tampabay.com/news/business/frontier-communications-pledges-smooth-take-over-of-verizon-fios-and-land/2271256


"I would never say we're 100 percent certain it will go perfectly," Mike Flynn, 
Frontier's regional president overseeing operations in Florida and the 
Carolinas. "But we're doing everything we can within our power … from the 
experience we've gleaned from every conversion we have done to make the next 
one better. So I'd just say we're pretty experienced at it."
He said customers may experience brief service interruptions in the early 
morning hours Friday, though the company is not expecting that to affect a 
significant numbers of customers.



Our Tampa office has been offline for nine hours and Frontier support is saying 
not to worry, it should be resolved in the next four days.

David


On 4/1/16, 4:42 PM, "NANOG on behalf of Jay R. Ashworth" 
 wrote:

>I heard speculation from many quarters that this may have been related to
>the Verizon splashcut to Frontier -- which, regardless of what Frontier
>was telling us, I was pretty sure would be more than just varying which light
>switch for a building sign was the one turned on.
>
>Has anyone heard anything they're allowed to repeat, yet, which confirms
>or denies?
>
>On a related[1] topic: I see on Craigs that Frontier is hiring through a sub
>for transport staff for their new NOC, which I think is going to be located
>in SPBGFLXA89H; that building has about 3 completely empty floors, and has 
>for over 20 years.  (OK, it did when I toured it in 91; dunno what's there
>now.  :-)
>
>Cheers,
>-- jra
>[1]maybe
>
>-- 
>Jay R. Ashworth  Baylink   j...@baylink.com
>Designer The Things I Think   RFC 2100
>Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
>St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

10gig pricing with Verizon crazy?

[David Hubbard]

Curious if anyone has had similar experience; looking for a 10gig transit 
circuit at a colo, contacted VZ as they’re on net in the facility, quoted me an 
astronomical amount at 10-20x going rates these days.  I’m curious if I just 
happened across a bad rep and should dig further, or that’s par for the course? 
 Rep was comfortable talking about BGP, v4/v6, etc. so I felt like I was 
talking to the right person until I saw the price lol.

Thanks,

David

Re: SMS gateways

[David Hubbard]

Scott, I was interested in that as well, it was in my original post.  I’m 
considering that and the SMSEagle; both are from Europe.  I can’t find too much 
on them from a real world war stories perspective, but there has been mention 
of the FoxBox on nanog in years past, so there are some users out there.

I am not going the Microtik+cell modem route that Faisal mentioned in his reply 
post because the intent is to tie the SMS alerting into other systems using 
some form of API, and both FoxBox and SMSEagle make that incredibly easy by 
having a simple http interface for sending texts, or a full API if you need to 
do two way.  The nagios plugin (and Zabbix too) are super simple since it’s 
just HTTP POST to send the alerts.

FoxBox claims it will work on Verizon networks because of the 3G support, but 
that doesn’t leave me with a comfortable feeling, so if we buy in, we’d 
probably get accounts from a GSM carrier for it, although I can’t find whether 
or not AT, etc. offer machine accounts, and I would not want to pay $50/mo 
per device just to send random texts.

I did get an off list reply from someone who let me know that our existing 
OpenGear devices (cell+ethernet console servers that run linux) have the 
ability to send SMS using a utility already present in the OS install.  Since 
we already have those in every location we’d also be putting an SMS gateway, 
I’m going to investigate if we could put a cgi script or something similar on 
them to accomplish the same goal with no additional equipment.

David




On 1/7/16, 3:34 PM, "NANOG on behalf of Scott Fisher"  wrote:

>Does anyone having experience getting this to work on US networks?
>
>http://www.smsfoxbox.it/en/foxbox-lx800-gateway-100.html/
>
>I am interested on getting this working with our Nagios notifications.
>
>On Wed, Jan 6, 2016 at 9:40 PM, John Levine  wrote:
>>>Thanks for those pointers. The "mega bill" problem is one I have to avoid. 
>>>We used to use ISDN as backup to T1 circuits,
>>>but had to abandon that after some wayward fail-overs resulted in $5000 
>>>phone bills. I'll check the plan overage terms
>>>carefully!
>>
>> Sounds like an excellent application for a $10/mo prepaid plan on
>> something like Tracfone.  If disaster strikes and you need a lot of
>> data one month, you can add extra credit directly from the phone.
>>
>
>
>
>-- 
>Scott

Re: SMS gateways

[David Hubbard]

The specific phrase you’ll want to use with your VZ rep is a “machine to 
machine” plan.  It’s the same type of plans alarm companies purchase for 
cell-backups.  They have plans with data allowances as low as 1 MB/month for a 
few dollars, but you get destroyed if you go over the plan because the data 
rates are very high.  If you just use them for emergency OOB ssh over cell 
they’re great and economical.

David




On 1/6/16, 5:14 PM, "NANOG on behalf of Mel Beckman"  wrote:

>The problem with Internet-based services is that they depend on the very thing 
>most of us are trying to monitor. For reliable SMS you need out-of-band text 
>transmission at least, and ideally out-of-band TCP/IP data. So far cellular 
>modems provide lots of options for the latter, but I've seen few 
>universally-available choices for the former. I plan to check out the Verizon 
>options mentioned here -- the last time I tried to talk to our business exec, 
>they claimed there were no cheap options.
>
> -mel

SMS gateways

[David Hubbard]

Hey all, was curious if anyone has opinions on the FoxBox vs SMS Eagle boxes 
for sending SMS alerts directly to the cell network?

http://www.smsfoxbox.it/en/foxbox-iq.html/
http://www.smseagle.eu/store/en/devices/1-sms-eagle.html

Any alternative options would be appreciated too.  I saw Microcom’s iSMS modem 
mentioned in the list archives but it’s only 2G so likely won’t be viable much 
longer.

The other question, given the fact that they’re both GSM-based, is whether or 
not you know if AT or T-Mobile have cheap ‘machine’ plans for use by these 
types of devices.  We have all of our OpenGear out of band console servers on 
Verizon and they have these special ‘machine’ plans for $10/mo with very 
limited bandwidth, so that has allowed us to deploy a bunch of them without 
worrying about a huge phone bill.

Thanks,

David

Opinions on Arista 7280?

[David Hubbard]

Curious if anyone's used the 7280 and wants to share their experience?
I'm looking at it primarily for three reasons, MLAG (i.e. multi-chassis
LACP), large ARP/MAC table (256k entries) and large IPv6 neighbor table
(256k entries).  For the table sizes we would like out of one pair of
switches, we'd be into the Cisco 7000 series, but that's dramatically
more expensive and we don't need much of anything else that it offers.

Looked at Brocade too, but they don't have devices that can do the multi
chassis LACP, has the huge table sizes and has a reasonable number of
10gig ports.  It was possible to construct a workable solution using
VDX's for switching and CER's for routing, but that's more complex than
Arista's option if it's a usable option.

Thanks,

David

How to force rapid ipv6 adoption

[David Hubbard]

Had an idea the other day; we just need someone with a lot of cash
(google, apple, etc) to buy Netflix and then make all new releases
v6-only for the first 48 hours.  I bet my lame Brighthouse and Fios
service would be v6-enabled before the end of the following week lol.

David

Question re session hijacking in dual stack environments w/MacOS

[David Hubbard]

Hey all, as we've slowly deployed IPv6 to our end users, it has begun to
cause some issues for those on Mac's specifically.  Apple apparently has
an algorithm at some point in the network stack to decide whether IPv4
or IPv6 is, perhaps, 'better' or 'faster' at any given point in time
during an ongoing session.  This allows a computer talking to a dual
stack remote website to flip flop between v4 and v6 as activity is
conducted.

Websites that require some type of authentication that is handled via
session cookies have been booting our users out randomly with "your ip
address has changed" type message.  This occurs when their Mac decides
to switch between protocols because the site views it as a session
hijacking attempt when Joe User with session ID xyz switches from
192.0.2.10 to 2001:db8::1:1:a or vice versa.

Has anyone run into this?  Our users on other platforms don't seem to
have this issue; linux and MS desktops seem to just use v6 if it's
available and v4 if not.

Thanks,

David

RE: Ear protection

[David Hubbard]

I wear one of two things:

1) The 3M Peltor 105 ear muffs which offer 30db reduction.
I keep them in my car because I also use them for the gun
range, they fit snug but not annoying.  They're only $18
on amazon: http://tinyurl.com/peltor105
There's also a behind the head bar if you don't like the over
the top kind.

2) A lot more expensive, but with a side benefit; I have
a custom set of ear plugs that I use for go kart racing so
I can have radio communication.  You can get them online
or at most race tracks on a race day.  Someone, or DIY at
home, will use a big syringe to squirt the mold liquid in
your ear, it sits for 60 seconds, then they pull it out and
send it off to have the ear plugs made.  They're very good
at eliminating noise but have the side benefit of a
headphone plug so you can still use your phone, ipod, etc.
while you're in the data center. :-)

David

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of 
> Nick Hilliard
> Sent: Wednesday, September 23, 2015 5:34 AM
> To: nanog@nanog.org
> Subject: Ear protection
> 
> What are people using for ear protection for datacenters 
> these days?  I'm down to my last couple of corded 3M 1110:
> 
> http://www.shop3m.com/3m-corded-earplugs-hearing-conservation-
> 1110.html
> 
> These work reasonably well in practice, with a rated nominal 
> noise reduction rate of 29dB.  Some people find them 
> uncomfortable, but they work well for me.
> 
> There are other ear plugs with rated NRR of up to 32-33dB.  
> Anyone have any opinions on what brands work well for them?
> 
> Nick
> 
> 

RE: another tilt at the Verizon FIOS IPv6 windmill

[David Hubbard]

From: Mel Beckman [mailto:m...@beckman.org] 
 
 David,
 Did you consider running an IPv6 tunnel through HE.net?
 

We couldn't get the desired throughput via HE tunnel.  We tried it, then
switched to v6 through VPN using a slice of our own allocation, but
ultimately didn't want that overhead either.

David

RE: another tilt at the Verizon FIOS IPv6 windmill

[David Hubbard]

On Mon, 13 Jul 2015, Paul B. Henson wrote:

 Seems to be a lot less noise on this iteration of the shake fist at 
 Verizon's lack of IPv6 thread, I guess everybody is pretty much burned

 out and given up 8-/. Verizon should just update their IPv6 status 
 page with a link to hurricane electric's tunnel broker page sigh.

I think that's exactly what's occurred.  There was a point where I spent
several years wasting time sending notes to the sales rep, opening
support tickets, trolling them on twitter and their own forum, etc., all
with either no useful answer or no answer at all; ultimately I gave up
and replaced the inexpensive Fios connection with a more $$ TWTC
circuit.  I'd flip it back to Fios if they rolled out v6 since it was a
lot less expensive and had been perfectly reliable at the location that
used it.

David

Huge Level 3 ipv4 issues?

[David Hubbard]

Experiencing packetloss all over the place (chicago, tampa, atlanta) on
Level 3's network; can't even reach them from Brighthouse residential.
IPv6 seems to still be working fine, but of course Brighthouse doesn't
offer that lol.

Anyone seeing the same?

David