RE: IPv6 Pain Experiment

[Naslund, Steve]

>Another misconception. Humans (by and large) count in decimal, base 10. 
>IPv4 is not that. It only LOOKS like that. In fact, the similarity to familiar 
>decimal numbers is one of the reasons that people who are new to networking 
>stumble early on, find CIDR challenging, etc.

Go ahead and read your v4 address over the phone and then do the same with your 
v6 address.  Which is easier?  I do understand all about these addresses both 
being binary underneath ( I've been doing this for over 30 years now).  However 
it is much easier to communicate using four decimal octets.

>I do understand that the hex thing presents a (small) learning curve. 
>But work with it for a little while and it will become familiar, just like 
>IPv4 did.

The question here is how do I convince an enterprise of the need to feel the 
pain of learning it and do I want to be the guy going under the bus the first 
time someone screws up and takes some business critical system down.  People 
generally do not like change and being forced to learn something new.  That is 
just human nature.  You have to give them a reason to want to do it (more 
money, better service, less long term cost, etc.).  It is hard to make the case 
to eliminate v4 in use cases where it is working perfectly fine (especially 
RFC1918 inside an enterprise).

Steven Naslund
Chicago IL

RE: IPv6 Pain Experiment

[Naslund, Steve]

I don’t think the issue is the readability of the addresses (although hex does 
confuse some people), mainly it is the length and ability to deal with any 
string of numbers that long for a human, and I do realize that you can do 
static addressing in IPv6 (but I sure would not want to since the manual entry 
of the addresses is going to be error prone on both the host and into DNS).  It 
is just way harder for a human to deal with hex v6 address than to easily 
memorize four decimal numbers in v4.  Most system admins and engineers can 
rattle off the IPv4 address of a lot of their systems like gateways, DNS 
servers, domain controllers, etc.  Can you imagine keeping those v6 addresses 
in your head the same way?  Think about reading them over the phone, typing 
them into a support case, typing a configuration sheet to be entered by some 
remote hands etc.  I am not saying it is insurmountable, it is just something 
people need to get used to.  To me, that is the biggest reason not to do more 
manual assignments than we need to.
I do understand why they need to be the way they are but I can't see anyone 
thinking IPv6 addresses are easier to read and handle.  

It is not a misconception that most server guys are used to static addressing 
and not auto-assignment.  I also takes some time to get people to stop 
hardcoding static addressing into system configurations.  There are lots of 
applications that have dialog boxes asking for addresses instead of names.  
That needs to stop in an auto-configured or DHCP environment (yeah, I know all 
about DHCP reservations but I hate them).

Your comment regarding small networks not needing IPv6 is exactly my point.  
The original post was talking about MANDATING the use of IPv6 to the exclusion 
of (or taxation of) IPv4.  My point is that there is not really a need to do so 
in a lot of use cases.  

The basic issue is that many system administrators know how to set up and 
configure IPv4 and a lot less of them know how to do IPv6, over time that will 
change but for now it is an indisputable fact.   If I want to go to IPv6 across 
the board, I suppose I could do the education and drag them into it.  However 
as long as my public facing interfaces are mostly fronted by firewalls and load 
balancers I can just do IPv6 at the border and be done with it for now.  Will 
it hurt anyone the leave the existing v4 address assignments there as well?  
No, not really.  Is there a huge advantage to stop using RFC1918 addressing 
within our network?  No, not really.  Would I build a completely new enterprise 
on IPv4...probably not.   Would I recommend that every global enterprise 
eradicate the use of IPv4 in the next couple of yearsno.

Steven Naslund
Chicago IL

On 10/2/19 5:54 PM, Matt Hoppes wrote:
> I disagree on that. Ipv4 is very human readable. It is numbers.
> 
> Ipv6 is not human numbers. It’s hex, which is not how we normally county.
> 
> It is all water under the bridge now, but I really feel like ipv6 could have 
> been made more human friendly and ipv4 interoperable.
> 
>> On Oct 2, 2019, at 8:49 PM, Doug Barton  wrote:
>>
>>> On 10/2/19 3:03 PM, Naslund, Steve wrote:
>>> The next largest hurdle is trying to explain to your server guys that you 
>>> are going to go with all dynamically assigned addressing now
>>
>> Completely false, but a very common misconception. There is nothing about 
>> IPv6 that prevents you from assigning static addresses.
>>
>>> and explaining to your system admin that can’t get a net mask in v4 figured 
>>> out, how to configure their systems for IPv6.
>>
>> If they only need an outbound connection, they probably don't need any 
>> configuration. The instructions for assigning a static address for inbound 
>> connections vary by OS, but I've seen a lot of them, and none of them are 
>> more than 10 lines long.
>>
>> Regarding the previous comments about all the drama of adding DNS records, 
>> etc.; that is what IPAM systems are for. If you're small enough that you 
>> don't need an IPAM for IPv4, you almost certainly don't for IPv6.
>>
>> IPv6 is different, but it's not any more difficult to learn than IPv4. (You 
>> weren't born understanding IPv4 either.)
>>
>> Doug

RE: IPv6 Pain Experiment

[Naslund, Steve]

In my experience, the biggest hurdle to installing a pure IPv6 has nothing to 
do with network gear or network engineers.  That stuff I expect to support v6.  
This biggest hurdle is the dumb stuff like machinery interfaces, surveillance 
devices, the must have IP interface on such and such of an obsolete appliance, 
etc.  The dumb legacy app that supports the ancient obsolete pen plotter that 
we must keep forever, etc.

The next largest hurdle is trying to explain to your server guys that you are 
going to go with all dynamically assigned addressing now and explaining to your 
system admin that can’t get a net mask in v4 figured out, how to configure 
their systems for IPv6.  There are a lot of people in the IT industry who are 
not nearly ready for v6.  In large enterprise networks, there is lots of 
East/West communications between systems and that is very difficult to 
transition through a dual stack process without tripping over a bug or serious 
incident.

It is really hard to look at cost difference in v6/v4 but there will be a 
definite learning curve with the associate oops moments and re-education that 
all costs time/money/downtime.  The simply reality is that there are more IT 
people that understand v4 and do not understand v6 yet.

Steven Naslund
Chicago IL


>I'd strongly disagree that it anywhere near doubles costs. Ultimately you're 
>buying hardware X and it's going to cost whatever it costs. So what more do 
>you really need to do to support IPv6? >Well, let's say you're using OSPF. 
>This means you'll also need to use OSPFv3, but that's not hard because your 
>OSPFv3 configs are going to basically mirror your OSPF configs. You'll need to 
>>run IPv6 over iBGP, perhaps, and over eBGP to your peers and transits, but 
>that's just another set of addresses bound to interfaces, sessions that mirror 
>the IPv4 ones, and policy rules/filters. If >you're doing super heavy TE, then 
>the filter configs might take some effort, but if we're talking about smaller 
>shops, doing heavy TE is unlikely. At that point, you just add a v6 address to 
>your >layer3 interfaces and you're good to go for the network side.

>Most of the time you spend configuring things won't be v4 or v6 specific, and 
>the v4 specific configurations can be copy/pasted with a quick string swap to 
>support v6 in a lot of cases.

RE: IPv6 Thought Experiment

[Naslund, Steve]

It's certainly financial but it's not just companies being cheap. For example 
for smaller companies with a limited staff and small margins. They may want to 
have v6 everywhere but lack the resources to do it. It would for certain speed 
up the process but there would be collateral damage in the process.

Here is the question being dealt with in the corporate environment.  Why should 
I prioritize moving everything to IPv6 now instead of my other zillion IT 
projects that actually are visible to my customers and business users?  It is 
simply, almost always, a cost benefit question.  I would have to convince the 
company that it is in their financial best interest to go that route.  I think 
over time the migration happens organically as more people are familiar with v6 
and all the equipment and setup schemes start using v6 as the default.  I would 
be hard pressed to come up with a reason for a hard deadline.  Making life 
easier for NANOG engineers is not high on most corporate priority lists ☺

Steven Naslund
Chicago IL

RE: IPv6 Thought Experiment

[Naslund, Steve]

A few thoughts:


1.  What global organization has the ability to impose a tax on any 
nation’s citizens?

2.  Do you not see an issue with making everyone worldwide get rid of every 
device that supports v4?  Kind of a burden for a developing country, no?  Also, 
a bit of an e-waste problem I would think.

3.  Do you think that any organization with the power to tax some Internet 
usage (like v6) will stop there and not figure a way of continuing the cash 
flow forever?

4.  The FCC and other standardization organizations often have statutory 
authority to manage things like spectrum management and consumer safety.  What 
would be their authority to mandate v6 usage?

5.  Why not just get carriers to make v4 service an optional extra just 
like static address requests?  There is no reason to empower government more 
than they already are.  Simple economic pressure would work.

6.  Why is your issue more important than any other so-called global issue 
like carbon taxes, endangered species, human trafficking, etc?  Do you want to 
go to a world government to encourage adoption of IPv6?  Why should anyone care 
about that other than us engineers working under the hood

7.  If someone like say Botswana says we are not paying your tax, do you 
intend to send in UN Peacekeeping Forces to collect the money owed?  Are we 
going to war with North Korean if they won’t let us check their routers for the 
presence of v4 addresses?

8.  What is the economic or social reasoning behind obsoleting ipV6?  Is 
this really an existential global issue or are you just inconvenienced by 
dealing with both address families?  While we think it a big deal here on 
NANOG, do you really think that the public sees that issue somewhere in their 
top 20 priorities?  I doubt it.

9.  Some world government enforcing global network standard migrations?  
What could possibly go wrong there ☺.  Do permanent UN Security Council members 
retain the right to veto these standards?

10.   I think at one time the US Government demanded POSIX compliance for all 
of their systems.  That did not even work on the scale of the US Government 
managing their own systems.  Why would this work any better?  Governments are 
notoriously bad at managing their own IT systems, I don’t think we want them 
managing all of ours as well.

Steven Naslund
Chicago IL

From: NANOG  On Behalf Of Antonios Chariton
Sent: Wednesday, October 2, 2019 11:38 AM
To: NANOG 
Subject: Re: IPv6 Thought Experiment

To clarify that further, this would be a monthly tax. So $2 / month.


On 2 Oct 2019, at 19:33, Antonios Chariton 
mailto:daknob@gmail.com>> wrote:

Dear list,
First of all, let me apologize if this post is not allowed by the list. To my 
best interpretation of the guidelines [1] it is allowed, but may be in a gray 
area due to rule #7.

I would like to propose the following thought experiment about IPv6, and I 
would like your opinion on what you believe would happen in such a case. Feel 
free to reply on or off list.

What if, globally, and starting at January 1st, 2020, someone (imagine a 
government or similar, but with global reach) imposed an IPv4 tax. For every 
IPv4 address on the Global Internet Routing Table, you had to pay a tax. Let’s 
assume that this can be imposed, must be paid, and cannot be avoided using some 
loophole. Let’s say that this tax would be $2, and it would double, every 3 or 
6 months.

What do you think would happen? Would it be the only way to reach 100% IPv6 
deployment, or even that wouldn’t be sufficient?

And for bonus points, consider the following: what if all certification bodies 
of equipment, for certifications like FCC’s or CE in Europe, for applications 
after Jan 1st 2023 would include a “MUST NOT support IPv4”..

What I am trying to understand is whether deploying IPv6 is a pure financial 
problem. If it is, in this scenario, it would very very soon become much more 
pricey to not deploy it.

I know there are a lot of gaps in this, for example who imposes this, what is 
the "Global Internet Routing Table", etc. but let’s try to see around them, to 
the core idea behind them.

Thanks,
Antonis

---
Links
---
1: 
https://nanog.org/resources/usage-guidelines/

RE: NetworkLayer

[Naslund, Steve]

Just a tip, but you cannot really determine packet loss on an MPLS network with 
a traceroute.  The nodes between the provider edge routers may not even 
represent your real path.  Also, provider routers within their network will be 
handling pings much differently than they handle your actual traffic.  The 
pings require processing whereas your MPLS traffic will be label switched.  
Much different performance.

Now from the trace below :

If you have zero packet loss to the final hop, how could you possibly have 
packet loss along the path?  Answer, you can't and there is no loss between you 
and the final hop for your traffic.  What you are seeing is the intermediate 
node not bothering or being too busy to answer your ping.

Steven Naslund
Chicago IL

From: NANOG  On Behalf Of K MEKKAOUI
Sent: Monday, September 23, 2019 4:10 PM
To: nanog@nanog.org
Subject: NetworkLayer

Hi

Anyone  from Network Layer to help? We are experiencing packet loss on the 
their MPLS network. I have attached a screenshot. Thank you

4 38.140.46.161  0%   31  16.3ms18.6 6.4   153.1
24.9
 5 154.54.45.113  0%   31   9.8ms13.9   919.2   
  2.9
 6 62.115.168.46  0%   31  12.5ms16.310.2  66   
  9.7
 7 62.115.134.48  0%   31  25.6ms23.416.539.2   
  5.7
 8 62.115.51.98   0%   31  19.4ms22.616.649.9   
  6.1
 9 50.97.19.120%   31  20.2ms  2212.837.9   
  4.3 
10 50.97.19.150%   31  35.9ms  3328.641.7   
3 
11 50.97.17.30   71%   31 timeout35.329.842.8   
  4.9 
12 169.45.18.4   40%   3158ms56.450.661.6   
  3.1 
13 169.45.18.41   0%   30 149.6ms62.750.5   149.6   
 23.4
14 169.48.118.131 0%   30 133.4ms59.850.2   133.4   
 14.7


Karim

RE: 44/8

[Naslund, Steve]

So, if ARIN allocates a v6 assignment to ARDC how do you plan to use it without 
a router or BGP.  Whether it's v4 or v6 you need to route it somewhere.  If you 
have a PC, you can have a router and if you don't have a PC you probably don't 
need to worry about any of this.   If your club can't afford the address 
allocation then you are probably in too expensive a hobby.  That is one of the 
cheaper things you need to get to do radio data.

Steven Naslund
Chicago IL

>Yeah because v6 only is the answer plus tour assuming all of these clubs have 
>routers and BGP and the money to get an allocation and ASN

RE: 44/8

[Naslund, Steve]

Why bother purchasing space?  CGNAT or v6 would both be better ways to go and 
future proof.  The v4 space you purchase today will be essentially worthless.

Steven Naslund
Chicago IL


>I really just want to know how I can purchase some more of that 44. 
>space :)

RE: 44/8

[Naslund, Steve]

How about this?  If you guys think your organization (club, group of friends, 
neighborhood association, whatever...) got screwed over by the ARDC, then why 
not apply for your own v6 allocation.  You would then have complete control 
over its handling and never have to worry about it again.  If you are not sure 
how to get started, visit ARINs website.  It is not that difficult or expensive 
and it would not be hard to justify.

Steven Naslund
Chicago IL

>And after 75 messages, nobody has asked the obvious question. When is ARDC 
>going to acquire IPv6 resources on our behalf? Instead being all worried about 
>legacy resources >we're highly underutilizing.
>
>Ham Radio is supposed to be about pushing the art forward. Let's do that.
>
>-KC8QAY

RE: 44/8

[Naslund, Steve]

>I can guarantee you that Akamai is very much run by beancounters in addition 
>to engineers. I have first hand experience with that.
>
>I can also assure you that it’s quite unlikely that any of Comcast, Netflix, 
>Facebook, Google, AT, T-Mobile, or Verizon just to name a few of the biggest 
>are managed without >due consideration of input from the bean counters. (I’d 
>bet at each of those companies, the day that engineer beats beancounter in a 
>disagreement is rare, indeed).
>
>Each and every one of those large companies has deployed IPv6. Some to a 
>greater extent than others. Facebook and T-Mo stand out as the prime examples, 
>having gone all->IPv6 in as much of their network as practicable today.
>
>The problem with the approach you are taking to IPv6 cost-benefit analysis is 
>that your claim of no ROI doesn’t actually hold true.
>
>The cost savings from a full-on deployment of IPv6 and moving to IPv4 as a 
>service at the edge can be significant. They are hard to capture without very 
>good cost accounting >and the problem really tends to be that engineers are 
>lousy cost-accountants and good cost accountants have a hard time 
>understanding what IPv6 brings to the table.
>
>It’s also true that some fraction (though now diminishing) of the ROI from a 
>v6 deployment cannot be realized until some other parties also deploy IPv6, 
>but there’s good news >on that front, too… More and more of those parties are 
>realizing the need to deploy IPv6.
>
>Owen

The common denominator for all of the companies listed is the size of their 
deployment.  The carriers needed to handle very large scale mobile networks 
that they could not possibly get a large enough allocation for.  The 
alternative CGN gear would have doubtless been extremely expensive as well.  
They also have the engineering and financial horsepower to hold their suppliers 
to the fire to make all of the devices together well with v6.Another 
advantage they have is that the lifespan of a mobile device and it's 
infrastructure is pretty short so they are not dealing with a lot of legacy 
devices.   It helped a lot that the v4 allocations were drying up at around the 
same time that the mobile networks were in full upgrade mode deploying 4G and 
LTE.

Facebook, Google, and Netflix deployed v6 mostly because there were so many 
mobile devices using it that it could not be ignored.  These are all 
market/financial forces at work, not some pioneering engineering drive.  In the 
corporate world we have to provide a reason to spend money so unless there is a 
business drive to deploy v6, it's not going to happen.  That pressure is 
mounting but not at a breaking point yet.  The two main pressures would be the 
cost of expanding into more v4 space and what your customers want.  The move to 
cloud based services means that the corporate demand for v4 space is actually 
decreasing since they are not hosting as many Internet facing applications as 
they once were.  They don't need to move to v6 since their cloud providers are 
doing it for them.  There is also a move in a lot of corporate networks to get 
away from dedicated circuits and use the Internet as transport.  As this 
happens, the corporate network does not even care that the service provider is 
using v6 transport to carry a tunnel.  V4 vs v6 becomes a non-issue over time 
and the pressure to change everything over to v6 goes away since the v4 space 
is not growing.

Steven Naslund
Chicago IL

RE: 44/8

[Naslund, Steve]

In defense of John and ARIN, if you did not recognize that ARDC represented an 
authority for this resource, who would be?  The complaints would have been even 
more shrill if ARIN took it upon themselves to “represent” the amateur radio 
community and had denied the request or re-allocated the assignment.  IANA 
would have been just as out of line making decisions for the community.  In my 
opinion the community is at fault for not recognizing the value of their 
assigned resource and putting mechanisms in place for its management (or maybe 
just not understanding the power that the ARDC represented).  At the end of the 
day, someone has to represent an authority for an assignment and ARDC was as 
close as you could get.  About the closest other organization would have been 
someone like the ARRL but even then you have a US organization representing a 
worldwide loosely coupled community.  I suppose there is a UN basis for 
worldwide management but does anyone here think that any UN organization would 
be a trustworthy administrator.

I think the decision to sell off some of the block makes sense given the size 
and current usage.   After all, by definition amateur radio is about advancing 
the state of the art and experimenting, v6 allocations are not scarce at all.  
The problem here is that the amateur radio community does appear to think they 
were well represented by the ARDC which I would have to say is their fault.   
The original allocation was made a long time ago when there was so much space 
that it had no real value.  What everyone seems to be bent out of shape about 
is that there was a value to this block and someone got paid for it.  This does 
not seem to put the community in peril of running out of space.  How you share 
in the dividends of this sale is another unsolvable problem.  How do you 
allocate this dividend to a worldwide community with no centralized membership 
database?

The original assignment of this block seems to have been a couple of people 
with an idea that was forward looking.  The fact that the authority for 
something like that was somewhat murky is not at all surprising.  In general, a 
lot of older assignments become clouded and disputable through numerous 
acquisitions and changes in control especially at a time that those assignment 
were not seen as having real value.  ARIN is in a sticky position here no 
matter what call they make.  I don’t envy John’s job in the least ☺

Steven Naslund
Chicago IL

>Respectfully John, this wasn't a DBA or an individual figuring the org name 
>field on the old email template couldn't be blank. A class-A was >allocated to 
>a _purpose_. You've not only allowed but encouraged that valuable resource to 
>be reassigned to an organization, this ARDC, and then >treated the 
>organization as a proxy for the purpose. No one asked you to do that. Nothing 
>in the publicly vetted policies demanded that you attach >organizations to the 
>purpose-based allocations and certainly nothing demanded that you grant such 
>organizations identical control over the >resources as the control possessed 
>by folks who were the intended direct recipients of assignments.
>
>This is a rare day, indeed, but I find myself largely agreeing with Bill here.
>
>The only thing I dispute here is that I’m pretty sure that the principals of 
>ARDC did request ARIN to make ARDC the controlling organization of the 
>resource. >The question here is whether or not it was appropriate or correct 
>for ARIN to do so.
>
>IMHO, it was not. IMHO, ARIN should have recognized that this particular block 
>was issued for a purpose and not to an organization or individual. That 
>contacts >were volunteers from the community that agreed to take on a task. 
>Even if the block ended up contactless, it should not have been open to claim 
>and certainly not >to 8.3 or 8.4 partial transfer to another organization away 
>from that purpose.

RE: 44/8

[Naslund, Steve]

I think the Class E block has been covered before.  There were two reasons to 
not re-allocate it.


1.  A lot of existing code base does not know how to handle those addresses 
and may refuse to route them or will otherwise mishandle them.

2.  It was decided that squeezing every bit of space out of the v4 
allocations only served to delay the desired v6 deployment.

This is my recollection and might be flawed.

Steven Naslund
Chicago IL

>Whatever happened to the entire class E block? I know it's reserved for future 
>use, but sounds like that future is now given that we've exhausted all 
>existing >allocations.

RE: Mexico

[Naslund, Steve]

You might want to check with a company called Transtelco.  They are an 
alternate fiber provider (outside of Telmex).

Steven Naslund
Chicago IL

From: NANOG  On Behalf Of Mehmet Akcin
Sent: Thursday, May 30, 2019 9:20 AM
To: nanog 
Subject: Mexico

Hi there

I am looking for dark fibre in several markets in México and waves from mexico 
city to LA and Dallas.

If you know someone in wholesale who can help, please contact me offlist

Thank you.

Mehmet
--
Mehmet
+1-424-298-1903

RE: Special Counsel Office report web site

[Naslund, Steve]

Agreed, I remember the biggest problem when the Starr Report was released was 
that our dial-up PoPs had all lines busy.  It was a different Internet then.

Steven Naslund
Chicago IL

> Hey Mike.
> 
> Agreed. But the scale of a 400 page document with global interest? 
> Should be highly cached with a good ratio of served to pull bits. I'm 
> willing to bet you a beer its just another day on the Internet. 
> However, I could be wrong. Hope to see you in DC to collect! I already 
> know Brett is in. :)

RE: Amazon Peering

[Naslund, Steve]

AKA Too Big To Care.  Happens a lot.

Steven Naslund
Chicago IL

On Wed, Jan 30, 2019 at 11:36 AM Mike Hammett 
mailto:na...@ics-il.net>> wrote:
Oh, you ordered cross connects for a PNI and they stopped responding 
mid-project? Isn't that nice!


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Luca Salvatore via NANOG" mailto:nanog@nanog.org>>
To: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Wednesday, January 30, 2019 9:45:29 AM
Subject: Re: Amazon Peering
Similar experiences here with Amazon.  Initially had semi-regular responses 
from their peering team, they issued LOAs, I ordered the x-connects and then 
radio silence for months.
At the point now where I'm disconnecting x-connects since it's a waste of money.

On Tue, Jan 29, 2019 at 10:49 AM Brooks Swinnerton 
mailto:bswinner...@gmail.com>> wrote:
I also saw sessions come up this weekend, no routes yet though.

On Mon, Jan 28, 2019 at 4:56 PM Tom Beecher 
mailto:beec...@beecher.cc>> wrote:
Mike-

Definitely moving forward now. Someone from Amazon was working with my peering 
group and things started coming up this weekend, so it seems like they're 
catching up pretty good now.

On Thu, Jan 24, 2019 at 2:45 PM Mike Hammett 
mailto:na...@ics-il.net>> wrote:
Let us know your success as well. I'll hold off following up on my requests 
until I see that other people are successful. I don't want to contribute to 
flooding them with requests.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Tom Beecher" mailto:beec...@beecher.cc>>
To: "Jason Lixfeld" mailto:jason%2bna...@lixfeld.ca>>
Cc: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Thursday, January 24, 2019 1:38:51 PM
Subject: Re: Amazon Peering
Thanks Jason. I'll have my peering team take another crack at reaching out and 
see what happens. Appreciate it!

On Thu, Jan 24, 2019 at 2:21 PM Jason Lixfeld 
mailto:jason%2bna...@lixfeld.ca>> wrote:
We circled back with them yesterday on a request we made in late November where 
at the time they said they wouldn’t be turned up until 2019 due to holiday 
network change freeze.

They responded within about 4 hours, thanked us for our patience and 
understanding and said we should expect them to be turned up in about 6 weeks, 
which is apparently their typical timing.

On Jan 24, 2019, at 2:13 PM, Tom Beecher 
mailto:beec...@beecher.cc>> wrote:

I hate to necro-thread , but has anyone seen any movement from Amazon on this? 
I just got a Strongly Worded Message about it, and according to my peering team 
, it's been radio silence for months.


On Sat, Nov 24, 2018 at 12:32 PM JASON BOTHE via NANOG 
mailto:nanog@nanog.org>> wrote:
This is a note I received on Oct18 when checking on a peering request submitted 
on Aug7..

“Apologies for the delays here. We have temporarily frozen IX peering as we 
revise some of our automation processes. I’m hopeful this will be unblocked by 
early November. Thank you for your continued patience.”

On Nov 24, 2018, at 10:59, Darin Steffl 
mailto:darin.ste...@mnwifi.com>> wrote:
It seems wasteful for Amazon to connect to an IX but then ignore peering 
requests for a year.

They have 40G of connectivity but are unresponsive. I'll try emailing all the 
other contacts listed in peeringdb.

Thanks

On Sat, Nov 24, 2018, 10:38 AM Mike Hammett 
mailto:na...@ics-il.net> wrote:
I've e-mailed my contacts there a couple times on people's behalf. No response 
yet.

It seems like a lot of organizations need 1 more person in their peering 
departments.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Darin Steffl" mailto:darin.ste...@mnwifi.com>>
To: "North American Network Operators' Group" 
mailto:nanog@nanog.org>>
Sent: Friday, November 23, 2018 10:21:51 PM
Subject: Amazon Peering
Hey all,

Does anyone have a direct contact to get a peering session established with 
Amazon at an IX? I sent a peering request Dec 2017 and two more times this Sept 
and Nov with no response.

I sent to peer...@amazon.com and received one 
automated response back so I know they received my email but nothing since.



--
Darin Steffl
Minnesota WiFi
www.mnwifi.com
507-634-WiFi
[http://www.snoitulosten.com/wp-content/uploads/2010/01/facebook-small.jpg]
 Like us on Facebook

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>And apparently fire. I wasn’t going to chime in but one of my >providers 
>*just* alerted us to an electrical fire in a Minneapolis pop >causing loads to 
>failover to ups. Unknown whether weather >conditions contributed to the 
>incident.

Yes, in Chicago we will see an increase in home fires because heating systems 
are being pushed to their limits and people tend to do stupid things like run 
unattended space heaters and try to thaw frozen stuff in crazy ways.  In a 
datacenter you might be pushing electrical loads while external electrical 
components are stressed with temperature.  I have seen transformer fires caused 
by the oil inside not circulating correctly.  You end up with hot and cold 
zones in them.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

Ironically you don’t really save a lot of energy when it’s this cold because 
the loops are running at high speed and the humidification coils are working 
overtime to keep the RH up in the room.

People think we can bring in all the outside cold we want but the issue then is 
humidity stability.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>Exactly what he said.  We actually run cooling and supplemental heating >in 
>extreme cold.  We need to keep the chiller pulling heat into itself and >pumps 
>moving on high to keep the outdoor components from freezing >up.  During the 
>summer you might run close to or slightly below freezing >on the coolant loops 
>but in these conditions you cannot run that low a >temp because things will 
>freeze up before the coolant returns.  We also >have to keep the room 
>reasonably warm (50F +).  You also need to watch >out for fast temp excursions 
>to keep humidity under control.

A good HVAC team is critical because we have noted that the building management 
systems often are not flexible enough to automatically deal with super extremes 
and require some human intervention to tell them to do things like run heat and 
cooling simultaneously.  Other actions like closing down louvers on evaporators 
may or may not be automated depending on your systems.  If any part of the 
system does fail in these conditions you have to move super quick or you could 
get serious damage fast.  Our biggest monitoring points are flow rate/pressure 
(which could indicate a freeze up beginning or a pump failing), output and 
return temp on the loops.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

>To the 'infrastructure' question, I think the biggest concerns would >be power 
>related. Although we have a DC in Buffalo that is cooled >on ambient outside 
>air that has the opposite problem ; it's TOO cold >at the moment, so we are 
>cycling most of the hot server exhaust >back into the computer rooms to 
>maintain temperatures.

Exactly what he said.  We actually run cooling and supplemental heating in 
extreme cold.  We need to keep the chiller pulling heat into itself and pumps 
moving on high to keep the outdoor components from freezing up.  During the 
summer you might run close to or slightly below freezing on the coolant loops 
but in these conditions you cannot run that low a temp because things will 
freeze up before the coolant returns.  We also have to keep the room reasonably 
warm (50F +).  You also need to watch out for fast temp excursions to keep 
humidity under control.

The wind speed does make some difference since it is like a fan on your 
evaporator pulling heat out of the cooling loops faster than still air will.

Steven Naslund
Chicago IL

RE: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

[Naslund, Steve]

The main issue is infrastructure like power, cable damage, and heating/cooling 
systems.

Power lines tend to go down because anything weak becomes brittle and any 
accident involving a pole tends to cause them to break rather than absorb 
impact.  Also, conduits and manholes that normally might be above freezing 
underground can have standing water freeze breaking splice cases and such.

Large chiller plants need to be run at higher temperatures and speeds to keep 
any outdoor components from freezing up.

Of course, repairs on anything outdoors tend to take a lot longer to resolve.  
Anything that requires digging might be near impossible in these conditions.

So far though in my area (Chicago) my carriers AT, Comcast, Cogent, and Level 
3 all seem to be fine so far.  Our current temp is -18.  Wind chill reports at 
-50 to -60 depending where you are.

Steven Naslund
Chicago IL

From: NANOG  On Behalf Of Mark Tinka
Sent: Wednesday, January 30, 2019 10:38 AM
To: North American Network Operators' Group 
Subject: Effects of Cold Front on Internet Infrastructure - U.S. Midwest

For anyone running IP networks in the Midwest, are you having to do anything 
special to keep your networks up?

For the data centres, is this cold front a chance to reduce air conditioning 
costs, or is it actually straining the infrastructure?

I'm curious, from a +27-degree C summer's day here in Johannesburg.

Mark.

RE: BGP Experiment

[Naslund, Steve]

Sorry.  Correction.

If it IS RFC compliant they should accept the attribute.  If it is NOT, they 
should drop (and maybe log it).

Steve

>Contact your hardware vendor.  That is not acceptable behavior.  If it is not 
>RFC compliant they need to accept the attribute, if it's not RFC compliant 
>they should gracefully >ignore it.  Now we all know that anyone using that 
>gear is vulnerable to a DoS attack.  Won't be long until anyone else sends 
>that to you.

>Steven Naslund
>Chicago IL 

RE: BGP Experiment

[Naslund, Steve]

Agreed, do you think you will not see that attribute again now that the public 
knows that you are vulnerable to this DoS method.  Expect to see an attack 
based on this method shortly.  They just did you a favor by exposing your 
vulnerability, you should take it as such.  I would be putting in emergency 
patches tonight if available.

Steven Naslund
Chicago IL
>This experiment should be continued.
>
>It's the only way to get people to patch stuff.
>And if all it takes to break things is a single announcement, than that's 
>something that should be definitely fixed.
>
>Blacklisting an ASN is not a solution, that's ignorance.
>
>Regards,
>Filip Hruska

RE: BGP Experiment

[Naslund, Steve]

Contact your hardware vendor.  That is not acceptable behavior.  If it is not 
RFC compliant they need to accept the attribute, if it's not RFC compliant they 
should gracefully ignore it.  Now we all know that anyone using that gear is 
vulnerable to a DoS attack.  Won't be long until anyone else sends that to you.

Steven Naslund
Chicago IL 

>Well, here, when you receive this particular attribute and if you're 
>vulnerable, your equipment automatically gets disconnected from the Internet, 
>so the issue kinda solves >itself.
>
>--
>Töma

RE: BGP Experiment

[Naslund, Steve]

I hope you are as critical of your hardware vendor that cannot accept BGP4 
compliant attributes or have you just not updated your code?  You can black 
hole anything you want but as long as the “Internet” is sending you an RFC 
compliant BGP you better be able to handle it.

Steven Naslund
Chicago IL

On Wed, Jan 23, 2019 at 12:00 PM Ben Cooper 
mailto:b...@packet.gg>> wrote:
Can you stop this?

You caused again a massive prefix spike/flap, and as the internet is not 
centered around NA (shock horror!) a number of operators in Asia and Australia 
go effected by your “expirment” and had no idea what was happening or why.

Get a sandbox like every other researcher, as of now we have black holed and 
filtered your whole ASN, and have reccomended others do the same.

RE: does emergency (911) dispatch uses IP ?

[Naslund, Steve]

That's right.  Check out www.west.com

So, your town will hire someone to put the systems in the 911 center and that 
might be a company like West or say you are a local network provider doing VOIP 
you could hire west to put in a gateway for you.  State and local governments 
can hire them to maintain the databases behind all of this.   You dump all of 
your 911 calls to their gateway (which also has access to your customer address 
data) and they will handle all of the routing out of their centers for you.  
They also provide recording keeping an recording services if you want.  Often 
the first choice from your site to West's datacenters is VOIP over Internet, 
they often then have backup landline routes and any other kind of redundancy 
you want to pay for.  Putting in multiple datacenters and having multiple 
Internet feeds you can easily outperform the reliability of the traditional 
POTS infrastructure.   This industry really took off when the CLEC/wireless 
industry launched to avoid creating a bunch of duplicative expensive 
infrastructure to meet the public safety regulations.  Better to have a few 
giants like West take on the liability and build a nice infrastructure anyone 
can use than to roll you own and have to keep up with the changing technology.  
This kind of technology also allows smaller municipalities to get better more 
advanced systems by purchasing them as a service rather than a one time monster 
budget hit that traditional 911 centers used to be.

One more great thing about doing this is that the network becomes much more 
flexible and calls can be routed around failed centers or center that may 
themselves be in the middle of a disaster.  

Disclaimer:  I don't make anything from West just know of their services and 
roll in the industry.  There are others that might be as good or better.

Steven Naslund

>Yes.
>
>Look for NENA (National Emergency Number Association).
>
>20+ years ago, 911 routing required telco connections in each LATA. Some
>legacy (e.g. copper) still uses LATA-based 911 routing, but a lot of 911 
>routing (i.e. cell, voip, next-gen voice, etc) has been consolidated to a few 
>service providers' data centers connected via IP.
>
>NextGeneration 911 will be essentially 100% IP.
>
>Cost, efficiency, etc.

RE: does emergency (911) dispatch uses IP ?

[Naslund, Steve]

There are multiple ways this outage can impact CL 911 service not just related 
to IP.  Here are a few of them:


1.   You have a POTS line and you dial 911 which gets to your central 
office but the CO switch had no trunks out, either because they were TDM but 
riding one of the optical carriers or IP based.  Both go down when the optical 
mux chokes.

2.  911 center is probably connected to multiple COs (usually at least two) 
but since this outage was nationwide it is very likely that adjacent COs could 
have lost transmission trunks isolating the 911 center.

3.  If you are a cellular customer, that data is getting backhauled to a 
central office and then getting inserted into the same CO switches that lost 
their transmission capacity.

It is important to remember that a lot of stuff is using IP for transport but 
the IP network is also controlling the L1 infrastructure.  In this case it 
looks like the IP network transmitted bad control data to the underlying fiber 
network.

Steven Naslund
Chicago IL

>Guys,
>While reading CL network down impacting 911 services, was trying to get more 
>information about how does this network looks like. From end user to center, I 
>guess VOIP is used. Wondering what is the communication method >from  
>Emergency service center to end units (Police, Fire or any other services). Do 
>they also use IP ? or its Still Radio only ?  if it is IP, do they use Unicast 
>or multicast or broadcast ?
>
>Tried googling, but did not get much information. Any insight would be 
>appreciated.
>
>Thanks
>Mankamana

RE: does emergency (911) dispatch uses IP ?

[Naslund, Steve]

So, to explain the whole system…..


1.  From your location to the your serving CO would be IP, POTS, Cellular 
however your normal phone call route.

2.  From your CO to the CO(s) serving your 911 center.  Might be a 
dedicated trunk or may have high priority to seize channels within the normal 
trunking between COs.  The transport being TDM or VOIP is up to the local 
carrier to decide.

3. From the 911 center to the to the responders can vary by area and size.  
Major metros can route to dispatch centers that handle police fire and other 
services and they can usually communicate over a dedicated trunked radio system 
through the area they cover to directly communicate with responders and command 
elements.  For a small town volunteer department the call might go to a county 
level dispatcher who pages out the responders.  It might even be ringing a 
single phone at the local PD.  It is up to each municipality to determine how 
they want the calls routed and the phone company assigns each number they 
assigned to a 911 center based on your street address.  There is a global table 
(used to be maintained by Bellcore, then Telcordia, I’m not sure now) that 
shows ranges of street addresses mapped to the correct 911 center that is used 
to populate the phone system.

4.   Large Metros and advanced 911 centers send voice and packet/cellular 
radio to responding units often giving them maps, aerial views, known hazmat on 
site, and other data.  If you have a large enterprise you can buy systems that 
allow you to communicate data like this from your organization to the 911 
center.  My company has a data link that sends mapping data, entrance 
information, and even has our security people meet the responders at the door 
every time 911 is called.   The wrong questions is IP or radio only, they are 
not mutually exclusive anymore.  A lot of these systems today are cellular data 
transmission or packet over radio.  Big county wide systems are often hybrids 
of both.  They can have their own radios covering major population density and 
use cellular data to fill in the shadows in their coverage.  The radio user 
just sees the device as a walkie talkie and all that switching is transparent 
to them.

Steven Naslund
Chicago IL


>Guys,
>While reading CL network down impacting 911 services, was trying to get more 
>information about how does this network looks like. From end user to center, I 
>guess VOIP is used. Wondering what is the communication method >from  
>Emergency service center to end units (Police, Fire or any other services). Do 
>they also use IP ? or its Still Radio only ?  if it is IP, do they use Unicast 
>or multicast or broadcast ?
>
>Tried googling, but did not get much information. Any insight would be 
>appreciated.
>
>Thanks
>Mankamana

RE: How to choose a transport(terrestrial/subsea)

[Naslund, Steve]

This was a product available from the earliest Bell System days.  You could 
specify a couple of options.  One is local path redundancy or diversity - 
intended to get you to another central office and not use the same cable as 
another specified circuit.  A second option is called avoidance where you could 
tell the phone company to avoid a certain area.  AT would let you order a 
SONET node which guaranteed two different entrances for fiber ring going 
through at least two different COs, expensive and you pay the install or agree 
to megalong contract terms for a minimum number of access circuits.  


As an example, the US Government ordered command and control circuits and 
explicitly had them avoid major metro areas (that were likely nuclear targets). 
 The deal in practice though is that these options were rarely ordered since 
whoever orders it pays all the initial construction costs.  If you building 
wants a connection to other than your home CO, you have to pay for all of the 
plant construction to reach a point where you could catch a cable going that 
way.

As to who is selling it?  Almost anyone if you pay for that level of custom 
engineering.  More importantly, can you verify it?  The US Government had a 
deal with AT to show them the entire AT backhaul network architecture.  Not 
many customers get that level of access.  If anything they are going to show 
you the "lines between cities" type map that we all know has little to do with 
reality on the ground.

Steven Naslund
Chicago IL

>Who is selling this product? I know SLA compensations on service disruptions 
>is a thing, but there the downside seller is carrying is limited to MRC, that 
>is seller gets higher margin on SLA products than nonSLA, when when outages 
>>are factored in. I don't see the business case for the seller in contractual 
>terms you are proposing. The amendment has to make more money for the seller, 
>otherwise there is no point for them to sell it, unless of course the product 
>>is unmarketable without the amendment.
>I would anticipate if this product is available the seller limits downside in 
>the contracts in such way that it will always be profitable to the seller to 
>sell the insurance to you.
>
>--
>  ++ytti

RE: How to choose a transport(terrestrial/subsea)

[Naslund, Steve]

All true but it is becoming increasingly difficult to determine if a provider 
is using another providers infrastructure (all are at some level).  For 
example, in the SIP world there are several national level carriers that are 
using Level 3s core SIP network and if you were not aware of that you could buy 
trunks from two of the largest SIP trunk providers in the US and actually be 
running on the same network.  Carriers are also very often reliant on the ILEC 
for fiber and last mile access.  Especially in non-metro areas getting diverse 
last mile access could be impossible or have huge construction costs.  It is 
pretty complicated to ensure that your carriers are really diverse and much 
harder to ensure that they stay that way.  I have many examples of carrier 
grooming their own primary and backup circuits onto the same L1 path and not 
realize they have done so.

Contractual diversity is a great idea that does not work since the carriers do 
not actually know what each other’s network looks like.  So let’s say that 
Sprint and CenturyLink choose the same fiber carrier between areas, do you 
think they would notify each other of that fact?  Do you think the fiber 
carrier would tell them what another customer’s network looks like?  You can 
tell Sprint to not use CenturyLink but there is no way to get both of them not 
to use the same third party.  I suppose you could contractually tell a carrier 
to avoid xxx cable but I would have little faith that they maintain that over 
time.  I seriously doubt they review all existing contracts when re-grooming 
their networks.

Steven Naslund
Chicago IL


>I'm of the opinion that, if you need resiliency, you should order explicitly 
>diverse circuits from a primary provider and then a secondary circuit from a 
>second vendor.
>
>Ultimately, If you want contractually-enforced physical diversity then the 
>best options will be single-vendor solutions: Obviously you also want to avoid 
>an unknown single-vendor single-point-of-failure, hence the >secondary 
>provider. Having two vendors is usually a less than optimal solution since 
>neither has visibility into the others' network to ensure the physical 
>diversity required for a truly resilient service: what happens if >an undersea 
>cable is cut, etc?
>
>The cost of such solutions is often unpleasant to justify, mind.
>
>~a

RE: CenturyLink RCA?

[Naslund, Steve]

A note for the guys hanging on to those POTS lines…It won’t really help.  One 
of our sites in Dubuque Iowa had ten CenturyLink PRIs (they are the LEC there) 
homed off of a 5ESS switch.  These all were unable to process calls during the 
CenturyLink problem.  The ISDN messaging returned indicated that the CL phone 
switch had no routes.  This tells me that either their inter-switch trunking or 
SS7 network or both are being transported over the same optical network as the 
Internet services.  So, even if your local line is POTS or traditional TDM it 
won’t matter if all of their transport is dependent on the IP world.

Looking at the Reddit comments on the Infinera devices being a problem, that 
makes more sense because that device blurs the line between optical mux and IP 
enabled devices with its Ethernet mapping functions.  One advantage of the pure 
optical mux is that it does not need, care, or understand L2 and L3 network 
protocols and are largely unaffected by those layers.  Convergence in devices 
moving across more network layers exposes it to more potential bugs.  
Convergence can easily lead to more single points of failure and the traffic 
capacity of these devices kind of encourages carriers to put more stuff in one 
basket than they traditionally did.  I understand the motivation to build a 
single high speed IP centric backbone but it makes everything dependent on that 
backbone.

Steven Naslund
Chicago IL

RE: CenturyLink RCA?

[Naslund, Steve]

They shouldn’t need OOB to operate existing lambdas just to configure new ones. 
 One possibility is that the management interface also handles master timing 
which would be a really bad idea but possible (should be redundant and it 
should be able to free run for a reasonable amount of time).  The main issue 
exposed is that obviously the management interface is critical and is not 
redundant enough.  That is if we believe the OOB explanation in the first place 
(which by the way is obviously not OOB since it wiped out the in band network 
when it failed).

Steven Naslund
Chicago IL


>This seems entirely plausible given that DWDM amplifiers and lasers being a 
>complex analog system, they need OOB to align.
>--
>Eric

RE: CenturyLink RCA?

[Naslund, Steve]

I agree 100%.  Now they need to figure out why bricking the management network 
stopped forwarding on the optical side.  > (Forgive my top posting, not on my 
desktop as I’m out of town)

Steven Naslund
Chicago IL
>
>Wild guess, based on my own experience as a NOC admin/head of operations at a 
>large ISP - they have an automated deployment system for new firmware for a 
>(mission critical) piece of backbone hardware.
>
>They may have tested said firmware on a chassis with cards that did not 
>exactly match the hardware they had in actual deployment (ie: card was older 
>hw revision in deployed hardware), and while it worked fine there, it 
>proceeded >shit the bed in the production.
>
>Or, they missed a mandatory low level hardware firmware upgrade that has to be 
>applied separately before the other main upgrade.
>
>Kinda picturing in my mind that they staged all the updates, set a timer, 
>staggered reboot, and after the first hit the fan, they couldn’t stop the rest 
>as it fell apart as each upgraded unit fell on its own sword on reboot.
>
>I’ve been bit by the ‘this card revision is not supported under this 
>platform/release’ bug more often then I’d like to admit.
>
>And, yes, my eyes did start to get glossy and hazy the more I read their 
>explanation as well.  It’s exactly the kind of useless post I’d write when I 
>want to get (stupid) people off my back about a problem.

RE: CenturyLink RCA?

[Naslund, Steve]

See my comments in line.

Steve

>Hey Steve,

>I will continue to speculate, as that's all we have.

> 1.  Are you telling me that several line cards failed in multiple cities in 
> the same way at the same time?  Don't think so unless the same software fault 
> was propagated to all of them.  If the problem was that they needed to be 
> reset, >couldn't that be accomplished by simply reseating them?

>L2 DCN/OOB, whole network shares single broadcast domain. 

Bad design if that’s the case, that would be a huge subnet.  However even if 
that was the case, you would not need to replace hardware in multiple places.  
You might have to reset it but not replace it.  Also being an ILEC it seems 
hard to believe how long their dispatches to their own central office took.  It 
might have taken awhile to locate the original problem but they should have 
been able to send a corrective procedure to CO personnel who are a lot closer 
to the equipment.  In my region (Northern Illinois) we can typically get access 
to a CO in under 30 minutes 24/7.  They are essentially smart hands technicians 
that can reseat or replace line cards.

> 2.  Do we believe that an OOB management card was able to generate so much 
> traffic as to bring down the optical switching?  Very doubtful which means 
> that the systems were actually broken due to trying to PROCESS the "invalid 
> >frames".  Seems like very poor control plane management if the system is 
> attempting to process invalid data and bringing down the forwarding plane.

>L2 loop. You will kill your JNPR/CSCO with enough trash on MGMT ETH.
>However I can be argued that optical network should fail up in absence of 
>control-plane, IP network has to fail down.

Most of the optical muxes I have worked with will run without any management 
card or control plane at all.  Usually the line cards keep forwarding according 
to the existing configuration even in the absence of all management functions.  
It would help if we knew what gear this was.  True optical muxes do not require 
much care and feeding once they have a configuration loaded.  If they are truly 
dependent on that control plane, then it needs to be redundant enough with 
watch dogs to reset them if they become non responsive and they need policers 
and rate limiter on their interfaces.  Seems they would be vulnerable to a DoS 
if a bad 
BPDU can wipe them out.

> 3.  In the cited document it was stated that the offending packet did not 
> have source or destination information.  If so, how did it get propagated 
> throughout the network?

>BPDU

Maybe, it would be strange that it was invalid but valid enough to continue 
forwarding.  In any case loss of the management network should not interrupt 
forwarding.  I also would not be happy with an optical network that relies on 
spanning tree to remain operational.

> My guess at the time and my current opinion (which has no real factual basis, 
> just years of experience) is that a bad software package was propagated 
> through their network.

>Lot of possible reasons, I choose to believe what they've communicated is what 
>the writer of the communication thought that happened, but as they likely are 
>not SME it's broken radio communication. BCAST storm on L2 DCN >would 
>plausibly fit the very ambiguous reason offered and is something people 
>actually are doing.

My biggest problem with their explanation is the replacement of line cards in 
multiple cities.  The only way that happens is when bad code gets pushed to 
them.  If it took them that long to fix an L2 broadcast storm, something is 
seriously wrong with their engineering.  Resetting the management interfaces 
should be sufficient once the offending line card is removed.  That is why I 
think this was a software update failure or a configuration push.  Either way, 
they should be jumping up and down on their vendor as to why this caused such 
large scale effects.

RE: CenturyLink RCA?

[Naslund, Steve]

Not buying this explanation for a number of reasons :

1.  Are you telling me that several line cards failed in multiple cities in the 
same way at the same time?  Don't think so unless the same software fault was 
propagated to all of them.  If the problem was that they needed to be reset, 
couldn't that be accomplished by simply reseating them?

2.  Do we believe that an OOB management card was able to generate so much 
traffic as to bring down the optical switching?  Very doubtful which means that 
the systems were actually broken due to trying to PROCESS the "invalid frames". 
 Seems like very poor control plane management if the system is attempting to 
process invalid data and bringing down the forwarding plane.

3.  In the cited document it was stated that the offending packet did not have 
source or destination information.  If so, how did it get propagated throughout 
the network?

My guess at the time and my current opinion (which has no real factual basis, 
just years of experience) is that a bad software package was propagated through 
their network.

Steven Naslund
Chicago IL

>
> One thing that is troubling when reading that URL is that it appears several 
> steps of restoration required teams to go onsite for local login, etc.,. 
> Granted, to troubleshoot hardware you need to be physically present to pop a 
> line card in and out, but CTL/LVL3 should have full out-of-band console and 
> power control to all core devices, we shouldn't be waiting for someone to 
> drive to a location to get console or do power cycling. And I would imagine 
> the first step to alot of the troubleshooting was power cycling and local 
> console logs.
>
>
> -John
>
>
>
> On 12/30/18 10:42 AM, Mike Hammett wrote:
>
> It's technical enough so that laypeople immediately lose interest, yet 
> completely useless to anyone that works with this stuff.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> 
> From: "Saku Ytti" 
> To: "nanog list" 
> Sent: Sunday, December 30, 2018 7:42:49 AM
> Subject: CenturyLink RCA?
>
> Apologies for the URL, I do not know official source and I do not 
> share the URLs sentiment.
> https://fuckingcenturylink.com/
>
> Can someone translate this to IP engineer? What did actually happen?
> From my own history, I rarely recognise the problem I fixed from 
> reading the public RCA. I hope CenturyLink will do better.
>
> Best guess so far that I've heard is
>
> a) CenturyLink runs global L2 DCN/OOB
> b) there was HW fault which caused L2 loop (perhaps HW dropped BPDU, 
> I've had this failure mode)
> c) DCN had direct access to control-plane, and L2 congested 
> control-plane resources causing it to deprovision waves
>
> Now of course this is entirely speculation, but intended to show what 
> type of explanation is acceptable and can be used to fix things.
> Hopefully CenturyLink does come out with IP-engineering readable 
> explanation, so that we may use it as leverage to support work in our 
> own domains to remove such risks.
>
> a) do not run L2 DCN/OOB
> b) do not connect MGMT ETH (it is unprotected access to control-plane, 
> it  cannot be protected by CoPP/lo0 filter/LPTS ec)
> c) do add in your RFP scoring item for proper OOB port (Like Cisco 
> CMP)
> d) do fail optical network up
>
> --
>   ++ytti
>


--
  ++ytti

RE: CenturyLink

[Naslund, Steve]

We see slow recovery.  Dallas data service came back up, Dubuque voice service 
still down.

Steven Naslund
Chicago IL

>
>Seems like things have stabilized as of about an hour ago for us.
>

CenturyLink

[Naslund, Steve]

Anyone have any insight to the nationwide CenturyLink issues/outages today?  
Just wondering.  Know for sure that our connections to them from Florida, Iowa, 
and Washington State are all affected.  Voice and data.

Steven Naslund
Chicago IL

RE: Stupid Question maybe?

[Naslund, Steve]

>Why do you think the network portion needs to be contiguous?

Just because some equipment at one time let you configure a non-contiguous mask 
does not make it correct configuration.  Please come up with any valid use case 
for a non-contiguous network (note NETWORK, not any other purpose) mask.

>Well, it does now. But that was not always the case.

It has ALWAYS been the only correct way to configure equipment and is a 
requirement under CIDR.  Here were your commonly used netmasks before CIDR/VLSM 
:

255.0.0.0
255.255.0.0
255.255.255.0

Which one is not contiguous?

>https://www.quora.com/Why-is-the-subnet-mask-255-255-255-64-invalid/answer/Patrick-W-Gilmore

In this example, the writer used it as a parlor trick to actually break a 
network.  That's why you don't do it and it was never a good configuration.  It 
just exploited a UI that did not validate the netmask.

>https://www.quora.com/Why-is-the-subnet-mask-255-255-255-64-invalid

In the second cited link, they are talking about using a non-contiguous mask in 
an access control function.  That is perfectly valid to do, it just is not a 
NETmask anymore.  By definition a netmask identifies the network portion of an 
address.  In the cited example they are defining a group of subnets to an ACL.

Steven Naslund
Chicago IL

>
>--
>TTFN,
>patrick

RE: Stupid Question maybe?

[Naslund, Steve]

I am wondering how a netmask could be not contiguous when the network portion 
of the address must be contiguous.  I suppose a bit mask could certainly be 
anything you want but a netmask specifically identifies the network portion of 
an address.

Steve

> I seem to remember that before the advent of VLSM and CIDR there was 
> no requirement for the 1 bits in the netmask to be contiguous with no 
> intervening 0 bits and there was always someone who tested it out on a 
> production network just to prove a point (usually only once)

RE: rfd

[Naslund, Steve]

I will grant you that no customer ever asked for route dampening.  I also 
realize that RFD is much less important now than in the past.  I come from the 
ARPANET/DDN ages of the Internet and can tell you that RFD was absolutely 
critical in the days of very under powered routers and very unstable data 
links.  I remember when it was quite hard to maintain a 64k link to some 
locations at all.  There might be less of a need for such a simple RFD but it 
did serve its purpose.  In fact, my main argument on this whole topic is that 
RFD is not relevant enough to waste a lot of effort on a global accepted 
mechanism.  It is just not the low hanging fruit of routing performance 
improvements.  I see two major improvements to global routing...congestion 
avoidance (which goes a little bit with bandwidth awareness but not exactly) 
and multipath load balancing (which kind of requires a congestion avoidance 
awareness).  Both of these are going to be extremely difficult issues on a 
global scale of adoption but that's what is needed.

Steven Naslund
Chicago IL

>Dear Steve,
>
>No worries, I have not forgotten the transitive properties of the LOCAL_PREF 
>BGP Path Attribute! :-) You are right that any LOCAL_PREF modifications (and 
>the attribute itself), are local to the Autonomous System in which they were 
>>set, but the effects of such settings can percolate further into the routing 
>system.
>
>A great example is the "BGP Graceful Shutdown" mechanism (science partially 
>documented in https://tools.ietf.org/html/rfc6198, actual specification here 
>https://tools.ietf.org/html/rfc8326). What is interesting is that by 
>>considering a path (any path, could be flapping) my network will propagate 
>alternative paths to my neighboring networks, or possibly even *withdraw* my 
>announcement in favor of alternative
>(stable?) paths via competitors.
>
>By attaching a lower LOCAL_PREF value to a given path for a period of time as 
>a 'penalty' for flapping, I suspect the visiblity of that flapping will be 
>greatly reduced. This of course doesn't hold true when the only origin of the 
>path is >flapping, but in many flapping cases I triaged it was clear that only 
>one out of many links was the root of the flapping.
>
>I'm not sure I share your concerns about scale, it appears that so far we seem 
>to be doing just fine without "route flap dampening, penalty
>type: suppress". No customers ask for it, in fact many are relieved we don't 
>use it. None of our peering partners ask for it either. When we see 
>oscillating paths we reach out to the offending party and ask them to fix it, 
>or take >unilateral action within a specific time frame.
>
>Kind regards,
>
>Job

RE: rfd

[Naslund, Steve]

I think you will find that very hard to evaluate since the value of RFD will be 
different in different network regions.  For example, it is probably good 
practice to run RFD toward a customer on an unstable access link.  It might not 
be a good idea to run it on a major backbone link that could possibly flap a 
large number of times in a very short period due to something like a 
maintenance activity.  Also, in areas that are largely on a fiber 
infrastructure will see RFD in a much different light than a largely wireless 
infrastructure that might be subject to momentary interference or 
interruptions.  I think it is most safe to say that RFD needs to be evaluated 
and tuned for what you want it to do.  Penalties are never a pleasant thing but 
they prevent lawlessness.  That is exactly what RFD does.  You are the cop that 
decides how to enforce the laws.

In fact in my experience people could also get much better network performance 
overall by properly tuning BGP timers but very few actually do it.  I bet you 
could improve the Internet stability way more by doing that.

Steven Naslund
Chicago IL

>What would really be of interest to me would be for those that run RFD to 
>measure its impact to their network (positive or otherwise) so we have 
>something scientific to base on.
>
>The theory (and practice of old) tells us that RFD is either very good, or 
>very bad. There are probably more folk that have turned it off than run it, or 
>vice versa. Ultimately, if we can get the state >of RFD's performance in 2018 
>on an axis, our words will likely carry more weight.
>Mark.

RE: Stupid Question maybe?

[Naslund, Steve]

I see it more used in terms of firewall operations on what are normally network 
routing devices.  I suppose someone with Cisco IOS architecture inside 
knowledge could tell us why they use that notation with ACLs primarily.  

 I have never seen a computer want or accept an inverse mask so it is 
irrelevant to ARP.  The question with ARP is "are we on the same network".

The naming of inverse net mask is really tragic.  It should be called net mask 
and host mask because that is what they really are.  In a net mask the 1s 
denote the network portion, in the host mask (nee inverse netmask) the 1s 
denote the host portion.  That's all there is too it.

The inverse mask could be used to figure out whether to ARP or not.  You just 
have to decide if the 1s or 0s mean that something is significant or not 
significant to your calculation.  Using the inverse mask I could decide to dump 
the portion = 1.  Using the network mask I can dump the portion = 0.  Nothing 
states how you have to use the information.

Steve

>Hi Steve,
>
>That's like saying the inverse mask is technically correct when the computer 
>wants to decide whether to arp for the next hop. No sale man.
>
>A AND NETMASK ?= B AND NETMASK
>
>is exactly the same operation as
>
>A OR inverse NETMASK ?= B OR inverse NETMASK
>
>While A AND inverse NETMASK ?= B AND inverse NETMASK *never* yields useful 
>knowledge.
>
>No sale.
>
>Regards,
>Bill Herrin

RE: rfd

[Naslund, Steve]

It is an interesting article but confirms a few things to me.  

1.  There are only a very small percentage of flapping routes causing an 
inordinate amount of BGP processing.  Would it be more effective to implement 
this route damping mechanism world wide or try to eliminate the source of the 
instability?

2.  The paper does not suggest how you would implement this on a global basis 
and what the "some people have it and some don't" scenario looks like.  The guy 
in the middle pays the price for your unstable customer.

3.  This affects such a small subset of global routes that we should be 
spending our time solving more globally impactful changes to BGP (like path 
bandwidth awareness).

Steven Naslund
Chicago IL

>Mainly because propagating a flapping route across the entire Internet is 
>damaging...
>
>
>https://www.researchgate.net/publication/220850232_Route_Flap_Damping_Made_Usable
>
>scott

RE: Stupid Question maybe?

[Naslund, Steve]

Two reasons :


1.  Legacy configuration portability, people learned a certain way and all 
versions of code understand a certain way.  The best way to correct that issue 
it to accept either of them.

2.  The inverse mask is indeed a pain in the neck but is technically 
correct.  The subnet mask is used where the equipment cares to work with the 
network portion of the address (ignoring the host).  The inverse mask is 
important where the equipment cares more about the host we are referring to 
(ignoring the network).  It’s a bit of a cheat to allow for code used in 
routing to be used for ACL and firewall without modification to the code.  For 
example, the same code piece that routes a network toward an Ethernet interface 
can be reused to route a host toward a null interface.

Steven Naslund
Chicago IL

>Why do we still have network equipment, where half the configuration requires 
>netmask notation, the other half requires CIDR and to throw you off, they also 
>included inverse netmasks.

RE: rfd

[Naslund, Steve]

Remember always that the local pref is just that, YOUR local preference.  
Sending that flapping route upstream does not give your peer the option to 
ignore it.  In any case, the downside is that you have to process that route 
and then choose whether or not to use it.  It’s like saying “now that you have 
processed this unstable route and burned your CPU cycles, I am now giving you 
to option not to install it into your table”.  Remember also that we are only 
talking about default behavior here.  You always have the option to override it 
by changing timer, penalties, or shutting down RFD all together.  We are only 
talking about day-to-day operation here.

Also, keep in mind that when we are talking about alterative stable paths we 
are only talking about what your network sees, not the entire Internet.  If you 
as a service provider are experiencing major issues, you may see a route to me 
as stable or unstable but making global routing decisions based on that is not 
sound.  What might be best for your customer or your business might not be best 
for the Internet community as a whole.   It is a matter of scale, how many 
services providers can allow how many unstable routes before the entire network 
becomes regionally or globally unstable.  It’s important to remember that 
flapping routes leave a certain amount of data in flight with no destination 
which is detrimental to overall performance.  As we move into a V6 world we are 
again worried about the size of the global routing tables and pushing routing 
performance.  Instability of routes is dangerous to system running near the 
limits.  Propagating a known unstable route would be a major shift in routing 
policy.  Today, you either say you can reach something or you don’t say 
anything.  Using the suggested alternative adds the option of “I might be able 
to reach this but not reliably” which then brings about metrics of “how 
reliably?” and that is a huge shift in how global routing works.  We have been 
struggling with a backbone routing protocol that does not really do a good job 
of understanding bandwidth and multiple paths so I would suggest that adding 
“maybe” routes is not a good idea.
At least using RFD you can explain to your customer why they are not reachable 
rather than explaining how you made a manual decision to dump them for the 
“good of the Internet”.  There is also a business penalty to the service 
provider that exposes instability to network.  People don’t want to peer or 
send traffic through unstable network regions.

Steve


>Hi Steve,
>
>Lowering the LP would achieve the outcome you desire, provided there are 
>(stable) alternative paths.
>
>What you advocate results in absolute outages in what may already be 
>precarious situations (natural disasters?) - what Saku Ytti suggests like a 
>less painful alternative with desirable properties.
>
>Kind regards,
>
>Job

RE: rfd

[Naslund, Steve]

Mainly because propagating a flapping route across the entire Internet is 
damaging to performance of things other your own equipment and that of your 
customer.  It is just "bad manners" to propagate a flapping route to your peers 
and it helps maintain a minimum level of stability that it required to keep you 
"on the Internet".  Imagine a table where 1000s of providers are each sending 
100s of unstable routes and that those unstable routes might be redistributing 
into various IGPs that may not respond very gracefully to rapid table changes 
(like most distance vector IGPs).  Also think of this scenario, your link to 
your customer might be flapping but that same customer might have other 
carriers advertising the same address space over a stable link.  In that case 
you would be doing a dis-service by not withdrawing that route and having a 
local-pref does not help since you don't necessarily have visibility to all of 
your customers other carrier networks.

You do have the ability to clear the RFD timers for a route if you need to 
manually intervene for example when you know for a fact that you fixed the 
problem.  That means that if no one is watching or intervening the network will 
"do the safe thing".

Steven Naslund
Chicago IL

>
>I always wondered why does it have to be so binary.
>
>I don't want to decide for my customers if partial visibility is better than 
>busy CPU, but I do appreciate stability. Why can't we have local-pref penalty 
>for flapping route. If it's only option, keep offering it, if there are other, 
>more >stable options, offer those.
>
>--
>  ++ytti

RE: Stupid Question maybe?

[Naslund, Steve]

It is a matter of machine readability vs human readability.  Remember the IP 
was around when routers did not have a lot of horsepower.  The dotted decimal 
notation was a compromise between pure binary (which the equipment used) and 
human readability.  VLSM seems obvious now but in the beginning organizing 
various length routes into very expensive memory and low horsepower processors 
meant that it was much easier to break routes down along byte boundaries.  This 
meant you only had four different lengths of route to deal with.  It was 
intended to eliminate multiple passes sorting the tables.  I am not quite sure 
what you mean about interspersing zeros, that would be meaningless.  Remember 
that it is a mask.  The address bits which are masked as 1s are significant to 
routing.  The bits that are masked with 0s are the host portion and don't 
matter to the network routing table.  

Steven Naslund
Chicago IL


>/24 is certainly cleaner than 255.255.255.0.
>
>I seem to remember it was Phil Karn who in the early 80's suggested that 
>expressing subnet masks as the number of bits from the top end of the address 
>word was efficient, since subnet masks were always a series of ones followd 
>>by zeros with no interspersing, which was incorporated (or independently 
>invented) about a decade later as CIDR a.b.c.d/n notation in RFC1519.
>   - Brian

RE: IGP protocol

[Naslund, Steve]

Yeah there are those.  

Steve

-Original Message-
From: Valdis Kletnieks  On Behalf Of valdis.kletni...@vt.edu
Sent: Monday, November 12, 2018 2:29 PM
To: Naslund, Steve 
Cc: nanog@nanog.org
Subject: Re: IGP protocol

On Mon, 12 Nov 2018 20:21:26 +, "Naslund, Steve" said:

> 2.  Most corporate networks will be running OSPF and/or EIGRP as an IGP.

And I'm sure there's still some crazies out there using RIPv2. :)

RE: IGP protocol

[Naslund, Steve]

I don't know where you heard that but it is probably incorrect.  Here is what I 
think you will find.

1.  Most large networks (service providers) supporting MPLS will be using ISIS 
as their IGP.  Some will have islands of OSPF because not everything speaks 
ISIS.
2.  Most corporate networks will be running OSPF and/or EIGRP as an IGP.

Steven Naslund
Chicago IL

-Original Message-
From: NANOG  On Behalf Of im
Sent: Friday, November 9, 2018 9:03 AM
To: nanog@nanog.org
Subject: IGP protocol

goodmorning nanog,

I heard that OSPF is only famous in asia region...
So that, please could you explain me

  1. what is your backbone's IGP protocol?
  2. why you choose it?


thanks,

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

>Make a second account at your bank.  One account is
>'storage' and has all your money.  You never use
>the 'storage account' ATM card for anything outside
>your bank's ATM machines.

Doubling the service fees from your bank.

>The second one is where you only keep $50-$100 in
>it.  When you use your ATM card it's only this account
>that's used.  Just before you make a purchase, move
>money from your 'storage account' into your 'active
>account' and make the purchase.

Don’t really want to be doing transfers with service fees every time I decide 
to fill up the gas tank.  Also, lots of banks will allow overdrafts which 
creates even more fees and some even auto transfer from one account to another 
to cover your overdrafts.  Also, does nothing for credit cards at all.

Steven Naslund
Chicago IL

RE: ifIndex

[Naslund, Steve]

I see this all the time.  Especially in module chassis.  It seems like 
sometimes it has to do with when each board goes to a ready state as the system 
boots.  We also see renumbering due to virtual interface and board additions.  
While you are running they seem to get the next ifindex available but when you 
reboot the seem to be in the order they come up or the order they are in the 
configuration.  It is a real pain and some software allows us to rescan a 
device and other software we have no easy way other than to delete and the 
re-add the device.  I feel your pain on this one.

I have no idea why most NMS systems can't seem to understand this and just 
rescan at a set interval or after an up/down device event.

Steven Naslund
Chicago IL

> do folk have experience with platforms where ifIndexes are not stable 
> across reboots etc?  how do you deal with it?  do some of those 
> platforms trap on change?

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Remember we are talking about classified intelligence systems and large IT 
organization infrastructure (Google, Yahoo, Apple) here (in the original 
Supermicro post).

That would be information whose unauthorized disclosure would cause grave or 
exceptional grave harm (definition of secret and top secret) to the National 
Security of the United States.  Seems like that warrants a default deny all 
(which is DoD and NSA policy).  I would argue that ANY datacenter server should 
be protected that way unless it is intended to be publicly accessible.

Steven Naslund


>To be fair, the idea that your security costs shouldn't outweigh
>potential harm really shouldn't be controversial.  You don't spend a
>billion dollars to protect a million dollars worth of product.
>
>That's hardly trolling.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

It only proves that you have seen the card at some point.  Useless.

Steven Naslund
Chicago IL

>I'm pretty sure the "entire point" of inventing CVV was to prove you
>physically have the card.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Mr Herrin, you are asking us to believe one or all of the following :

1.  You believe that it is good security policy to NOT have a default DENY ALL 
policy in place on firewalls for DoD and Intelligence systems handling 
sensitive data.

2.  You managed to convince DoD personnel of that fact and actually got them to 
approve an Authorization to Operate such a system based on cost savings.

3.  You are just trolling to start a discussion.

The reason I asked what system it is would be to question the authorities at 
DoD on who and why this was approved.  If you don't want to disclose that then 
you are either trolling or don't want anyone to look into it.  It won't be hard 
to determine if you actually had any government contracts since that is public 
data.  There are very few systems whose EXISTENCE is actually classified, but 
you were the one that cited it as an example supporting your policy.  If you 
cannot name the system then it doesn't support your argument very well does it. 
 Completely unverifiable.

In any case I believe the smart people here on NANOG can accept or reject your 
security advice based on the factors above.  I'm done talking about this one.

Steven Naslund


>> Want to tell us what system this is?

>Yes, I want to give you explicit information about a government system
>in this public forum and you should encourage me to do so. I thought
>you said you had some skill in the security field?
>
>Regards,
>Bill Herrin

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

If there was a waiver issued for your ATO, it would have had to have been 
issued by a department head or the OSD and approved by the DoD CIO after 
Director DISA provides a recommendation and it is mandatory that it be posted 
at https://gtg.csd.disa.mil.  Please see this DoD Instruction 
http://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/831001p.pdf (the 
waiver process is on page 23).  If it did not go through that process, then it 
is not approved not matter what anyone told you.  I know your opinion did not 
make it through that process.

Want to tell us what system this is?
  

Steven Naslund 
Chicago IL


>And yet I got my DoD system ATOed my way earlier this year by
>demonstrating to the security controls assessment team that the cost
>of default-deny-all exceeded the risk cost of default-allow with IDS
>alerts on unexpected traffic.
>
>Because not spending more on a security implementation than the amount
>by which it reduces the risk cost, is a CORE SECURITY PRINCIPLE while
>default-deny-all is merely a standard policy.
>
>Regards,
>Bill Herrin

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

It is good but has several inherent problems (other than almost no one using 
it).  Your card number is static and so is your pin.  If they get compromised, 
you are done.  Changing token/pin resolve the static number problem completely, 
compromise of a used token has no impact whatsoever.

Steven Naslund
Chicago IL

>
>IVR credit card PIN entry is a thing
>
>For example - 
>https://www.hdfcbank.com/personal/making-payments/security-measures/ivr-3d-secure

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

True and that should be mandatory but does not solve the telephone agent 
problem.

Steven Naslund
Chicago IL

  >  I understand that in some countries the common practice is that the
  >  waiter or clerk brings the card terminal to you or you go to it at the
  >  cashier's desk, and you insert or swipe it, so the card never leaves
  >  your hand.  And you have to enter the PIN as well.  This seems
  >  notably more secure against point-of-sale compromise.
  > - Brian

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Sure and with the Exp Date, CVV, and number printed on every card you are open 
to compromise every time you stay in the hotel or go to a restaurant where you 
hand someone your card.  Worse yet, the only option if you are compromised is 
to change all your numbers and put the burden on your of notifying everyone and 
that evening you hand your card to the waiter and the cycle starts over.  The 
system is so monumentally stupid it’s unbelievable.

  Steven Naslund

 Chicago IL



>  Well,

  >  Once you get the Expiry Date (which is the most prevalent data that is not 
encoded with the CHD)

  >  CVV is only 3 digits, we saw ppl using parallelizing tactics to find the 
correct sequence using acquirers around the world.

  >  With the delays in the reporting pipeline, they have the time to 
completely abuse that CHD/Date/CVV before getting caught.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Having gone through this I know that it's all on you which is why no one really 
cares.  You have to notice a fraudulent charge (in most cases), you have to 
dispute it, you have to prove it was not you that made the charge, and if they 
agree then they change all of your numbers at which point you have to contact 
everyone that might be auto charging your accounts for you.  It is a super pain 
in the neck.  So many merchants have been compromised that it seems to be 
having less and less impact on their reputation.


>They actually profit from fraud; and my theory is that that's why issuers have 
>mostly ceased allowing consumers to generate one time use card numbers via 
>portal or app, even though they claim it's >simply because "you're not 
>responsible for fraud."  When a stolen credit card is used, the consumer 
>disputes the resulting fraudulent charges.  The dispute makes it to the 
>merchant account issuer, who >then takes back the money their merchant had 
>collected, and generally adds insult to injury by charging the merchant a 
>chargeback fee for having to deal with the issue (Amex is notable for not 
>doing >this).  The fee is often as high as $20, so the merchant loses whatever 
>merchandise or service they sold, loses the money, and pays the merchant 
>account bank a fee on top of that.
>
>Regarding CVV; PCI permits it being stored 'temporarily', but with specific 
>conditions on how that are far more restrictive than the card number.  Suffice 
>it to say, it should not be possible for an >intrusion to obtain it, and we 
>know how that goes
>
>These days javascript being inserted on the payment page of a compromised 
>site, to steal the card in real time, is becoming a more common occurrence 
>than actually breaching an application or database.  >Websites have so much 
>third party garbage loaded into them now, analytics, social media, PPC ads, 
>etc. that it's nearly impossible to know what should or shouldn't be present, 
>or if a given block of JS >is sending the submitted card in parallel to some 
>other entity.  There's technologies like subresource integrity to ensure the 
>correct code is served by a given page, but that doesn't stop someone from 
>>replacing the page, etc.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

The entire point of the CVV has become useless.  Recently my wife was talking 
to an airline ticket agent on the phone (American Airlines) and one of the 
things they ask for on the phone is the CVV.  If you are going to read that all 
out over the phone with all the other data you are completely vulnerable to 
fraud.  It would be trivial to implement a system where you make a charge over 
the phone like that and get a text asking you to authorize it instead of asking 
for a CVV.

After all this time it is stupid to have the same data being used over and 
over.  We have had SecurID and other token/pin systems in the IT world forever. 
 I have a token on my iPhone right now that I use for certain logins to 
systems.  The hardware tokens cost very little (especially compared to the 
credit card companies revenue).  The soft tokens are virtually free.  A token 
should be useful for one and only one transaction.  You would be vulnerable 
from the time you read your token to someone (or something) until the charge 
hit your account.  You would also not have to worry about a call center agent 
or waiter stealing that data because it could only be used once (and if it is 
not their employer it would become apparent really quickly).   Recurring 
transactions should be unique tokens for a set amount range from a particular 
entity (i.e. 12 transactions, one per month, not more than $500 each, Comcast 
only).  For example, my reusable token given to my cable company should not be 
usable by anyone else.  Why hasn’t this been done yet…..simple there is no 
advantage to the retailers and processors.There has been some one-time use 
numbers for stuff like that but it is inconvenient for the user so it won’t be 
that popular.  The entire system is archaic and dates back to the time of 
imprinting on paper.

Tokenized transactions exist today between some entities and the processors but 
it is time to extend that all the way from card holder to processor.

Steven Naslund
Chicago IL

 >   Well,

 >   Once you get the Expiry Date (which is the most prevalent data that is not 
 > encoded with the CHD)

 >   CVV is only 3 digits, we saw ppl using parallelizing tactics to find the 
 > correct sequence using acquirers around the world.

 >   With the delays in the reporting pipeline, they have the time to 
 > completely abuse that CHD/Date/CVV before getting caught.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

You are free to disagree all you want with the default deny-all policy but it 
is a DoD 5200.28-STD requirement and NSA Orange Book TCSEC requirement.  It is 
baked into all approved secure operating systems including SELINUX so it is 
really not open for debate if you have meet these requirements.  Remember we 
were talking about Intel agency systems here, not the general public.  It is 
SUPPOSED to be painful to open things to the Internet in those environments.  
It needs to take an affirmative act to do so.  It is a simple matter of knowing 
what each and every connection outside the network is there for.  It also 
reveals application vulnerabilities and compromises as well as making it easy 
to identify apps that are compromised.  

In several of the corporate networks I have worked on, they had differing 
policies for different network zones.  For example, you might allow your users 
out to anywhere on the Internet (at least for common public protocols like 
HTTP/HTTPS) but not allow any servers out to the Internet except where they are 
in a DMZ offering public services or destination required for support (like 
patching and remote updates).  Seemed like good workable policy.

Steven Naslund
Chicago IL


>Hi Steve,
>
>I respectfully disagree.
>
>Deny-all-permit-by-exception incurs a substantial manpower cost both
>in terms of increasing the number of people needed to do the job and
>in terms of the reducing quality of the people willing to do the job:
>deny-all is a more painful environment to work in and most of us have
>other options. As with all security choices, that cost has to be
>balanced against the risk-cost of an incident which would otherwise
>have been contained by the deny-all rule.
>
>Indeed, the most commonplace security error is spending more resources
>securing something than the risk-cost of an incident.  By voluntarily
>spending the money you've basically done the attacker's damage for
>them!
>
>Except with the most sensitive of data, an IDS which alerts security
>when an internal server generates unexpected traffic can establish
>risk-costs much lower than the direct and indirect costs of a deny-all
>rule.
>
>Thus rejecting the deny-all approach as part of a balanced and well
>conceived security plan is not inherently an error and does not
>necessarily recommend firing anyone.
>
>Regards,
>Bill Herrin

RE: Oct. 3, 2018 EAS Presidential Alert test

[Naslund, Steve]

I agree 100% and also have noticed that severe weather systems tend to more 
severe in rural areas due to either open spaces (the plains) or trees (forested 
areas) doing more damage.  I can tell you from living the in Midwest that the 
storms in Iowa and Nebraska are way worse than the ones that hit Chicago.  A 
weather guy I know told me it has something to do with convective heat rising 
from major cities which is why you rarely see tornados hitting downtown Chicago 
and New York.  I have noticed that for some reason local weather alerts seem to 
be more reliable than the national level tests on cellular.  Don't know if it 
has to do with shear volume or what.  Also, like I said earlier in rural areas 
you are less likely to run into a bystander that knows what is going on.

Steven Naslund 
Chicago IL


>How quickly we forget.  Puerto Rico's catastrophe was only a year ago. 
>Per capita fatalities in rural areas are usually higher than cities after 
>a disaster.  Telecommunications are even more important in rural areas 
>because you have fewer disaster response resources than in cities.
>Rural areas receive warnings later, have fewer emergency responders, fewer 
>advanced trauma hospitals. There are more neighbors helping neighbors in 
>cities, and more potential sources of help in densely populated areas.
>
>Telecommunication providers are less likely to spend money hardening
>infrastructure in rural areas, because there is less business.  Its easy 
>to find alternative telecommunications in New York City. Its hard to find 
>backup telecommunications in Idaho.
>
>A nation-wide WEA and EAS system helps warn people in both cities and 
>rural areas. But they still depend on carriers and broadcasters. If there 
>are no backup batteries in cell towers, or backup transmitters for 
>broadcasters, you end up with communication blackouts like in Puerto Rico 
>for months.

RE: Oct. 3, 2018 EAS Presidential Alert test

[Naslund, Steve]

I am wondering if this seems common to most of you on here.  In my area it 
seems that all cellular sites have backup generators and battery backup.  Seems 
like the biggest issues we see are devices remote from the central offices that 
lose power and cause disruptions, like RSTs and SLCs.  During hurricane Katrina 
we saw a lot of active outside plant devices underwater and that caused lots of 
disruption even when the CO survived.

Steven Naslund
Chicago IL

 >On October 8, 2018 at 16:37 s...@donelan.com (Sean Donelan) wrote:
 > A nation-wide WEA and EAS system helps warn people in both cities and 
 > rural areas. But they still depend on carriers and broadcasters. If there 
 > are no backup batteries in cell towers, or backup transmitters for 
 > broadcasters, you end up with communication blackouts like in Puerto Rico 
 > for months.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Yet this data gets compromised again and again, and I know for a fact that the 
CVV was compromised in at least four cases I personally am aware of.  As long 
as the processors are getting the money, do you really think they are going to 
kick out someone like Macy's or Home Depot?  After all, it is really only an 
inconvenience to you and neither of them care much about that.

Steve



>It's been a while since I've had to professionally worry about this,
>but as I recall, compliance with PCI [Payment Card Industry] Data
>Security Standards prohibit EVER storing the CVV.  Companies which
>do may find themselves banned from being able to process card
>payments if they're found out (which is unlikely).
>   - Brian

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Allowing an internal server with sensitive data out to "any" is a serious 
mistake and so basic that I would fire that contractor immediately (or better 
yet impose huge monetary penalties.  As long as your security policy is 
defaulted to "deny all" outbound that should not be difficult to accomplish.  
Maybe if a couple contractors feel the pain, they will straighten up.  The 
requirements for securing government sensitive data is communicated very 
clearly in contractual documents.  Genuine mistake can get you in very deep 
trouble within the military and should apply to contractors as well.  I can 
tell you that the "oh well, it's just a mistake" gets used far too often and 
its why your personal data is getting compromised over and over again by all 
kinds of entities.  For example, with tokenization there is no reason at all 
for any retailer to be storing your credit card data (card number, CVV, exp 
date) at all (let alone unencrypted) but it keeps happening over and over.   
There needs to be consequences especially for contractors in the age of cyber 
warfare. 

Steven Naslund
Chicago IL

> Important distinction; You fire any contractor who does it *repeatedly* after 
> communicating the requirements for securing your data.
>
> Zero-tolerance for genuine mistakes (we all make them) just leads to high 
> contractor turnaround and no conceivable security improvement; A a rotating 
> door of mediocre contractors is a much larger >attack surface than a small 
> set of contractors you actively work with to improve security.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

You just need to fire any contractor that allows a server with sensitive data 
out to an unknown address on the Internet.  Security 101.

Steven Naslund

>From: Eric Kuhnke 
>
 >many contractors *do* have sensitive data on their networks with a gateway 
 >out to the public Internet. 
>
>
>I could definitely imagine that happening.
>
>scott

RE: v6 DNSSEC fail, was Buying IPv4 blocks

[Naslund, Steve]

>On 10/5/18 1:53 AM, Mark Andrews wrote:
> If you don’t want fragmented IPv6 UDP responses use
> 
>   server ::/0 { edns-udp-size 1232; };
> 
> That’s 1280 - IPv6 header - UDP header.  Anything bigger than that can 
> theoretically be fragmented.  You will then have to deal with PMTUD 
> failures as the servers switch over to TCP.

That is true provided that you accept that some people may not be able to 
respond without the packet getting fragmented due to tunneling or a million 
other reasons they may not support that MTU.   Nonstandard MTU has always and 
seems will continue to be problematic.  It all really began with tunneling 
which by its nature lowers the MTU available to the application.  Firewalls 
really have to just deal with it and do the re-assembly they need to.  It does 
create tremendous performance issues for these devices at high bandwidth.  
Bottom line is fragmentation sucks and V6 does not make it any better.

Steven Naslund
Chicago IL

RE: Oct. 3, 2018 EAS Presidential Alert test

[Naslund, Steve]

A few cases come to mind.  I also think there are lots of alerts that will not 
send people screaming into the streets.  9/11 did not really have that effect 
in most places and it took quite some time for word to spread to people who did 
not have full time media access.   You also have to account for non-urban areas 
(the majority of our land mass).  In a lot of this country you might not see 
anyone other than the ones you live with for many hours or days at a time.

Here is a few times I know I would not get an alert unless it came via cellular.

1.  2 AM when most people are sleeping.
2.  Riding in my car for an hour to / from work and listening to a podcast or 
music on my device.
3.  Out in the boat or at the beach.  Usually not listening to broadcast media.

Alerting cell phones should not be too high a bar to reach.  They certainly 
don't seem to have a problem getting notifications to you from every app you 
have.  It's pretty long overdue coming from companies that constantly brag 
about their super advanced high speed data networks.  It's pretty clear that 
they are not taking this real serious if you look at how long this executive 
order is taking to realize.  We are talking about years and years.  With more 
and more people cutting their cable and using Internet media vs broadcast 
media, alerting has actually gotten worse recently.

Steven Naslund
Chicago IL

> I wonder, if there were a real alert, what the odds are that one 
> wouldn't hear about it in 1 minute, 5 minutes, etc even if they didn't 
> personally get it.
> 
> Obviously edge cases are possible, you were deep in a cave with your 
> soccer team, but there must be mathematical modeling of that sort of 
> information dispersion.
> 
> It would have to account for other possible channels, word of mouth, 
> facebook, twitter,  posts or really any informatonal source you were 
> on on the internet (e.g., news sites), TV, radio, people screaming in 
> the streets, etc.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

It would be really noticeable.  In the secure networks I have worked with 
"default routes" were actually strictly forbidden.  Also, ACLs and firewall 
policy is all written with Deny All policy first.  Everything talking through 
them is explicitly allowed.

The government especially in the three letter intel agencies is not a clownish 
as they are depicted.

Steven Naslund
Chicago IL

>Which makes the traffic that wanders towards the default route where
>nothing should go *very* noticeable.
>
>Regards,
>Bill Herrin

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Remember it's the data that is classified, not the network.  It does not matter 
if you have IP connectivity, it matters if the classified data is allowed to 
move over the connection. When a government agency talks about a "classified 
network" they are talking about a network that has been approved to transport 
the data and has appropriate access controls.  Just because your email server 
is attached to the Internet does not mean I have access to its data.  Same in 
the classified world, just because you can send an email from the Internet to 
SIPRNET does not mean you have SIPRNET access.

Steven Naslund
Chicago IL

>> Classified networks do not connect to other networks unless
>> they are equally or higher classified.

>that sentence makes no sense.  if A can connect to B because B is more
>highly classified than A, then B is connecting to a less classified
>network A.
>
>randy

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

>> Classified networks do not connect to other networks unless they are 
>> equally or higher classified.  No internet connection.
>> Period.

Not quite but there are at least application level gateways.  For example, 
there are usually gateway that can let unclassified email flow into classified 
systems.  However there is an application gateway to allow ONLY email protocols 
and only in the desired direction.

>Well, if your classified network is connecting to a higher classified net, then
>*that* network is connecting to a lower classified net, right?

In a very highly controlled manner.  The lower classified network may only be 
allowed to send data to the higher classified network.  If the higher level 
network is multilevel capable it will be allowed to move documents to the lower 
level network if they are at the right level of classification.  Again this is 
application layer security and all levels below that would not be trusted 
between the two networks.  A gateway with a specialized application would have 
vetted connectivity to both networks.

>That, plus I think the Snowden escapade was ample proof that security rules 
>will get bent when needed to get work done - it turned out that Snowden was 
>able to walk off with terabytes of data because >security restrictions had 
>been disabled because they were putting a crimp in the analysts' style...

That is completely different.  We are talking HUMINT instead of ELINT or 
SIGINT.  Snowden flat out stole the data as an insider.

Steven Naslund 
Chicago IL

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

Quite different really.  FIREWALK is really an intercept device to get data out 
of a firewalled or air gapped network.  The exploit Bloomberg describes would 
modify or alter data going across a server’s bus.  The big difference is the 
Bloomberg device needs command and control and a place to dump the tapped data 
to over the server’s network connection.  That device is not going to be able 
to do so out of any classified military network I have ever worked on.  Or 
anyone with a halfway decent firewall (which I would assume Apple and Amazon 
would have for the internal servers).  I think this article is unlikely to be 
true for the following reasons :


1.   Separate chip is much more detectable physically than an altered 
chipset that is already on the board.

2.   Requires motherboard redesign to get access to power and buses needed 
(again easily detectable during any design mods “hey does anyone know what 
these are for?”)

3.   Does not have onboard communications so it will be sending data 
traffic on the network interfaces (will definitely trigger even the most 
rudimentary IDP systems).It relies on these backbone Internet companies and 
Intelligence agencies to have absolutely abysmal security on their networks to 
be at all useful.

4.   Parts would have to be brought into the plant, stored somewhere, and 
all the internal systems would need a trail of  where the part came from, how 
ordered it, where it is warehoused, loaded into pick/place, etc.  Much better 
to compromised an existing chips supply chain.

Does anyone think that someone somewhere is trying to kill Supermicro?  They 
sure have had a lots of bad news lately.

Steven Naslund
Chicago IL

>To me this looks like a Chinese version of the NSA FIREWALK product. Which is 
>a network implant built into a RJ45 jack intended to be soldered onto a 
>motherboard. The FIREWALK info came out with the Snowden leaks in 2013 and the 
>tech was >years old at that time.
>
>https://en.wikipedia.org/wiki/NSA_ANT_catalog
>
>I am not able to say a lot more, but when I worked for a major defence 
>contractor in 2006-2007 in Afghanistan, building WAN links in and out of the 
>country by satellite, hardware implants were found in equipment. Not our 
>equipment, but it was close >enough to our operations that we were briefed on 
>it and made aware.

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

I can read but I am really finding it hard to believe that they all agreed to 
even comment on it at all.  Especially the PRC.  Next question would be that if 
Bloomberg was calling me for "months to a year" why not get out in front of it 
in the first place?  The whole story and its responses are very flakey at least 
to my BS detector.

Steven Naslund
Chicago IL


>As they mentioned in their responses, Bloomberg has been calling each
>of the companies for comment on the developing article for months to a
>year. That's why they all knew it was coming.
>
>Regards,
>Bill Herrin

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

It is definitely more desirable to try and tap a serialized data line than the 
parallel lines.  The thing that made me most suspicious of the article is why 
would anyone add a chip.  It requires power and connections that a highly 
detectable.  Motherboard designs are very complex in the characteristics of 
data buses so it is not so easy to just extend or tap into them without having 
negative effects (which brings the board under scrutiny that we don't want).  
Why not embed our rogue chip inside the case of a chip that is already 
controlling the bus or memory we want to play with?  It would be really hard to 
detect without x-ray of all of the system chipsets.

The other thing I am highly skeptical of is the suggestion of attempting to tap 
sensitive intel agency systems this way.  Talking to a C server is suicide 
from within their network.  How long do you think it would take them to detect 
a reach out to the Internet from inside?  How are you going to get the data 
from the outside back into their network?  You still have to defeat their 
firewalls to do it.  If this was targeted to specialized video processing 
server then would it not be unusual for them to be talking to some random IP 
address on the Internet?


Steven Naslund
Chicago IL

>Just theory - tapping on same lines as SPI flash (let's assume it is not 
>QSPI), so we are "in parallel", as "snooper" chip.
>First - it can easily snoop by listening MISO/MOSI/CS/CLK.
>When required data pattern and block detected during snooping, it can 
>remember offset(s) of required data.
>When, later, BMC send over MOSI request for this "offset", we override 
>BMC and force CS high (inactive), so main flash chip will not answer, 
>and answer instead of him our, different data from "snooper".
>Voila... instead of root:password we get root:nihao

RE: bloomberg on supermicro: sky is falling

[Naslund, Steve]

I was wondering about where this chip tapped into all of the data and timing 
lines it would need to have access to.  It would seem that being really small 
creates even more problems making those connections.  I am a little doubtful 
about the article.  It would seem to me better to create a corrupted copy of 
something like a front side bus chipset, memory controller or some other 
component that handles data lines than create a new component that would then 
require a motherboard redesign to integrate correctly.  It would seem that as 
soon as the motherboard design was changed someone would wonder "hey, where are 
all those data lines going?"  It would also require less people in on the plan 
to corrupt or replace a device already in the design.  All you need is a way to 
intercept the original chip supply and insert your rogue devices.

On the opposite side of the argument, does anyone think it is strange that all 
of the companies mentioned in the article along with the PRC managed to get a 
simultaneous response back to Bloomberg.  Seems pretty pre-calculated to me.  
Or did some agency somewhere tell everyone they better shut up about the whole 
thing?

Steven Naslund
Chicago IL 


>Though Bloomberg didn't go out of their way to say it, the photos were
>"representative" of the chip supposedly found. Were they in possession
>of any hard evidence of the chips' existence, they'd have said so.
>
>Regards,
>Bill Herrin

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Don't panic though about the 70 meter rise though.  According to this article 
by National Geographic, it would take around 5000 years to melt that much ice 
even assuming the current temperature rise continues.

Steven Naslund
Chicago IL


>Here is a simple question to answer while you are at it.  Once the arctic ice 
>and glaciers melt, what will cause the ocean levels to continue to rise at 
>this incredible rate?  The total estimate for sea >level rise would be 70 
>meters if absolutely all ice on the face of the Earth melted.  A radical 
>change no doubt but it will not continue forever.
>
>The Earth right now is about as warm as it was during the previous 
>interglacial period which was about 125,000 years ago.  At that time sea level 
>was actually 4 METERS HIGHER THAN IT IS RIGHT NOW.  So >we know that before 
>humans were widespread on Earth, sea level was 4 METERS higher than it is 
>right now.  I guess this goes against the "worse than it has ever been" kind 
>of arguments".
>
>Steven Naslund
>Chicago IL

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Here is a simple question to answer while you are at it.  Once the arctic ice 
and glaciers melt, what will cause the ocean levels to continue to rise at this 
incredible rate?  The total estimate for sea level rise would be 70 meters if 
absolutely all ice on the face of the Earth melted.  A radical change no doubt 
but it will not continue forever.

The Earth right now is about as warm as it was during the previous interglacial 
period which was about 125,000 years ago.  At that time sea level was actually 
4 METERS HIGHER THAN IT IS RIGHT NOW.  So we know that before humans were 
widespread on Earth, sea level was 4 METERS higher than it is right now.  I 
guess this goes against the "worse than it has ever been" kind of arguments".

Steven Naslund
Chicago IL



>Pretty hard to accept 198 inches since NASA's own data shows no more than 
>250mm or 9.4 inches since 1888.  You would have to assume there are no 
>balancing factors.  If the earth gets warmer then there >is also more 
>evaporation of the oceans which causes more rainfall which helps moderate 
>temperature and moves oceanic water inland.  I agree the climate is getting 
>warmer but doubt that trend continues >forever.  History says it won't.  
>Common sense says that in any closed system, things do not change 
>exponentially forever.  I really do need an answer to the question of why in 
>certain years ocean >levels were actually lower than the year before like 
>2010.  I honestly want to know why that happens.
>
>Steven Naslund
>Chicago IL

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Pretty hard to accept 198 inches since NASA's own data shows no more than 250mm 
or 9.4 inches since 1888.  You would have to assume there are no balancing 
factors.  If the earth gets warmer then there is also more evaporation of the 
oceans which causes more rainfall which helps moderate temperature and moves 
oceanic water inland.  I agree the climate is getting warmer but doubt that 
trend continues forever.  History says it won't.  Common sense says that in any 
closed system, things do not change exponentially forever.  I really do need an 
answer to the question of why in certain years ocean levels were actually lower 
than the year before like 2010.  I honestly want to know why that happens.

Steven Naslund
Chicago IL

>Let's run the math.  1mm/additional per year. So 1 the first year, 2 aditional 
>the second, ... and the century year then adds 100mm or 4 inches *by itself*.
>But we need to add years 1 to 99's contributions too...
>
>sum(1..100) = 101 * 50 or 5050mm.  Divide by 25.4 and you get 198 inches 
>cumulative.
>
>
>Be glad the actual rate of acceleration is less than 1mm/year.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

There are lots of ways to construct a graph to look scary.  Just try to redraw 
that graph as the change in overall depth of the ocean.  It would be so flat as 
to be useless.  Wikipedia (might be right or not) says the average depth of the 
ocean is 3,688 meters or 12,100 feet.  If we take that and convert to mm we get 
3,688,000.  So let's take some of the number thrown around here.

If we have a 250 mm rise since 1888 we have a percentage change of 0.01 %

If we have around a mm per year, we have a an annual change of .27%, if it 
is 3 mm per year that is .81%

Just for reference 1 mm is about the size of the average pin head and a flea is 
approximately 1.5mm in length.

Not exactly an alarming graph when you look at it that way.  By the way I am 
also really interested in why sea level actually fell in 2010 according to the 
NASA satellite graph from 56 mm to 48 mm.  Were we especially good people that 
year?

Steven Naslund
Chicago IL


>> JUST BARELY curve upwards. So I dug into THEIR actual data - and even 
>> THEIR data shows something like a cumulative 1mm/year increase - and - 
>> it took ~40 years or so to get to that 1mm increase (to be extra 
>> clear, this is a reported increase over how much oceans are rising now 
>> compared to ~40 years ago. But I'm not even sure this added up to even 
>> a full 1 mm.)

>Compound interest is a bitch.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

And just to be abundantly clear.  I am not denying climate change and I am all 
for eliminating pollution and our impact on the planet in general.  However I 
firmly believe that there will be further climate change regardless of what 
humans do.  That is the cycle of the planet so far and way before we were here.

My main argument is against the case that rising sea level constitutes some 
kind of emergency for network infrastructure.  I don't believe it is an 
emergency and believe that our infrastructure is constantly evolving and all of 
these routes will be handled in the natural course of things.  I say this 
coming from years of network infrastructure engineering and installations and 
time spent in trenches, central offices, and manholes all around the world.  
Given that the backbone of the Internet is mostly running on fiber optics which 
are highly water resistant, localized coastal flooding is less of an issue than 
ever.  Did you ever look into how far an oceanic cable comes ashore before it 
hits the landing station?  If you did you would not be worried about a few mm 
of ocean rise.

Steven Naslund
Chicago IL

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

I agree with this.  I suppose you could take tons of measurements and average 
them out to be pretty accurate but I am not sure how you would account for 
tidal gravitational effects which vary all the time.  Seems like the precision 
claimed would be really hard to pull off without knowing exactly the 
gravitational effect at that location at exactly the same time.  I am also 
wondering how many points on the ocean you would have to take this measurement 
and how often to get that level of precision.  Given the altitude of the 
satellites the percentage of error here is super small, not even OTDR units can 
get that kind of precision on a measurement.

You would also have to align the satellite measurements with the pre-satellite 
measurement which could not have possibly have the same level of precision.  A 
statistics person would have to tell me if that methodology is even valid.  The 
level charts go back to the 1880s and I can't imagine a global network of 
measurement for that time.  I'm also pretty sure they were not taking 
measurements in mm in the United States so there is that conversion error to 
deal with not to mention the lack of international measurement standards.

Steven Naslund
Chicago IL

>Chris,
>
>I understand how radar altimetry works. I would like to understand how
>they achieve the claimed precision. 3.2mm is one heck of a precise
>measurement from a flying platform hundreds of kilometers away,
>particularly when that requires the platform itself to be located with
>even higher precision against some reference points deemed stable for
>the purpose of making the measurement.
>
>

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Well, the problem might be that I am an old guy and remember very well in the 
70s when the "scientific community" screamed at us about the coming ice age.  
Next, we had global warming.  Now we just call it climate change because we 
just don't know which way it's going to go.  Those same anthropologists also 
know that in my area which is Illinois, it was once much hotter and we had 
Tyrannosaurus running around.  We also had ice ages that carved the Great Lakes 
that I am sitting next to.  All of that way before we were here.  Did we warm 
up the climate and prevent the coming ice age in the 70's or could it be that 
the Earth is still in the cycle of coming out of the last ice age?  We know the 
climate has changed without our input.  As an engineer I would like to know how 
we separate what would be happening without us from what effect we are having.  
I am not denying that there are changes but I am not confident in what effect 
we have and whether our effect is counter or accelerating the normal trend.

So, I am not sure what is causing climate change but I am very sure that there 
will be major climate changes on Earth.  Whether a species survives or not is a 
matter of whether they adapt.  

Steven Naslund

>Anthropologists say (there was a pretty good article on this in The
>Atlantic a year or two ago) that's what we (humanity) have done
>historically, adapted, eventually learned to eat acorns or rats or
>whatever*.
>
>And very little if anything to combat the basic problem even if we
>understood it well enough.
>
>We'll adapt and adapt because the problems tend to evolve slowly.
>
>Unfortunately I tend to think that's the likely outcome here simply
>because whatever we (more developed countries) do several billion
>people out there will undo faster because let's face it they want to
>eat regularly, have reliable electricity, etc. etc. etc.
>
>And a lot of what could be done works against their getting all that,
>at least if it's limited to their means.
>
>Perhaps not in theory.
>
>But call me when the G8 or G20 proposes to plunk down the many
>trillions it would likely cost to provide the rest of them with
>fertilizer and farming techniques and energy generation plants and so
>on which aren't contributing to the problem.
>
>Didn't India recently state that they won't even talk about slowing
>down the rate of increase (2nd derivative) of coal usage for at least
>ten years?
>
>Not picking on India, they have their reasons, but just trying to be
>realistic and move past these late-night dorm room bull sessions about
>how the world ought to work.
>
>* One significant exception was crop and field rotation which worked
>very well where it was possible.
>
>-- 
>-Barry Shein
>
>Software Tool & Die| b...@theworld.com | 
>http://www.TheWorld.com
>Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
>The World: Since 1989  | A Public Information Utility | *oo*

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

In 2000 the network runs on completely different infrastructure than it did in 
1900 (what little network existed).  By 2100 I am pretty sure we will be on 
different infrastructure by then.  Are you saying there will be no changes in 
network topology to account for that?  By 2100 neither you or I will have to 
worry about it in any case unless you know of a more major breakthrough in the 
near future.

Steve


>Steve,

>

>You are simply wrong. The sea level is rising at an increasing rate. The 
>average sea level will go up by 30 centimeters to 1 meter by 2100. And of 
>course, the storm surge will increase by a multiple of that.

>

>Sources: NOAA.

>

>It means access networks along the two coasts will be increasingly down.

>r.

-

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

If you live near a coast, you are going to experience bigger storms and loss of 
power more often than someone that lives inland.  If you live in the Himalayas 
you are going to get more snow and cold weather.  Not my problem if you like 
your beach front property.  However I have not seen any major damage to fiber 
based networks and I was a first responder during Hurricane Katrina.  The 
majority of damage was to flooded central offices which is going to happen from 
time to time no matter where you are at.  Overall network reliability has 
increased greatly over the years which is undeniable.  This all just seems very 
alarmist to me.  Also, even if we assume you are correct about ocean levels 
threatening networks (which I don't believe to be even close to the biggest 
threats), what exactly can the NANOG community do about it for you.There 
are real, live, NOW, threats like the BGP hijackings that have been responded 
to that are real operational responses.

Steven Naslund
Chicago IL


>But the reality is that if you get bigger storm surges, your Internet access 
>will be knocked. You will get loss of power and even if the backbone holds up, 
>the access networks will not. Every time we get a severe flood here in 
>Budapest, power is >knocked out and we are down hard. The general population 
>may not take much comfort in your installation of thousands of miles of fiber. 
>Just a fact.

>

>Regards,

>

>Roderick.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Again, the original argument was about rising ocean levels not all causes of 
floods.  Are floods a threat, yep but not as much as it used to be before 
fiber.  Is the rise of ocean levels by 10” per century the cause of all floods, 
no its not.


Steven Naslund
Chicago IL

>From: Rod Beck [mailto:rod.b...@unitedcablecompany.com]
>Sent: Thursday, July 26, 2018 12:13 PM
>To: Naslund, Steve; nanog@nanog.org
>Subject: Re: Rising sea levels are going to mess with the internet
>

>Easy way to settle it. Look at Hurricane Sandy and Katrina. If they had no 
>effect on terrestrial cables, then this is probably a misplaced concern.

>

>- R.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Don't know but the backbone of the Internet is not running on it.  Also, a 
hurricane is not the same as a rise in sea level at less than 10" per century 
which was the threat described here.  There are all kinds of floods for reasons 
other than rising sea levels.

Steven Naslund
Chicago IL

>-Original Message-
>From: Valdis Kletnieks [mailto:val...@vt.edu] On Behalf Of 
>valdis.kletni...@vt.edu
>Sent: Thursday, July 26, 2018 12:09 PM
>To: Naslund, Steve
>Cc: nanog@nanog.org
>Subject: Re: Rising sea levels are going to mess with the internet
>
>
>Have they finished fixing all the corroded copper wiring from Sandy pumping 
>sea water into lower Manhattan?

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

I know of tons of manholes that are continuously full of water every time I 
have been out to them, I am pretty sure those cables have dealt with the 
immersion for quite a number of years.

Steven Naslund
Chicago IL



>I don't have a strong feeling on this matter, but it is not the average 
>increase that matters. Every small increase in average has a multiplier effect 
>on storm surge.

>http://www.pnas.org/content/early/2017/10/23/1715895114.

>

>Nonetheless, my guess is that the real threat is to general property close to 
>the shore, not the terrestrial cables even though they are not waterproof 
>(only submarine cable can handle long term immersion).

>

>Regards,

>

>Roderick.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

So, I accept the data.  Going back to 1880 I will be generous and say that you 
have a 250 mm rise in sea level (which is about 10 inches for us Imperial 
types).  I think we will probably be ready to outrun that problem.  Let's get 
back to real network threats like BGP Hijacking which can wipe you out tomorrow.

Steven Naslund
Chicago IL 

>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jason Kuehl
>Sent: Thursday, July 26, 2018 11:58 AM
>To: rod.b...@unitedcablecompany.com
>Cc: NANOG
>Subject: Re: Rising sea levels are going to mess with the internet
>
>Science https://climate.nasa.gov/vital-signs/sea-level/
>
>Give the data yourself.

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

BTW, I have installed thousands of miles of fiber and been submerged in plenty 
of manholes over the years.  If you have been in a manhole in the spring you 
would know what a non-event you are talking about here.  A lot of your Internet 
is under water a lot of the time anyway (not even counting all of the oceanic 
stuff).


>Since we have been able to cope with train derailments, backhoes, forest 
>fires, traffic accidents, etc, I am pretty confident that the networks will 
>keep up with the lightning fast 1/8" per year rise in >sea level.  
>


Steven Naslund
Chicago IL

RE: California fires: smart speakers and emergency alerts

[Naslund, Steve]

Almost everyone with a cell phone gets real time alerts too.  I am not sure how 
many more ways we can make people aware of things around them.  Seems like yet 
another government mandate to dictate what a device must do.

>People in tornado areas seem to be the most aware that alert radios 
>already exist. No internet access required.

Steven Naslund
Chicago IL

RE: Rising sea levels are going to mess with the internet

[Naslund, Steve]

Since we have been able to cope with train derailments, backhoes, forest fires, 
traffic accidents, etc, I am pretty confident that the networks will keep up 
with the lightning fast 1/8" per year rise in sea level.  

Steven Naslund
Chicago IL

RE: China Showdown Huawei vs ZTE

[Naslund, Steve]

>I'm sure all these companies have legal entities in all countries the operate 
>in. So Huawei in US is US company and Huawei products bought in US from US 
>Huawei are good,. but bad >when bought from Huawei China?

IANAL however I was a network engineer for the US Air Force for over ten years. 
 Here is how the US DoD looks at it.  There are three tiers of defense 
contractors.

Yes - Cisco, Juniper and other US controller entities that the DoD has already 
vetted and does business with on a routine basis.  Also includes systems 
pre-integrated by defense contractors like Boeing and Lockheed that are sold as 
complete turn-key systems.

Maybe - Allied (usually NATO) defense contractors that also have vetted 
security policy.  That would be companies like BAE Systems, Dausault, and 
Siemens.  This would also include US suppliers that may never have done 
business with the DoD before and would have to undergo further review prior to 
being awarded a contract.  There are also some "buy American" consideration 
that required us to use US suppliers unless there was a valid reason why the 
foreign manufacturer was the better choice (say we have an air defense system 
from BAE that has been designed to work with a specific device as part of a 
system).  That is an economic/political concern in addition to the security 
concern and is covered under contracting regulations.  

No way - entities considered to be under to control of or part of the military 
industrial complex of rival nations.  That would include most Russian, Chinese, 
Iranian, etc companies.  Also companies that refuse to comply with certain 
government sanctions or disclosure requirements.  Also companies that employ 
specifically banned individuals under the export control act.

This is not necessarily a technical legal thing like having a corporate entity 
in the US (every multinational does), it is an intelligence assessment of risk. 
 For sensitive software there is a long laundry list of requirements 
surrounding source code control and signing.  In almost all cases I am aware of 
the US DoD acquires a Restricted Software License which actually means that 
they have access to view to source code for whatever they are running and 
require a cryptographically secure way of knowing the running code matches.  
For many of the systems I worked with there were actually special software 
loads signed by DISA (Defense Information Systems Agency) that we had to run.  
DISA software loads also tended to block certain configurations known to be 
insecure and a lot of times enforced higher security or encryption requirement. 
 Our hardware had to come off a list of approved devices and in very sensitive 
service the device were sent to an NSA lab for analysis and returned under 
courier control before they could enter certain areas or networks.  If the 
device ever exited the facility they had to go back for recertification.  This 
was for assurance against embedded hardware taps or bugging devices.  They also 
compared the device against known good models to make sure the hardware was the 
same.

The US Government considers Huawei and ZTE to have "close ties" to the Chinese 
government according to the Director of National Intelligence along with the 
heads of CIA, FBI, and the NSA as stated in testimony before the Senate 
Intelligence Committee.  The founder of Huawei is the former engineering 
officer of the People's Liberation Army of China.

Now, this only applies to US Government agencies according to their acquisition 
rules but there have been moves by the FCC to ban these devices from US 
cellular network.  I am not advocating for or against any of these policies and 
you can run what you want (assuming it can be imported).  I myself would be 
nervous running Huawei code in a device if a cyber war broke out between the US 
and China.

Steven Naslund
Chicago IL  

RE: China Showdown Huawei vs ZTE

[Naslund, Steve]

>
> > Yes looks like they are both under pressure. I feel bad for the USA based
> > employees. I know Huawei has quite a few in Plano, Texas.
>
> Feel sorry for US based consumers. Historically protectionism always
> hurts the local economy most. By creating artificial demand on local
> products, over time local products become uncompetitive for export.
>
> I wonder, in what fundamental way Cisco and Juniper are US products,
> Huawei and ZTE Chinese products? To me it looks like Cisco has no
> development on IOS-XR outside India, components and assembly is in
> China. Shareholders are people holding Vanguard/Blackrock. What makes
> US company a US company?
>

Easy one, what law is the company incorporated under?  Nothing against the 
Chinese companies (some of their stuff is really great), but it is admittedly 
hard to separate China's military industrial complex from their communications 
suppliers.  I can understand other countries not wanting critical 
infrastructure under their software control given that the Chinese government 
has been very active in industrial espionage.  It is not that a US company 
cannot be compromised but I think they might at least be held accountable (by 
their markets) when they get caught.

Steven Naslund
Chicago IL

RE: Is WHOIS going to go away?

[Naslund, Steve]

>Now we're way off-topic, but our constitution acknowledges that is a 
>pre-existing right.  The constitution didn't grant it to you.  (Rights are 
>inherent, privileges are granted)
>
>People have the right to speak, write, and publish whatever they want.
>
>-A

Our Constitution does not equal worldwide law and that is what we are really 
talking about here

They were under British law before the Declaration of Independence (and it was 
vague until the war concluded).  So they did not have that right under the 
current law in their jurisdiction.  They were in fact criminals at the time.  I 
don’t think that was right but it was the fact.

You are way off base in your argument here because I am not disputing that you 
have the right to publish whatever you want on the Internet.  Go ahead and put 
up any web site you like.  I am just saying that you do NOT have the right to 
privacy when broadcasting information in a public forum.   No broadcast media 
is private by definition.

Steven Naslund
Chicago IL


>Now we're way off-topic, but our constitution acknowledges that is a 
>pre-existing right.  The constitution didn't grant it to you.  (Rights are 
>inherent, privileges are granted)
>
>People have the right to speak, write, and publish whatever they want.
>
>-A

RE: Is WHOIS going to go away?

[Naslund, Steve]

>Steve,
>
>I think you should re-examine the early history of the USA.  Anonymous
>pamphleteering was the origin of our rebellion against England,
>with Benjamin Franklin and many of the other founding fathers
>publishing without their identities being registered anywhere.  The
>Federalist Papers which form the basis for our system of government
>were published anonymously.  It's a fundamental part of our liberties.

They did not in fact have the "right" to publish those pamphlets.  They were in 
fact considered sedition by England.  Just because something was done or seems 
correct does not make it a legal right.  Freedom of speech is a right, 
anonymity is not a right, and privacy is not a right you have when you do 
things in public.  That is simple well established law.

>No COMMERCIAL publisher will do that himself, but any individual
>who wants to may do so.  "Freedom of the Press is guaranteed only
>to those who own one", and with the Internet, for the first time
>in many years, it is again practical to publish anonymously.

And you would be violating the law if it was ruled that your publication was in 
fact a publication under the law.  Freedom of the Press is not absolute because 
you do not have the right to violate MY rights by publishing slanderous 
materials, you do not have the right to communicate a threat.  Publications are 
responsible for what they say.  That is also well established law.

Freedom of the Press does not equal right to anonymity.

>It is the entrenched powers who want to require strict identification
>of all sources.

ICANN already has all of the data and they report to the world governments 
ultimately.  ICANN is a non-profit corporation under California law so 
ultimately whatever they do is subject to US law and they could be compelled to 
comply with California or US court orders.  I would say the powers that be 
already have the data. 


>I refer you to the Electronic Frontier Foundation website, and to
>the Internet law blog, and the Reporters Committee for freedom of
>the press, and any good American History book for further information.
>   - Brian

I refer you to the LAW of whatever country you are in.  They don't care what 
the EFF thinks and that blog won't keep you out of jail.

Steven Naslund

RE: Is WHOIS going to go away?

[Naslund, Steve]

>...in every other form of communication, the phrase "get a warrant" comes to 
>mind.
>Except on the internet where we require the information to be public so that 
>anyone and their dog can view it without a warrant.

Wrong on several counts.  You can publicly access the records of who owns every 
radio station, television station, and newspaper in the US and a lot of other 
countries.  All of those organizations are REQUIRED by law to file ownership 
statements. Every periodical published in the United States has a block in it 
identifying the publisher.  Every book sold has a publisher listed even if the 
author chooses to remain anonymous.  It is a violation of the law for a 
telemarketer to call you without identifying themselves (which is what we 
complain about with phone scammers).

Get a warrant only applies to communications (like your phone calls and your 
personal Internet traffic) that have a reasonable expectation of privacy.  If 
you are in the public square shouting to the world you have no expectation of 
anonymity and you can actually be held responsible for false statements about 
another individual.  A publicly accessible website’s published pages would not 
have any expectation of privacy whatsoever.  Besides we are talking about 
identification of ownership of a communications site not the communications 
going through it.  Just because I have your WHOIS data does not mean I have 
root access to your server.   The government needs a warrant to listen to your 
phone calls but not to know you have a phone and where it is.   We are not 
letting people monitor your traffic through WHOIS, we are only identifying who 
is responsible for all communications coming from that site.

Another point is that “get a warrant” does not apply in totalitarian countries 
in any case.  Try saying get a warrant in North Korean or China.  Pretty moot 
point there.


"Those who would give up essential Liberty, to purchase a little temporary 
Safety, deserve neither Liberty nor Safety."

No one ever had the liberty of publishing information to the public without 
accountability.  There are tons of laws protecting you from false statements 
and communications intended to harm your reputation or damage your business.

You are giving up an essential liberty here which is knowing who is saying what 
about you.  Do you not want the right to know the sources of information 
presented to the public?  Do you think I should be able to post anything I want 
about you in the public square without accountability?  Can I put up a 
billboard criticizing you personally and keep my identity a complete secret?  
Might it be nice to know that the source of political news might have an axe to 
grind or an ideological bent, would you like to know that the news story you 
just read was actually from an opposition candidate?  Are we not making a huge 
deal about Russia messing around with elections and trolling?  How would you 
ever know that was going on with no accountability of the source of information?

The whole protecting you from the government point is nothing but a straw man.  
There is no nation state that does not have enough resources to recover that 
information from you or your communications carrier.  Even if your traffic is 
encrypted, it is trivial to figure out who is posting to social media or 
underground websites via other intelligence or simple traffic analysis.  They 
can deny their entire populations access to just about any communications media 
they like.  Most of them don’t because it is actually a more lucrative source 
of intelligence than a threat.  If you are a dissident I might be better off 
leaving you on the Internet and trying to map your network of people even 
though it would be easy to just interrupt your comms.

From a technical perspective, the domain naming system and Internet addressing 
system are assets you do not own.  They are assigned to you and are considered 
a type of resource under quasi governmental control.  If you keep WHOIS data 
secret all you are really doing is keeping the public out and the government 
in.  Do you really believe that ICANN will stand up to the world governments if 
they ask for the data?   If so, you probably also believe that the UN is 
effective at keeping the world at peace.

Steven Naslund
Chicago IL

RE: Is WHOIS going to go away?

[Naslund, Steve]

I don't see why there should not be a way to know who is publishing data on the 
Internet.  In almost all other forms of communication, there is some 
accountability for the origination of information.  Newspaper publishers are 
known, radio stations are usually licensed and publicly known, television is 
licensed as well.  Your phone and Internet traffic is available to the 
government and law enforcement.  People need to be held legally accountable for 
the information they present to the public otherwise you would have absolutely 
no recourse in the event that you were slandered, scammed, or otherwise harmed 
by this information.  People being scared of their government is a real thing, 
however it is not up to the Internet to protect people from their own 
governments, that is a political problem not a technical one.  Always think of 
the negative side of the argument.  If a website was distributing unauthorized 
compromising photos of your children would you want them to be completely 
anonymous? 

Think of how aggravated we all are with the spam we receive every day and how 
much you like spoofed caller ID data when you talk about anonymity.   

Publishing information for access by the entire public should have some sort of 
accountability with it.

When you get into the business of "protecting" people from their own 
"oppressive" governments, you are also protecting "enemies and criminals" from 
another perspective.  Most all nation states would have the ability to track 
the communications to their source in any case so all you are really doing is 
protecting the data from the public.

It would appear to me that the ICANN proposal is nothing more than a means to 
monetize what used to be public data.  Why should Google have all the fun?

Steven Naslund
Chicago IL





-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of b...@theworld.com
Sent: Friday, April 20, 2018 2:11 PM
To: Tei
Cc: nanog@nanog.org
Subject: Re: Is WHOIS going to go away?


On April 20, 2018 at 12:03 oscar.vi...@gmail.com (Tei) wrote:
 > Maybe a good balance for whois is to include organization information
 > so I know where a website is hosted, but not personal information, so
 > I can't show in their house and steal their dog.
 > 
 > I feel uneasy about having my phone available to literally everyone on
 > the internet.

There are various privacy options available when one registers a
domain, generally a matter of checking a box and usually free.

 > 
 > 
 > -- 
 > --
 > ℱin del ℳensaje.

-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

RE: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Naslund, Steve]

Got it.  Do any of those trunks add a new VLAN to the switch that was not 
active before?  If so, that would cause a BPDU over all trunks that allow that 
VLAN.  Even if the port is not up yet, by adding the VLAN to ANY trunk you are 
implying that it should be active on ALL trunks that are not VLAN limited.

Steve

>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joseph Jenkins
>Sent: Thursday, April 05, 2018 4:34 PM
>To: nanog@nanog.org
>Subject: Re: Juniper Config Commit causes Cisco Etherchannels to go into 
>err-disable state
>
>Steve let me clarify the config I am applying has nothing to do with an LACP 
>trunk or any of my existing LACP trunks. It is a completely different 
>>configuration on a completely different interface, the only similarity is 
>that I am trying to configure a trunk interface on the Juniper side for 
>>multiple vlans. There is no LACP configuration involved.

RE: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Naslund, Steve]

It really does not resolve anything it just allows a bad configuration to work. 
 The guard is there so that if one side is configured as a channel and the 
other side is not, the channel gets shut down.  Allowing it to remain up can 
cause a BPDU loop.  Your spanning tree is trying to tell you something, you 
should listen or you could get really hard to isolate issues.

Steven Naslund
Chicago IL  

>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joseph Jenkins
>Sent: Thursday, April 05, 2018 4:16 PM
>To: Robert Webb
>Cc: nanog@nanog.org
>Subject: Re: Juniper Config Commit causes Cisco Etherchannels to go into 
>err-disable state
>
>No there isn't, but from what I am getting responses both onlist and off list 
>is to just run this on the Cisco switches:
>
>no spanning-tree etherchannel guard misconfig
>
>and that should resolve the issue.
>
>Thanks Everyone.

RE: Juniper Config Commit causes Cisco Etherchannels to go into err-disable state

[Naslund, Steve]

I am kind of confused by your configuration.  If the Cisco side is configured 
as LACP trunk, then the Juniper side also needs to be configured as LACP 
trunks.  Spanning-tree would be getting confused because the Cisco is treating 
the LACP trunk as a single interface for purposes of spanning-tree (which 
should be configured at the port-channel level),  Juniper is considering them 
to all be individual ports and would be sending BPDUs over each individual 
interface.  The Cisco is correctly error disabling the port because it detects 
individual port BPDUs and determines that the channel is misconfigured.  Or am 
I missing something in your config completely?

If you are configuring ports other than the connected ports as trunks then your 
case makes sense.  One thing that might cause you issue is the VLAN access of 
the LACP trunk.  If one side has an vlan access list and the other side does 
not, you might get a spanning tree error when you configure a port on a new 
VLAN.  Essentially you have a "trunk all" on one side and a new VLAN is showing 
up on a trunk that is not allowed on the other side.  It would also help to see 
your spanning tree configuration (i.e. are both side running the same spanning 
tree mode?).  The clue here is that the event triggers even though the port is 
not up yet.  If you configure a new port on a VLAN that is not currently up, 
the VLAN will come up on all trunks that are allowed to have all VLANs 
immediately.

Steven Naslund
Chicago IL

>-Original Message-
>From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joseph Jenkins
>Sent: Thursday, April 05, 2018 3:58 PM
>To: nanog@nanog.org
>ubject: Juniper Config Commit causes Cisco Etherchannels to go into 
>err-disable state
>
>I have cases open with both Cisco and Juniper on this, but wanted to see if 
>anyone else had seen an issue like this because support has no idea.
>
>I have a Juniper QFX 5100 Core running in Virtual Chassis mode with 4 
>switches. I have 4 separate stacks of Cisco 3750 switches with 2x1GB uplinks 
>>bound into 4 different LACP trunks. I have had it happen twice now where I 
>apply a trunk port config(not an LACP trunk) to a port that isn't a part of 
>>any of the LACP trunks and it causes all 4 of the Etherchannels on the Cisco 
>stacked switches to go into an err-disable state with these
>messages:
>
>Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on 
>Gi1/0/48, putting Gi1/0/48 in err-disable state
>
>Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on 
>Po17, putting Gi1/0/48 in err-disable state
>
>Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on 
>Po17, putting Po17 in err-disable state
>
>Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
>GigabitEthernet1/0/48, changed state to down
>
>Mar 14 07:11:33: %PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on 
>Gi2/0/48, putting Gi2/0/48 in err-disable state (CA-TOR-1-7-2)
>
>Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
>GigabitEthernet2/0/48, changed state to down
>
>Mar 14 07:11:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
>Port-channel17, changed state to down
>
>Here is the config I am applying to the port that has caused this issue to 
>happen twice now:
>
>set interfaces ge-0/0/67 description "Firewall Port"
>set interfaces ge-0/0/67 unit 0 family ethernet-switching interface-mode trunk 
>set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members >9-10 
>set interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 29 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members >31-32 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 43 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members >50-51 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 56 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members >58 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 66 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 68 >set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 90 set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 143 >set 
>interfaces ge-0/0/67 unit 0 family ethernet-switching vlan members 170
>
>The issue happens within a couple of minutes of committing the config on the 
>Juniper side, there are no cables plugged into port 0/0/67 so technically 
>>there shouldn't be any BPDU's sent out since there isn't a port change.
>
>Juniper Support wants me to turn on trace option and then run though a bunch 
>of scenarios, the issue is that testing this takes down my network.
>
>Just wanted to put it out there to see if anyone else had run into a situation 
>similar to this.
>
>TIA
>
>Joe

RE: Are any of you starting to get AI robocalls?

[Naslund, Steve]

There are plenty of ways to handle that.

There are P-asserted identities that can be passed with the call in addition to 
the CLID.  In SIP, there is also call history data that can give you all of the 
PBX hops identified.

If a customer with a PBX wants to forward calls back into the PSTN then the 
carrier can have an option to allow them to do that but they better also have a 
way of tracking those calls since they are open to abuse and they are obscuring 
the routing of the call.  I am OK with that as long as the carrier is 
responsible for tracking back any abuse complaints.

I do think every PBX in the call path should be identified.  We have had 
instances where some stupid PBX is forwarding calls to the wrong number 
generating abuse complaints that track back to the wrong place because the PBX 
forwarded the original caller-id.  So you call that person back and they 
correctly claim that they never called you.

Steven Naslund
Chicago IL

>-Original Message-
>From: Dovid Bender [mailto:do...@telecurve.com] 
>Sent: Thursday, April 05, 2018 9:07 AM
>To: Naslund, Steve; NANOG list
>Subject: Re: Are any of you starting to get AI robocalls?
>
>Steve,
>
>Any customer with a PBX has a valid reason to pass CLI that isn't theirs if 
>they are passing through a call.  
>
>
>Regards,
>
>Dovid