Re: AWS contact?

[Carsten Bormann]

On 20. Feb 2021, at 01:16, Andras Toth  wrote:
> 
> 00:10:28.921224 IP6 (flowlabel 0x4901f, hlim 54, next-header TCP (6) payload 
> length: 1460) 

Weird.  Why would an IPv6 server ever send a full-sized packet?

You have to sacrifice ~100 bytes to the firewall gods.

Grüße, Carsten

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Eric Kuhnke]

Sierra Leone is very much *not* French speaking, in the context of ISPs and
telecom.

There may be a significant minority of people who do speak French due to
its regional proximity to other countries, for business, but the language
of higher education, business, finance, telecom, real estate and so forth
is all in English.

On Fri, Feb 19, 2021 at 3:20 AM Rod Beck 
wrote:

> I am sure South Africa is better. I am really referring to French speaking
> Western Africa.
>
> -R.
>
> --
> *From:* NANOG 
> on behalf of Mark Tinka 
> *Sent:* Friday, February 19, 2021 5:09 AM
> *To:* nanog@nanog.org 
> *Subject:* Re: Carrier Neutral Site - Freetown, Sierra Leone?
>
>
>
> On 2/18/21 19:45, Rod Beck wrote:
>
> Every time I try to bring a circuit into Africa it is like a complete tour
> of Dante's Hell.
>
>
> A broad brush for such a large place.
>
> Mark.
>

Re: No AM/FM/TV broadcaster outages reported to the FCC in Texas

[Luke Guillory]

Yea, we didn’t have any outages either when hurricane Zeta came through. /S

Didn’t we see a video of a tower collapsing due to ice? Might have been further 
up north possibly.






Sent from my iPhone

On Feb 19, 2021, at 8:19 PM, Sean Donelan  wrote:

*External Email: Use Caution*

The Federal Communications Commission has been reporting a summary of outage
reports (but not the details) from the Texas severe winter weather emergency.

No AM/FM/TV broadcasters have reported any outages in Texas or Oklahoma
during the last week.  More than likely there were outages, but station
owners declined to report them to the FCC.

Outage reports from telecommunications, wireless and VOIP providers

State   Feb 19  Feb 18  Feb 17
Oklahoma:   4   8   7
Texas:  153 208 140

4 PSAP (9-1-1 answering points) were impacted by power outages.  Unclear
what the impact was.

No AM/FM/TV broadcaster outages reported to the FCC in Texas

[Sean Donelan]

The Federal Communications Commission has been reporting a summary of outage
reports (but not the details) from the Texas severe winter weather emergency.

No AM/FM/TV broadcasters have reported any outages in Texas or Oklahoma 
during the last week.  More than likely there were outages, but station 
owners declined to report them to the FCC.


Outage reports from telecommunications, wireless and VOIP providers

State   Feb 19  Feb 18  Feb 17
Oklahoma:   4   8   7
Texas:  153 208 140

4 PSAP (9-1-1 answering points) were impacted by power outages.  Unclear 
what the impact was.

Re: AWS contact?

[Andras Toth]

Hey Michael,

Given the fact that the TCP 3-way handshake is established, sounds like
some Path MTU blackholing happening. Due to it happening during TLS
handshake it's likely from the server towards you.

2a04:4e42::272 and 2a04:4e42:2f::272 belong to Fastly (AS54113) as they
host a share of images-na.ssl-images-amazon.com. Looking at a tcpdump, the
first large packet in the flow is from the server. I have a full-sized
native ipv6 connection so large packets are received:

00:10:28.921224 IP6 (flowlabel 0x4901f, hlim 54, next-header TCP (6)
payload length: 1460) 2a04:4e42:2f::272.443 >
2600:1f18:2fe:904:4341:3edf:79e3:de1d.42114: Flags [.], cksum 0xc122
(correct), seq 1:1429, ack 518, win 131, options [nop,nop,TS val 3517605680
ecr 572934936], length 1428

Using https://github.com/falling-sky/mtu1280d to emulate a smaller MTU, in
response to the large packet (#1) we send back an icmpv6 packet too big
response (#2), triggering Fastly to send smaller packets (#3):

00:11:22.179423 IP6 (flowlabel 0xa9776, hlim 53, next-header TCP (6)
payload length: 1460) 2a04:4e42:2f::272.443 >
2600:1f18:2fe:904:4341:3edf:79e3:de1d.42116: Flags [.], cksum 0xc7f0
(correct), seq 1:1429, ack 518, win 131, options [nop,nop,TS val 3934482883
ecr 572988194], length 1428
00:11:22.179527 IP6 (hlim 255, next-header ICMPv6 (58) payload length:
1240) 2600:1f18:2fe:904:4341:3edf:79e3:de1d > 2a04:4e42:2f::272: [icmp6 sum
ok] ICMP6, packet too big, mtu 1280
00:11:22.180175 IP6 (flowlabel 0xa9776, hlim 53, next-header TCP (6)
payload length: 1236) 2a04:4e42:2f::272.443 >
2600:1f18:2fe:904:4341:3edf:79e3:de1d.42116: Flags [.], cksum 0x8c51
(correct), seq 1:1205, ack 518, win 131, options [nop,nop,TS val 3934482884
ecr 572988196], length 1204

Either your system does not send back an ICMPv6 packet too big reply, or
something drops it on the way and it never reaches Fastly. You should check
your firewall settings in the path to ensure you don't block ICMP and
ICMPv6 packets.

Regards,
Andras


On Sat, Feb 20, 2021 at 7:11 AM Michael Crapse  wrote:

> I would like to know as well who best to reach out to. We are experiencing
> ipv6 related issues with AWS, unable to load even amazon.com completely
> when any of our customers have ipv6 connectivity
>
> curl -vvv
> https://images-na.ssl-images-amazon.com/images/I/11EIQ5IGqaL._RC%7C01ZTHTZObnL.css
> *   Trying 2a04:4e42::272...
> * TCP_NODELAY set
> * Connected to images-na.ssl-images-amazon.com (2a04:4e42::272) port 443
> (#0)
> * schannel: SSL/TLS connection with images-na.ssl-images-amazon.com port
> 443 (step 1/3)
> * schannel: checking server certificate revocation
> * schannel: sending initial handshake data: sending 202 bytes...
> * schannel: sent initial handshake data: sent 202 bytes
> * schannel: SSL/TLS connection with images-na.ssl-images-amazon.com port
> 443 (step 2/3)
> * schannel: failed to receive handshake, SSL/TLS connection failed
> * Closing connection 0
> * schannel: shutting down SSL/TLS connection with
> images-na.ssl-images-amazon.com port 443
> * Send failure: Connection was reset
> * schannel: failed to send close msg: Failed sending data to the peer
> (bytes written: -1)
> * schannel: clear security context handle
> curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed
>
>
>
>
> On Tue, 26 Jan 2021 at 11:00, Josh Baird  wrote:
>
>> Are you sure it's not due to the Verizon outage?  As a non-customer, your
>> options for contacting support are limited.
>>
>> On Tue, Jan 26, 2021 at 12:55 PM Justin Wilson (Lists) 
>> wrote:
>>
>>> What is the best avenue for contacting support for AWS? I have
>>> several ISPs experiencing reachability issues with AWS hosted sites.  These
>>> are from different backbones, different gear, etc.  The common denominator
>>> is AWS.
>>>
>>> Been googling around and can’t seem to find a contact.
>>>
>>>
>>>
>>> Justin Wilson
>>> j...@mtin.net
>>>
>>> —
>>> https://j2sw.com - All things jsw (AS209109)
>>> https://blog.j2sw.com - Podcast and Blog
>>>
>>>

Re: Famous operational issues

[Tom Hill]

On 16/02/2021 22:08, Jared Mauch wrote:
> I was thinking about how we need a war stories nanog track. My favorite was 
> being on call when the router was stolen. 

Enough time has (probably) elapsed since my escapades in a small data
centre in Manchester. The RFO was ten pages long, and I don't want to
spoil the ending, but ... I later discovered that Cumulus' then VP of
Engineering had elevated me to a veritable 'Hall of Infamy' for the
support ticket attached to that particular tale.

One day I'll be able to buy the guy that handled it a *lot* of whisky.
He deserved it.

-- 
Tom

Re: CGNAT

[Tom Hill]

On 19/02/2021 20:11, Tony Wicks wrote:
> Because then a large part of the Internet won't work

Hey, look on the bright side: customers won't be able to use Twitter to
complain! :D

Ofc, IPv4aaS has many good success stories out there; Sky Italia are
running MAP-T, many, many mobile ISPs are running 464XLAT with great
success.

We're in a situation where making IPv6 a *prerequisite* of your IPv4
connectivity can realistically improve your margins when some sort of
CGNAT gateway is a requirement.

Yes it requires looking at your CPE support, but if you're doing even
00,000's of subs, I'm sure the benefits aren't trivial when viewed
through the lens of the number of connections that a single Chrome tab
can happily chew through.

-- 
Tom

Re: [EXTERNAL] Re: dumb question: are any of the RIR's out of IPv4 addresses?

[Max Tulyev]

We can help, of course ;)

Mail me off-list for details. Or isn't it off-topic right here?

17.02.21 06:53, Mann, Jason via NANOG пише:

Any recommendations for legitimate ip brokers?


*From:* NANOG  on behalf of 
Michael Thomas 

*Sent:* Tuesday, February 16, 2021 5:46 PM
*To:* Fred Baker 
*Cc:* nanog@nanog.org 
*Subject:* [EXTERNAL] Re: dumb question: are any of the RIR's out of 
IPv4 addresses?


On 2/16/21 4:18 PM, Fred Baker wrote:
You may find this article interesting: 
https://urldefense.com/v3/__https://blog.apnic.net/2019/12/13/keep-calm-and-carry-on-the-status-of-ipv4-address-allocation/__;!!GaaboA!999i8DMj5mceMG2R6J8wgZ29XjBhQvAJU3QMixqhvjqpQCsdAvcck6BpWKVqMw$ 
 





So aside from Afrinic, this is all being done on the gray market?
Wouldn't you expect that price to follow something like an exponential
curve as available addresses become more and more scarce and unavailable
for essentially any price?

Mike



Sent from my iPad


On Feb 16, 2021, at 3:07 PM, Michael Thomas  wrote:


Basically are there places that you can't get allocations? If so, 
what is happening?


Mike

Re: CGNAT

[Mark Andrews]

I’m sure the large parts of the world already doing this would disagree.
-- 
Mark Andrews

> On 20 Feb 2021, at 07:11, Tony Wicks  wrote:
> 
> 
> Because then a large part of the Internet won't work
> 
> From: NANOG  on behalf of Mark 
> Andrews 
> Sent: Saturday, 20 February 2021, 9:04 am
> To: Steve Saner
> Cc: nanog@nanog.org
> Subject: Re: CGNAT
> 
> Why not go whole hog and provide IPv4 as a service? That way you are not 
> waiting for your customers to turn up IPv6 to take the load off your NAT box.
> 
> Yes, you can do it dual stack but you have waited so long you may as well 
> miss that step along the deployment path.
> -- 
> Mark Andrews
> 
>>> On 20 Feb 2021, at 01:55, Steve Saner  wrote:
>>> 
>> 
>> We are starting to look at CGNAT solutions. The primary motivation at the 
>> moment is to extend current IPv4 resources, but IPv6 migration is also a 
>> factor.
>> 
>> We've been in touch with A10. Just wondering if there are some alternative 
>> vendors that anyone would recommend. We'd probably be looking at a solution 
>> to support 5k to 15k customers and bandwidth up to around 30-40 gig as a 
>> starting point. A solution that is as transparent to user experience as 
>> possible is a priority.
>> 
>> Thanks
>> 
>> -- 
>> Steve Saner
>> ideatek HUMAN AT OUR VERY FIBER
>> This email transmission, and any documents, files or previous email messages 
>> attached to it may contain confidential information. If the reader of this 
>> message is not the intended recipient or the employee or agent responsible 
>> for delivering the message to the intended recipient, you are hereby 
>> notified that any dissemination, distribution or copying of this 
>> communication is strictly prohibited. If you are not, or believe you may not 
>> be, the intended recipient, please advise the sender immediately by return 
>> email or by calling 620.543.5026. Then take all steps necessary to 
>> permanently delete the email and all attachments from your computer system.
>> 
> 

Re: CGNAT

[JORDI PALET MARTINEZ via NANOG]

IPv4 as a Service such as 464XLAT, will allow them to use less IPv4 public 
addresses than CGNAT, less costly equipment (or open source) and still provide 
dual-stack inside the customers networks.

 

There is nothing from Internet that will not work. I’ve many deployments based 
on this, and this is the transition mechanism that have more millions of 
customers, even if compared with all the others together.

 

Regards,

Jordi

@jordipalet

 

 

 

El 19/2/21 21:15, "NANOG en nombre de Tony Wicks" 
 escribió:

 

Because then a large part of the Internet won't work

 

From: NANOG  on behalf of Mark 
Andrews 
Sent: Saturday, 20 February 2021, 9:04 am
To: Steve Saner
Cc: nanog@nanog.org
Subject: Re: CGNAT


Why not go whole hog and provide IPv4 as a service? That way you are not 
waiting for your customers to turn up IPv6 to take the load off your NAT box.

 

Yes, you can do it dual stack but you have waited so long you may as well miss 
that step along the deployment path.

-- 

Mark Andrews




On 20 Feb 2021, at 01:55, Steve Saner  wrote:



We are starting to look at CGNAT solutions. The primary motivation at the 
moment is to extend current IPv4 resources, but IPv6 migration is also a factor.

 

We've been in touch with A10. Just wondering if there are some alternative 
vendors that anyone would recommend. We'd probably be looking at a solution to 
support 5k to 15k customers and bandwidth up to around 30-40 gig as a starting 
point. A solution that is as transparent to user experience as possible is a 
priority.

 

Thanks


-- 

Steve Saner

ideatek HUMAN AT OUR VERY FIBER

This email transmission, and any documents, files or previous email messages 
attached to it may contain confidential information. If the reader of this 
message is not the intended recipient or the employee or agent responsible for 
delivering the message to the intended recipient, you are hereby notified that 
any dissemination, distribution or copying of this communication is strictly 
prohibited. If you are not, or believe you may not be, the intended recipient, 
please advise the sender immediately by return email or by calling 
620.543.5026. Then take all steps necessary to permanently delete the email and 
all attachments from your computer system.

 



**
IPv4 is over
Are you ready for the new Internet ?
http://www.theipv6company.com
The IPv6 Company

This electronic message contains information which may be privileged or 
confidential. The information is intended to be for the exclusive use of the 
individual(s) named above and further non-explicilty authorized disclosure, 
copying, distribution or use of the contents of this information, even if 
partially, including attached files, is strictly prohibited and will be 
considered a criminal offense. If you are not the intended recipient be aware 
that any disclosure, copying, distribution or use of the contents of this 
information, even if partially, including attached files, is strictly 
prohibited, will be considered a criminal offense, so you must reply to the 
original sender to inform about this communication and delete it.

Re: Famous operational issues

[Sabri Berisha]

- On Feb 19, 2021, at 3:07 AM, Daniel Karrenberg d...@ripe.net wrote:

Hi,

> Lessons: HW/SW mono-cultures are dangerous. Input testing is good
> practice at all levels software. Operational co-ordination is key in
> times of crisis.

Well... Here is a very similar, fairly recent one. Albeit in this case, the
opposite is true: running one software train would have prevented an outage.
Some members on this list (hi, Brian!) will recognize the story.

Group XX within $company decided to deploy EVPN. All of backbone was running
single $vendor, but different software trains. Turns out that between an
early draft, implemented in version X, and the RFC, implemented in version Y,
a change was made in NLRI formats which were not backwards compatible.

Version X was in use on virtually all DC egress boxes, version Y was in use
on route reflectors. The moment the first EVPN NLRI was advertised, the 
entire backbone melted down. Dept-wide alert issued (at night), people trying
to log on to the VPN. Oh wait, the VPN requires yubikey, which requires the
corp network to access the interwebs, which is not accessible due to said
issue.

And, despite me complaining since the day of hire, no out of band network.

I didn't stay much longer after that.

Thanks,

Sabri 

Re: CGNAT

[Tony Wicks]

Because then a large part of the Internet won't workFrom: NANOG  on behalf of Mark Andrews Sent: Saturday, 20 February 2021, 9:04 amTo: Steve SanerCc: nanog@nanog.orgSubject: Re: CGNATWhy not go whole hog and provide IPv4 as a service? That way you are not waiting for your customers to turn up IPv6 to take the load off your NAT box.Yes, you can do it dual stack but you have waited so long you may as well miss that step along the deployment path.-- Mark AndrewsOn 20 Feb 2021, at 01:55, Steve Saner  wrote:We are starting to look at CGNAT solutions. The primary motivation at the moment is to extend current IPv4 resources, but IPv6 migration is also a factor.We've been in touch with A10. Just wondering if there are some alternative vendors that anyone would recommend. We'd probably be looking at a solution to support 5k to 15k customers and bandwidth up to around 30-40 gig as a starting point. A solution that is as transparent to user experience as possible is a priority.Thanks-- Steve Sanerideatek HUMAN AT OUR VERY FIBERThis
 email transmission, and any documents, files or previous email messages
 attached to it may contain confidential information. If the reader of 
this message is not the intended recipient or the employee or agent 
responsible for delivering the message to the intended recipient, you 
are hereby notified that any dissemination, distribution or copying of 
this communication is strictly prohibited. If you are not, or believe 
you may not be, the intended recipient, please advise the sender 
immediately by return email or by calling 620.543.5026. Then take all steps necessary to permanently delete the email and all attachments from your computer system.

Re: AWS contact?

[Michael Crapse]

I would like to know as well who best to reach out to. We are experiencing
ipv6 related issues with AWS, unable to load even amazon.com completely
when any of our customers have ipv6 connectivity

curl -vvv
https://images-na.ssl-images-amazon.com/images/I/11EIQ5IGqaL._RC%7C01ZTHTZObnL.css
*   Trying 2a04:4e42::272...
* TCP_NODELAY set
* Connected to images-na.ssl-images-amazon.com (2a04:4e42::272) port 443
(#0)
* schannel: SSL/TLS connection with images-na.ssl-images-amazon.com port
443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 202 bytes...
* schannel: sent initial handshake data: sent 202 bytes
* schannel: SSL/TLS connection with images-na.ssl-images-amazon.com port
443 (step 2/3)
* schannel: failed to receive handshake, SSL/TLS connection failed
* Closing connection 0
* schannel: shutting down SSL/TLS connection with
images-na.ssl-images-amazon.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer
(bytes written: -1)
* schannel: clear security context handle
curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed




On Tue, 26 Jan 2021 at 11:00, Josh Baird  wrote:

> Are you sure it's not due to the Verizon outage?  As a non-customer, your
> options for contacting support are limited.
>
> On Tue, Jan 26, 2021 at 12:55 PM Justin Wilson (Lists) 
> wrote:
>
>> What is the best avenue for contacting support for AWS? I have
>> several ISPs experiencing reachability issues with AWS hosted sites.  These
>> are from different backbones, different gear, etc.  The common denominator
>> is AWS.
>>
>> Been googling around and can’t seem to find a contact.
>>
>>
>>
>> Justin Wilson
>> j...@mtin.net
>>
>> —
>> https://j2sw.com - All things jsw (AS209109)
>> https://blog.j2sw.com - Podcast and Blog
>>
>>

Re: CGNAT

[Mark Andrews]

Why not go whole hog and provide IPv4 as a service? That way you are not 
waiting for your customers to turn up IPv6 to take the load off your NAT box.

Yes, you can do it dual stack but you have waited so long you may as well miss 
that step along the deployment path.
-- 
Mark Andrews

> On 20 Feb 2021, at 01:55, Steve Saner  wrote:
> 
> 
> We are starting to look at CGNAT solutions. The primary motivation at the 
> moment is to extend current IPv4 resources, but IPv6 migration is also a 
> factor.
> 
> We've been in touch with A10. Just wondering if there are some alternative 
> vendors that anyone would recommend. We'd probably be looking at a solution 
> to support 5k to 15k customers and bandwidth up to around 30-40 gig as a 
> starting point. A solution that is as transparent to user experience as 
> possible is a priority.
> 
> Thanks
> 
> -- 
> Steve Saner
> ideatek HUMAN AT OUR VERY FIBER
> This email transmission, and any documents, files or previous email messages 
> attached to it may contain confidential information. If the reader of this 
> message is not the intended recipient or the employee or agent responsible 
> for delivering the message to the intended recipient, you are hereby notified 
> that any dissemination, distribution or copying of this communication is 
> strictly prohibited. If you are not, or believe you may not be, the intended 
> recipient, please advise the sender immediately by return email or by calling 
> 620.543.5026. Then take all steps necessary to permanently delete the email 
> and all attachments from your computer system.

Re: Anyone from Cloudflare peering lurking?

[Mike Hammett]

Ditto. 




Mine (multiple ASes) were up and then went down, not to be heard from again. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Bryan Holloway"  
To: "NANOG list"  
Sent: Friday, February 19, 2021 1:41:36 PM 
Subject: Anyone from Cloudflare peering lurking? 

Trying to get a few pubIX sessions up ... ping me off-list, s.v.p.? 

E-mails to the usual contacts aren't working. 

Thanks! 

Anyone from Cloudflare peering lurking?

[Bryan Holloway]

Trying to get a few pubIX sessions up ... ping me off-list, s.v.p.?

E-mails to the usual contacts aren't working.

Thanks!

Re: Texas internet connectivity declining due to blackouts

[Haudy Kazemi via NANOG]

Griddy's model makes sense for customers who have the ability to
automatically shed load and switch remaining critical load to backup
generation when wholesale prices spike above the cost of using the backup
generation. Might also make sense if the minimum load after load shedding
is small enough that $9/kWh is not going to break the budget. A 10 watt,
1000 lumen LED can run for 100 hours on 1 kWh.

Customers without such load shedding and/or backup are taking the risk of
possibly seeing $9/kWh (current TX cap?), assuming power is available at
all. Do their customers understand the risk?

It also appears Griddy is planning to roll out a 'price protection option'
for customers. I guess that option probably will look similar to variable
rate plans offered by other retail providers.



On Fri, Feb 19, 2021, 12:19 Tim Burke  wrote:

> CYA measure more than anything else, so Griddy can say they warned their
> customers that prices would be high when faced with chargebacks or bad
> press.
>
> Based on past experience, they are just passing through actual electric
> costs and profiting off of a ~$10 membership fee. After the absurd energy
> rates they had to pass through in 2019 (somewhere around $80/kWh), I'm
> amazed anyone still uses them.
>
> V/r
> Tim
>
> -Original Message-
> From: NANOG  On Behalf Of Mark Tinka
> Sent: Thursday, February 18, 2021 10:26 PM
> To: nanog@nanog.org
> Subject: Re: Texas internet connectivity declining due to blackouts
>
>
>
> On 2/17/21 16:09, Ben Cannon wrote:
>
> > https://www.dallasnews.com/business/energy/2021/02/16/electricity-reta
> > iler-griddys-unusual-plea-to-texas-customers-leave-now-before-you-get-
> > a-big-bill/
> >  > ailer-griddys-unusual-plea-to-texas-customers-leave-now-before-you-get
> > -a-big-bill/>
> >
> >
> > The power market in Texas has utterly failed.
>
> Griddy aren't greedy. Pity about the grid.
>
> Mark.
>

Re: Famous operational issues

[Warren Kumari]

At a previous company we had a large number of Foundry Networks layer-3
switches. They participated in our OSPF network and had a *really* annoying
bug. Every now and then one of them would get somewhat confused and would
corrupt its OSPF database (there seemed to be some pointer that would end
up off by one).

It would then cleverly realize that its LSDB was different to everyone
else's and so would flood this corrupt database to all other OSPF speakers.
Some vendors would do a better job of sanity checking the LSAs and would
ignore the bad LSAs, other vendors would install them... and now you have
different link state databases on different devices and OSPF becomes
unhappy.

Nov 24 22:23:53.633 EST: %OSPF-4-BADLSAMASK: Bad LSA mask: Type 5,
LSID 0.9.32.5

Mask 10.160.8.0 from 10.178.255.252
NOTE: This route will not be installed in the routing table.
Nov 26 11:01:32.997 EST: %OSPF-4-BADLSAMASK: Bad LSA mask: Type 5, LSID 0.4.2.3

Mask 10.2.153.0 from 10.178.255.252
NOTE: This route will not be installed in the routing table.
Nov 27 23:14:00.660 EST: %OSPF-4-BADLSAMASK: Bad LSA mask: Type 5, LSID 0.4.2.3

Mask 10.2.153.0 from 10.178.255.252
NOTE: This route will not be installed in the routing table.

 If you look at the output, you can see that there is some garbage in the
LSID field and the bit that should be there is now in the Mask section. I
also saw some more extreme version of the same bug, in my favorite example
the mask was 115.104.111.119 and further down there was 105.110.116.114 --
if you take these as decimal number and look up their ASCII values we get
"show" and "inte" -- I wrote a tool to scrape bits from these errors and
ended up with a large amount of the CLI help text.




Many years ago I worked for a small Mom-and-Pop type ISP in New York state
(I was the only network / technical person there) -- it was a very free
wheeling place and I built the network by doing whatever made sense at the
time.

One of my "favorite" customers (Joe somebody) was somehow related to the
owner of the ISP and was a gamer. This was back in the day when the gaming
magazines would give you useful tips like "Type 'tracert $gameserver' and
make sure that there are less than N hops".  Joe would call up tech
support, me, the owner, etc and complain that there was N+3 hops and most
of them were in our network. I spent much time explaining things about
packet-loss, latency, etc but couldn't shake his belief that hop count was
the only metric that mattered.

Finally, one night he called me at home well after midnight (no, I didn't
give him my home phone number, he looked me up in the phonebook!) to
complain that his gaming was suffering because it was "too many hops to get
out of your network". I finally snapped and built a static GRE tunnel from
the RAS box that he connected to all over the network -- it was a thing of
beauty, it went through almost every device that we owned and took the most
convoluted path I could come up with. "Yay!", I figured, "now I can
demonstrate that latency is more important than hop count" and I went to
bed.

The next morning I get a call from him. He is ecstatic and wildly impressed
by how well the network is working for him now and how great his gaming
performance is. "Oh well", I think, "at least he is happy and will leave me
alone now". I don't document the purpose of this GRE anywhere and after
some time forget about it.

A few months later I am doing some routine cleanup work and stumble across
a weird looking tunnel -- its bizarre, it goes all over the place and is
all kinds of crufty -- there are static routes and policy routing and
bizarre things being done on the RADIUS server to make sure some user
always gets a certain IP... I look in my pile of notes and old configs and
then decide to just yank it out.

That night I get an enraged call (at home again) from Joe *screaming* that
the network is all broken again because it is now way too many hops to get
out of the network and that people keep shooting him...

*What I learnt from this:*
1: Make sure you document everything (and no, the network isn't
documentation)
2: Gamers are weird.
3: Making changes to your network in anger provides short term pleasure but
long term pain.



On Fri, Feb 19, 2021 at 1:10 PM Andrew Gallo  wrote:

>
>
> On 2/16/2021 2:37 PM, John Kristoff wrote:
> > Friends,
> >
> > I'd like to start a thread about the most famous and widespread Internet
> > operational issues, outages or implementation incompatibilities you
> > have seen.
> >
> > Which examples would make up your top three?
>
>
> I don't believe I've seen this in any of the replies, but the AT
> cascading switch crashes of 1990 is a good one.  This link even has some
> pseudocode
> https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse
>
>

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra

RE: CGNAT

[Tony Wicks]

Not the Cheapest option out there but the most rock solid one I have found is 
to install the extended service/multi service cards in the BNG and do it 
locally there. We are currently using both Juniper MX480/960 with MS-MPC cards 
and Nokia 7750 SR with ISA or ESA cards. Its also well worth running dual stack 
IPv6 as you can bypass 40%+ traffic from the CGN process for all that CDN 
traffic.

 

From: NANOG  On Behalf Of Steve Saner
Sent: Friday, 19 February 2021 5:39 am
To: nanog@nanog.org
Subject: CGNAT

 

We are starting to look at CGNAT solutions. The primary motivation at the 
moment is to extend current IPv4 resources, but IPv6 migration is also a factor.

 

We've been in touch with A10. Just wondering if there are some alternative 
vendors that anyone would recommend. We'd probably be looking at a solution to 
support 5k to 15k customers and bandwidth up to around 30-40 gig as a starting 
point. A solution that is as transparent to user experience as possible is a 
priority.

 

Thanks


-- 

Steve Saner

ideatek HUMAN AT OUR VERY FIBER

This email transmission, and any documents, files or previous email messages 
attached to it may contain confidential information. If the reader of this 
message is not the intended recipient or the employee or agent responsible for 
delivering the message to the intended recipient, you are hereby notified that 
any dissemination, distribution or copying of this communication is strictly 
prohibited. If you are not, or believe you may not be, the intended recipient, 
please advise the sender immediately by return email or by calling  
 620.543.5026. Then take all steps necessary to permanently 
delete the email and all attachments from your computer system.

RE: Texas internet connectivity declining due to blackouts

[Tim Burke]

CYA measure more than anything else, so Griddy can say they warned their 
customers that prices would be high when faced with chargebacks or bad press. 

Based on past experience, they are just passing through actual electric costs 
and profiting off of a ~$10 membership fee. After the absurd energy rates they 
had to pass through in 2019 (somewhere around $80/kWh), I'm amazed anyone still 
uses them. 

V/r
Tim

-Original Message-
From: NANOG  On Behalf Of Mark Tinka
Sent: Thursday, February 18, 2021 10:26 PM
To: nanog@nanog.org
Subject: Re: Texas internet connectivity declining due to blackouts



On 2/17/21 16:09, Ben Cannon wrote:

> https://www.dallasnews.com/business/energy/2021/02/16/electricity-reta
> iler-griddys-unusual-plea-to-texas-customers-leave-now-before-you-get-
> a-big-bill/ 
>  ailer-griddys-unusual-plea-to-texas-customers-leave-now-before-you-get
> -a-big-bill/>
>
>
> The power market in Texas has utterly failed.

Griddy aren't greedy. Pity about the grid.

Mark.

Re: CGNAT

[Douglas Fischer]

I recommend you to take a look at DANOS.

https://danosproject.atlassian.net/wiki/spaces/DAN/pages/416153601/Carrier+Grade+NAT+CGNAT

- A very active open-source project.
- Sponsored by AT
- Uses Vyatta (and DPDK for good performance)
- The Routing Engine is based on FRR.
- Syntax sounds like Junos.
- Is the ONLY ONE open source project(at least that I know) that implements
CGNAT on Bulk Port Allocation mode(not deterministic/predefined).
- Had very good improvements on PCP recently.
- Supports a few NAT-ALGs.

I and some good friends here in Brazil had some good experiences with it.

Marcelo Gondin wrote this tutorial in pt_BR, mentioning about a case with:
26Gbps / 1.5Mpps / 11502 simultaneous clients / 192 used Públic IPv4
addresses.
https://wiki.brasilpeeringforum.org/w/CGNAT_Bulk_Port_Allocation_com_DPDK



Em sex., 19 de fev. de 2021 às 11:57, Steve Saner 
escreveu:

> We are starting to look at CGNAT solutions. The primary motivation at the
> moment is to extend current IPv4 resources, but IPv6 migration is also a
> factor.
>
> We've been in touch with A10. Just wondering if there are some alternative
> vendors that anyone would recommend. We'd probably be looking at a solution
> to support 5k to 15k customers and bandwidth up to around 30-40 gig as a
> starting point. A solution that is as transparent to user experience as
> possible is a priority.
>
> Thanks
>
> --
> Steve Saner
> ideatek HUMAN AT OUR VERY FIBER
>
> This email transmission, and any documents, files or previous email
> messages attached to it may contain confidential information. If the reader
> of this message is not the intended recipient or the employee or agent
> responsible for delivering the message to the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you are not, or believe you may
> not be, the intended recipient, please advise the sender immediately by
> return email or by calling 620.543.5026. Then take all steps necessary to
> permanently delete the email and all attachments from your computer system.
>


-- 
Douglas Fernando Fischer
Engº de Controle e Automação

Re: Famous operational issues

[Andrew Gallo]

On 2/16/2021 2:37 PM, John Kristoff wrote:

Friends,

I'd like to start a thread about the most famous and widespread Internet
operational issues, outages or implementation incompatibilities you
have seen.

Which examples would make up your top three?



I don't believe I've seen this in any of the replies, but the AT 
cascading switch crashes of 1990 is a good one.  This link even has some 
pseudocode

https://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse

Weekly Routing Table Report

[Routing Analysis Role Account]

This is an automated weekly mailing describing the state of the Internet
Routing Table as seen from APNIC's router in Japan.

The posting is sent to APOPS, NANOG, AfNOG, SANOG, PacNOG, SAFNOG
TZNOG, MENOG, BJNOG, SDNOG, CMNOG, LACNOG and the RIPE Routing WG.

Daily listings are sent to bgp-st...@lists.apnic.net

For historical data, please see http://thyme.rand.apnic.net.

If you have any comments please contact Philip Smith .

Routing Table Report   04:00 +10GMT Sat 20 Feb, 2021

Report Website: http://thyme.rand.apnic.net
Detailed Analysis:  http://thyme.rand.apnic.net/current/

Analysis Summary


BGP routing table entries examined:  846303
Prefixes after maximum aggregation (per Origin AS):  39
Deaggregation factor:  2.63
Unique aggregates announced (without unneeded subnets):  403355
Total ASes present in the Internet Routing Table: 70577
Prefixes per ASN: 11.99
Origin-only ASes present in the Internet Routing Table:   60708
Origin ASes announcing only one prefix:   25065
Transit ASes present in the Internet Routing Table:9869
Transit-only ASes present in the Internet Routing Table:298
Average AS path length visible in the Internet Routing Table:   4.3
Max AS path length visible:  62
Max AS path prepend of ASN (266299)  59
Prefixes from unregistered ASNs in the Routing Table:   974
Number of instances of unregistered ASNs:   978
Number of 32-bit ASNs allocated by the RIRs:  35117
Number of 32-bit ASNs visible in the Routing Table:   29149
Prefixes from 32-bit ASNs in the Routing Table:  136109
Number of bogon 32-bit ASNs visible in the Routing Table:16
Special use prefixes present in the Routing Table:1
Prefixes being announced from unallocated address space:598
Number of addresses announced to Internet:   2914957184
Equivalent to 173 /8s, 190 /16s and 183 /24s
Percentage of available address space announced:   78.7
Percentage of allocated address space announced:   78.7
Percentage of available address space allocated:  100.0
Percentage of address space in use by end-sites:   99.5
Total number of prefixes smaller than registry allocations:  288946

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:   221371
Total APNIC prefixes after maximum aggregation:   65337
APNIC Deaggregation factor:3.39
Prefixes being announced from the APNIC address blocks:  217208
Unique aggregates announced from the APNIC address blocks:88272
APNIC Region origin ASes present in the Internet Routing Table:   11277
APNIC Prefixes per ASN:   19.26
APNIC Region origin ASes announcing only one prefix:   3214
APNIC Region transit ASes present in the Internet Routing Table:   1602
Average APNIC Region AS path length visible:4.6
Max APNIC Region AS path length visible: 30
Number of APNIC region 32-bit ASNs visible in the Routing Table:   6431
Number of APNIC addresses announced to Internet:  770503680
Equivalent to 45 /8s, 236 /16s and 244 /24s
APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911, 45056-46079, 55296-56319,
   58368-59391, 63488-64098, 64297-64395, 131072-143673
APNIC Address Blocks 1/8,  14/8,  27/8,  36/8,  39/8,  42/8,  43/8,
49/8,  58/8,  59/8,  60/8,  61/8, 101/8, 103/8,
   106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8,
   116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8,
   123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8,
   163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8,
   203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8,
   222/8, 223/8,

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:243913
Total ARIN prefixes after maximum aggregation:   112217
ARIN Deaggregation factor: 2.17
Prefixes being announced from the ARIN address blocks:   244427
Unique aggregates announced from the ARIN address blocks:116579
ARIN Region origin ASes present in the Internet Routing Table:18734
ARIN Prefixes per ASN:13.05
ARIN 

Re: Famous operational issues

[Andrey Kostin]

Jen Linkova писал 2021-02-19 00:04:


OK, Warren, achievement unlocked. You've just made a network engineer
to google 'router'


He meant that we call "frezer" machine... (in our language ;)

I heard a similar story from my colleague who was working at that time 
for Huawei as DWDM engineer and had to fly frequently with testing 
devices.
One time he tried to explain at airport security control what DWDM 
spectrum analyser is for, the officer called another for help and he 
said something like this: "DWDM spectrum analyser? Pass it, usual 
thing..."


--
Kind regards,
Andrey Kostin

Re: Famous operational issues

[Aaron C. de Bruyn via NANOG]

All these stories remind me of two of my own from back in the late 90s.
I worked for a regional ISP doing some network stuff (under the real
engineer), and some software development.

Like a lot of ISPs in the 90s, this one started out in a rental house.
Over the months and years rooms were slowly converted to host more and more
equipment as we expanded our customer base and presence in the region.
If we needed a "rack", someone would go to the store and buy a 4-post metal
shelf [1] or...in some cases the dump to see what they had.

We had one that looked like an oversized filing cabinet with some sort of
rails on the sides.  I don't recall how the equipment was mounted, but I
think it was by drilling holes into the front lip and tapping the screws
in.  This was the big super-important rack.  It had the main router that
connected lines between 5 POPs around the region, and also several
connections to Portland Oregon about 60 miles away.  Since we were
making tons of money, we decided we should update our image and install
real racks in the "bedroom server room".  It was decided we were going to
do it with no downtime.

I was on the 2-man team that stood behind and in front of the rack with
2x4s dead-lifting them as equipment was unscrewed and lowered onto the
boards.  I was on the back side of the rack.  After all the equipment was
unscrewed, someone came in with a sawzall and cut the filing cabinet thing
apart.  The top half was removed and taken away, then we lifted up on the
boards and the bottom half was slid out of the way.  The new rack was
brought in, bolted to the floor, and then one by one equipment was taken
off the pile we were holding up with 2x4s, brought through the back of the
new rack, and then mounted.

I was pleasantly surprised and very relieved when we finished moving the
big router, several switches, a few servers, and a UPS unit over to the new
rack with zero downtime.  The entire team cheered and cracked beers.  I
stepped out from behind the rack...
...and snagged the power cable to the main router with my foot.  I don't
recall the Cisco model number after all this time...but I do remember the
excruciating 6-8 minutes it took for the damn thing to reboot, and the
sight of the 7 PRI cards in our phone system almost immediately jumping
from 5 channels in-use to being 100% full.

It's been 20 years, but I swear my arms are still sore from holding all
that equipment up for ~20 minutes, and I always pick my feet up very slowly
when I'm near a rack. ;)

The second story is a short one from the same time period.  Our POPs
consisted of the afore-mentioned 4-post metal shelves stacked with piles of
US Robotics 56k modems [2] stacked on top of each other.  They were wired
back to some sort of serial box that was in-turn connected to an ISA card
stuck in a Windows NT 4 server that used RADIUS to authenticate sessions
with an NT4 server back at the main office that had user accounts for all
our customers.  Every single modem had a wall-wart power brick for power,
an RJ11 phone line, and a big old serial cable.  It was an absolute rats
nest of cables.  The small POP (which I think was a TuffShed in someone's
yard about 50 feet from the telco building) was always 100 degrees--even in
the dead of winter.

One year we made the decision to switch to 3Com Total Control Chassis with
PRI cards.  The cut-over was pretty seamless and immediately made shelves
stacked full of hundreds of modems completely useless.  As we started
disconnecting modems with the intent of selling them for a few bucks to
existing customers who wanted to upgrade or giving them to new customers to
get them signed up, we found a bunch of the stacks of modems had actually
melted together due to the temps.  That explained the handful of numbers in
the hunt group that would just ring and ring with no answer.  In the end we
went from a completely packed 10x20 shed to two small 3Com TCH boxes packed
with PRI cards and a handful of PRI cables with much more normal
temperatures.

I thoroughly enjoyed the "wild west" days of the internet.

If Eric and Dan are reading this, thanks for everything you taught me about
networking, business, hard work, and generally being a good person.

-A

[1] -
https://www.amazon.com/dp/B01D54TICS/ref=redir_mobile_desktop?_encoding=UTF8=Pe4xuew1D1PkrRA9cq8Cdg_cr_id=5048111780901_rd_plhdr=t_rd_r=4d9e3b6b-3360-41e8-9901-d079ac063f03_rd_w=uRxXq_rd_wg=CDibq_=sbx_be_s_sparkle_td_asin_0_img

[2] - https://www.usr.com/products/56k-dialup-modem/usr5686g/



On Tue, Feb 16, 2021 at 11:39 AM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two 

Re: CoPP on NXOS

[Jay Ford]

Setting the "conform" & "violate" actions to "drop" for a class with
appropriate ACL matching seems to work:

   policy-map type control-plane copp-policy-whatever
 ! other classes ...
 class copp-class-undesirable-junk
   set cos 0
   police cir 32 kbps bc 310 ms conform drop violate drop
 ! other classes ...

The rates are irrelevant in that case, but still required.

_
Jay Ford, Network Engineering, University of Iowa
email: jay-f...@uiowa.edu, phone: 319-335-

On Wed, 17 Feb 2021, Drew Weaver wrote:

This might be a little too platform/vendor specific for this group so I 
apologize in advance
if that is the case.

 

Does anyone have a working example of CoPP on NXOS which limits things like 
BGP, SSH, and the
NXAPI HTTPS interface to a specific remote /32 and blocks everything else that 
is not
specifically allowed in the ACLs attached to the classes?

 

I’ve had a ticket open /w TAC for a month and I’m actually getting nowhere.

 

Thank you so much,

-Drew

CGNAT

[Steve Saner]

We are starting to look at CGNAT solutions. The primary motivation at the
moment is to extend current IPv4 resources, but IPv6 migration is also a
factor.

We've been in touch with A10. Just wondering if there are some alternative
vendors that anyone would recommend. We'd probably be looking at a solution
to support 5k to 15k customers and bandwidth up to around 30-40 gig as a
starting point. A solution that is as transparent to user experience as
possible is a priority.

Thanks

-- 
Steve Saner
ideatek HUMAN AT OUR VERY FIBER

This email transmission, and any documents, files or previous email
messages attached to it may contain confidential information. If the reader
of this message is not the intended recipient or the employee or agent
responsible for delivering the message to the intended recipient, you are
hereby notified that any dissemination, distribution or copying of this
communication is strictly prohibited. If you are not, or believe you may
not be, the intended recipient, please advise the sender immediately by
return email or by calling 620.543.5026. Then take all steps necessary to
permanently delete the email and all attachments from your computer system.

Re: Texas internet connectivity declining due to blackouts

[Mal via NANOG]

On 18/02/2021 7:54 am, Milt Aitken wrote:
> The bill arrived today.  $391.26 got me 3459kwh.  That is 11.3cents/kwh net 
> for business power from Cobb EMC, who charges a good bit more than GPC (they 
> buy a lot of their power from GPC).  N the past, I’ve had GPC bills from 
> customers’ homes that net to about 7.5 cents/kwh.

Try paying for electricity in South Australia.. its around $0.34/kwh
(plus %10 tax).  We have a regime where essentially those without solar
are paying for those with.

South Australia too had a complete power grid failure in 2016. 
Regardless of the ample coal supply the state has, we had Power stations
knocked down for green sources, like wind and solar.   During the storm,
we had a cascading transmission network failure with 23 pylons taken out
which effected our supply from other Australian states..  And the lights
went dark, statewide.

Its 38°C here today (100°F).  Thaw out Texas.

Re: Famous operational issues

[Daniel Karrenberg]

On 16 Feb 2021, at 20:37, John Kristoff wrote:

I'd like to start a thread about the most famous and widespread 
Internet

operational issues, outages or implementation incompatibilities you
have seen.

Which examples would make up your top three?



My absolute top one happened 1995. Traffic engineering was not a widely 
used term then. A bright colleague who will remain un-named decided that 
he could make AS paths longer by repeating the same AS number more than 
once. Unfortunately the prevalent software on CISCO routers was not 
resilient to such trickery and reacted with a reboot. This caused an 
avalanche of jo-jo-ing routers. Think it through!


It took some time before that offending path could be purged from the 
whole Internet; yes we all roughly knew the topology and the players of 
the  BGP speaking parts of it at that time.  Luckily this happened 
during the set-up for the Danvers IETF and co-ordination between major 
operators was quick because most of their routing geeks happened to be 
in the same room, the ‘terminal room’; remember those?


Since at the time I personally had no responsibility for operations any 
more I went back to pulling cables and crimping RJ45s.


Lessons: HW/SW mono-cultures are dangerous. Input testing is good 
practice at all levels software. Operational co-ordination is key in 
times of crisis.


Daniel

Re: Famous operational issues

[Jen Linkova]

On Fri, Feb 19, 2021 at 9:40 AM Warren Kumari  wrote:
> 4: Not too long after I started doing networking (and for the same small ISP 
> in Yonkers), I'm flying off to install a new customer. I (of course) think 
> that I'm hot stuff because I'm going to do the install, configure the router, 
> whee, look at me! Anyway, I don't want to check a bag, and so I stuff the 
> Cisco 2501 in a carryon bag, along with tools, etc (this was all pre-9/11!). 
> I'm going through security and the TSA[0] person opens my bag and pulls the 
> router out. "What's this?!" he asks. I politely tell him that it's a router. 
> He says it's not. I'm still thinking that I'm the new hotness, and so I tell 
> him in a somewhat condescending way that it is, and I know what I'm talking 
> about. He tells me that it's not a router, and is starting to get annoyed. I 
> explain using my "talking to a 5 year old" voice that it most certainly is a 
> router. He tells me that lying to airport security is a federal offense, and 
> starts looming at me. I adjust my attitude and start explaining that it's 
> like a computer and makes the Internet work. He gruffly hands me back the 
> router, I put it in my bag and scurry away. As I do so, I hear him telling 
> his colleague that it wasn't a router, and that he certainly knows what a 
> router is, because he does woodwork...

OK, Warren, achievement unlocked. You've just made a network engineer
to google 'router'

P.S. I guess I'm obliged to tell a story if I respond to this thread...so...
"Servers and the ice cream factory".
Late spring/early summer in Moscow. The temperature above 30C (86°F).
I worked for a local content provided.
Aircons in our server room died, the technician ETA was 2 days ( I
guess we were not the only ones with aircon problems).
So we drove to the nearby ice cream factory  and got *a lot* of  dry
ice. Then we have a roaster: every few hours one person took a deep
breath, grabbed a box of dry ice, ran into the server room and emptied
the box on top of the racks. The backup person was watching through
the glass door - just in case, you know, ready to start the rescue
operation.
We (and the servers) survived till the technician arrived. And we had
a lot of dry ice to cool the beer..

-- 
SY, Jen Linkova aka Furry

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Ge DUPIN]

As you can guess from the name of the capital, Sierra Leone is an English 
speaking country
Ge

> Le 19 févr. 2021 à 12:19, Rod Beck  a écrit :
> 
> I am sure South Africa is better. I am really referring to French speaking 
> Western Africa. 
> 
> -R. 
> 
> From: NANOG  > on behalf 
> of Mark Tinka mailto:mark@tinka.africa>>
> Sent: Friday, February 19, 2021 5:09 AM
> To: nanog@nanog.org   >
> Subject: Re: Carrier Neutral Site - Freetown, Sierra Leone?
>  
> 
> 
> On 2/18/21 19:45, Rod Beck wrote:
> 
>> Every time I try to bring a circuit into Africa it is like a complete tour 
>> of Dante's Hell. 
> 
> A broad brush for such a large place.
> 
> Mark.

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Mark Tinka]

On 2/19/21 14:34, Martijn Schmidt wrote:

My own admittedly relatively limited experience with networking in 
Africa has always been that people truly, honestly want to help make 
things better whenever it's reasonably possible (e.g. you obviously 
can't break local laws, or ignore a regulator). The attitude is 
generally very very good, compared to what we've seen in some other 
regions of the world. That being said, I have no experience with 
Sierra Leone in particular so can't help out in this case..


MainOne, Glo-1 and ACE, would be the current major routes into West Africa.

I know some people there and can reach out, if they aren't lurking here 
already.


Mark.

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Martijn Schmidt via NANOG]

My own admittedly relatively limited experience with networking in Africa has 
always been that people truly, honestly want to help make things better 
whenever it's reasonably possible (e.g. you obviously can't break local laws, 
or ignore a regulator). The attitude is generally very very good, compared to 
what we've seen in some other regions of the world. That being said, I have no 
experience with Sierra Leone in particular so can't help out in this case..

Best regards,
Martijn

On 2/19/21 12:57 PM, Mark Tinka wrote:


On 2/19/21 13:47, Jörg Kost wrote:


Hello,

I am sure it could resolve the discussion and the topic easier and more 
helpful, if you can line out what exactly is the issue and where help is 
needed, and not comparing and generalizing the order of a circuit for a whole 
continent to walking or passing by the idea of a hell. That is pretty unfair.

We don’t have any business in Africa, but I’d like to listen and understand.

This is what I'm also trying to get to, albeit the long-way around.

That said, the OP did speak of Freetown, so.

If I can get more detail, I can reach out to some folk that run network in 
those parts and see if they can help.

My/our expertise is more eastern and southern Africa.

Mark.

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Mark Tinka]

On 2/19/21 13:47, Jörg Kost wrote:


Hello,

I am sure it could resolve the discussion and the topic easier and 
more helpful, if you can line out what exactly is the issue and where 
help is needed, and not comparing and generalizing the order of a 
circuit for a whole continent to walking or passing by the idea of a 
hell. That is pretty unfair.


We don’t have any business in Africa, but I’d like to listen and 
understand.




This is what I'm also trying to get to, albeit the long-way around.

That said, the OP did speak of Freetown, so.

If I can get more detail, I can reach out to some folk that run network 
in those parts and see if they can help.


My/our expertise is more eastern and southern Africa.

Mark.

Re: Famous operational issues

[Owen DeLong]

In the case of Exodus when I was working there, it was literally dictated to us 
by
the fire marshal of the city of Santa Clara (and enough other cities where we 
had
datacenters to make a universal policy the only sensible choice).

Owen
 
> On Feb 18, 2021, at 1:07 AM, Eric Kuhnke  wrote:
> 
> On that note, I'd be very interested in hearing stories of actual incidents 
> that are the cause of why cardboard boxes are banned in many facilities, due 
> to loose particulate matter getting into the air and setting off very 
> sensitive fire detection systems.
> 
> Or maybe it's more mundane and 99% of the reason is people unpack stuff and 
> don't always clean up properly after themselves.
> 
> On Wed, Feb 17, 2021, 6:21 PM Owen DeLong  > wrote:
> Stolen isn’t nearly as exciting as what happens when your (used) 6509 arrives 
> and
> gets installed and operational before anyone realizes that the conductive 
> packing
> peanuts that it was packed in have managed to work their way into various 
> midplane
> connectors. Several hours later someone notices that the box is quite 
> literally
> smoldering in the colo and the resulting combination of panic, fire drill, and
> management antics that ensue.
> 
> Owen
> 
> 
> > On Feb 16, 2021, at 2:08 PM, Jared Mauch  > > wrote:
> > 
> > I was thinking about how we need a war stories nanog track. My favorite was 
> > being on call when the router was stolen. 
> > 
> > Sent from my TI-99/4a
> > 
> >> On Feb 16, 2021, at 2:40 PM, John Kristoff  >> > wrote:
> >> 
> >> Friends,
> >> 
> >> I'd like to start a thread about the most famous and widespread Internet
> >> operational issues, outages or implementation incompatibilities you
> >> have seen.
> >> 
> >> Which examples would make up your top three?
> >> 
> >> To get things started, I'd suggest the AS 7007 event is perhaps  the
> >> most notorious and likely to top many lists including mine.  So if
> >> that is one for you I'm asking for just two more.
> >> 
> >> I'm particularly interested in this as the first step in developing a
> >> future NANOG session.  I'd be particularly interested in any issues
> >> that also identify key individuals that might still be around and
> >> interested in participating in a retrospective.  I already have someone
> >> that is willing to talk about AS 7007, which shouldn't be hard to guess
> >> who.
> >> 
> >> Thanks in advance for your suggestions,
> >> 
> >> John
> 

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Jörg Kost]

Hello,

I am sure it could resolve the discussion and the topic easier and more 
helpful, if you can line out what exactly is the issue and where help is 
needed, and not comparing and generalizing the order of a circuit for a 
whole continent to walking or passing by the idea of a hell. That is 
pretty unfair.


We don’t have any business in Africa, but I’d like to listen and 
understand.


Regards
Jörg


On 19 Feb 2021, at 12:19, Rod Beck wrote:

I am sure South Africa is better. I am really referring to French 
speaking Western Africa.


-R.


From: NANOG  
on behalf of Mark Tinka 

Sent: Friday, February 19, 2021 5:09 AM
To: nanog@nanog.org 
Subject: Re: Carrier Neutral Site - Freetown, Sierra Leone?



On 2/18/21 19:45, Rod Beck wrote:

Every time I try to bring a circuit into Africa it is like a complete 
tour of Dante's Hell.


A broad brush for such a large place.

Mark.

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Mark Tinka]

On 2/19/21 13:19, Rod Beck wrote:

I am sure South Africa is better. I am really referring to French 
speaking Western Africa.


Well, South Africa is just one country out of 54.

And French-speaking West Africa is also a multitude of countries.

Mark.

Re: Carrier Neutral Site - Freetown, Sierra Leone?

[Rod Beck]

I am sure South Africa is better. I am really referring to French speaking 
Western Africa.

-R.


From: NANOG  on behalf 
of Mark Tinka 
Sent: Friday, February 19, 2021 5:09 AM
To: nanog@nanog.org 
Subject: Re: Carrier Neutral Site - Freetown, Sierra Leone?



On 2/18/21 19:45, Rod Beck wrote:

Every time I try to bring a circuit into Africa it is like a complete tour of 
Dante's Hell.

A broad brush for such a large place.

Mark.

Re: Famous operational issues

[Wolfgang Tremmel]

Do you remember the Cisco HDCI connectors? 
https://en.wikipedia.org/wiki/HDCI

I once shipped a Cisco 4500 plus some cables to a remote data center and asked 
the local guys to cable them for me.
With Cisco you could check the cable type and if they were properly attached. 
They were not.

I asked for a check and the local guy confirmed me three times that the cables 
were properly plugged. 
At the end I gave up, and took the 3 hour drive to the datacenter to check 
myself.

Problem was that, while the casing of the connector is asymmetrical, the pins 
inside are symmetrical.
And the local guy was quite strong.

Yes, he managed to plug in the cables 180° flipped, bending the case, but he 
got them in.
He was quite embarrassed when I fixed the cabling problem in 10 seconds.

That must have been 1995 or so

Wolfgang



> On 16. Feb 2021, at 20:37, John Kristoff  wrote:
> 
> Which examples would make up your top three?

-- 
Wolfgang Tremmel 

Phone +49 69 1730902 0  | wolfgang.trem...@de-cix.net
Executive Directors: Harald A. Summa and Sebastian Seifert | Trade Registry: AG 
Cologne, HRB 51135
DE-CIX Management GmbH | Lindleystrasse 12 | 60314 Frankfurt am Main | Germany 
| www.de-cix.net

Re: Famous operational issues

[Mark Tinka]

On 2/19/21 10:40, Suresh Ramasubramanian wrote:

He is. He asked a perfectly relevant question based on what he saw of 
the physical setup in front of him.


And he kept his cool when being talked down to.

I’d hire him the next minute, personally speaking.



In the early 2000's, with that level of deduction, I'd have been 
surprised if he wasn't snatched up quickly. Unless, of course, it 
ultimately wasn't his passion.


Mark.

Re: Famous operational issues

[Suresh Ramasubramanian]

He is. He asked a perfectly relevant question based on what he saw of the 
physical setup in front of him.

And he kept his cool when being talked down to.

I’d hire him the next minute, personally speaking.

From: Sabri Berisha 
Date: Friday, 19 February 2021 at 2:02 PM
To: Suresh Ramasubramanian 
Cc: nanog 
Subject: Re: Famous operational issues
On Feb 18, 2021, at 11:51 PM, Suresh Ramasubramanian  
wrote:

>> On 2/19/21 00:37, Warren Kumari wrote:

>> and says "'K. So, you doing a full iBGP mesh, or confeds?". I really hadn't
>> intended to be a condescending ass, but I think of that every time I realize 
>> I
>> might be assuming something about someone based on thier attire/job/etc.

> Did you at least hire the janitor?

Well, it's funny that you mention that because I worked at a place where the
company ended up hiring a young lady who worked in the cafeteria. When she
graduated she was offered a job in HR, and turned out to be absolutely awesome.

At some point in my life, I was carrying 50lbs bags of potato starch. Now I have
two graduate degrees and am working on a third. That janitor may be awesome, 
too!

Thanks,

Sabri

Re: Famous operational issues

[Sabri Berisha]

On Feb 18, 2021, at 11:51 PM, Suresh Ramasubramanian  
wrote: 

>> On 2/19/21 00:37, Warren Kumari wrote:

>> and says "'K. So, you doing a full iBGP mesh, or confeds?". I really hadn't
>> intended to be a condescending ass, but I think of that every time I realize 
>> I
>> might be assuming something about someone based on thier attire/job/etc.

> Did you at least hire the janitor?

Well, it's funny that you mention that because I worked at a place where the
company ended up hiring a young lady who worked in the cafeteria. When she
graduated she was offered a job in HR, and turned out to be absolutely awesome.

At some point in my life, I was carrying 50lbs bags of potato starch. Now I have
two graduate degrees and am working on a third. That janitor may be awesome, 
too!

Thanks, 

Sabri