Re: Opengear alternatives that support 5g?

[Jared Mauch]

If someone wants to assemble some of the freetserv boxes, I have some of the 
PCBs and components here if you want them.

- Jared

> On Apr 26, 2024, at 1:27 PM, Andrew Latham  wrote:
> 
> If anyone is interested in https://freetserv.github.io/ but does not want to 
> build one I have sort of documented an alternative at 
> https://lathama.net/Tech/Hardware/USB-32COM-RM so you can use anything to 
> connect the 5G or dialup to
> 
> On Fri, Apr 26, 2024 at 11:21 AM Michel Blais  
> wrote:
> Doesn't meet #3 but I'm testing Banana Pi BPI-R3 and seems way better than 
> RPI for this purpose.
> 
> You need to add the mini-pci modem of your choice but their 2 SIM card slots 
> on board. There are also 5 RJ45 ports if your devices have OOB ethernet ports.
> 
> There are 2 onboard storage (NOR and NAND) and you can add a M2 SSD so it is 
> possible to have failover disks.
> 
> I also like the fact that there are 2 SFP ports. There are some places in our 
> area where the LTE / 5G network is really awful so we can use a fiber 
> wavelength instead. It depends on the same fiber but at least, doesn't depend 
> on any active devices on site.
> 
> The bad is that you still need a USB to serial ports adapter. Also, you can 
> customise OpenWRT as much as you like.
> 
> For me, it's an advantage but in your case, it seems like an issue. For the 
> OP, having several VPN options like zerotier seems like an advantage. 
> 
> Le ven. 26 avr. 2024, à 12 h 44, Warren Kumari  a écrit :
> 
> 
> 
> 
> On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti  wrote:
> On Fri, 26 Apr 2024 at 03:11, David H  wrote: 
> Curious if anyone has particular hardware they like for OOB / serial 
> management, similar to OpenGear, but preferably with 5G support, maybe even 
> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
> machine accounts out of Verizon, and the added speed would be nice too. Or do 
> you separate the serial from the access device (cell+firewall, etc.)? 
> You could get a 5G Catalyst with an async NIM or SM. 
> But I think you're setting up yourself for unnecessary costs and failures by 
> designing your OOB to require static IP. You could design it so that the OOB 
> spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
> they come from, using certificates or PSK for identity, instead of IP.
> 
> 
> Yup, I agree — but that simply rewrites the question to be:
> "Curious if anyone has particular hardware they like for OOB / serial 
> management, similar to OpenGear, but preferably with 5G support, which can be 
> a spoke that dials in to the central OOB hub, and the OOB hub doesn't care 
> what IP they come from, using certificates or PSK for identity, instead of 
> IP."
> 
> I've been on the same quest, and I have some additional requests / features. 
> Ideally it:
> 
> 1: would be small - my particular use-case is for a "traveling rack", and so 
> 0U is preferred.
> 
> 2: would be fairly cheap.
> 
> 3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
> that for a while, and it was clunky — the SD card died a few times (and 
> jumped out entirely once!), people kept futzing with the OS and fighting over 
> which console software to use, installing other packages, etc.
> 
> 4: support modern SSH clients (it seems like you shouldn't have to say this, 
> but… )
> 
> 5: actually be designed as a termserver - the current thing we are using 
> doesn't really understand terminals, and so we need to use 'socat 
> -,raw,echo=0,escape=0x1d TCP::' to get things like 
> tab-completion and "up-arrow for last command" to work.
> 
> 6: support logging of serial (e.g crash-messages) to some sort of log / 
> buffer / similar (it's useful to be able to see what a device barfed all over 
> the console when it crashes. 
> 
> 
> The Get Console Airconsole TS series meets many of these requirements, but it 
> doesn't do #6. It also doesn't really feel like they have been updating / 
> maintaining these.
> 
> Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts 
> when I do this" camp, but, well…
> 
> W
> 
> 
> 
> -- 
> ++ytti
> 
> 
> 
> -- 
> - Andrew "lathama" Latham -

Re: Whitebox Routers Beyond the Datasheet

[Jared Mauch]

> On Apr 13, 2024, at 12:15 AM, 7ri...@gmail.com wrote:
> 
> 
>> I feel like this shouldn't be listed on a data sheet for just the whitebox 
>> hardware. The software running on it would be the gating factor.
> There would be two things ... BGP convergence, and then the time required to 
> get routes from the RIB into the hardware forwarding tables. These are 
> completely separate things. Both are gated on software for the most part, and 
> it would be hard to measure them unless you know a lot more about the 
> environment. Even then it would be a bit of a guess.
> 
> Contact me off list if you're interested in prior experience in this area.
> 
> :-) /r


Yeah, I think the question is coming from the wrong direction, which is what 
route scale do you need then match it to the hardware.  You can load a variety 
of software on these devices, including putting something like cRPD on top of 
it so you have the Juniper software and policy language, or roll your own with 
FRR, BIRD or something else.

The kernel -> FIB (hardware) download performance will vary as will the way the 
TCAM is carved up into the various routes and profiles.

It also depends on what you download to the FIB vs what you have in your RIB, 
for example a fib-filter in Juniper parlance may give you the ability to carry 
a full routing table but just a default and your local stub routes depending on 
the device role.  (Connected/static + local iBGP+eBGP learned)

- Jared

Re: Akamai contact

[Jared Mauch]

On Wed, Mar 20, 2024 at 06:26:24PM +, Liviu Danicel wrote:
> Hello,
> 
> Anyone from Akamai on the list ? I have a banned /24 subnet and would like to 
> discuss it.

Yes, I would suggest checking this page as well from an IP where
you are experiencing the issue.

https://www.akamai.com/us/en/clientrep-lookup/

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Fiber aggregators and such

[Jared Mauch]

With all the $ being spent expanding fiber in the last mile, I’ve got a theory 
that a lot of new and diverse fiber routes are being built between locations.

There’s a few places I know that roll up some of this information, but I’m 
wondering if there’s anyone rolling this all up either publicly or privately.

On-list and off-list replies welcome.

- Jared

Re: puck not responding

[Jared Mauch]

On Sat, Mar 02, 2024 at 11:55:45AM +0100, Bjørn Mork wrote:
> George Herbert  writes:
> 
> > If it wasn’t for how clunky they are with email sites, I’d suggest
> > moving to a cloud somewhere.  But …
> 
> I believe statistics point in favour of the single puck.nether.net
> host
> 
> BTW, for anyone else taking advantage of the excellent secondary service
> provided by puck: You might want to update your AXFR ACLs.  It seems the
> IPv6 address has changed.
> 
> I must admit that such transfer failures go unnoticed due to the large
> volume of unwanted requests.  So I appreciate the extra effort sending
> an email warning when a zone i disabled.

Yes, I'm notifying people now and have updated the FAQ/docs
page.  I also said there that I would notify people if the geography of
the machine changed and it has.

I still need to get my upstreams to notify all their upstreams
to permit packets as there's one provider that does uRPF in the mix, so
I have blocked their routes for now.

> Thanks for running all these high quality services!

it's the sustained community efforts that have allowed technology to
improve to the point where auto-updates and many other things are
without trouble, sadly i had to do a bit of physical moving of things,
but the machine should now have a ~10g uplink and if I can find the
right 100g device that I'm happy with I'm in a better position to
update/upgrade it now compared to a week ago.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: puck not responding

[Jared Mauch]

> On Feb 29, 2024, at 10:56 AM, Jay Acuna  wrote:
> 
> On Thu, Feb 29, 2024 at 9:22 AM Jared Mauch  wrote:
>> 
> 
> Apparently some of the most important email lists, Outages, etc, are
> being kept online by 1 person's  Unix/Linux server.
> 

There’s other people who have access etc, but when it comes to hardware that is 
quite old, last substantive refresh was in 2011, it’s served its purpose well.

Obligatory xkcd https://xkcd.com/2347/

Re: puck not responding

[Jared Mauch]

> On Feb 28, 2024, at 1:30 AM, Daniel Marks via NANOG  wrote:
> 
> We’re getting rocked by storms here in Michigan, could be related.
> 

[ brief version of what happened from what I can tell reconstructing things]

I was alerted ~4am US/E yesterday about the issue.  This machine has been 
generously hosted by my previous employer for quite some time, funnily enough 
it was 7 years ago almost to the day since I started my current employment.

The IPMI was not responsive and the machine was located in 350 Cermak, on a 
floor that was not impacted with the heat/cold event.

I have been meaning to move things off and on, but never quite had the 
motivation to tackle the task.  Yesterday forced my hand.

Once I confirmed that we could get the machine out of the colocation facility 
(thank you again NTT) I drove from Michigan to Chicago, got lunch and picked up 
the machine and headed back to the colocation that I have in Michigan at the 
123Net/DetroitIX site.

Once I had a console on it, I determined that this old machine had a few things 
that had been gradually updated and upgraded over time, not all the filesystem 
options were set correctly and after some tune2fs options were set and fstab 
updated to ensure everything is migrated fully from ext2 -> ext4 the system was 
able to be booted without issues.

Afterwards I’ve determined that there is still a hardware related problem, so I 
am now going to move it to new hardware later today schedule permitting as I 
want to go onsite and make sure that the I/O is performant.

Feb 28 22:09:05 kernel: Memory: 32816872K/33544380K available (20480K kernel 
code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 727248K reserved, 0K 
cma-reserved)
Feb 29 00:20:07 kernel: Memory: 16326408K/16767164K available (20480K kernel 
code, 3276K rwdata, 14748K rodata, 4588K init, 4892K bss, 440496K reserved, 0K 
cma-reserved)

Not quite a great thing when nobody is onsite and the machine requires being 
power cycled and the amount of memory changes.

If you are seeing any other issues, do let me know, I did move the IPv4 space 
but have renumbered for v6, so if you use my free secondary dns service, and 
your own vanity name, you will need to update your  records.

If you are seeing any reachability issues let me know, there should be ROA and 
other objects in place for things.

Sorry everyone got this email, feel bad it’s like when warren asked the list 
some personal details :-)

- Jared


(Even more details: changing disk images from qcow -> qcow2 and other things 
like ext2 -> ext3/4 over all the years as the machine has gone from Linux -> 
FreeBSD -> Linux again and other things is always a fun way to keep bringing 
your legacy around with you, it’s good overall)

sendgrid.net contact

[Jared Mauch]

If you have a contact there can you please contact me off-list.  Thanks.

- Jared

Re: Internet Exchange Visualization

[Jared Mauch]

> On Aug 15, 2023, at 3:35 PM, Randy Bush  wrote:
> 
>> actually, i am amazed by the extent of "remote peering."  if one
>> measures rtt to all the peers on the six, for example, the curve goes
>> out to well over 200ms.  the six has seen remote peers from the gulf
>> states, and i do not mean louisiana.
>> 
>> graph below is one way to visualize ix connectivity, the op's
>> question.
> 
> i guess the list does not like graphs.  decline of net predicted; news
> at eleven.  if you care, unicast.

At $priorjob we had a latency requirement, must be I think it was ~2ms away.

The number of IX that are effectively a global transit backbone is quite odd.

I would be interested in seeing the graphic, if you can unicast or post URL.

- Jared

Re: Hawaiian ILEC infrastructure and fire

[Jared Mauch]

> On Aug 17, 2023, at 1:55 PM, TJ Trout  wrote:
> 
> I'm familiar with the island, it's it's puzzling that the major 3 cell 
> carriers would accept a single point of failure like that, you would think 
> they had microwave backup at minimum. Maybe it was a generator issue. 
> 

It’s common for a lot of cell backhaul to be on poles because it’s seen as 
cheap and easy to repair, except when you have a lot of failures at once, and 
I’m guessing replacement poles require a bit more effort to get there like 
everything else.

I’m reminded of the fiber routes in/out of Madrid where they are all (largely) 
on a single route.  There is a diverse route but it’s $$$ and some providers 
don’t want to pay for it.

There’s a number of towns in the US that don’t even have any fiber connectivity 
at all, the voice/data come in via licensed links.

It’s also common that something is missed or gets groomed off a diverse path 
without your knowledge.  Seen this many times.

- Jared

Re: Flapping Transport

[Jared Mauch]

> On Aug 1, 2023, at 2:18 PM, Mike Hammett  wrote:
> 
> I have a wave transport vendor that suffered issues twice about ten days 
> apart, causing my link to flap a bunch. I put in a ticket on the second set 
> of occurrences. I was told that there was a card issue identified and would 
> be notified when the replacement happened. Ticket closed.
> 
> Three weeks later, I opened a new ticket asking for the status. The new card 
> arrived the next day, but since no more flaps were happening, the card would 
> not be replaced. Ticket closed.
> 
> 
> A) It doesn't seem like they actually did anything to fix the circuit.
> B) They admitted a problem and sent a new card.
> C) They later decided to not do anything.
> 
> 
> Is that normal?
> Is that acceptable?
> 
> 
> To avoid issues flapping causes, I disabled that circuit until repaired, but 
> it seems like they're not going to do anything and I only know that because I 
> asked.


With passive components like amplifiers and such, or they might have had 
someone do work that they don’t want to fess up to (which is kinda silly) I get 
that.

I have our junipers configured with a 5 second up timer, eg: "hold-time up 5000”

This way a flapping circuit must be stable for at least a few seconds before it 
can be placed back into service, otherwise if you have a prefix that comes from 
connected/direct/static/qualified-next-hop it won’t go into another protocol 
and possibly cause a globally visible BGP event.

Some providers have a much more disruptive layer-1 infrastructure and will ask 
you to configure a 1s+ up timer.  I think there’s an interesting question that 
could go either way, do you want transport side faults to be exposed to you, or 
should the client interface in a system be held up so you don’t have that fault 
condition forward (sometimes called FDI) to the client interface.

They may have had the system misconfigured so you saw a fault on a protected 
path when there was a switch.  It can take some time for the transponder to 
re-tune if the timing is different if your A path is 25km and B side is 5km and 
you have a optical switch, with the higher PHY rates it will take some extra 
time.

I know that Cisco also has these interface timers, but some of the others may 
not (eg: I don’t know if Mikrotik has them, but queue the wiki in a reply).

If it’s stable for 48 hours, I would place it back into service, but you should 
escalate at the same time and determine if they were truly hands off.  It may 
be a fiber was bent and is now fixed and that actually was the root cause.

Hope this helps you and a few others.

- jared

Re: Do ISP's collect and analyze traffic of users?

[Jared Mauch]

> On May 16, 2023, at 2:57 PM, Michael Thomas  wrote:
> 
> 
> On 5/16/23 7:35 AM, Livingood, Jason via NANOG wrote:
>> +1 to what Josh writes below. I would also differentiate between mobile 
>> networks (service provisioned to individual devices & often carrier s/w on 
>> the device) and wireline networks (home devices behind a router/gateway/NAT).
>>  
>> I just don't think sale of data is a business for wireline ISPs. If it were 
>> - given most companies are public - you'd see it in SEC 10K filings and on 
>> earnings calls. Indeed, they'd be required to talk about it with investors 
>> if it was a material revenue stream. I see none of that. Rather, the focus 
>> is on subscription revenue. If you want to know about data monetization - 
>> focus on services you don't pay for...
>> 
> Why would there be a difference between wireless and wired? 

If you purchase MVNO from someone, those providers may do something else.  I 
think it’s also a bit interesting because some providers previously attempted 
to monetize this data, either through DNS wildcarding vs NXDOMAIN.  If it’s 
just generic “someone asked for this name” vs “this CIDR or IP requested data”.

https://tech.slashdot.org/story/14/03/11/1813226/crowdsourcing-confirms-websites-inaccessible-on-comcast

A reminder that what’s old is new again on the internet, so I’m sure we’ll see 
things come back around, bad ideas continue to come up with revenue ideas.

- Jared

Re: New addresses for b.root-servers.net

[Jared Mauch]

I know when I did the openresolver project stuff I saw a number that would send glue or referrals from before they moved to the root servers domain names. - Jared Sent via RFC1925 compliant deviceOn Jun 2, 2023, at 8:49 AM, Nathan Ward  wrote:

On 2/06/2023 at 10:22:46 AM, Wes Hardaker  wrote:



2. I'll note that we are still serving DNS requests at the addresses thatwe switched away from in 2017 [1][2].  At that time we actually onlypromised 6 months and we've doubled that time length with our latestannounced change.  But we do need a date after which we can turn offservice to an address block if some reason demands it.



Hi Wes,Seems to me that this could be heavily informed by historical data from this earlier renumbering.Do you have query rates over time for the old and new addresses since this change in 2017?Even if you end up with the same answer of 12mo, data supporting it may give comfort to the community.Maybe you make a call that once it’s at say 1% or 0.1% or something like that, then it’s OK to turn off - and make a prediction for when that might be based on the historical data.--Nathan Ward

Re: Routed optical networks

[Jared Mauch]

> On May 11, 2023, at 11:11 AM, Mark Tinka  wrote:
> 
> 
> 
> On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote:
> 
>> Hi Jared,
>> Could I make a conclusion from your comments: "only Carrier itself 
>> understand the traffic - see many examples in the text".
>> I would very agree to this.
> 
> I wouldn't say only carriers understand the traffic as much as I would say 
> carrier's traffic is more transparent, and perhaps, even more readily 
> available.
> 
> I just think that it is not relevant to try and lump network and content 
> traffic into one growth pattern. For all intents & purposes, there are two 
> Internets running side-by-side between network and content. They converge at 
> some point, but really, they are very different.

And as I’ve seen, they continue to become a bit more divergent.  As someone who 
has an access network I see where the majority of my bits go, which is to the 
content folks.  There’s some to the other part, but mostly people want their 
MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, 
Metaverse, Game downloads, Streaming service, Email and cloud ramps 
(enterprise).

- jared

Re: Routed optical networks

[Jared Mauch]

> On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG 
>  wrote:
> 
> To clarify the table I linked to in the previous email:
> 
> Cisco estimates IP traffic exchanged over the access network by both 
> businesses and consumers with:
> 
> • endpoints over managed networks and 
> • endpoints over unmanaged networks (“Internet traffic”).
> 
> Both the mobile access network and the fixed access network are considered. 
> 
> Cisco considers IP traffic over managed networks to be characterized by 
> passage through a single service provider. 
> Without explicitly referring to quality of service (QoS), 
> the implication is clearly that the traffic is controlled to meet the QoS 
> demanded by the service level agreement (SLA). 
> 
> In contrast, “Internet traffic” crosses provider domains; 
> typically, this traffic is delivered on the basis of providers’ best effort. 
> These two kinds of traffic complement one another and collectively are 
> referred to as total global IP traffic.


I think there’s a lot of problems here.  While places like my employer will 
periodically disclose our traffic numbers, and DDoS providers, mitigation 
platforms and otherwise will disclose the peaks they see, much of this data is 
a bit opaque, and tools like AI that do in-metro or cross-metro 
datacenter-datacenter remote DMA type activities, those all count differently.

We have seen a continued trend of the privatization of traffic and localization 
of that over time.  I’ve watched all the big carriers retreat from their global 
network reaches to be more of regionalized networks.  A decade ago you would 
have seen European national incumbents peering and with market in Asia, and the 
complete global networks continue to shrink.

Meanwhile you have a mix of the content and cloud providers continue to build 
their business-purpose networks expanding into markets that the uppercase 
Internet may not need to reach.

You can look at the proposals in the EU about fees, and I have dual thoughts on 
this which are MY OWN and don’t represent my employer or otherwise, but if you 
read this post from Petra Arts - 
https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major 
interconnection points like Frankfurt, which are important but double as 
problematic.  The number of people that need to go to the near market (eg: 
Chicago, while I’m in Detroit area) for good connectivity is an issue, 
meanwhile there’s a robust need to keep traffic within the state of Michigan 
and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on 
the board).  There need to be some aggregation points, so not everyone needs to 
be in Detroit, but also not everyone needs to be in Frankfurt - and content 
localization needs to continue to happen, but is also very regionalized in 
popularity.

How to do this all and not have it all route via Chicago or Frankfurt is a 
challenge, but also not everyone will be in Berlin, Munich or these other 
markets.  This is where having a robust optical network capability (or 
backbone) can come into play, that you can deliver deeper from those hub 
points, but at the same time, I’ve been in meetings where companies have their 
own challenges accepting that content in those downstream locations as their 
network was also built to get to/from the major hub cities, or IP space wasn’t 
allocated in a way that can provide consistent routing results or behaviors.  
(This is where IPv6 can be super helpful, it gives the chance to possibly 
Greenfield, aka not screw it up - at least initially).

There’s huge volumes of IP traffic exchanged, but the largest volumes are being 
moved over private interconnects or a localized IX to those eyeball networks 
with the historical global backbones playing more of the long-distance carrier 
role, which is critical as you want a path to deliver those bits, without it 
following the ITU-style sender pays model, as the majority of IP traffic is 
actually requested by the customer of the end-user network.  (All of it if you 
remove network scans, ddos, web bots/crawlers).

Most networks have no SLA once things cross an unpaid boundary (SFI, or even 
private peering) - and if they are a customer and that path is congested, it’s 
up to the customer to upgrade that path.

- Jared (many hats)

Re: Routed optical networks

[Jared Mauch]

> On May 2, 2023, at 5:11 PM, Etienne-Victor Depasquale  wrote:
> 
> I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you 
> are at a hop where in a long distance system solution, you would end up with 
> OEO, but instead you get directionality capability with an IP/MPLS capable 
> device.  As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the 
> latest example of that.
> 
> Jared, I understand your point in the above statement to be that 
> directionality is cost-effectively implemented through label-switched paths, 
> rather than (ROADM-enabled) optical path switching. 
> 
> Do I understand right?

It might be based on the distances and bandwidth required, and the need or 
desire to have diverse paths.  Much of the economics of this vary based on the 
vendors and underlying either cost or pricing model.

I do believe a few more engineers would be aided with a TCO for networking once 
you add up all the costs, either internal or external.

We’ve seen the one-time-spend to build out the datacenter space exceed the cost 
of the equipment.  (Based on the number of XC/patch panel positions needed for 
example - which might be a direct expense while the hardware may be capitalized 
and depreciated over a period, also TCO to power on a device - it can be quite 
common for that to exceed the Capex as well).

- Jared

Re: Routed optical networks

[Jared Mauch]

> On May 4, 2023, at 6:21 AM, Mark Tinka  wrote:
> 
> 
> 
> On 5/4/23 11:40, Denis Fondras wrote:
> 
>> 
>> You may also take into account the time to deliver.
>> Laying fiber takes much more time than plugging a colored optic.
> 
> Indeed - part of the expense of running new fibre is the time it takes to 
> start making money from it.
> 

One of the advantages in the US right now is that capital cost is being 
subsidized or reimbursed by state or federal government, leaving room to 
expand.  Some providers have missed out on this, and others have capitalized on 
it.  It’s very much a mix of people who are chasing that $ and those that are 
not.  

I will say that merchant silicon has it’s place, but so does the vendor 
silicon.  At some point if you get the fiber to that location, unlocking the 
capacity becomes much easier with CEx or similar modules to overlay services.

Making the choice to build a quality fiber first network is important, and why 
I have already had to take routes that I had planned lower count fibers on and 
upgrade them.  I’m a bit shocked that I now need a 288F cable on some of my 
routes to support future expansion, but that fiber cost is still small compared 
to the labor.

- Jared

Re: Spectrum networks IPv6 access issue

[Jared Mauch]

> On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG  wrote:
> 
> This has been “resolved", I finally got through to some awesome engineer at 
> Spectrum who has rerouted traffic while they work with their hardware vendor 
> (thanks Jake):


One of the tools that I’ve used in the past is the RIPE Atlas service to 
measure these things.  It’s helped me isolate IP space reachability issues for 
new announcements, because you can get enough of a random sample of hosts to 
isolate things, and enough data about that endpoint to launch follow-up 
measurements.

- Jared

Re: Routed optical networks

[Jared Mauch]

> On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG 
>  wrote:
> 
> On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:
> > In short, the idea is that optical networks are wasteful and routers do a
> > better job making more use of a network's capacity than ROADMs. Take the
> > extra router hop (or 3 or 8) versus short-cutting it with an optical
> > network because the silicon is so low-latency anyway that it hardly makes a
> > difference now. Putting more GBs per second on fewer strands means saving a
> > lot of money on infrastructure costs.
> 
> This is a very convoluted way of backing into the ole packet-switched
> vs. circuit switched decision.
> 
> I don't follow. 
> While ROADMs can be thought of as circuit-switchers,
> the number of concurrent clients and switching latency put ROADMs on a 
> different operational level than packet switchers, right?


I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are 
at a hop where in a long distance system solution, you would end up with OEO, 
but instead you get directionality capability with an IP/MPLS capable device.  
As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest 
example of that.

I know of a few companies that have looked at solutions like this, and can 
expect there to be some interesting solutions that would appear as a result.  
Optical line systems tend to have pretty low power requirements compared to a 
router, but some of the routers are getting pretty low power as well when it 
comes to the power OPEX/bit, and if you have the ability to deliver services as 
an integrated packet optical you could see reduced costs and simplified 
components/sparing.

I’ll also say that I’ve not yet seen the price compression that I had expected 
in the space yet, but I figure that’s coming.  We are seeing the bits/watt 
ratio improve though, so for the same or less power consumption you get more 
bits.  Some of this technology stuff is truly magical.

- Jared

Re: Dormant space on blacklists, how can I resolve this?

[Jared Mauch]

I’ll help you off-list.  

- Jared

> On Apr 27, 2023, at 9:46 AM, Matthew Crocker  wrote:
> 
> 
> Hello,
>  
> I run Crocker Communications (AS7849) and have ARIN allocations of 
> 161.77.0.0/16 & 66.59.48.0/20.   The 66.58.48.0/20 space was used for our 
> datacenter which shutdown a couple years ago.  The space has mostly been 
> dormant for the past couple years.   I’m now starting to assign 
> 66.59.[55-60].0/24 to a new group of residential FTTH customers.   The 
> customers are getting access denied messages from Akamai based websites.
>  
> What can I do to get Akamai to unblock the 66.59.48.0/20 space.
> Is there a website I can look to research the reputation of the subnets?  
> They haven’t been used in years so I would expect them to be pretty clean.
>  
> Thanks
>  
> -Matt

Re: 100G-LR1 (DR/FR)

[Jared Mauch]

> On Apr 3, 2023, at 4:54 PM, Tony Wicks  wrote:
> 
> I have been using the  QSFP-100G-CWDM4 2k optics for within rack/DC for a 
> couple of years now. They are about the same price as SR optics but allow the 
> use of simple duplex single mode patches without blasting 10K optics at each 
> other over a 2M patch. Never had one fail or any compatibility issues. 

We saw some issues with the CWDM4 optics failing that caused us to make some 
changes in how we used those optics.  I’ve also heard rumors some of the CWDM4 
optics would go 10x the published spec, but I have not tested that myself.

- Jared

Re: 100G-LR1 (DR/FR)

[Jared Mauch]

We are willing to do 100G-LR1 if someone asks these days.  It lets us be able 
to roll it up into 400G optics on our side as appropriate.  

The big difference in DR/FR is the receiver sensitivity, they are all 
compatible optically, so it’s really about the DR/FR being yield rejects for 
LR1.  It’s also less components in the LR1 vs 100G-LR4 since you don’t need 4 
transmitters and 4 receivers and if one fails you toss the optic, so fewer 
components is also lower cost.

- Jared

> On Apr 2, 2023, at 8:14 PM, David Siegel  wrote:
> 
> At this point, I'd be happy to see others happily deploy a single-lambda 
> optic of almost any variety!  Since deploying 400G in a clients network (but 
> 100G still being the preferred connection choice), any inquiry with respect 
> to LR1, FR1 or DR+ is met with "no thanks, LR4 please."
> 
> If asked, I'd recommend FR1.  They're available at a great price-point, and 
> 2km reach is adequate for most applications.
> 
> On Fri, Mar 31, 2023 at 7:25 AM Jared Mauch  wrote:
> The common tech is 100G-LR4 these days - I'm wondering how many operators are 
> supporting the LR1 to allow its use on 400G and future 800G optics as those 
> use breakout to support 100G ports. 
> 
> Would you rather do a 400G port on a router vs 100LR1?
> 
> Curious what others think. 
> 
> Sent via RFC1925 compliant device

SF union square area fiber

[Jared Mauch]

Can someone who is familiar with the fiber assets around the union square area 
in SF ping me off-list?

Thanks.

- jared

100G-LR1 (DR/FR)

[Jared Mauch]

The common tech is 100G-LR4 these days - I'm wondering how many operators are 
supporting the LR1 to allow its use on 400G and future 800G optics as those use 
breakout to support 100G ports. 

Would you rather do a 400G port on a router vs 100LR1?

Curious what others think. 

Sent via RFC1925 compliant device

Re: Increasing problems with geolocation/IPv4 access

[Jared Mauch]

> On Jan 20, 2023, at 11:29 PM, Crist Clark  wrote:
> 
> Are you sure it’s really geolocation blocks? Or is it anonymizer and VPN 
> service detection? The geoIP vendors typically sell both since one of 
> anonymizers’ top applications is to evade geolocation. Have customers using 
> peer-to-peer anonymizers wittingly or unwittingly? Customers with malware or 
> other PUPs hosting anonymizer services?

I know in the case of one provider it was a geolocation related issue.  I don’t 
know if they fixed it, as I said the customers left that provider so the 
complaint went away.

There seem to be a few issues happening.  If I’m not getting the bot/threat 
feeds for those places, I’m happy to follow-up with that customer, but some is 
just flat out things like “This isn’t IP space in US” or the feedback from the 
customer says the provider places them in Mexico.

As I said, looking for any place that has 23.138.114.0/24 in a feed to be 
blocked as some of the ISD (intermediate school district) that aggregates tech 
for several around the area started blocking over the winter break anyone in 
that /24, can ping from other subnets but not that one *smh*.

I’m a bit grasping at straws, but also looking for any ideas or information 
that people may have around it.  I get some people may update monthly, or take 
time to get the changes through their systems, but parts of this have been 
going on now since mid-late September.  If it’s going to take 1.5-2 quarters to 
have the IP space be viable, this is something I’ll be taking up eventually 
with folks at ARIN - similar to issues with other things that may not be easily 
fixed, there’s a level of effort that I’m willing to undertake here, but at 
some point there is a question about if it’s fit for any purpose.

The reality is I expect if I can find where the feed is that has the space 
flagged, that will likely address this part of the long tail.  I would hate to 
end up doing more NAT-PT/44 due to one or a few vendors with bad data sources.

- Jared

Re: Increasing problems with geolocation/IPv4 access

[Jared Mauch]

> On Jan 20, 2023, at 8:02 PM, Owen DeLong  wrote:
> 
> I will repeat what I have been saying since the first discussions of the 
> concept of ip geo-location some decades ago…
> 
> An IP address is not tied to any of the following:
>   Location
>   Person
> 
> An IP address may be transiently tied to a host. The definition of transient 
> in this case can vary widely from a few seconds to multiple years.
> IP Addresses may be tied to an organization (though this is also usually some 
> level of transient).
> 
> Trying to pretend otherwise in any useful way is fraught.


I think sadly the counterbalance item is that there is some insurance 
underwriter or similar that wants a checkbox saying “yes there is a firewall” 
or “you do X,Y,Z”.

Or: Sure, I agree with you, and when I’m in Europe or similar and can’t access 
my (home) government stuff because they just have off-continent blocked is also 
an issue.

Also: water wet.

What I’m actually looking for isn’t so much a soapbox but to find where the 
[bad] data is coming from so it can be updated as appropriate.  I’m also fine 
with telling the customer to phone the service/bank/whatnot (which is what I 
did in other cases and as much as I also personally dislike the centralization 
of the internet etc) - my customers do seem to really have good experience with 
a modern service like YoutubeTV (for example) - oh and it does IPv6 too.

If you see this and go back to the original post, I am interested if you have 
seen that prefix or any IP space within it, and if it comes from a feed or set 
of aggregated feeds etc, even the name of the company or source/resources there 
so I can try knocking on the door.

- Jared

Increasing problems with geolocation/IPv4 access

[Jared Mauch]

I’ve been seeing an increasing problem with IP space not having the ability to 
be used due to the behaviors of either geolocation or worse, people blocking IP 
space after it’s been in-use for a period of time.

Before I go back to someone at ARIN and say “your shiny unused 4.10 IP space” 
is non-functional and am at a place where I need to start/restart/respawn the 
timer, I have a few questions for people:

1) Do you see 23.138.114.0/24 in any feeds from a security provider that say it 
can/should be blocked?  If so, I’d love to hear from you to track this down.  
Over the new year we had some local schools start to block this IP space.

2) many companies have geolocation feeds and services that exist and pull in 
data.  The reputable people are easy to find, there are those that are 
problematic from time-to-time (I had a few customers leave Sling due to the 
issues with that service).

3) Have you had similar issues?  How are you chasing all the issues?  We’ve 
seen things from everything works except uploading check images to banks, to 
other financial service companies block the space our customers are in.  If we 
move them to another range this solves the problem.

4) We do IPv6, these places aren’t IPv6 modern at all, so that’s no help.

5) IRR+geofeed are published of course.  I’m thinking that it might be 
worthwhile that IP space have published placeholders when it’s well understood, 
eg: ARIN 4.9 space, I can predict what our next allocation would be, it would 
be great to have it be pre-warmed. 

I’ve only seen a few complaints against all our IP space over time, so I don’t 
think there’s anything malicious coming from the IP space to justify it, but 
it’s also possible they didn’t make it through.

If you’re with the FKA Savvis side, can you also ping me, I’d like to see if 
you can reach out to our most recent complaint source to see if we can find who 
is publishing this.  Same if you’re with Merit or the Michigan Statewide 
Educational Network - your teachers stopped being able to post to powerschool 
for their students over the new year break.  They’ve fed it up to their tech 
people towards the ISD.  Details available off-list.

Any insights are welcome, and as I said, I’d like to understand where the 
source list is as it starts out working then gradually breaks, so someone is 
publishing things and they are going out further.

- Jared

Re: Google Speed Test

[Jared Mauch]

On Tue, Jan 03, 2023 at 09:31:27AM -0600, Mike Hammett wrote:
> Is there enough available capacity for {insert whatever the customer is 
> trying to do here}. 
> 
> Can they run 4 YouTubeTV streams or can they run 20? 
> Can they download a file at 5 megabits/s or 15 gigabits/s? 
> 
> 
> There's not a problem to be solved, but information of a variety of types to 
> be gleaned for a variety of purposes. 

Most SLAs only cover on-net services, I recommend having a good
on-net server for testing purposes and knowing your immediate upstream
and peer upstreams test points.  I do recommend that most carriers have
an iperf3/iperf2 test point.  You may find your carriers have one as
well, even if it's not listed in their support pages.  

I've found this useful when you suspect some problem, including
a link hashing problem that only impacts a few flows.

    - jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: ROAs Expire

[Jared Mauch]

On Tue, Jan 03, 2023 at 08:56:28AM -0600, Mike Hammett wrote:
> ROAs expire. Creating new ones doesn't seem to be terribly difficult, but why 
> do they expire in the first place? 

There's several reasons I can see why one would want this:

1) to ensure that proper care is maintained over the IP space, domains,
certificiates (ROA is a certificiate), etc expire and require renewal.

2) If there's a new cipher algo flaw or similar, it may become necessary
to rotate things.

3) to help avoid some of the problems that exist with unmaintained IRR
objects.

There's more I'm sure, but this is one of the reasons that I
personally have been hesitatant to roll out some tools, eg: DNSSEC
(which suffers from a variety of ciphers and for some cases lack of
ability to publish to parents - i think this was largely resolved).

With this increased security also comes to increased fragility,
which I suspect is what you are writing about, something likely broke
for you or someone else due to lack of checking the status of the ROA
expiration.

This has happened in the past with domains, including big name
ones, so having something setup (eg: roa watch, similar to x509watch on
*nix systems) would be appropriate.

I'm sure others can refer to tools or services that can do this,
but it's always a good idea to check your objects and watch when they go
away or expire.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Geoip database update

[Jared Mauch]

I've been waiting for some updates since September, there are good and 
reputable geoIP services that accept updates and those that refuse to 
acknowledge issues or have no way to update. 

Some of the financial institutions and government agencies are examples of bad 
consumers of them, but it's not limited to them. 

If you run a service or engage in geo blocking because of licensing or 
something else, provide an internal escalation path for the customer to have 
you contact the ISP, or compare the data with published Geofeeds or similar. 

The issue is even with ARIN 4.10 space that was previously unused. 

- Jared

Sent via RFC1925 compliant device

> On Dec 17, 2022, at 12:42 PM, Norman Jester  wrote:
> 
> I have found that to be impossible.
> There are a few geoip databases but they all tend to operate the same, with 
> specific update periods. I’ve been on the phone with maxmind etc and they 
> simply have no way to help and say we must wait. It’s really sad how they 
> control so much important data yet care not about the impact their control 
> has when they are wrong.
> 
> Norman Jester
> 
> 
>> On Dec 17, 2022, at 9:31 AM, Mehmet Akcin  wrote:
>> 
>> Hi
>> 
>> Is there a way to force update/flush geoip database faster than 30 days?
>> 
>> I am trying to help a customer resolve some issues due to geoip.
>> 
>> Thanks in advance
>> -- 
>> Mehmet
>> +1-424-298-1903

Re: Random shower thought: GBIC with LC connector...

[Jared Mauch]

> On Nov 15, 2022, at 2:09 PM, Warren Kumari  wrote:
> 
> 
> 
> 
> 
> On Tue, Nov 15, 2022 at 12:55 PM, Eric Litvin  wrote:
> A, Gbics: 
> 
> If you google ws-g5483, 84, 86, 87 - you’ll see the whole line up. All had sc 
> connectors except 83 which was copper rj45 connector.
> 
> 
> 
> Well, clearly that's not true.
> 
> As examples:
> The Cisco GigaStack GBIC - 
> https://www.mtmnet.com/PDF_FILES/WS-X3500-XL_DataSheet.pdf - uses the 6 pin 
> IEEE1394 (Firewire) connector
> 
> 1000Base-CX uses twinax shielded twisted pair and the "HSSDC" connector - 
> e.g: https://www.ebay.com/itm/372539150316
> 
> I've seen (but cannot find a photo at the moment) a DB-9 GBIC (to allow 
> connection to DB-9 fiber channel stuff)
> 
> Someone used to sell an HD-BNC GBIC  - basically the GBIC version of 
> https://www.bhphotovideo.com/c/product/1430372-REG/wohler_sfp_sdi_output_3g_sdi_transceiver_hd_bnc_connectors.html
> 
> I've also somewhere seen an "F-style" coax GBIC to allow reuse of in-building 
> coax - basically looked like a GBIC form factor version of 
> http://www.mdslink.com/magic-sfp/ 


There’s a lot of things here that are feasible, because many modules are in the 
same form factor, eg XFP that is EDFA etc.

The specs (SFF-8024) have a variety of types and encodings with GBIC 2.4.2 and 
table 4.2

The SFF-8024 is being revved to v4.9.2 right now, so there’s always 
enhancements.

The SFP can be used to do SDI or other types of output that are non-IP.

It is nice to seeing things move to CMIS if you’ve dealt with all the 
lack-of-clarity in the hardware and software specifications for what bits are 
set or required to be set, which is why some optics haven’t always worked 
because the EEPROM had TX_Disable set or similar.

- Jared

Re: BCP38 For BGP Customers

[Jared Mauch]

On Thu, Nov 10, 2022 at 10:27:02AM -0800, William Herrin wrote:
> On Thu, Nov 10, 2022 at 10:08 AM Grant Taylor via NANOG  
> wrote:
> > I wonder if Feasible Path uRPF or Enhanced Feasible Path uRPF might help
> > the situation.  However I suspect they both suffer from the FIB != RIB
> > problem and associated signaling.
> 
> Hi Grant,
> 
> That's a fairly good way to think about it. BGP knows -a- path and
> sometimes it knows more than one but it simply doesn't have signal on
> the totality of feasible paths for a particular IP address. No
> distance-vector protocol can.

There's more than this going on as well, because there's a
number of other things going on, the IETF has created a SAVNET working
group to see if it's possible to do something here, and there's also
work in the SIDROPS WG that isn't yet adopted but may be.

The intent would be to include things like the ASPA work with
the SIDR/RPKI work to permit a proof to exist for SAV purposes.  This
may not include all the p2p IP space that would exist between the
networks, and if one doesn't publish ASPA data for things like all those
cloud on-ramp type services, you may end up with traffic blackholed or
other side-effects.

Simply put, SAV/BCP-38 et al is hard, and nearly impossible when
you get much further away from the subnet that traffic originates from.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Geo-IP Sling.com and/or Dish Network Contact.

[Jared Mauch]

Yes, the IP space is updated, not sure what Sling is using, but
this customer will be redirected to YoutubeTV which doesn't seem to have
the same geolocation issues.

There's a significant [growing] issue here as if I'm several
months into the ARIN allocation and can not utilize the space, I expect
a number of providers are facing similar issues.

- Jared

On Mon, Nov 07, 2022 at 02:57:03PM -0500, Josh Luthman wrote:
> Did you update the block with those services listed in the link?  They
> usually update ~weekly and of course there's a delay to the providers as
> well.
> 
> On Fri, Nov 4, 2022 at 5:35 PM Jared Mauch  wrote:
> 
> > Anyone figure this out? Have a new block that works with everything else
> > it seems but them and about to tell this customer to switch from their
> > service.
> >
> > Or if someone knows why 23.138.114.0/24 would be geolocated outside
> > US/Michigan would be great to know.
> >
> > Thanks!
> >
> > Sent via RFC1925 compliant device
> >
> > On May 11, 2022, at 10:35 AM, Josh Luthman 
> > wrote:
> >
> > 
> > Dish/Sling isn't on here but check this list:
> >
> > https://thebrotherswisp.com/index.php/geo-and-vpn/
> >
> > On Tue, May 10, 2022 at 5:18 PM Nicholas Warren 
> > wrote:
> >
> >> Does anyone know how to get ahold of Sling.com or Dish to update location
> >> information on IPv4 addresses?
> >>
> >> I don’t know if meta discussion is allowed on-list, but maybe geolocation
> >> contacts could be listed on the community site? 
> >>
> >> - Nich Warren
> >>
> >

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: BCP38 For BGP Customers

[Jared Mauch]

On Mon, Nov 07, 2022 at 02:47:57PM -0500, Tom Beecher wrote:
> >
> > Are you taking the stance of "if you don't send us the prefix, then
> > we don't accept the traffic"?
> >
> 
> If you were one of my upstreams, and you implemented that, you would very
> quickly no longer be one of my upstreams.

Yes, I suffer from having two upstreams that each have a shared
transit supplier.  They are most likely to only have a single best path
on their network and i can observe in the flow data it's not the one I
expect it to be.

I'm not sure how that provider (3356) would make it happen.

I can tell you that the uRPF that 7018 does made me not able to
utilize one of the providers for outbound traffic because they never
opened the proper ticket for routing that IP space until I had
side-escalated to some people that could help me after several months.

Thankfully it's not a lot of bits but was still annoying to
diagnose and triage.

- Jared

> On Mon, Nov 7, 2022 at 2:22 PM Charles Rumford via NANOG 
> wrote:
> 
> > Hello -
> >
> > I'm are currently working on getting BCP38 filtering in place for our BGP
> > customers. My current plan is to use the Juniper uRPF feature to filter
> > out
> > spoofed traffic based on the routing table. The mentality would be: "If
> > you
> > don't send us the prefix, then we don't accept the traffic". This has
> > raised
> > some issues amongst our network engineers regarding multi-homed customers.
> >
> > One of the issues raised was if a multi-homed BGP customer revoked a
> > prefix from
> > one of their peerings, but continued sending us traffic on the link then
> > we
> > would drop the traffic.
> >
> > I would like to hear what others are doing for BCP38 deployments for BGP
> > customers. Are you taking the stance of "if you don't send us the prefix,
> > then
> > we don't accept the traffic"? Are you putting in some kind of fall back
> > filter
> > in based on something like IRR data?
> >
> > Thanks!
> >
> > --
> > Charles Rumford (he/his/him)
> > Network Engineer | Deft
> > 1-312-268-9342 | charl...@deft.com
> > deft.com
> >

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Geo-IP Sling.com and/or Dish Network Contact.

[Jared Mauch]

Anyone figure this out? Have a new block that works with everything else it seems but them and about to tell this customer to switch from their service. Or if someone knows why 23.138.114.0/24 would be geolocated outside US/Michigan would be great to know. Thanks!Sent via RFC1925 compliant deviceOn May 11, 2022, at 10:35 AM, Josh Luthman  wrote:Dish/Sling isn't on here but check this list:https://thebrotherswisp.com/index.php/geo-and-vpn/On Tue, May 10, 2022 at 5:18 PM Nicholas Warren  wrote:Does anyone know how to get ahold of Sling.com or Dish to update location information on IPv4 addresses?

I don’t know if meta discussion is allowed on-list, but maybe geolocation contacts could be listed on the community site? 

- Nich Warren

Re: AKAMAI Contact

[Jared Mauch]

Please ping me off list. Thanks. 

Sent via RFC1925 compliant device

> On Sep 28, 2022, at 3:47 PM, Joshua Pool via NANOG  wrote:
> 
> 
> Anyone have a contact for AKAMAI?
> 
> Thanks in advance.
> 
> Josh

Re: Akamai Contact/Issues

[Jared Mauch]

You can ping me off list. Thanks. Sent via RFC1925 compliant deviceOn Sep 26, 2022, at 1:39 PM, Dustin Brooks  wrote:







Anyone having issues with sites hosted by Akamai?  We have several users within a single /24 that are not able to access several (go.microsoft.com,
www.irs.gov, adobe.com, npr.org, just to name a few).  I’ve tried emailing Akamai but I get a canned response “you don’t have an account with us.
 
Anyone know how to get in touch with them?
 


-- 


Dustin Brooks
Sr. Network Engineer

 

 
2028 Highway 115, Mansura, LA 71350
318.597.0303 Office | 337.781.4506 Mobile |  www.Conterra.com


 

This e-mail may contain information that is confidential or privileged. If you are not the intended recipient, do not read, copy or distribute the e-mail or any attachments. Instead, please notify the sender and delete the e-mail and any attachments. Thank
 you.

Samsung too?

[Jared Mauch]

I tried to place some new IP space under 4.10 ARIN into service and there's some Samsung thing that doesn't work now per customer reports so I moved people back out, so if there's someone who knows about that I'd also appreciate a ping. Sent via RFC1925 compliant deviceOn Sep 17, 2022, at 2:34 PM, Christopher Munz-Michielin  wrote:
  

  
  
Thanks for the link, I did actually check out the links on that
  page and they all seem to return the correct geoip.  I didn't see
  any specific contacts for Level3, unless I missed it?
Someone did reach out to me off-list, so fingers crossed they can
  get it resolved.


On 2022-09-15 14:19, Josh Luthman
  wrote:


  
  Did you check these services?


https://thebrotherswisp.com/index.php/geo-and-vpn/

  
  
  
On Wed, Sep 14, 2022 at 6:34
  PM Christopher Munz-Michielin 
  wrote:

Hey
  All,
  
  For my dayjob I help run an Anycast DNS network and we've
  recently had 
  complaints from our users in the APAC region that sites using
  Level 3's 
  CDN have poor performance.  Upon looking into this, it seems
  that they 
  (Level 3) have incorrect geolocation data for some of our
  Singapore 
  servers and are returning IPs for CDN servers in Australia. 
  I've 
  validated the WHOIS information for the blocks in question is
  correct, 
  and every GeoIP site I check against comes back as Singapore,
  so this 
  must be some internal database.
  
  I've tried emailing the whois contact, as well as the
  technical contact 
  for the domain footprint.net
  but have yet to receive a response.  Wonder 
  if anyone else has been able to get in touch with the CDN
  people at Level 3?
  
  Cheers!
  Chris
  

  

  

Re: NANOG List posts and DMARC

[Jared Mauch via NANOG]

> On Aug 2, 2022, at 4:31 PM, John Levine via NANOG  wrote:
> 
> It appears that Michael Thomas via NANOG  said:
>> 
>> On 8/2/22 12:30 PM, Jim Popovitch via NANOG wrote:
>>> It's been doing it for ages for p=reject, but not p=none (the latter
>>> being Jared's situation)
> 
> I don't understand Jared's concern.  His DMARC policy, like mine, is p=none
> which tells receivers to do nothing DMARC-y with our messages.  I don't get
> any sort of blowback from nanog posts that I can recall seeing.
> 
>> I'm sort of surprised that an org would have p=reject when its users use 
>> outside mailing lists. 
> 
> Unfortunately, we lost that battle a long time ago.  It's "more secure" and
> "best practice" so go away.


Much like inline replies v top-posting and etc.. 

I did manage to get someone to flip the setting so hopefully I’m not getting a 
lot of bounce back from this e-mail.

Thanks to the kind soul who flipped the setting.

- jared

NANOG List posts and DMARC

[Jared Mauch]

Can someone flip the option in Mailman for DMARC please, it’s problematic as if 
one posts and does DMARC and has feedback on, our messages are  possibly 
rejected, and the feedback from a post is quite large.

Not sure who manages it anymore these days.

- Jared

Re: Allegedly Tier 1s in Wikipedia

[Jared Mauch]

> On Aug 2, 2022, at 11:58 AM, Tom Beecher  wrote:
> 
> This conventional interpretation is the one I'm applying in this question.
> 
> I would argue even the 'conventional' definition of 'Tier 1' has been 
> nebulous for long enough that it doesn't really matter much anymore. 
> 
> Who a network connects with and how is all that matters, regardless of what 
> label they want to apply to themselves. 


Yeah, I would generally agree with this.  The interesting thing as I see it is 
so much depends of if it’s long-distance or not.  If you look at what the 
content side generally does (Netflix, Akamai, Fastly, AWS/Cloudfront, 
Cloudflare, Yahoo, Edgecast, Apple, etc).. you see that placed close to the 
end-users and generally you aren’t going more than a few metros over hopefully.

This generally means you are doing on-ramp to a cloud (Microsoft, Google, AWS, 
etc) or get content, or rarely go to something that’s much further away (voice, 
etc).

Those places can’t wait for the traditional peering issues to be resolved will 
move their traffic to another provider, the day of the traditional SFI/Tier1 is 
largely history as the volumes are localized, but that long distance 
performance matters as much as ever.

If you are seeing traffic stuck on any particular provider/path you really 
should be looking at a regional provider that gives you a good blend vs going 
to the big “name brand” places that don’t maintain good local connectivity and 
are more likely to trombone your traffic.

I do this for my small ISP, I purchase from two regional providers that roll up 
everything nicely so I’m unlikely to have any single outage/issue.


To the other question from Mike, does it matter?  Yes, if you are a corporate 
place and just go to a national provider because of a national agreement, we 
have all seen how this is problematic in the past, and when there is a big 
outage, some companies would literally pay that cost for a diverse link.

- Jared

Re: Akamai Peering

[Jared Mauch]

I'll provide a bit more detail - We have certainly been
standardizing on 100G for a number of years now and have a decreasing
number of devices where 10G is appropriate.

for public peering we certainly do have an open peering policy,
if you are encountering an issue please reach out and I can identify
what the root cause is.  If you have a ticket number, etc.. that can
help as well.  I don't personally monitor the ticket queue.

For private interconnect, 100G is the port speed for most of the
americas, some markets may vary.

For public peering, so much depends on the IX/IXP.  EdgeconneX
in Denver does not have access to the Denver IX and we are working to
extend there.  There's at least 4 different sites in Denver for
interconnection, and it's impractical to be in them all.

Some more details would be helpful (in private) so we can move
the traffic to a direct path.

If you have a 10G port and want to swap it to a 100G port, we
should have that conversation.

- Jared

On Tue, Jul 26, 2022 at 08:27:09AM -0500, Paul Emmons wrote:
> Akamai isn't supporting 10g ports on IXPs.  I'd be surprised if the allowed
> it on PNIs.  As for not being on the IXPs, that's odd.
> 
> On Tue, Jul 26, 2022 at 8:23 AM Jawaid Bazyar 
> wrote:
> 
> > Hi,
> >
> >
> >
> > We had Akamai servers in our data center for many years until a couple
> > years ago, when they said they’d changed their policies and decommissioned
> > the servers.
> >
> >
> >
> > I understand that, maintaining many server sites and being responsible for
> > that hardware, even if you pay nothing for power or collocation, must be
> > costly. And at the time, we didn’t have much traffic to them.
> >
> >
> >
> > Today, however, we’re hitting 6 Gbps with them nightly. Not sure what
> > traffic it is they’re hosting but it’s surely video of some sort.
> >
> >
> >
> > We are in the same data center with them, Edgeconnex Denver, and they
> > refuse to peer because they say their minimum traffic level for peering is
> > 30 Gbps.
> >
> >
> >
> > Their peeringdb entry says “open peering”, and in my book that’s not open
> > peering.
> >
> >
> >
> > So this seems to be exactly backward from where every other major content
> > provider is going – free peering with as many eyeball networks as possible.
> >
> >
> >
> > Google – no bandwidth minimum, and, they cover costs on 1st and every
> > other cross connect
> >
> > Amazon – peers are two Denver IX
> >
> > Apple – peers at two Denver IX
> >
> > Netflix – free peering everywhere
> >
> >
> >
> > And, on top of that, Akamai is not at either of the two Denver exchange
> > points, which push together probably half a terabit of traffic.
> >
> >
> >
> > What is the financial model for Akamai to restrict peering this way?
> > Surely it’s not the 10G ports and optics, which are cheap as dirt these
> > days.
> >
> >
> >
> > Doesn’t this policy encourage eyeballs to move this traffic to their
> > cheapest possible transit links, with a potential degradation of service
> > for Akamai’s content customers?
> >
> >
> >
> > Thanks for the insight,
> >
> >
> >
> > Jawaid
> >
> >
> >
> >
> >
> > *[image: uc%3fid=1CZG_hGEeUP_KD95fSHu2oBRA_6dkOo6n]*
> >
> > *Jawaid Bazyar*
> >
> > Chief Technical Officer
> >
> > VERO Broadband
> >
> > [image: signature_3735065359]
> >
> > 303-815-1814
> >
> > [image: signature_3363732610]
> >
> > jbaz...@verobroadband.com
> >
> > [image: signature_60923]
> >
> > https://verobroadband.com
> >
> > [image: signature_4057438942]
> >
> > 2347 Curtis St, Denver, CO 80205
> >
> >
> >

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: HE.net and BGP Communities

[Jared Mauch]

You always want to prefer customer routes over non customer routes as a service 
provider. Of course having a robust set of communities to let adjustments 
happen helps. 

Without proper tiering of routes you may see unstable routing. 

Having a standard set of customer, peer, transit set of local preferences would 
go a long way. Same for geographic scope of routes, only use these on same 
continent. Makes using a provider if you do something like anycast hard if they 
haul you long distance.  

- Jared 

Sent via RFC1925 compliant device

> On Jul 25, 2022, at 6:49 AM, Forrest Christian (List Account) 
>  wrote:
> 
> 
> I wish they'd add one more that turns off their "prefer routes learned from a 
> customer" rule.   I'm having to split my blocks in half and announce them 
> that way to get them to send my traffic directly to me through our IX peering 
> session as opposed to one of my transit providers.
> 
> I'd rather they just let shortest path selection work. 
> 
>> On Sun, Jul 24, 2022, 1:43 PM Siyuan Miao  wrote:
>> They do have BGP communities ... but for black-hole only :-(
>> 
>>> On Sun, Jul 24, 2022 at 9:39 PM Ryan Hamel  
>>> wrote:
>>> Yes.
>>> 
>>> Ryan
>>> 
>>> -Original Message-
>>> From: NANOG  On Behalf Of Rubens 
>>> Kuhl
>>> Sent: Sunday, July 24, 2022 12:36 PM
>>> To: Nanog 
>>> Subject: HE.net and BGP Communities
>>> 
>>> The last mention I found on NANOG about HE.net and BGP communities for 
>>> traffic engineering is from April 2021 and said they provided none.
>>> 
>>> Is that still the case a year later ?
>>> 
>>> 
>>> Rubens
>>> 

Re: Rogers Outage Canada

[Jared Mauch]

> On Jul 11, 2022, at 11:15 AM, Victor Kuarsingh  wrote:
> 
> This is the most they can and will say.  For liabilities reasons, specifics 
> are likely not in the cards.  As most services ride over common service 
> networks, its quite possible that a network substrate failure can have a 
> number of upstream service impacts.  The point here is that the CEO is 
> directly addressing the customer base, which is needed here.
> 

Yes, unless you get a side-leak from someone in-the-know, I expect either 
direct or CRTC will be the best source.  I know a few of us are concerned about 
if we have any similar risk of impact to our environments, so we will be 
watching close for any hints.

- Jared

Re: is nanog really in the spoofer report?

[Jared Mauch]

I can confirm that the hardware that NANOG does use can't filter it all 
automatically I think for v6 as I helped them look at it. 

Sent via RFC1925 compliant device

> On Jul 10, 2022, at 5:22 PM, Matthew Luckie  wrote:
> 
> 
>> 
>> I just realized that many automatically put emails with the subject
>> line of "Spoofer Report for NANOG" in the trash, so I changed it.
>> 
>> Is that for real or a spoof itself?  If it's real I know a buncha
>> guys that will help. ;)
> 
> This is real:
> 
> https://spoofer.caida.org/recent_tests.php?as_include=19230
> 
> These are correlated with conference network deployments at each NANOG
> event.  Some NANOG conference networks have SAV deployed (more so
> before 2017), but my understanding is that networking equipment does
> not come with SAV enabled by default, so it is easy to overlook.
> 
> Matthew

signature.asc
Description: Binary data

Re: FCC BDC engineer?

[Jared Mauch]

Yeah the big thing I’ve seen is that companies have historically over claimed 
on their 477 reports in weird and interesting ways.  I understand why and how 
it happens, for example, if we do a HH meet for service at location X in census 
tract 2020-01 and I have a 2 mile loop to location Y in census tract 2020-02, 
what is the service address? When there’s a new service, how does it get 
re-geocoded?  Did you get all the exceptions handled properly?  

The new BDC rules are also a bit odd compared to the 477 ones, which if at an 
address I sold 2 services, I might have 2 locations but BDC says it’s 1 even if 
duplex.

Things just get a bit sticky around this is all when it comes to this.  I 
appreciate better accuracy as Comcast still claims to offer service at my home 
which isn’t true.  So do a few other providers as well which is inaccurate.

I already filed my 477 for 1H22, now to get this BDC done.

- jared

> On Jul 5, 2022, at 1:58 PM, Andrew Latham  wrote:
> 
> I read https://docs.fcc.gov/public/attachments/DA-22-543A1.pdf and a PE is 
> not required.
> 
> On Tue, Jul 5, 2022 at 9:47 AM KCI Dave Logan via NANOG  
> wrote:
> Hi all.  We operate a small regional ISP in Colorado, but no size is too 
> small to ignore the FCC, as you all know.
> 
> We're really struggling to find the required engineer for the filing, and 
> we're small enough that we don't have an officer with engineering credentials.
> 
> Any pointers in the CO/WY/NE/KS area would be great, on or off list.
> 
> I sure hope we're the only org with this problem still, and all the rest of 
> you are good to go.
> 
> Thanks,
> dave
> 
> -- 
> 
> Dave Logan
> Kentec Communications, Inc.
> 970-522-8107
> 
> 
> 
> -- 
> - Andrew "lathama" Latham -

Re: Serious Juniper Hardware EoL Announcements

[Jared Mauch]

> On Jun 14, 2022, at 12:42 PM, Shawn L via NANOG  wrote:
> 
> With the current shortages and lead times, I almost feel like I did back in 
> the beginning of my career --- 
>  
> Then it was "what can we do with what we can afford" now it's more like  
> "What can we do with what we have (or can actually get)"?

I’m definitely feeling a bit more of this - we are seeing quite a bit of 
mismatch as well in hardware where higher speeds are coming but without a firm 
consensus around optics.  At least for the 400G space it seems to be largely 
sorted, as DR, FR and LR are all interchangeable it’s largely that receiver 
sensitivity which comes into scope, and the LR8 being there, but unlikely to 
see a lot of volume over time.

Reminds me a lot of the DS3 vs OC3 vs gigE days of “what speed, how many”, but 
at least we have bundling figured out at this point.

- Jared

Re: Comcast Network Peer Survey on DSCP/ECN for L4S

[Jared Mauch]

This seems to be missing some of the reasons/why things are remarked, perhaps 
it would be wise to bring some of the people interested in this to the various 
vendor-specific lists or such?

For example, for some hardware types, enabling any sort of rate shaping at all 
will rewrite the DSCP values, even for packets that do not traverse the shaped 
interfaces.

- Jared

> On Jun 10, 2022, at 9:31 AM, Livingood, Jason via NANOG  
> wrote:
> 
> Hi – Comcast is working on the implementation of ultra-low latency 
> networking, leveraging the IETF’s upcoming L4S standard. This standard will 
> require passing ECN and DSCP markings across network boundaries. As a result, 
> we are interested in your perspective on this and in how you handle markings 
> today. We have a short survey that should only take a few minutes to 
> complete. Take the survey at https://forms.office.com/r/vGb0LUXfS1
>  
> While any network operator is welcome to take this, we are particularly 
> interested in any networks that are directly interconnected with us today.
>  
> Thank you!
> Jason Livingood
> Comcast – Technology Policy & Standards
> jason_living...@comcast.com
>  
> PS – Apologies if any of you get a duplicate of this request via other 
> channels.

Re: [Story] When IPv6 Fixes IPv4 Peering Issues

[Jared Mauch]

> On Jun 13, 2022, at 10:25 AM, Brielle  wrote:
> 
> I quickly reconfigured the Cys WireGuard node to connect to the Den node over 
> IPv6 and, after WireGuard did its magic dynamically reconfiguring endpoints, 
> suddenly the connection was back up and routing at full speed. Hell yeah!
> 
> So, moral / TLDR of the story?
> 
> Don't discount taking the time to set up IPv6, even if it's just for your 
> important devices. Also, WireGuard > IPsec.

This is great to hear.  I know that many things will operate better in IPv6 
land vs IPv4 land as in IPv4 land there’s a lot of port-based filtering that 
happens in networks which isn’t the same in IPv6-land.

Super glad to hear that IPv6 saved the day for you!

- Jared

Re: Upstream bandwidth usage

[Jared Mauch]

On Fri, Jun 10, 2022 at 10:31:47AM +0200, Mark Tinka wrote:
> 
> 
> On 6/10/22 09:52, Vasilenko Eduard via NANOG wrote:
> 
> > I did believe that it is about the cost of SFP on the CPE/ONT side: 5$ 
> > against 7$ makes a big difference if you multiply by 100.
> > 
> > By the way, there are many deployments of 10G symmetric PON. It was 
> > promoted for "Enterprise clients".
> > CPE cost hurts in this case.
> > But some CPE could be 10GE and another 1GE upstream (10G downstream) on the 
> > same tree.
> 
> Yes, XG-PON.
> 
> Most FTTH operator stories I've heard of are still running regular GPON,
> thought.
> 
> Seems XG-PON has a high barrier-to-entry for el-cheapo home consumers.

You would be surprised.  The equipment isn't that expensive in
the grand scheme of things.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Jared Mauch]

On Wed, Jun 01, 2022 at 08:49:13PM -0700, Seth Mattinen wrote:
> On 6/1/22 8:12 PM, Mitchell Tanenbaum via NANOG wrote:
> > Believe it or not, there is cable within 500 yards, but they won’t
> > extend it. (:
> 
> 
> 50 feet across the street from me on the east side of the road is AT FTTH
> territory. My side of the street is not. F the west side apparently.

This is common sadly.  I had fiber 1200' from my house that was
unused and there may be no record of it, etc.. so it's just not possible
to happen.  Same goes for areas that have long-haul fiber passing them
but can't get service.

Not everyone is that lucky, but I've seen places with 2-3 fiber
providers that pass them and none offer service.

    - Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Jared Mauch]

> On May 31, 2022, at 8:41 AM, Livingood, Jason via NANOG  
> wrote:
> 
> All the large DOCSIS networks of which I am aware are in fact working on 
> changing their spectrum plan and physical layer to enable higher US speeds 
> and in some cases symmetric multi-gig services.


Yes, I’ve seen Comcast claim to offer 2G symmetric services in their 
applications for funding from state/local authorities to expand their networks 
to unserved or underserved areas.  I have no reason to disbelieve this claim.  
I’ve been talking to vendors about what’s going on for the next generation of 
FTTx/PON and it looks quite attractive at this point, I’m excited to see the 
latency drops and speeds go up for people once they’re off DSL or DOCSIS over 
time.

In my early days of research for my “hobby ISP” as I call it, I looked at 
getting older docsis systems as an option/alternative, and it seemed to be 
worthwhile as without a TV overlay, there were more options for speed.

The reality is today once you have the infrastructure in place, if you planned 
well you can easily upgrade with overlays.

- Jared

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Jared Mauch]

Sadly thus us repeating the same problematic data based on average usage by 
older Americans vs usage by younger people or those of us with several 
children. 

I agree with the average utilization but when it comes to those peaks my 
customers can finish their uploads or restores quickly when they do need it. If 
they are behind a limiter at 25m suddenly that FedEx or carrier pigeon seems 
best. 

Business I was at today says they need 40mbps 

- Jared 

Sent via RFC1925 complaint device

> On May 28, 2022, at 4:22 PM, Mike Hammett via NANOG  wrote:
> 
> 
> Most households have no practical use for more than 25 megs. More is better, 
> but let's not just throw money into a fire because of a marketing machine.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> 
> Midwest-IX
> http://www.midwest-ix.com
> 
> From: "Aaron Wendel" 
> To: nanog@nanog.org
> Sent: Monday, May 23, 2022 1:49:13 PM
> Subject: Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers
> 
> The Fiber Broadband Association estimates that the average US household 
> will need more than a gig within 5 years.  Why not just jump it to a gig 
> or more?
> 
> 
> On 5/23/2022 1:40 PM, Sean Donelan wrote:
> >
> > https://www.fcc.gov/document/fcc-proposes-higher-speed-goals-small-rural-broadband-providers-0
> >  
> >
> >
> > The Federal Communications Commission voted [May 19, 2022] to seek 
> > comment on a proposal to provide additional universal service support 
> > to certain rural carriers in exchange for increasing deployment to 
> > more locations at higher speeds. The proposal would make changes to 
> > the Alternative Connect America Cost Model (A-CAM) program, with the 
> > goal of achieving widespread deployment of faster 100/20 Mbps 
> > broadband service throughout the rural areas served by rural carriers 
> > currently receiving A-CAM support.
> >
> 
> 

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Jared Mauch]

> On May 26, 2022, at 9:31 AM, Livingood, Jason via NANOG  
> wrote:
> 
>> Latency is a limitation for things that are generally relatively low 
>> bandwidth (interactive audio, zoom, etc.).
>> Higher bandwidth won’t solve the latency problem
> 
> +1
> IMO as we enter the 'post-gigabit era', an extra 1 Gbps to the home will 
> matter less than 100 ms or 500 ms lower working latency (optimally sub-50 ms, 
> if not sub-25 ms). The past is exclusively speed-focused -- the future will 
> be speed + working latency + reliability/resiliency + consistency of QoE + 
> security/protection + WiFi LAN quality.

This is always cute when posted from the haves vs the have-nots.  I’m watching 
a lot of people who don’t want to take government money, or play along flail at 
all of this.  They see the internet as for e-mail vs some futuristic use-case.

A few realities:

1) material cost is overall small for a fiber network (Even with the 250% price 
increase in the past ~24 months in materials)
2) Labor is the killer (this also has inputs of diesel fuel costs as the trucks 
that move the stuff are all diesel) reflecting 80%+ of the direct hard costs
3) There’s a lot of variable soft costs in permitting, engineering (Drawings) 
and network design inputs.
4) Many electric utilities have poor quality poles and want to charge tenants 
to upgrade them when they’ve ignored them for decades
5) Several companies have zero incentive to improve the QOE of the end-user 
service

Of course speed, latency, reliability matter.

It’s possible to hit people with varying technologies, and when you stick to 
one, be it PON, HFC, xDSL + FTTx, the other inputs come into play, be it the 
spectrum reserved for RF overlay on PON and HFC or otherwise.

You’re also seeing carriers walk away from new developments if they can’t be 
the monopoly option there, so it’s quite interesting watching what happens with 
my FTTH hat on.

I would say, if you’re looking to build or expand your networks, focus on how 
you can get the fiber out there, there’s a lot of money available if you’re 
willing to take it.  It might mean taking the USF money and the obligations 
that go with that in reporting, compliance, etc.. but those costs don’t have to 
be onerous if you are mindful of how the programs work and have the right 
integration/reporting.

- Jared

Re: BGP Javascript Map/Visualization

[Jared Mauch]

On Thu, May 26, 2022 at 10:43:24PM +, Adam Thompson wrote:
> Neat.  Any idea who to ask questions of, regarding the incorrectness of the 
> data?  I would have assumed Job, but he's long gone from NTT, is this 
> abandonware or maintained?  Anyone know?
> 

I know where it's hosted, and it was a bit amusing when I left
NTT that he gave me grief about the as2914.net domain registration ... 

I would love to see it updated, even if it's not from the 2914
vantage point, i think he left some docs about how it was done.

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: IP Multicast Persistent Duplicate Packet Issue

[Jared Mauch]

You can likely set "ip pim dr-priority" to get what you want

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipmulti/command/imc-cr-book/imc_i3.html#wp1384657000

- Jared

On Thu, Apr 07, 2022 at 09:07:34AM +1000, Roman Islam wrote:
> Thanks Jared. Yes it is SSM. I will try to flip PIM priority and align HSRP
> active.
> 
> My initial thought was keeping everything as it is. Can I simply flip PIM
> assert winner to PE1. Same way we can manually select STP root. Looks like
> this is probably not an option. Didn't find relevant doc on how to
> manipulate PIM assert winner in Cisco boxes though. Highest IP is the
> winner if IGP costs are the same to the source.
> 
> R
> 
> 
> On Thu, Apr 7, 2022 at 3:02 AM Jared Mauch  wrote:
> 
> > If this is ASM, what device is the RP?  You may want to configure MSDP
> > between PE1/PE2 to help if that’s the case, or is this SSM or something
> > else you may want to just flip the PIM priority to make it pick what you
> > want and see if you can tie it to your HSRP (Cisco, might I suggest VRRP so
> > you could do dual-vendor later?) state, perhaps with EEM script to keep the
> > config in sync if required.
> >
> > - Jared
> >
> > > On Apr 6, 2022, at 3:25 AM, Roman Islam  wrote:
> > >
> > > Hi Everyone,
> > >
> > > Has anyone experienced a TETRA Radio application issue if underlying IP
> > multicast transport sends persistent duplicate packets?
> > >
> > > Here is my scenario as below:
> > >
> > > PIM is running on the MPLS L3 VPN environment. C multicast is running on
> > a single VRF (TETRA) only. Source is running behind a dual home PE. HSRP,
> > PIM DR path is via PE1 to the source. Anycast RP is configured with PE1 &
> > PE2.
> > >
> > > On the receiver side there is a single PE. When I check (S,G) route on
> > the receiver side PE default MDT is working as expected. After the
> > threshold exceeds it switches to the data MDT. PIM Assert mechanism winner
> > is PE2 though (Since PE 2 has the highest IP and source is behind dual home
> > PE with equal cost I guess).
> > >
> > > Can this be a reason for persistent duplicate multicast packets at the
> > receiver side; since the assert winner is PE2 but HSRP, PIM DR path is via
> > PE1 to the source?
> > >
> > > If this is the case is there any way to manually configure the assert
> > winner to PE1?
> > >
> > > Roman
> >
> >

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: IP Multicast Persistent Duplicate Packet Issue

[Jared Mauch]

If this is ASM, what device is the RP?  You may want to configure MSDP between 
PE1/PE2 to help if that’s the case, or is this SSM or something else you may 
want to just flip the PIM priority to make it pick what you want and see if you 
can tie it to your HSRP (Cisco, might I suggest VRRP so you could do 
dual-vendor later?) state, perhaps with EEM script to keep the config in sync 
if required.

- Jared

> On Apr 6, 2022, at 3:25 AM, Roman Islam  wrote:
> 
> Hi Everyone,
> 
> Has anyone experienced a TETRA Radio application issue if underlying IP 
> multicast transport sends persistent duplicate packets?
> 
> Here is my scenario as below:
> 
> PIM is running on the MPLS L3 VPN environment. C multicast is running on a 
> single VRF (TETRA) only. Source is running behind a dual home PE. HSRP, PIM 
> DR path is via PE1 to the source. Anycast RP is configured with PE1 & PE2. 
> 
> On the receiver side there is a single PE. When I check (S,G) route on the 
> receiver side PE default MDT is working as expected. After the threshold 
> exceeds it switches to the data MDT. PIM Assert mechanism winner is PE2 
> though (Since PE 2 has the highest IP and source is behind dual home PE with 
> equal cost I guess).
> 
> Can this be a reason for persistent duplicate multicast packets at the 
> receiver side; since the assert winner is PE2 but HSRP, PIM DR path is via 
> PE1 to the source? 
> 
> If this is the case is there any way to manually configure the assert winner 
> to PE1? 
> 
> Roman

Re: Anyone have contacts for Akamai GeoIP

[Jared Mauch]

Yup.. ping me with the details off-list

- Jared

> On Mar 2, 2022, at 7:02 PM, Christopher Munz-Michielin 
>  wrote:
> 
> Hey All,
> 
> Hoping someone has a contact at Akamai who can assist.
> 
> As part of my day job I run a DNS network and we've been having issues with 
> Akamai mis-locating the geolocation of some of our revolvers.  The most 
> egregious example is our resolver in Frankfurt being classified as 
> Australian, but there are some other instances as well.
> 
> This is an issue because we run fully recursive revolvers, so when a customer 
> queries our DNS server, we attempt to resolve the domain directly against the 
> authoritative name servers (Akamai in this case).  Because Akamai has 
> mis-located our IPs, we get handed an IP in the wrong hemisphere and our 
> customers experience 200+ ms latency to sites that should be regional.
> 
> We've tried reaching out via a couple of channels but have not gotten 
> anything back, wondering if anyone on the list knows a contact address for 
> Akamai GeoIP we can submit corrections to.
> 
> Cheers,

Re: home router battery backup

[Jared Mauch]

> On Jan 13, 2022, at 12:28 PM, Chris Adams  wrote:
> 
> Once upon a time, Brandon Martin  said:
>> AT and Comcast don't seem to provide battery by default if you buy
>> voice service from them.
> 
> The only major power outage I've experienced at my house (I've been here
> over 20 years) was the May 2011 tornado outbreak, when TVA lost hundreds
> of distribution towers, and my local utility lost all feeds.  At the
> time, I had AT POTS, Comcast cable/Internet, and T-Mobile cell.

You are lucky.  In my areas we have many power outages and in my ~20 years in 
my current house, we have had several outages that went past 3 days in various 
weather (spring, summer, fall, winter)

Not only were we impacted by the NE blackout, but just in the short time I’ve 
had a generator it’s run around 0.5% of the time due to grid outage, with the 
prior year we had 7 different outages.

I also don’t believe the reporting is accurate from this source I’m doing a 
quick SWAG of, I think it’s been longer.  It’s so bad I monitor the grid 
voltage and have it in influxdb+grafana dashboard

(I recommend iotawatt if you want a neat device)

- Jared

Re: Can it really be this quiet?

[Jared Mauch]

> On Jan 3, 2022, at 2:53 PM, Job Snijders via NANOG  wrote:
> 
> Hi Allen,
> 
> Yes, it can be this quiet. It’s good news, it means the thing is mostly 
> working :-)
> 
> I wish everyone a happy and calm 2022!
> 

I’m surprised nobody talked about the major provider outages that happened over 
the break, there were several of them, but mostly in Asia.

- jared

Re: Log4j mitigation

[Jared Mauch]

> On Dec 13, 2021, at 2:24 PM, Owen DeLong  wrote:
> 
> The bigger problem seems to be the ever growing list of products you may be 
> using which depend on it potentially without your knowledge.

This isn’t a new problem.

This is an great modern example showing how deeply embedded things could be, 
and they get worse with each of these nesting technologies as well, it may be 
embedded in a docker or VM image, or the class could be in some other JAR or 
zip you are not aware of, or could come back with an overlapping class 
definition based on the order things get loaded.

The same was always true with shared libraries and too-generic function names.

It’s such a blast from the past as I had felt we had moved past many of these 
interpreted environment or parser things by properly encoding strings with a 
function.

I’m really amazed at how widespread this is and what enterprise applications 
have had to get patched due to them embedding this software.

- jared

Re: Log4j mitigation

[Jared Mauch]

This is largely a patching exercise for people that use the software. If you 
use it, please patch. 

Sent via RFC1925 complaint device

> On Dec 10, 2021, at 10:59 PM, Andy Ringsmuth  wrote:
> 
> The intricacies of Java are over my head, but I’ve been reading about this 
> Log4j issue that sounds pretty bad.
> 
> What do we know about this? What, if anything, can a network operator do to 
> help mitigate this? Or even an end user?
> 
> 
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> a...@andyring.com

Re: private 5G networks?

[Jared Mauch]

> On Dec 5, 2021, at 3:42 AM, Eliot Lear  wrote:
> 
> 
> On 01.12.21 15:17, Tom Beecher wrote:
>> While you are correct that it's just as illegal to intentionally interfere 
>> with the unlicensed wifi bands as it is with CBRS, the difference is that 
>> the FCC and regulatory bodies are much more likely to investigate and take 
>> action against intentional interference in these frequency ranges than they 
>> would be in the unlicensed wifi bands.
> 
> And there's a practical reason for that: establishing proof of unauthorized 
> use of a frequency is a heck of a lot easier than intentional interference.  
> All the former requires is triangulation of the offending station.  The 
> latter requires that plus a finding of intent. It CAN happen; but more often 
> than not what is actually found is a faulty piece of equipment that is 
> emitting something and everything else catching a bad harmonic. There was a 
> famous case about this in Wales in which an old television set took out a 
> town.[1]
> 
> Eliot
> 
> https://www.openreach.com/news/second-hand-tv-wipes-out-broadband-for-entire-village/
> 
> 

There was one in Oregon where it was transmitting on one of the ELB frequencies 
as well

https://www.nytimes.com/2004/11/01/technology/the-tv-that-sent-out-a-cry-for-help-via-satellite.html

- Jared

Re: Redeploying most of 127/8, 0/8, 240/4 and *.0 as unicast

[Jared Mauch]

On Fri, Nov 19, 2021 at 09:43:26AM -0800, Michael Thomas wrote:
> 
> On 11/19/21 8:27 AM, Randy Bush wrote:
> > these measurements would be great if there could be a full research-
> > style paper, with methodology artifacts, and reproducible results.
> > otherwise it disappears in the gossip stream of mailimg lists.
> > 
> Maybe an experimental rfc making it a rfc 1918-like subnet and implementing
> it on openwrt or something like that to see what happens. how many ip
> cameras and the like roll over and die? same for class E addresses too, I
> suppose. The question with anything that asks about legacy is how long the
> long tail actually is.
> 
> Mike, not that have any position on whether this is a good idea or not

I can tell you it's observable out there and if i use my home network
to follow default i can tell it is working through those devices at
least.

I agree with Randy it would be good if someone did this, it shouldn't be
too hard with ripe atlas and a provider deciding to announce something
like 240.2.3.0/24 to see if it can be reached.

That's at least a decent measurement and report, but the client
side OS will still be a variable that is difficult to digest.  Not sure
how many people are running very old IP stacks.  This is another hard to
measure problem.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Redeploying most of 127/8, 0/8, 240/4 and *.0 as unicast

[Jared Mauch]

> On Nov 18, 2021, at 4:31 PM, Randy Bush  wrote:
> 
> as a measurement kinda person, i wonder if anyone has looked at how much
> progress has been made on getting hard coded dependencies on D, E, 127,
> ... out of the firmware in all networked devices.


At least the E space is largely usable last I checked (eg: IOS-XR, JunOS).  You 
can even measure traceroutes that have the class-e space in them from some 
cloud providers if your network and the intermediaries don’t do u-rpf as well.

Most OS’es (MacOS, Linux, various *BSD) also permit this as well.  My 
understanding is that the RedmondOS may not handle this, but if you need to 
number your infrastructure these addresses may work well.

- Jared

Re: IPv6 and CDN's

[Jared Mauch]

On Fri, Oct 22, 2021 at 05:13:09PM +0200, Job Snijders via NANOG wrote:
> Hi everyone, goedenmiddag Marco!
> 
> On Fri, Oct 22, 2021 at 01:40:42PM +0200, Marco Davids via NANOG wrote:
> > We currently live in times where is actually fun to go IPv6-only. In my
> > case, as in: running a FreeBSD kernel compiled without the IPv4-stack.
> 
> Indeed, this is fun experimentation. Shaking the (source code) trees
> through excercises like these is a valuable way to identify gaps.
> 
> > It turns out that there underlying CDN's with domain names such as
> > ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
> > reside on authoritative name servers that *only* have an IPv4 address.
> 
> As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
> Fastly is working hard with select customers and friends to support IPv6
> for everyone.
> 
> ** SNIP **
> 
> as BGP traffic engineering) might be reluctant to offer IPv6 services
> "as if they are the same as IPv4". More study is required.
> 
> Tl;DR - work in progress! :-)

Some of the other CDNs do have IPv6 on the authorities and
should work without issues.

eg:

dig -6 +trace www.akamai.com.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: DNS pulling BGP routes?

[Jared Mauch]

This is quite common to tie an underlying service announcement to BGP 
announcements in an Anycast or similar environment.  It doesn’t have to be 
externally visible like this event for that to be the case.

I would say more like Application availability caused the BGP routes to be 
withdrawn.  

I know several network operators that run DNS internally (even on raspberry pi 
devices)  and may have OSPF or BGP announcements internally to ensure things 
work well.  If the process dies (crash, etc) they want to route to the next 
nearest cluster.

Of course if they all are down there’s negative outcomes.

- Jared


> On Oct 6, 2021, at 1:42 PM, Michael Thomas  wrote:
> 
> So if I understand their post correctly, their DNS servers have the ability 
> to withdraw routes if they determine are sub-optimal (fsvo). I can certainly 
> understand for the DNS servers to not give answers they think are unreachable 
> but there is always the problem that they may be partitioned and not the 
> routes themselves. At a minimum, I would think they'd need some consensus 
> protocol that says that it's broken across multiple servers.
> 
> But I just don't understand why this is a good idea at all. Network topology 
> is not DNS's bailiwick so using it as a trigger to withdraw routes seems 
> really strange and fraught with unintended consequences. Why is it a good 
> idea to withdraw the route if it doesn't seem reachable from the DNS server? 
> Give answers that are reachable, sure, but to actually make a topology 
> decision? Yikes. And what happens to the cached answers that still point to 
> the supposedly dead route? They're going to fail until the TTL expires anyway 
> so why is it preferable withdraw the route too?
> 
> My guess is that their post while more clear that most doesn't go into enough 
> detail, but is it me or does it seem like this is a really weird thing to do?
> 
> Mike
> 
> 
> On 10/5/21 11:56 PM, Bjørn Mork wrote:
>> Masataka Ohta  writes:
>> 
>>> As long as name servers with expired zone data won't serve
>>> request from outside of facebook, whether BGP routes to the
>>> name servers are announced or not is unimportant.
>> I am not convinced this is true.  You'd normally serve some semi-static
>> content, especially wrt stuff you need yourself to manage your network.
>> Removing all DNS servers at the same time is never a good idea, even in
>> the situation where you believe they are all failing.
>> 
>> The problem is of course that you can't let the servers take the
>> decision to withdraw from anycast if you want to prevent this
>> catastrophe.  The servers have no knowledge of the rest of the network.
>> They only know that they've lost contact with it.  So they all make the
>> same stupid decision.
>> 
>> But if the servers can't withdraw, then they will serve stale content if
>> the data center loses backbone access. And with a large enough network
>> then that is probably something which happens on a regular basis.
>> 
>> This is a very hard problem to solve.
>> 
>> Thanks a lot to facebook for making the detailed explanation available
>> to the public.  I'm crossing my fingers hoping they follow up with
>> details about the solutions they come up with.  The problem affects any
>> critical anycast DNS service. And it doesn't have to be as big as
>> facebook to be locally critical to an enterprise, ISP or whatever.
>> 
>> 
>> 
>> Bjørn

Re: Disaster Recovery Process

[Jared Mauch]

> On Oct 5, 2021, at 10:05 AM, Karl Auer  wrote:
> 
> On Tue, 2021-10-05 at 08:50 -0400, Jared Mauch wrote:
>> A few reminders for people:
>> [excellent list snipped]
> 
> I'd add one "soft" list item:
> 
> - in your emergency plan, have one or two people nominated who are VERY
> high up in the organisation. Their lines need to be open to the
> decisionmakers in the emergency team(s). Their job is to put the fear
> of a vengeful god into any idiot who tries to interfere with the
> recovery process by e.g. demanding status reports at ten-minute
> intervals.

At $dayjob we split the technical updates on a different bridge from the 
business updates.

There is a dedicated team to coordinate an entire thing, they can be low 
severity (risk) or high severity (whole business impacting).

They provide the timeline to next update and communicate what tasks are being 
done.  There’s even training on how to be a SME in the environment.  

Nothing is perfect but this runs very smooth at $dayjob

- Jared

Disaster Recovery Process

[Jared Mauch]

> On Oct 4, 2021, at 4:53 PM, Jorge Amodio  wrote:
> 
> How come such a large operation does not have an out of bound access in case 
> of emergencies ???
> 
> 

I mentioned to someone yesterday that most OOB systems _are_ the internet.  It 
doesn’t always seem like you need things like modems or dial-backup, or access 
to these services, except when you do it’s critical/essential.

A few reminders for people:

1) Program your co-workers into your cell phone
2) Print out an emergency contact sheet
3) Have a backup conference bridge/system that you test
  - if zoom/webex/ms are down, where do you go?  Slack?  Google meet? Audio 
bridge?
  - No judgement, but do test the system!
4) Know how to access the office and who is closest.  
  - What happens if they are in the hospital, sick or on vacation?
5) Complacency is dangerous
  - When the tools “just work” you never imagine the tools won’t work.  I’m 
sure the lessons learned will be long internally.  
  - I hope they share them externally so others can learn.
6) No really, test the backup process.



* interlude *

Back at my time at 2914 - one reason we all had T1’s at home was largely so we 
could get in to the network should something bad happen.  My home IP space was 
in the router ACLs.  Much changed since those early days as this network became 
more reliable.  We’ve seen large outages in the past 2 years of platforms, 
carriers, etc.. (the Aug 30th 2020 issue is still firmly in my memory).  

Plan for the outages and make sure you understand your playbook.  It may be 
from snow day to all hands on deck.  Test it at least once, and ideally with 
someone who will challenge a few assumptions (eg: that the cell network will be 
up)

- Jared

Re: IPv6 woes - RFC

[Jared Mauch]

I mostly agree with this. Even new hardware like eero doesn't do v6 by default. 
It's just off. So many things are like this. It's nice that LTE and other 
deployments have v6 by default. Last time I knew providers like t mobile are 
great but their MVNOs like Ultra Mobile do not do v6. 

All this and we will get there. I'm expecting another decade or two though. 

Sent from my TI-99/4a

> On Sep 18, 2021, at 4:29 PM, John R. Levine  wrote:
> 
> IPv4 isn't going away any time soon.

Re: IPv6 woes - RFC

[Jared Mauch]

> On Sep 16, 2021, at 9:23 AM, Ca By  wrote:
> 
> 
> 
> 
> This has nothing to do with IPv6, of course, other than that modern phones use
> VoLTE so within a mobile carrier's network your voice call is probably handled
> using IPv6 transport.
> 
> Good point John. 
> 
> A lot of folks missed that ipv6 absorbed the scale growth in mobile, and 
> mobile is what most eyeballs and be big content consider the internet to be. 
> And, yes, mobile voice is called VoLTE and is most commonly deployed in the 
> usa with ipv6. 
> 
> And most internet is called youtube / fb, and that is ipv6 too. 
> 
> This is where i live and work , 87% of mobiles on v6, voice and data
> 
> https://www.worldipv6launch.org/apps/ipv6week/measurement/images/graphs/CombinedUSMobileCarriers.png
> 
> This is where nanog seems to be (old man yells at cloud meme)
> 
> https://static.wikia.nocookie.net/memepediadankmemes/images/0/01/297.jpg/revision/latest?cb=20180908193511
> 
> I don’t see the failure of ipv6 in 2021. It is globally deployed, providing 
> global address, to billions of things and PB/s of content 
> 
> There are laggards in adopting v6, but they have not stopped the frontier of 
> internet to reaching billions of people and things. 
> 


Yeah, I think this is the thing that I see people most often missing.  Yes your 
provider may not be doing IPv6, but many applications and providers may yet be 
IPv6 internally or IPv6 to the popular content. 

I also say the number of people who store an IP as integer in a mysql(mariadb) 
backend is not to be underestimated.  

I still see some people doing the split up the IPv6 to store it in multiple 
columns thing even in 2021 which is disappointing as it shows this 
backwards/legacy thinking.

If you’re seeing majority IPv4 bits, you likely have some choke point (CGN/FW) 
or similar gateway on the content side that needs a major update.  I saw a post 
yesterday that someone used a unicode char to add an account nickname at their 
bank and it caused the entire bank to pause and them to get a phone call.  Not 
sure if it’s true, but it rings true.

Be the one enabling the next-generation access and you’ll similarly see graphs 
like the mobile carriers see.

- Jared

Re: do bgp optimizers think?

[Jared Mauch]

> On Sep 9, 2021, at 11:44 AM, Randy Bush  wrote:
> 
> to control inbound traffic, how do bgp optimizers decide how to tune
> what they announce?  slfow?  exploration?  ouija board?
> 
> randy


Generally via sFlow or other traffic detail models.

- Jared

9500 BGP Contact

[Jared Mauch]

Hello,

I’m looking for someone at 9500 that can look into why you started announcing 
some IP space starting on August 25th intermittently that is causing an 
operational network issue.

Can you please contact me off-list?

Thanks

- jared

Re: Reminder: Never connect a generator to home wiring without transfer switch

[Jared Mauch]

> On Aug 25, 2021, at 10:04 AM, Mark Tinka  wrote:
> 
> You need to make these things fool-proof. We haven't traveled in over a year 
> but the day we do, it's a recipe for disaster if the person that deals with 
> this stuff is on the road when the power goes out back at home.

This is why I personally spent the $$ on a proper standby generator with 
multiple ATS for the multiple panels.

- Jared

Re: Setting sensible max-prefix limits

[Jared Mauch]

> On Aug 18, 2021, at 9:38 AM, John Kristoff  wrote:
> 
> Maybe because there isn't a simple, universal approach to setting it.
> Probably like a lot of people, historically I'd set it to
> some % over the current stable count and then manually adjust when the
> limits were about to be breached, or often was the case when they were
> and I wasn't ready for it. Not ideal.
> 
> I've never felt the automation of this setting however was worth the
> effort.  Of course I am not usually responsible for hundreds of routers
> and thousands of peering sessions.

We did a variant of this at NTT, with certain baseline settings.  Sometimes 
networks would advertise more routes because they onboarded a large customer 
and it would cause manual updates to be necessary.

Polling daily and snapshotting these values is important to understand what is 
changing.  The reason I just posted a message about Akamai max-prefix is we 
have been giving some general guidance that is out of line/norm compared to 
what perhaps what we want.  This won’t cause a service outage per-se but will 
cause suboptimal routing as we continue to make improvements and upgrades to 
our network.

- Jared

AS20940 Max Prefix and as-path filtering

[Jared Mauch]

Hello,

Akamai has been increasing the routes we are advertising in various locations 
as part of ongoing network changes.  If you have a max-prefix set for Akamai 
can you please increase v4 to 1k and ensure you are accepting our additional 
ASNs that may live behind the clusters.  

We are going to be updating PeeringDB to reflect this change as well.

An appropriate as-path access-list would look like:

20940_(21342|16625|33905|36183|24319|34164|35204|35994|43639|18680|39836|35993|23454|18717|31108|36183)

Feel free to reach out if you have any questions.

- Jared

Re: Global Akamai Outage

[Jared Mauch]

Work hat is not on, but context is included from prior workplaces etc. 

> On Jul 25, 2021, at 2:22 AM, Saku Ytti  wrote:
> 
> It doesn't seem like a tenable solution, when the solution is 'do
> better', since I'm sure whoever did those checks did their best in the
> first place. So we must assume we have some fundamental limits what
> 'do better' can achieve, we have to assume we have similar level of
> outage potential in all work we've produced and continue to produce
> for which we exert very little control over.

I have seen a very strong culture around risk and risk avoidance whenever 
possible at akamai. Some minor changes are taken very seriously. 

I appreciate that on a daily basis, and when we make mistakes (I am human after 
all) are made, reviews of the mistakes and corrective steps are planned and 
followed up on. I'm sure this time will not be different. 

I also get how easy it is to be cynical about these issues. There's always 
someone with power who can break things, but those can also often fix them just 
as fast. 

Focus on how you can do a transactional routing change and roll it back, how 
you can test etc. 

This is why for years I told one vendor that had a line-by-line parser their 
system was too unsafe for operation. 

There's also other questions like:

How can we improve response times when things are routed poorly? Time to 
mitigate hijacks is improved my majority of providers doing RPKI OV, but 
interprovider response time scales are much longer. I also think about the two 
big CTL long haul and routing issues last year. How can you mitigate these 
externalities. 

- Jared 

Re: Global Akamai Outage

[Jared Mauch]

> On Jul 22, 2021, at 12:56 PM, Andy Ringsmuth  wrote:
> 
> The outage appears to have, ironically, taken out the outages and 
> outages-discussion lists too.
> 
> Kinda like having a fire at the 911 dispatch center…



Should not have impacted me in my hosting of the list.  Obviously if the domain 
names were impacted in the lookups for sending e-mail as well, there would be 
problems.



- Jared

Re: Alien waves

[Jared Mauch]

I looked at this before and go far enough in the conversations with one carrier 
that had sold this as a product before and had a poor experience with the 
customers they were no longer offering it. You are likely better off getting a 
volume deal on waves, which can be had for pretty cheap these days.

I can get 400g waves from carriers now, the capacity involved is short of what 
makes sense for IRU long haul.

Dark in the metro continues to work out. As Saku mentioned there's significant 
risks opening up a system even with channel blockers etc to protect each other 
it hasn't worked out. 

Ask some carriers about a dedicated managed system in your rack if XC fees are 
killing you. 

Sent from my TI-99/4a

> On Jul 20, 2021, at 4:35 PM, Lady Benjamin Cannon of Glencoe, ASCE 
>  wrote:
> 
> Does anyone have a comprehensive (or any) list of carriers doing alien 
> wavelengths? (background: 
> https://thecinict.com/2021/03/05/adding-alien-wavelengths/ 
> https://www.ekinops.com/solutions/optical-transport/alien-wavelength )
> 
> Emphasis on subsea operators.
> —L.B.
> 
> Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
> 6x7 Networks & 6x7 Telecom, LLC 
> CEO 
> l...@6by7.net
> "The only fully end-to-end encrypted global telecommunications company in the 
> world.”
> FCC License KJ6FJJ
> 
> 
> 

Re: 100G, input errors and/or transceiver issues

[Jared Mauch]

> On Jul 19, 2021, at 1:50 PM, Saku Ytti  wrote:
> 
> On Mon, 19 Jul 2021 at 20:19, Graham Johnston
>  wrote:
> 
>> I don't at this point have long term data collection compiled for the issues 
>> that we've faced. That said, we have two 100G transport links that have a 
>> regular background level of input errors at ranges that hover between 
>> 0.00055 to 0.00383 PPS on one link, and none to 0.00135 PPS (that jumped to 
>> 0.03943 PPS over the weekend). The range is often directionally associated 
>> rather than variable
> 
> On typical 100G link we do not get single FCS error in a typical day.
> However Ethernet spec still allows very high error rate of 10**-12. So
> 1 error per 1Tb (b not B). I.e. 1 error per 10s, or 0.1PPS would be
> in-spec. We see much better performance to this and vendors generally
> accept lower error rates as legitimate errors.
> 

I will confirm my experience with this at $dayjob as well.  We see interfaces 
with no errors over much longer periods of time inclusive of several of the 
technology options.  If you are seeing errors, there’s likely something wrong 
like unclean fiber or bad optic/linecard etc.

- Jared

Re: A crazy idea

[Jared Mauch]

> On Jul 19, 2021, at 9:04 AM, Stephen Satchell  wrote:
> 
> On 7/19/21 5:41 AM, Feldman, Mark wrote:
>> What you propose is not outlandish; some ISPs have been dual stack
>> and providing some combination of these services for years.  They
>> already provide IPv6 ip6.arpa delegations should their business
>> customers want them.  Some even provide at least a /56 so customers
>> can have multiple /64 subnets.  And we, I mean they, can also provide
>> RFC2317 in-addr.arpa delegations for those smaller IPv4 blocks.
> 
> The part that is missing isn't the "some ISPs", it's "all ISPs".  Also, I 
> don't know of any DNS service provider that offers a product to handle 
> delegations from the IN-ADDR.ARPA and IP6.ARPA trees.
> 
> I'm focusing on the SOHO customer market with my proposal.

Most are regional carriers w/ monopoly and no incentive to offer anything else. 
 This is especially the case with AT in my area.  The same applies to other 
regional providers like Frontier and even services like Verizon FIOS that do 
not offer IPv6 at all.

You really want a SMB connection w/ dedicated IP space, and they may not be 
able to sell that to you on their consumer network.

- Jared

Internet Quality workshop CFP for the internet architecture board (IAB)

[Jared Mauch]

I know that many operators shift traffic based on network and internet quality 
(or don’t use certain networks at all).  This is a great chance to share 
information about things operators have experienced or do to actively measure 
or otherwise to inform those decisions.

- snip -

For complete details, please see:
https://www.iab.org/activities/workshops/network-quality/

Submissions Due: Monday 2nd August 2021, midnight AOE (Anywhere On Earth)
Invitations Issued by: Monday 16th August 2021

Workshop Date: This will be a virtual workshop, spread over three days:

1400-1800 UTC Tue 14th September 2021
1400-1800 UTC Wed 15th September 2021
1400-1800 UTC Thu 16th September 2021

Workshop co-chairs: Wes Hardaker, Evgeny Khorov, Omer Shapira

The Program Committee members:

Jari Arkko, Olivier Bonaventure, Vint Cerf, Stuart Cheshire, Sam
Crowford, Nick Feamster, Jim Gettys, Toke Hoiland-Jorgensen, Geoff
Huston, Cullen Jennings, Katarzyna Kosek-Szott, Mirja Kuehlewind,
Jason Livingood, Matt Mathias, Randall Meyer, Kathleen Nichols,
Christoph Paasch, Tommy Pauly, Greg White, Keith Winstein.

Send Submissions to: network-quality-workshop...@iab.org.

Position papers from academia, industry, the open source community and
others that focus on measurements, experiences, observations and
advice for the future are welcome. Papers that reflect experience
based on deployed services are especially welcome. The organizers
understand that specific actions taken by operators are unlikely to be
discussed in detail, so papers discussing general categories of
actions and issues without naming specific technologies, products, or
other players in the ecosystem are expected. Papers should not focus
on specific protocol solutions.

The workshop will be by invitation only. Those wishing to attend
should submit a position paper to the address above; it may take the
form of an Internet-Draft.

All inputs submitted and considered relevant will be published on the
workshop website. The organisers will decide whom to invite based on
the submissions received. Sessions will be organized according to
content, and not every accepted submission or invited attendee will
have an opportunity to present as the intent is to foster discussion
and not simply to have a sequence of presentations.

Position papers from those not planning to attend the virtual sessions
themselves are also encouraged. A workshop report will be published
afterwards.

Overview:

"We believe that one of the major factors behind this lack of progress
is the popular perception that throughput is the often sole measure of
the quality of Internet connectivity. With such narrow focus, people
don’t consider questions such as:

What is the latency under typical working conditions?
How reliable is the connectivity across longer time periods?
Does the network allow the use of a broad range of protocols?
What services can be run by clients of the network?
What kind of IPv4, NAT or IPv6 connectivity is offered, and are there firewalls?
What security mechanisms are available for local services, such as DNS?
To what degree are the privacy, confidentiality, integrity and
authenticity of user communications guarded?

Improving these aspects of network quality will likely depend on
measurement and exposing metrics to all involved parties, including to
end users in a meaningful way. Such measurements and exposure of the
right metrics will allow service providers and network operators to
focus on the aspects that impacts the users’ experience most and at
the same time empowers users to choose the Internet service that will
give them the best experience."

Re: Muni broadband sucks (was: New minimum speed for US broadband connections)

[Jared Mauch]

> On Jun 2, 2021, at 12:44 PM, Andy Ringsmuth  wrote:
> 
>>> On Mon, May 31, 2021 Mike Hammett wrote:
 Muni broadband does suck, but that's another thread for another day.
>>> Excluding cases where muni broadband doesn't suck, why does muni broadband 
>>> suck?
>>> 
>>> Personally I wouldn't mind more access to dark fiber à la Stokab, much like 
>>> the dry copper pairs of yesterday.
>>> 
>>> If the default state of muni broadband of is suck, what is the root cause? 
>>> Is it a people problem and/or can something be done to improve on the 
>>> default state?
>> 
>> 
>> Muni broadband sucks for several reasons but the most important one is:
>> 
>> Competition. Municipal broadband eliminates it. If it's not obvious
>> why, feel free to Google how competition and monopolization impact
>> product quality. It's a pretty universal trait.
>> 
>> 
>> If you were to structure muni broadband to enhance competition rather
>> than limit it, you might get a different result. For example, if
>> municipalities installed and leased fiber optic cables to every
>> structure but didn't provide any services on those cables, relying
>> instead on third parties directly billing the customer to do so, it
>> could work out as well as having municipalities pay for roads and
>> letting people buy their own cars and trucks to use on them.
> 
> In many municipalities, you can choose your electricity provider. And yet 
> there are not multiple companies running power lines to every house.
> 
> It is easy to make the argument “muni broadband sucks because no competition” 
> but it is much more difficult to back it up with hard data.
> 
> Take a look at Nebraska for instance. Here, by law, electricity is a public 
> utility. And yet we have some of the lowest rates and highest uptime in the 
> country. No competition, low prices, stellar service record.
> 
> I’m generally all for private enterprise. But when those private enterprises 
> take public money, don’t do what they are supposed to do with it, squander 
> it, and nothing changes, again and again, well, what’s that definition of 
> insanity?
> 
> 
> Here in Lincoln, Nebraska, we actually do have fiber available at every 
> address in the city. And a private company did it. 100 percent underground, 
> all 96 square miles of the city. They did it all in about two years. I can 
> get 50Mbps synchronous for $45, 500 for $70 or gig for $99. TV and phone also 
> if I want it. Local support too, not India. 
> 
> They now have fiber in 15 Nebraska cities and two in Colorado and are 
> expanding rapidly. Why? A great product at a great price with outstanding 
> customer service. Spectrum is losing customers like crazy as a result, and 
> precisely zero people are shedding any tears (Spectrum salesmen excepted).
> 
> It can be done. Is it an investment? Yes. Just like anything else. Some 
> investments have a quicker return on capital than others.

+1 on the investment lifecycle requirement.  It can require a 10-20 year 
vision.  The problem we have right now is due to squirrel chasing on the part 
of some companies the money that could have been invested in locking in markets 
was spent on other investments.  You see a big difference between forward 
looking companies and their network performance and those that are backwards 
looking.

I had to build fiber to my house because the fiber near my home (about 1200’ 
away) was not in a position to be upgraded or maintained in such a way to 
deliver service to our area.  This is a very common trend I’ve observed.

My county did a large broadband survey where a contractor drove by every 
home/property to determine what was there.  Many addresses without service have 
multiple fiber providers at the road, it’s just not the “right fiber”.  This 
also includes spare conduit and space that was built out in a forward thinking 
model that others have to duplicate later because the assets are lost or 
forgotten in paperwork.

I also see a number of the smaller ISPs (and some bigger ones) who are like 
“you can watch Netflix and zoom, what’s your problem?” When there are end-users 
that are willing to pay for the higher speeds.  Not every home is going to 
spend $8k or $100k to get service, but they certainly do exist and make the 
business case more feasible.

- Jared

Re: Google IP Geolocation

[Jared Mauch]

I've had a similar issue in the past trying to get ready to peer with them. I 
wanted portal access to look at things. I may yet post a geofeed file just 
because. 

(I was also rejected a portal account, didn't escalate to friends at google 
because I know my traffic is under a gig for now). 

- Jared 
 
Ps: Akamai can take geofeed if you have issues

Sent from my TI-99/4a

> On Apr 10, 2021, at 4:57 AM, Laura Smith via NANOG  wrote:
> 
> Yup. I've had this problem with Google for two years now.
> 
> "We're Google. We know better than you. We're not interested in discussion. 
> And no, you can't have access to the ISP portal you silly little person"  
> .. is the summary of my experience.
> 
> And all this is despite my network peering with Google over a major IXP.  
> They *STILL* can't get the right geolocation despite having a direct peering 
> session with us over the exchange !
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐ Original Message ‐‐‐
>> On Monday, 29 March 2021 21:12, Troy Kelly via NANOG  wrote:
>> 
>> We've also been denied access to the ISP portal.
>> 
>> When we replied as to why, we were told to not open another ticket. They 
>> aren't interested in conversation.
>> 
>> Sent from ProtonMail mobile
>> 
>>  Original Message 
>>> On 30 Mar 2021, 6:53 am, Mike Hammett < na...@ics-il.net> wrote:
>>> 
>>> I've had others at Google specifically say that portal should be used for 
>>> that purpose, so maybe they need to make sure right and left hands know 
>>> what the other is doing.
>>> 
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>> 
>>> From: "Josh Luthman" 
>>> To: "Christopher Morrow" 
>>> Cc: nanog@nanog.org
>>> Sent: Monday, March 29, 2021 1:52:48 PM
>>> Subject: Re: Google IP Geolocation
>>> 
>>> https://isp.google.com
>>> 
>>> I signed up for an account there and they said:
>>> 
>>> "Currently, the Google ISP portal is designed for our partners of GGC, PNI 
>>> or IX programs.
>>>  
>>> The access to portal is granted on request only to our partners." 
>>> 
>>> Josh Luthman
>>> 24/7 Help Desk: 937-552-2340
>>> Direct: 937-552-2343
>>> 1100 Wayne St
>>> Suite 1337
>>> Troy, OH 45373
>>> 
 On Mon, Mar 29, 2021 at 2:08 PM Christopher Morrow 
  wrote:
>>> 
 On Mon, Mar 29, 2021 at 1:59 PM Josh Luthman  
 wrote:
 
> Google ISP specifically told me they didn't want to do deal with 
> geolocation on the ISP portal.
 
 unsure who 'google isp' is here, but ... I think if you point at a 
 properly formed geo-location data file it'll get eaten and produce proper 
 results for you.
 you do have to add that in your isp portal:
tri-pipe-thingy -> configuration -> IP geolocation
 and /register feed/ button on that page.
  
 
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
> 
> On Sat, Mar 27, 2021 at 3:28 PM Christopher Morrow 
>  wrote:
> 
>> As a note, on the ISP portal there's a place to put a link to your 
>> RFC8805 format geolocation feed...
>> these are scraped out 'regularly' and help keep things oriented better 
>> for folks.
>> 
>> the ietf noc folk use this method to tell google (and other folk who 
>> scrape out our data) where meetings are before the meetings get there:
>>   https://noc.ietf.org/geo/google.csv
>> 
>> -chris
>> volunteer noc persona
>> 
>> On Fri, Mar 26, 2021 at 6:43 PM Michael K. Spears  
>> wrote:
>> 
>>> Awesome, I think I’ve figured out the Google ISP portal signup, but it 
>>> definitely seems semi-complicated in a way, notably finding the link…
>>> 
>>>  
>>> 
>>> Thank you,
>>> 
>>> Michael K. Spears
>>> 
>>> 727.656.3347
>>> 
>>>  
>>> 
>>> From: Mike Hammett 
>>> Sent: Friday, March 26, 2021 6:30 PM
>>> To: Michael K. Spears 
>>> Cc: nanog@nanog.org
>>> Subject: Re: Google IP Geolocation
>>> 
>>>  
>>> 
>>> We're working on a video to show people how to sign up for the ISP 
>>> portal and get to that part of the portal once signed up. We'll drop a 
>>> link to it near the Google section of our geolocation page.
>>> 
>>> -
>>> 
>>> Mike Hammett
>>> 
>>> Intelligent Computing Solutions
>>> 
>>> http://www.ics-il.com
>>> 
>>> Midwest-IX
>>> 
>>> http://www.midwest-ix.com
>>> 
>>> From: "Michael K. Spears" 
>>> 
>>> To: nanog@nanog.org
>>> 
>>> Sent: Friday, March 26, 2021 5:10:23 PM
>>> 
>>> Subject: Google IP Geolocation
>>> 
>>> Anyone have a good contact at Google who can help with IP geolocation? 
>>> I have a /24 where anything related to Google is in the wrong language.
>>> 
>>> Thank you,
>>> 

Re: wow, lots of akamai

[Jared Mauch]

On Thu, Apr 01, 2021 at 08:09:24PM +, Luke Guillory wrote:
> IX’s don’t really help the source doesn’t use them.
> 
> Akamai traffic.
> 17G via Local Cache
> 17G via Transit
> 8G via IXs.
> 
> Plenty of room on IXs for more on our side.

Often we can see the ports at the IX flatline in these cases.
I've been working to improve some of them, for exampe at the DetroitIX
we have 300G of capacity and you can see the impact at the IX here:

https://www.detroitix.com/#stats

If you look at the monthly graph you can see other game download events.

It can be incredibly hard to right-size this stuff, please do reach out
to myself or Niels if things didn't work right for your network.  Also
you should reach out to your carriers who may have had congestion to
ensure they are upgrading their networks as well.  I've been told by
some teams they won't upgraded for a day a month of thsi big traffic,
but when one of their customers has issues they immediately escalate to
us to ask us to move (and we try hard to do that).

It's not all throw bandwidth at the problem, but with 100g optics being
so cheap these days, it may be the lower bar

    - jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: wow, lots of akamai

[Jared Mauch]

On Thu, Apr 01, 2021 at 03:31:39PM -0400, Dave Brockman - DVS wrote:
> On 4/1/2021 3:21 PM, Niels Bakker wrote:
> > * nanog@nanog.org (Jean St-Laurent via NANOG) [Thu 01 Apr 2021, 21:03
> > CEST]:
> >> An artificial roll out penalty somehow? Probably not at the ISP level,
> >> but more at the game level. Well, ISP could also have some mechanisms
> >> to reduce the impact or even Akamai could force a progressive roll out.
> > 
> > It's an online game. You can't play the game with outdated assets. You'd
> > not see walls where other players would, for example.
> > 
> > What you're suggesting is the ability of ISPs to market Internet access
> > at a certain speed but not have to deliver it based on conditions they
> > create.
> > 
> > 
> > -- Niels.
> 
> 
> There was at least one online gaming platform that used to stage their
> game updates, where clients would download the update over the previous
> week or so, and then on go-live day, the game updated itself.  There
> were a few that had to download on go-live day, but the clients I ran,
> and friends ran, had downloaded prior.  I don't know if it is still that
> way, but it seems like reasonable solution to me.

Actually in the past year many of the gaming studios have
started to do this for the release process.  It will start during an
off-peak hour and then flow through the day.  The end-user induced
demand and stress it places on networks doesn't fit into the historical
95/5 30-day peak planning model.  I know some carriers are struggling to
adjust.  There's a few of us here on-list, please reach out so we can
help.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: wow, lots of akamai

[Jared Mauch]

> On Apr 1, 2021, at 11:15 AM, Töma Gavrichenkov  wrote:
> 
> Peace,
> 
> On Thu, Apr 1, 2021, 6:09 PM  wrote:
> That was a lot of traffic coming out of akamai aanp clusters the last couple 
> nights!  What was it?
> 
> "Call of Duty" update again, obviously.
> 
> https://www.eurogamer.net/articles/2021-03-29-this-weeks-call-of-duty-warzone-update-is-over-50gb


Yes, we have had a number of big customer traffic in recent days.  Hopefully 
traffic is flowing well for many networks.  If this is negatively impacting 
you, please reach out.

- Jared

Re: Trouble Playing Multiplayer Games from Reallocated IP Space

[Jared Mauch]

Tim, I'll get you off-list.

- Jared

On Wed, Mar 03, 2021 at 10:37:44AM -0500, Tim Nowaczyk wrote:
> Hey NANOG,
> 
> We have seen an issue where our customers who have IP addresses that are 
> directly allocated to us can play online multiplayer games (NBA2k, NBA2k21, 
> Fallout 76, and Stardew Valley were mentioned specifically) but when they 
> have an IP that was reallocated to us by one of our Upstreams (Server 
> Central), these games no longer work. I know NBA2k is behind Prolexic, but 
> not sure about the others. 2k Games says that we, the ISP, must be blocking 
> something, but there’s absolutely no difference in how data moves from our 
> internet edge to the customer whether they have one of our IPs or one of the 
> reallocated ones. We noticed that one geocoding provider has our reallocated 
> IPs flagged as “hosting” IPs and maybe 2k is using some metadata like that to 
> block our IPs.
> 
> Does anyone have a contact at Prolexic who might be able to help?
> 
> Thanks,
> Tim Nowaczyk
> 
> --  
> Timothy Nowaczyk  |  Senior Network Manager
> office  703.554.6622   |  mobile  571.318.9434 
> 

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Famous operational issues

[Jared Mauch]

On Thu, Feb 18, 2021 at 01:07:01AM -0800, Eric Kuhnke wrote:
> On that note, I'd be very interested in hearing stories of actual incidents
> that are the cause of why cardboard boxes are banned in many facilities,
> due to loose particulate matter getting into the air and setting off very
> sensitive fire detection systems.
> 
> Or maybe it's more mundane and 99% of the reason is people unpack stuff and
> don't always clean up properly after themselves.

We had a plastic bag sucked into the intake of a router in a
datacenter once that caused it to overheat and take the site down.  We
had cameras in our cage and I remember seeing the photo from the site of
the colo (I'll protect their name just because) taken as the tech was on
the phone and pulled the bag out of the router.

The time from the thermal warning syslog that it's getting warm
to overheat and shutdown is short enough you can't really get a tech to
the cage in time to prevent it.

I assume also the latter above, which is people have varying
definitons of clean.

    - Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: Famous operational issues

[Jared Mauch]

The he.net side is interesting as you can see who their v4 transits are but 
they suppress their routes via v6, but (last I knew) lacked community support 
for their customers to do similar route suppression.

I’m not a fan of it, but it makes the commercial discussions much easier each 
time those networks come by to shop services to me in a personal or 
professional capacity.  “No, I need all the internet”.

- Jared

> On Feb 17, 2021, at 12:07 PM, David Guo via NANOG  wrote:
> 
> Cogentco still did not peer with Google and HE over IPv6 I guess.
> 
> From: NANOG  on behalf of Justin 
> Wilson (Lists) 
> Sent: Thursday, February 18, 2021 00:53
> To: Miles Fidelman
> Cc: nanog@nanog.org
> Subject: Re: Famous operational issues
>  
> I remember when the big carriers de-peered with Cogent in the early 2000s.  
> The underestimated the amount of web-sites being hosted by people using 
> cogent exclusively. 
> 
> 
> Justin Wilson
> j...@j2sw.com
> 
> —
> https://j2sw.com - All things jsw (AS209109)
> https://blog.j2sw.com - Podcast and Blog
> 
> > On Feb 17, 2021, at 10:29 AM, Miles Fidelman  
> > wrote:
> > 
> > John Kristoff wrote:
> >> Friends,
> >> 
> >> I'd like to start a thread about the most famous and widespread Internet
> >> operational issues, outages or implementation incompatibilities you
> >> have seen.
> >> 
> > Well... pre-Internet, but the great Northeast fiber cut comes to mind 
> > (backhoe vs. fiber, backhoe won).
> > 
> > Miles Fidelman
> > 
> > -- 
> > In theory, there is no difference between theory and practice.
> > In practice, there is.   Yogi Berra
> > 
> > Theory is when you know everything but nothing works. 
> > Practice is when everything works but no one knows why. 
> > In our lab, theory and practice are combined: 
> > nothing works and no one knows why.  ... unknown

Re: Famous operational issues

[Jared Mauch]

I was thinking about how we need a war stories nanog track. My favorite was 
being on call when the router was stolen. 

Sent from my TI-99/4a

> On Feb 16, 2021, at 2:40 PM, John Kristoff  wrote:
> 
> Friends,
> 
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?
> 
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
> 
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
> 
> Thanks in advance for your suggestions,
> 
> John

Re: Texas internet connectivity declining due to blackouts

[Jared Mauch]

> On Feb 16, 2021, at 8:25 AM, Mike Hammett  wrote:
> 
> It's cheaper to build 2x, 3x, 4x the aerial plant than to build 1x the 
> underground plant.
> 
> The actual cost per foot is more like 10x difference, but there are right of 
> way, maintenance, etc. costs to factor in as well.
> 

Labor is something in 8x but the permit/engineering cost is usually the same 
per foot, but the make-ready on poles can make underground competitive or more 
like 1.5x when you fully bake the costs.  Really depends on pole distances and 
quality.  O long-term is lower on underground vs aerial.

- Jared

Re: Texas internet connectivity declining due to blackouts

[Jared Mauch]

Almost exactly 4 years ago we were out up here in Michigan for over 120 hours 
after a wind storm took out power to 1 million homes. Large scale restoration 
takes time. When the load and supply are imbalanced it can make things worse as 
well. 

I'm hoping things return to normal soon but also am reminded it can take some 
time. 

We now have a large generator with automatic switchover after that event. 
Filling gas cans every 12 hours to feed the generator is no fun. 

Sent from my TI-99/4a

> On Feb 15, 2021, at 11:54 PM, Cory Sell via NANOG  wrote:
> 
>  Total population is a pretty big difference as you go north, as is how well 
> infrastructure is actually prepared for snow/ice and cold temperatures in 
> general.
> 
> I’ve been without power all day and have no doubt I’ll cross the 24-hour mark 
> here in a handful of hours.
> 
> Sent from ProtonMail Mobile
> 
> 
>> On Mon, Feb 15, 2021 at 10:42 PM, Sean Donelan  wrote:
>> 
>> 
>> On Tue, 16 Feb 2021, Cory Sell via NANOG wrote:
>> > adoption. Sure, wind isn’t perfect, but looks like solution relied on 
>> > failed
>> > in a massive way.
>> 
>> Strange the massive shortages and failures are only in one state.
>> 
>> The extreme cold weather extends northwards across many states, which
>> aren't reporting rolling blackouts.
> 
> 

Re: New York Carrier Hotels

[Jared Mauch]

I’m expecting many people to move out to 165 Halsey but as with many things the 
future is still hazy.  I also wonder if at some point Google will decide that 
WFH is viable and they don’t need the office space in 111 8th and things will 
swing back..

(Yes, I know that 165 isn’t in NY)

- Jared

> On Feb 11, 2021, at 1:51 PM, Rod Beck  wrote:
> 
> Hey Folks, 
> 
> I am looking for a list of the ten most important NYC telecom hotels. Over 
> the last 15 years carrier business has shifted to a large extent to Secaucus 
> Equinix & Google has taken over a big part of 111 8th Avenue. What the 
> important sites today and are any new facilities on the horizon?
> 
> Roderick Beck
> Global Network Capacity Procurement
> United Cable Company
> www.unitedcablecompany.com
> https://unitedcablecompany.com/video/
> New York City & Budapest
> rod.b...@unitedcablecompany.com
> Budapest: 36-70-605-5144
> NJ: 908-452-8183 

Summary: advertise-peer-as

[Jared Mauch]

I’ve closed the form for responses.

There were 102 people who self-selected to participate. 

Around 25% of you have this configured as a default in your network.

At 20940 we are asking our networking partners to configure this facing us such 
that we can receive the routes from some of the stub networks we operate.

I was also told there’s some interesting behavior in IOS-XR, if a peer is in 
the same peer-group and based on the order of the peers coming up, the behavior 
may be different where routes may be suppressed.

You may also want to read up on the "send-transit-peer” command for your 
network.

Thanks for responding, and if you have a BGP peer with 20940 please set 
advertise-peer-as or similar.  It may help keep more traffic on direct links as 
a result.

- jared

advertise-peer-as

[Jared Mauch]

I’m curious how many networks have this as part of their default configuration?

If you’re bored on a weekend please click through to this and say yes/no.  I’ll 
summarize in a week or so.  If I configured the form right, it should also show 
you the results as well.

https://forms.gle/xxLnA7KgQL48VMNe8

- Jared

Re: 10g residential CPE

[Jared Mauch]

> On Dec 26, 2020, at 10:35 AM, Mikael Abrahamsson via NANOG  
> wrote:
> 
> Here the "truth" is that if you game, you need to have a wired connection to 
> your gaming computer. All gamers "know" this.

My sons switch is hard wired, he gets considerable advantage (apparently) due 
to using the USB adapter vs wifi when playing online.

- Jared

Re: [External] 10g residential CPE

[Jared Mauch]

> On Dec 25, 2020, at 5:32 PM, John Levine  wrote:
> 
> I agree it is odd to make 100/100 the top speed. The fiber service I
> have from my local non-Bell telco offers 100/100, 500/500, and
> 1000/1000. FiOS where you can get it goes to 940/880.
> 
> The obvious guess is that their upstream bandwidth is
> underprovisioned, or maybe they figure 100/100 is all they need to
> compete in that particular market.

My TV (wired) pulls at higher bitrates when doing the initial fetches of the 
buffering.  Not unusual to see it pulling more than 150Mb/s at the start of a 
(non-4K) show.

I think the extent that end-users are impacted by these slower speeds while 
buffering is under appreciated in the experience.

At $dayjob many servers are 10G or 100G so the limiting factor is most likely 
the CPE or ISP.  I was hearing last night about someone with a device that 
didn’t appear to be hitting the line-rate but was dropping 0.5% of packets when 
running at 3Gb/s until they upgraded to one of the major networking vendors we 
all know here.

In my small FTTH network the slowest link is at the customer home and all the 
devices are hardware ASIC forwarded vs offload as you find in some of the 
low/mid-tier devices (eg: Tik/UBNT).

Many streaming things do 8 second waits between chunks, so if you’re pulling a 
video stream at 6Mb/s you really are pulling 6*8 (lets say 50) then idle for 7 
seconds.  If you’re on a 25Mb/s service or even a 50Mb/s service it won’t work 
the way you expect if there’s any other activity.

- Jared

Re: 10g residential CPE

[Jared Mauch]

> On Dec 25, 2020, at 12:48 PM, Bryan Fields  wrote:
> 
> 10g to the home is a great idea to think about, it's just not terribly
> practical for most customers unless they want to drop 1-2k on routing gear and
> nics.  This is always changing, but it's going to be a few years until we
> reach the right performance and price point.

Think more using your PON network to also serve commercial customers so you 
don't need high end CPE to hit 1-5Gbps or WDM setups. 

Re: Need someone with clue @ Network Solutions.

[Jared Mauch]

Matthew,

I haven’t seen this problem in a long time where someone else submits data to 
cause the out-of-zone glue to appear.  It’s possible there’s something 
happening at NETSOL that is causing this, but the best way is for you to go 
into your registrar and ensure they’re publishing the proper host records for 
your in-zone glue which should address this if nobody got back to you yet.  It 
may also be easier to find someone on the dns-operations list than NANOG these 
days.

- Jared

> On Dec 15, 2020, at 11:43 AM, Matthew Crocker  
> wrote:
> 
> I need to get Network Solutions to remove glue records for hosts in my 
> domain.   My domain isn’t registered with Network Solutions and they refuse 
> to speak with me as I’m not a customer.
>  
> I’ve had my customer attempt to update their domain through Network Solutions 
> but the only thing they can change is the NS record, not the underlying host 
> glue record.   I don’t think the glue records even need to exist as they are 
> published by my domain already.
>  
> Does anyone have any contacts at Network Solutions that can help?
>  
> Example:
>  
> dig .com NS @i.gtld-servers.net.
>  
> ; <<>> DiG 9.10.6 <<>> .com NS @i.gtld-servers.net.
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24593
> ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 3
> ;; WARNING: recursion requested but not available
>  
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 4096
> ;; QUESTION SECTION:
> ;.com.IN NS
>  
> ;; AUTHORITY SECTION:
> .com.  172800 IN NS dns-auth4.crocker.com.
> .com.  172800 IN NS dns-auth3.crocker.com.
>  
> ;; ADDITIONAL SECTION:
> dns-auth4.crocker.com.  172800 IN A  66.59.48.95
> dns-auth3.crocker.com.  172800 IN A  66.59.48.94
>  
> ;; Query time: 73 msec
> ;; SERVER: 192.43.172.30#53(192.43.172.30)
> ;; WHEN: Tue Dec 15 11:34:41 EST 2020
> ;; MSG SIZE  rcvd: 124
>  
> The correct servers are:
>  
> dns-auth3.crocker.com.  299IN A  66.59.61.10
> dns-auth4.crocker.com.  299IN A  66.59.61.194