Re: Meta outage

[Dave Taht]

My first thought, I'll admit, is that they turned a LLM loose on their
configuration controls, and it "naturally" put itself in route 666.

I sure hope from whatever subbasement clueful admins are still in over
there, they can put it back together. Otherwise, productivity
might improve as users flee to Office 365 to get some work done.

I keep worrying about KEYTRAP

--
https://www.youtube.com/watch?v=N0Tmvv5jJKs Epik Mellon: Making the
Internet Better
Dave Täht CSO, LibreQos

starlink ixp peering progress

[Dave Taht]

One of the things I learned today was that starlink has published an
extensive guide as to how existing BGP AS holders can peer with them
to get better service.

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink

I am curious if there is a way to see how many have peered already,
how many they could actually peer with?, and progress over time since
inception what would be the right tools for that? This is pretty
impressive for peering so far:

https://www.peeringdb.com/net/18747

Is there a better email list to discuss ixp stuff?


-- 
https://blog.cerowrt.org/post/2024_predictions/
Dave Täht CSO, LibreQos

Re: [External] Re: IPv6 uptake

[Dave Taht]

OpenWrt, from which much is derived, is default deny on ipv4 and ipv6.

The ipv6 firewall on most cable devices prior to the XB6 is very, very limited.

On Mon, Feb 19, 2024 at 12:44 PM William Herrin  wrote:
>
> On Mon, Feb 19, 2024 at 9:23 AM Hunter Fuller  wrote:
> > On Mon, Feb 19, 2024 at 11:16 AM William Herrin  wrote:
> > > > There isn't really an advantage to using v4 NAT.
> > > I disagree with that one. Limiting discussion to the original security
> > > context (rather than the wider world of how useful IPv6 is without
> > > IPv4), IPv6 is typically delivered to "most people" without border
> > > security, while IPv4 is delivered with a stateful NAT firewall.
> >
> > Maybe this is the disconnect. Who delivers v6 without a firewall?
> >
> > I've done a lot of T-Mobile and Comcast business connections lately,
> > and those certainly both provide a firewall on v4 and v6. I'll admit
> > I'm not currently well-versed in other providers (except ones that
> > don't provide v6 at all...).
>
> Hi Hunter,
>
> You may be right. I haven't ordered SOHO service in a long time and in
> fairness you were talking about Joe's Taco Shop not Joe's home
> network.
>
> I -suspect- that the wifi router provided for Joe's home network
> doesn't do much more than plain routing on the IPv6 side but I do not
> know that for a truth. I ordered my wave and comcast services without
> a router and I didn't keep the centurylink router long enough to test
> whether it did any filtering on IPv6. I noticed no knobs for IPv6
> filtering or port forwarding, so I suspect it did not.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: [External] Re: IPv6 uptake

[Dave Taht]

On Mon, Feb 19, 2024 at 11:13 AM Hunter Fuller via NANOG
 wrote:
>
> On Mon, Feb 19, 2024 at 9:29 AM Mike Hammett  wrote:
> > "In IPv6's default operation, if Joe has two connections then each of
> > his computers has two IPv6 addresses and two default routes. If one
> > connection goes down, one of the routes and sets of IP addresses goes
> > away."
> >
> > This sounds like a disaster.
>
> You know, I thought so too, until I deployed it and it worked fine.

Years ago we made "source specific routing" the default in openwrt.
This means all hosts get both sets of prefixes, and naturally retry
other src addresses.

To what extent anyone else has adopted this is unknown. The popular
mwan3 code is kind of hairy vs a vs ipv6 here.



> I have done it twice now, once on MikroTik RouterOS and once on
> Ubiquiti EdgeOS. You just have to make sure the timers are pretty
> short, and that the router will stop sending RAs for the route if it's
> not working. This is definitely something that a COTS SOHO dual WAN
> router, that Joe would buy, could and should do by default (hopefully
> they do; I just haven't checked).
>
> --
> Hunter Fuller (they)
> Router Jockey
> VBH M-1C
> +1 256 824 5331
>
> Office of Information Technology
> The University of Alabama in Huntsville
> Network Engineering



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: [External] Re: IPv6 uptake

[Dave Taht]

mdns can still be "fun" in a wide variety of situations.

https://www.reddit.com/r/k12sysadmin/comments/9yghdx/chromebooks_and_peer_to_peer_updates_can_be/

I do not know to what extent the upgrade to unicast feature long
gestating in the IETF has been adopted.

On Mon, Feb 19, 2024 at 11:10 AM Hunter Fuller via NANOG
 wrote:
>
> On Mon, Feb 19, 2024 at 9:17 AM William Herrin  wrote:
> > There's also the double-ISP loss scenario that causes Joe to lose all
> > global-scope IP addresses. He can overcome that by deploying ULA
> > addresses (a third set of IPv6 addresses) on the internal hosts, but
> > convincing the internal network protocols to stay on the ULA addresses
> > is wonky too.
>
> In the real world today, most applications seem to use mDNS and
> link-local addresses to keep this connectivity working. (I am guessing
> Joe's Taco Shop uses something like Square, and just needs his
> register to talk to his printer. This already works with mDNS and
> link-locals today.)
>
> --
> Hunter Fuller (they)
> Router Jockey
> VBH M-1C
> +1 256 824 5331
>
> Office of Information Technology
> The University of Alabama in Huntsville
> Network Engineering



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

dnssec keytrap vuln

[Dave Taht]

Really long list of fixed dns servers here:

https://www.linkedin.com/posts/bwoodcock_a-bunch-of-really-hard-work-over-the-past-activity-7163284274660532224-vYKv
-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: The Reg does 240/4

[Dave Taht]

I attempted with as much nuance and humor as I could muster, to
explain and summarize the ipv4 exhaustion problem, and CGNAT, the
240/4 controversy as well as the need to continue making the IPv6
transition, on this podcast yesterday.

https://hackaday.com/2024/02/14/floss-weekly-episode-769-10-more-internet/

Enjoy.

Re: Utilizing USG networks for internal purposes (Re: route: 0.0.0.0/32 in LEVEL3 IRR)

[Dave Taht]

Excellent summary of the USG position as of 2019. It is, um, nearly 5
years later, has any of these stuff evolved?

On Tue, Feb 13, 2024 at 9:58 PM John Curran  wrote:
>
> On Jan 31, 2024, at 12:48 AM, Rubens Kuhl  wrote:
>
> DoD's /8s are usually squatted by networks that run out of private IPv4 space.
> Even though it is very risky to steal resources from an organization
> that can deploy a black helicopter or a nuclear warhead over you, for
> some reason like it not appearing in the DFZ people seem to like it.
>
>
> Folks -
>
> A network that wants to be creative and utilize an address block that’s 
> assigned to others
> for their own internal purposes runs two distinct risks:
>
> 1. An address block that’s not utilized today may easily become publicly 
> routed tomorrow
> (either by the original address holder or by their assignee/successor) 
> and it is not possible
> to reliably predict whether your customers will need access to the 
> resources that end up
> on that address space.
>
> 2. If you should leak routes publicly for another's address space, there are 
> organizations that
> will object – and in the case US government networks, this can include 
> some uncomfortable
> conversations.  [1]
>
> None of this suggests that one cannot configure their routers any way that 
> they wish – just that
> it’d be best if done with appropriate care and an upfront understanding of 
> the risks involved.
>
> Thanks!
> /John
>
> John Curran
> President and CEO
> American Registry for Internet Numbers
>
> [1] 
> https://pc.nanog.org/static/published/meetings/NANOG77/2108/20191028_Elverson_Your_As_Is_v1.pdf
>  pg 4.
>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: The Reg does 240/4

[Dave Taht]

On Tue, Feb 13, 2024 at 2:18 AM Jay R. Ashworth  wrote:
>
> - Original Message -
> > From: "Dave Taht" 
>
> > The angst around ipv6 on hackernews that this triggered was pretty
> > revealing and worth thinking about independently.
> > https://news.ycombinator.com/item?id=39316266
>
> Thanks; the source where I got the other link mentioned that, and I meant
> to include it...
>
> > I was inspired to try a couple traceroutes. It used to be 240 escaped
> > my prior comcast router and wandered around a while; it does not do
> > that anymore. I would be dryly amused if that box was actually running
> > my old OpenWrt bcp38 stuff which blocked 240 for a couple years. My
> > cloud works, my aws stack works, openwrt works.
>
> Damn; I haven't touched the bcp38 wiki in some time.

In what way do you plan to touch it?

> Thanks for the reminder.

The bcp38 code for OpenWrt was not updated in light of the nftables
switch, as of a few years ago, but I have not looked at it in a long
time. Maybe someone else fixed it. I have not been doing much
development.

As it is bcp38 needs to be applied carefully by an ISP given the
sordid mess of other rfc1918 addresses along the path nowadays.

I doubt it is a good idea for consumer devices anymore. I liked the
side-effects of running it then tho, stopping random worms for chewing
up my external bandwidth. (the code was not just bcp38 related)

A plug - that I have NO IDEA made it into other ipv6 implementations -
is that we put ipv6 source specific routing into the OpenWrt stack to
elegantly make bcp38-like behavior the default there, back in 2013.

ip route add :: from my:ipv6:address:ranges/mask dest:addr:of:your:choice.

And also made the idea work in babel and ISIS to help with poor man´s
multihoming.

Most distro kernels I have seen lately do not seem to support "from" anymore.

>
> > Peering into a murky crystal ball, say, 5 years in the future:
> >
> > Another thing that I worry about is port space exhaustion, which is
> > increasingly a thing on firewalls and CGNs. If I can distract you - in
> > this blog cloudflare attempted to cut the number of ipv4 addresses
> > they use from 2 to 1, after observing some major retry issues. With a
> > nice patch, reducing the problem.
> >
> > https://blog.cloudflare.com/linux-transport-protocol-port-selection-performance/
>
> Interesting.  Isn't that something CGNAT implementers would have had to deal 
> with
> already?

I do not know of a single CGNAT that gives an operator a report on syn
retries, and thus exhaustion is hidden by the native retry behaviors
of the host stacks. Is there one?

The cloudflare work seems helpful here.

>
> > Peering further into the soi-distant decades ahead, perhaps we should
> > just allocate all the remaining protocol space in the IP header to a
> > quic native protocol, and start retiring the old ones.
>
> Well, I've been able to avoid thinking about it for some time, but ISTR my
> reaction to QUIC as violating a number of organized religions' blasphemy
> rules...
>
> > /me hides
>
> Indeed.

I enjoy being offline ever the more, these days. The internet
addiction level out there has become rather depressing.

https://www.linkedin.com/feed/update/urn:li:activity:7162457657210044416/


>
> Cheers,
> -- jra
> --
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: The Reg does 240/4

[Dave Taht]

The angst around ipv6 on hackernews that this triggered was pretty
revealing and worth thinking about independently.
https://news.ycombinator.com/item?id=39316266

In the tik world, people are struggling to deploy ipv6 as even linux
kernel 5.7 in routerOS 7.XX still has some needed missing features. It
also appears 240 ain´t working there, either. And routerOS is one of
the more up to date platforms.

if I could use the controversy to talk to why it has been so hard to
deploy ipv6 to the edge and how to fix that problem instead rather
than triggering people, it would be helpful.

...

I was inspired to try a couple traceroutes. It used to be 240 escaped
my prior comcast router and wandered around a while; it does not do
that anymore. I would be dryly amused if that box was actually running
my old OpenWrt bcp38 stuff which blocked 240 for a couple years. My
cloud works, my aws stack works, openwrt works.

My comcast ipv6 connection is LOVELY - ssh stays nailed up for days. I
still reflexively use mosh because it survives me moving from AP to
AP.

I do wish there was some way I could escape the painful policy debate
and just focus on the code-related problems. (my position is basically
that all new devices not waste cycles blocking the 240 and 0/8 ranges,
and merely it move it from reserved for bezos^H^H^H^H^Hfuture use to
unicast and recognize deployed reality).

Peering into a murky crystal ball, say, 5 years in the future:

Another thing that I worry about is port space exhaustion, which is
increasingly a thing on firewalls and CGNs. If I can distract you - in
this blog cloudflare attempted to cut the number of ipv4 addresses
they use from 2 to 1, after observing some major retry issues. With a
nice patch, reducing the problem.

https://blog.cloudflare.com/linux-transport-protocol-port-selection-performance/

Their problems remain the same if they also just use one ipv6 address
(which would be silly, of course). QUIC is going to make this worse.

In there, they mention udp-lite, but don´t mention that this protocol
has worked for over a decade, and has all this unallocated port space.
Firewalling and natting it is easy.

Peering further into the soi-distant decades ahead, perhaps we should
just allocate all the remaining protocol space in the IP header to a
quic native protocol, and start retiring the old ones.

/me hides

On Tue, Feb 13, 2024 at 1:21 AM Jay R. Ashworth  wrote:
>
> I know we had a thread on this last month, but I can't remember what it
> was titled.
>
> ElReg has done a civilian-level backgrounder on the 240/4 issue, for anyone
> who wants to read and scoff at it.  :-)
>
> https://www.theregister.com/2024/02/09/240_4_ipv4_block_activism/
>
> Cheers,
> -- jra
>
> --
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: route: 0.0.0.0/32 in LEVEL3 IRR

[Dave Taht]

That's pretty cool, actually. I keep wondering when someone will offer
up a 0.0.0.0/8...

https://www.ietf.org/archive/id/draft-schoen-intarea-unicast-0-00.html

There must be more people out there than just amazon and google that
ran out of 10/8.

On Tue, Jan 30, 2024 at 11:29 AM Frank Habicht  wrote:
>
> Hi,
>
> I got 2 bounces for the email addresses seen below for an email similar
> to the below...
>
> Anyone want to remove this IRR entry before anyone notices...???   ;-)
>
> Frank
>
>
> I believe that the entry of
> route:  0.0.0.0/32
>
> does not serve any good purpose?
>
> I was surprised to see it in a list of prefixes from bgpq4 and the very
> good https://irrexplorer.nlnog.net/prefix/0.0.0.0 guided me that it's in
> "Level3".
>
> I'm wondering how many auto-generated filters contain this unnecessary
> prefix
>
> PS: Oh, just seen - it's from TODAY. Maybe remove before anyone sees it...?
>
> Thanks for looking into this,
> Frank
>
>
> [frank@fisi ~]$ whois -h rr.level3.com 0.0.0.0/32
> [Querying rr.level3.com]
> [rr.level3.com]
> route:  0.0.0.0/32
> origin: AS10753
> mnt-by: TCCGlobalNV-MNT
> changed:ankita.gre...@lumen.com
> source: LEVEL3
> last-modified:  2024-01-30T11:04:49Z
>
>
> [frank@fisi ~]$ whois -h rr.level3.com TCCGlobalNV-MNT
> [Querying rr.level3.com]
> [rr.level3.com]
> mntner: TCCGlobalNV-MNT
> descr:  TCC Global N.V.
> auth:   CRYPT-PW DummyValue  # Filtered for security
> upd-to: ripehostmas...@eu.centurylink.net
> tech-c: LTHM
> admin-c:LTHM
> mnt-by: TCCGlobalNV-MNT
> changed:ankita.gre...@lumen.com
> source: LEVEL3
> last-modified:  2024-01-30T11:01:52Z
>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: 202401100645.AYC Re: IPv4 address block

[Dave Taht]

On Thu, Jan 11, 2024 at 12:26 PM Tom Beecher  wrote:
>
> Christopher-
>
>> Reclassifying this space, would add 10+ years onto the free pool for each 
>> RIR. Looking at the APNIC free pool, I would estimate there is about 1/6th 
>> of a /8 pool available for delegation, another 1/6th reserved. 
>> Reclassification would see available pool volumes return to pre-2010 levels.
>
>
> Citing Nick Hilliard from another reply, this is an incorrect statement.
>
>> on this point: prior to RIR depletion, the annual global run-rate on /8s
>> measured by IANA was ~13 per annum. So that suggests that 240/4 would
>> provide a little more than 1Y of consumption, assuming no demand
>> back-pressure, which seems an unlikely assumption.
>
>
>> I share Dave's views, I would like to see 240/4 reclassified as unicast 
>> space and 2 x /8s delegated to each RIR with the /8s for AFRINIC to be held 
>> until their issues have been resolved.
>
>
> This has been discussed at great length at IETF. The consensus on the 
> question has been consistent for many years now; doing work to free up 12-ish 
> months of space doesn't make much sense

Thought that 12 month argument was purest BS in light of all the
events since 2011. We have been "out" of IPv4 space for many years
now, and there is a functioning market for IPv4 space that seems to be
serving the purpose. 240/4 is only marginally useful today, but useful
it is.

> when IPv6 exists, along with plenty of transition/translation mechanisms. 
> Unless someone is able to present new arguments that change the current 
> consensus, it's not going to happen.

Instead, bigcos like google and amazon have been able to squat on
240/4 and take advantage of it for 5+ years now. I do kind of hope
others are using it up in the same ways they are.

Consensus, no. Just the few, like my team, that looked clearly at the
future internet's needs getting shouted down by those in power over
there. We cited many other arguments in favor of opening it up. There
are rumblings far outside the realm of the ietf.

I was once naive enough to consider the internet a vast, global,
shared, and beloved space with resources that needed to be conserved
and spread to and for all mankind. And while I still do feel that, our
existing bureaucracies and gatekeepers have seemingly infinite power
to say no, to even simple improvements to how the internet could work.

There is no reason whatsoever for 240/4 to remain "reserved". There
are plausible debates as to how it should be used, but rfc1918-style
only benefits the few.

> On Thu, Jan 11, 2024 at 5:54 AM Christopher Hawker  
> wrote:
>>
>> There really is no reason for 240/4 to remain "reserved". I share Dave's 
>> views, I would like to see 240/4 reclassified as unicast space and 2 x /8s 
>> delegated to each RIR with the /8s for AFRINIC to be held until their issues 
>> have been resolved.
>>
>> Reclassifying this space, would add 10+ years onto the free pool for each 
>> RIR. Looking at the APNIC free pool, I would estimate there is about 1/6th 
>> of a /8 pool available for delegation, another 1/6th reserved. 
>> Reclassification would see available pool volumes return to pre-2010 levels.
>>
>> https://www.apnic.net/manage-ip/ipv4-exhaustion/
>>
>> In the IETF draft that was co-authored by Dave as part of the IPv4 Unicast 
>> Extensions Project, a very strong case was presented to convert this space.
>>
>> https://www.ietf.org/archive/id/draft-schoen-intarea-unicast-240-00.html
>>
>> Regards,
>> Christopher Hawker
>>
>> On Thu, 11 Jan 2024 at 20:40, Dave Taht  wrote:
>>>
>>> On Wed, Jan 10, 2024 at 11:06 AM Tom Beecher  wrote:
>>> >>
>>> >> There's a whole bunch of software out there that makes certain
>>> >> assumptions about allowable ranges. That is, they've been compiled with
>>> >> a header that defines ..
>>> >
>>> >
>>> > Of course correct. It really depends on the vendor / software / versions 
>>> > in an environment. A lot of vendors removed that years ago, because 
>>> > frankly a lot of large networks have been using 240/4 as pseudo RFC1918 
>>> > for years. Others have worked with smaller vendors and open source 
>>> > projects to do the same.
>>> >
>>> > It's consistently a topic in the debates about 240/4 reclassification.
>>>
>>> There's debates still? I gave up. After making 240/4 and 0/8 work
>>> across all of linux and BSD and all the daemons besides bird (which
>>> refused the patch , I took so much flack that I decided I would just
>>

Re: 202401100645.AYC Re: IPv4 address block

[Dave Taht]

On Wed, Jan 10, 2024 at 11:06 AM Tom Beecher  wrote:
>>
>> There's a whole bunch of software out there that makes certain
>> assumptions about allowable ranges. That is, they've been compiled with
>> a header that defines ..
>
>
> Of course correct. It really depends on the vendor / software / versions in 
> an environment. A lot of vendors removed that years ago, because frankly a 
> lot of large networks have been using 240/4 as pseudo RFC1918 for years. 
> Others have worked with smaller vendors and open source projects to do the 
> same.
>
> It's consistently a topic in the debates about 240/4 reclassification.

There's debates still? I gave up. After making 240/4 and 0/8 work
across all of linux and BSD and all the daemons besides bird (which
refused the patch , I took so much flack that I decided I would just
work on other things. So much of that flack was BS - like if you kill
the checks in the OS the world will end - that didn't happen. Linux
has had these two address ranges just work for over 5 years now.

240/4 is intensely routable and actually used in routers along hops
inside multiple networks today, but less so as a destination.

I would really like, one day, to see it move from reserved to unicast
status, officially. I would have loved it if 0/8 was used by a space
RIR, behind CGNAT, for starters, but with a plan towards making it
routable. I am not holding my breath.

The principal accomplishment of the whole unicast extensions project
was to save a nanosecond across all the servers in the world on every
packet by killing the useless 0/8 check. That patch paid for itself
the first weekend after that linux kernel deployed. It is the
simplest, most elegant, and most controversial patch I have ever
written.

https://news.ycombinator.com/item?id=20430096


>
> On Wed, Jan 10, 2024 at 10:45 AM Michael Butler  
> wrote:
>>
>> On 1/10/24 10:12, Tom Beecher wrote:
>> > Karim-
>> >
>> > Please be cautious about this advice, and understand the full context.
>> >
>> > 240/4 is still classified as RESERVED space. While you would certainly
>> > be able to use it on internal networks if your equipment supports it,
>> > you cannot use it as publicly routable space. There have been many
>> > proposals over the years to reclassify 240/4, but that has not happened,
>> > and is unlikely to at any point in the foreseeable future.
>>
>> While you may be able to get packets from point A to B in a private
>> setting, using them might also be .. a challenge.
>>
>> There's a whole bunch of software out there that makes certain
>> assumptions about allowable ranges. That is, they've been compiled with
>> a header that defines ..
>>
>> #define IN_BADCLASS(i)  (((in_addr_t)(i) & 0xf000) == 0xf000)
>>
>> Michael
>>


-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: Sufficient Buffer Sizes

[Dave Taht]

Hoo, boy. This is now such an old debate that I do not know where to
start anymore.

I am of the firm opinion nowadays that if you are buffering more than
a few ms at these enormous speeds, you are doing it wrong, and
regardless https://arxiv.org/abs/2109.11693 seems to hold as for
highly multiplexed traffic.

My outside number for a FIFO buffer in the modern CDN´d world is a
mere 30ms at lower speeds which allows for good gaming and
videoconferencing experiences, and good performance with modern paced
transports (in general linux now does packet pacing across all
congestion controls) and with a good head drop AQM and FQ algo, far,
far less buffering is feasible across the board.

https://blog.cerowrt.org/post/juniper/

But regrettably not available at 100Gbit yet (tho libreqos is coming close)

On Tue, Jan 2, 2024 at 6:22 PM William Herrin  wrote:
>
> On Tue, Jan 2, 2024 at 3:02 PM Mike Hammett  wrote:
> > While attempting to ascertain how big of switch buffers I needed in a 100G 
> > switch, I rediscovered this article where I first learned about switch 
> > buffers.
> >
> > https://fasterdata.es.net/network-tuning/router-switch-buffer-size-issues/#:~:text=Optimum%20Buffer%20Size=The%20general%20rule%20of%20thumb,1G%20host%20across%20the%20WAN.
> >
> > It suggests that 60 meg is what you need at 10G. Is that per interface? 
> > Would it be linear in that I would need 600 meg at 100G?
> >
> > Some 100G switches I was looking at only had 36 megs, so that's 
> > insufficient either way you look at it.
>
> Hi Mike,
>
> My thoughts:
>
> 1. 50 ms is -way- too much buffer. A couple links like that in the
> path and the user will suffer jitter in excess of 100ms which is
> incredibly disruptive for interactive applications.
>
> 2. The article discussed how much buffer to apply to the -slower-
> interfaces, not the faster ones, the idea being that data entering
> from the faster interfaces could otherwise overwhelm the slower ones
> resulting in needless retransmission and head-end blocking. Are the
> 100G interfaces on your switch the -slower- ones?
>
>
> I don't know the best number, but I suspect the speed at which packets
> clear an interface is probably a factor in the equation, so that the
> reasonable buffer depth in ms when a packet clears in 1ms is probably
> different than the reasonable buffer depth when a packet clears in 1
> us.
>
> Regards,
> Bill Herrin
>
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



-- 
40 years of net history, a couple songs:
https://www.youtube.com/watch?v=D9RGX6QFm5E
Dave Täht CSO, LibreQos

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Dave Taht]

On Fri, Dec 1, 2023 at 6:21 PM Tom Samplonius  wrote:
>
>
> > https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
> >
> > Us bufferbloat folk have been putting together a response to the FCC's
> > NOI (notice of inquiry) asking for feedback as to increasing the
> > broadband speeds beyond 100/20 Mbit.
>
>
>   The era of “buffer bloat” has passed.  Buffer bloat is just about jitter, 
> and jitter mitigation is just better now.

I am really not sure what world you live in. Certainly the problem has
got much better since 2010, but nearly
every network I ever explore still has vast amounts of it. The global
south, every coffee shop, and every hotel
I have been in, all have major issues here.  I make a habit of testing
the conference wifi and posting results
at every talk. The apartment complex I am staying in this week has a completely
saturated backhaul and oversubscribed wifi, with tremendous amounts of
delay and packet loss, where more
constructive packet loss would help. Google docs is nearly unusable
most of the day... and so it goes.

I do hope the solutions for bloat continue to roll out globally and we
cross this chasm in the next few years.

>   I don’t think jitter needs to be part of public policy.

Congress has mandated a report from the FCC every year about the state
of the internet, which is what
this attempt at a constructive policy change of focus was about.

>
> Tom



-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Dave Taht]

On Sat, Dec 2, 2023 at 2:30 AM Stephen Satchell  wrote:
>
> On 12/1/23 5:27 PM, Mike Hammett wrote:
> > It would be better to keep the government out of it altogether, but that 
> > has little chance of happening.
> >
>
> I agree.  But I do have a question: is there a Best Practices RFC for
> setting buffer sizes in the existing corpus?  The Internet community has
> been pretty good at setting reasonable standards and recommendations, so
> a pointer to a BCP or RFC would go much farther to solve the bufferbloat
> problem, as I believe administrators would prefer the "suggestion"
> instead of ham-handed regulation.

I too!

The IETF recommends universal AQM: see RFC7567.

However, as a consensus document, it was impossible to get the group
to agree to fair (flow) queuing also being deployed universally,
(although it is discussed extensively) where I feel that that
technology - breaking up bursts, making it possible for all flows to
multiplex better to a destination, and isolating problematic flows -
is more important than AQM. A lot of FQ is implicit - packet pacing
from the hosts does it e2e, switches naturally multiplex from multiple
ports, but anywhere it is not, explicitly having it helps in
downshifting from one rate to another. A pure AQM tends to react late
to bursts otherwise. "Flow Queuing", is a real advance over
conventional fair queueing, IMHO.

PIE has a delay target of 16ms, Codel 5ms - but these are targets that
take a while to hit, absorbing bursts, and then draining the queue to
a steady, smaller state gradually. I recommend VJ's talk on this
highly:

https://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos/#van-jacobson-of-google-introduces-the-codel-solution-and-the-packet-fountain-analogy-at-the-ietf84-conference-july-august-2012

(he has also been heavily involved in BBR and so many other things)

If all you have is a FIFO, I personally would recommend no more than
30ms in a byte fifo, if available. A packet fifo, limited thusly,
might have issues with swallowing enough acks, but either way, with
the advent of "packet pacing" from the hosts, some pretty good
experimental evidence that 100ms is too much... (I can supply more
links), and the rise of unclassified interactive traffic like webrtc
with tighter delay constraints, I still lean strongly towards 30ms as
the outside figure in most cases.

Aside from incast traffic, you only need even this much buffering when
a link is oversubscribed. Try not to do that, but test. We got back a
lot of good data from the level3 outage showing that a lot of core
seemed to have 250ms of buffering, or more. I can dig up that
research.

For more RFCs from the now closed IETF AQM working group, see:

https://datatracker.ietf.org/group/aqm/documents/

> But that's just me.  I do know there has been academic research on the
> subject, but don't recall seeing the results published as a practical
> operational RFC.

I too would like a practical operational RFC.

It is becoming increasingly practical, but the big vendors are lagging
behind on support for advanced FQ and AQM techniques. There has been
some decent work in P4. In my case on problematic links I just slap
CAKE in front of it on a whitebox. Example of the generally pleasing
results here: https://blog.cerowrt.org/post/juniper/

(this blog also references the debate about the BDP in relation to the
number of flows controversy)

That blog also shows the degradation in tcp performance once buffers
crack 250ms - a sea of retransmits with ever decreasing goodput.

Cisco has AFD (approximate fair drop). I have zero reports from the
field from those configuring it.

I hear good things about Juniper's RED implementation but have not
torn it apart, and few configure it that I know of.

I would love it if more people slapped LibreQos on an oversubscribed
link (it's good to well past 25Gbit and pushes cake to about
10Gbits/core destination), and observed what happened to user
satisfaction, packet drops, RFC3168 ecn, and flow collisions in the 8
way set associative hash , and so on. We've produced some good tools
for it, notably tcp sampling of the rtt, as well as nearly live "mice
and elephants" plots from the 2002 paper on it.

>
> (And this is very much on-topic for NANOG, as it is about encouraging
> our peers to implement effective operation in their networks, and in
> their connections with others.)

I too encourage everyone.

-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Dave Taht]

On Fri, Dec 1, 2023 at 1:55 PM Mel Beckman  wrote:
>
> bufferbloat is rarely fatal

This task will put me in my grave, sooner rather than later!
>
>
> LOL! I know one person taht may disagree with that :)
>
>  -mel
>
> On Dec 1, 2023, at 9:41 AM, Tom Mitchell  wrote:
>
> 
> Not sure we need the FCC telling us how to build products or run networks.  
> Seat belts are life-or-death, but bufferbloat is rarely fatal ;-)  Let it be 
> a point of differentiation.
>
> -- Tom
>
>
> On Thu, Nov 30, 2023 at 4:56 PM Dave Taht  wrote:
>>
>> Over here:
>>
>> https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
>>
>> Us bufferbloat folk have been putting together a response to the FCC's
>> NOI (notice of inquiry) asking for feedback as to increasing the
>> broadband speeds beyond 100/20 Mbit.
>>
>> "Calls for further bandwidth increases are analogous to calling for
>> cars to have top speeds of 100, 200, or 500 miles per hour. Without
>> calling also for better airbags, bumpers, brakes, or steering wheels,
>> (or roads designed to minimize travel delay), these initiatives will
>> fail (and are failing) to meet the needs of present and future users
>> of the internet."
>>
>> Comments (and cites) welcomed also! The text is still somewhat in flux...
>>
>>
>> --
>> :( My old R campus is up for sale: https://tinyurl.com/yurtlab
>> Dave Täht CSO, LibreQos



-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Dave Taht]

On Fri, Dec 1, 2023 at 12:46 PM Shane Ronan  wrote:
>
> If you want money from the government to subsidize your network, you'll 
> follow their rules...

It is the misguided focus on too many too simple things from the
regulator and Congress, without even understanding what a packet is,
and enormous subsidies for things that don't matter, and great
mis-understanding of the things that do, that bothers me the most.

Anyway, for 14 years, I have been trying to get bufferbloat fixed,
universally, and great progress is being made. I felt that with one
political push like this, it might begin to turn the tide. We are
accepting signatures on the FCC filing until 2PM EST today.

https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit

> On Fri, Dec 1, 2023 at 12:39 PM Tom Mitchell  wrote:
>>
>> Not sure we need the FCC telling us how to build products or run networks.  
>> Seat belts are life-or-death, but bufferbloat is rarely fatal ;-)  Let it be 
>> a point of differentiation.
>>
>> -- Tom
>>
>>
>> On Thu, Nov 30, 2023 at 4:56 PM Dave Taht  wrote:
>>>
>>> Over here:
>>>
>>> https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
>>>
>>> Us bufferbloat folk have been putting together a response to the FCC's
>>> NOI (notice of inquiry) asking for feedback as to increasing the
>>> broadband speeds beyond 100/20 Mbit.
>>>
>>> "Calls for further bandwidth increases are analogous to calling for
>>> cars to have top speeds of 100, 200, or 500 miles per hour. Without
>>> calling also for better airbags, bumpers, brakes, or steering wheels,
>>> (or roads designed to minimize travel delay), these initiatives will
>>> fail (and are failing) to meet the needs of present and future users
>>> of the internet."
>>>
>>> Comments (and cites) welcomed also! The text is still somewhat in flux...
>>>
>>>
>>> --
>>> :( My old R campus is up for sale: https://tinyurl.com/yurtlab
>>> Dave Täht CSO, LibreQos



-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Dave Taht]

Over here:

https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit

Us bufferbloat folk have been putting together a response to the FCC's
NOI (notice of inquiry) asking for feedback as to increasing the
broadband speeds beyond 100/20 Mbit.

"Calls for further bandwidth increases are analogous to calling for
cars to have top speeds of 100, 200, or 500 miles per hour. Without
calling also for better airbags, bumpers, brakes, or steering wheels,
(or roads designed to minimize travel delay), these initiatives will
fail (and are failing) to meet the needs of present and future users
of the internet."

Comments (and cites) welcomed also! The text is still somewhat in flux...


-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

The rise and fall of the 90's telecom bubble

[Dave Taht]

Aside from me pinning the start of the bubble closer to 1992 when
commercial activity was allowed, and M for ISPs at insane valuations
per subscriber by 1995 (I had co-founded an ISP in 93, but try as I
might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and
crashed by 97 (?)), this was a whacking good read, seems accurate, and
moves to comparing it across that to the present day AI bubble.

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and

In the end we sold (my ISP, founded 93) icanect for 3 cents on the
dollar in 99, and I lost my shirt (not for the first time) on it, only
to move into embedded Linux (Montavista) after the enormous pop
redhat's IPO had had in 99. The company I was part of slightly prior
(Mediaplex) went public December 12, 1999 and cracked 100/share, only
to crash by march, 2000 to half the IPO price (around $7 as I recall),
wiping out everyone that had not vested yet. I lost my shirt again on
that and Montavista too and decided I would avoid VCs henceforth.

I am always interested in anecdotal reports of personal events in this
increasingly murky past, and in trying to fact check the above link.

So much fiber got laid by 2000 that it is often claimed that it was at
least a decade before it was used up, (the article says only 2.7% was
in use by 2002) and I have always wondered how much dark, broken,
inaccessible fiber remains that nobody knows where it even is anymore
due to many lost databases. I hear horror stories...

The article also focuses solely on the us sector, and I am wondering
what it looked like worldwide.

I believed in the 90s we were seeing major productivity gains. The
present expansion of the internet in my mind should not be much
associated with "productivity gains", as, imho, reducing the general
population to two thumbs and a 4 inch screen strikes me as an enormous
step backwards.

(I have a bad habit of cross posting my mails to where older denizens
of the internet reside, sorry! If you end up posting to one of my
lists I will add a sender allows filter for you)
-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

Re: AS8003 mysteries

[Dave Taht]

Well, I found it odd that they announce no IPv6 space.

On Thu, Nov 9, 2023 at 11:49 AM Tom Beecher  wrote:
>
> Didn't think there was much confusion about it at this point.
>
> The DOD has the assignments for the space, they can announce it whenever they 
> want, even if it's from a shell ASN.
>
> On Wed, Nov 8, 2023 at 4:52 PM Dave Taht  wrote:
>>
>> Anyone have an update as to where this effort, announcing qute a bit
>> of usa government space, stands?
>>
>> https://www.kentik.com/blog/the-mystery-of-as8003/
>> --
>> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

AS8003 mysteries

[Dave Taht]

Anyone have an update as to where this effort, announcing qute a bit
of usa government space, stands?

https://www.kentik.com/blog/the-mystery-of-as8003/
-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: Congestion/latency-aware routing for MPLS?

[Dave Taht]

On Wed, Oct 18, 2023 at 7:38 AM Tom Beecher  wrote:
>
> Auto-bandwidth won't help here if the bandwidth reduction is 'silent' as 
> stated in the first message. A 1G interface , as far as RSVP is concerned, is 
> a 1G interface, even if radio interference across it means it's effectively a 
> 500M link.
>
> Theoretically, you could have some sort of automation in place that 
> dynamically detected available bandwidth over the path, and then re-configure 
> the RSVP configured bandwidth for the interface to reflect that so the next 
> auto-bandwidth calculation would take that into account. However, the 
> efficacy of this would depend on the length of the RF disruption that caused 
> BW reduction. Assuming your detection time was near instant ( which is saying 
> something ) ,you'd still have to have very aggressive auto-BW timers to 
> adjust to it quickly enough, and there are other downsides to doing that.

I have always been curious as to what extent RED is deployed on junOS?
(in for example mpls networks) I had had some pretty bad results with
some mx gear out of my control, a while back, couldn't fix it, slapped
cake on it, grumpily blogged, moved on.

https://blog.cerowrt.org/post/juniper/

What kind of latency swings are observable today?

>
> On Wed, Oct 18, 2023 at 10:16 AM Jason R. Rokeach via NANOG  
> wrote:
>>
>> Hi Adam,
>> This sounds like a use case for MPLS-TE with TWAMP-Light. TWAMP-Light 
>> handles the latency concern and can encode your measured latency in IS-IS. 
>> Juniper docs: 
>> https://www.juniper.net/documentation/us/en/software/junos/is-is/topics/topic-map/enable-link-delay-advertise-in-is-is.html.
>>  The configuration in steps 5 and 7 is all thats required (from a config 
>> standpoint) to get the data into IS-IS.
>> You then, when building an RSVP LSP, would specify a constraint for the 
>> latency. Alternatively you can route by latency on its own by setting the 
>> metric to latency, but as you've alluded to, this can be pretty dangerous in 
>> environments with mixed bandwidth availability.
>>
>> The other option afforded for the second point on traffic balance is to use 
>> auto-bandwidth 
>> (https://www.juniper.net/documentation/us/en/software/junos/mpls/topics/topic-map/basic-lsp-configurtion.html#id-configuring-automatic-bandwidth-allocation-for-lsps
>>  - see also 
>> https://archive.nanog.org/sites/default/files/tues.general.steenbergen.autobandwidth.30.pdf).
>>
>> Other vendors support this as well.
>> SR supports the use of TWAMP-Light as well if you prefer that over RSVP, but 
>> it doesn't support auto-bandwidth.
>>
>> ___
>> Jason R. Rokeach
>> m: 603.969.5549
>> e: ja...@rokea.ch
>> tg: jasonrokeach
>>
>> Sent with ProtonMail secure email. Get my PGP Public Key.
>>
>> --- Original Message ---
>> On Wednesday, October 18th, 2023 at 9:13 AM, Adam Thompson - athompson at 
>> merlin.mb.ca  wrote:
>>
>> Using a mix of Juniper hardware...
>>
>> Network provides VPLS to customer, over MPLS (obviously) in a 
>> dual-redundant-ring radio topology.  Each site is connected to one or more 
>> neighbors, generally with two radios, in two different bands, to *each* 
>> neighbor.  So an ordinary node might have 4 radios, 2 pointing in each 
>> direction.
>>
>> Every single radio link has different bandwidth, different latency, and 
>> different interference characteristics.
>>
>> These radio links do run at 100% capacity at least some of the time.
>>
>> It's possible to set each link's relative cost in OSPF or IS-IS, of course, 
>> but I haven't found a way to make the router react to latency changes on one 
>> link or the other.  (Right now, I think costs are set equal so traffic will 
>> use both links.)  This means interference in one band invisibly diminishes 
>> the Ethernet bandwidth available and silently increases the latency on that 
>> link, sometimes dramatically.  This seems to do interestingly unpleasant 
>> things to the client's flows.
>>
>> It's generally true that one band will be much more severely affected than 
>> the other, in any interference event.  Before anyone asks, I'm told the 
>> network is a mixture of licensed and unlicensed bands, that's not changing 
>> anytime soon.
>>
>> In a perfect world, I'd like the routers to dynamically adjust traffic 
>> balance, but even just temporarily halting use of the impaired link would be 
>> helpful (or so I believe right now, at least).
>>
>> Is this a pipe dream?  I'm not seeing anything in JunOS that could 
>> accomplish this...  I'm not even sure if a mesh protocol could handle dual 
>> active links like this?
>>
>> Ideas, comments, etc. all appreciated.
>>
>> Also, I'm not the direct operator of use network. I'm involved, but mostly 
>> just trying to help them find better solutions.  Nor am I an MPLS expert, as 
>> is obvious here.
>>
>> Thanks,
>> -Adam
>>
>> Adam Thompson
>>
>> Consultant, Infrastructure Services
>>
>> MERLIN
>>
>> 100 - 135 Innovation Drive
>>
>> 

Re: Congestion/latency-aware routing for MPLS?

[Dave Taht]

We have been hoping to find use cases for the babel protocol's rtt
metric, which builds on ideas from ntp, and is primarily used today in
overlay networks:
https://datatracker.ietf.org/doc/draft-ietf-babel-rtt-extension/


On Wed, Oct 18, 2023 at 7:17 AM Jason R. Rokeach via NANOG
 wrote:
>
> Hi Adam,
> This sounds like a use case for MPLS-TE with TWAMP-Light. TWAMP-Light handles 
> the latency concern and can encode your measured latency in IS-IS. Juniper 
> docs: 
> https://www.juniper.net/documentation/us/en/software/junos/is-is/topics/topic-map/enable-link-delay-advertise-in-is-is.html.
>  The configuration in steps 5 and 7 is all thats required (from a config 
> standpoint) to get the data into IS-IS.
> You then, when building an RSVP LSP, would specify a constraint for the 
> latency. Alternatively you can route by latency on its own by setting the 
> metric to latency, but as you've alluded to, this can be pretty dangerous in 
> environments with mixed bandwidth availability.
>
> The other option afforded for the second point on traffic balance is to use 
> auto-bandwidth 
> (https://www.juniper.net/documentation/us/en/software/junos/mpls/topics/topic-map/basic-lsp-configurtion.html#id-configuring-automatic-bandwidth-allocation-for-lsps
>  - see also 
> https://archive.nanog.org/sites/default/files/tues.general.steenbergen.autobandwidth.30.pdf).
>
> Other vendors support this as well.
> SR supports the use of TWAMP-Light as well if you prefer that over RSVP, but 
> it doesn't support auto-bandwidth.
>
> ___
> Jason R. Rokeach
> m: 603.969.5549
> e: ja...@rokea.ch
> tg: jasonrokeach
>
> Sent with ProtonMail secure email. Get my PGP Public Key.
>
> --- Original Message ---
> On Wednesday, October 18th, 2023 at 9:13 AM, Adam Thompson - athompson at 
> merlin.mb.ca  wrote:
>
> Using a mix of Juniper hardware...
>
> Network provides VPLS to customer, over MPLS (obviously) in a 
> dual-redundant-ring radio topology.  Each site is connected to one or more 
> neighbors, generally with two radios, in two different bands, to *each* 
> neighbor.  So an ordinary node might have 4 radios, 2 pointing in each 
> direction.
>
> Every single radio link has different bandwidth, different latency, and 
> different interference characteristics.
>
> These radio links do run at 100% capacity at least some of the time.
>
> It's possible to set each link's relative cost in OSPF or IS-IS, of course, 
> but I haven't found a way to make the router react to latency changes on one 
> link or the other.  (Right now, I think costs are set equal so traffic will 
> use both links.)  This means interference in one band invisibly diminishes 
> the Ethernet bandwidth available and silently increases the latency on that 
> link, sometimes dramatically.  This seems to do interestingly unpleasant 
> things to the client's flows.
>
> It's generally true that one band will be much more severely affected than 
> the other, in any interference event.  Before anyone asks, I'm told the 
> network is a mixture of licensed and unlicensed bands, that's not changing 
> anytime soon.
>
> In a perfect world, I'd like the routers to dynamically adjust traffic 
> balance, but even just temporarily halting use of the impaired link would be 
> helpful (or so I believe right now, at least).
>
> Is this a pipe dream?  I'm not seeing anything in JunOS that could accomplish 
> this...  I'm not even sure if a mesh protocol could handle dual active links 
> like this?
>
> Ideas, comments, etc. all appreciated.
>
> Also, I'm not the direct operator of use network. I'm involved, but mostly 
> just trying to help them find better solutions.  Nor am I an MPLS expert, as 
> is obvious here.
>
> Thanks,
> -Adam
>
> Adam Thompson
>
> Consultant, Infrastructure Services
>
> MERLIN
>
> 100 - 135 Innovation Drive
>
> Winnipeg, MB R3T 6A8
>
> (204) 977-6824 or 1-800-430-6404 (MB only)
>
> https://www.merlin.mb.ca
>
> Chat with me on Teams
>
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Dave Taht]

For starters I would like to apologize for cc-ing both nanog and my
new nn list. (I will add sender filters)

A bit more below.

On Sun, Oct 15, 2023 at 9:32 AM Tom Beecher  wrote:
>>
>> So for now, we'll keep paying for transit to get to the others (since it’s 
>> about as much as transporting IXP from Dallas), and hoping someone at Google 
>> finally sees Houston as more than a third rate city hanging off of Dallas. 
>> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
>> peering to Kansas City. Yeah, I think the former is more likely. 
>
>
> There is often a chicken/egg scenario here with the economics. As an eyeball 
> network, your costs to build out and connect to Dallas are greater than your 
> transit cost, so you do that. Totally fair.
>
> However think about it from the content side. Say I want to build into to 
> Houston. I have to put routers in, and a bunch of cache servers, so I have 
> capital outlay , plus opex for space, power, IX/backhaul/transit costs. 
> That's not cheap, so there's a lot of calculations that go into it. Is there 
> enough total eyeball traffic there to make it worth it? Is saving 8-10ms 
> enough of a performance boost to justify the spend? What are the long term 
> trends in that market? These answers are of course different for a company 
> running their own CDN vs the commercial CDNs.
>
> I don't work for Google and obviously don't speak for them, but I would 
> suspect that they're happy to eat a 8-10ms performance hit to serve from 
> Dallas , versus the amount of capital outlay to build out there right now.

The three forms of traffic I care most about are voip, gaming, and
videoconferencing, which are rewarding to have at lower latencies.
When I was a kid, we had switched phone networks, and while the sound
quality was poorer than today, the voice latency cross-town was just
like "being there". Nowadays we see 500+ms latencies for this kind of
traffic.

As to how to make calls across town work that well again, cost-wise, I
do not know, but the volume of traffic that would be better served by
these interconnects quite low, respective to the overall gains in
lower latency experiences for them.



>
> On Sat, Oct 14, 2023 at 11:47 PM Tim Burke  wrote:
>>
>> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
>> good bit less than $900 on the wholesale market.
>>
>> Sadly, IXP’s are seemingly turning into a pay to play game, with rates 
>> almost costing as much as transit in many cases after you factor in loop 
>> costs.
>>
>> For example, in the Houston market (one of the largest and fastest growing 
>> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
>> several thousand for a 100g wave, plus several thousand for a 100g port on 
>> one of those major IXes. Or, a better option, we can get a 100g flat 
>> internet transit for just a little bit more.
>>
>> Fortunately, for us as an eyeball network, there are a good number of major 
>> content networks that are allowing for private peering in markets like 
>> Houston for just the cost of a cross connect and a QSFP if you’re in the 
>> right DC, with Google and some others being the outliers.
>>
>> So for now, we'll keep paying for transit to get to the others (since it’s 
>> about as much as transporting IXP from Dallas), and hoping someone at Google 
>> finally sees Houston as more than a third rate city hanging off of Dallas. 
>> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
>> peering to Kansas City. Yeah, I think the former is more likely. 
>>
>> See y’all in San Diego this week,
>> Tim
>>
>> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
>> >
>> > This set of trendlines was very interesting. Unfortunately the data
>> > stops in 2015. Does anyone have more recent data?
>> >
>> > https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>> >
>> > I believe a gbit circuit that an ISP can resell still runs at about
>> > $900 - $1.4k (?) in the usa? How about elsewhere?
>> >
>> > ...
>> >
>> > I am under the impression that many IXPs remain very successful,
>> > states without them suffer, and I also find the concept of doing micro
>> > IXPs at the city level, appealing, and now achievable with cheap gear.
>> > Finer grained cross connects between telco and ISP and IXP would lower
>> > latencies across town quite hugely...
>> >
>> > PS I hear ARIN is planning on dropping the price for, and bundling 3
>> > BGP AS numbers at a time, as of the end of this year, also.
>> >
>> >
>> >
>> > --
>> > Oct 30: 
>> > https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>> > Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: transit and peering costs projections

[Dave Taht]

On Sat, Oct 14, 2023 at 9:12 PM Tim Burke  wrote:
>
> It’s better for customer experience to keep it local instead of adding 200 
> miles to the route. All of the competition hauls all of their traffic up to 
> Dallas, so we easily have a nice 8-10ms latency advantage by keeping transit 
> and peering as close to the customer as possible.
>
> Plus, you can’t forget to mention another ~$10k MRC per pair in DF costs to 
> get up to Dallas, not including colo, that we can spend on more transit or 
> better gear!

Texas's BEAD funding and broadband offices are looking for proposals
and seem to have dollars to spend. I have spent much of the past few
years attempting to convince these entities that what was often more
needed was better, more local IXPs. Have you reached out to them?


> On Oct 14, 2023, at 23:03, Ryan Hamel  wrote:
>
> 
> Why not place the routers in Dallas, aggregate the transit, IXP, and PNI's 
> there, and backhaul it over redundant dark fiber with DWDM waves or 400G 
> OpenZR?
>
> Ryan
>
> 
> From: NANOG  on behalf of Tim Burke 
> 
> Sent: Saturday, October 14, 2023 8:45 PM
> To: Dave Taht 
> Cc: Network Neutrality is back! Let´s make the technical aspects heard this 
> time! ; libreqos 
> ; NANOG 
> Subject: Re: transit and peering costs projections
>
> Caution: This is an external email and may be malicious. Please take care 
> when clicking links or opening attachments.
>
>
> I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
> good bit less than $900 on the wholesale market.
>
> Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
> costing as much as transit in many cases after you factor in loop costs.
>
> For example, in the Houston market (one of the largest and fastest growing 
> regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
> several thousand for a 100g wave, plus several thousand for a 100g port on 
> one of those major IXes. Or, a better option, we can get a 100g flat internet 
> transit for just a little bit more.
>
> Fortunately, for us as an eyeball network, there are a good number of major 
> content networks that are allowing for private peering in markets like 
> Houston for just the cost of a cross connect and a QSFP if you’re in the 
> right DC, with Google and some others being the outliers.
>
> So for now, we'll keep paying for transit to get to the others (since it’s 
> about as much as transporting IXP from Dallas), and hoping someone at Google 
> finally sees Houston as more than a third rate city hanging off of Dallas. 
> Or… someone finally brings a worthwhile IX to Houston that gets us more than 
> peering to Kansas City. Yeah, I think the former is more likely. 
>
> See y’all in San Diego this week,
> Tim
>
> On Oct 14, 2023, at 18:04, Dave Taht  wrote:
> >
> > This set of trendlines was very interesting. Unfortunately the data
> > stops in 2015. Does anyone have more recent data?
> >
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrpeering.net%2Fwhite-papers%2FInternet-Transit-Pricing-Historical-And-Projected.php=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=nQeWrGi%2BblMmtiG9u7SdF3JOi1h9Fni7xXo%2FusZRopA%3D=0
> >
> > I believe a gbit circuit that an ISP can resell still runs at about
> > $900 - $1.4k (?) in the usa? How about elsewhere?
> >
> > ...
> >
> > I am under the impression that many IXPs remain very successful,
> > states without them suffer, and I also find the concept of doing micro
> > IXPs at the city level, appealing, and now achievable with cheap gear.
> > Finer grained cross connects between telco and ISP and IXP would lower
> > latencies across town quite hugely...
> >
> > PS I hear ARIN is planning on dropping the price for, and bundling 3
> > BGP AS numbers at a time, as of the end of this year, also.
> >
> >
> >
> > --
> > Oct 30: 
> > https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnetdevconf.info%2F0x17%2Fnews%2Fthe-maestro-and-the-music-bof.html=05%7C01%7Cryan%40rkhtech.org%7Cc8ebae9f0ecd4b368dcb08dbcd319880%7C81c24bb4f9ec4739ba4d25c42594d996%7C0%7C0%7C638329385118876648%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=ROLgtoeiBgfAG40UZqS8Zd8vMK%2B0HQB7RV%2FhQRvIcFM%3D=0
> > Dave Täht CSO, LibreQos



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

transit and peering costs projections

[Dave Taht]

This set of trendlines was very interesting. Unfortunately the data
stops in 2015. Does anyone have more recent data?

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php

I believe a gbit circuit that an ISP can resell still runs at about
$900 - $1.4k (?) in the usa? How about elsewhere?

...

I am under the impression that many IXPs remain very successful,
states without them suffer, and I also find the concept of doing micro
IXPs at the city level, appealing, and now achievable with cheap gear.
Finer grained cross connects between telco and ISP and IXP would lower
latencies across town quite hugely...

PS I hear ARIN is planning on dropping the price for, and bundling 3
BGP AS numbers at a time, as of the end of this year, also.



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

new net neutrality/title ii mailing list

[Dave Taht]

Since network neutrality and title ii regulation is back in the news,
and the issues
so fraught with technical and political mis-conceptions, I have
started a new mailing list to discuss it, and try (for once) to feed
back valid techical feedback into the FCC´s normal processes. I kind
of expect some EU-derived regulatory ideas to emerge, in particular.

Sign up here: https://lists.bufferbloat.net/listinfo/nnagain

The FCC chair has put out the following statement, and is circulating
a proposed NPRM to be released Oct 19th.

https://docs.fcc.gov/public/attachments/DOC-397257A1.pdf

...

There is another FCC effort on a cybersecurity label due october 6th,
which amazingly, a FCC commissioner had reached out to hackernews on,
and got back loads of wonderful feedback.

https://news.ycombinator.com/item?id=37392676

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: what is acceptible jitter for voip and videoconferencing?

[Dave Taht]

On Thu, Sep 21, 2023 at 3:34 PM William Herrin  wrote:

> On Thu, Sep 21, 2023 at 6:28 AM Tom Beecher  wrote:
> > My understanding has always been that 30ms was set based on human
> perceptibility. 30ms was the average point at which the average person
> could start to detect artifacts in the audio.
>
> Hi Tom,
>
> Jitter doesn't necessarily cause artifacts in the audio. Modern
> applications implement what's called a "jitter buffer." As the name
> implies, the buffer collects and delays audio for a brief time before
> playing it for the user. This allows time for the packets which have
> been delayed a little longer (jitter) to catch up with the earlier
> ones before they have to be played for the user. Smart implementations
> can adjust the size of the jitter buffer to match the observed
> variation in delay so that sound quality remains the same regardless
> of jitter.
>
> Indeed, on Zoom I barely noticed audio artifacts for a friend who was
> experiencing 800ms jitter. Yes, really, 800ms. We had to quit our
> gaming session because it caused his character actions to be utterly
> spastic, but his audio came through okay.
>
> The problem, of course, is that instead of the audio delay being the
> average packet delay, it becomes the maximum packet delay.


Yes. I talked to this point in my apnic session here:
https://blog.apnic.net/2020/01/22/bufferbloat-may-be-solved-but-its-not-over-yet/

I called it "riding the TCP sawtooth"- the compensating voip delay becomes
equal to the maximum size of the buffer, and thus controls the jitter that
way. Sometimes, to unreasonable extents, like 800ms in your example.


> You start
> to have problems with people talking over each other because when they
> start they can't yet hear the other person talking. "Sorry, go ahead.
> No, you go ahead."
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>


-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: what is acceptible jitter for voip and videoconferencing?

[Dave Taht]

Thank you all for your answers here, on the poll itself, and for
papers like this one. The consensus seems to be settling around 30ms for
VOIP with a few interesting outliers and viewpoints.

https://scholarworks.gsu.edu/cgi/viewcontent.cgi?article=1043=cs_theses

Something that came up in reading that... that I half remember from my
early days of working with VOIP (on asterisk) was that silence suppression
(and comfort noise) that did not send any packets was in general worse than
sending silence (or comfort noise) - for two reasons - one was nat closures,
but the other was that steady stream also helped control congestion and had
less jitter swings.

So in the deployments I was doing then, I universally disabled this feature
on the phones I was using then.

In my mind (particularly in a network that is packet (not byte) buffer
limited), this showed that point, (to an extreme!)

https://www.duo.uio.no/bitstream/handle/10852/45274/1/thesis.pdf

But my question is now, are we doing silence suppression (not sending
packets) on voip nowadays?

On Thu, Sep 21, 2023 at 2:55 PM Eric Kuhnke  wrote:

> Artifacts in audio are a product of packet loss or jitter resulting in
> codec issues issues leading to human subject perceptible audio anomalies,
> not so much latency by itself. Two way voice is remarkably NOT terrible on
> a 495ms RTT satellite based two-way geostationary connection as long as
> there is little or no packet loss.
>
> On Thu, Sep 21, 2023 at 12:47 PM Tom Beecher  wrote:
>
>> My understanding has always been that 30ms was set based on human
>> perceptibility. 30ms was the average point at which the average person
>> could start to detect artifacts in the audio.
>>
>> On Tue, Sep 19, 2023 at 8:13 PM Dave Taht  wrote:
>>
>>> Dear nanog-ers:
>>>
>>> I go back many, many years as to baseline numbers for managing voip
>>> networks, including things like CISCO LLQ, diffserv, fqm prioritizing
>>> vlans, and running
>>> voip networks entirely separately... I worked on codecs, such as oslec,
>>> and early sip stacks, but that was over 20 years ago.
>>>
>>> The thing is, I have been unable to find much research (as yet) as to
>>> why my number exists. Over here I am taking a poll as to what number is
>>> most correct (10ms, 30ms, 100ms, 200ms),
>>>
>>> https://www.linkedin.com/feed/update/urn:li:ugcPost:7110029608753713152/
>>>
>>> but I am even more interested in finding cites to support various
>>> viewpoints, including mine, and learning how slas are met to deliver it.
>>>
>>> --
>>> Oct 30:
>>> https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
>>> Dave Täht CSO, LibreQos
>>>
>>

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

what is acceptible jitter for voip and videoconferencing?

[Dave Taht]

Dear nanog-ers:

I go back many, many years as to baseline numbers for managing voip
networks, including things like CISCO LLQ, diffserv, fqm prioritizing
vlans, and running
voip networks entirely separately... I worked on codecs, such as oslec, and
early sip stacks, but that was over 20 years ago.

The thing is, I have been unable to find much research (as yet) as to why
my number exists. Over here I am taking a poll as to what number is most
correct (10ms, 30ms, 100ms, 200ms),

https://www.linkedin.com/feed/update/urn:li:ugcPost:7110029608753713152/

but I am even more interested in finding cites to support various
viewpoints, including mine, and learning how slas are met to deliver it.

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

Re: So what do you think about the scuttlebutt of Musk interfering in Ukraine?

[Dave Taht]

This is one of those threads where I do think folk would benefit from
hearing from the horses' mouths. In a recent bio of musk published this
past week, the author claimed that starlink withdrew service over crimea
based on the knowledge it was going to be used for a surprise attack.
Starlink - and that author - now state that (
https://twitter.com/elonmusk/status/1700345943105638636 )

The onus is meaningfully different if I refused to act upon a request from
Ukraine vs. made a deliberate change to Starlink to thwart Ukraine. At no
point did I or anyone at SpaceX promise coverage over Crimea. Moreover, our
terms of service clearly prohibit Starlink for offensive military action,
as we are a civilian system, so they were again asking for something that
was expressly prohibited. SpaceX is building Starshield for the US
government, which is similar to, but much smaller than Starlink, as it will
not have to handle millions of users. That system will be owned and
controlled by the US government.
Quote
Walter Isaacson
@WalterIsaacson
·
Sep 8
To clarify on the Starlink issue: the Ukrainians THOUGHT coverage was
enabled all the way to Crimea, but it was not. They asked Musk to enable it
for their drone sub attack on the Russian fleet. Musk did not enable it,
because he thought, probably correctly, that would cause a… Show more


Furthermore, Musk stated yesterday that had the request come from the us
government, he would have complied.

I will refrain from editorializing.


On Thu, Sep 14, 2023 at 5:56 AM Aaron de Bruyn via NANOG 
wrote:

> Starlink isn't a monopoly. Ukraine could have guided their munitions with
> Iridium or another satellite Internet system.
>
>
> Don't forget GLONASS. 
>
> On Thu Sep 14, 2023, 03:10 AM GMT, William Herrin  wrote:
>
> On Wed, Sep 13, 2023 at 5:47 PM Michael Thomas  wrote:
>
> Doesn't this bump up against common carrier protections?
>
>
> Hi Michael,
>
> Internet providers aren't common carriers. If they were, it'd be
> unlawful to stop your customers from sending email spam that was
> merely offensive rather than illegal. It's also why Internet providers
> aren't required to follow network neutrality. Internet providers gain
> their immunity through section 230 and the DMCA instead.
>
> Common carrier status typically applies to shipping companies and
> basic telephone service. Part of the mess with unwanted phone calls is
> that the caller has to break the law (e.g. by calling a number on the
> do-not-call list) before the phone company is allowed to act against
> them.
>
> I sure don't
> want my utilities weaponizing their monopoly status to the whims of any
> random narcissist billionaire.
>
>
> Starlink isn't a monopoly. Ukraine could have guided their munitions
> with Iridium or another satellite Internet system.
>
> That said, volunteering services to the military of a nation at war
> and then pulling the rug out from under them is so classless, one
> wonders if Musk isn't trying to build a communist utopia.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>
>

-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html
Dave Täht CSO, LibreQos

starlink deluge test and 33 (-2) engine static fire

[Dave Taht]

I know it is a bit off topic for nanog, but this test was very exciting today:

https://twitter.com/SpaceX/status/1695158759717474379

Happy friday! While they have filed for a launch license for august
31st, it is impossible for me to believe that date! It was also
difficult to believe the deluge ("bidet") system would actually work.

I look forward to a launch vehicle capable of putting up the next
generation of starlink sats which are estimated to have 4x the
capacity of the old, and further improvements on their wifi, backbone
and satellite switching technologies.

-- 
Podcast: https://www.youtube.com/watch?v=bxmoBr4cBKg
Dave Täht CSO, LibreQos

Re: Internet Exchange Visualization

[Dave Taht]

I hear the cybergeography project is making a comeback.

https://personalpages.manchester.ac.uk/staff/m.dodge/cybergeography/atlas/atlas.html

On Mon, Aug 21, 2023 at 5:17 PM Matthew Petach  wrote:
>
>
>
> On Sun, Aug 20, 2023 at 11:06 PM Thomas Beer  wrote:
>>
>> Hi Matt,
>>
>>>
>>> You might mean "exchange inter-connections" as "how are the different 
>>> internet exchanges connected to each other?"
>>> in which case the answer is generally "through the Internet".  ^_^;
>>
>>
>> I meant ix internet exchange path visualization and an online tool to take a 
>> look at it in (near) real time!
>>
>> Cheers
>>
>
>
> Ah, thank you for the enlightening clarification.
>
> No such tool exists, sorry.
>
> Thanks!
>
> Matt
>



-- 
Podcast: https://www.youtube.com/watch?v=bxmoBr4cBKg
Dave Täht CSO, LibreQos

Re: Last Mile ISP Quality Measurements

[Dave Taht]

I am reluctant to respond because it might end up sounding like an ad
for libreqos.io.

Leaving aside the tcp rtt tracking, the cake shaping, the mark and
drop statistics in that product, the (mostly wireless) ISPs we work
with typically have a dashboard of long term SNMP statistics of key
parameters like signal strength (RSSI), a heatmap of rtts to those
routers, and the upstreams, (smokeping cannot handle this kind of
density), a bandwidth tracker (usually on a 5 minute interval), a few
raspberry pi or equivalents at the towers doing active measurements on
demand, and a set of actionable items derived from that that the
support techs work off of.
Then there is a per customer screen that captures as much as possible
useful about the customer and every hop along the way.

There are a lot of pics of dashboards like this on the web, see
preseem, paraqum, and of course libreqos for examples. People use a
variety of backend products for it (grafina, redis, influx are
popular).

hope this helps.

On Wed, Aug 9, 2023 at 5:23 AM Steve Pointer  wrote:
>
> Have you considered hosting a Ripe Anchor?
>
> https://atlas.ripe.net/anchors/about/
>
> Minimal cost, good of the Internet project, good insights, answers the use 
> case you describe.
>
> Steve P
>
>


-- 
Podcast: https://www.youtube.com/watch?v=bxmoBr4cBKg
Dave Täht CSO, LibreQos

Re: Picking a RIR/obtaining an AS/ressurrecting a legacy space

[Dave Taht]

On Thu, Jul 6, 2023 at 3:26 PM William Herrin  wrote:
>
> On Thu, Jul 6, 2023 at 1:43 PM John Curran  wrote:
> > You are correct, insomuch if one intends to alter answers about the 
> > purported history
> > of an address block from what was said the first time, we won’t simply 
> > disregard the
> > previously supplied story.
>
> With due respect John, any given set of facts can be presented more
> than one way, without altering any of the facts. When a use case is
> dead center for ARIN's normal operation, it doesn't much matter. When
> it's more of an edge case, I recommend having a friendly ear help you
> get your ducks in a row before saying anything to ARIN. And let's face
> it, it doesn't get much more edge case than updating a dormant
> pre-ARIN (legacy) address block.
>
> Regards,
> Bill Herrin


The history here goes back 30 years! (
http://www.ccil.org/home/about-ccil/history-of-ccil ) Eric is mostly
retired, rarely answers email, and all memories of this period faint.
I have reached out to them to see if they still have use for the other
/24.

Thank you everyone for explaining how things work nowadays.

>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



-- 
Podcast: 
https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos

Re: Picking a RIR/obtaining an AS/ressurrecting a legacy space

[Dave Taht]

thank you for your research into this on our behalf and the steer to
the right things.

Yes, these two ip address ranges are erics, and if anything he´s more
allergic to paperwork than I am.

On Thu, Jul 6, 2023 at 11:03 AM William Herrin  wrote:
>
> On Thu, Jul 6, 2023 at 8:03 AM Dave Taht  wrote:
> > https://bgpview.io/prefix/198.177.242.0/24
>
> This is registered to Thyrsus Enterprises via ARIN, managed by an Eric
> Raymond of Pennsylvania. Refer to
> https://search.arin.net/rdap/?query=198.177.242.0
>
> If your friend happens to be Eric Raymond, his best bet is to simply
> leave it alone as a legacy address under his control rather than try
> to prove himself the legal successor in interest to Thyrsus
> Enterprises. As long as there is no current Thyrsus Enterprises, and
> as the guy on the whois, he'll be able to submit an LOA to an ISP and
> get them to accept the route.

Cool. That seems way simpler than the other alternatives running though my head.

> If your friend isn't Eric Raymond or Thyrsus Enterprises still exists
> and is someone else... you're done. Save yourself some grief and just
> go to an address broker. Let them help you through the process of
> getting addresses.
>
>
> > https://bgpview.io/prefix/198.177.243.0/24
>
> This is registered to Chester County Freenet care of Chester County
> Hospital. Refer to https://search.arin.net/rdap/?query=198.177.243.0
>
> Raymond again controls it, but since he's neither the Freenet nor the
> hospital you're going to run into trouble getting it routed let alone
> getting ARIN to recognize you as the legal successor in interest.

I do not know the history here, I will ask.

>
> > the whole /22 was obtained to support the (long since deceased)
> > chester county freenet, but he has no record of that. Neither does
> > anyone else.
>
> Those would form a /22 with 240 and 241. Both are registered to other
> people. Unclear why you thought otherwise. If you were thinking 244
> and 245 (which do not form a /22 with 242 and 243), I'm sorry to tell
> you that they're also registered to someone else.

OK, good to know, that you for straightening this out a bit.

>
>
> > I presently have one vote for ARIN and another for RIPE. We are us
> > based, but more of the folk using libreqos are located elsewhere.
>
> The addresses are registered at ARIN. Until ARIN recognizes your
> friend as the registrant organization, they will remain so. At which
> point there's not a lot of benefit to moving them.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/



-- 
Podcast: 
https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos

Re: Picking a RIR/obtaining an AS/ressurrecting a legacy space

[Dave Taht]

Fixing the ausnog cc... more below.

On Thu, Jul 6, 2023 at 8:46 AM Bill Woodcock  wrote:
>
> The ASN really isn’t a big deal.  There’s no scarcity of them, you can get a 
> 16-bit one by asking.

Heh. Trying to reaquire my old 16 bit AS number is merely a matter of
vanity... just navigating through the modern processes to become real
again in some fashion the larger problem. I remember how much I
resented faxing allocations way back when, modern processes seem
worse!

> The legacy IPv4 space, well, if there’s a clear chain of custody to the 
> current holder, and the current holder is responsive, they can use it or 
> transfer it.  But also, IPv4 space isn’t scarce…  it just costs money, now, 
> to buy.

The holder has a clear chain of custody for the bottom two /24s

https://bgpview.io/prefix/198.177.242.0/24
https://bgpview.io/prefix/198.177.243.0/24

But the whole /22 was obtained to support the (long since deceased)
chester county freenet, but he has no record of that. Neither does
anyone else. The most ideal outcome is we get a /22 out of this, a
less ideal but still nice would be to have the upper pool released to
someone.

Eric is willing to loan his IPs to our good cause if we can clear it
up but wants to reserve his rights to actually sell them at some point
in the future.

>
> If you’re in the US, just use ARIN.  ARIN’s processes aren’t arcane, 
> particularly compared with RIPE, and fees are predictable and relatively low.

I presently have one vote for ARIN and another for RIPE. We are us
based, but more of the folk using libreqos are located elsewhere.


>
> -Bill
>
>
>
> > On Jul 6, 2023, at 16:29, Dave Taht  wrote:
> >
> > I have an old friend still holding onto some legacy IP space that he
> > has not used in 30 years. The origin goes back to the early 90s, and
> > originally through ARIN. In the relevant databases it is a /23, but
> > actually a /22 - but the top 2 addresses are not registered or
> > announced anywhere I can find. I do not mind losing those to the pool
> > but getting the /23 up and running would help... and a /22 far more
> > useful for our purposes. Sadly I also have a lovely 16 bit BGP AS
> > number AS5768 still unused from my first company of that era but in
> > the hands of a admin that has been unresponsive about either using it
> > or giving it back for many years. Sentimentally I would like to find a
> > way to get that back... but it is ok if that doesn't happen.
> >
> > Anyway, LibreQos would really like to obtain a BGP AS number from some
> > RIR (or is there an unused BGP AS transfer market?) and have some real
> > IPv4 addresses to vector some traffic through, in our testbeds
> > initially, and perhaps later on as means to shape traffic for other
> > services. Most of our market is outside the USA actually and I would
> > be inclined to get that AS from the simplest AR to deal with, but my
> > list of preferences is merely based on where we have installations
> > rather than cost/contacts/customer service... and especially,
> > "hassle". Honestly coping with figuring out the fee and registration
> > schedules are is just beyond me. I have heard ripe was easiest to deal
> > with regarding legacy space. (?)
> >
> > Anyone out there that can help sort out this legacy space in a sane
> > manner? We are subsisting on a tiny amount of donations/month
> > presently, and the up front cost and yearly costs are quite a lot to
> > make this step.
> >
> > Finding someone(s) to help us become real in this fashion, navigating
> > the RIRs process, setting up bird or FRR for us (with a touch of
> > anycast), would help, and help (at some price) moving forward, would
> > be great. I have not got BGP running myself in over 25 years!
> >
> > --
> > Podcast: 
> > https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
> > Dave Täht CSO, LibreQos
>


-- 
Podcast: 
https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos

Picking a RIR/obtaining an AS/ressurrecting a legacy space

[Dave Taht]

I have an old friend still holding onto some legacy IP space that he
has not used in 30 years. The origin goes back to the early 90s, and
originally through ARIN. In the relevant databases it is a /23, but
actually a /22 - but the top 2 addresses are not registered or
announced anywhere I can find. I do not mind losing those to the pool
but getting the /23 up and running would help... and a /22 far more
useful for our purposes. Sadly I also have a lovely 16 bit BGP AS
number AS5768 still unused from my first company of that era but in
the hands of a admin that has been unresponsive about either using it
or giving it back for many years. Sentimentally I would like to find a
way to get that back... but it is ok if that doesn't happen.

Anyway, LibreQos would really like to obtain a BGP AS number from some
RIR (or is there an unused BGP AS transfer market?) and have some real
IPv4 addresses to vector some traffic through, in our testbeds
initially, and perhaps later on as means to shape traffic for other
services. Most of our market is outside the USA actually and I would
be inclined to get that AS from the simplest AR to deal with, but my
list of preferences is merely based on where we have installations
rather than cost/contacts/customer service... and especially,
"hassle". Honestly coping with figuring out the fee and registration
schedules are is just beyond me. I have heard ripe was easiest to deal
with regarding legacy space. (?)

Anyone out there that can help sort out this legacy space in a sane
manner? We are subsisting on a tiny amount of donations/month
presently, and the up front cost and yearly costs are quite a lot to
make this step.

Finding someone(s) to help us become real in this fashion, navigating
the RIRs process, setting up bird or FRR for us (with a touch of
anycast), would help, and help (at some price) moving forward, would
be great. I have not got BGP running myself in over 25 years!

-- 
Podcast: 
https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Dave Taht]

On Sat, Jun 17, 2023 at 5:41 PM Tom Beecher  wrote:
>>
>> You are also assuming their only product is Home Internet. Providing 
>> Internet to ships at sea, planes in the sky and other more unconventional 
>> uses will provide a lot more revenue than the home Internet will.
>
>
> I am not assuming that at all.
>
> There is absolutely a market for sat internet. It's just not a $30B revenue a 
> year business as Musk has said.
>
> On land , why do wireline providers not build out into rural areas? There is 
> not enough subscriber density to recover buildout costs in an acceptable 
> timeframe.  Starlink has the same problem ; the number of possible 
> subscribers is exceptionally low relative to the buildout cost.

No it does not. Reduced density in any area makes for a compelling
market for starlink. The buildout cost is fixed (cover the globe with
sats), once the globe is covered, taking advantage of any area under
that is straightforward. It is quite unlike wires in this case, or
even FWA, there is no power to towers, no need for power or cable
anything but a downlink site located somewhere within a few hundred
miles.

We are also seeing rural 5G FWA expand rapidly, in part because the
gear costs the same no matter how many people are on it.

> There won't ever be high demand for Starlink in urban areas because it's not 
> needed, and performance is bad when users are clustered like that.

Agreed.

> Again, I agree there is a market for sat internet. It's just never going to 
> be anywhere close to as large as what is claimed.

I think we are arguing the difference between 10m people and 30m? 10m
people is quite a substantial business, barely cracking the ranks of
the larger ISPS, and yet ~$1B/month. Hard to complain...

I would have liked it if starlink´s business service included BGP
peering, and other classic aspects of the internet that it does not
have as yet.

>
> On Sat, Jun 17, 2023 at 7:25 PM  wrote:
>>
>> You are also assuming their only product is Home Internet. Providing 
>> Internet to ships at sea, planes in the sky and other more unconventional 
>> uses will provide a lot more revenue than the home Internet will.
>>
>>
>>
>> On Jun 17, 2023, at 7:04 PM, Tom Beecher  wrote:
>>
>> 
>>>
>>> You’re assuming the launches are costing them something, which in fact may 
>>> not be true. Rumor has it, they are piggybacking on other payloads which 
>>> pay for the launches, particularly government contracts.
>>
>>
>> Assuming they are, they aren't doing enough of those launches to piggyback 
>> enough sats to reach the 40k claim.
>>
>> Zero out the launch costs, subscriber revenue still doesn't doesn't come 
>> close to touching the sat costs.
>>
>>
>> On Sat, Jun 17, 2023 at 6:27 PM  wrote:
>>>
>>> You’re assuming the launches are costing them something, which in fact may 
>>> not be true. Rumor has it, they are piggybacking on other payloads which 
>>> pay for the launches, particularly government contracts.
>>>
>>>
>>>
>>>
>>> On Jun 17, 2023, at 5:54 PM, Tom Beecher  wrote:
>>>
>>> 

 As I mentioned elsewhere, I'm not sure that the current economics are the 
 real economics. I'm pretty sure they've been purposefully throttling 
 demand because they still don't have the capacity so it would make sense 
 to overcharge in the mean time. Is there something inherent in their cpe 
 that makes them much more expensive than, say, satellite tv dishes? I can 
 see marginally more because of the LEO aspect, but isn't that mainly just 
 software? It wouldn't surprise me that the main cost is the truck roll.
>>>
>>> - Starlink currently reports around 1.5M subscribers. At $110 a month, 
>>> that's $165M in revenue,
>>>
>>> - A Falcon 9 launch is billed out at $67M. A Falcon 9 can carry up to 60 
>>> Starlink sats. That's ~667 launches to reach the stated goal of 40k sats in 
>>> the constellation. So roughly $45B in just launch costs, if you assume the 
>>> public launch price. (Because if they are launching their own stuff, they 
>>> aren't launching an external paying customer.)
>>> - The reported price per sat is $250k.
>>>
>>> Assuming they give themselves a friendly internal discount, the orbital 
>>> buildout cost are in the neighborhood of $30B for launches, and $10B for 
>>> sats.
>>>
>>> - The satellite failure rate is stated to be ~ 3% annually. On a 40K 
>>> cluster, that's 1200 a year.
>>>
>>> That's about 20 more launches a year, and $300M for replacement sats. Let's 
>>> round off and say that's $1B a year there.
>>>
>>>  So far, that's a $40B buildout with a $1B annual run rate. And that's just 
>>> the orbital costs. We haven't even calculated the manufacturing costs of 
>>> the receiver dishes, terrestrial network infra cost , opex from staff , 
>>> R, etc .
>>>
>>> Numbers kinda speak for themselves here.
>>>
 I mean, I get that Musk is sort of a cuckoo bird but say what you will he 
 does have big ambitions.
>>>
>>>
>>> Ambition is good. But 

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Dave Taht]

On Sat, Jun 17, 2023 at 5:16 PM Tom Beecher  wrote:
>>
>> Also: they plan to use Starship when it's available which has 10x more 
>> capacity. If it really is fully reusable as advertised, that is going to 
>> really drive down the launch cost.
>
>
> Starship is years away from being flight ready. The most recent test launch 
> from Texas was not a 'successful failure' as widely portrayed in the media.  
> Reputable people who have been working in this field for decades have pointed 
> out tons of massive problems that are not quick fixes.

1) I agree that they are years from flight ready, however the
improvements in the queue for the next launch are already impressive.
A lot of nay-saying concerns have been addressed since the launch.

The environmental impact was far less than believed. An analysis of
the dust spread across town was shown to just be sand, not vaporised
fondag,
as thought.

While the everyday astronaut and starbase_csi can be thought of as
fanbois, they are also producing the most quality reporting and
analysis that exists:

https://twitter.com/Erdayastronaut
https://twitter.com/CSI_Starbase

They are good folk to track.

Eric Burger is a more conventional tech journalist covering all of space:

https://twitter.com/SciGuySpace

https://arstechnica.com/author/ericberger/

There are an amazing number of individuals reporting on daily
progress, with live video feeds.

There is really massive construction going on, replacing the existing
megabay, the damaged tanks are being replaced rapidly, the launch site
has been dug out and partially repaired,  and a new launch license was
issued for the next 6 months last week.

The principal barriers to another launch are a successful test of the
new water deluge system, and qualifying a more advanced flight
termination system. The next ship and booster will possibly be tested
next month, and these have replaced the hydrolic controls with
electric and have better motor shielding in general.

Yes, an utterly amazing amount of things need to go right to launch a
spaceship, but ... my best bet for another launch of starship would be
early september.


>
> On Sat, Jun 17, 2023 at 6:56 PM Michael Thomas  wrote:
>>
>> Whether or not it makes business sense isn't really what I was talking 
>> about. I was talking about the home dish costing $1k. That sounds like it 
>> could easily be reduced significantly unless there is some underlying tech 
>> reason.
>>
>> Also: they plan to use Starship when it's available which has 10x more 
>> capacity. If it really is fully reusable as advertised, that is going to 
>> really drive down the launch cost.
>>
>> But your calculations don't take into account that they are not at anywhere 
>> close to a full constellation: they are only at 4k out of the 40k they need 
>> so they literally can't support higher numbers. Their new generation of 
>> satellite is also suppose to be doing some in-orbit routing or something 
>> like that which would I would assume will really help on the bandwidth 
>> front. How much that affects their maximum subscriber base when they are 
>> fully deployed I don't know but it's bound to be a lot more possible subs 
>> than they have now.
>>
>> I mean, this could be a spectacular flop like Iridium but a lot has changed 
>> in 20 some years not least of which is the cost of launch.
>>
>> Mike
>>
>> On 6/17/23 2:53 PM, Tom Beecher wrote:
>>>
>>> As I mentioned elsewhere, I'm not sure that the current economics are the 
>>> real economics. I'm pretty sure they've been purposefully throttling demand 
>>> because they still don't have the capacity so it would make sense to 
>>> overcharge in the mean time. Is there something inherent in their cpe that 
>>> makes them much more expensive than, say, satellite tv dishes? I can see 
>>> marginally more because of the LEO aspect, but isn't that mainly just 
>>> software? It wouldn't surprise me that the main cost is the truck roll.
>>
>> - Starlink currently reports around 1.5M subscribers. At $110 a month, 
>> that's $165M in revenue,
>>
>> - A Falcon 9 launch is billed out at $67M. A Falcon 9 can carry up to 60 
>> Starlink sats. That's ~667 launches to reach the stated goal of 40k sats in 
>> the constellation. So roughly $45B in just launch costs, if you assume the 
>> public launch price. (Because if they are launching their own stuff, they 
>> aren't launching an external paying customer.)
>> - The reported price per sat is $250k.
>>
>> Assuming they give themselves a friendly internal discount, the orbital 
>> buildout cost are in the neighborhood of $30B for launches, and $10B for 
>> sats.
>>
>> - The satellite failure rate is stated to be ~ 3% annually. On a 40K 
>> cluster, that's 1200 a year.
>>
>> That's about 20 more launches a year, and $300M for replacement sats. Let's 
>> round off and say that's $1B a year there.
>>
>>  So far, that's a $40B buildout with a $1B annual run rate. And that's just 
>> the orbital costs. We haven't even 

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Dave Taht]

I am happy to see the conversation about starlink escaping over here,
because it is increasingly a game-changing technology (I also run the
starlink mailing list, cc´d)...

On Sat, Jun 17, 2023 at 3:56 PM Tom Beecher  wrote:
>>
>> As I mentioned elsewhere, I'm not sure that the current economics are the 
>> real economics.

There is a whole other cluster on the drawing boards, called
Starshield, which you can read about here:
https://www.spacex.com/starshield/

The current "retail"economics are limited to US allies as a result of
the ukraine war showing how important information and bandwidth are to
modern warfare. There are also political implications to downlinks in
each country.

I imagine, for example, that India is holding off on licensing until
Musk gets them a tesla factory.

Multiple other countries are making a huge investment into retaining
control of the "spacewaves", so there´s that also.

>I'm pretty sure they've been purposefully throttling demand because they still 
>don't have the capacity so it would make sense to overcharge in the mean time.

Throttling demand is not how I would put it. Each cell has a limited
capacity, so starlink has been running promotions to get more
subscribers into more rural cells where the capacity exists.

I have kvetched elsewhere about how poorly starlink manages bandwidth
and bufferbloat currently, but they are largely better than modern day
5G and DSL, so...

>  Is there something inherent in their cpe that makes them much more expensive 
> than, say, satellite tv dishes?

The original cost/dish was about 2k, so they were selling those at
well below the install price, with a ROI of about 12 months, given
that figure. I imagine with mass manufacturing the cost/dish has come
down substantially, and they also charge a realistic price on the
business quality dish of $2500. It would not surprise me if the basic
dishy essentially cost less than 500 to manufacture nowadays.

The default wifi router, which many replace, cannot be more than 50
dollars on the BOM.

> I can see marginally more because of the LEO aspect, but isn't that mainly 
> just software? It wouldn't surprise me that the main cost is the truck roll.

There is no truck roll. They have gone to amazing extants in - put the
dish in a clear area, power it up, you are on.

Establishing infrastructure, like downlinks, connected near fiber in
civilization does have a large cost, takes time, and is also subject
to government regulation.

>
> - Starlink currently reports around 1.5M subscribers. At $110 a month, that's 
> $165M in revenue,

Creating A 2B dollar/year business in 4 years is quite impressive. A
reasonable projection would be 10m subs in 4 more years, e.g.
10B/year. That aint' chicken scratch. In fact, I think it funds
humanity´s expansion into the solar system quite handily.

> - A Falcon 9 launch is billed out at $67M. A Falcon 9 can carry up to 60 
> Starlink sats. That's ~667 launches to reach the stated goal of 40k sats in 
> the constellation. So roughly $45B in just launch costs, if you assume the 
> public launch price. (Because if they are launching their own stuff, they 
> aren't launching an external paying customer.)

> - The reported price per sat is $250k.

There are multiple sat types, the mini v2 (which can only be flown on
the falcon 9, is rumored to cost about that much)

Starship had had a much larger, much more highly capable sat designed
for it, but it is running a few years behind schedule. The hope for
that was that launch costs would decline even further.

Also OPEX - running this network - is probably a substantial cost. I
have lost track of the number of downlink stations established (over
200 now) but I would guess those are about 1m per.

There is a really amazing site that looks at this stuff called starlink.sx.

>
> Assuming they give themselves a friendly internal discount, the orbital 
> buildout cost are in the neighborhood of $30B for launches, and $10B for sats.

The present day capacity, even if they were to do no more launches, is
still underused. Roughly half the USA has no starlink service yet,
multiple countries have been slow to license, and nearly all of Africa
remains uncovered. Maritime and air are big sources of new business. I
try to stress it is where  people are but infrastructure isn´t is
where starlink really shines,

and that very little bandwidth is required for things like email and chat.

>
> - The satellite failure rate is stated to be ~ 3% annually. On a 40K cluster, 
> that's 1200 a year.

Where did you see that? So far as I can tell, the failure rate,
exclusive of one launch lost to solar expansion, is trending towards
zero. Also, maneuvering thrust (documented somewhere) has been quite
under expectations, in terms of operating fuel they could use the
existing sats for far, far longer than the intended 5 year operational
lifetime, in this regard.

>
> That's about 20 more launches a year, and $300M for replacement sats. Let's 
> round off 

Re: Do ISP's collect and analyze traffic of users?

[Dave Taht]

On Sat, Jun 10, 2023 at 9:46 AM John van Oppen  wrote:
>
> As a decent sized north American ISP I think I need totally agree with this 
> post.There simply is not any economically justifiable reason to collect 
> customer data, doing so is expensive, and unless you are trying to traffic 
> shape like a cell carrier

They shape? News to me...

>has zero economic benefit. In our case we do 1:4000 netflow samples and 
>that is literally it, we use that data for peering analytics and failure 
>modeling.
>
> This is true for both large ISPs I've been involved with and in both cases I 
> would have overseen the policy.
>
> What I see in this thread is a bunch of folks guessing that clearly have not 
> been involved in large eyeball ISP operations.

The smaller (mostly rural) WISPs I work with do not have time or
desire to monetize traffic either! Pretty much all of them have their
hands full just solving tech support problems.

They do collect extensive metrics on bandwidth, packet loss, latency,
snmp stats of all sorts, airtime, interference, cpu stats, routing
info,
(common tools are things like UISP, splynx, opennms), and keep
amazingly good (lidar, even) maps of the local terrain. If the bigger
ISPs are only doing netflow once in a while, no wonder the little
wisps survive.

The ones shaping via libreqos.io now are totally in love[1] with our
in-band RTT metrics as that is giving them insight into their backhaul
behaviors in rain and snow and sleet, instead of out of band snmp, as
well as gaining insight into when it is the customer wifi that is the
real problem. It is the combination of all these metrics that helps
narrow down problems.

But the only monetization that happens is the monthly bill. Most of
these cats are actually rather ornery and *very* insistent about
protecting their customers privacy, from all comers, and resistant to
cloud based applications in general.

There are some bad apples in the wisp world that do want to rate limit
(via dpi) netflix above all else in case of running low on backhaul,
but they are not in my customer base.

[1] we (and they) *are* passionately interesting in identifying the
characteristics of multiple traffic types and mitigating attacks, and
a couple are publishing some anonymized movies of what traffic looks
like: https://www.youtube.com/@trendaltoews7143/videos


>
>
> -Original Message-
> From: NANOG  On Behalf Of Saku Ytti
> Sent: Tuesday, May 16, 2023 7:56 AM
> To: Tom Beecher 
> Cc: nanog@nanog.org
> Subject: Re: Do ISP's collect and analyze traffic of users?
>
> I can't tell what large is. But I've worked for enterprise ISP and consumer 
> ISPs, and none of the shops I worked for had capability to monetise 
> information they had. And the information they had was increasingly low 
> resolution. Infraprovider are notoriously bad even monetising their infra.
>
> I'm sure do monetise. But generally service providers are not interesting or 
> have active shareholders, so very little pressure to make more money, hence 
> firesales happen all the time due infrastructure increasingly seen as a 
> liability, not an asset. They are generally boring companies and internally 
> no one has incentive to monetise data, as it wouldn't improve their personal 
> compensation. And regulations like GDPR create problems people rather not 
> solve, unless pressured.
>
> Technically most people started 20 years ago with some netflow sampling 
> ratio, and they still use the same sampling ratio, despite many orders of 
> magnitude more packets. Meaning previously the share of flows captured was 
> magnitude higher than today, and today only very few flows are seen in very 
> typical applications, and netflow is largely for volumetric ddos and high 
> level ingressAS=>egressAS metrics.
>
> Hardware offered increasingly does IPFIX as if it was sflow, that is,
> 0 cache, immediately exported after sampled, because you'd need like
> 1:100 or higher resolution, to have any significant luck in hitting the same 
> flow twice. PTX has stopped supporting flow-cache entirely because of this, 
> at the sampling rate where cache would do something, the cache would overflow.
>
> Of course there are other monetisation opportunities via other mechanism than 
> data-in-the-wire, like DNS
>
>
> On Tue, 16 May 2023 at 15:57, Tom Beecher  wrote:
> >
> > Two simple rules for most large ISPs.
> >
> > 1. If they can see it, as long as they are not legally prohibited, they'll 
> > collect it.
> > 2. If they can legally profit from that information, in any way, they will.
> >
> > Now, ther privacy policies will always include lots of nice sounding 
> > clauses, such as 'We don't see your personally identifiable information'. 
> > This of course allows them to sell 'anonymized' sets of that data, which 
> > sounds great , except as researchers have proven, it's pretty trivial to 
> > scoop up multiple, discrete anonymized data sets, and cross reference to 
> > identify individuals. 

Re: LibreQos

[Dave Taht]

On Sat, May 13, 2023 at 12:28 AM Mark Tinka  wrote:
>
>
>
> On 5/12/23 17:59, Dave Taht wrote:
>
> > :blush:
> >
> > We have done a couple podcasts about it, like this one:
> >
> > https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/
> >
> > and have perhaps made a mistake by using matrix chat, rather than a
> > web forum, to too-invisibly, do development and support in, but it has
> > been a highly entertaining way to get a better picture of the real
> > problems caring ISPs have.
> >
> > I see you are in Africa? We have a few ISPs playing with this in kenya...
>
> DM me, please, the ones you are aware about that would be willing to
> share their experiences. I'd like to get them to talk about what they've
> gathered at the upcoming SAFNOG meeting in Lusaka.
>
> We have a fairly large network in Kenya, so would be happy to engage
> with the operators running the LibreQoS there.

I forwarded your info.

Slide 40 here, has an anonymized libreqos report of observed latencies
in Africa. The RTTs there are severely bimodal (30ms vs 300ms), which
mucks with a default assumption in sch_cake of a default 100ms RTT.

http://www.taht.net/~d/Misunderstanding_Residential_Bandwidth_Latency.pdf

There are two ways to deal with this, right now we are recommending
cake rtt 200ms setting to keep throughput up. The FQ component
dominates for most traffic, anyway. With a bit more work we hope to
come up with a way of more consistent queuing delay. Or, we could just
wait for more CDNs to and IXPs to deploy there.

/me hides

A note about our public plots:  We had a lot of people sharing
screenshots, so we added a "klingon mode" to consistently
transliterate the more private data to that language.

Another fun fact was by deploying this stuff several folk found
sufficient non-paying clients on their network to pay for the hardware
inside of a month or two.

>
> > We do not know. Presently our work is supported by equinix´s open
> > source program, with four servers in their Dallas DC, and they are
> > 25Gbit ports. Putting together enough dough to get to 100Gbit or
> > finding someone willing to send traffic through more bare metal at
> > that data center or elsewhere is on my mind. In other words, we can
> > easily spin up the ability to L2 route some traffic through a box in
> > their DCs, if only we knew where to find it. :)
> >
> > If you assume linearity to cores (which is a lousy assumption, ok?),
> > 64 Xeon cores could do about 200Gbit, running flat out. I am certain
> > it will not scale linearly and we will hit multiple bottlenecks on a
> > way to that goal.
> >
> > Limits we know about:
> >
> > A) Trying to drive 10s of gbits of realistic traffic through this
> > requires more test clients and servers than we have, or someone with
> > daring and that kind of real traffic in the first place. For example
> > one of our most gung-ho clients has 100Gbit ports, but not anywhere
> > near that amount of inbound traffic. (they are crazy enough to pull
> > git head, try it for a few minutes in production, and then roll back
> > or leave it up)
> >
> > B) A brief test of a 64 core AMD + Nvidia ethernet was severely
> > outperformed by our current choice of a 20 core xeon gold + intel 710
> > or 810 card. It is far more the ethernet card that is the dominating
> > factor. I would kill if I could find one that did a LPM -> CPU
> > mapping... (e.g. instead of a LPM->route mapping, LPM to what cpu to
> > interrupt). We also tried an 80 core arm to inconclusive results early
> > on.
> >
> > Tests of the latest ubuntu release are ongoing. I am not prepared to
> > bless that or release any results yet.
> >
> > C) A single cake instance on one of the more high end Xeons can
> > *almost* push 10Gbit/sec while eating a core.
> >
> > D) Our model is one cake instance per subscriber + the ability to
> > establish trees emulating links further down the chain. One ISP is
> > modeling 10 mmwave hops. Another is just putting in multiple boxes
> > closer to the towers.
> >
> > So in other words, 100s of gbits is achievable today if you throw
> > boxes at it, and more cost effective to do that way. We will of
> > course, keep striving to crack 100gbit native on a single box with
> > multiple cards. It is a nice goal to have.
> >
> > E) In our present, target markets, 10k typical residential subscribers
> > only eat 11Gbit/sec at peak. That is a LOT of the smaller ISPs and
> > networks that fit into that space, so of late we have been focusing
> > more on anal

LibreQos

[Dave Taht]

Changing the topic...

On Fri, May 12, 2023 at 7:11 AM Mark Tinka  wrote:
>
>
>
> On 5/12/23 15:03, Dave Taht wrote:
>
> > Libreqos is free software, working as a bridge, you can plug it in
> > between any two points on your network, and on cheap (350 bucks off of
> > ebay) xeon gold hardware easily cracks 25Gbits while shaping with a
> > goal of cracking 100Gbits one day soon.
>
> This is fantastic!

:blush:

We have done a couple podcasts about it, like this one:

https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/

and have perhaps made a mistake by using matrix chat, rather than a
web forum, to too-invisibly, do development and support in, but it has
been a highly entertaining way to get a better picture of the real
problems caring ISPs have.

I see you are in Africa? We have a few ISPs playing with this in kenya...

>
> I also found your post about it here:
>
> https://www.reddit.com/r/HomeNetworking/comments/11pmc9a/a_latency_on_the_internet_update_bufferbloat_sqm/
>
> If you could throw more hardware at it, could it do several 100's of Gbps?

We do not know. Presently our work is supported by equinix´s open
source program, with four servers in their Dallas DC, and they are
25Gbit ports. Putting together enough dough to get to 100Gbit or
finding someone willing to send traffic through more bare metal at
that data center or elsewhere is on my mind. In other words, we can
easily spin up the ability to L2 route some traffic through a box in
their DCs, if only we knew where to find it. :)

If you assume linearity to cores (which is a lousy assumption, ok?),
64 Xeon cores could do about 200Gbit, running flat out. I am certain
it will not scale linearly and we will hit multiple bottlenecks on a
way to that goal.

Limits we know about:

A) Trying to drive 10s of gbits of realistic traffic through this
requires more test clients and servers than we have, or someone with
daring and that kind of real traffic in the first place. For example
one of our most gung-ho clients has 100Gbit ports, but not anywhere
near that amount of inbound traffic. (they are crazy enough to pull
git head, try it for a few minutes in production, and then roll back
or leave it up)

B) A brief test of a 64 core AMD + Nvidia ethernet was severely
outperformed by our current choice of a 20 core xeon gold + intel 710
or 810 card. It is far more the ethernet card that is the dominating
factor. I would kill if I could find one that did a LPM -> CPU
mapping... (e.g. instead of a LPM->route mapping, LPM to what cpu to
interrupt). We also tried an 80 core arm to inconclusive results early
on.

Tests of the latest ubuntu release are ongoing. I am not prepared to
bless that or release any results yet.

C) A single cake instance on one of the more high end Xeons can
*almost* push 10Gbit/sec while eating a core.

D) Our model is one cake instance per subscriber + the ability to
establish trees emulating links further down the chain. One ISP is
modeling 10 mmwave hops. Another is just putting in multiple boxes
closer to the towers.

So in other words, 100s of gbits is achievable today if you throw
boxes at it, and more cost effective to do that way. We will of
course, keep striving to crack 100gbit native on a single box with
multiple cards. It is a nice goal to have.

E) In our present, target markets, 10k typical residential subscribers
only eat 11Gbit/sec at peak. That is a LOT of the smaller ISPs and
networks that fit into that space, so of late we have been focusing
more on analytics and polish than pushing more traffic. Some of our
new R/T analytics break down at 10k cake instances (that is 40 million
fq_codel queues, ok?), and we cannot sample at 10ms rates, falling
back to (presently) 1s conservatively.

We are nearing putting out a v1.4-rc7 which is just features and
polish, you can get a .deb of v1.4-rc6 here:

https://github.com/LibreQoE/LibreQoS/releases/tag/v1.4-rc6

There is an optional, and anonymized reporting facility built into
that. In the last two months, 44404 cake shaped devices shaping
.19Tbits that we know of have come online. Aside from that we have no
idea how many ISPs have picked it up! a best guess would be well over
100k subs at this point.

Putting in libreqos is massively cheaper than upgrading all the cpe to
good queue management, (it takes about 8 minutes to get it going in
monitor mode, but exporting shaping data into it requires glue, and
time) but better cpe remains desirable - especially that the uplink
component of the cpe also do sane shaping natively.

"And dang, it, ISPs of the world, please ship decent wifi!?", because
we can see the wifi going south in many cases from this vantage point
now. In the past year mikrotik in particular has done a nice update to
fq_codel and cake in RouterOS, eero 6s have got quite good, much of
openwifi/openwrt, evenroute  is good...

It feels

Re: Routed optical networks

[Dave Taht]

On Thu, May 11, 2023 at 7:32 AM Mark Tinka  wrote:
>
>
>
> On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote:
>
> Sandvine is not representative of global traffic because DPI is installed 
> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – 
> it is not the biggest source. Moreover, Mobiles would look better growing 
> because the limiting factor was on technology (5G proposed more than 4G, 4G 
> proposed much more than 3G) – it would probably would less disruptive in the 
> future.
>
> Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
> publicly. Sandvine could not see it.
>
>
> And QUIC is making the the DPI model so loved by MNO's quickly irrelevant.


What we have been doing with libreqos is add in FQ + AQM (sch_cake),
while pioneering some nifty new eBPF traffic monitoring abilities like
being able to do "mice and elephants" plots, and more importantly
sample TCP RTT statistics using a variant of kathie nichols' "passive
ping" (pping) utility.  This latter feature is proving very useful in
diagnosing problems deeper in the network.

Quic does respond to good ole queuing delays, packet drop, and in some
cases ecn, but I am going to miss pping as it deploys more. It is the
FQ that helps the most on videoconferencing and voip traffic.

Libreqos is free software, working as a bridge, you can plug it in
between any two points on your network, and on cheap (350 bucks off of
ebay) xeon gold hardware easily cracks 25Gbits while shaping with a
goal of cracking 100Gbits one day soon.

demo here: https://payne.taht.net , running a variety of flent based
stress tests, tcp up and downloads, the rrul test, etc. The
interesting part to me, is that *real* traffic actually looks nothing
like that on a per subscriber basis. Here´s a screen-movie example of
real-world netflix queuing depth and delay:

https://www.youtube.com/watch?v=C-2oSBr2200

Our new "piano roll" mice and elephants plotter is pretty neat, if I
can say so myself.

...

What we see from those ISPs that have shared their data from their
Libreqos deployment is that the principal usage of major bandwidth is
movie streaming, and there is essentially no difference in average
usage between a 50Mbit residential plan and a gbit plan and only
barely discernible at 25 vs 50.

Given how new this codebase is we do not have any long term statistics
for traffic growth or decline, or other patterns.

But I kind of hope more folk here fire it up in their labs, at least.
Please drop in on our chat channel #libreqos:matrix.org if you need
help getting it setup. Sampling as we can at 10ms is a very different
view of the net from 5 minute averages.



Specifically plugging because I would like to understand CDN traffic
patterns better, and do not have much data as yet on that side,
collected this way.

> Mark.



-- 
Podcast: 
https://www.linkedin.com/feed/update/urn:li:activity:7058793910227111937/
Dave Täht CSO, LibreQos

Re: Routed optical networks

[Dave Taht]

On Thu, May 11, 2023 at 4:25 AM Vasilenko Eduard
 wrote:
>
> I did investigate traffic for every Carrier while dealing with it as a 
> consultant (repeated many dozens of times).
>
> I have seen over a decade how traffic growth dropped year-over-year (from 60% 
> to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel).

There was a covid spike, but the trendline appeared to be down to 5%
projected this year in a british study for fixed residential that I
cannot find right now.

> Sometimes I talk to old connections and they confirm that it is even less now.
>
> In rear cases, It is typically possible to find this information on the 
> public Internet (I remember the case when Google disclosed traffic for 
> Pakistan at the conference with the explanation that 30% is attributed to new 
> subscribers, and an additional +30% is to more heavy content per subscriber).
>
> But mostly, it was confidential information from a discussion with Carriers – 
> they all know very well their traffic growth.
>
> In general, traffic stat is pretty confidential. I did not have the 
> motivation to aggregate it.
>
>
>
> Sandvine is not representative of global traffic because DPI is installed 
> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – 
> it is not the biggest source. Moreover, Mobiles would look better growing 
> because the limiting factor was on technology (5G proposed more than 4G, 4G 
> proposed much more than 3G) – it would probably would less disruptive in the 
> future.
>
> Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
> publicly. Sandvine could not see it.
>
>
>
> VNI is claiming so many things. Please show where exactly they show traffic 
> growth (I am not interested in prediction speculations). Is it possible to 
> understand CAGR for the 5 last years? Is it declining or growing? (traffic 
> itself is for sure still growing)
>
>
>
> Of course, the disruption could come at any year and add a new S-curve 
> (Metaverse?). But disruption is by definition not predictable.
>
>
>
> PS: Everything above and below in this thread is just my personal opinion.
>
>
>
> Eduard
>
> From: Etienne-Victor Depasquale [mailto:ed...@ieee.org]
> Sent: Thursday, May 11, 2023 12:48 PM
> To: Vasilenko Eduard 
> Cc: Dave Taht ; Phil Bedard ; 
> NANOG 
> Subject: Re: Routed optical networks
>
>
>
> Eduard, academics cite the VNI (and the Sandvine Global reports).
>
>
>
> Do you know of alternative sources that show traffic growth data you're more 
> comfortable with?
>
>
>
> Cheers,
>
>
>
> Etienne
>
>
>
> On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard 
>  wrote:
>
> But it is speculation, not a trend yet.
>
> I remember 10y ago every presentation started from the claim that 100B of IoT 
> would drive XXX traffic. It did not happen.
>
> Now we see presentations that AI would be talking to AI that generates  
> traffic.
>
> Maybe some technology would push traffic next S-curve, maybe not. It is still 
> speculation.
>
>
>
> The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 
> 2) video quality for subscribers. Nothing else yet.
>
> It is almost finished for both trends. We are close to the plateau of these 
> S-curves.
>
> For some years (2013-2020) I was carefully looking at numbers for many 
> countries: it was always possible to split CAGR for these 2 components. The 
> video part was extremely consistent between countries. The subscriber part 
> was 100% proportional to subscriber CAGR.
>
> Everything else up to now was “marketing” to say it mildly.
>
>
>
> Reminder: nothing in nature could grow indefinitely. The limit always exists. 
> It is only a question of when.
>
>
>
> PS: Of course, marketing people could draw you any traffic growth – it 
> depends just on the marketing budget.
>
>
>
> Eduard
>
> From: Dave Taht [mailto:dave.t...@gmail.com]
> Sent: Tuesday, May 9, 2023 11:41 PM
> To: Vasilenko Eduard 
> Cc: Phil Bedard ; Etienne-Victor Depasquale 
> ; NANOG 
> Subject: Re: Routed optical networks
>
>
>
> Up until this moment I was feeling that my take on the decline of traffic 
> growth was somewhat isolated, in that I have long felt that we are nearing 
> the top of the S curve of the data we humans can create and consume. About 
> the only source of future traffic growth I can think of comes from getting 
> more humans online, and that is a mere another doubling.
>
>
>
> On the other hand, predictions such as 640k should be enough for everyone did 
> not pan out.
>
>
>
> On the gripping ha

Re: Routed optical networks

[Dave Taht]

Up until this moment I was feeling that my take on the decline of traffic
growth was somewhat isolated, in that I have long felt that we are nearing
the top of the S curve of the data we humans can create and consume. About
the only source of future traffic growth I can think of comes from getting
more humans online, and that is a mere another doubling.

On the other hand, predictions such as 640k should be enough for everyone
did not pan out.

On the gripping hand, there has been an explosion of LLM stuff of late,
with enormous models being widely distributed in just the past month:

https://lwn.net/Articles/930939/

Could the AIs takeoff lead to a resumption of traffic growth? I still don´t
think so...


On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG 
wrote:

> Disclaimer: Metaverse has not changed Metro traffic yet. Then …
>
>
>
> I am puzzled when people talk about 400GE and Tbps in the Mero context.
>
> For historical reasons, Metro is still about 2*2*10GE (one “2” for
> redundancy, another “2” for capacity) in the majority of cases worldwide.
>
> How many BRASes serve more than 4/1.5=27k users in the busy hour?
>
> It means that 50GE is the best interface now for the majority of cases.
> 2*50GE=100Gbps is good room for growth.
>
> Of course, exceptions could be. I know BRAS that handles 86k subscribers
> (do not recommend anybody to push the limits – it was so painful).
>
>
>
> We have just 2 eyes and look at video content about 22h per week (on
> average). Our eyes do not permit us to see resolution better than
> particular for chosen distance (4k for typical TV, HD for smartphones, and
> so on). Color depth 10bits is enough for the majority, 12bits is sure
> enough for everybody. 120 frames/sec is enough for everybody. It would
> never change – it is our genetics.
>
> Fortunately for Carriers, the traffic has a limit. You have probably seen
> that every year traffic growth % is decreasing. The Internet is stabilizing
> and approaching the plateau.
>
> How much growth is still awaiting us? 1.5? 1.4? It needs separate
> research. The result would be tailored for whom would pay for the research.
>
> IMHO: It is not mandatory that 100GE would become massive in the metro. (I
> know that 100GE is already massive in the DC CLOS)
>
>
>
> Additionally, who would pay for this traffic growth? It also limits
> traffic at some point.
>
> I hope it would happen after we would get our 22h/4k/12bit/120hz.
>
>
>
> Now, you could argue that Metaverse would jump and multiply traffic by an
> additional 2x or 3x. Then 400GE may be needed.
>
> Sorry, but it is speculation yet. It is not a trend like the current
> (declining) traffic growth.
>
>
>
> Ed/
>
> *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org]
> *On Behalf Of *Phil Bedard
> *Sent:* Thursday, May 4, 2023 8:32 PM
> *To:* Etienne-Victor Depasquale ; NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> It’s not necessarily metro specific although the metro networks could lend
> themselves to overall optimizations.
>
>
>
> The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your
> adoption of 400G since today they require a QDD port.   There are 100G QDD
> ports but that’s not all that popular yet.   Of course there is work to do
> something similar in QSFP28 if the power can be reduced to what is
> supported by an existing QSFP28 port in most devices.   In larger networks
> with higher speed requirements and moving to 400G with QDD, using the DCO
> optics for connecting routers is kind of a no-brainer vs. a traditional
> muxponder.   Whether that’s over a ROADM based optical network or not,
> especially at metro/regional distances.
>
>
>
> There are very large deployments of IPoDWDM over passive DWDM or dark
> fiber for access and aggregation networks where the aggregate required
> bandwidth doesn’t exceed the capabilities of those optics.  It’s been done
> at 10G for many years.  With the advent of pluggable EDFA amplifiers, you
> can even build links up to 120km* (perfect dark fiber)  carrying tens of
> terabits of traffic without any additional active optical equipment.
>
>
>
> It’s my personal opinion we aren’t to the days yet of where we can simply
> build an all packet network with no photonic switching that carries all
> services, but eventually (random # of years) it gets there for many
> networks.  There are also always going to be high performance applications
> for transponders where pluggable optics aren’t a good fit.
>
>
>
> Carrying high speed private line/wavelength type services as well is a
> different topic than interconnecting IP devices.
>
>
>
> Thanks,
>
> Phil
>
>
>
>
>
> *From: *NANOG  on behalf
> of Etienne-Victor Depasquale via NANOG 
> *Date: *Monday, May 1, 2023 at 2:30 PM
> *To: *NANOG 
> *Subject: *Routed optical networks
>
> Hello folks,
>
>
>
> Simple question: does "routed optical networks" have a clear meaning in
> the metro area context, or not?
>
>
>
> Put differently: 

Re: SF union square area fiber

[Dave Taht]

That is a very illustrative photo of the state of the internet plant,
even today. I hope you find it!

On Tue, Apr 4, 2023 at 8:51 AM Bill Woodcock  wrote:
>
> > On Apr 4, 2023, at 5:39 PM, Jared Mauch  wrote:
> > Can someone who is familiar with the fiber assets around the union square 
> > area in SF ping me off-list?
>
> Heh.  Somewhere, I have photos that Steve Feldman and I took while spelunking 
> around under there trying to find fiber for the NANOG that was held there in 
> 1997.  We found a gas-chandelier maintenance department that people had just 
> locked up and walked away from.  Tools still spread out on workbenches, lamps 
> half-rebuilt, everything.  It was like electrification had hit one day during 
> lunch-hour.
>
> -Bill
>


-- 
AMA March 31: https://www.broadband.io/c/broadband-grant-events/dave-taht
Dave Täht CEO, TekLibre, LLC

Re: Suggestions for those attending NANOG 88 in Seattle

[Dave Taht]

ed to be 
>> > purchased in person in Seattle.
>> >
>> > Monday Social Event Guest Pass: $50 per guest (purchase separately when 
>> > you register, limit 2)
>> >
>> > Tuesday Night Beer N Gear Pass: $50 per guest (purchase separately when 
>> > you register, limit 1)
>> >
>> >
>> > Registration Cancellation Fees:
>> >
>> > NANOG hopes everyone who registers for the meeting will be able to attend; 
>> > however, we know extenuating circumstances do occur. The NANOG 
>> > cancellation and refund policies are as follows:
>> >
>> > Registrations canceled on 28 March to 28 May, 2023 is refundable but will 
>> > incur a $50.00 fee
>> >
>> > Registrations canceled on 29 May to June 10, 2023 is refundable but will 
>> > incur a $100.00 fee
>> >
>> > Registrations canceled on or after 11 June, 2023 will not receive a refund
>> >
>> >
>> > Hotel Guest Room Block
>> >
>> > Hyatt Regency Seattle
>> >
>> > 808 Howell Street
>> >
>> > Seattle, Washington  98101
>> >
>> > Online Reservations: click here
>> >
>> > Group Name:NANOG 88
>> >
>> > Room Rate:  $299.00 USD  Standard Room Occupancy* (up to 2 people)
>> >
>> > $324.00 USD Triple Occupancy*
>> >
>> > $349.00 USD Quadruple Occupancy*
>> >
>> > *PLUS 15.7% Local Occupancy Tax + $4.00 Tourism Fee
>> >
>> > Group Rate Expires: 18 May, 2023 OR once the NANOG block is filled
>> > Reservations:  +206. 973.1234 OR Toll Free 1.800.233.1234
>> >
>> > Check In Time:  4:00PM
>> > Check Out Time:   11:00AM
>> >
>> > Hotel Cancellation Policy: Guests can cancel their reservations 48 hours 
>> > prior to their arrival to receive a full refund of their deposit. Any 
>> > reservation canceled after 48 priors to arrival will forfeit a fee of one 
>> > night's room and tax.
>> >
>> > Self-Parking Only: (prices subject to change)
>> >
>> > 0-1 Hour $17.00
>> >
>> > 1-2 Hours $20.00
>> >
>> > 2-3 Hours $23.00
>> >
>> > 3-4 Hours $26.00
>> >
>> > 4-5 Hours $29.00
>> >
>> > 5-12 Hours $33.00
>> >
>> > 12-24 Hours $48.00
>> >
>> >
>> > City & State taxes included in the above rates.  Overnight parking 
>> > includes in & out privileges.  Guest room charge available, room key used 
>> > for access after arrival.
>> >
>> >
>> > VISA Requests
>> >
>> > A letter of invitation is issued solely for the purpose of assisting 
>> > participants with visa applications for their attendance at the 
>> > conference. If you require a letter of invitation, please register and pay 
>> > for the NANOG meeting. Once the payment has been received, all requests 
>> > for Letters of Invitation should be addressed by email directly to 
>> > nanog-supp...@nanog.org
>> >
>> >
>> > The following information is required before a letter of invitation will 
>> > be issued:
>> >
>> > Name as it appears on your passport
>> >
>> > Passport Number
>> >
>> > Email Address
>> >
>> > Hotel name and reservation number
>> >
>> > Company name and address
>> >
>> >
>> >   4. Room Block and Attendee List Poachers
>> >
>> > Please be aware that companies not affiliated with the NANOG organization 
>> > may contact you in an  attempt to sell you a hotel room. This is 
>> > fraudulent activity as NANOG does not utilize third party service 
>> > providers for housing services.
>> >
>> >
>> > If you have any questions about the meeting, please contact us directly 
>> > at: nanog-supp...@nanog.org.
>> >
>> >
>> > We look forward to seeing you in Seattle!
>> >
>> >
>> > Sincerely, the NANOG Staff
>> >
>> > nanog-supp...@nanog.org
>>
>>
>>
>> --
>> For hire. https://bill.herrin.us/resume/



-- 
AMA March 31: https://www.broadband.io/c/broadband-grant-events/dave-taht
Dave Täht CEO, TekLibre, LLC

Re: A blatant podcast plug

[Dave Taht]

On Sun, Mar 5, 2023 at 3:02 PM Alexander Huynh via NANOG
 wrote:
>
> On 2023-03-05 12:34:40 -0800, Dave Taht wrote:
> >I rather enjoyed doing this podcast a few weeks ago, (and enjoy this
> >podcast a lot, generally), and it talks to what I've been up to for
> >the past year or so on fixing bufferbloat for ISPs.
> >
> >https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/
>
> Thank you for the link! I'll give it a listen this evening.

Pathetically, what did you think?

> >I am kind of curious as to how much XDP and EBPF now exist in the
> >nanog universe and other applications y'all are finding for it?
>
> We at Cloudflare use both XDP and eBPF extensively for our load
> balancing and DoS mitigation applications:
> https://www.google.com/search?q=site%3Ablog.cloudflare.com+xdp+OR+ebpf

David Tubes (of cloudflare) gave a pretty good talk at our recently
concluded "understanding latency conference.
Toke gave a pretty good talk on the state of bpf
Kathie Nichols talked about some nifty packet analysis techniques
Vodaphone opened with surprising candor about there being no demand for > 1gbit
Nokia talked about L4S
Stuart Cheshire of apple talked about their RPM metric
Had a good panel with ookla

And me, I channeled Roy Beatty from Blade Runner for all the network
problems I have seen and attempted to fix in just the past few weeks,
for a 3 minute monologue.

https://www.youtube.com/watch?v=tAVwmUG21OY=6483s

:)

I would really like to start a "back to packet captures" movement!
Have MS-clippy show up and say "Your network is being weird, would you
like me to take a packet capture?", and/or daveGPT3 chime in.

Anyway, last blatant plug on this list, if you want to feel some mild
winds of positive change, please cue up:

https://www.understandinglatency.com/recordings-2023

and pass around.





> --
> Alex



--
Come Heckle Mar 6-9 at: https://www.understandinglatency.com/
Dave Täht CEO, TekLibre, LLC

A blatant podcast plug

[Dave Taht]

I rather enjoyed doing this podcast a few weeks ago, (and enjoy this
podcast a lot, generally), and it talks to what I've been up to for
the past year or so on fixing bufferbloat for ISPs.

https://packetpushers.net/podcast/heavy-networking-666-improving-quality-of-experience-with-libreqos/

I am kind of curious as to how much XDP and EBPF now exist in the
nanog universe and other applications y'all are finding for it?

-- 
Come Heckle Mar 6-9 at: https://www.understandinglatency.com/
Dave Täht CEO, TekLibre, LLC

Re: Upstream bandwidth usage

[Dave Taht]

I dug out this old thread again...

https://www.broadband.io/c/get-broadband-grant-alerts-news/the-brothers-wisp-podcast

What is the request/grant latency in various gpons? DOCSIS-LL has it
below 2ms, I think.

On Sun, Jun 12, 2022 at 12:00 AM Mark Tinka  wrote:
>
>
>
> On 6/11/22 22:20, Karsten Thomann via NANOG wrote:
>
> >
> > Does anyone know the Asian market where they are using E-PON?
> > After my very short search it seems they provide best effort up to 1G 
> > without
> > any real plans...
>
> When I was in Malaysia years back, we did use ZTE for some EPON
> services. But we retired those and moved to GPON.
>
> Mark.



-- 
A pithy note on VOQs vs SQM: https://blog.cerowrt.org/post/juniper/
Dave Täht CEO, TekLibre, LLC

Re: Yondoo provided router, has "password" as admin pw, won't let us change it

[Dave Taht]

I am also a big fan of installing cake (sqm-scripts) in front cable devices.

On Thu, Feb 9, 2023 at 5:59 AM Todd Stiers  wrote:
>
> [OP here]
>
> Just some minor follow up:
>
>  - The tech was able to swap out their RG with the modem-only one that I had 
> sent (after making a couple phone calls). It didn't seem like they could 
> provision a user-supplied modem remotely for some reason, but it also sounded 
> like maybe this wasn't something they normally do, if ever.
>
>  - The outgoing RG was an Evolution Digital EVO3000GW. The screenshots that 
> dropped were meant to show me attempting an admin password change, and it not 
> letting me.
>
>  - AFAIK, no WAN ports were open, but UPnP was on by default. I neglected to 
> do a port scan on the WAN port before the equipment swap, but that probably 
> would've been prudent.
>
>  - Sorry for not being clear about this before, but I'm fairly remote (~5 
> hour drive), so my mom was acting as remote [somewhat arthritic] hands in all 
> this.
>
>  - Since I'm remote, I had previously sent a raspberry pi that is running 
> both pi-hole (to mitigate the possibility of her or her partner clicking on a 
> malicious ad or pop-up that may compel them to inadvertently connect with a 
> call center scammer again) and ZeroTier. I use ZT to log in to this device, 
> which double NAT breaks, which is why I brought that up. Totally 
> understandable that most average customers don't use this, and a double-NAT 
> situation is probably fine for my mom's demographic. That said, to be sure, 
> the much bigger issue is that they're provisioning CPE with an unchangeable 
> "password."
>
>  - I understand that this forum may not be quite the right fit for a post 
> like this, and am looking for others that may be more appropriate. My hope is 
> that this eventually gets to someone at Yondoo, or parent Mid-Atlantic 
> Broadband (AS29914), since something like this probably falls outside of the 
> wheelhouse of their tier 1 support, which was all we could get a hold of.
>
> Thanks to everyone who's responded -- I value all of your input.
>
> Cheers,
> Todd
>
> On Wed, Feb 8, 2023 at 5:09 PM Jason R. Rokeach via NANOG  
> wrote:
>>
>> It’s been a while, but attacks that take advantage of this are (or at least 
>> in the past have been) real.
>>
>> https://blog.sucuri.net/2014/09/website-security-compromised-website-used-to-hack-home-routers.html
>>
>> https://www.digitaltrends.com/web/javascript-malware-mobile/
>>
>> I recall when this stuff first started to come out, leaning on RG vendors to 
>> fix their firmware to make their default passwords unpredictable based on 
>> information readily available on the LAN.
>> In this case we’re not even talking about taking action this sophisticated… 
>> It seems to me that, having a customer willing and ready to secure 
>> themselves, preventing them from doing so is wildly inappropriate.
>>
>>
>> On Wed, Feb 8, 2023 at 7:57 PM, Eric Kuhnke  wrote:
>>
>> I agree, but if we start listing every massive security vulnerability that 
>> can be found on the intra-home LAN in consumer-grade routers and home 
>> electronics equipment, or things that people operate in their homes with the 
>> factory-default passwords, we'd be here all month in a thread with 300 
>> emails.
>>
>> I'm sure this ISP will realize what a silly thing they did if and when some 
>> sort of worm or trojan tries a set of default logins/passwords on whatever 
>> is the default gateway of the infected PC, and does something like rewrite 
>> the IPs entered for DNS servers to send peoples' web browsing to advertising 
>> for porn/casinos/scams, male anatomy enlargement services or something.
>>
>>
>>
>> On Wed, Feb 8, 2023 at 3:28 PM William Herrin  wrote:
>>>
>>> On Wed, Feb 8, 2023 at 2:36 PM Eric Kuhnke  wrote:
>>> > I would hope that this router's admin "password" interface is only 
>>> > accessible from the LAN side.
>>> > This is bad, yes, but not utterly catastrophic.
>>>
>>> It means that any compromised device on the LAN can access the router
>>> with whatever permissions the password grants. While there are
>>> certainly worse security vulnerabilities, I'm reluctant to describe
>>> this one as less than catastrophic. Where there's one grossly ignorant
>>> security vulnerability there are usually hundreds.
>>>
>>> Regards,
>>> Bill Herrin
>>>
>>>
>>> --
>>> For hire. https://bill.herrin.us/resume/



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Does anyone know of a martin broadhurst?

[Dave Taht]

He was apparently based in cambridge, but his site is down.

Recently I stumbled across a series of articles by him that were
lovingly illustrative, incredibly clearly well written, and appeared
to use a C library that I don't have, covering a ton of fundamental
algorithms that I would like to have in my toolkit, still, in C,
because that is how I still think.

I first found him whilst I was looking for a binpacking algorithm(s)
and benchmarks for how well they performed on modern hardware. (I am
trying to wedge 40m queues into 16 cores every 10ms)

https://web.archive.org/web/20190530024749/http://www.martinbroadhurst.com/tag/bin-packing

And then I started reading the rest of his canon: Wow:

https://web.archive.org/web/20190704202816/http://www.martinbroadhurst.com/author/martin

Stuff like (hit the archive per above to get these)

http://www.martinbroadhurst.com/bellman-ford-algorithm-in-c.html

really clean avl example:

http://www.martinbroadhurst.com/avl-tree-in-c.html

Anyway, given that the site is down, and he hasn´t posted anything in
a year, I fear he has passed on. I did find an obituary that had about
the right timing in 2021, and he had the neckbeard... Did/does anyone
here know him?

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: starlink downlink/internet access

[Dave Taht]

I maintain an email list for issues specific to starlink here:
https://lists.bufferbloat.net which has multiple experts on it. There
are also quite a few folk on twitter covering what's going on there.

The latest information I had was that they'd started off hooked up to
google's stuff but have been building out their own network wherever
they can.


On Tue, Jan 10, 2023 at 8:50 PM Ong Beng Hui  wrote:
>
> Hi folks,
>
> Anyone know/advise if Starlink internet downlink is in US Google fiber ?
> I thought I saw a message before that Starlink was using Google fiber.
>
> I was referring to the actual internet transit, not the Satellite
> downlink station.
>
> Please advise.
>


-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: Google Speed Test

[Dave Taht]

I maintain a fleet of 15 "flent" servers across the globe, leveraging
irtt, iperf, netperf, and a few other tools. I do not have the
resources to publish them widely (flent.org's tools are by design,
intended more for folk to quickly spin up a server and client for
internal tests, because most of the results are very embarrassing for
the ISPs and vendors). They are widely available for linux and osx, as
part of their package repositories, with packages for openwrt as well.

My favorite tests in that suite are the rrul, rrul_be, "squarewave"
and tcp_nup tests, which are still the only few that sample tcp_info
enough to get cwnd and rtt statistics directly from the streams.

waveform's speedtest leverages cloudflare's cdn.

The new speedtest.net apps test for the presence of "FQ" more than
bufferbloat, but also come in command line versions.

Re: Google Speed Test

[Dave Taht]

Waveform leverages cloud flare's CDN.

I have a worldwide fleet of iperf, netperf, flent and irtt servers folk are
welcome to use, as I don't trust the web best tests

On Wed, Dec 28, 2022, 11:44 AM Douglas Fischer 
wrote:

> I have recollection of something like embeded quality testing on youtube.
> I don't remember if it was a speed test or a latency/jitter test.
>
> I looked quickly to see if I could find it... But I couldn't find it.
>
> Em qua., 28 de dez. de 2022 às 13:43, Mike Hammett 
> escreveu:
>
>> Does AS15169 have a speed test? It would be nice to gauge the capacity to
>> a particular network that's something laypeople could do. I could host
>> something in GCP myself, but cloud is expensive.
>>
>> -
>> Mike Hammett
>> [ http://www.ics-il.com/ | Intelligent Computing Solutions ]
>> [ https://www.facebook.com/ICSIL ] [
>> https://plus.google.com/+IntelligentComputingSolutionsDeKalb ] [
>> https://www.linkedin.com/company/intelligent-computing-solutions ] [
>> https://twitter.com/ICSIL ]
>> [ http://www.midwest-ix.com/ | Midwest Internet Exchange ]
>> [ https://www.facebook.com/mdwestix ] [
>> https://www.linkedin.com/company/midwest-internet-exchange ] [
>> https://twitter.com/mdwestix ]
>> [ http://www.thebrotherswisp.com/ | The Brothers WISP ]
>> [ https://www.facebook.com/thebrotherswisp ] [
>> https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg ]
>>
>
>
> --
> Douglas Fernando Fischer
> Engº de Controle e Automação
>

Re: Large RTT or Why doesn't my ping traffic get discarded?

[Dave Taht]

There's this thing called bufferbloat...

On Wed, Dec 21, 2022 at 11:58 AM William Herrin  wrote:
>
> On Wed, Dec 21, 2022 at 9:10 AM Jason Iannone  wrote:
> > Here's a question I haven't bothered to ask until now. Can someone please 
> > help me understand why I receive a ping reply after almost 5 seconds?
> >
> > 64 bytes from 4.2.2.2: icmp_seq=398 ttl=54 time=4915.096 ms
> > 64 bytes from 4.2.2.2: icmp_seq=399 ttl=54 time=4310.575 ms
> > 64 bytes from 4.2.2.2: icmp_seq=400 ttl=54 time=4196.075 ms
> > 64 bytes from 4.2.2.2: icmp_seq=401 ttl=54 time=4287.048 ms
> > 64 bytes from 4.2.2.2: icmp_seq=403 ttl=54 time=2280.466 ms
> > 64 bytes from 4.2.2.2: icmp_seq=404 ttl=54 time=1279.348 ms
> > 64 bytes from 4.2.2.2: icmp_seq=405 ttl=54 time=276.669 ms
>
> Hi Jason,
>
> This usually means a problem on the Linux machine originating the
> packet. It has lost the ARP for the next hop or something similar so
> the outbound ICMP packet is queued. The glitch repairs itself,
> briefly, releasing the queued packets. Then it comes right back.
>
> Regards,
> Bill Herrin
>
>
> --
> For hire. https://bill.herrin.us/resume/



-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

the ipv4 vs ipv6 growth debate

[Dave Taht]

I use a web plugin tool called ipvfoo to track my actual ipv4 vis ipv6
usage. I wish it worked over time. With very few exceptions I am still
regularly calling ipv4 addresses in most webpages. Has anyone done a
more organized study of say, the top 1 million, and how many still
require at least some ipv4 to exist, and those trends over time?

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: ipv4/25s and above

[Dave Taht]

Before this conversation forked off in a direction I didn't want it to go,
I'd like to thank everyone, privately and publicly, that gave me a hint as
to the distribution of /25s and greater in their networks.

I was at the time, trying to get "libreqos.io" to crack the 32k customer
barrier, which we did soon afterwards, and to get a decent histogram for
our trie emulations of what real customer traffic might look like once we
started scaling past 40Gbit. (the XDP trie lookup is pretty expensive for
us currently, but a lot cheaper than the /32s)

Some idea of failover mechanisms other than BGP (igps are all over the map)
came from this conversation also. I would certainly like to learn more
about how well that works.

A new question I have... is there any decent rules for CGNAT vs bandwidth
vs customer allocations of ipv4 space? I've had a few horror stories shared
with me privately about that, of note was 9000 *fiber* users behind a
single /32. with no way to haul ipv6 or more ips there

Re: Alternative Re: ipv4/25s and above Re: 202211201009.AYC

[Dave Taht]

On Mon, Nov 21, 2022 at 4:05 PM David Conrad  wrote:
>
> Barry,
>
> On Nov 21, 2022, at 3:01 PM, b...@theworld.com wrote:
>
> We've been trying to get people to adopt IPv6 widely for 30 years with very 
> limited success
>
>
> According to https://www.google.com/intl/en/ipv6/statistics.html, it looks 
> like we’ve gone from ~0% to ~40% in 12 years. 
> https://stats.labs.apnic.net/ipv6 has it around 30%. Given an Internet 
> population of about 5B, this can (simplistically and wrongly) argued to mean 
> 1.5-2B people are using IPv6. For a transition to a technology that the vast 
> majority of people who pay the bills will neither notice nor care about, and 
> for which the business case typically needs projection way past the normal 
> quarterly focus of shareholders, that seems pretty successful to me.
>
> But back to the latest proposal to rearrange deck chairs on the IPv4 Titanic, 
> the fundamental and obvious flaw is the assertion of "commenting out one line 
> code”. There isn’t “one line of code”. There are literally _billions_ of 
> instances of “one line of code”, the vast majority of which need to be 
> changed/deployed/tested with absolutely no business case to do so that isn’t 
> better met with deploying IPv6+IPv4aaS. I believe this has been pointed out 
> numerous times, but it falls on deaf ears, so the discussion gets a bit 
> tedious.

I have been trying to steer clear of this debate this time around, but
since I'm the one that made that analogy to begin with...

There are now billions and billions of *non-instances* of this one
line of code, saving nanoseconds on every connection, since 2008 in
the case of 240/4 and 2018 in the case of 0/8 - and that savings
alone, I felt was worth it. No additional future use is required from
my perspective to have realized real economic value from these address
spaces.

It would be rather nice, if, over time, we pretty much agreed that
embedding an 1981 policy into future OS kernels and routers transport
mechanisms was silly.

Full stop. Can someone citing me about the non-wisdom of "delete 1
line of code from everything" try to explain why our OSes MUST
continue enforcing some distinction between 240/4 and 0/8 and the rest
of the known unicast internet?

...

To take the next step - towards some sort of allocation policy - is a
matter of years and years. The subject of current research is what
does trying to make it work, break? I regularly use 240 nowadays
myself where I am not sure where the rfc1918 space is... or on a vpn -
eating my dogfood - but I do think it would be a tragic waste if we
didn't make an effort to make them globally usable in the long run.

I also tend to be upset by the argument that "this must work
internet-wide, on everything, forever, and immediately", which of
course, doesn't apply to ipv6 either.

No, it just needs to work on islands with limited address space,
initially. Tunnels between forward thinking providers, perhaps.
Starlink could use it to address terminals if they wanted - they still
don't have ipv6 working worth a darn -

I've also said a lot, that "the prospect of a portion of the internet
completely immune to windows-born viruses and worms is really
pleasing..." and I get a lot of laughs from that, because it's true -
If you've been in the trenches, fighting those off for the last few
decades, knowing that *some* piece of your infrastructure couldn't be
subject to those sort of attacks from old or windows OSes is a relief.

Arguments for deploying ipv6 remain! There's no escaping ipv6, and I
spend a lot of time trying to convince ISPs nowadays to deploy that,
but *all* of the ones I'm presently working with still must provide
IPv4 space, and thus are deploying CGNAT more rapidly than ipv6. I
will keep trying to get ipv6 deployed at every chance I get! I'm very
happy to have finally got ipv6 trie support into libreqos.io a few
weeks ago - but the demand is all cgnat, and mpls and vlans and ipv4
tunnels - I'd love to find a customer to try out our new ipv6 support,
because despite trying for months, we don't have any, as yet.

Blatant plug: 
https://github.com/LibreQoE/LibreQoS/tree/main/v1.3#v13-ipv4--ipv6-beta

Anyway... some use of these new ipv4 address spaces is inevitable, and
I really do wish y'all cared more about nanoseconds,
here or there, or anywhere.

>
> Regards,
> -drc
>


-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

ipv4/25s and above

[Dave Taht]

I am kind of curious as to the distribution of connections to smaller
companies and other entities that need more than one ipv4 address, but
don't run BGP. So, for as an ISP or infrastructure provider, what is
the typical percentage nowadays of /32s /31s /30s... /25s of stuff
that gets run "elsewhere"?

Is there any correlation between the number of IPs a customer gets and
the amount of bandwidth they buy?

Obviously "retail", home use is /32s and there's an increasing amount
of CGNAT, but I can't help but imagine there are thousands of folk
running /27s and /29s for every /24 or /22 out there.

I've been paying 15/month for a /29 for forever, but barely use it.

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: jon postel

[Dave Taht]

That book needs a sequel.

+10 on the internet history mailing list also.

2:45 apple's talk on responsiveness tuesday

[Dave Taht]

It's "fun" to run their new go based responsiveness test on  the
conference wifi: https://github.com/network-quality/goresponsiveness -
especially
with two or more people at the same time. Hotel wifi is *even* more
fun. in-Bar 5G comparing it (or the new speedtest app which is also
measuring roughly the same things) to 4G, oh my! Would love it if more
folk tried it and shared data!

Also: https://github.com/Zoxc/crusader is shaping up, and has a
staggered start test now.

And of course, there's always flent.

I'm not at this nanog, tho I wish I was.


-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

Re: jon postel

[Dave Taht]

On Sun, Oct 16, 2022 at 2:21 PM Randy Bush  wrote:
>
> my favorite is
>
> It's perfectly appropriate to be upset.  I thought of it in a slightly
> different way--like a space that we were exploring and, in the early days,
> we figured out this consistent path through the space: IP, TCP, and so on.
> What's been happening over the last few years is that the IETF is filling
> the rest of the space with every alternative approach, not necessarily any
> better.  Every possible alternative is now being written down.  And it's not
> useful.  -- Jon Postel

I wish I'd met him. I know I would have liked him a lot. We wear the
same sandals.

-- 
This song goes out to all the folk that thought Stadia would work:
https://www.linkedin.com/posts/dtaht_the-mushroom-song-activity-698135607352320-FXtz
Dave Täht CEO, TekLibre, LLC

bufferbloat-beating customer shaping via LibreQoS

[Dave Taht]

There's been a huge uptake in interest lately in doing better per
device and per customer shaping, especially for
ISPs, in the libreQoS.io project, which is leveraging the best ideas
bufferbloat project members have had over the
past decade (cake, bpf, xdp) to push an x86 middlebox well past the
10Gbit barrier, on sub-2k boxes, with really
good stats on backlogs, drops, and ecn marks. I've long primarily
tried to get fq_codel and cake running on the CPE (most recently
mikrotik), and that's been taking too long.

I have no idea to what extent members of this list have interest in
this, but if you know of a smaller ISP with bad bufferbloat,
please pass that link along? It's got ridiculously easier to set up as
a vm of late.

There is presently a design discussion going on over here:

https://github.com/rchac/LibreQoS/issues/57

And by mentioning it here, today, I'm mostly asking what other real
life use cases we should try to tackle? What backend tools should we
try to integrate with?

-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: End of Cogent-Sprint peering wars?

[Dave Taht]

On Wed, Sep 7, 2022 at 1:48 PM Sean Donelan  wrote:
>
>
> Are Sprint AS1239 and Cogent AS174 finally going to settle their peering
> disputes?
>
> T-Mobile sells legacy Sprint wireline business to Cogent for $1, expects
> hefty charge
> https://www.reuters.com/markets/deals/cogent-communications-acquire-t-mobiles-wireline-business-2022-09-07/
>
> 1,400 customers
> 1,300 employees
>
> 19,000 long-haul route miles
> 1,300 metro route miles
> 16,800 leased route miles

That's a dollar well spent. It also explains the layoffs.
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: RFC: BOGONs over BGP, adding some ranges

[Dave Taht]

On Tue, Aug 30, 2022 at 10:17 AM James Shank  wrote:
>
> Dear NANOG!
>
> As many of you know, Team Cymru runs a free service delivering updated
> BOGONS to networks around the world. We've been doing this for decades
> at this point. For more information about this service, please see
> https://www.team-cymru.com/bogon.
>
> Recently, we've discussed internally a discrepancy between our BGP based
> feed and the other formats we deliver for our Traditional BOGONS feed.
> For historic reasons, we previously omitted delivery of some ranges that
> are BOGONS from our BGP advertisements. We are considering adding the
> below ranges back in, but want to hear feedback from the community on
> these ranges prior to advertising them, out of an abundance of caution.
>
> The below ranges are what we are currently considering advertising:
> 0/8 (this one we already have concluded is safe as it is already
> advertised in our "FULL BOGONS" set.)
> 127/8
> 224/4
> 240/4
>
> Note that 224/4 and 240/4 may be aggregated differently in our
> advertisement, but are broken out here to facilitate discussion.

I would like to make an effort to debogon 240/4 at least, or have an
authorized experiment to at least determine how fully bogon'd it is.

Some recent data about amazon and verizon:

https://labs.ripe.net/author/qasim-lone/2404-as-seen-by-ripe-atlas/

One question for me is how many folks rely on regular bogon updates
such as yours. There are others.

> So, fellow NANOGers, what say ye? I would love to hear your feedback,
> pro or con, well-reasoned with data points or general "argh! there be
> dragons!" sentiments.
>
> Looking forward to seeing folks in Hollywood for N86!
>
> Cheers!
>
> James
>
> --
> *James Shank*
> Chief Architect of Community Services and Sr. Security Evangelist
> e: jsh...@cymru.com
> o: +1 847 378-3365
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: CC: s to Non List Members (was Re: 202203080924.AYC Re: 202203071610.AYC Re: Making Use of 240/4 NetBlock)

[Dave Taht]

On Wed, Mar 9, 2022 at 9:11 AM Tim Howe  wrote:
>
> On Wed, 9 Mar 2022 11:22:49 -0500
> Tom Beecher  wrote:
>
> > > It doesn't take any OS upgrades for "getting everything to work on
> > > IPv6".  All the OS's and routers have supported IPv6 for more than a
> > > decade.
> > >
> >
> > There are lots of vendors, both inside and outside the networking space,
> > that have consistently released products with non-existant or broken IPv6
> > implementations. That includes smaller startups, as well as very big
> > names. An affirmative choice is often made to make sure v4 works , get the
> > thing out the door, and deal with v6 later, or if a big client complains.
>
> This a thousand times.  Don't believe the claims of IPv6
> support until you have fully tested it.  Almost no vendor is including
> any IPv6 testing in their QA process and nobody is including it in any
> of their support staff training.  Their labs may not even have v6
> capability.  Some of our biggest vendors who have supposedly supported
> v6 for over a decade have rudimentary, show-stopping bugs.  The support
> staff at these vendors have often never even seen a customer using v6,
> and they have no idea what it looks like on their own gear.

I have worked really hard to make sure ipv6 "just works" (still) in
the upcoming openwrt 22.03 release, treating it as *my* primary ip
stack, at least.

But I spent most of my time fixing a string of fq-codel & ATF wifi
regressions on the mt76 chips and (especially) not on testing the
various encapsulations, and am out of time.

If y'all care about ipv6, please lean in, test, and file bugs on these
last release candidates before it goes final?

https://downloads.openwrt.org/releases/22.03.0-rc6/targets/

The network you save may be your own.
>
> A subset of these vendors will listen to you and fix the
> problems.  Give them your support and loyalty.  I want to name names so
> bad...
>
> --TimH



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: 400G forwarding - how does it work?

[Dave Taht]

On Sun, Aug 7, 2022 at 11:24 PM Masataka Ohta
 wrote:
>
> sro...@ronan-online.com wrote:
>
> > There are MANY real world use cases which require high throughput at
> > 64 byte packet size.
>
> Certainly, there were imaginary world use cases which require
> to guarantee so high throughput of 64kbps with 48B payload
> size for which 20(40)B IP header was obviously painful and 5B
> header was used. At that time, poor fair queuing was assumed,
> which requires small packet size for short delay.
>
> But as fair queuing does not scale at all, they disappeared
> long ago.

What do you mean by FQ, exactly?

"5 tuple FQ" is scaling today on shaping middleboxes like preseem and
LibreQos to over 10gbits. ISP reported results of customer calls about
speed simply vanish. Admittedly the AQM is dropping or marking some
.x% of packets, but tests with fq with short buffers vs aqm alone
showed the former the clear winner, and fq+aqm took it in for the
score.

On linux fq_codel is the near universal default, also. The linux tcp
stack does fq+pacing at nearly 100gbits today on "BIG" tcp.

"disappeared", no. invisible, possible. transitioning from +10 gbit
down to 1gbit or less, really, really, useful. IMHO, and desparately
needed, in way more places.

Lastly VOCs and LAG and switch fabrics essentially FQ ports. In the
context of aggregating up to 400Gbit from that, you are FQing also.

Now fq-ing inline against the ip headers at 400gbit appeared
impossible until this convo when the depth of the pipeline and
hardware hashing was discussed, but I'll settle for more rfc7567
behavior just in stepping down from that, to 100, and from 100
stepping down, adding in fq+aqm.


>
>  > Denying those use cases because they don’t fit
>  > your world view is short sighted.
>
> That could have been a valid argument 20 years ago.
>
> Masataka Ohta



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: 400G forwarding - how does it work?

[Dave Taht]

If it's of any help... the bloat mailing list at lists.bufferbloat.net has
the largest concentration of
queue theorists and network operator + developers I know of. (also, bloat
readers, this ongoing thread on nanog about 400Gbit is fascinating)

There is 10+ years worth of debate in the archives:
https://lists.bufferbloat.net/pipermail/bloat/2012-May/thread.html as one
example.

On Sun, Aug 7, 2022 at 10:14 AM dip  wrote:

>
> Disclaimer: I often use the M/M/1 queuing assumption for much of my work
> to keep the maths simple and believe that I am reasonably aware in which
> context it's a right or a wrong application :). Also, I don't intend to
> change the core topic of the thread, but since this has come up, I couldn't
> resist.
>
> >> With 99% load M/M/1, 500 packets (750kB for 1500B MTU) of
> >> buffer is enough to make packet drop probability less than
> >> 1%. With 98% load, the probability is 0.0041%.
>
> To expand the above a bit so that there is no ambiguity. The above assumes
> that the router behaves like an M/M/1 queue. The expected number of packets
> in the systems can be given by
>
> [image: image.png]
> where [image: image.png] is the utilization. The probability that at
> least B packets are in the system is given by  [image: image.png] where B
> is the number of packets in the system. for a link utilization of .98, the
> packet drop probability is .98**(500) = 0.41%. for a link utilization
> of 99%,  .99**500 = 0.00657%.
>
>
Regrettably, tcp ccs, by design do not stop growth until you get that drop,
e.g. 100+% utilization.


>> When many TCPs are running, burst is averaged and traffic
> >> is poisson.
>
> M/M/1 queuing assumes that traffic is Poisson, and the Poisson assumption
> is
> 1) The number of sources is infinite
> 2) The traffic arrival pattern is random.
>
> I think the second assumption is where I often question whether the
> traffic arrival pattern is truly random. I have seen cases where traffic
> behaves more like self-similar. Most Poisson models rely on the Central
> limit theorem, which loosely states that the sample distribution will
> approach a normal distribution as we aggregate more from various
> distributions. The mean will smooth towards a value.
>
> Do you have any good pointers where the research has been done that
> today's internet traffic can be modeled accurately by Poisson? For as many
> papers supporting Poisson, I have seen as many papers saying it's not
> Poisson.
>
> https://www.icir.org/vern/papers/poisson.TON.pdf
> https://www.cs.wustl.edu/~jain/cse567-06/ftp/traffic_models2/#sec1.2
>

I am firmly in the not-poisson camp, however, by inserting (esp) FQ and AQM
techniques on the bottleneck links it is very possible to smooth traffic
into this more easily analytical model - and gain enormous benefits from
doing so.



> On Sun, 7 Aug 2022 at 04:18, Masataka Ohta <
> mo...@necom830.hpcl.titech.ac.jp> wrote:
>
>> Saku Ytti wrote:
>>
>> >> I'm afraid you imply too much buffer bloat only to cause
>> >> unnecessary and unpleasant delay.
>> >>
>> >> With 99% load M/M/1, 500 packets (750kB for 1500B MTU) of
>> >> buffer is enough to make packet drop probability less than
>> >> 1%. With 98% load, the probability is 0.0041%.
>>
>> > I feel like I'll live to regret asking. Which congestion control
>> > algorithm are you thinking of?
>>
>> I'm not assuming LAN environment, for which paced TCP may
>> be desirable (if bandwidth requirement is tight, which is
>> unlikely in LAN).
>>
>> > But Cubic and Reno will burst tcp window growth at sender rate, which
>> > may be much more than receiver rate, someone has to store that growth
>> > and pace it out at receiver rate, otherwise window won't grow, and
>> > receiver rate won't be achieved.
>>
>> When many TCPs are running, burst is averaged and traffic
>> is poisson.
>>
>> > So in an ideal scenario, no we don't need a lot of buffer, in
>> > practical situations today, yes we need quite a bit of buffer.
>>
>> That is an old theory known to be invalid (Ethernet switches with
>> small buffer is enough for IXes) and theoretically denied by:
>>
>> Sizing router buffers
>> https://dl.acm.org/doi/10.1145/1030194.1015499
>>
>> after which paced TCP was developed for unimportant exceptional
>> cases of LAN.
>>
>>  > Now add to this multiple logical interfaces, each having 4-8 queues,
>>  > it adds up.
>>
>> Having so may queues requires sorting of queues to properly
>> prioritize them, which costs a lot of computation (and
>> performance loss) for no benefit and is a bad idea.
>>
>>  > Also the shallow ingress buffers discussed in the thread are not delay
>>  > buffers and the problem is complex because no device is marketable
>>  > that can accept wire rate of minimum packet size, so what trade-offs
>>  > do we carry, when we get bad traffic at wire rate at small packet
>>  > size? We can't empty the ingress buffers fast enough, do we have
>>  > physical memory for each port, do we share, how do we share?

Re: FEC AO 2022-14

[Dave Taht]

I welcomed bulk mail after I switched to reading news online - needed
something to start the fireplace.

If I could I'd ban plastic envelope windows.

Re: 4 gbit capable bufferbloat flent test target close to/traversing MegaIX in sydney?

[Dave Taht]

On Wed, Jul 27, 2022 at 2:48 PM Dave Taht  wrote:
>
> I am curious if there is anyone out there willing to run a server with
> irtt and netperf on it that I could do some bufferbloat testing
> against (in off peak hours)? I've been getting some severely bloated
> (250ms!) results on the 27ms path I'm on now at rates slightly above
> 1.2gbit (can share the ugly details privately)
>
> --
> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
> Dave Täht CEO, TekLibre, LLC

I would like to clarify that the issue I'm having is NOT with MegaIX,
but another that doesn't want to listen, on a path to linode...




-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

4 gbit capable bufferbloat flent test target close to/traversing MegaIX in sydney?

[Dave Taht]

I am curious if there is anyone out there willing to run a server with
irtt and netperf on it that I could do some bufferbloat testing
against (in off peak hours)? I've been getting some severely bloated
(250ms!) results on the 27ms path I'm on now at rates slightly above
1.2gbit (can share the ugly details privately)

-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: 400G forwarding - how does it work?

[Dave Taht]

This convo is giving me some hope that the sophisticated FQ and AQM
algorithms I favor can be made to run in more hardware at high rates,
but most of the work I'm aware of has targetted tofino and P4.

The only thing I am aware of shipping is AFD in some cisco hw. Anyone
using that?

Re: Health Tips for Desk Junkies + N86 Sponsorships Available!

[Dave Taht]

On Thu, Jul 7, 2022 at 5:25 PM William Herrin  wrote:
>
> On Thu, Jul 7, 2022 at 2:01 PM Nanog News  wrote:
> > Health Tips for Desk Junkies
>
> The idiom is "Desk Jockey" not "Desk Junkie." Because you "ride a
> desk" (another idiom) all day, not get addicted to sitting or
> something.

It's not the seat. It's the 50 inch monitor that's addicting.
>
> Regards,
> Bill Herrin
>
>
> --
> For hire. https://bill.herrin.us/resume/



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: What say you, nanog re: Starlink vs 5G?

[Dave Taht]

On Fri, Jun 24, 2022 at 10:06 AM Chris Wright
 wrote:
>
> The term "5G" among technical circles started vague, became better defined 
> over the course of several years, and is becoming vague again. This nuance 
> was never well understood in the public eye, nor by mass publications like 
> CNN. This is a battle for 12GHz, not 5G.

I second that. I will try to use that last sentence if I have to get
involved that fight. Elsewhere, though, I do wish that starlink would
adopt
an fq_codel derived algorithm on the dishy and headends to smooth out
the wildly variable latencies some.

https://www.phoronix.com/forums/forum/phoronix/latest-phoronix-articles/1330193-spacex-starlink-internet-experience-performance/page5
>
> Chris
>
>
> -Original Message-
> From: NANOG  On 
> Behalf Of John Levine
> Sent: Thursday, June 23, 2022 9:45 PM
> To: nanog@nanog.org
> Subject: Re: What say you, nanog re: Starlink vs 5G?
>
> It appears that Eric Kuhnke  said:
> >Adding a terrestrial transmitter source mounted on towers and with CPEs
> >that stomps on the same frequencies as the last 20 years of existing
> >two way VSAT terminals throughout the US seems like a bad idea. Even if
> >you ignore the existence of Starlink, there's a myriad of low bandwidth
> >but critical SCADA systems out there and remote locations on ku-band
> >two way geostationary terminals right now.
>
> I think the original thought was that the satellite service would be used in 
> rural areas and 5G in cities so there'd be geographic separation, but 
> Starlink is selling service all over the place.
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Congrats to AS701

[Dave Taht]

On Wed, Jun 15, 2022 at 6:09 PM Christopher Morrow
 wrote:
>
>
>
> On Mon, Jun 13, 2022 at 12:00 PM Justin Streiner  wrote:
>>
>> I might call Verizon and ask about v6 availability as I periodically do.  
>> I'll check if I see anything different on my gear later today.  I have a 
>> GPON business service with static IPv4 at one location and an older BPON 
>> business service with static IPv4 in another location.
>>
>
> As a short and not totally complete update to this problem... A 'long time 
> listener, first time caller' sort of person
> noted to me off-list that:
>   "Hey, once upon a time I dealt with hardware/vendor things... and we 
> wouldn't send 'RA type' packets (solicits/etc)
> down the customer leg UNLESS they had already sent a 
> RouterSolicitation... on the BNG platform."
>
> So... I copy/pasta'd some comcast facing config and.. low and behold my link 
> sends me a /56 if I ask for one via PD!
> for  I can't personally use 
> the v6 (yet) here, but this is super encouraging!
>
> Perhaps this is 'CPE configuration away' from working in a bunch more places?

Well, in a FIOS location[1] with the most ancient fios provided CPE I
know of (at least 10 years old), connected to a modern day openwrt
router, it's been putting out dhcpv6 solicits with no response for
roughly... 10 years.

It would be great if there was someone to call at FIOS about what gear
will be v6 capable. This router also happens to have my longest
running hurricane electric ipv6 tunnel.

Never did I imagine 10+ years ago that tunnel would still be in operation.

[1] http://esr.ibiblio.org/?p=4162
>
> -chris
>
>>
>> Thank you
>> jms
>>
>> On Mon, Jun 13, 2022 at 11:18 AM Nimrod Levy  wrote:
>>>
>>> Also, it doesn't seem to be enabled on ports that have static ipv4
>>>
>>> but progress is progress. we'll take it.
>>>
>>> Nimrod
>>>
>>>
>>> On Mon, Jun 13, 2022 at 11:17 AM Matthew Huff  wrote:

 Still no IPv6 in Westchester County, NY ☹



 Great sign though, maybe NY will get it eventually



 From: NANOG  On Behalf Of Joe 
 Loiacono
 Sent: Monday, June 13, 2022 10:55 AM
 To: nanog@nanog.org
 Subject: Re: Congrats to AS701



 FiOS from Maryland (anonymized):

 enp3s0: flags=4163  mtu 1500
 inet 192.168.1.164  netmask 255.255.255.0  broadcast 192.168.1.255
 inet6 fe80::b104:8f4d:e5b2:e13b  prefixlen 64  scopeid 0x20
 inet6 2600:4040:b27f:cb00:a9b1:5f59::  prefixlen 64  
 scopeid 0x0
 inet6 2600:4040:b27f:cb00:24a8:7b31::  prefixlen 64  
 scopeid 0x0
 inet6 2600:4040:b27f:cb00:e1b6:8b83::  prefixlen 64  
 scopeid 0x0
 ether d0:67:e5:23:ec:fe  txqueuelen 1000  (Ethernet)
 RX packets 2518066  bytes 1448982813 (1.4 GB)
 RX errors 0  dropped 0  overruns 0  frame 0
 TX packets 2157395  bytes 260073952 (260.0 MB)
 TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 a@b:~$ ping 2607:f8b0:4004:c09::6a
 PING 2607:f8b0:4004:c09::6a(2607:f8b0:4004:c09::6a) 56 data bytes
 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=1 ttl=59 time=24.0 ms
 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=2 ttl=59 time=17.6 ms
 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=3 ttl=59 time=20.4 ms
 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=4 ttl=59 time=23.4 ms
 ^C
 --- 2607:f8b0:4004:c09::6a ping statistics ---
 4 packets transmitted, 4 received, 0% packet loss, time 3004ms
 rtt min/avg/max/mdev = 17.618/21.351/23.983/2.555 ms



 On 6/12/2022 1:55 PM, Christopher Morrow wrote:





 On Sat, Jun 11, 2022 at 11:03 PM Darrel Lewis (darlewis) 
  wrote:

 I, for one, am having a hard time finding the proper words to express the 
 joy that I am feeling at this momentous moment!





 It's quite amazing, I think... that it's taken so long to get to 
 deployment you can actually see on the fios plant :)

 I'd note I can't see the below on my homestead, but I can at a relative's 
 (where the ifconfig data is from).

 I also can't tell if the upstream will PD a block to the downstream... and 
 the VZ CPE is 'not something I want to fiddle with',

 because everytime I have tried at my house I've just taken it out behind 
 the woodshed with a maul... and replaced it with

 something I CAN configure successfully. (plus.. don't want that TR 069 in 
 my home...)



 -chris



 -Darrel



 On Jun 11, 2022, at 7:05 PM, Christopher Morrow  
 wrote:

 



 Looks like FIOS customers may be getting ipv6 deployed toward them, 
 finally:

 ifconfig snippet from local machine:
 inet6 2600:4040:2001:2200:73d2:6bcc:1e6b:43a1  prefixlen 64  
 scopeid 0x0
   

Re: Serious Juniper Hardware EoL Announcements

[Dave Taht]

On Tue, Jun 14, 2022 at 9:44 AM Shawn L via NANOG  wrote:
>
> With the current shortages and lead times, I almost feel like I did back in 
> the beginning of my career ---
>
>
>
> Then it was "what can we do with what we can afford" now it's more like  
> "What can we do with what we have (or can actually get)"?

Like, working on better software...

>
>
> Shawn
>
> -Original Message-
> From: "Adam Thompson" 
> Sent: Tuesday, June 14, 2022 12:36pm
> To: "Mark Tinka" , "nanog@nanog.org" 
> Subject: RE: Serious Juniper Hardware EoL Announcements
>
> [Not specific to the Juniper EoLs...]
>
> I sort of agree with Mark:
>
> I've been sampling a fairly wide variety of sources in various parts of the 
> global supply chain, and my synthesis of what they're saying is that we 
> probably won't *consistently* have the ready availability of "stuff" (both 
> electronic and not) we had pre-pandemic, for the rest of my career 
> (10-15yrs), and maybe not in the lifetimes of anyone reading this today, 
> either.
>
> Whether those sources are accurate, their interpretation is accurate, my 
> synthesis is accurate, whether I'm listening to the right people in the first 
> place... all debatable. I sure hope the above conclusion is wrong.
>
> One possible upside: it might slow down the incessant upgrade hamster-wheel 
> we're all running on? Imagine having enough time to do your job thoroughly 
> and properly... Yes, I know I'm dreaming :-).
>
>
> Adam Thompson
> Consultant, Infrastructure Services
> MERLIN
> 100 - 135 Innovation Drive
> Winnipeg, MB R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> https://www.merlin.mb.ca
> Chat with me on Teams: athomp...@merlin.mb.ca
>
> > -Original Message-
> > From: NANOG  On Behalf
> > Of Mark Tinka
> > Sent: Tuesday, June 14, 2022 11:19 AM
> > To: nanog@nanog.org
> > Subject: Re: Serious Juniper Hardware EoL Announcements
> >
> >
> >
> > On 6/14/22 18:06, JASON BOTHE via NANOG wrote:
> >
> > > Saw this coming a mile away. With chips and technology progressing
> > despite ability to manufacture, I’m certain many are going to do this.
> >
> > All this will do is keep these boxes off the open market, which will
> > simply bump up open market prices, with no incentive for the majority
> > of
> > folk to buy directly from the OEM.
> >
> > I suspect supply chain will improve within the next 12 months, but
> > then
> > regress and hit a massive crunch from around Q4'23 onward. How long
> > for,
> > I can't say...
> >
> > Mark.



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Upstream bandwidth usage

[Dave Taht]

On Sat, Jun 11, 2022 at 1:22 PM Karsten Thomann via NANOG
 wrote:
>
> On Friday, 10 June 2022 10:15:15 CEST Chris Hills wrote:
> > On 10/06/2022 00:31, Mel Beckman wrote:
> > > Your point on asymmetrical technologies is excellent. But you may not be
> > > aware that residential optical fiber is also asymmetrical. For example,
> > > GPON, the latest ITU specified PON standard, and the most widely
> > > deployed, calls for a 2.4 Gbps downstream and a 1.25 Gbps upstream
> > > optical line rate.
> > Not all residential fiber is asymmetric. Nokia XGS-PON supports 9.953
> > Tx/Rx (e.g. LTF7226 transceiver).
> XGS-PON isn't Nokia specific and can be bought from many other vendors.
> Even as probably no one is deploying XG-PON in new deployments (10/2.5G), I
> don't believe ISP start selling symmetrical services to residential customers
> as a standard, even if the PON itself is symmetrical.
> I know you can get from many providers a symmetrical service on G-PON, but
> that is an option, not the default.
>
> Does anyone know the Asian market where they are using E-PON?
> After my very short search it seems they provide best effort up to 1G without
> any real plans...
>

My question is always: how are people using these technologies doing
queue management. I took apart one ONT so far, it supported RED and
hardware flow control, but if it were configured or not, couldn't
tell. Sonic (SF) had about 90ms of buffering in their upstream.


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Comcast Network Peer Survey on DSCP/ECN for L4S

[Dave Taht]

I would argue that question 9 needs an option of "Both".

Secondly, two additional good questions to ask would be: are the ECN
values presently being treated as RFC3168?

Are the ECN values being modified by any AQM implementations (WRED,
FQ_CODEL, etc) on any switch or router in transit?

Re: Aftermarket switches that were manufactured in any sort of quantity?

[Dave Taht]

I am mostly searching for switches that can have custom firmware on
them. The very long list of those
compatible with SONIC is here:

https://github.com/Azure/sonic-buildimage/tree/master/device

Re: [EXTERNAL] Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Tue, Jun 7, 2022 at 8:55 AM Livingood, Jason
 wrote:
>
> > I think peak demand should be flattening in the past year? There's
> only so much 4k video to consume, so many big games to download?
>
> I doubt it - demand continues to grow at a pretty normal year-over-year rate 
> and has been doing so for 25+ years. I don't see that sort of trajectory 
> changing.

Dennard scaling ended in 2006. The US birthrate is negative. Wage
growth is illusory. There are only 16 hours in a day where content can
be consumed, and time on the internet is at an all time high.

> JL
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Tue, Jun 7, 2022 at 8:24 AM Livingood, Jason via NANOG
 wrote:
>
> A related observation – years ago we gave cable modem bootfiles to a group of 
> customers that had no rate shaping according to their subscription and 
> compared that to existing customers (with an academic researcher). The 
> experiment group did not know of the change, so it could not influence their 
> behavior. We observed that peak demand generally hit a plateau that was well 
> below available capacity & this was driven by existing applications & 
> associated user behavior. There’s obviously a chicken-or-egg problem with 
> capacity & apps to use that capacity, but most ISPs raise end user speeds at 
> least annually and try to stay ahead of increases in peak demand.

I think peak demand should be flattening in the past year? There's
only so much 4k video to consume, so many big games to download?

My curve seems closer to a doubling of the average usage over 10
years. It would be really radical of me to
start yelling "peak bandwidth" a la peak oil without more study...

A very informal survey of those that had deployed higher rates on
mikrotik stuff at WISPAMERICA had all 5 of the people rolling their
eyes and saying avg downloads had gone from 2 to 3Mbit upon doubling
or more their allocated bandwidth, and they had no congestive issues
on their network peering.

There was also a technical limitation in the mikrotik deployment in
that they use very short queues by default (50 packets) for either the
fifo or (the common) SFQ deployments. Shapers were universally used by
this small group, and they were
unaware of the sideffects of such short queues.  I also took apart a
recent ubnt 60Ghz radio's behaviors, and that was FQ'd
and also with very short queues... and what looked like ack
synthesis... with no options to change the configuration. I am
thinking in part the lack of measured WISP "demand" for more bandwidth
is in part due to overly short (as opposed to bufferbloated) queues!

There's a really long  thread over here with the mikrotik userbase
going to town on fq_codel and cake:
https://forum.mikrotik.com/viewtopic.php?p=937633#p925485

>
>
> JL
>
>
>
> From: NANOG  on 
> behalf of Jim Troutman 
> Date: Monday, June 6, 2022 at 19:29
> To: Tony Wicks 
> Cc: "nanog@nanog.org" 
> Subject: Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers
>
>
>
> Some usage data:
>
>
>
> On a rural FTTX XGS-PON network with primarily 1Gig symmetric customers, I 
> see about 1.5mbit/customer average inbound across 7 days, peaks at about 
> 10mbit/customer, with 1 minute polling.  Zero congestion in middle mile, 
> transit or peering.
>
>



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Tue, Jun 7, 2022 at 7:47 AM Denis Fondras  wrote:
>
> Le Tue, Jun 07, 2022 at 08:12:07AM -0500, Mike Hammett a écrit :
> > Would it matter if it took 10 minutes or an hour?
> >
>
> Yes, it means the computer could be off for 50 minutes.
> Also everyone who had a connection reset when uploading a big file after 55
> minutes understands why it is good if it only would take 10 minutes.
>
> Peace of mind is under-rated :)

I have often wished rsync was the default file transfer protocol.

> >
> > What's the OneDrive rate limit?
> >
> >
> >
> >
> > -
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> > - Original Message -
> >
> > From: "Tony Wicks" 
> > To: nanog@nanog.org
> > Sent: Monday, June 6, 2022 5:36:13 PM
> > Subject: RE: FCC proposes higher speed goals (100/20 Mbps) for USF providers
> >
> >
> >
> > >This whole thread is about hypothetical futures, so it's not hard to 
> > >imagine downloads filling to available capacity.
> > >Mike
> >
> > So, a good example of how this capacity is used, In New Zealand we have a 
> > pretty broad fibre network covering most of the population. My niece asked 
> > me to share my backup copy of her wedding photo’s/video’s the other day. I 
> > have a 4Gb/s / 4Gb/s XGSPON connection and she’s got a 1Gb/s / 500Mb/s GPON 
> > connection. I simply dropped a copy of the 5.1G directory into a one drive 
> > folder and shared it, 10 minutes later (one drive is still limited in how 
> > fast you can upload) she had it all and she was very happy. With these 
> > speeds its not even a consideration to think about capacity, everything 
> > just works.



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Mon, Jun 6, 2022 at 3:38 PM Tony Wicks  wrote:
>
> >This whole thread is about hypothetical futures, so it's not hard to imagine 
> >downloads filling to available capacity.
>
> >Mike
>
>
>
> So, a good example of how this capacity is used, In New Zealand we have a 
> pretty broad fibre network covering most of the population. My niece asked me 
> to share my backup copy of her wedding photo’s/video’s the other day. I have 
> a 4Gb/s / 4Gb/s XGSPON connection and she’s got a 1Gb/s / 500Mb/s GPON 
> connection. I simply dropped a copy of the 5.1G directory into a one drive 
> folder and shared it, 10 minutes later (one drive is still limited in how 
> fast you can upload) she had it all and she was very happy. With these speeds 
> its not even a consideration to think about capacity, everything just works.

"New Zealand is approximately 268,838 sq km, while United States is
approximately 9,833,517 sq km, making United States 3,558% larger than
New Zealand. Meanwhile, the population of New Zealand is ~4.9 million
people (327.7 million more people live in United States)."

To finish up the math here, how much did NZ's fiber buildout cost?
-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Mon, Jun 6, 2022 at 7:46 AM Mike Hammett  wrote:
>
> "I find it sad that so many would argue for never needing anything
> more than we have today."

My principal argument is that we've made a huge mistake with buffering
in general, all fixed now by various RFCs and widely
available source code, and that if we focused on improving the routers
rather than digging more holes in the ground, the internet would
become vastly better, faster, and cheaper - faster.  If somehow I
could wave a wand and get everyone to reflash a junked router
to openwrt/dd-wrt/merlin/etc and configure SQM calls for moah
bandwidth would decrease. If somehow getting those RFCs mandated in
more RFPs, we'd also be making real progress.

Lower latency does not need more bandwidth it needs better bandwidth.

https://www.bitag.org/documents/BITAG_latency_explained.pdf

If further, folk would stop using oversize wifi channels and
co-ordinate spectrum, again with wifi chips that do the right thing
under contention
(example: https://blog.cerowrt.org/post/fq-codel-unifi6/), it would be
a better network

Or you can burn 10s of k per mile to massively overbuild a fiber
network, that doesn't solve the real last mile problem in wifi queue
management.

>
> Few to none are doing that. Upgrades are an organic part of the process. Some 
> places they're hard, but most places they're comparatively easy. Let's stop 
> putting the cart before the horse just to feel good about ourselves. That's 
> too expensive.


>
> "totally fail to provide the same to everyone."
>
> Why should that be desirable?
>
>
> "If we had moved to fibre everywhere then perhaps"

gbit fiber everywhere would actually work pretty well, as very few
pieces of gear can keep up with gbit fiber:

https://forum.openwrt.org/t/so-you-have-500mbps-1gbps-fiber-and-need-a-router-read-this-first/90305

I was really astonished at how few device at a recent conference could
do a gbit in both directions at the same time. None.

> Negative. DOCSIS works well enough. Modern DSL implementations are good 
> enough. Fixed wireless in many cases is good enough. Next gen satellite is 
> good enough.

... With good queue management. Starlink still has lousy queue
management. Got a blog post coming up A lot of fixed wireless,
notably ubnt and now mikrotik, have good queue management. Most DSL
doesn't.

>
>
>
> "If you build it they will come."
>
> So then build the hypothetical content that needs this?
>
>
> Gigabit download level service is available to enough (at least in the US) 
> that if such a downstream heavy service were on our doorstep, it would work 
> for most Americans. Once people got tired of being proven that you need such 
> forward-looking downstream capacity in the regulatory world, they just moved 
> to upstream and cried wolf there too. Yes, many services do have mildly 
> inadequate upstream, but certainly not anything to change the regulatory 
> environment over.

I really would, actually, at this depressing point... like regulators,
to mandate that ipv6 be deployed, and rfc7567 at every fast->slow
transition.
It would raise the bar for adaquate internet services over and above
lost cries for more bandwidth.

>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> 
> From: "Brandon Butterworth" 
> To: "Mike Hammett" 
> Cc: "Michael Thomas" , nanog@nanog.org
> Sent: Monday, June 6, 2022 9:31:13 AM
> Subject: Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers
>
> On Mon Jun 06, 2022 at 08:06:50AM -0500, Mike Hammett wrote:
> > "So what happens if the Next Big Thing..."
>
> I find it sad that so many would argue for never needing anything
> more than we have today. It's like why did we bother coming out of
> the trees, or the oceans even (yes Apple digital watches are a pretty
> neat idea).
>
> The non fibre installations we have today, while working for some,
> totally fail to provide the same to everyone. While fixing that
> globally should be a priority it should not be done in a manner
> that will require it all doing again in 10 years.
>
> Building in some headroom for growth makes sense, we're not talking
> lots it's only 10x ish to do gigabit ish, so within error margin.
>
> > I see this said a lot, but it doesn't really mean anything. We
> > are sufficiently close to whatever is likely to come that it
> > can come and bandwidths will have to catch up upon its launch.
>
> If we had moved to fibre everywhere then perhaps, but until
> then we face many decades trying to get that done. So if
> something comes up we may be stuck waiting. Stuff always
> comes up.
>
> When I started the BBC streaming we were told not to bother by
> ISPs, the quality was rubbish, the network couldn't handle it and
> never will. I did it anyway and the net grew but it was a long
> slow process with lots of screaming. It'd be nice to not have to
> wait so long next time 

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Mon, Jun 6, 2022 at 5:47 AM Masataka Ohta
 wrote:
>
> Dave Taht wrote:
>
> > Looking back 10 years, I was saying the same things, only then I felt
> > it was 25Mbit circa mike belshe's paper. So real bandwidth
> > requirements only doubling every decade might be a new equation to
> > think about...
>
> Required resolution of pictures is bounded by resolution of our
> eyes, which is fixed.
>
> For TVs at homes, IMHO, baseband 2k should be enough, quality of
> which may be better than highly compressed 4k.
>
> Masataka Ohta

Yep. And despite our best efforts, nobody can hear the difference
between 48khz/24 bit audio and 96khz/24 bit audio. The difference
between 16 bit and 24 bit audio can be heard... but not so much on
bluetooth earbuds! Attempts to make 10 channel audio more popular
(like Atmos) appeal to a very narrow market.

Belshe's paper on "more bandwidth doesn't matter (much) from 2008:
https://docs.google.com/a/chromium.org/viewer?a=v=sites=Y2hyb21pdW0ub3JnfGRldnxneDoxMzcyOWI1N2I4YzI3NzE2



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

On Fri, Jun 3, 2022 at 9:12 AM Masataka Ohta
 wrote:
>
> Livingood, Jason via NANOG wrote:
>
> > That shows up as increased user demand (usage), which means that the
> > CAGR will rise and get factored into future year projections.
>
> You should recognize that Moore's law has ended.
>
> Masataka Ohta

For a long time now...

I have had the opinion that we have reached the age of "peak
bandwidth", that nearly nobody's 4 person home needs more than 50Mbit
with good queue management. Certainly increasing upload
speeds dramatically (and making static IP addressing and saner
firewalling feasible) might shift some resources from the cloud, which
I'd like (anyone using tailscale here?), but despite
8k video (which nobody can discern), it's really hard to use up >
50Mbit for more than a second or three with current applications. Even
projected applications like VR, or adding other senses to the internet
like smell or taste, are not bandwidth intensive.

Looking back 10 years, I was saying the same things, only then I felt
it was 25Mbit circa mike belshe's paper. So real bandwidth
requirements only doubling every decade might be a new equation to
think about...

... check in with me again and wipe egg off my face in another decade.

-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Dave Taht]

re: 
https://www.fcc.gov/document/fcc-proposes-higher-speed-goals-small-rural-broadband-providers-0

On Thu, May 26, 2022 at 7:36 AM Livingood, Jason via NANOG
 wrote:
>
> > Latency is a limitation for things that are generally relatively low 
> > bandwidth (interactive audio, zoom, etc.).
> > Higher bandwidth won’t solve the latency problem
>
> +1
> IMO as we enter the 'post-gigabit era', an extra 1 Gbps to the home will 
> matter less than 100 ms or 500 ms lower working latency (optimally sub-50 ms, 
> if not sub-25 ms). The past is exclusively speed-focused -- the future will 
> be speed + working latency + reliability/resiliency + consistency of QoE + 
> security/protection + WiFi LAN quality.

I'd settle for the 100Mbit era having sub 25ms working latency. Which
we've been achieving in fq_codel, cake, and even pie, for 10 years.

I will file on this nprm, some variant of
https://docs.google.com/document/d/1FjRo9MNnVOLh733SNPNyqaR1IFee7Q5qbMrmW1PlPr8/edit

But I keep hoping more will sign on board. Perhaps finding a lawyer to
proof it. And I'm not sure what hook to use on this nprm out of my
existing evolving document without tieing myself to a chair with a
variety of calming drugs handy.

I'm an engineer, dang it, not a politician!
>
> Jason
>
>


-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Any sign of supply chain returning to normal?

[Dave Taht]

On Thu, May 19, 2022 at 11:01 AM Jason Biel  wrote:
>
> > The oem ain't gonna support the resold device either.
>
> Many vendors support resold gear through a recertification cost in order to 
> bring it back under a support contract.

In my world, support ends after 6 months. Period.

It's even worse than that. Mediatek, for example, provides a devkit to
new customers, still, based on the obsolete
LEDE-17 release of openwrt, e.g. 6+ year code. I recently pointed out
to a marketing manager pimping how wifi-7 was going to fix latency on
wifi in 3-4 years, how crappy the factory driver was, compared to
what's now in linux

https://blog.cerowrt.org/post/fq-codel-unifi6/

and asked when they were going to ship that instead, to a blank
stare... And yes, I know of several "new" customers for that chipset
that are using that obsolete code, too scared and incompentent to make
the jump to a more current OS.

If you think that's bad, qualcomm is worse, and I just established a
new record, I think, with truly ancient broadcom's openwrt based
devkit that just shipped with the "NEW" triband tp-link deco series...
https://www.tp-link.com/us/deco-mesh-wifi/product-family/deco-xe75/ -
I can hardly bring myself to talk to the sea of CVEs and incompetence
in there... you can start with them STILL shipping dnsmasq 2.62... and
linux 3.3.8.

I used to be really proud that openwrt was used by all these major
manufacturers, but I'd also thought that they'd
have been responsible enough to at least keep up with CVEs, and stay
within a few years of the mainline.

If you are wondering why WiFi-6 works so badly out of the box, or why
ipv6 is not rolling out, you don't have to look much further. The
really, really sad thing, is that the ODM in these cases, just slaps
the devkit and a fancy gui
on top of it, and ships the product, with no further support.

So I kind of view recycling routers, with newer software, as a great
way to clean up the present ecosystem. And if you  looked at the first
url I pasted above, with 4x more throughput, and 10x less latency, on
"obsolete", hw.

> On Thu, May 19, 2022 at 10:58 AM Dave Taht  wrote:
>>
>> On Thu, May 19, 2022 at 10:33 AM Jason Biel  wrote:
>> >
>> > Who's going to support that reflashed device? Certainly not the OEM vendor.
>>
>> The oem ain't gonna support the resold device either.
>>
>> Yes, arguably, someone or someones doing a value add would have to be
>> making money at it somehow.
>>
>> However, at least in my world, volunteers make the world round, still.
>> It would kind of suck,
>> I suppose, if someone unleashed a few hundred thousand reflashed
>> routers like the TIP openwifi
>> effort ( https://telecominfraproject.com/ ) seem to intent on doing...
>>
>> ... but if the OS is good enough to not need support, the impact is minimal.
>> >
>> > On Thu, May 19, 2022 at 10:30 AM Dave Taht  wrote:
>> >>
>> >> On Thu, May 19, 2022 at 9:07 AM NetEquity Sales  
>> >> wrote:
>> >> >
>> >> > As someone who works within the "secondary market" for networking 
>> >> > hardware, there is a ton of demand spilling over into the 
>> >> > "pre-owned/vendor refurbished" market.
>> >>
>> >> I just wish there were people putting in a value-add, like reflashing
>> >> with better software, first.
>> >>
>> >> >
>> >> > Market prices on pre-owned equipment are rapidly increasing in step 
>> >> > with increased demand and dwindling supply.
>> >> >
>> >> > Market prices on 1G - 10G switching products, wireless infrastructure 
>> >> > devices, etc have been rising precipitously. Even semi "legacy" stuff 
>> >> > going back 2-3 generations (EOL/EOS) from current gen have doubled, 
>> >> > tripled, even quadrupled in price.
>> >> >
>> >> > I've been involved in the hardware business for 20 years and the 
>> >> > current market landscape is unprecedented.
>> >> >
>> >> >
>> >> >
>> >> > Cory J. Andrews
>> >> > ++
>> >> > NetEquity.com (Formerly CiscoBuy.com)
>> >> > 4519 Northgate Court
>> >> > Sarasota, FL 34234
>> >> > ++
>> >> > TF/FAX 877.582.4726
>> >> > E - sa...@netequity.com
>> >> >
>> >> > On Thu, May 19, 2022, 9:48 AM Josh Luthman 
>> >> >  wrote:
>> >> >>
>> >> >> I'd bet it's cheaper and easier to quantify new hardware tha

Re: Any sign of supply chain returning to normal?

[Dave Taht]

On Thu, May 19, 2022 at 10:33 AM Jason Biel  wrote:
>
> Who's going to support that reflashed device? Certainly not the OEM vendor.

The oem ain't gonna support the resold device either.

Yes, arguably, someone or someones doing a value add would have to be
making money at it somehow.

However, at least in my world, volunteers make the world round, still.
It would kind of suck,
I suppose, if someone unleashed a few hundred thousand reflashed
routers like the TIP openwifi
effort ( https://telecominfraproject.com/ ) seem to intent on doing...

... but if the OS is good enough to not need support, the impact is minimal.
>
> On Thu, May 19, 2022 at 10:30 AM Dave Taht  wrote:
>>
>> On Thu, May 19, 2022 at 9:07 AM NetEquity Sales  wrote:
>> >
>> > As someone who works within the "secondary market" for networking 
>> > hardware, there is a ton of demand spilling over into the 
>> > "pre-owned/vendor refurbished" market.
>>
>> I just wish there were people putting in a value-add, like reflashing
>> with better software, first.
>>
>> >
>> > Market prices on pre-owned equipment are rapidly increasing in step with 
>> > increased demand and dwindling supply.
>> >
>> > Market prices on 1G - 10G switching products, wireless infrastructure 
>> > devices, etc have been rising precipitously. Even semi "legacy" stuff 
>> > going back 2-3 generations (EOL/EOS) from current gen have doubled, 
>> > tripled, even quadrupled in price.
>> >
>> > I've been involved in the hardware business for 20 years and the current 
>> > market landscape is unprecedented.
>> >
>> >
>> >
>> > Cory J. Andrews
>> > ++
>> > NetEquity.com (Formerly CiscoBuy.com)
>> > 4519 Northgate Court
>> > Sarasota, FL 34234
>> > ++
>> > TF/FAX 877.582.4726
>> > E - sa...@netequity.com
>> >
>> > On Thu, May 19, 2022, 9:48 AM Josh Luthman  
>> > wrote:
>> >>
>> >> I'd bet it's cheaper and easier to quantify new hardware than software.  
>> >> Labor was super expensive and now it is ready to implode.
>> >>
>> >> On Thu, May 19, 2022 at 9:27 AM Dave Taht  wrote:
>> >>>
>> >>> As I've been saying for a while, instead of buying new kit, perhaps we
>> >>> could spend some time on getting better software onto our older kit?
>> >>> Getting stuff to multiplex better, be more reliable, last longer?
>> >>>
>> >>> It isn't just me wanting to upgrade a billion+ routers with existing
>> >>> crappy software to openwrt, is it?
>> >>>
>> >>> https://docs.google.com/document/d/1T21on7g1MqQZoK91epUdxLYFGdtyLRgBat0VXoC9e3I/edit
>>
>>
>>
>> --
>> FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
>> Dave Täht CEO, TekLibre, LLC
>
>
>
> --
> Jason



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Any sign of supply chain returning to normal?

[Dave Taht]

On Thu, May 19, 2022 at 9:07 AM NetEquity Sales  wrote:
>
> As someone who works within the "secondary market" for networking hardware, 
> there is a ton of demand spilling over into the "pre-owned/vendor 
> refurbished" market.

I just wish there were people putting in a value-add, like reflashing
with better software, first.

>
> Market prices on pre-owned equipment are rapidly increasing in step with 
> increased demand and dwindling supply.
>
> Market prices on 1G - 10G switching products, wireless infrastructure 
> devices, etc have been rising precipitously. Even semi "legacy" stuff going 
> back 2-3 generations (EOL/EOS) from current gen have doubled, tripled, even 
> quadrupled in price.
>
> I've been involved in the hardware business for 20 years and the current 
> market landscape is unprecedented.
>
>
>
> Cory J. Andrews
> ++
> NetEquity.com (Formerly CiscoBuy.com)
> 4519 Northgate Court
> Sarasota, FL 34234
> ++
> TF/FAX 877.582.4726
> E - sa...@netequity.com
>
> On Thu, May 19, 2022, 9:48 AM Josh Luthman  
> wrote:
>>
>> I'd bet it's cheaper and easier to quantify new hardware than software.  
>> Labor was super expensive and now it is ready to implode.
>>
>> On Thu, May 19, 2022 at 9:27 AM Dave Taht  wrote:
>>>
>>> As I've been saying for a while, instead of buying new kit, perhaps we
>>> could spend some time on getting better software onto our older kit?
>>> Getting stuff to multiplex better, be more reliable, last longer?
>>>
>>> It isn't just me wanting to upgrade a billion+ routers with existing
>>> crappy software to openwrt, is it?
>>>
>>> https://docs.google.com/document/d/1T21on7g1MqQZoK91epUdxLYFGdtyLRgBat0VXoC9e3I/edit



-- 
FQ World Domination pending: https://blog.cerowrt.org/post/state_of_fq_codel/
Dave Täht CEO, TekLibre, LLC

Re: Any sign of supply chain returning to normal?

[Dave Taht]

As I've been saying for a while, instead of buying new kit, perhaps we
could spend some time on getting better software onto our older kit?
Getting stuff to multiplex better, be more reliable, last longer?

It isn't just me wanting to upgrade a billion+ routers with existing
crappy software to openwrt, is it?

https://docs.google.com/document/d/1T21on7g1MqQZoK91epUdxLYFGdtyLRgBat0VXoC9e3I/edit

Re: are underwater routers a thing?

[Dave Taht]

On Sat, Apr 16, 2022 at 1:57 PM Mark Tinka  wrote:
>
>
>
> On 3/18/22 06:21, Joel Jaeggli wrote:
>
> >
> > The mean depth of the worlds oceans is around ~3700 meters below MSL
> > which means most service calls involve deploying to the proximate
> > location of the fault, fishing around for a while and then carefully
> > re-laying  several kilometers of cable on a splice has been made.
> > which typically takes weeks.
>
> And lots, and lots of $$.

Just arrange for a volcano to go off, and splice then.

>
> Mark.



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

gitlab contact?

[Dave Taht]

Most cloud operations websites are kept internal. gitlab's is not,
which is pretty cool. In looking over this issue, today:

https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6768

They are tracking tcp syn retransmits, but not drops or other
congestion control related info. And
also using:

sysctl net.ipv4.tcp_notsent_lowat=4294967295;

where we've been getting good results with that set as low as 32k.
Anyone know anyone at gitlab?


-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: V4 via V6 and IGP routing protocols

[Dave Taht]

The other question for this list I'd basically had was this:

> https://datatracker.ietf.org/doc/html/draft-ietf-babel-v4viav6

> Please let me know if you feel that it should be possible to
> completely disable v4-via-v6 even on newer kernels, and whether you
> feel that v4-via-v6 should be disabled by default.  (The "Security
> Considerations" section of the draft cited above might be
> interesting.)"

Re: V4 via V6 and IGP routing protocols

[Dave Taht]

On Mon, Apr 4, 2022 at 5:16 AM Mark Tinka  wrote:
>
>
>
> On 4/4/22 03:06, Dave Taht wrote:
>
> > I'm actually not here to start a debate... happy to learn that the v4
> > over v6 feature I'm
> > playing with actually exists in another protocol, mainly. I'm
> > critically dependent on
> > source specific routing, also, so I am hoping there's an isis or ospf
> > that can do
> > what I need, or now that I have more routers with enough memory, switch back
> > to an ibgp.
>
> Are MPLS or SR too heavy a bat?

MPLS was not an option at the time. It might become one. Don't know
diddly about the current state of SR.

What happened in the mesh wifi market circa 2015 is that eero
perfected a superset of (layer 2) 802.11s that scaled well enough (3
hops max) to create a market. the eero 5 was great, the 6 a step
backwards, I'm rather impressed by their new 6E product... on paper.

It was bridging ethernet multicast to wifi multicast that was a real
killer to performance then. Various solutions have appeared to lighten
that load, arp proxying (don't know about ND), and a version of mdns
that can upgrade to unicast, and fq_codel for wifi is out there on a
lot of products now, like the ath9k, ath10k, pretty much all of intel
and mediateks chips, iOS and OSX.

Trying to revisit a few assumptions as to where the real problems are now.

>
>
> > yes, and for smaller networks that are interconnecting, bgp can be
> > too heavyweight also.
>
> I know tons of small networks using Mikrotik to run their BGP backbone.
> Seems to work :-).

BGP may end up what I switch to, but I note I have other requirements
than just a robust routing protocol.

middlebox fq_codel solutions like preseem's are now pretty common in
that market, but the rightest answer was to move good queue management
to every fast->slow transition as close to the hw as possible
(rfc7567).

Mikrotik only recently added fq_codel and cake support. There's some
marvelous pictures of how well that's working if you login here:

https://forum.mikrotik.com/viewtopic.php?t=179307#p885613


Regrettably their IPv6 support was broken entirely when I last checked
a month or two back, I don't think they have the fq_codel native for
wifi code operational either. Now that ISP rates are so much better
than before, WiFi has become the bloated bottleneck for many. There's
a real cliff for range, I was recently at a large apartment building
where the usable range was under 8 feet due to all the APs competing.
Sells more APs though...

One of my holy grails from a working combination of fq_codel for
wifi/ethernet meshes and a good routing protocol was a workable RTT,
as opposed to loss based, metric. I was kind of agnostic as to the
underlying routing protocol, but I really wanted it to fit in memory
and scale well past 3 hops.  Since then I mostly gave up on homenet's
ideas, am revisiting older ones (like ospf and RPL) to see what
progress was made there.

It's a little OT for nanog, aside from a goal being better e2e
connectivity and simplicity. I'll take another look at the current
state of ospfv3, isis, and mpls.

> Mark.



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC

Re: V4 via V6 and IGP routing protocols

[Dave Taht]

I'm actually not here to start a debate... happy to learn that the v4
over v6 feature I'm
playing with actually exists in another protocol, mainly. I'm
critically dependent on
source specific routing, also, so I am hoping there's an isis or ospf
that can do
what I need, or now that I have more routers with enough memory, switch back
to an ibgp.

Is there a lightweight bgp client worth fiddling with? gobgp looked
interesting. Presently
I run bird in some places

On Sun, Apr 3, 2022 at 6:49 AM Masataka Ohta
 wrote:
>
> Dave Taht wrote:
>
> > Periodically I still do some work on routing protocols. 12? years ago I had 
> > kind
> > of given up on ospf and isis, and picked the babel protocol as an IGP
> > for meshy networks because I felt link-state had gone as far as it
> > could and somehow unifying BGP DV with an IGP that was also DV
> > (distance vector) seemed like a path forward.
>
> As DV depends other routers to choose the best path from
> several candidates updated asynchronously, which means
> it is against the E2E principle and decisions by other
> routers are delayed a lot to wait all the candidates
> are updated, it is hopeless.

I like to think babel solved most of the problems that RIP had, and
while an optimal state is slow to arrive in babel, a working state is
immediate, it's loop free, and it had a rtt metric.

>
> OTOH, LS only allows routers distribute the current most link
> states instantaneously and let end systems of individual
> routers compute the best path, LS converges quickly.

Ages ago, I'd written a tool to stress out various igps in a scenario where lots
of routes were coming and going, called rtod. It pretty much broke all the
daemons and protocols I'd had available to me at fairly low levels of churn.

https://github.com/dtaht/rtod

> BGP is DV because there is no way to describe policies of
> various domains and, even if it were possible, most, if
> not all, domains do not want to publish their policies
> in full detail.

yes, and for smaller networks that are interconnecting, bgp can be
too heavyweight also.

>
> > My question for this list is basically, has anyone noticed or fiddled
> > with babel?
>
> No.
>
> Masataka Ohta



-- 
I tried to build a better future, a few times:
https://wayforward.archive.org/?site=https%3A%2F%2Fwww.icei.org

Dave Täht CEO, TekLibre, LLC