Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Dent wrote:

The explanation I got (which seems fair) from someone was that they 
only way to roll a new transport was to squat on some existing stuff 
that would make it through firewalls.

If there's clearly a two-way flow occurring,


Clearly, it should be DOS amplification.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

On Wed, Feb 19, 2020 at 1:28 AM Daniel Dent
 wrote:
>
> On 2020-02-18 4:25 p.m., Jared Mauch wrote:
> > The members of the QUIC WG at IETF have not thought this was a problem as 
> > they don't observe it across the board.
> >
> > The cost for payloads with QUIC is much higher on the CPU side vs TCP as 
> > well - this is also not considered an issue either.

Jared, you mean: "Cost on the server" here not "cost on the router".
I think, and I ask because at least some of Daniel's note implies, to
my reading, that 'on the router' was your concern?

> There's plenty of room for system call/interface improvements and
> hardware acceleration in UDP stacks, both of which should help with CPU
> concerns. Now that UDP may represent a significant portion of internet
> traffic, it will be easier for the necessary engineering expense to be
> justified.
> > The explanation I got (which seems fair) from someone was that they only 
> > way to roll a new transport was to squat on some existing stuff that would 
> > make it through firewalls.
> If there's clearly a two-way flow occurring, i.e. as is the case with a

2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.

-chris

Re: QUIC traffic throttled on AT residential

[Daniel Dent]

On 2020-02-18 4:25 p.m., Jared Mauch wrote:

The members of the QUIC WG at IETF have not thought this was a problem as they 
don't observe it across the board.

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - 
this is also not considered an issue either.
There's plenty of room for system call/interface improvements and 
hardware acceleration in UDP stacks, both of which should help with CPU 
concerns. Now that UDP may represent a significant portion of internet 
traffic, it will be easier for the necessary engineering expense to be 
justified.

The explanation I got (which seems fair) from someone was that they only way to 
roll a new transport was to squat on some existing stuff that would make it 
through firewalls.
If there's clearly a two-way flow occurring, i.e. as is the case with a 
stream of QUIC traffic, that is a clearly distinguishable case from a 
DoS where the recipient is going to be dropping traffic. While this may 
be considerably harder to implement at scale than simply throttling UDP 
indiscriminately, hopefully there can be enough user suffering that AT 
feels compelled to do the right thing and allow the internet to continue 
to develop new protocols. Not everything fits neatly into a 
circuit-switched head-of-line-blocking request-response/HTTP paradigm.

We have an internet that is largely fixed around running NATs and TCP and UDP 
only these days. I for one find this sad and don't see a good way forward on 
either side.


Hopefully situations like this where Google swings their Chrome hammer 
for good instead of for evil can help get us there... now if we can just 
get those 100 gigabyte game downloads to get delivered over UDP too...


---

Daniel Dent

https://www.danieldent.com

Re: QUIC traffic throttled on AT residential

[Töma Gavrichenkov]

Peace,

On Wed, Feb 19, 2020 at 7:49 AM Daniel Sterling
 wrote:

> May I naively ask if Google staff have considered scrapping using UDP
> and instead proposing a new, first-class transport protocol that OSes
> can implement on top of IP?

The IETF WG did, at some point.  The opinion overall I think is that
this would probably bring in even more troubles both to the protocol
(e.g. in plenty of networks which do not allow any IP proto except 1,
6 and 17) and to networks (we have things like RFC 8086 for a reason).
There might've been other reasons.

--
Töma

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Michael Brown wrote:


Blocking a (for you) undesirable option when an established fallback
exists is a much better end user experience than introducing breakage
into that option


Are you saying AT should block UDP entirely?

Damian Menscher via NANOG wrote:

> As much as I would on principle rather not stick to a legacy, TCP-only
> home network --

Never stick to UDP.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 12:51 AM Damian Menscher  wrote:
> [snip impressive debugging story]

lol fair. I didn't umm mean to just brag -- my point was that:
generally available SoHo internet is worse than mobile networks esp.
for UDP traffic!

> Rather than hobble your home network to work around a misbehaving ISP, have 
> you considered simply getting a new ISP?

I could get a leased line, sure, or route all my traffic through a
UDP/TCP VPN -- but the average user won't and can't, no.

-- Dan

Re: QUIC traffic throttled on AT residential

[Damian Menscher via NANOG]

On Tue, Feb 18, 2020 at 8:48 PM Daniel Sterling 
wrote:

[snip impressive debugging story]

As much as I would on principle rather not stick to a legacy, TCP-only
> home network --
>
> I can say that right now, my home internet, blocking UDP 443, and
> making tons of insecure DNS queries -- is the most stable, fastest,
> most usable and enjoyable home internet I've ever had. And my users
> agree -- they no longer turn off wifi.
>

Rather than hobble your home network to work around a misbehaving ISP, have
you considered simply getting a new ISP?

Damian

Re: QUIC traffic throttled on AT residential

[Mark Andrews]

> On 19 Feb 2020, at 15:47, Daniel Sterling  wrote:
> 
> On Tue, Feb 18, 2020 at 8:05 PM Michael Brown  wrote:
>> Blocking a (for you) undesirable option when an established fallback
>> exists is a much better end user experience than introducing breakage
>> into that option
> 
>> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
>> terrible slowdowns due to packet loss when it broke
> 
> All the +1s.
> 
> 
> I have one goal for my home internet: it should work better than my cell 
> phone!
> 
> In our house, "screen time" is "all the time". Everyone is on their
> phone non-stop. So you'd think getting fiber, installing UBNT APs and
> swapping out AT's CPE for a Core i5 linux box would provide a better
> internet experience than a tree-obstructed cell tower a mile or two
> down the road.
> 
> But you'd be wrong.
> 
> 
> Everyone in the house was, on a daily basis, turning off wifi in favor
> of AT LTE!
> 
> Why was everyone switching off wifi? I couldn't blame them -- I was
> toggling it on and off myself to get the occasional website or IM
> conversation to load. Why was my network broken?? How was it possible
> that a fairly high-latency mobile connection could provide a better
> experience than 802.11ac to an AP in the same room backed by a gigabit
> PON?
> 
> 
> I banged against this for *years*. I punted on using my own router and
> tried just AT's CPE (reset to factory defaults). That does work
> decently, but there are some maddening quirks, not least of which is
> insanely high jitter.
> 
> I tried SoHo devices running vendor stock firmware driving hardware
> NAT. Those also work well -- until they inevitably crash.
> 
> 
> I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
> running upstream kernels; downstream kernels; I tried custom patches.
> I tried HFSC and CoDel; I compiled iproute2 so I could have some
> tc_cake.
> 
> 
> On the link-layer (wifi) side I tried one AP; two APs; three APs. I
> tested any number of combinations of SSID name, channel frequency and
> width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
> AP; in desperation I even tried dedicating one AP and an entire 5ghz
> frequency range for just one phone.
> 
> But nothing mattered until I finally figured it out:
> 
> It was DNS. Of course it was DNS. It's always DNS.
> 
> 
> When DNS was solid and fast, everything else fell into place. Toggling
> wifi worked because it was the same as re-querying DNS! And the DNS
> service on mobile works well -- better than the average CPE.
> 
> *** I cannot stress this enough. No manner of tuning or tweaking to my
> home network stopped users from fleeing it, until I had solid DNS. ***
> 
> 
> For fast DNS, you of course need fast UDP. And, as we've empirically
> discovered, well-behaved UDP is by no means guaranteed on residential
> connections.
> 
> It turns out dnsmasq has a couple of tunables that can make a huge
> difference to home internet DNS performance. First, you need to be
> querying the DNS servers AT tells you to via DHCP. They're the least
> likely to be throttled, unfortunately. But I've found even that alone
> isn't enough.
> 
> You need to set dnsmasq's "query-port" option. By default it's random
> to protect against CVE-2008-1447, but apparently sending a ton of
> random-source-port UDP traffic does not impress the AT network flow
> control systems, and your DNS traffic becomes unbearably slow (or is
> simply dropped entirely)

If dnsmasq supports DNS COOKIE turn it on.  DNS COOKIE provides protection
against CVE-2008-1447 provides the other end supports DNS COOKIE without
having to play games with ports.

> AT gives you two DNS servers via DHCP. You can query more --
> 8.8.8.8, 4.2.2.2, 2606:4700:4700:: -- but if you do, you'll want
> to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
> query to every server on your list and use whatever comes back first.
> If you've angered the UDP flow restrictor, no matter -- with luck at
> least one of your packets is going to a server that's up and not
> throttled or overloaded!
> 
> 
> Of course DNS is just the beginning -- you still need a proper gateway
> device with a good NAT stack and/or firewall; you still need a strong
> wifi signal; you still need tc_cake so everyone can watch Netflix at
> the same time.
> 
> But DNS is the *core*. Nothing works well until DNS works well. That
> means nothing works well unless UDP works well. And if I have learned
> anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
> Not. Work. Well.
> 
> 
> Enter QUIC. It may be the perfect transport-layer protocol; but by
> putting it on top of UDP it's hobbled. It breaks extant v4 internet in
> a way that nothing else we've yet seen does -- it takes what would be
> your TCP traffic and gives it inconsistent and intermittently poor
> performance. Maybe it's sometimes fast. Maybe it is. But I can tell
> you, it sometimes Is Very Much Not So.
> 
> 
> As 

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 8:05 PM Michael Brown  wrote:
> Blocking a (for you) undesirable option when an established fallback
> exists is a much better end user experience than introducing breakage
> into that option

> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
> terrible slowdowns due to packet loss when it broke

All the +1s.


I have one goal for my home internet: it should work better than my cell phone!

In our house, "screen time" is "all the time". Everyone is on their
phone non-stop. So you'd think getting fiber, installing UBNT APs and
swapping out AT's CPE for a Core i5 linux box would provide a better
internet experience than a tree-obstructed cell tower a mile or two
down the road.

But you'd be wrong.


Everyone in the house was, on a daily basis, turning off wifi in favor
of AT LTE!

Why was everyone switching off wifi? I couldn't blame them -- I was
toggling it on and off myself to get the occasional website or IM
conversation to load. Why was my network broken?? How was it possible
that a fairly high-latency mobile connection could provide a better
experience than 802.11ac to an AP in the same room backed by a gigabit
PON?


I banged against this for *years*. I punted on using my own router and
tried just AT's CPE (reset to factory defaults). That does work
decently, but there are some maddening quirks, not least of which is
insanely high jitter.

I tried SoHo devices running vendor stock firmware driving hardware
NAT. Those also work well -- until they inevitably crash.


I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
running upstream kernels; downstream kernels; I tried custom patches.
I tried HFSC and CoDel; I compiled iproute2 so I could have some
tc_cake.


On the link-layer (wifi) side I tried one AP; two APs; three APs. I
tested any number of combinations of SSID name, channel frequency and
width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
AP; in desperation I even tried dedicating one AP and an entire 5ghz
frequency range for just one phone.

But nothing mattered until I finally figured it out:

It was DNS. Of course it was DNS. It's always DNS.


When DNS was solid and fast, everything else fell into place. Toggling
wifi worked because it was the same as re-querying DNS! And the DNS
service on mobile works well -- better than the average CPE.

*** I cannot stress this enough. No manner of tuning or tweaking to my
home network stopped users from fleeing it, until I had solid DNS. ***


For fast DNS, you of course need fast UDP. And, as we've empirically
discovered, well-behaved UDP is by no means guaranteed on residential
connections.

It turns out dnsmasq has a couple of tunables that can make a huge
difference to home internet DNS performance. First, you need to be
querying the DNS servers AT tells you to via DHCP. They're the least
likely to be throttled, unfortunately. But I've found even that alone
isn't enough.

You need to set dnsmasq's "query-port" option. By default it's random
to protect against CVE-2008-1447, but apparently sending a ton of
random-source-port UDP traffic does not impress the AT network flow
control systems, and your DNS traffic becomes unbearably slow (or is
simply dropped entirely).


AT gives you two DNS servers via DHCP. You can query more --
8.8.8.8, 4.2.2.2, 2606:4700:4700:: -- but if you do, you'll want
to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
query to every server on your list and use whatever comes back first.
If you've angered the UDP flow restrictor, no matter -- with luck at
least one of your packets is going to a server that's up and not
throttled or overloaded!


Of course DNS is just the beginning -- you still need a proper gateway
device with a good NAT stack and/or firewall; you still need a strong
wifi signal; you still need tc_cake so everyone can watch Netflix at
the same time.

But DNS is the *core*. Nothing works well until DNS works well. That
means nothing works well unless UDP works well. And if I have learned
anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
Not. Work. Well.


Enter QUIC. It may be the perfect transport-layer protocol; but by
putting it on top of UDP it's hobbled. It breaks extant v4 internet in
a way that nothing else we've yet seen does -- it takes what would be
your TCP traffic and gives it inconsistent and intermittently poor
performance. Maybe it's sometimes fast. Maybe it is. But I can tell
you, it sometimes Is Very Much Not So.


As much as I would on principle rather not stick to a legacy, TCP-only
home network --

I can say that right now, my home internet, blocking UDP 443, and
making tons of insecure DNS queries -- is the most stable, fastest,
most usable and enjoyable home internet I've ever had. And my users
agree -- they no longer turn off wifi.


May I naively ask if Google staff have considered scrapping using UDP
and instead proposing a new, first-class transport protocol 

Re: QUIC traffic throttled on AT residential

[nanog-list]

On 2020-02-18 4:32 p.m., Michael Brown wrote:

With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
falls back to IPv4, everybody's happy.


The IPv6 deployment landscape might look considerably better if browser 
developers were instead to work together to co-ordinate an "unhappy 
eyeballs" algorithm that was rolled out over time, gradually degrading 
performance for IPv4-only connectivity situations. Now that 
non-evergreen browsers are pretty much dead, this might even be a 
realistic proposal, especially if it were done gradually enough. Google 
announcing a ranking penalty for IPv4-only sites wouldn't hurt either.


A while after implementing "unhappy eyeballs", once the performance 
penalties are severe enough that IPv6 adoption is high, the algorithm 
could be further adjusted to gradually impose penalties on sites that 
dual-stack instead of going v6-only. Growing prevalence of IPv6-only 
sites is probably the only thing that will get a lot of access networks 
to support v6.


The problem with happy eyeballs is that it works too well: excellent 
backwards compatibility reduces or eliminates the incentives for progress.


Happy eyeballs was necessary for it be realistic for sites to deploy 
IPv6 support, but now it holds us back.


--

Daniel Dent

https://www.danieldent.com

Re: QUIC traffic throttled on AT residential

[Michael Brown]

On 2020-02-18 7:07 p.m., Ross Tajvar wrote:
> Are you suggesting that ATT block all QUIC across their network?
Blocking a (for you) undesirable option when an established fallback
exists is a much better end user experience than introducing breakage
into that option

When you throttle or subtly break things you get:

On 2020-02-18 7:12 p.m., Daniel Sterling wrote:
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT ipv4 residential connection (and a web search
> suggests I'm not alone).

Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
terrible slowdowns due to packet loss when it broke

Or: some AT customers cannot connect to our customers due to IPv6
HTTPS interception: https://meta.discourse.org/t/-/140769/3

Or (probably the same problem):
https://tutanota.com/blog/posts/att-blocks-tutanota/

With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
falls back to IPv4, everybody's happy.

Re: QUIC traffic throttled on AT residential

[Dan Wing]

Yes, other than AT increasing their permitted incoming UDP traffic -- the 
easiest thing AT can do -- AT could ask the vendor of their flow 
restricting device to use bi-directional UDP traffic on same 5-tuple to 
indicate "desire to receive", rather than solely examining incoming UDP traffic 
as they are doing today; this is not easy but also not impossible.

While it is true Youtube/Google could treat AT connections as 'bad' 
and force TCP, but I am sure Youtube/Google is already gathering those 
statistics and already aware that AT is throttling.  For all we know, you and 
the others noticing the issue have fallen into the pit of A/B testers checking 
for their current throttling, and others aren't being throttled.  Perhaps it's 
everyone, the tests are not well described or well announced.  Youtube/Google 
is hoping customers complain to AT so that AT removes or improves the flow 
restrictor, because otherwise AT customers won't be able to get QUIC.  
Similarly, AT could protect their users from AT's own rate limiting by 
blocking QUIC towards major servers that support QUIC but such blocking becomes 
problematic as QUIC rolls out beyond Google to Cloudflare and elsewhere.

This tussle has similarities to IPv6 vs IPv4.

-d


> On Feb 18, 2020, at 4:00 PM, Ca By  wrote:
> 
> 
> 
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling  > wrote:
> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
> 
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
> 
> I assume traffic is being throttled on AT's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
> 
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
> 
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
> 
> May I naively ask if Google staff are working with AT to address this?
> 
> Sounds like you found the answer, ATT just needs to scale up your rejection 
> approach that is proven to work well. 
> 
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they 
> would be mixing with bad company in the pool of ipv4 udp traffic ... but they 
> have reasons. 
> 
> I am not a fan of quic or any udp traffic. My suggestion was that Google use 
> an new L4 instead of UDP, but that was too hard for the Googlers. 
> 
> So here we are.
> 
> Just say no to udp. 
> 
> 
> 
> 
> -- Dan

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 7:20 PM Dan Wing  wrote:
> For all we know, you and the others noticing the issue have fallen into the 
> pit of A/B testers checking for their current throttling, and others aren't 
> being throttled.

Ouch, I hope not -- do A/B tests that result in extreme performance
degradation continue indefinitely ?

> Youtube/Google is hoping customers complain to AT

It seems like this is one area where AT has no reason to care
(unless management decides to get involved). The default stance from
support would be, I assume: "Google is slow for you? Well obviously it
can't be Google's fault. Have a new NAT box"

I would bet dollars to donuts that if Google took a proper look at
their stats they would see this issue manifesting as users simply not
using services that have been QUIC-holed.

-- Dan

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

The members of the QUIC WG at IETF have not thought this was a problem as they 
don't observe it across the board. 

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - 
this is also not considered an issue either. 

The explanation I got (which seems fair) from someone was that they only way to 
roll a new transport was to squat on some existing stuff that would make it 
through firewalls. 

We have an internet that is largely fixed around running NATs and TCP and UDP 
only these days. I for one find this sad and don't see a good way forward on 
either side. 

Sent from my iCar

> On Feb 18, 2020, at 7:13 PM, Daniel Sterling  
> wrote:
> 
> 9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar  wrote:
>> 
>> Are you suggesting that ATT block all QUIC across their network?
> 
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT ipv4 residential connection (and a web search
> suggests I'm not alone).

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar  wrote:
>
> Are you suggesting that ATT block all QUIC across their network?

One might argue they already *are* doing so; QUIC is essentially
unusable on my AT ipv4 residential connection (and a web search
suggests I'm not alone).

Re: QUIC traffic throttled on AT residential

[Ross Tajvar]

Are you suggesting that ATT block all QUIC across their network?

On Tue, Feb 18, 2020, 7:02 PM Ca By  wrote:

>
>
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
> wrote:
>
>> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
>> from google (esp. youtube) becomes very slow after a time.
>>
>> This especially occurs with ipv4 connections. I'm not the only one to
>> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
>> notes the issue.
>>
>> I assume traffic is being throttled on AT's side. And it's not done
>> with their CPE; I'm bypassing their NAT box and connecting my laptop
>> directly to the ONT.
>>
>> A quick google search shows people are aware that QUIC can look like
>> DoS traffic -- but how widely known is this problem? It may become
>> worse if / when sites transition to HTTP/3
>>
>> For now I reject UDP 443 on the client firewall. Applications using
>> QUIC client libraries then fallback to TCP. This works well and I have
>> no issues with latency or throughput when using TCP.
>>
>> May I naively ask if Google staff are working with AT to address this?
>
>
> Sounds like you found the answer, ATT just needs to scale up your
> rejection approach that is proven to work well.
>
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
> would be mixing with bad company in the pool of ipv4 udp traffic ... but
> they have reasons.
>
> I am not a fan of quic or any udp traffic. My suggestion was that Google
> use an new L4 instead of UDP, but that was too hard for the Googlers.
>
> So here we are.
>
> Just say no to udp.
>
>
>
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Ca By]

On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
wrote:

> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
>
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
>
> I assume traffic is being throttled on AT's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
>
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
>
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
>
> May I naively ask if Google staff are working with AT to address this?


Sounds like you found the answer, ATT just needs to scale up your rejection
approach that is proven to work well.

Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
would be mixing with bad company in the pool of ipv4 udp traffic ... but
they have reasons.

I am not a fan of quic or any udp traffic. My suggestion was that Google
use an new L4 instead of UDP, but that was too hard for the Googlers.

So here we are.

Just say no to udp.



>
> -- Dan
>

QUIC traffic throttled on AT residential

[Daniel Sterling]

I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
from google (esp. youtube) becomes very slow after a time.

This especially occurs with ipv4 connections. I'm not the only one to
notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
notes the issue.

I assume traffic is being throttled on AT's side. And it's not done
with their CPE; I'm bypassing their NAT box and connecting my laptop
directly to the ONT.

A quick google search shows people are aware that QUIC can look like
DoS traffic -- but how widely known is this problem? It may become
worse if / when sites transition to HTTP/3

For now I reject UDP 443 on the client firewall. Applications using
QUIC client libraries then fallback to TCP. This works well and I have
no issues with latency or throughput when using TCP.

May I naively ask if Google staff are working with AT to address this?

-- Dan

Re: puertorico internet exchange

[Mehmet Akcin]

Hey there everyone!

Puerto Rico IX is growing stronger each day.

This is the last utilization,

https://ibb.co/2hpp1CT

Now next question is how can we get more caches? If you are experienced
running small isps snd want to help please join our discussion @
www.puertoricoix.net

Mwhmwt

On Fri, Dec 13, 2019 at 18:04 Mehmet Akcin  wrote:

> Hey there
>
> as the first bits started flowing through PRIX, I wanted to give a few
> exciting updates.
>
> Current peer lists can be found https://www.peeringdb.com/ix/2353
>
> One of the major social media companies who have a presence in PR has
> committed to join PRIX.
>
> We have also started a slack channel to have better communication with
> onsite (in PR) people and remote folks to collaborate. You can join the
> slack
> https://join.slack.com/t/puertoricoix/shared_invite/enQtODcyMTY4ODEzMTI0LTc0NzgzNjIwNjI4MDFkNzY5NzFkNGUyNGY3NDhjOTNkZWM1Mjc4NmEwMTg1NWRkODcyZTY3NmJmOTEwZTA4MTU
>
> not much traffic yet but here is an attached traffic exchange graph.
> https://ibb.co/vBS6N30
>
> thanks, everyone for your help
>
> Mehmet
>
>
> --
Mehmet
+1-424-298-1903

RE: akamai yesterday - what in the world was that

[Darrin Veit via NANOG]

Some clarifications on Xbox One behavior as it pertains to game updates, 
standby, and external hard drives:

If an Xbox is set to 'Instant On' mode and it has settings configured to keep 
the console OS and games/apps updated, the Xbox will check for updates during 
an overnight maintenance window when it is in standby (adjusted to local time) 
and automatically download/apply those updates. If an Xbox customer launches a 
game that has a content update that is required that hasn't yet been installed, 
the customer will receive a prompt that the game needs an update and the 
customer can then decide if they want to take a foreground content update or 
play the game offline. These foreground content updates are most commonly 
encountered if the Xbox hasn't yet had an overnight maintenance window in 
standby prior to a game update becoming required. 

If an Xbox has games/apps installed on an external hard drive that have an 
available update during the maintenance window, the drive will power up to 
enable downloading and applying of the update. As others have mentioned, there 
is a setting to turn off external drives while in standby that is enabled by 
default. This setting is used to control external hard drives so that they will 
enter into a low power state while the console is in standby for power saving 
purposes. With this setting enabled, external hard drives should power up when 
called by the Xbox to enable content updates while the console is in standby. 
With that said, there may be some external hard drives that don't work properly 
with the power management commands being used, so this setting can be disabled 
to work around interop issues with those hard drives by keeping them powered up 
while the Xbox is in standby. Net-net, content on external hard drives should 
be updated when the Xbox is in standby with this setting enabled, barring any 
hard drive firmware interop issues.


-Original Message-
From: NANOG  On Behalf Of Chris Adams
Sent: Wednesday, February 12, 2020 12:04 PM
To: nanog@nanog.org
Subject: [EXTERNAL] Re: akamai yesterday - what in the world was that

Once upon a time, Mike Hammett  said:
> Aren't most modern consoles on whether they're "on" or not? IE: It's not a 
> full power up from a dead stop, 0 watts power usage. 

The Xbox One kind of does that - it can receive updates (both game and
OS) in that state, but it depends on other settings.  If you plug in an 
external hard drive, there's a separate setting that is off by default (so if a 
game is on the external drive, it doesn't get updates).
--
Chris Adams 

Re: ATT Microcell in Austin, TX

[Brandon Martin]

On 2/18/20 2:25 PM, Matt Hoppes wrote:

Power goes out and the pole mounted nodes go out eventually.


Where "eventually" seems to vary a LOT.  I've observed hold up times as 
long as 8+ hours all the way down to "well, I guess there was a minor 
power glitch at the nearest power injection point because it dropped 
everything for 30 seconds while stuff came back up".  Lots of factors 
seem to play in this both in terms of design and maintenance.  For a 
while, some MSOs took their job seriously as they were offering 
relatively popular business-oriented voice products.  MSOs targeting 
only consumer service and still in the mindset of linear TV (or having 
not touched their plant since that was the major use case) often have no 
battery at all.


The same seems true of many FTTN deployments.  Hold-up time on nodes 
varies a lot.  The older the deployment, the more design hold-up time it 
seems to have, but of course maintenance varies a lot.  Newer 
deployments, especially fiber-to-the-curb often have essentially no 
hold-up at the local node unless it's back powered from the customer 
prem (in which case the customer can keep it up themselves).

--
Brandon Martin

Re: ATT Microcell in Austin, TX

[Matt Hoppes]

This is already how much of the cable networks operate. 

Power goes out and the pole mounted nodes go out eventually. 

> On Feb 18, 2020, at 2:15 PM, Constantine A. Murenin  
> wrote:
> 
>> On Tue, 18 Feb 2020 at 10:10, Darin Steffl  wrote:
> 
>> I believe that when this happens, they should proactively block or limit 
>> video and file download/upload traffic as much as possible to make sure 
>> communications like calls and texts can go through with the highest success 
>> rate possible. Netflix and YouTube should never hinder more important 
>> communications in my opinion. Maybe it's as simple as putting a rate limit 
>> for each cellphone connected to these now overloaded sectors so no one can 
>> hog the cell capacity.
> 
> This is very easy to do, thanks for the widespread adoption and use of HTTP, 
> which lets you easily filter these sorts of things in times of need and to 
> suit the requirements.
> 
> Or, what, we no longer use HTTP, because it's not "secure"?
> 
> Nevermind, folks.  Don't forget to update your certs and thank IETF, Mozilla, 
> Cloudflare and Google Chrome for your lack of connectivity.  But at least 
> you're secure, as no bad traffic can reach you now!
> 
> C.

Re: ATT Microcell in Austin, TX

[Constantine A. Murenin]

On Tue, 18 Feb 2020 at 10:10, Darin Steffl  wrote:

> I believe that when this happens, they should proactively block or limit
> video and file download/upload traffic as much as possible to make sure
> communications like calls and texts can go through with the highest success
> rate possible. Netflix and YouTube should never hinder more important
> communications in my opinion. Maybe it's as simple as putting a rate limit
> for each cellphone connected to these now overloaded sectors so no one can
> hog the cell capacity.
>

This is very easy to do, thanks for the widespread adoption and use of
HTTP, which lets you easily filter these sorts of things in times of need
and to suit the requirements.

Or, what, we no longer use HTTP, because it's not "secure"?

Nevermind, folks.  Don't forget to update your certs and thank IETF,
Mozilla, Cloudflare and Google Chrome for your lack of connectivity.  But
at least you're secure, as no bad traffic can reach you now!

C.

Re: ATT Microcell in Austin, TX

[Shane Ronan]

Agreed, specifically talking about small/micro cells.

Shane

On Tue, Feb 18, 2020, 2:11 PM Jared Mauch  wrote:

> On Tue, Feb 18, 2020 at 02:05:15PM -0500, Shane Ronan wrote:
> > I can tell you that most carriers have neither type, at least in the US.
>
> Most towers can survive a brown/blackout lasting a few minutes
> (usually enough to last the safety system cycling and the fuses blowing
> on a grounded leg).
>
> Major towers tend to have 8-12h of battery, with sometimes 24h.
>
> Most utility outages are restored in that time with the exception
> being major storms.
>
> In michigan we can get a credit for lack of restoration in 16
> hours:
>
> -- snip --
> A customer is eligible for a credit under normal
> conditions if the utility fails to restore service
> within 16 hours after an outage resulting from
> conditions other than catastrophic conditions.
> -- snip --
>
> this lines up with the planning strategy of the utilities.
>
> - Jared
>
> --
> Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
> clue++;  | http://puck.nether.net/~jared/  My statements are only
> mine.
>

Re: ATT Microcell in Austin, TX

[Jared Mauch]

On Tue, Feb 18, 2020 at 02:05:15PM -0500, Shane Ronan wrote:
> I can tell you that most carriers have neither type, at least in the US.

Most towers can survive a brown/blackout lasting a few minutes
(usually enough to last the safety system cycling and the fuses blowing
on a grounded leg).

Major towers tend to have 8-12h of battery, with sometimes 24h.

Most utility outages are restored in that time with the exception
being major storms.

In michigan we can get a credit for lack of restoration in 16 hours:

-- snip --
A customer is eligible for a credit under normal
conditions if the utility fails to restore service
within 16 hours after an outage resulting from
conditions other than catastrophic conditions.
-- snip --

this lines up with the planning strategy of the utilities.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: ATT Microcell in Austin, TX

[Shane Ronan]

I can tell you that most carriers have neither type, at least in the US.

Shane

On Tue, Feb 18, 2020, 1:18 PM Stephen Satchell  wrote:

> There is power backup and then there is power backup.
>
> The former is a small power pack (batteries, supercapacitors, whatever)
> that will allow the microcell to weather a short blackout or brownout.
> We are talking seconds, to bridge switching transits.  To be useful in a
> deployment, such a holdover battery needs to be very low maintenance.
> (Think about cars that use supercapacitors as a battery replacement --
> good for short needs like a single engine start.)
>
> The latter is longer-life power to keep the microcell going for days or
> weeks.  It's debatable whether this is mandatory.
>

Re: ATT Microcell in Austin, TX

[Stephen Satchell]

There is power backup and then there is power backup.

The former is a small power pack (batteries, supercapacitors, whatever) 
that will allow the microcell to weather a short blackout or brownout. 
We are talking seconds, to bridge switching transits.  To be useful in a 
deployment, such a holdover battery needs to be very low maintenance. 
(Think about cars that use supercapacitors as a battery replacement -- 
good for short needs like a single engine start.)


The latter is longer-life power to keep the microcell going for days or 
weeks.  It's debatable whether this is mandatory.

Re: ATT Microcell in Austin, TX

[Ben Cannon]

Most of the small or micro cells are there to add data capacity not necessary 
device count, which are two different things.  

However, where they are added to augment device count we will have problems if 
they are not backed up.

As the tech shrinks and battery tech improves this will become solvable, but we 
are a ways out still.

-Ben

> On Feb 18, 2020, at 8:45 AM, sro...@ronan-online.com wrote:
> 
> The feasibility of back hauling power from a central location is almost 
> zero. Conduit can be direct buried and then fiber shot through it, this would 
> be almost impossible with DC power cables.
> 
> Keep in mind that WPS already provides priority to “priority” traffic.
> 
> Shane
> 
> Sent from my iPhone
> 
>>> On Feb 18, 2020, at 11:09 AM, Darin Steffl  wrote:
>>> 
>> 
>> Matt, 
>> 
>> You're correct that if most of these small cells goes offline during a power 
>> outage, the remaining macro cells would not be able to handle the load well. 
>> 
>> Data would be nearly useless and phone/texts may be sporadic. 
>> 
>> I believe that when this happens, they should proactively block or limit 
>> video and file download/upload traffic as much as possible to make sure 
>> communications like calls and texts can go through with the highest success 
>> rate possible. Netflix and YouTube should never hinder more important 
>> communications in my opinion. Maybe it's as simple as putting a rate limit 
>> for each cellphone connected to these now overloaded sectors so no one can 
>> hog the cell capacity.
>> 
>> It would be pretty sweet though if small cells all had a linked power source 
>> following the same fiber paths that all hook back into a large battery 
>> backup or generator somewhere. Maybe 30-40 small cells can have backup power 
>> from one macro cell generator. I'm not sure if they're installed that way or 
>> not but it would ideal. Otherwise, you're losing 10 to 100x of the capacity 
>> of a cell network during power outages if the small cells go down. 
>> 
>>> On Tue, Feb 18, 2020, 9:46 AM Matt Erculiani  wrote:
>>> It will be interesting to see how this plays out as reliance on these small 
>>> cells for capacity grows. I'd imagine demand for cellular bandwidth goes up 
>>> during a power outage and not down. 
>>> 
>>> Is it reasonable to think that there could be a situation where cell 
>>> capacity is not available during a time of need because these sites will 
>>> simply go down and significantly reduce coverage/quality in dense 
>>> metropolitan areas?
>>> 
>>> -Matt
>>> 
 On Sun, Feb 16, 2020, 19:15 Shane Ronan  wrote:
 This is a small cell. They are very common across all of the carriers.
 
 It is NOT intended to provide primary coverage for the area.
 
 It IS intended to provide additional capacity to the immediate area.
 
 Think of the large cell towers as providing blanket coverage, while small 
 cells provide hot spots of increased capacity.
 
 Most small cells have no battery backup or generator at all, as it's not 
 feasible given the real estate available.
 
> On Sun, Feb 16, 2020, 5:58 PM Chris Boyd  wrote:
> Since people on here like to talk about the generatorn run time on cell 
> towers, I thought y’all might like to see an ATT microcell in downtown 
> Austin, TX.  No apparent generator or battery on it.
> 
> https://imgur.com/a/RY9Tg7h
> 
> —Chris

Re: ATT Microcell in Austin, TX

[sronan]

The feasibility of back hauling power from a central location is almost zero. 
Conduit can be direct buried and then fiber shot through it, this would be 
almost impossible with DC power cables.

Keep in mind that WPS already provides priority to “priority” traffic.

Shane

Sent from my iPhone

> On Feb 18, 2020, at 11:09 AM, Darin Steffl  wrote:
> 
> 
> Matt, 
> 
> You're correct that if most of these small cells goes offline during a power 
> outage, the remaining macro cells would not be able to handle the load well. 
> 
> Data would be nearly useless and phone/texts may be sporadic. 
> 
> I believe that when this happens, they should proactively block or limit 
> video and file download/upload traffic as much as possible to make sure 
> communications like calls and texts can go through with the highest success 
> rate possible. Netflix and YouTube should never hinder more important 
> communications in my opinion. Maybe it's as simple as putting a rate limit 
> for each cellphone connected to these now overloaded sectors so no one can 
> hog the cell capacity.
> 
> It would be pretty sweet though if small cells all had a linked power source 
> following the same fiber paths that all hook back into a large battery backup 
> or generator somewhere. Maybe 30-40 small cells can have backup power from 
> one macro cell generator. I'm not sure if they're installed that way or not 
> but it would ideal. Otherwise, you're losing 10 to 100x of the capacity of a 
> cell network during power outages if the small cells go down. 
> 
>> On Tue, Feb 18, 2020, 9:46 AM Matt Erculiani  wrote:
>> It will be interesting to see how this plays out as reliance on these small 
>> cells for capacity grows. I'd imagine demand for cellular bandwidth goes up 
>> during a power outage and not down. 
>> 
>> Is it reasonable to think that there could be a situation where cell 
>> capacity is not available during a time of need because these sites will 
>> simply go down and significantly reduce coverage/quality in dense 
>> metropolitan areas?
>> 
>> -Matt
>> 
>>> On Sun, Feb 16, 2020, 19:15 Shane Ronan  wrote:
>>> This is a small cell. They are very common across all of the carriers.
>>> 
>>> It is NOT intended to provide primary coverage for the area.
>>> 
>>> It IS intended to provide additional capacity to the immediate area.
>>> 
>>> Think of the large cell towers as providing blanket coverage, while small 
>>> cells provide hot spots of increased capacity.
>>> 
>>> Most small cells have no battery backup or generator at all, as it's not 
>>> feasible given the real estate available.
>>> 
 On Sun, Feb 16, 2020, 5:58 PM Chris Boyd  wrote:
 Since people on here like to talk about the generatorn run time on cell 
 towers, I thought y’all might like to see an ATT microcell in downtown 
 Austin, TX.  No apparent generator or battery on it.
 
 https://imgur.com/a/RY9Tg7h
 
 —Chris

Re: ATT Microcell in Austin, TX

[Tom Beecher]

Net neutrality!*

*Except if someone drives through a power pole.

On Tue, Feb 18, 2020 at 11:11 AM Darin Steffl 
wrote:

> Matt,
>
> You're correct that if most of these small cells goes offline during a
> power outage, the remaining macro cells would not be able to handle the
> load well.
>
> Data would be nearly useless and phone/texts may be sporadic.
>
> I believe that when this happens, they should proactively block or limit
> video and file download/upload traffic as much as possible to make sure
> communications like calls and texts can go through with the highest success
> rate possible. Netflix and YouTube should never hinder more important
> communications in my opinion. Maybe it's as simple as putting a rate limit
> for each cellphone connected to these now overloaded sectors so no one can
> hog the cell capacity.
>
> It would be pretty sweet though if small cells all had a linked power
> source following the same fiber paths that all hook back into a large
> battery backup or generator somewhere. Maybe 30-40 small cells can have
> backup power from one macro cell generator. I'm not sure if they're
> installed that way or not but it would ideal. Otherwise, you're losing 10
> to 100x of the capacity of a cell network during power outages if the small
> cells go down.
>
> On Tue, Feb 18, 2020, 9:46 AM Matt Erculiani  wrote:
>
>> It will be interesting to see how this plays out as reliance on these
>> small cells for capacity grows. I'd imagine demand for cellular bandwidth
>> goes up during a power outage and not down.
>>
>> Is it reasonable to think that there could be a situation where cell
>> capacity is not available during a time of need because these sites will
>> simply go down and significantly reduce coverage/quality in dense
>> metropolitan areas?
>>
>> -Matt
>>
>> On Sun, Feb 16, 2020, 19:15 Shane Ronan  wrote:
>>
>>> This is a small cell. They are very common across all of the carriers.
>>>
>>> It is NOT intended to provide primary coverage for the area.
>>>
>>> It IS intended to provide additional capacity to the immediate area.
>>>
>>> Think of the large cell towers as providing blanket coverage, while
>>> small cells provide hot spots of increased capacity.
>>>
>>> Most small cells have no battery backup or generator at all, as it's not
>>> feasible given the real estate available.
>>>
>>> On Sun, Feb 16, 2020, 5:58 PM Chris Boyd 
>>> wrote:
>>>
 Since people on here like to talk about the generatorn run time on cell
 towers, I thought y’all might like to see an ATT microcell in downtown
 Austin, TX.  No apparent generator or battery on it.

 https://imgur.com/a/RY9Tg7h

 —Chris
>>>
>>>

Re: ATT Microcell in Austin, TX

[Darin Steffl]

Matt,

You're correct that if most of these small cells goes offline during a
power outage, the remaining macro cells would not be able to handle the
load well.

Data would be nearly useless and phone/texts may be sporadic.

I believe that when this happens, they should proactively block or limit
video and file download/upload traffic as much as possible to make sure
communications like calls and texts can go through with the highest success
rate possible. Netflix and YouTube should never hinder more important
communications in my opinion. Maybe it's as simple as putting a rate limit
for each cellphone connected to these now overloaded sectors so no one can
hog the cell capacity.

It would be pretty sweet though if small cells all had a linked power
source following the same fiber paths that all hook back into a large
battery backup or generator somewhere. Maybe 30-40 small cells can have
backup power from one macro cell generator. I'm not sure if they're
installed that way or not but it would ideal. Otherwise, you're losing 10
to 100x of the capacity of a cell network during power outages if the small
cells go down.

On Tue, Feb 18, 2020, 9:46 AM Matt Erculiani  wrote:

> It will be interesting to see how this plays out as reliance on these
> small cells for capacity grows. I'd imagine demand for cellular bandwidth
> goes up during a power outage and not down.
>
> Is it reasonable to think that there could be a situation where cell
> capacity is not available during a time of need because these sites will
> simply go down and significantly reduce coverage/quality in dense
> metropolitan areas?
>
> -Matt
>
> On Sun, Feb 16, 2020, 19:15 Shane Ronan  wrote:
>
>> This is a small cell. They are very common across all of the carriers.
>>
>> It is NOT intended to provide primary coverage for the area.
>>
>> It IS intended to provide additional capacity to the immediate area.
>>
>> Think of the large cell towers as providing blanket coverage, while small
>> cells provide hot spots of increased capacity.
>>
>> Most small cells have no battery backup or generator at all, as it's not
>> feasible given the real estate available.
>>
>> On Sun, Feb 16, 2020, 5:58 PM Chris Boyd  wrote:
>>
>>> Since people on here like to talk about the generatorn run time on cell
>>> towers, I thought y’all might like to see an ATT microcell in downtown
>>> Austin, TX.  No apparent generator or battery on it.
>>>
>>> https://imgur.com/a/RY9Tg7h
>>>
>>> —Chris
>>
>>

Re: ATT Microcell in Austin, TX

[Matt Erculiani]

It will be interesting to see how this plays out as reliance on these small
cells for capacity grows. I'd imagine demand for cellular bandwidth goes up
during a power outage and not down.

Is it reasonable to think that there could be a situation where cell
capacity is not available during a time of need because these sites will
simply go down and significantly reduce coverage/quality in dense
metropolitan areas?

-Matt

On Sun, Feb 16, 2020, 19:15 Shane Ronan  wrote:

> This is a small cell. They are very common across all of the carriers.
>
> It is NOT intended to provide primary coverage for the area.
>
> It IS intended to provide additional capacity to the immediate area.
>
> Think of the large cell towers as providing blanket coverage, while small
> cells provide hot spots of increased capacity.
>
> Most small cells have no battery backup or generator at all, as it's not
> feasible given the real estate available.
>
> On Sun, Feb 16, 2020, 5:58 PM Chris Boyd  wrote:
>
>> Since people on here like to talk about the generatorn run time on cell
>> towers, I thought y’all might like to see an ATT microcell in downtown
>> Austin, TX.  No apparent generator or battery on it.
>>
>> https://imgur.com/a/RY9Tg7h
>>
>> —Chris
>
>