Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
 wrote:
> A problem of QUIC with NAT is that existing NAT can not detect
> graceful shutdown of QUIC and must depends on timeout.
>
> So, port numbers may be used up before timeout.

Hmm, this is not what is happening.

I managed to (fairly easily!) reproduce the issue earlier tonight. I
generated a fair bit of UDP traffic via xbox, a corporate VPN, and
youtube over quic.

Sure enough, after about 45 minutes, the YouTube app on my iPad paused
and then auto-reset the stream quality to "144p" resolution.

I was able to set it back to 720p60, but that only lasted about 2
minutes before the stream stopped playing. I waited several minutes
for it to resume; it did not. So, I then blocked UDP 443 on my router.
The video then *immediately* resumed at 720p60.

I didn't run tcpdump but I did grab some screenshots of iftop. It
looks like my iPad connected to AS15169 with a single UDP connection.
I see one consistent source and dest IP / port combo for those 10s of
minutes, up until UDP is blocked. Local port 58053, remote port 443 on
the same IP for the whole time.

At first the connection averages around 2mbps; when the issue occurs,
I see it has dropped to a rate of under 200kbps.

I've no idea what to make of that. Surely Google isn't throttling
their traffic to me? If so why allow a fallback to TCP?

When I originally discovered this issue, I was of course not trying to
do anything odd with my internet connection. And I didn't immediately
know QUIC was the issue.

Only after it happened several times did I dig into the traffic and
then block QUIC, and I was shocked to see that both resolve the issue
and prevent its recurrence.

So again -- I hit this issue repeatedly without trying to --

And just now, I was able to reproduce it simply by generating a bit of
UDP traffic on purpose!

I only wish I were insane; but from where I'm sitting, QUIC has broken
my internet, and the resolution is blocking QUIC.

-- Dan

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Sterling wrote:


I received a comment that maybe the issue is not AT's "core"
network, but rather to do with the NAT device in my house.


A problem of QUIC with NAT is that existing NAT can not detect
graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Mark Andrews]

> On 20 Feb 2020, at 11:29, Daniel Sterling  wrote:
> 
> On Tue, Feb 18, 2020 at 11:47 PM Daniel Sterling
>  wrote:
>> random-source-port UDP traffic does not impress the AT network flow
>> control systems, and your DNS traffic becomes unbearably slow (or is
> 
> I received a comment that maybe the issue is not AT's "core"
> network, but rather to do with the NAT device in my house.
> 
> Oh, I wish this were the case!
> 
> Unfortunately, there is no NAT CPE from AT involved:
> 
> I've taken AT's RG CPE out completely. That device, which otherwise
> would indeed be my gateway, is unused on my home network, completely
> unpowered and unplugged.
> 
> Instead I've got a linux box hooked directly up to the ONT. This works
> fine; occasionally I *do* have do reconnect their RG and let it
> re-auth to the PON, but once the connection is live, I fully unplug
> their RG again, and plug my laptop (spoofing its MAC) back in directly
> to the ONT.
> 
> The laptop is running ubuntu 19.10, so there should be no artificial
> limits. It's running NAT directly from the v4 IP I get from DHCP.
> 
> 
> You may have noticed v4 is a theme here. I have nothing against using
> v6 -- , I must admit the truth is I have no idea how to make ubuntu
> acquire a v6 -- address? block ? I don't even know the right term --
> from uverse.

It should just be a DHCPv6 PREFIX DELEGATION (PD).  See RFC 8415.

If AT are still using 6RD (IPv6 Rapid Deployment) that is described in
RFC 5969.  There is a DHCPv4 option that you can request that gives
you the details for configuring the IPv6 in IPv4 tunnel and how to
compute the IPv6 prefix from the allocated IPv4 address.

CPE’s just try both and use DHCPv6 PD if both exist.

> I know v6 works cuz AT's device supports it, and openwrt does it out
> of the box-- but heck if I know how it works. At the risk of asking
> this list for tech support -- does anyone want to ping me off list and
> point me in the right direction?? Maybe somebody from Google -- you
> can make up for breaking my v4 internet!!
> 
> Thanks,
> Dan

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

FCC: Sharing outage reports with State-Local-Other Government Agencies

[Sean Donelan]

I missed this in the announcement of the FCC's February meeting agenda 
next week.


In the early 2000's (after 9/11), the FCC ruled that outage reports were 
"presumptively confidential."  Personnally, I think the real reason was a 
specific carrier used the outage reporting in its sales/marketing, which 
upset the other carriers.  They complained to the FCC about the "misuse" 
of outage reports for several years from the late 1990's.  Yes, I was one 
of those people that used to visit the FCC Public Reading Room to read the 
outage report filings, before things were on-line.




At its February commission meeting, the FCC proposes to change its 
information sharing rules for outage reports to allow sharing with the 
States, District of Columbia, Tribal nations, territories and other 
federal government agencies.


https://docs.fcc.gov/public/attachments/DOC-362365A1.pdf

- Propose to provide direct, read-only access to NORS and DIRS filings to 
qualified agencies of the 50 states, the District of Columbia, Tribal 
nations, territories, and federal government.


- Propose to allow these agencies to share NORS and DIRS information with 
other public safety officials that reasonably require NORS and DIRS 
information to prepare for and respond to disasters.


- Propose to allow participating agencies to publicly disclose NORS or 
DIRS filing information that is aggregated and anonymized across at least 
four service providers.


- Propose to condition a participating agency’s direct access to NORS and 
DIRS filings on their agreement to treat the filings as confidential and 
not disclose them absent a finding by the Commission that allows them to 
do so.


- Propose to establish an application process that would grant agencies 
access to NORS and DIRS after those agencies certify to certain 
requirements related to maintaining confidentiality of the data and the

security of the databases.

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 11:47 PM Daniel Sterling
 wrote:
> random-source-port UDP traffic does not impress the AT network flow
> control systems, and your DNS traffic becomes unbearably slow (or is

I received a comment that maybe the issue is not AT's "core"
network, but rather to do with the NAT device in my house.

Oh, I wish this were the case!

Unfortunately, there is no NAT CPE from AT involved:

I've taken AT's RG CPE out completely. That device, which otherwise
would indeed be my gateway, is unused on my home network, completely
unpowered and unplugged.

Instead I've got a linux box hooked directly up to the ONT. This works
fine; occasionally I *do* have do reconnect their RG and let it
re-auth to the PON, but once the connection is live, I fully unplug
their RG again, and plug my laptop (spoofing its MAC) back in directly
to the ONT.

The laptop is running ubuntu 19.10, so there should be no artificial
limits. It's running NAT directly from the v4 IP I get from DHCP.


You may have noticed v4 is a theme here. I have nothing against using
v6 -- , I must admit the truth is I have no idea how to make ubuntu
acquire a v6 -- address? block ? I don't even know the right term --
from uverse.

I know v6 works cuz AT's device supports it, and openwrt does it out
of the box-- but heck if I know how it works. At the risk of asking
this list for tech support -- does anyone want to ping me off list and
point me in the right direction?? Maybe somebody from Google -- you
can make up for breaking my v4 internet!!

Thanks,
Dan

Re: Tell me about AS19111

[John Sweeting]

UPDATE: ARIN has completed coordination with the organization and whois is now 
correct. Thanks!



On 2/5/20, 8:30 PM, "NANOG on behalf of John Levine"  wrote:

1800vitamins.org has a web site at 12.180.219.234 which looks like
they would sell me vitamins should I or my dog need any.

Routeviews tells me that IP is in AS19111, routed via AS7018.  AS7018
is AT which isn't surprising for a 12/8 address, but ARIN says
AS19111 doesn't exist.  Huh?

Signed,
Confused
-- 
Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for 
Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:
> Yeah, that was a nice surprise to find that my tethered LTE connection
> was out performing my wired cable modem service. Of course, I had
> already signed up for a year of service and there were early termination
> fees for cancelling... that and there are no other wireline providers
> available at my home (not even ATT).

So we're left with some questions:

1. It's clear I'm not the only one experiencing this issue. How
widespread is this problem, really? Has it gotten rather worse over
the past ~year?

2. Are customers of larger ISPs much more impacted than customers of
smaller ones that (assumedly) don't have to deprioritize UDP so much?
2a. If users *are* impacted, as Blake notes, they may not be able to
switch ISPs to improve their lot.. will customers complain to their
ISP or to Google?

3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
only works on v6 (and if it in fact continues to actively BREAK
v4-only users), then is this v6's "killer app" that will drive
adoption?
3a. Or will this issue hinder HTTP/3 deployment (or cause mass
blocking of UDP on clients)?

4. Will ISPs be willing to give UDP traffic higher priority to improve
user experience? Will that only happen once HTTP/3 is widely deployed?

5. We can only assume Google is aware of this issue; will Google work
to improve QUIC fallback to TCP, or will they work with ISPs to get
QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
actively encourage QUIC to break v4 at the expensive of current user
experience?
5a. Will another company that wants HTTP/3 to succeed take the mantle
and work with ISPs to improve the situation? I'm reminded of when
Microsoft worked with ISPs to ensure xbox UDP traffic would transit
properly

-- Dan

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

If by targeting DDoSes you mean rate limiting all UDP (because some UDP 
is bad), then I would expect that policy to be published on a provider's 
website so that folks using UDP would at least have the option to 
research and know this info before signing up or agreeing to a service 
commitment. Similarly, application developers could point to a 
provider's policy when designing their applications or assisting users. 
Even if Net Neutrality is dead, I believe the "Restoring Internet 
Freedom" order is still a thing in the US.



On 2/19/2020 3:03 PM, Mike Hammett wrote:
Net Neutrality likely wouldn't have impacted this at all. AT isn't 
targeting QUIC, they're targeting DDoSes.




-
Mike Hammett
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 


*From: *"Brian J. Murrell" 
*To: *nanog@nanog.org
*Sent: *Wednesday, February 19, 2020 3:01:20 PM
*Subject: *Re: QUIC traffic throttled on AT residential

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:
>
> Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

> So that people (or
> companies) are free to develop new applications or enhance existing
> ones
> without running into a quagmire of different policies implemented by
> any
> number of different networks between the application developer and
> the
> application's users?

Yes, this is a very prominent reason for Net Neutrality.  Too bad the
FCC killed that out from under the people and companies that would
utilize it to develop new applications.

Cheers,
b.

Re: QUIC traffic throttled on AT residential

[Mike Hammett]

Net Neutrality likely wouldn't have impacted this at all. AT isn't targeting 
QUIC, they're targeting DDoSes. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Brian J. Murrell"  
To: nanog@nanog.org 
Sent: Wednesday, February 19, 2020 3:01:20 PM 
Subject: Re: QUIC traffic throttled on AT residential 

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote: 
> 
> Isn't this exactly why Net Neutrality is a thing: 

Isn't it a "dead" thing in the USofA? 

> So that people (or 
> companies) are free to develop new applications or enhance existing 
> ones 
> without running into a quagmire of different policies implemented by 
> any 
> number of different networks between the application developer and 
> the 
> application's users? 

Yes, this is a very prominent reason for Net Neutrality. Too bad the 
FCC killed that out from under the people and companies that would 
utilize it to develop new applications. 

Cheers, 
b. 

Re: QUIC traffic throttled on AT residential

[Brian J. Murrell]

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:
> 
> Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

> So that people (or 
> companies) are free to develop new applications or enhance existing
> ones 
> without running into a quagmire of different policies implemented by
> any 
> number of different networks between the application developer and
> the 
> application's users?

Yes, this is a very prominent reason for Net Neutrality.  Too bad the
FCC killed that out from under the people and companies that would
utilize it to develop new applications.

Cheers,
b.



signature.asc
Description: This is a digitally signed message part

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/19/2020 2:01 PM, Daniel Sterling wrote:

On Wed, Feb 19, 2020 at 2:55 PM Blake Hudson  wrote:

I'm guessing ATT doesn't disclose this policy transparently either.

they disclose it pretty transparently to their customers in the form
of very slow youtube traffic when using v4 QUIC ;)


Yeah, that was a nice surprise to find that my tethered LTE connection 
was out performing my wired cable modem service. Of course, I had 
already signed up for a year of service and there were early termination 
fees for cancelling... that and there are no other wireline providers 
available at my home (not even ATT).

Re: QUIC traffic throttled on AT residential

[Brandon Martin]

On 2/19/20 2:54 PM, Fred Baker wrote:

The argument I have heard is that residential firewalls often block anything 
that is*not*  UDP or TCP. The question for the googlers was existential - can 
it work at all?


I'm not sure that they "block" it, per se, though some probably do have 
an explicit rule to that effect.  I would think the bigger issue is that 
they don't know how to 1:N NAT arbitrary L4s (and how would they), so 
the absolute best you might get is that the first device behind the NAT 
to establish a mapping sees all the relevant L4 traffic and everybody 
else is locked out.  I'd suspect the normal case is simply that they 
drop it on the floor unless there's a specified "DMZ" host.


Perhaps this is just a semantic difference, but I think it's actually an 
even more difficult issue to resolve.  If it were simply blocked, that's 
usually "easy" (either for the user, via a management interface, or for 
the vendor, via policy template) to fix.  Writing an entirely new L4 NAT 
helper is a different matter entirely.


IPv6 would of course render this moot, but we all know how well IPv6 
traffic gets treated...

--
Brandon Martin

Re: QUIC traffic throttled on AT residential

[Michael Brown]

On 2020-02-19 1:06 a.m., Masataka Ohta wrote:
> Are you saying AT should block UDP entirely?

No; while I don't presume to have all the answers they should at the
minimum take into account how it affects the end-user (CUSTOMER!)
experience when making decisions like this.

(No they shouldn't block UDP entirely, but if they're having UDP/443
DDOS problems then blocking it while they get a proper scalable solution
in place is better than throttling it).

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 2:55 PM Blake Hudson  wrote:
> I'm guessing ATT doesn't disclose this policy transparently either.

they disclose it pretty transparently to their customers in the form
of very slow youtube traffic when using v4 QUIC ;)

Re: QUIC traffic throttled on AT residential

[Fred Baker]

> On Feb 18, 2020, at 4:00 PM, Ca By  wrote:
> 
> I am not a fan of quic or any udp traffic. My suggestion was that Google use 
> an new L4 instead of UDP, but that was too hard for the Googlers.

The argument I have heard is that residential firewalls often block anything 
that is *not* UDP or TCP. The question for the googlers was existential - can 
it work at all?


signature.asc
Description: Message signed with OpenPGP

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/18/2020 6:00 PM, Ca By wrote:
On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
mailto:sterling.dan...@gmail.com>> wrote:


I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
from google (esp. youtube) becomes very slow after a time.

This especially occurs with ipv4 connections. I'm not the only one to
notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
notes the issue.

I assume traffic is being throttled on AT's side. And it's not done
with their CPE; I'm bypassing their NAT box and connecting my laptop
directly to the ONT.

A quick google search shows people are aware that QUIC can look like
DoS traffic -- but how widely known is this problem? It may become
worse if / when sites transition to HTTP/3

For now I reject UDP 443 on the client firewall. Applications using
QUIC client libraries then fallback to TCP. This works well and I have
no issues with latency or throughput when using TCP.

May I naively ask if Google staff are working with AT to address
this?


Sounds like you found the answer, ATT just needs to scale up your 
rejection approach that is proven to work well.


Yes, most ddos traffic is ipv4 udp and yes Google was made aware that 
they would be mixing with bad company in the pool of ipv4 udp traffic 
... but they have reasons.


I am not a fan of quic or any udp traffic. My suggestion was that 
Google use an new L4 instead of UDP, but that was too hard for the 
Googlers.


So here we are.

Just say no to udp.


Isn't this exactly why Net Neutrality is a thing: So that people (or 
companies) are free to develop new applications or enhance existing ones 
without running into a quagmire of different policies implemented by any 
number of different networks between the application developer and the 
application's users?


For what it's worth, Cox cable also treats UDP traffic differently - 
impacting the performance of VPNs using UDP. Cox provides a list of 
blocked ports on their website, but makes no mention that they throttle 
or otherwise treat UDP as a special case. I'm guessing ATT doesn't 
disclose this policy transparently either.

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Christopher Morrow wrote:


2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.


Subject suggests it's retail ISP to homes, which are unlikely to
be multihomed.


It probably depends on where you want to do such limiting, right?
"At peering/transit edge" - save your core, dont' carry traffic you
"know" you will throw away anyway.
"At the customer edge" - scaling of state management could be
problematic && you'll carry this 'bad' traffic across your network.


Limiting is not by edges but by ISPs.

In transit ISPs, there are numerous flows generated by customers
of other ISPs that customer wise rate limiting does not scale.

In access ISPs, revenue increases proportional to the number
of customers that you can prepare rate limiting devices proportional
to the number of customers to make customer wise rate limiting scale.

Masataka Ohta

[NANOG-announce] Promoting Diversity + Access in Internet Tech 

[NANOG Marketing]

*A conversation with Charisse Stokes + Trent Edwards of the Montgomery,
Alabama Chamber of Commerce.*

“Our diversity of background, thought, and experience — as well as race and
gender — give us the best opportunity to compete, and succeed," says Trent
Edwards of the Montgomery, Alabama Chamber of Commerce.

Trent and his colleague Charisse Stokes hosted NANOG Executive Director
Edward McNair during his visit to Alabama this fall, for two days of
STEM-focused talks, lunch + learn sessions, and community meetings to share
NANOG’s perspective on the latest industry trends and careers in Internet
tech.

We recently had a conversation with Trent and Charisse, where we got to
talk about the initiatives they’re currently working on, and how those
align with NANOG’s mission to empower people and inspire change.

Read the Interview


*Get the latest NANOG news, delivered right to your inbox!*
Sign up for the NANOG Newsletter to keep up with our innovative community,
via the latest NANOG events + programs, member features + stories — all in
one place. Subscribe Now 
___
NANOG-announce mailing list
NANOG-announce@nanog.org
https://mailman.nanog.org/mailman/listinfo/nanog-announce

Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

On Wed, Feb 19, 2020 at 3:18 AM Masataka Ohta
 wrote:
>
> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.

It probably depends on where you want to do such limiting, right?
"At peering/transit edge" - save your core, dont' carry traffic you
"know" you will throw away anyway.
"At the customer edge" - scaling of state management could be
problematic && you'll carry this 'bad' traffic across your network.

Note: I'm not really casting aspersions on either part of the whole
argument here, just attempting to provide some color to the parts
which seem 'unlikely to be successful'

I think for a bunch of this discussion there are N folks with M goals,
and eventually 'the market will dictate' which final direction we
travel.

Re: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

"yes, also received"
"thannks Toma!" (and nanog owner)

On Wed, Feb 19, 2020 at 4:55 AM Töma Gavrichenkov  wrote:
>
> Peace,
>
> nanog-ow...@nanog.org
>
> On Wed, Feb 19, 2020 at 12:51 PM Dave Bell  wrote:
> > Is anyone else receiving this spam?
> Yes
>
> > Is there a better way to report this?
>
> nanog-ow...@nanog.org (CC'd) helped me in the past.
>
> --
> Töma

Re: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Töma Gavrichenkov]

Peace,

nanog-ow...@nanog.org

On Wed, Feb 19, 2020 at 12:51 PM Dave Bell  wrote:
> Is anyone else receiving this spam?
Yes

> Is there a better way to report this?

nanog-ow...@nanog.org (CC'd) helped me in the past.

--
Töma

Fwd: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Dave Bell]

Is anyone else receiving this spam?

Is there a better way to report this? I couldn't see anything from a quick
look through the various list pages.

Dave

-- Forwarded message -
From: Electric Forest Festival 
Date: Wed, 19 Feb 2020 at 09:42
Subject: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on
AT residential
To: 



*Electric Forest 2020 will take place on June 25-28, 2020.*

Forest HQ has received your email. Help save precious resources by
reviewing the information below and looking up common questions in The
Forest Frequently Asked Questions: Experience.ElectricForestFestival.com


Please contact Festival Ticketing Support at 855-279-6941 for all issue
regarding your purchase or for account troubleshooting.

*Electric Forest is sold out. *Lyte is the only HQ endorsed way to get
passes now that it’s sold out.


To know when all things Electric Forest 2020 are happening sign up to the
EF Newsletter. 

*Happy Forest!*

118323:553794

Re: QUIC traffic throttled on AT residential

[Dave Bell]

On Wed, 19 Feb 2020 at 08:18, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.
>
>
>
My network has multiple ingress/egress points, and I do my filtering on the
edge.

The Chris's statement applies to my retail home customers too.

Dave

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Christopher Morrow wrote:


2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.


Subject suggests it's retail ISP to homes, which are unlikely to
be multihomed.

Masataka Ohta