Re: understanding IPv6 (was: Re: QUIC traffic throttled on AT residential)

[Daniel Sterling]

On Sun, Jun 7, 2020 at 2:00 AM Fred Baker  wrote:
> I'm sorry you have chosen to ignore documents like RFC 3315, which is where 
> DHCP PD was first described (in 2003). It's not like anyone's hiding it.

I am sorry as well!

I openly admit I am not the smartest bear in the woods. I struggle to
read through RFCs. Researching IPv6 for me means doing a web search
for "what ipv6 halp", to which google frustratingly returns: No
results found for "what ipv6 halp".

In all seriousness, I have been trying to understand IPv6 for a long
time, and the documentation that I read (again, admittedly not often
RFCs, but certainly Wikipedia, linux distro docs, etc) never mentioned
DHCP PD, or at least never mentioned it as something important for how
end-users would use IPv6.

So while it may be true that no one is hiding this information, in my
experience no one is shining a spot light on it either, and until I
was told about it, I was simply unable to understand IPv6.

Once we know something we can forget that everyone else doesn't know
that same information, and figuring out what inexperienced people need
to know to understand a topic can be difficult. So I am offering to
the community that "DHCP PD" is such a thing: it's something that
everyone who already knows how things works takes for granted.

So when someone points me in the right direction by mentioning such an
important piece of the puzzle, I am genuinely grateful!

-- Dan

understanding IPv6 (was: Re: QUIC traffic throttled on AT residential)

[Daniel Sterling]

On Wed, Feb 19, 2020 at 7:51 PM Mark Andrews  wrote:
> > I have nothing against using
> > v6 -- , I must admit the truth is I have no idea how to make ubuntu
> > acquire a v6 -- address? block ? I don't even know the right term --
> > from uverse.
>
> It should just be a DHCPv6 PREFIX DELEGATION (PD).  See RFC 8415.

Mark -- thank you so much for this information.

Knowing that I needed to understand something called "DHCP PD" was the
most valuable thing I've learned about ipv6 in the past decade.

At $WORK we don't use v6 at all. (Fortune 500 company; I can deploy v6
as much as I want on the segments I control, but the larger company
does nothing with it.) So while I know how to set up and use ipv4
pretty well, I knew essentially nothing about v6 other than what I'd
read on my own time.

Not knowing how my home ISP equipment even acquired a v6 address (must
less how it handed them out to my home LAN devices) was very
frustrating, and reading about v6 in general was not very enlightening
for me. So this nugget of info -- "DHCP PD" -- was a godsend. By
looking up information related to it I was able to dramatically
increase my understanding of v6.

So again, thank you!

-- Dan

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Mon, Mar 2, 2020 at 4:29 PM Daniel Sterling
 wrote:
> Also: I think ipv6 isn't working for me cuz it's being dropped by a switch 
> I'm using!
>
> I will swap that out / remove that and try ipv6 again

OK, ipv6 is working for me now.

The switch that was dropping ipv6 traffic was: Windows 10 (1909)
hyper-v switch! I was running Windows -> hyper-v -> openwrt, since
openwrt's kernel didn't have support for a USB NIC I'm using. I know,
this is ridiculous.

I switched to ubuntu 19.10 -> kvm -> openwrt, and ipv6 works out of the box.

Next step may be to see if I can get ipv6 working with just plain
ubuntu (and to stop using USB NICs)

-- Dan

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

No voice service on my line, or TV. Just gigabit internet.

Also: I think ipv6 isn't working for me cuz it's being dropped by a switch
I'm using!

I will swap that out / remove that and try ipv6 again

-- Dan

On Thu, Feb 27, 2020, 9:10 AM Hiers, David  wrote:

> We find that they usually impose pretty harsh QOS on a link that has an
> ATT voice service.
>
> David
>
>
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay Hennigan
> Sent: Thursday, February 20, 2020 12:13 AM
> To: nanog@nanog.org
> Subject: Re: QUIC traffic throttled on AT residential
>
> On 2/18/20 18:40, nanog-l...@contactdaniel.net wrote:
>
> > Growing prevalence of IPv6-only
> > sites is probably the only thing that will get a lot of access
> > networks to support v6.
>
> I recall a similar idea called "The Great IPv6 Experiment" back in 2007.
> ;-)
>
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>
> --
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>

Re: QUIC traffic throttled on AT residential

[Tom Hill]

On 21/02/2020 23:37, Owen DeLong wrote:
> What’s next? Why not simply eliminate port numbers altogether in favor
> of a single 16-bit client-side unique session identifier.


I see what you did there.

-- 
Tom

RE: QUIC traffic throttled on AT residential

[Hiers, David]

We find that they usually impose pretty harsh QOS on a link that has an ATT 
voice service.

David



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay Hennigan
Sent: Thursday, February 20, 2020 12:13 AM
To: nanog@nanog.org
Subject: Re: QUIC traffic throttled on AT residential

On 2/18/20 18:40, nanog-l...@contactdaniel.net wrote:

> Growing prevalence of IPv6-only
> sites is probably the only thing that will get a lot of access 
> networks to support v6.

I recall a similar idea called "The Great IPv6 Experiment" back in 2007. ;-)


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

--
This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, notify the sender immediately by return email and delete the message 
and any attachments from your system.

Re: QUIC traffic throttled on AT residential

[Paul Timmins]

It's okay though, because we freed up UDP/53 by moving DNS to TCP/443, 
so then we can move HTTPS to UDP/53.


On 2/21/20 6:37 PM, Owen DeLong wrote:

First we moved the entire internet to TCP/443.

Now we propose moving it all to UDP/53.

What’s next? Why not simply eliminate port numbers altogether in favor 
of a single 16-bit client-side unique session identifier.


Owen

On Feb 21, 2020, at 15:20 , Matthew Petach > wrote:




On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski > wrote:



[...]

Now… once we are aware, the only question is — where we go from here?

—
./



Well, it's clear the UDP 443 experiment wasn't entirely successful.

So clearly, it's time to use the one UDP port that is allowed through 
at the top of everyone's ACL rules, and update QUIC in the next 
iteration to use UDP/53.


*THAT* should solve the whole problem, once and for all.

;)

Matt

Re: QUIC traffic throttled on AT residential

[Łukasz Bromirski]

At this pace and having adopted CI/CD methodology, we may QUICkly run out of 
UDP ports to use.

I’d actually switch to ICMP. Type 8 code 0 and Type 0, code 0. Then staging a 
war on rate-limiters around the world.

Also, 123/udp seems to look interesting ;)

-- 
./

> On 22 Feb 2020, at 00:21, Matthew Petach  wrote:
> 
> 
> 
> 
>> On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski  wrote:
>> 
>> [...]
>> 
>> Now… once we are aware, the only question is — where we go from here?
>> 
>> — 
>> ./
> 
> 
> 
> Well, it's clear the UDP 443 experiment wasn't entirely successful.
> 
> So clearly, it's time to use the one UDP port that is allowed through at the 
> top of everyone's ACL rules, and update QUIC in the next iteration to use 
> UDP/53.
> 
> *THAT* should solve the whole problem, once and for all.
> 
> ;)
> 
> Matt
> 

Re: QUIC traffic throttled on AT residential

[Owen DeLong]

First we moved the entire internet to TCP/443.

Now we propose moving it all to UDP/53.

What’s next? Why not simply eliminate port numbers altogether in favor of a 
single 16-bit client-side unique session identifier.

Owen

> On Feb 21, 2020, at 15:20 , Matthew Petach  wrote:
> 
> 
> 
> On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski  > wrote:
> 
> [...]
> 
> Now… once we are aware, the only question is — where we go from here?
> 
> — 
> ./
> 
> 
> Well, it's clear the UDP 443 experiment wasn't entirely successful.
> 
> So clearly, it's time to use the one UDP port that is allowed through at the 
> top of everyone's ACL rules, and update QUIC in the next iteration to use 
> UDP/53.
> 
> *THAT* should solve the whole problem, once and for all.
> 
> ;)
> 
> Matt
> 

Re: QUIC traffic throttled on AT residential

[Matthew Petach]

On Fri, Feb 21, 2020, 13:31 Łukasz Bromirski  wrote:

>
> [...]
>
> Now… once we are aware, the only question is — where we go from here?
>
> —
> ./
>


Well, it's clear the UDP 443 experiment wasn't entirely successful.

So clearly, it's time to use the one UDP port that is allowed through at
the top of everyone's ACL rules, and update QUIC in the next iteration to
use UDP/53.

*THAT* should solve the whole problem, once and for all.

;)

Matt

Re: QUIC traffic throttled on AT residential

[Łukasz Bromirski]

Hi Dan!

> On 21 Feb 2020, at 20:22, Dan Wing  wrote:
> 
> There are choices, such as making connection initiation, connection 
> acceptance, and connection termination parsable by network elements on the 
> path so state can be established, maintained, and cleared, DoS can be 
> identified, and so on.  The decision was to hide all that from network 
> elements.

Because monetization of content delivery should be constrained only
for those, that are able to make new standards, while ignoring
openness and cooperation.

Google is the new AOL? With AMP, QUIC and other nice, shiny,
closed and proprietary stuff? Oh, and BTW, please sync your life
with our cloud.

Now… once we are aware, the only question is — where we go from here?

— 
./

Re: [External] Re: QUIC traffic throttled on AT residential

[Ca By]

On Fri, Feb 21, 2020 at 2:22 PM Hunter Fuller  wrote:

> On Fri, Feb 21, 2020 at 2:42 PM Jared Mauch  wrote:
> > I can already hear the QUIC WG types blaming the network in abstentia,
> because well, why would an operator want to keep their network functioning?
> :-)
>
> In fairness, it's not actually functioning. For one thing, it's
> passing some traffic at an abysmal rate. ;)
>

Life and engineering are full of trade-offs.

Re: [External] Re: QUIC traffic throttled on AT residential

[Hunter Fuller]

On Fri, Feb 21, 2020 at 2:42 PM Jared Mauch  wrote:
> I can already hear the QUIC WG types blaming the network in abstentia, 
> because well, why would an operator want to keep their network functioning? 
> :-)

In fairness, it's not actually functioning. For one thing, it's
passing some traffic at an abysmal rate. ;)

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 21, 2020, at 2:22 PM, Dan Wing  wrote:
> 
> There are choices, such as making connection initiation, connection 
> acceptance, and connection termination parsable by network elements on the 
> path so state can be established, maintained, and cleared, DoS can be 
> identified, and so on.  The decision was to hide all that from network 
> elements.

I think those design choices lead directly to these known cases where a UDP 
policer is encountered.  I can drive along the road and not crash into the 
trees, but when I make a choice to deploy something without the proper safety 
equipment and it crashes, blaming the road seems like a poor response.

I can already hear the QUIC WG types blaming the network in abstentia, because 
well, why would an operator want to keep their network functioning? :-)

- Jared

Re: QUIC traffic throttled on AT residential

[Dan Wing]

There are choices, such as making connection initiation, connection acceptance, 
and connection termination parsable by network elements on the path so state 
can be established, maintained, and cleared, DoS can be identified, and so on.  
The decision was to hide all that from network elements.

-d


> On Feb 20, 2020, at 7:54 PM, Matthew Kaufman  wrote:
> 
> 
> 
> On Thu, Feb 20, 2020 at 8:10 AM Ca By  > wrote:
> 
> 
> Not indiscriminate. 
> 
> As Google was informed by network operators all along since 2014, ipv4 UDP is 
> a major uptime threat via DDoS to access networks.  
> ...
> 
> Google choose not to be sensitive to that, they were told where the landmines 
> were, they choose to go on anyhow. 
> 
> 
> It isn’t like they had a choice. You can’t build a transport protocol like 
> QUIC on top of TCP (I know, I built one of its ancestors, which also uses UDP 
> underneath). And if you think getting UDP through existing networks is hard, 
> try using a novel IP protocol number.
> 
> Matthew Kaufman
> 

Re: QUIC traffic throttled on AT residential

[Matthew Kaufman]

On Thu, Feb 20, 2020 at 8:10 AM Ca By  wrote:

>
>
> Not indiscriminate.
>
> As Google was informed by network operators all along since 2014, ipv4 UDP
> is a major uptime threat via DDoS to access networks.
> ...
>
> Google choose not to be sensitive to that, they were told where the
> landmines were, they choose to go on anyhow.
>
>
It isn’t like they had a choice. You can’t build a transport protocol like
QUIC on top of TCP (I know, I built one of its ancestors, which also uses
UDP underneath). And if you think getting UDP through existing networks is
hard, try using a novel IP protocol number.

Matthew Kaufman

>
>

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Sterling wrote:


A problem of QUIC with NAT is that existing NAT can not detect
graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.


Hmm, this is not what is happening.


I thought so. My point is that the problem can be another reason
to avoid QUIC.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Lukas Tribus wrote:


IPv6 UDP is currently not broken, that doesn't mean v6 is the solution
to this problem. It's just means the particular ISP did not yet deploy
the same policies or "mitigations" for v6 traffic.


It is more likely that the ISP does not support v6 at all.


In a much smaller eyeball environment (with
much smaller chokepoints), we have mapped possibly amplificated
packets (ip frag, dns, ntp, memcached, et all) to a specific queue.
Unless the links are congested, this traffic passes just as any other
traffic and during congestion it only uses whatever bandwidth the
queue has - no static rate-limits.


That is a bad idea.

Static rate limit is necessary to discourage DoS attackers.

If the attacker send 10Mbps stream to an amplifier and the stream
is redirected to a victim at 100Mbps, 10Mbps rate limiting negates
the amplification.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Lukas Tribus]

Hello,


On Thu, 20 Feb 2020 at 21:30, Daniel Sterling  wrote:
> As has been continually noted, this issue goes away if you use v4 TCP or v6 
> UDP.

IPv6 UDP is currently not broken, that doesn't mean v6 is the solution
to this problem. It's just means the particular ISP did not yet deploy
the same policies or "mitigations" for v6 traffic. As v6 adoption
increases, so will abuse/misuse, especially when attackers notice that
their attack traffic is rate-limited on v4 but not on v6 and P2P
gaming switches from v4 to v6. And at some point this will lead to
"feature parity" in IPv6. So while v6 UDP currently works, I don't
think we can assume that's permanent.

I disagree this approach is necessary to keep the network running and
the pagers from buzzing. In a much smaller eyeball environment (with
much smaller chokepoints), we have mapped possibly amplificated
packets (ip frag, dns, ntp, memcached, et all) to a specific queue.
Unless the links are congested, this traffic passes just as any other
traffic and during congestion it only uses whatever bandwidth the
queue has - no static rate-limits. I'm not saying this will fix
whatever the policies discussed here are supposed to fix, but there is
always a way to improve and make the mitigations more nuanced.

Of course ISPs will protect the network and the customers. But whether
you apply a simple rate-limiting for some traffic or some AI-assisted
auto-mitigation for traffic misuse, you gotta be prepared to monitor
and update it, staying on top of at least the major false-positives,
short-term but long-term as well. After all, things tend to change
over time.



lukas

Re: [External] Re: QUIC traffic throttled on AT residential

[Hunter Fuller]

On Thu, Feb 20, 2020 at 3:45 PM Jared Mauch  wrote:
> I can think of many legitimate cases, but i think this is where you have 
> internet for everyone and internet for the tech-savvy/business split that 
> becomes interesting.
>
> I’ve generally been willing to pay more for a business class service for 
> support and improved response SLA.  The average user isn’t going to detect 
> that 10% of their UDP has gone missing, nor should they be expected to.

I really hope my constituents don't have to get business class
connections just to get decent performance out of our services, such
as UDP based tunnels. They barely care what a VPN is, much less what
UDP is. And if our VPN software detects that UDP is available, it will
use it, so I suspect it would be (or is being) affected by this.

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 4:53 PM, Blake Hudson  wrote:
> 
> 
As a network operator my goal was always to ensure customers receive
 the traffic they expected, high rates of UDP were often not what they 
 wanted.
 
Adusting the limits may be useful but I still think the question of
 what rate of UDP traffic is acceptable is a practical one for the future.
 
- Jared
>>> I think that's a fair statement Jared. How about this question: Would it be 
>>> reasonable for one to presume that someone purchasing a 25Mbps internet 
>>> connection might potentially want to send or receive 25Mbps of UDP traffic? 
>>> I can think of a few (not uncommon) applications where this would be the 
>>> case (VPNs, security cameras using RTP, teleconferencing, web browsers 
>>> implementing QUIC, DNS servers, hosted PBX, etc).
>> I can think of many legitimate cases, but i think this is where you have 
>> internet for everyone and internet for the tech-savvy/business split that 
>> becomes interesting.
>> 
>> I’ve generally been willing to pay more for a business class service for 
>> support and improved response SLA.  The average user isn’t going to detect 
>> that 10% of their UDP has gone missing, nor should they be expected to.
>> 
>> - Jared
> And here I think is where one crosses the threshold between providing an 
> "internet connection" and providing a connection "that can be used to access 
> specific applications or services" (as defined by your provider). This is one 
> step away from your ISP selling you a connection to access Facebook, if you 
> want to access Twitter that's available on their premium package. Oh, you 
> want to access Slack, sorry we don't offer that as a package yet. Call back 
> in a month. You need to esss-esss-achhh? I've never heard of that, why would 
> you want to do that?

AT has rarely offered internet service, their required devices for their 
U-Verse often munged traffic.  I recall when you could reboot their boxes by 
sending SIP packets to devices behind them and it would intercept them and 
think it was for itself for their POTS service.

If you have any NAT/ALG in there, it’s not pure internet, but most people want 
access to the “web” and aren’t running ftp/finger/ytalk/uucp/sip etc.. This is 
why SSL VPNs on 443 became a thing over time.

- jared

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

I think that's a fair statement Jared. How about this question: Would it be 
reasonable for one to presume that someone purchasing a 25Mbps internet 
connection might potentially want to send or receive 25Mbps of UDP traffic? I 
can think of a few (not uncommon) applications where this would be the case 
(VPNs, security cameras using RTP, teleconferencing, web browsers implementing 
QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have 
internet for everyone and internet for the tech-savvy/business split that 
becomes interesting.

I’ve generally been willing to pay more for a business class service for 
support and improved response SLA.  The average user isn’t going to detect that 
10% of their UDP has gone missing, nor should they be expected to.

- Jared
And here I think is where one crosses the threshold between providing an 
"internet connection" and providing a connection "that can be used to 
access specific applications or services" (as defined by your provider). 
This is one step away from your ISP selling you a connection to access 
Facebook, if you want to access Twitter that's available on their 
premium package. Oh, you want to access Slack, sorry we don't offer that 
as a package yet. Call back in a month. You need to esss-esss-achhh? 
I've never heard of that, why would you want to do that?

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 4:42 PM, Blake Hudson  wrote:
> 
> 
> 
> On 2/20/2020 1:10 PM, Jared Mauch wrote:
>> On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
>>> On 2/20/2020 10:34 AM, Ca By wrote:
 On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson >>> > wrote:
 Dropping udp is not from a “best practice” doc from a vendor, it is
 deployed by network ops folks that are trying to sleep at night.
>>> I get it Ca, I happen to be one of those network ops folks that likes to
>>> sleep at night. However, I've never thought it was a good practice to break
>>> applications in fun ways for my customers to discover on their own and I've
>>> never sold someone a 150Mbps package that actually only delivers 10Mbps for
>>> certain applications. Regardless of the intent, ATT and Cox's policies are
>>> not transparent, open, or neutral on this topic. This leaves us to speculate
>>> on what their intentions might have been and whether their actions are an
>>> appropriate response to any concerns they might have had.
>>  I was responsible for deploying such policies in the past, going back as
>> far as the UDP/1434 filters I was forced to deploy due to persistent network
>> congestion.  Rolling these back took some time.
>> 
>>  The same is true for UDP policers we ended up rolling out for NTP, 
>> chargen
>> and other activities.
>> 
>>  Extending these to consumer side where the traffic often originated 
>> makes
>> sense until the devices can be secured.  You can blame the providers for 
>> deploying
>> filters, or not disconnecting consumers that have devices that can be 
>> exploited
>> or whatever other reason you believe.
>> 
>>  As a network operator my goal was always to ensure customers receive
>> the traffic they expected, high rates of UDP were often not what they wanted.
>> 
>>  Adusting the limits may be useful but I still think the question of
>> what rate of UDP traffic is acceptable is a practical one for the future.
>> 
>>  - Jared
> 
> I think that's a fair statement Jared. How about this question: Would it be 
> reasonable for one to presume that someone purchasing a 25Mbps internet 
> connection might potentially want to send or receive 25Mbps of UDP traffic? I 
> can think of a few (not uncommon) applications where this would be the case 
> (VPNs, security cameras using RTP, teleconferencing, web browsers 
> implementing QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have 
internet for everyone and internet for the tech-savvy/business split that 
becomes interesting.

I’ve generally been willing to pay more for a business class service for 
support and improved response SLA.  The average user isn’t going to detect that 
10% of their UDP has gone missing, nor should they be expected to.

- Jared

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 1:10 PM, Jared Mauch wrote:

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:

On 2/20/2020 10:34 AM, Ca By wrote:

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson mailto:bl...@ispn.net>> wrote:
Dropping udp is not from a “best practice” doc from a vendor, it is
deployed by network ops folks that are trying to sleep at night.

I get it Ca, I happen to be one of those network ops folks that likes to
sleep at night. However, I've never thought it was a good practice to break
applications in fun ways for my customers to discover on their own and I've
never sold someone a 150Mbps package that actually only delivers 10Mbps for
certain applications. Regardless of the intent, ATT and Cox's policies are
not transparent, open, or neutral on this topic. This leaves us to speculate
on what their intentions might have been and whether their actions are an
appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as
far as the UDP/1434 filters I was forced to deploy due to persistent network
congestion.  Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, 
chargen
and other activities.

Extending these to consumer side where the traffic often originated 
makes
sense until the devices can be secured.  You can blame the providers for 
deploying
filters, or not disconnecting consumers that have devices that can be exploited
or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared


I think that's a fair statement Jared. How about this question: Would it 
be reasonable for one to presume that someone purchasing a 25Mbps 
internet connection might potentially want to send or receive 25Mbps of 
UDP traffic? I can think of a few (not uncommon) applications where this 
would be the case (VPNs, security cameras using RTP, teleconferencing, 
web browsers implementing QUIC, DNS servers, hosted PBX, etc).

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 3:30 PM, Daniel Sterling  
> wrote:
> 
> On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch  wrote:
>> if the question is will the browser vendor (google) or the broadband 
>> provider (att)
>> move first, i can already predict the answer.  my experience (again) with 
>> the quic
>> wg is they seem to think there's many options and bad providers will be 
>> replaced
>> which seems disconnected from reality.
> 
> Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where
> there are multiple ISPs.
> 
> But not everyone is, and local-loop unbundling is no longer a thing :(
> 
> The average user is at the mercy of their local ISP. So the major
> providers should (but may not have much incentive to) provide decent
> service.
> 

Honestly, there’s part of me that wants to say be lucky you have a choice.

My options here are (1.5m DSL, fixed wireless ISP behind several layers of NAT,
Cellular data or dial-up).

“Watch this space” for my solution.  Several people know what it is, but
my solution isn’t for everyone.

- Jared

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch  wrote:
> if the question is will the browser vendor (google) or the broadband provider 
> (att)
> move first, i can already predict the answer.  my experience (again) with the 
> quic
> wg is they seem to think there's many options and bad providers will be 
> replaced
> which seems disconnected from reality.

Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where
there are multiple ISPs.

But not everyone is, and local-loop unbundling is no longer a thing :(

The average user is at the mercy of their local ISP. So the major
providers should (but may not have much incentive to) provide decent
service.

As has been continually noted, this issue goes away if you use v4 TCP or v6 UDP.

And one may argue that most users *will* by default have v6 UDP
connectivity from their ISP -- to Google at least.

But HTTP/3 is coming -- and site operators may not realize deploying
HTTP/3 on v4 only will break connectivity for their users.

We're currently in a limbo where v4, which had been working fine, has
been *broken* -- and not everyone uses v6.

Site operators aren't used to deploying services that break when
served over ipv4. HTTP/3 will be the first widely-deployed service
that only properly works when served from v6.

So something is going to give:

* site operators will have to deploy v6 before they deploy HTTP/3, or

* network operators will have to make v4 UDP as reliable as v4 TCP (or
block HTTP/3)

If neither happens, then as more sites start pushing v4 UDP, more and
more users will start seeing issues, even if they *do* have v6 working
well at their house.

It may become common practice to block HTTP/3 on the client side to
improve connectivity to v4-only services.

Exciting times lay ahead, indeed

-- Dan

Re: QUIC traffic throttled on AT residential

[Tom Beecher]

>
> i don't think you've addressed the "replace your broken ISP" action that
> is clearly sane and would fix this, right?
>

The sanity presumes two things:

A: That he could do so without having to change addresses as well.
(Something that is still all too true for much of the US.)
B: The other provider is not doing anything on their network that might
similarly be impacting this specific traffic. (But could very well be doing
something to impact other traffic.)


On Thu, Feb 20, 2020 at 2:53 PM Todd Underwood  wrote:

> and just to check one thing...
>
> On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
> wrote:
>
>> I don't particularly *want* to block or advocate blocking QUIC, but if
>> I keep hitting the issue and can't help people troubleshoot, what
>> other sane option have I?
>>
>
> i don't think you've addressed the "replace your broken ISP" action that
> is clearly sane and would fix this, right?
>
> i'm assuming that this is not an option to get a functional IP layer?
>
> t
>
>
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

On Thu, Feb 20, 2020 at 02:50:58PM -0500, Todd Underwood wrote:
> and just to check one thing...
> 
> On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
> wrote:
> 
> > I don't particularly *want* to block or advocate blocking QUIC, but if
> > I keep hitting the issue and can't help people troubleshoot, what
> > other sane option have I?
> >
> 
> i don't think you've addressed the "replace your broken ISP" action that is
> clearly sane and would fix this, right?
> 
> i'm assuming that this is not an option to get a functional IP layer?

often one has to live in the world of reality.  if google is very worried about
the broadband experience they can continue to invest in this space and 
challenges.

it seems the corporate overlords has led it to not continue to persue this 
option.

it's hard to have it both ways, wanting unlimited UDP, low cost (free?) ports 
for
bits, etc..

there's natural dynamics at play here with business interests that haven't quite
balanced out yet.  perhaps the LTE/5G overlay will replace fixed broadband at 
home
with managed CPE from the cellular providers.

if the question is will the browser vendor (google) or the broadband provider 
(att)
move first, i can already predict the answer.  my experience (again) with the 
quic
wg is they seem to think there's many options and bad providers will be replaced
which seems disconnected from reality.

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: QUIC traffic throttled on AT residential

[Todd Underwood]

and just to check one thing...

On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
wrote:

> I don't particularly *want* to block or advocate blocking QUIC, but if
> I keep hitting the issue and can't help people troubleshoot, what
> other sane option have I?
>

i don't think you've addressed the "replace your broken ISP" action that is
clearly sane and would fix this, right?

i'm assuming that this is not an option to get a functional IP layer?

t


>
> -- Dan
>

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Thu, Feb 20, 2020 at 2:11 PM Jared Mauch  wrote:
> As a network operator my goal was always to ensure customers receive
> the traffic they expected, high rates of UDP were often not what they wanted.

Well, I wouldn't say I *want* UDP traffic, but if everyone is bound
and determined to send it to me for web / video traffic unless I block
it, I suppose we should all work together to improve my (average v4
end-user) experience.

I set up a rolling tcpdump to capture QUIC / HTTP/3 traffic.

tcpdump -C 100 -w quic -W 100 -i eth1 'udp and port 443'

I can make this dump available for inspection if (when) I organically
experience the issue again.

Is there anything else I can do to help Google or AT improve things?
Any magic debugging I can turn on on the client side?

Feel free to ping me off list...

I don't particularly *want* to block or advocate blocking QUIC, but if
I keep hitting the issue and can't help people troubleshoot, what
other sane option have I?

-- Dan

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
> On 2/20/2020 10:34 AM, Ca By wrote:
> > 
> > On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson  > > wrote:
> > Dropping udp is not from a “best practice” doc from a vendor, it is
> > deployed by network ops folks that are trying to sleep at night.
> 
> I get it Ca, I happen to be one of those network ops folks that likes to
> sleep at night. However, I've never thought it was a good practice to break
> applications in fun ways for my customers to discover on their own and I've
> never sold someone a 150Mbps package that actually only delivers 10Mbps for
> certain applications. Regardless of the intent, ATT and Cox's policies are
> not transparent, open, or neutral on this topic. This leaves us to speculate
> on what their intentions might have been and whether their actions are an
> appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as
far as the UDP/1434 filters I was forced to deploy due to persistent network
congestion.  Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, 
chargen
and other activities.

Extending these to consumer side where the traffic often originated 
makes
sense until the devices can be secured.  You can blame the providers for 
deploying
filters, or not disconnecting consumers that have devices that can be exploited
or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 10:41 AM Dave Bell  wrote:

>
> Not indiscriminate.
>>
>
> Indiscriminate - done at random or without careful judgement.
>
> Considering that Daniel is complaining that QUIC is broken, it certainly
> seems like some network operators are subjecting all UDP traffic on their
> network to the same policers. This feels pretty indiscriminate to me.
>
> I'm all for policing the known baddies, such as CHARGEN and NTP, but to
> discard UDP for fun is like taking a sledgehammer where a scalpel will do.
>
>

For fun?

We are done here

Access networks need controls to maintain uptime against the non-stop
>> barrage of ddos attacks. I am sure you have seen the headlines and mails on
>> this list, ddos is hard to deal with. Access network will use whatever
>> tools are required to keep the pagers quiet and customers happy.
>>
>
> I operate an access network that does not blanket police UDP. Google give
> me a dashboard that tell me 45% of requests  were served happily by QUIC,
> and I have no customers complaining about things not working, and our
> pagers are silent.
>

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 10:41 AM, Dave Bell wrote:


Not indiscriminate.


Indiscriminate - done at random or without careful judgement.

Considering that Daniel is complaining that QUIC is broken, it 
certainly seems like some network operators are subjecting all UDP 
traffic on their network to the same policers. This feels pretty 
indiscriminate to me.


I'm all for policing the known baddies, such as CHARGEN and NTP, but 
to discard UDP for fun is like taking a sledgehammer where a scalpel 
will do.


Access networks need controls to maintain uptime against the
non-stop barrage of ddos attacks. I am sure you have seen the
headlines and mails on this list, ddos is hard to deal with.
Access network will use whatever tools are required to keep the
pagers quiet and customers happy.


I operate an access network that does not blanket police UDP. Google 
give me a dashboard that tell me 45% of requests  were served happily 
by QUIC, and I have no customers complaining about things not working, 
and our pagers are silent.



Dave, just wanted to say that I 100% agree with your comments. The bad 
actors are well known. I believe treating all UDP as bad is misguided. 
Like you, I assist in operation of several access networks that do not 
blanket police UDP and my pager remains relatively silent.

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 10:34 AM, Ca By wrote:


On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson > wrote:



Your comments seem to differentiate IP4 vs IP6, but I don't
believe that
is relevant to the issue of an ISP throttling or breaking specific
applications. If you have evidence that UDP on IP4 is treated
differently than UDP on IP6 by your provider, without further
information I would suspect that this is simply an unintentional over
sight on their part.



This is your misunderstanding. The protections are to drop ipv4 udp 
because that is where the ddos / iot trash is , not v6 for now




Perhaps the attention you've generated on this topic, along with the
adoption of additional UDP based applications like QUIC, will
encourage
ISPs to treat UDP in a more neutral manner and not simply see UDP as
something that is "bad".


Dropping udp is not from a “best practice” doc from a vendor, it is 
deployed by network ops folks that are trying to sleep at night.




I get it Ca, I happen to be one of those network ops folks that likes to 
sleep at night. However, I've never thought it was a good practice to 
break applications in fun ways for my customers to discover on their own 
and I've never sold someone a 150Mbps package that actually only 
delivers 10Mbps for certain applications. Regardless of the intent, ATT 
and Cox's policies are not transparent, open, or neutral on this topic. 
This leaves us to speculate on what their intentions might have been and 
whether their actions are an appropriate response to any concerns they 
might have had.

Re: QUIC traffic throttled on AT residential

[Dave Bell]

> Not indiscriminate.
>

Indiscriminate - done at random or without careful judgement.

Considering that Daniel is complaining that QUIC is broken, it certainly
seems like some network operators are subjecting all UDP traffic on their
network to the same policers. This feels pretty indiscriminate to me.

I'm all for policing the known baddies, such as CHARGEN and NTP, but to
discard UDP for fun is like taking a sledgehammer where a scalpel will do.


> Access networks need controls to maintain uptime against the non-stop
> barrage of ddos attacks. I am sure you have seen the headlines and mails on
> this list, ddos is hard to deal with. Access network will use whatever
> tools are required to keep the pagers quiet and customers happy.
>

I operate an access network that does not blanket police UDP. Google give
me a dashboard that tell me 45% of requests  were served happily by QUIC,
and I have no customers complaining about things not working, and our
pagers are silent.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson  wrote:

>
>
> On 2/19/2020 3:21 PM, Daniel Sterling wrote:
> > On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:
> >> Yeah, that was a nice surprise to find that my tethered LTE connection
> >> was out performing my wired cable modem service. Of course, I had
> >> already signed up for a year of service and there were early termination
> >> fees for cancelling... that and there are no other wireline providers
> >> available at my home (not even ATT).
> > So we're left with some questions:
> >
> > 1. It's clear I'm not the only one experiencing this issue. How
> > widespread is this problem, really? Has it gotten rather worse over
> > the past ~year?
> >
> > 2. Are customers of larger ISPs much more impacted than customers of
> > smaller ones that (assumedly) don't have to deprioritize UDP so much?
> > 2a. If users *are* impacted, as Blake notes, they may not be able to
> > switch ISPs to improve their lot.. will customers complain to their
> > ISP or to Google?
> >
> > 3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
> > only works on v6 (and if it in fact continues to actively BREAK
> > v4-only users), then is this v6's "killer app" that will drive
> > adoption?
> > 3a. Or will this issue hinder HTTP/3 deployment (or cause mass
> > blocking of UDP on clients)?
> >
> > 4. Will ISPs be willing to give UDP traffic higher priority to improve
> > user experience? Will that only happen once HTTP/3 is widely deployed?
> >
> > 5. We can only assume Google is aware of this issue; will Google work
> > to improve QUIC fallback to TCP, or will they work with ISPs to get
> > QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
> > actively encourage QUIC to break v4 at the expensive of current user
> > experience?
> > 5a. Will another company that wants HTTP/3 to succeed take the mantle
> > and work with ISPs to improve the situation? I'm reminded of when
> > Microsoft worked with ISPs to ensure xbox UDP traffic would transit
> > properly
> >
> > -- Dan
> Dan, my experience with Cox is that their standard cable internet
> package (advertised as 150Mbps) rate limits UDP to ~10Mbps. This appears
> to be controlled via the cable modem config file which is enforced by
> both the cable modem and the CMTS. I do not know if this is per flow or
> per circuit or affects IP4 differently than IP6. I suspect that someone
> at Cox decided that the only applications using UDP were VoIP and DNS
> and that those applications never needed more than 1Mbps so anything
> else must be "bad" and should be stopped. Whether "bad" means harmful to
> network operation, harmful to support costs, or harmful to profits, I do
> not know.
>
> Your comments seem to differentiate IP4 vs IP6, but I don't believe that
> is relevant to the issue of an ISP throttling or breaking specific
> applications. If you have evidence that UDP on IP4 is treated
> differently than UDP on IP6 by your provider, without further
> information I would suspect that this is simply an unintentional over
> sight on their part.



This is your misunderstanding. The protections are to drop ipv4 udp because
that is where the ddos / iot trash is , not v6 for now

>
>
> Perhaps the attention you've generated on this topic, along with the
> adoption of additional UDP based applications like QUIC, will encourage
> ISPs to treat UDP in a more neutral manner and not simply see UDP as
> something that is "bad".
>

Dropping udp is not from a “best practice” doc from a vendor, it is
deployed by network ops folks that are trying to sleep at night.


> --Blake
>

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/19/2020 3:21 PM, Daniel Sterling wrote:

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:

Yeah, that was a nice surprise to find that my tethered LTE connection
was out performing my wired cable modem service. Of course, I had
already signed up for a year of service and there were early termination
fees for cancelling... that and there are no other wireline providers
available at my home (not even ATT).

So we're left with some questions:

1. It's clear I'm not the only one experiencing this issue. How
widespread is this problem, really? Has it gotten rather worse over
the past ~year?

2. Are customers of larger ISPs much more impacted than customers of
smaller ones that (assumedly) don't have to deprioritize UDP so much?
2a. If users *are* impacted, as Blake notes, they may not be able to
switch ISPs to improve their lot.. will customers complain to their
ISP or to Google?

3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
only works on v6 (and if it in fact continues to actively BREAK
v4-only users), then is this v6's "killer app" that will drive
adoption?
3a. Or will this issue hinder HTTP/3 deployment (or cause mass
blocking of UDP on clients)?

4. Will ISPs be willing to give UDP traffic higher priority to improve
user experience? Will that only happen once HTTP/3 is widely deployed?

5. We can only assume Google is aware of this issue; will Google work
to improve QUIC fallback to TCP, or will they work with ISPs to get
QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
actively encourage QUIC to break v4 at the expensive of current user
experience?
5a. Will another company that wants HTTP/3 to succeed take the mantle
and work with ISPs to improve the situation? I'm reminded of when
Microsoft worked with ISPs to ensure xbox UDP traffic would transit
properly

-- Dan
Dan, my experience with Cox is that their standard cable internet 
package (advertised as 150Mbps) rate limits UDP to ~10Mbps. This appears 
to be controlled via the cable modem config file which is enforced by 
both the cable modem and the CMTS. I do not know if this is per flow or 
per circuit or affects IP4 differently than IP6. I suspect that someone 
at Cox decided that the only applications using UDP were VoIP and DNS 
and that those applications never needed more than 1Mbps so anything 
else must be "bad" and should be stopped. Whether "bad" means harmful to 
network operation, harmful to support costs, or harmful to profits, I do 
not know.


Your comments seem to differentiate IP4 vs IP6, but I don't believe that 
is relevant to the issue of an ISP throttling or breaking specific 
applications. If you have evidence that UDP on IP4 is treated 
differently than UDP on IP6 by your provider, without further 
information I would suspect that this is simply an unintentional over 
sight on their part.


Perhaps the attention you've generated on this topic, along with the 
adoption of additional UDP based applications like QUIC, will encourage 
ISPs to treat UDP in a more neutral manner and not simply see UDP as 
something that is "bad".


--Blake

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 9:56 AM Dave Bell  wrote:

>
> On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:
>
>> UDP is broken
>>
>
> I would argue that UDP isn't broken. Networks which drop it
> indiscriminately are broken.
>
Not indiscriminate.

As Google was informed by network operators all along since 2014, ipv4 UDP
is a major uptime threat via DDoS to access networks.

Access networks need controls to maintain uptime against the non-stop
barrage of ddos attacks. I am sure you have seen the headlines and mails on
this list, ddos is hard to deal with. Access network will use whatever
tools are required to keep the pagers quiet and customers happy.

Google choose not to be sensitive to that, they were told where the
landmines were, they choose to go on anyhow.

Re: QUIC traffic throttled on AT residential

[Aled Morris via NANOG]

On Thu, 20 Feb 2020 at 15:57, Dave Bell  wrote:

>
> On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:
>
>> UDP is broken
>>
>
> I would argue that UDP isn't broken. Networks which drop it
> indiscriminately are broken.
>

Does this errant network behaviour not impact RTP applications like video
streams?

Aled

Re: Re: QUIC traffic throttled on AT residential {5403687}

[Dave Bell]

I didn't contact you. Fuck off.

On Thu, 20 Feb 2020 at 16:01, Dead.net Customer Service <
d...@wmgcustomerservice.com> wrote:

> Thank you for contacting Dead.net customer service.
>
> Our customer service team will reply to your email as soon as possible.
>
> Due to our current email volume, replies may take up to 2 business days.
>
>  We apologize for the inconvenience and thank you for your patience.
>
>  Sincerely,
>
>  The Dead.net Customer Service Team
>
>
>

Re: QUIC traffic throttled on AT residential

[Dave Bell]

On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:

> UDP is broken
>

I would argue that UDP isn't broken. Networks which drop it
indiscriminately are broken.

RE: QUIC traffic throttled on AT residential

[Keith Medcalf]

On Thursday, 20 February, 2020 08:31, Ca By  wrote:

>On Thu, Feb 20, 2020 at 8:34 AM Tom Beecher  wrote:

>   I only wish I were insane; but from where I'm sitting, QUIC
>has broken
>   my internet, and the resolution is blocking QUIC.
>
>   The QUIC protocol itself isn't breaking anything ; some middlebox is
>breaking QUIC. It's likely collateral damage from honest attempts to
>mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
>degree, but the upstream issue still remains.

>   I recall reading a draft from the WG about having things open a
>parallel TCP connection in case UDP got horked for seamless fallback, but
>I don't remember if it ever moved past that, or if anyone actually
>implemented it.

>UDP is broken
>
>The Google devs said it would in fine in 2014
>
>They said it would be “exciting”

UDP is not "broken".  UDP is the UNRELIABLE DATAGRAM PROTOCOL and it is living 
up to its name!  What is broken is something pretending that the "U" in UDP 
means something other than UNRELIABLE and then pretending their use of an 
unreliable delivery mechanism makes it reliable when that clearly is not the 
case.

In other words, applying a coat of paint to a sows ear to make it look like a 
silk purse does not make the sows ear into a silk purse.  It is still just a 
sows ear painted to look like a silk purse, and still behaves like a sows ear.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 8:34 AM Tom Beecher  wrote:

> I only wish I were insane; but from where I'm sitting, QUIC has broken
>> my internet, and the resolution is blocking QUIC.
>>
>
> The QUIC protocol itself isn't breaking anything ; some middlebox is
> breaking QUIC. It's likely collateral damage from honest attempts to
> mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
> degree, but the upstream issue still remains.
>
> I recall reading a draft from the WG about having things open a
> parallel TCP connection in case UDP got horked for seamless fallback, but I
> don't remember if it ever moved past that, or if anyone actually
> implemented it.
>
>
UDP is broken

The Google devs said it would in fine in 2014

They said it would be “exciting”


https://groups.google.com/a/chromium.org/forum/m/#!msg/proto-quic/09L5YD2u5xU/dK6oU2UA67YJ



> On Wed, Feb 19, 2020 at 11:33 PM Daniel Sterling <
> sterling.dan...@gmail.com> wrote:
>
>> On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
>>  wrote:
>> > A problem of QUIC with NAT is that existing NAT can not detect
>> > graceful shutdown of QUIC and must depends on timeout.
>> >
>> > So, port numbers may be used up before timeout.
>>
>> Hmm, this is not what is happening.
>>
>> I managed to (fairly easily!) reproduce the issue earlier tonight. I
>> generated a fair bit of UDP traffic via xbox, a corporate VPN, and
>> youtube over quic.
>>
>> Sure enough, after about 45 minutes, the YouTube app on my iPad paused
>> and then auto-reset the stream quality to "144p" resolution.
>>
>> I was able to set it back to 720p60, but that only lasted about 2
>> minutes before the stream stopped playing. I waited several minutes
>> for it to resume; it did not. So, I then blocked UDP 443 on my router.
>> The video then *immediately* resumed at 720p60.
>>
>> I didn't run tcpdump but I did grab some screenshots of iftop. It
>> looks like my iPad connected to AS15169 with a single UDP connection.
>> I see one consistent source and dest IP / port combo for those 10s of
>> minutes, up until UDP is blocked. Local port 58053, remote port 443 on
>> the same IP for the whole time.
>>
>> At first the connection averages around 2mbps; when the issue occurs,
>> I see it has dropped to a rate of under 200kbps.
>>
>> I've no idea what to make of that. Surely Google isn't throttling
>> their traffic to me? If so why allow a fallback to TCP?
>>
>> When I originally discovered this issue, I was of course not trying to
>> do anything odd with my internet connection. And I didn't immediately
>> know QUIC was the issue.
>>
>> Only after it happened several times did I dig into the traffic and
>> then block QUIC, and I was shocked to see that both resolve the issue
>> and prevent its recurrence.
>>
>> So again -- I hit this issue repeatedly without trying to --
>>
>> And just now, I was able to reproduce it simply by generating a bit of
>> UDP traffic on purpose!
>>
>> I only wish I were insane; but from where I'm sitting, QUIC has broken
>> my internet, and the resolution is blocking QUIC.
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Tom Beecher]

>
> I only wish I were insane; but from where I'm sitting, QUIC has broken
> my internet, and the resolution is blocking QUIC.
>

The QUIC protocol itself isn't breaking anything ; some middlebox is
breaking QUIC. It's likely collateral damage from honest attempts to
mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
degree, but the upstream issue still remains.

I recall reading a draft from the WG about having things open a
parallel TCP connection in case UDP got horked for seamless fallback, but I
don't remember if it ever moved past that, or if anyone actually
implemented it.


On Wed, Feb 19, 2020 at 11:33 PM Daniel Sterling 
wrote:

> On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
>  wrote:
> > A problem of QUIC with NAT is that existing NAT can not detect
> > graceful shutdown of QUIC and must depends on timeout.
> >
> > So, port numbers may be used up before timeout.
>
> Hmm, this is not what is happening.
>
> I managed to (fairly easily!) reproduce the issue earlier tonight. I
> generated a fair bit of UDP traffic via xbox, a corporate VPN, and
> youtube over quic.
>
> Sure enough, after about 45 minutes, the YouTube app on my iPad paused
> and then auto-reset the stream quality to "144p" resolution.
>
> I was able to set it back to 720p60, but that only lasted about 2
> minutes before the stream stopped playing. I waited several minutes
> for it to resume; it did not. So, I then blocked UDP 443 on my router.
> The video then *immediately* resumed at 720p60.
>
> I didn't run tcpdump but I did grab some screenshots of iftop. It
> looks like my iPad connected to AS15169 with a single UDP connection.
> I see one consistent source and dest IP / port combo for those 10s of
> minutes, up until UDP is blocked. Local port 58053, remote port 443 on
> the same IP for the whole time.
>
> At first the connection averages around 2mbps; when the issue occurs,
> I see it has dropped to a rate of under 200kbps.
>
> I've no idea what to make of that. Surely Google isn't throttling
> their traffic to me? If so why allow a fallback to TCP?
>
> When I originally discovered this issue, I was of course not trying to
> do anything odd with my internet connection. And I didn't immediately
> know QUIC was the issue.
>
> Only after it happened several times did I dig into the traffic and
> then block QUIC, and I was shocked to see that both resolve the issue
> and prevent its recurrence.
>
> So again -- I hit this issue repeatedly without trying to --
>
> And just now, I was able to reproduce it simply by generating a bit of
> UDP traffic on purpose!
>
> I only wish I were insane; but from where I'm sitting, QUIC has broken
> my internet, and the resolution is blocking QUIC.
>
> -- Dan
>

Re: QUIC traffic throttled on AT residential

[Jay Hennigan]

On 2/18/20 18:40, nanog-l...@contactdaniel.net wrote:

Growing prevalence of IPv6-only 
sites is probably the only thing that will get a lot of access networks 
to support v6.


I recall a similar idea called "The Great IPv6 Experiment" back in 2007. ;-)


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
 wrote:
> A problem of QUIC with NAT is that existing NAT can not detect
> graceful shutdown of QUIC and must depends on timeout.
>
> So, port numbers may be used up before timeout.

Hmm, this is not what is happening.

I managed to (fairly easily!) reproduce the issue earlier tonight. I
generated a fair bit of UDP traffic via xbox, a corporate VPN, and
youtube over quic.

Sure enough, after about 45 minutes, the YouTube app on my iPad paused
and then auto-reset the stream quality to "144p" resolution.

I was able to set it back to 720p60, but that only lasted about 2
minutes before the stream stopped playing. I waited several minutes
for it to resume; it did not. So, I then blocked UDP 443 on my router.
The video then *immediately* resumed at 720p60.

I didn't run tcpdump but I did grab some screenshots of iftop. It
looks like my iPad connected to AS15169 with a single UDP connection.
I see one consistent source and dest IP / port combo for those 10s of
minutes, up until UDP is blocked. Local port 58053, remote port 443 on
the same IP for the whole time.

At first the connection averages around 2mbps; when the issue occurs,
I see it has dropped to a rate of under 200kbps.

I've no idea what to make of that. Surely Google isn't throttling
their traffic to me? If so why allow a fallback to TCP?

When I originally discovered this issue, I was of course not trying to
do anything odd with my internet connection. And I didn't immediately
know QUIC was the issue.

Only after it happened several times did I dig into the traffic and
then block QUIC, and I was shocked to see that both resolve the issue
and prevent its recurrence.

So again -- I hit this issue repeatedly without trying to --

And just now, I was able to reproduce it simply by generating a bit of
UDP traffic on purpose!

I only wish I were insane; but from where I'm sitting, QUIC has broken
my internet, and the resolution is blocking QUIC.

-- Dan

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Sterling wrote:


I received a comment that maybe the issue is not AT's "core"
network, but rather to do with the NAT device in my house.


A problem of QUIC with NAT is that existing NAT can not detect
graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Mark Andrews]

> On 20 Feb 2020, at 11:29, Daniel Sterling  wrote:
> 
> On Tue, Feb 18, 2020 at 11:47 PM Daniel Sterling
>  wrote:
>> random-source-port UDP traffic does not impress the AT network flow
>> control systems, and your DNS traffic becomes unbearably slow (or is
> 
> I received a comment that maybe the issue is not AT's "core"
> network, but rather to do with the NAT device in my house.
> 
> Oh, I wish this were the case!
> 
> Unfortunately, there is no NAT CPE from AT involved:
> 
> I've taken AT's RG CPE out completely. That device, which otherwise
> would indeed be my gateway, is unused on my home network, completely
> unpowered and unplugged.
> 
> Instead I've got a linux box hooked directly up to the ONT. This works
> fine; occasionally I *do* have do reconnect their RG and let it
> re-auth to the PON, but once the connection is live, I fully unplug
> their RG again, and plug my laptop (spoofing its MAC) back in directly
> to the ONT.
> 
> The laptop is running ubuntu 19.10, so there should be no artificial
> limits. It's running NAT directly from the v4 IP I get from DHCP.
> 
> 
> You may have noticed v4 is a theme here. I have nothing against using
> v6 -- , I must admit the truth is I have no idea how to make ubuntu
> acquire a v6 -- address? block ? I don't even know the right term --
> from uverse.

It should just be a DHCPv6 PREFIX DELEGATION (PD).  See RFC 8415.

If AT are still using 6RD (IPv6 Rapid Deployment) that is described in
RFC 5969.  There is a DHCPv4 option that you can request that gives
you the details for configuring the IPv6 in IPv4 tunnel and how to
compute the IPv6 prefix from the allocated IPv4 address.

CPE’s just try both and use DHCPv6 PD if both exist.

> I know v6 works cuz AT's device supports it, and openwrt does it out
> of the box-- but heck if I know how it works. At the risk of asking
> this list for tech support -- does anyone want to ping me off list and
> point me in the right direction?? Maybe somebody from Google -- you
> can make up for breaking my v4 internet!!
> 
> Thanks,
> Dan

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 11:47 PM Daniel Sterling
 wrote:
> random-source-port UDP traffic does not impress the AT network flow
> control systems, and your DNS traffic becomes unbearably slow (or is

I received a comment that maybe the issue is not AT's "core"
network, but rather to do with the NAT device in my house.

Oh, I wish this were the case!

Unfortunately, there is no NAT CPE from AT involved:

I've taken AT's RG CPE out completely. That device, which otherwise
would indeed be my gateway, is unused on my home network, completely
unpowered and unplugged.

Instead I've got a linux box hooked directly up to the ONT. This works
fine; occasionally I *do* have do reconnect their RG and let it
re-auth to the PON, but once the connection is live, I fully unplug
their RG again, and plug my laptop (spoofing its MAC) back in directly
to the ONT.

The laptop is running ubuntu 19.10, so there should be no artificial
limits. It's running NAT directly from the v4 IP I get from DHCP.


You may have noticed v4 is a theme here. I have nothing against using
v6 -- , I must admit the truth is I have no idea how to make ubuntu
acquire a v6 -- address? block ? I don't even know the right term --
from uverse.

I know v6 works cuz AT's device supports it, and openwrt does it out
of the box-- but heck if I know how it works. At the risk of asking
this list for tech support -- does anyone want to ping me off list and
point me in the right direction?? Maybe somebody from Google -- you
can make up for breaking my v4 internet!!

Thanks,
Dan

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:
> Yeah, that was a nice surprise to find that my tethered LTE connection
> was out performing my wired cable modem service. Of course, I had
> already signed up for a year of service and there were early termination
> fees for cancelling... that and there are no other wireline providers
> available at my home (not even ATT).

So we're left with some questions:

1. It's clear I'm not the only one experiencing this issue. How
widespread is this problem, really? Has it gotten rather worse over
the past ~year?

2. Are customers of larger ISPs much more impacted than customers of
smaller ones that (assumedly) don't have to deprioritize UDP so much?
2a. If users *are* impacted, as Blake notes, they may not be able to
switch ISPs to improve their lot.. will customers complain to their
ISP or to Google?

3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
only works on v6 (and if it in fact continues to actively BREAK
v4-only users), then is this v6's "killer app" that will drive
adoption?
3a. Or will this issue hinder HTTP/3 deployment (or cause mass
blocking of UDP on clients)?

4. Will ISPs be willing to give UDP traffic higher priority to improve
user experience? Will that only happen once HTTP/3 is widely deployed?

5. We can only assume Google is aware of this issue; will Google work
to improve QUIC fallback to TCP, or will they work with ISPs to get
QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
actively encourage QUIC to break v4 at the expensive of current user
experience?
5a. Will another company that wants HTTP/3 to succeed take the mantle
and work with ISPs to improve the situation? I'm reminded of when
Microsoft worked with ISPs to ensure xbox UDP traffic would transit
properly

-- Dan

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

If by targeting DDoSes you mean rate limiting all UDP (because some UDP 
is bad), then I would expect that policy to be published on a provider's 
website so that folks using UDP would at least have the option to 
research and know this info before signing up or agreeing to a service 
commitment. Similarly, application developers could point to a 
provider's policy when designing their applications or assisting users. 
Even if Net Neutrality is dead, I believe the "Restoring Internet 
Freedom" order is still a thing in the US.



On 2/19/2020 3:03 PM, Mike Hammett wrote:
Net Neutrality likely wouldn't have impacted this at all. AT isn't 
targeting QUIC, they're targeting DDoSes.




-
Mike Hammett
Intelligent Computing Solutions <http://www.ics-il.com/>
<https://www.facebook.com/ICSIL><https://plus.google.com/+IntelligentComputingSolutionsDeKalb><https://www.linkedin.com/company/intelligent-computing-solutions><https://twitter.com/ICSIL>
Midwest Internet Exchange <http://www.midwest-ix.com/>
<https://www.facebook.com/mdwestix><https://www.linkedin.com/company/midwest-internet-exchange><https://twitter.com/mdwestix>
The Brothers WISP <http://www.thebrotherswisp.com/>
<https://www.facebook.com/thebrotherswisp><https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

*From: *"Brian J. Murrell" 
*To: *nanog@nanog.org
*Sent: *Wednesday, February 19, 2020 3:01:20 PM
*Subject: *Re: QUIC traffic throttled on AT residential

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:
>
> Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

> So that people (or
> companies) are free to develop new applications or enhance existing
> ones
> without running into a quagmire of different policies implemented by
> any
> number of different networks between the application developer and
> the
> application's users?

Yes, this is a very prominent reason for Net Neutrality.  Too bad the
FCC killed that out from under the people and companies that would
utilize it to develop new applications.

Cheers,
b.

Re: QUIC traffic throttled on AT residential

[Mike Hammett]

Net Neutrality likely wouldn't have impacted this at all. AT isn't targeting 
QUIC, they're targeting DDoSes. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Brian J. Murrell"  
To: nanog@nanog.org 
Sent: Wednesday, February 19, 2020 3:01:20 PM 
Subject: Re: QUIC traffic throttled on AT residential 

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote: 
> 
> Isn't this exactly why Net Neutrality is a thing: 

Isn't it a "dead" thing in the USofA? 

> So that people (or 
> companies) are free to develop new applications or enhance existing 
> ones 
> without running into a quagmire of different policies implemented by 
> any 
> number of different networks between the application developer and 
> the 
> application's users? 

Yes, this is a very prominent reason for Net Neutrality. Too bad the 
FCC killed that out from under the people and companies that would 
utilize it to develop new applications. 

Cheers, 
b. 

Re: QUIC traffic throttled on AT residential

[Brian J. Murrell]

On Wed, 2020-02-19 at 13:54 -0600, Blake Hudson wrote:
> 
> Isn't this exactly why Net Neutrality is a thing:

Isn't it a "dead" thing in the USofA?

> So that people (or 
> companies) are free to develop new applications or enhance existing
> ones 
> without running into a quagmire of different policies implemented by
> any 
> number of different networks between the application developer and
> the 
> application's users?

Yes, this is a very prominent reason for Net Neutrality.  Too bad the
FCC killed that out from under the people and companies that would
utilize it to develop new applications.

Cheers,
b.



signature.asc
Description: This is a digitally signed message part

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/19/2020 2:01 PM, Daniel Sterling wrote:

On Wed, Feb 19, 2020 at 2:55 PM Blake Hudson  wrote:

I'm guessing ATT doesn't disclose this policy transparently either.

they disclose it pretty transparently to their customers in the form
of very slow youtube traffic when using v4 QUIC ;)


Yeah, that was a nice surprise to find that my tethered LTE connection 
was out performing my wired cable modem service. Of course, I had 
already signed up for a year of service and there were early termination 
fees for cancelling... that and there are no other wireline providers 
available at my home (not even ATT).

Re: QUIC traffic throttled on AT residential

[Brandon Martin]

On 2/19/20 2:54 PM, Fred Baker wrote:

The argument I have heard is that residential firewalls often block anything 
that is*not*  UDP or TCP. The question for the googlers was existential - can 
it work at all?


I'm not sure that they "block" it, per se, though some probably do have 
an explicit rule to that effect.  I would think the bigger issue is that 
they don't know how to 1:N NAT arbitrary L4s (and how would they), so 
the absolute best you might get is that the first device behind the NAT 
to establish a mapping sees all the relevant L4 traffic and everybody 
else is locked out.  I'd suspect the normal case is simply that they 
drop it on the floor unless there's a specified "DMZ" host.


Perhaps this is just a semantic difference, but I think it's actually an 
even more difficult issue to resolve.  If it were simply blocked, that's 
usually "easy" (either for the user, via a management interface, or for 
the vendor, via policy template) to fix.  Writing an entirely new L4 NAT 
helper is a different matter entirely.


IPv6 would of course render this moot, but we all know how well IPv6 
traffic gets treated...

--
Brandon Martin

Re: QUIC traffic throttled on AT residential

[Michael Brown]

On 2020-02-19 1:06 a.m., Masataka Ohta wrote:
> Are you saying AT should block UDP entirely?

No; while I don't presume to have all the answers they should at the
minimum take into account how it affects the end-user (CUSTOMER!)
experience when making decisions like this.

(No they shouldn't block UDP entirely, but if they're having UDP/443
DDOS problems then blocking it while they get a proper scalable solution
in place is better than throttling it).

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 2:55 PM Blake Hudson  wrote:
> I'm guessing ATT doesn't disclose this policy transparently either.

they disclose it pretty transparently to their customers in the form
of very slow youtube traffic when using v4 QUIC ;)

Re: QUIC traffic throttled on AT residential

[Fred Baker]

> On Feb 18, 2020, at 4:00 PM, Ca By  wrote:
> 
> I am not a fan of quic or any udp traffic. My suggestion was that Google use 
> an new L4 instead of UDP, but that was too hard for the Googlers.

The argument I have heard is that residential firewalls often block anything 
that is *not* UDP or TCP. The question for the googlers was existential - can 
it work at all?


signature.asc
Description: Message signed with OpenPGP

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/18/2020 6:00 PM, Ca By wrote:
On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
mailto:sterling.dan...@gmail.com>> wrote:


I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
from google (esp. youtube) becomes very slow after a time.

This especially occurs with ipv4 connections. I'm not the only one to
notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
notes the issue.

I assume traffic is being throttled on AT's side. And it's not done
with their CPE; I'm bypassing their NAT box and connecting my laptop
directly to the ONT.

A quick google search shows people are aware that QUIC can look like
DoS traffic -- but how widely known is this problem? It may become
worse if / when sites transition to HTTP/3

For now I reject UDP 443 on the client firewall. Applications using
QUIC client libraries then fallback to TCP. This works well and I have
no issues with latency or throughput when using TCP.

May I naively ask if Google staff are working with AT to address
this?


Sounds like you found the answer, ATT just needs to scale up your 
rejection approach that is proven to work well.


Yes, most ddos traffic is ipv4 udp and yes Google was made aware that 
they would be mixing with bad company in the pool of ipv4 udp traffic 
... but they have reasons.


I am not a fan of quic or any udp traffic. My suggestion was that 
Google use an new L4 instead of UDP, but that was too hard for the 
Googlers.


So here we are.

Just say no to udp.


Isn't this exactly why Net Neutrality is a thing: So that people (or 
companies) are free to develop new applications or enhance existing ones 
without running into a quagmire of different policies implemented by any 
number of different networks between the application developer and the 
application's users?


For what it's worth, Cox cable also treats UDP traffic differently - 
impacting the performance of VPNs using UDP. Cox provides a list of 
blocked ports on their website, but makes no mention that they throttle 
or otherwise treat UDP as a special case. I'm guessing ATT doesn't 
disclose this policy transparently either.

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Christopher Morrow wrote:


2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.


Subject suggests it's retail ISP to homes, which are unlikely to
be multihomed.


It probably depends on where you want to do such limiting, right?
"At peering/transit edge" - save your core, dont' carry traffic you
"know" you will throw away anyway.
"At the customer edge" - scaling of state management could be
problematic && you'll carry this 'bad' traffic across your network.


Limiting is not by edges but by ISPs.

In transit ISPs, there are numerous flows generated by customers
of other ISPs that customer wise rate limiting does not scale.

In access ISPs, revenue increases proportional to the number
of customers that you can prepare rate limiting devices proportional
to the number of customers to make customer wise rate limiting scale.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

On Wed, Feb 19, 2020 at 3:18 AM Masataka Ohta
 wrote:
>
> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.

It probably depends on where you want to do such limiting, right?
"At peering/transit edge" - save your core, dont' carry traffic you
"know" you will throw away anyway.
"At the customer edge" - scaling of state management could be
problematic && you'll carry this 'bad' traffic across your network.

Note: I'm not really casting aspersions on either part of the whole
argument here, just attempting to provide some color to the parts
which seem 'unlikely to be successful'

I think for a bunch of this discussion there are N folks with M goals,
and eventually 'the market will dictate' which final direction we
travel.

Re: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

"yes, also received"
"thannks Toma!" (and nanog owner)

On Wed, Feb 19, 2020 at 4:55 AM Töma Gavrichenkov  wrote:
>
> Peace,
>
> nanog-ow...@nanog.org
>
> On Wed, Feb 19, 2020 at 12:51 PM Dave Bell  wrote:
> > Is anyone else receiving this spam?
> Yes
>
> > Is there a better way to report this?
>
> nanog-ow...@nanog.org (CC'd) helped me in the past.
>
> --
> Töma

Re: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Töma Gavrichenkov]

Peace,

nanog-ow...@nanog.org

On Wed, Feb 19, 2020 at 12:51 PM Dave Bell  wrote:
> Is anyone else receiving this spam?
Yes

> Is there a better way to report this?

nanog-ow...@nanog.org (CC'd) helped me in the past.

--
Töma

Fwd: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on AT residential

[Dave Bell]

Is anyone else receiving this spam?

Is there a better way to report this? I couldn't see anything from a quick
look through the various list pages.

Dave

-- Forwarded message -
From: Electric Forest Festival 
Date: Wed, 19 Feb 2020 at 09:42
Subject: Forest HQ Has Received Your Message: Re: QUIC traffic throttled on
AT residential
To: 



*Electric Forest 2020 will take place on June 25-28, 2020.*

Forest HQ has received your email. Help save precious resources by
reviewing the information below and looking up common questions in The
Forest Frequently Asked Questions: Experience.ElectricForestFestival.com
<https://madisonhousepresents.freshdesk.com/a/admin/automations/ticket_creation/2569347/Experience.ElectricForestFestival.com>

Please contact Festival Ticketing Support at 855-279-6941 for all issue
regarding your purchase or for account troubleshooting.

*Electric Forest is sold out. *Lyte is the only HQ endorsed way to get
passes now that it’s sold out.
<https://electricforestfestival.com/wristband-exchange>

To know when all things Electric Forest 2020 are happening sign up to the
EF Newsletter. <https://electricforestfestival.com/newsletter>

*Happy Forest!*

118323:553794

Re: QUIC traffic throttled on AT residential

[Dave Bell]

On Wed, 19 Feb 2020 at 08:18, Masataka Ohta <
mo...@necom830.hpcl.titech.ac.jp> wrote:

> Christopher Morrow wrote:
>
> > 2 way flow means something on your home host or home gateway.
> > It means very little at internet scale... since, in many cases, you ->
> > server and server -> you are not sharing many of the same links /
> > routers / etc.
>
> Subject suggests it's retail ISP to homes, which are unlikely to
> be multihomed.
>
>
>
My network has multiple ingress/egress points, and I do my filtering on the
edge.

The Chris's statement applies to my retail home customers too.

Dave

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Christopher Morrow wrote:


2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.


Subject suggests it's retail ISP to homes, which are unlikely to
be multihomed.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Dent wrote:

The explanation I got (which seems fair) from someone was that they 
only way to roll a new transport was to squat on some existing stuff 
that would make it through firewalls.

If there's clearly a two-way flow occurring,


Clearly, it should be DOS amplification.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Christopher Morrow]

On Wed, Feb 19, 2020 at 1:28 AM Daniel Dent
 wrote:
>
> On 2020-02-18 4:25 p.m., Jared Mauch wrote:
> > The members of the QUIC WG at IETF have not thought this was a problem as 
> > they don't observe it across the board.
> >
> > The cost for payloads with QUIC is much higher on the CPU side vs TCP as 
> > well - this is also not considered an issue either.

Jared, you mean: "Cost on the server" here not "cost on the router".
I think, and I ask because at least some of Daniel's note implies, to
my reading, that 'on the router' was your concern?

> There's plenty of room for system call/interface improvements and
> hardware acceleration in UDP stacks, both of which should help with CPU
> concerns. Now that UDP may represent a significant portion of internet
> traffic, it will be easier for the necessary engineering expense to be
> justified.
> > The explanation I got (which seems fair) from someone was that they only 
> > way to roll a new transport was to squat on some existing stuff that would 
> > make it through firewalls.
> If there's clearly a two-way flow occurring, i.e. as is the case with a

2 way flow means something on your home host or home gateway.
It means very little at internet scale... since, in many cases, you ->
server and server -> you are not sharing many of the same links /
routers / etc.

-chris

Re: QUIC traffic throttled on AT residential

[Daniel Dent]

On 2020-02-18 4:25 p.m., Jared Mauch wrote:

The members of the QUIC WG at IETF have not thought this was a problem as they 
don't observe it across the board.

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - 
this is also not considered an issue either.
There's plenty of room for system call/interface improvements and 
hardware acceleration in UDP stacks, both of which should help with CPU 
concerns. Now that UDP may represent a significant portion of internet 
traffic, it will be easier for the necessary engineering expense to be 
justified.

The explanation I got (which seems fair) from someone was that they only way to 
roll a new transport was to squat on some existing stuff that would make it 
through firewalls.
If there's clearly a two-way flow occurring, i.e. as is the case with a 
stream of QUIC traffic, that is a clearly distinguishable case from a 
DoS where the recipient is going to be dropping traffic. While this may 
be considerably harder to implement at scale than simply throttling UDP 
indiscriminately, hopefully there can be enough user suffering that AT 
feels compelled to do the right thing and allow the internet to continue 
to develop new protocols. Not everything fits neatly into a 
circuit-switched head-of-line-blocking request-response/HTTP paradigm.

We have an internet that is largely fixed around running NATs and TCP and UDP 
only these days. I for one find this sad and don't see a good way forward on 
either side.


Hopefully situations like this where Google swings their Chrome hammer 
for good instead of for evil can help get us there... now if we can just 
get those 100 gigabyte game downloads to get delivered over UDP too...


---

Daniel Dent

https://www.danieldent.com

Re: QUIC traffic throttled on AT residential

[Töma Gavrichenkov]

Peace,

On Wed, Feb 19, 2020 at 7:49 AM Daniel Sterling
 wrote:

> May I naively ask if Google staff have considered scrapping using UDP
> and instead proposing a new, first-class transport protocol that OSes
> can implement on top of IP?

The IETF WG did, at some point.  The opinion overall I think is that
this would probably bring in even more troubles both to the protocol
(e.g. in plenty of networks which do not allow any IP proto except 1,
6 and 17) and to networks (we have things like RFC 8086 for a reason).
There might've been other reasons.

--
Töma

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Michael Brown wrote:


Blocking a (for you) undesirable option when an established fallback
exists is a much better end user experience than introducing breakage
into that option


Are you saying AT should block UDP entirely?

Damian Menscher via NANOG wrote:

> As much as I would on principle rather not stick to a legacy, TCP-only
> home network --

Never stick to UDP.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Wed, Feb 19, 2020 at 12:51 AM Damian Menscher  wrote:
> [snip impressive debugging story]

lol fair. I didn't umm mean to just brag -- my point was that:
generally available SoHo internet is worse than mobile networks esp.
for UDP traffic!

> Rather than hobble your home network to work around a misbehaving ISP, have 
> you considered simply getting a new ISP?

I could get a leased line, sure, or route all my traffic through a
UDP/TCP VPN -- but the average user won't and can't, no.

-- Dan

Re: QUIC traffic throttled on AT residential

[Damian Menscher via NANOG]

On Tue, Feb 18, 2020 at 8:48 PM Daniel Sterling 
wrote:

[snip impressive debugging story]

As much as I would on principle rather not stick to a legacy, TCP-only
> home network --
>
> I can say that right now, my home internet, blocking UDP 443, and
> making tons of insecure DNS queries -- is the most stable, fastest,
> most usable and enjoyable home internet I've ever had. And my users
> agree -- they no longer turn off wifi.
>

Rather than hobble your home network to work around a misbehaving ISP, have
you considered simply getting a new ISP?

Damian

Re: QUIC traffic throttled on AT residential

[Mark Andrews]

> On 19 Feb 2020, at 15:47, Daniel Sterling  wrote:
> 
> On Tue, Feb 18, 2020 at 8:05 PM Michael Brown  wrote:
>> Blocking a (for you) undesirable option when an established fallback
>> exists is a much better end user experience than introducing breakage
>> into that option
> 
>> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
>> terrible slowdowns due to packet loss when it broke
> 
> All the +1s.
> 
> 
> I have one goal for my home internet: it should work better than my cell 
> phone!
> 
> In our house, "screen time" is "all the time". Everyone is on their
> phone non-stop. So you'd think getting fiber, installing UBNT APs and
> swapping out AT's CPE for a Core i5 linux box would provide a better
> internet experience than a tree-obstructed cell tower a mile or two
> down the road.
> 
> But you'd be wrong.
> 
> 
> Everyone in the house was, on a daily basis, turning off wifi in favor
> of AT LTE!
> 
> Why was everyone switching off wifi? I couldn't blame them -- I was
> toggling it on and off myself to get the occasional website or IM
> conversation to load. Why was my network broken?? How was it possible
> that a fairly high-latency mobile connection could provide a better
> experience than 802.11ac to an AP in the same room backed by a gigabit
> PON?
> 
> 
> I banged against this for *years*. I punted on using my own router and
> tried just AT's CPE (reset to factory defaults). That does work
> decently, but there are some maddening quirks, not least of which is
> insanely high jitter.
> 
> I tried SoHo devices running vendor stock firmware driving hardware
> NAT. Those also work well -- until they inevitably crash.
> 
> 
> I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
> running upstream kernels; downstream kernels; I tried custom patches.
> I tried HFSC and CoDel; I compiled iproute2 so I could have some
> tc_cake.
> 
> 
> On the link-layer (wifi) side I tried one AP; two APs; three APs. I
> tested any number of combinations of SSID name, channel frequency and
> width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
> AP; in desperation I even tried dedicating one AP and an entire 5ghz
> frequency range for just one phone.
> 
> But nothing mattered until I finally figured it out:
> 
> It was DNS. Of course it was DNS. It's always DNS.
> 
> 
> When DNS was solid and fast, everything else fell into place. Toggling
> wifi worked because it was the same as re-querying DNS! And the DNS
> service on mobile works well -- better than the average CPE.
> 
> *** I cannot stress this enough. No manner of tuning or tweaking to my
> home network stopped users from fleeing it, until I had solid DNS. ***
> 
> 
> For fast DNS, you of course need fast UDP. And, as we've empirically
> discovered, well-behaved UDP is by no means guaranteed on residential
> connections.
> 
> It turns out dnsmasq has a couple of tunables that can make a huge
> difference to home internet DNS performance. First, you need to be
> querying the DNS servers AT tells you to via DHCP. They're the least
> likely to be throttled, unfortunately. But I've found even that alone
> isn't enough.
> 
> You need to set dnsmasq's "query-port" option. By default it's random
> to protect against CVE-2008-1447, but apparently sending a ton of
> random-source-port UDP traffic does not impress the AT network flow
> control systems, and your DNS traffic becomes unbearably slow (or is
> simply dropped entirely)

If dnsmasq supports DNS COOKIE turn it on.  DNS COOKIE provides protection
against CVE-2008-1447 provides the other end supports DNS COOKIE without
having to play games with ports.

> AT gives you two DNS servers via DHCP. You can query more --
> 8.8.8.8, 4.2.2.2, 2606:4700:4700:: -- but if you do, you'll want
> to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
> query to every server on your list and use whatever comes back first.
> If you've angered the UDP flow restrictor, no matter -- with luck at
> least one of your packets is going to a server that's up and not
> throttled or overloaded!
> 
> 
> Of course DNS is just the beginning -- you still need a proper gateway
> device with a good NAT stack and/or firewall; you still need a strong
> wifi signal; you still need tc_cake so everyone can watch Netflix at
> the same time.
> 
> But DNS is the *core*. Nothing works well until DNS works well. That
> means nothing works well unless UDP works well. And if I have learned
> anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
> Not. Work. Well.
> 
> 
> Enter QUIC. It may be the perfect transport-layer protocol; but by
> putting it on top of UDP it's hobbled. It breaks extant v4 internet in
> a way that nothing else we've yet seen does -- it takes what would be
> your TCP traffic and gives it inconsistent and intermittently poor
> performance. Maybe it's sometimes fast. Maybe it is. But I can tell
> you, it sometimes Is Very Much Not So.
> 
> 
> As 

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 8:05 PM Michael Brown  wrote:
> Blocking a (for you) undesirable option when an established fallback
> exists is a much better end user experience than introducing breakage
> into that option

> Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
> terrible slowdowns due to packet loss when it broke

All the +1s.


I have one goal for my home internet: it should work better than my cell phone!

In our house, "screen time" is "all the time". Everyone is on their
phone non-stop. So you'd think getting fiber, installing UBNT APs and
swapping out AT's CPE for a Core i5 linux box would provide a better
internet experience than a tree-obstructed cell tower a mile or two
down the road.

But you'd be wrong.


Everyone in the house was, on a daily basis, turning off wifi in favor
of AT LTE!

Why was everyone switching off wifi? I couldn't blame them -- I was
toggling it on and off myself to get the occasional website or IM
conversation to load. Why was my network broken?? How was it possible
that a fairly high-latency mobile connection could provide a better
experience than 802.11ac to an AP in the same room backed by a gigabit
PON?


I banged against this for *years*. I punted on using my own router and
tried just AT's CPE (reset to factory defaults). That does work
decently, but there are some maddening quirks, not least of which is
insanely high jitter.

I tried SoHo devices running vendor stock firmware driving hardware
NAT. Those also work well -- until they inevitably crash.


I tried ddwrt and openwrt. I tried AQM; I tried QoS. I tried NUCs
running upstream kernels; downstream kernels; I tried custom patches.
I tried HFSC and CoDel; I compiled iproute2 so I could have some
tc_cake.


On the link-layer (wifi) side I tried one AP; two APs; three APs. I
tested any number of combinations of SSID name, channel frequency and
width. I tried with ipv6; no ipv6; I put all 2ghz devices on their own
AP; in desperation I even tried dedicating one AP and an entire 5ghz
frequency range for just one phone.

But nothing mattered until I finally figured it out:

It was DNS. Of course it was DNS. It's always DNS.


When DNS was solid and fast, everything else fell into place. Toggling
wifi worked because it was the same as re-querying DNS! And the DNS
service on mobile works well -- better than the average CPE.

*** I cannot stress this enough. No manner of tuning or tweaking to my
home network stopped users from fleeing it, until I had solid DNS. ***


For fast DNS, you of course need fast UDP. And, as we've empirically
discovered, well-behaved UDP is by no means guaranteed on residential
connections.

It turns out dnsmasq has a couple of tunables that can make a huge
difference to home internet DNS performance. First, you need to be
querying the DNS servers AT tells you to via DHCP. They're the least
likely to be throttled, unfortunately. But I've found even that alone
isn't enough.

You need to set dnsmasq's "query-port" option. By default it's random
to protect against CVE-2008-1447, but apparently sending a ton of
random-source-port UDP traffic does not impress the AT network flow
control systems, and your DNS traffic becomes unbearably slow (or is
simply dropped entirely).


AT gives you two DNS servers via DHCP. You can query more --
8.8.8.8, 4.2.2.2, 2606:4700:4700:: -- but if you do, you'll want
to enable dnsmasq's "all-servers" option. Packets are cheap -- send a
query to every server on your list and use whatever comes back first.
If you've angered the UDP flow restrictor, no matter -- with luck at
least one of your packets is going to a server that's up and not
throttled or overloaded!


Of course DNS is just the beginning -- you still need a proper gateway
device with a good NAT stack and/or firewall; you still need a strong
wifi signal; you still need tc_cake so everyone can watch Netflix at
the same time.

But DNS is the *core*. Nothing works well until DNS works well. That
means nothing works well unless UDP works well. And if I have learned
anything about AS7018, it's that UDP -- especially its v4 UDP -- Does.
Not. Work. Well.


Enter QUIC. It may be the perfect transport-layer protocol; but by
putting it on top of UDP it's hobbled. It breaks extant v4 internet in
a way that nothing else we've yet seen does -- it takes what would be
your TCP traffic and gives it inconsistent and intermittently poor
performance. Maybe it's sometimes fast. Maybe it is. But I can tell
you, it sometimes Is Very Much Not So.


As much as I would on principle rather not stick to a legacy, TCP-only
home network --

I can say that right now, my home internet, blocking UDP 443, and
making tons of insecure DNS queries -- is the most stable, fastest,
most usable and enjoyable home internet I've ever had. And my users
agree -- they no longer turn off wifi.


May I naively ask if Google staff have considered scrapping using UDP
and instead proposing a new, first-class transport protocol 

Re: QUIC traffic throttled on AT residential

[nanog-list]

On 2020-02-18 4:32 p.m., Michael Brown wrote:

With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
falls back to IPv4, everybody's happy.


The IPv6 deployment landscape might look considerably better if browser 
developers were instead to work together to co-ordinate an "unhappy 
eyeballs" algorithm that was rolled out over time, gradually degrading 
performance for IPv4-only connectivity situations. Now that 
non-evergreen browsers are pretty much dead, this might even be a 
realistic proposal, especially if it were done gradually enough. Google 
announcing a ranking penalty for IPv4-only sites wouldn't hurt either.


A while after implementing "unhappy eyeballs", once the performance 
penalties are severe enough that IPv6 adoption is high, the algorithm 
could be further adjusted to gradually impose penalties on sites that 
dual-stack instead of going v6-only. Growing prevalence of IPv6-only 
sites is probably the only thing that will get a lot of access networks 
to support v6.


The problem with happy eyeballs is that it works too well: excellent 
backwards compatibility reduces or eliminates the incentives for progress.


Happy eyeballs was necessary for it be realistic for sites to deploy 
IPv6 support, but now it holds us back.


--

Daniel Dent

https://www.danieldent.com

Re: QUIC traffic throttled on AT residential

[Michael Brown]

On 2020-02-18 7:07 p.m., Ross Tajvar wrote:
> Are you suggesting that ATT block all QUIC across their network?
Blocking a (for you) undesirable option when an established fallback
exists is a much better end user experience than introducing breakage
into that option

When you throttle or subtly break things you get:

On 2020-02-18 7:12 p.m., Daniel Sterling wrote:
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT ipv4 residential connection (and a web search
> suggests I'm not alone).

Or: I no longer use my ISP's IPv6 access (via 6rd) since it would cause
terrible slowdowns due to packet loss when it broke

Or: some AT customers cannot connect to our customers due to IPv6
HTTPS interception: https://meta.discourse.org/t/-/140769/3

Or (probably the same problem):
https://tutanota.com/blog/posts/att-blocks-tutanota/

With blocking in these cases, QUIC falls back to TCP, Happy Eyeballs
falls back to IPv4, everybody's happy.

Re: QUIC traffic throttled on AT residential

[Dan Wing]

Yes, other than AT increasing their permitted incoming UDP traffic -- the 
easiest thing AT can do -- AT could ask the vendor of their flow 
restricting device to use bi-directional UDP traffic on same 5-tuple to 
indicate "desire to receive", rather than solely examining incoming UDP traffic 
as they are doing today; this is not easy but also not impossible.

While it is true Youtube/Google could treat AT connections as 'bad' 
and force TCP, but I am sure Youtube/Google is already gathering those 
statistics and already aware that AT is throttling.  For all we know, you and 
the others noticing the issue have fallen into the pit of A/B testers checking 
for their current throttling, and others aren't being throttled.  Perhaps it's 
everyone, the tests are not well described or well announced.  Youtube/Google 
is hoping customers complain to AT so that AT removes or improves the flow 
restrictor, because otherwise AT customers won't be able to get QUIC.  
Similarly, AT could protect their users from AT's own rate limiting by 
blocking QUIC towards major servers that support QUIC but such blocking becomes 
problematic as QUIC rolls out beyond Google to Cloudflare and elsewhere.

This tussle has similarities to IPv6 vs IPv4.

-d


> On Feb 18, 2020, at 4:00 PM, Ca By  wrote:
> 
> 
> 
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling  > wrote:
> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
> 
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
> 
> I assume traffic is being throttled on AT's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
> 
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
> 
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
> 
> May I naively ask if Google staff are working with AT to address this?
> 
> Sounds like you found the answer, ATT just needs to scale up your rejection 
> approach that is proven to work well. 
> 
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they 
> would be mixing with bad company in the pool of ipv4 udp traffic ... but they 
> have reasons. 
> 
> I am not a fan of quic or any udp traffic. My suggestion was that Google use 
> an new L4 instead of UDP, but that was too hard for the Googlers. 
> 
> So here we are.
> 
> Just say no to udp. 
> 
> 
> 
> 
> -- Dan

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Tue, Feb 18, 2020 at 7:20 PM Dan Wing  wrote:
> For all we know, you and the others noticing the issue have fallen into the 
> pit of A/B testers checking for their current throttling, and others aren't 
> being throttled.

Ouch, I hope not -- do A/B tests that result in extreme performance
degradation continue indefinitely ?

> Youtube/Google is hoping customers complain to AT

It seems like this is one area where AT has no reason to care
(unless management decides to get involved). The default stance from
support would be, I assume: "Google is slow for you? Well obviously it
can't be Google's fault. Have a new NAT box"

I would bet dollars to donuts that if Google took a proper look at
their stats they would see this issue manifesting as users simply not
using services that have been QUIC-holed.

-- Dan

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

The members of the QUIC WG at IETF have not thought this was a problem as they 
don't observe it across the board. 

The cost for payloads with QUIC is much higher on the CPU side vs TCP as well - 
this is also not considered an issue either. 

The explanation I got (which seems fair) from someone was that they only way to 
roll a new transport was to squat on some existing stuff that would make it 
through firewalls. 

We have an internet that is largely fixed around running NATs and TCP and UDP 
only these days. I for one find this sad and don't see a good way forward on 
either side. 

Sent from my iCar

> On Feb 18, 2020, at 7:13 PM, Daniel Sterling  
> wrote:
> 
> 9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar  wrote:
>> 
>> Are you suggesting that ATT block all QUIC across their network?
> 
> One might argue they already *are* doing so; QUIC is essentially
> unusable on my AT ipv4 residential connection (and a web search
> suggests I'm not alone).

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

9On Tue, Feb 18, 2020 at 7:07 PM Ross Tajvar  wrote:
>
> Are you suggesting that ATT block all QUIC across their network?

One might argue they already *are* doing so; QUIC is essentially
unusable on my AT ipv4 residential connection (and a web search
suggests I'm not alone).

Re: QUIC traffic throttled on AT residential

[Ross Tajvar]

Are you suggesting that ATT block all QUIC across their network?

On Tue, Feb 18, 2020, 7:02 PM Ca By  wrote:

>
>
> On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
> wrote:
>
>> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
>> from google (esp. youtube) becomes very slow after a time.
>>
>> This especially occurs with ipv4 connections. I'm not the only one to
>> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
>> notes the issue.
>>
>> I assume traffic is being throttled on AT's side. And it's not done
>> with their CPE; I'm bypassing their NAT box and connecting my laptop
>> directly to the ONT.
>>
>> A quick google search shows people are aware that QUIC can look like
>> DoS traffic -- but how widely known is this problem? It may become
>> worse if / when sites transition to HTTP/3
>>
>> For now I reject UDP 443 on the client firewall. Applications using
>> QUIC client libraries then fallback to TCP. This works well and I have
>> no issues with latency or throughput when using TCP.
>>
>> May I naively ask if Google staff are working with AT to address this?
>
>
> Sounds like you found the answer, ATT just needs to scale up your
> rejection approach that is proven to work well.
>
> Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
> would be mixing with bad company in the pool of ipv4 udp traffic ... but
> they have reasons.
>
> I am not a fan of quic or any udp traffic. My suggestion was that Google
> use an new L4 instead of UDP, but that was too hard for the Googlers.
>
> So here we are.
>
> Just say no to udp.
>
>
>
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Ca By]

On Tue, Feb 18, 2020 at 5:44 PM Daniel Sterling 
wrote:

> I've AT fiber (in RTP, NC) (AS7018) and I notice UDP QUIC traffic
> from google (esp. youtube) becomes very slow after a time.
>
> This especially occurs with ipv4 connections. I'm not the only one to
> notice; a web search for e.g. "Extremely Poor Youtube TV Performance"
> notes the issue.
>
> I assume traffic is being throttled on AT's side. And it's not done
> with their CPE; I'm bypassing their NAT box and connecting my laptop
> directly to the ONT.
>
> A quick google search shows people are aware that QUIC can look like
> DoS traffic -- but how widely known is this problem? It may become
> worse if / when sites transition to HTTP/3
>
> For now I reject UDP 443 on the client firewall. Applications using
> QUIC client libraries then fallback to TCP. This works well and I have
> no issues with latency or throughput when using TCP.
>
> May I naively ask if Google staff are working with AT to address this?


Sounds like you found the answer, ATT just needs to scale up your rejection
approach that is proven to work well.

Yes, most ddos traffic is ipv4 udp and yes Google was made aware that they
would be mixing with bad company in the pool of ipv4 udp traffic ... but
they have reasons.

I am not a fan of quic or any udp traffic. My suggestion was that Google
use an new L4 instead of UDP, but that was too hard for the Googlers.

So here we are.

Just say no to udp.



>
> -- Dan
>