Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Damian Menscher via NANOG]

Amir: you're exactly correct -- but since you asked, here's their answer
from the last time I suggested they respond with RSTs:
https://seclists.org/nanog/2020/Jan/612

Damian

On Thu, Feb 20, 2020 at 5:36 PM Amir Herzberg  wrote:

> If I read your description correctly:
>
> - Attacker sends spoofed TCP SYN from your IP address(es) and different
> src ports, to some TCP servers (e.g. port 80)
> - TCP servers respond with SYN/ACK  ; many servers resend the SYN/ACK
> hence amplification .
> - *** your system does not respond ***
> - Servers may think you're doing SYN-Flood against them, since connection
> remains in SYN_RCVD, and hence complain. In fact, we don't really know what
> is the goal of the attackers; they may in fact be trying to do SYN-Flood
> against these servers, and you're just a secondary victim and not the even
> the target, that's also possible.
>
> Anyway, is this the case?
>
> If it is... may I ask, do you (or why don't you) respond to the
> unsolicited SYN/ACK with RST as per the RFC?
>
> I suspect you don't, maybe due to these packets being dropped by FW/NAT,
> that's quite common. But as you should understand by now from my text, this
> (non-standard) behavior is NOT recommended. The problem may disappear if
> you reconfigure your FW/NAT (or host) to respond with RST to unsolicited
> SYN/ACK.
>
> As I explained above, if my conjectures are true, then OVH as well as the
> remote servers may have a valid reason to consider you either as the
> attacker or as an (unknowning, perhaps) accomplice.
>
> I may be wrong - sorry if so - and would appreciate, in any case, if you
> can confirm or clarify, thanks.
>
> --
> Amir Herzberg
>
> Comcast professor of Security Innovations, University of Connecticut
>
> Homepage: https://sites.google.com/site/amirherzberg/home
>
> Foundations of Cyber-Security (part I: applied crypto, part II:
> network-security):
> https://www.researchgate.net/project/Foundations-of-Cyber-Security
>
>
>
> On Thu, Feb 20, 2020 at 5:23 PM Octolus Development 
> wrote:
>
>> A very old attack method called TCP-AMP ( https://pastebin.com/jYhWdgHn )
>> has been getting really popular recently.
>>
>> I've been a victim of it multiple times on many of my IP's and every time
>> it happens - My IP's end up getting blacklisted in major big databases. We
>> also receive tons of abuse reports for "Port Scanning".
>>
>> Example of the reports we're getting:
>> tcp: 51.81.XX.XX:19342 -> 209.208.XX.XX:80 (SYN_RECV)
>> tcp: 51.81.XX.XX:14066 -> 209.208.XX.XX:80 (SYN_RECV)
>>
>> OVH are threatening to kick us off their network, because we are victims
>> of this attack. And requesting us to do something about it, despite the
>> fact that there is nothing you can do when you are being victim of an DDoS
>> Attack.
>>
>> Anyone else had any problems with these kind of attacks?
>>
>> The attack basically works like this;
>> - The attacker scans the internet for TCP Services, i.e port 80.
>> - The attacker then sends spoofed requests from our IP to these TCP
>> Services, which makes the remote service attempt to connect to us to
>> initiate the handshake.. This clearly fails.
>> ... Which ends up with hundreds of request to these services, reporting
>> us for "port flood".
>>
>>
>>

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Damian Menscher via NANOG]

On Thu, Feb 20, 2020 at 3:40 PM Jean | ddostest.me via NANOG <
nanog@nanog.org> wrote:

> It doesn't sound to be a real amplification.. If it is, can anyone provide
> the amplification factor? 1x?
>
> It sounds more like a TCP spoofing.
>

Some reading for you:
https://www.usenix.org/conference/woot14/workshop-program/presentation/kuhrer

Damian

Re: QUIC traffic throttled on AT residential

[Matthew Kaufman]

On Thu, Feb 20, 2020 at 8:10 AM Ca By  wrote:

>
>
> Not indiscriminate.
>
> As Google was informed by network operators all along since 2014, ipv4 UDP
> is a major uptime threat via DDoS to access networks.
> ...
>
> Google choose not to be sensitive to that, they were told where the
> landmines were, they choose to go on anyhow.
>
>
It isn’t like they had a choice. You can’t build a transport protocol like
QUIC on top of TCP (I know, I built one of its ancestors, which also uses
UDP underneath). And if you think getting UDP through existing networks is
hard, try using a novel IP protocol number.

Matthew Kaufman

>
>

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Daniel Sterling wrote:


A problem of QUIC with NAT is that existing NAT can not detect
graceful shutdown of QUIC and must depends on timeout.

So, port numbers may be used up before timeout.


Hmm, this is not what is happening.


I thought so. My point is that the problem can be another reason
to avoid QUIC.

Masataka Ohta

Re: QUIC traffic throttled on AT residential

[Masataka Ohta]

Lukas Tribus wrote:


IPv6 UDP is currently not broken, that doesn't mean v6 is the solution
to this problem. It's just means the particular ISP did not yet deploy
the same policies or "mitigations" for v6 traffic.


It is more likely that the ISP does not support v6 at all.


In a much smaller eyeball environment (with
much smaller chokepoints), we have mapped possibly amplificated
packets (ip frag, dns, ntp, memcached, et all) to a specific queue.
Unless the links are congested, this traffic passes just as any other
traffic and during congestion it only uses whatever bandwidth the
queue has - no static rate-limits.


That is a bad idea.

Static rate limit is necessary to discourage DoS attackers.

If the attacker send 10Mbps stream to an amplifier and the stream
is redirected to a victim at 100Mbps, 10Mbps rate limiting negates
the amplification.

Masataka Ohta

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Amir Herzberg]

If I read your description correctly:

- Attacker sends spoofed TCP SYN from your IP address(es) and different src
ports, to some TCP servers (e.g. port 80)
- TCP servers respond with SYN/ACK  ; many servers resend the SYN/ACK hence
amplification .
- *** your system does not respond ***
- Servers may think you're doing SYN-Flood against them, since connection
remains in SYN_RCVD, and hence complain. In fact, we don't really know what
is the goal of the attackers; they may in fact be trying to do SYN-Flood
against these servers, and you're just a secondary victim and not the even
the target, that's also possible.

Anyway, is this the case?

If it is... may I ask, do you (or why don't you) respond to the unsolicited
SYN/ACK with RST as per the RFC?

I suspect you don't, maybe due to these packets being dropped by FW/NAT,
that's quite common. But as you should understand by now from my text, this
(non-standard) behavior is NOT recommended. The problem may disappear if
you reconfigure your FW/NAT (or host) to respond with RST to unsolicited
SYN/ACK.

As I explained above, if my conjectures are true, then OVH as well as the
remote servers may have a valid reason to consider you either as the
attacker or as an (unknowning, perhaps) accomplice.

I may be wrong - sorry if so - and would appreciate, in any case, if you
can confirm or clarify, thanks.

-- 
Amir Herzberg

Comcast professor of Security Innovations, University of Connecticut

Homepage: https://sites.google.com/site/amirherzberg/home

Foundations of Cyber-Security (part I: applied crypto, part II:
network-security):
https://www.researchgate.net/project/Foundations-of-Cyber-Security



On Thu, Feb 20, 2020 at 5:23 PM Octolus Development 
wrote:

> A very old attack method called TCP-AMP ( https://pastebin.com/jYhWdgHn )
> has been getting really popular recently.
>
> I've been a victim of it multiple times on many of my IP's and every time
> it happens - My IP's end up getting blacklisted in major big databases. We
> also receive tons of abuse reports for "Port Scanning".
>
> Example of the reports we're getting:
> tcp: 51.81.XX.XX:19342 -> 209.208.XX.XX:80 (SYN_RECV)
> tcp: 51.81.XX.XX:14066 -> 209.208.XX.XX:80 (SYN_RECV)
>
> OVH are threatening to kick us off their network, because we are victims
> of this attack. And requesting us to do something about it, despite the
> fact that there is nothing you can do when you are being victim of an DDoS
> Attack.
>
> Anyone else had any problems with these kind of attacks?
>
> The attack basically works like this;
> - The attacker scans the internet for TCP Services, i.e port 80.
> - The attacker then sends spoofed requests from our IP to these TCP
> Services, which makes the remote service attempt to connect to us to
> initiate the handshake.. This clearly fails.
> ... Which ends up with hundreds of request to these services, reporting us
> for "port flood".
>
>
>

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Jean | ddostest.me via NANOG]

It doesn't sound to be a real amplification.. If it is, can anyone 
provide the amplification factor? 1x?


It sounds more like a TCP spoofing.

Jean

On 2020-02-20 18:22, Töma Gavrichenkov wrote:

Peace,

On Fri, Feb 21, 2020, 1:57 AM Filip Hruska > wrote:


[..] OVH has been offering DDOS protection capable of soaking up
hundreds of gigabits+ per second as a standard with all their
services for a long time

They only do it for common trivial vectors like UDP-based 
amplification — and other types easily handleable through flowspec.


Which is honestly not their fault because they try to keep their costs 
down.  (Other means to keep the costs down may be of concern of Ronald 
G. though, but that's a different story.)


However, TCP amplification is not of that sort.

--
Töma

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Töma Gavrichenkov]

Peace,

On Fri, Feb 21, 2020, 1:57 AM Filip Hruska  wrote:

> [..] OVH has been offering DDOS protection capable of soaking up hundreds
> of gigabits+ per second as a standard with all their services for a long
> time
>
They only do it for common trivial vectors like UDP-based amplification —
and other types easily handleable through flowspec.

Which is honestly not their fault because they try to keep their costs
down.  (Other means to keep the costs down may be of concern of Ronald G.
though, but that's a different story.)

However, TCP amplification is not of that sort.

--
Töma

>

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Filip Hruska]

Hello,

Since OVH has been offering DDOS protection capable of soaking up 
hundreds of gigabits+ per second as a standard with all their services 
for a long time, I'm assuming this is a miscommunication / standard 
support response.


I would try to get in touch with the network team and include pcaps.

Filip

On 2/20/20 11:37 PM, Töma Gavrichenkov wrote:

Peace,

On Fri, Feb 21, 2020, 1:18 AM Octolus Development > wrote:


OVH are threatening to kick us off their network, because we are
victims of this attack.


Most of the hosting companies will do that to you because you're 
causing degradation of service quality for other customers.  
Especially the lowcosters which even cut the costs of processing the 
abuse, and OVH is the world-renowned cost cutter.


Look out for data centers with appropriate DDoS countermeasures.  
Those won't be cheap though.


--
Töma

Re: Forest HQ Has Received Your Message: Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Töma Gavrichenkov]

Help saving precious resources by unsubscribing from the NANOG mailing
list, or I will have to report the abuse.

On Fri, Feb 21, 2020, 1:39 AM Electric Forest Festival <
i...@electricforestfestival.com> wrote:

>
> *Electric Forest 2020 will take place on June 25-28, 2020.*
>
> Forest HQ has received your email. Help save precious resources by
> reviewing the information below and looking up common questions in The
> Forest Frequently Asked Questions: Experience.ElectricForestFestival.com
> 
>
> Please contact Festival Ticketing Support at 855-279-6941 for all issue
> regarding your purchase or for account troubleshooting.
>
> *Electric Forest is sold out. *Lyte is the only HQ endorsed way to get
> passes now that it’s sold out.
> 
>
> To know when all things Electric Forest 2020 are happening sign up to the
> EF Newsletter. 
>
> *Happy Forest!*
>
> 118554:553794
>

Re: TCP-AMP DDoS Attack - Fake abuse reports problem

[Töma Gavrichenkov]

Peace,

On Fri, Feb 21, 2020, 1:18 AM Octolus Development  wrote:

> OVH are threatening to kick us off their network, because we are victims
> of this attack.
>

Most of the hosting companies will do that to you because you're causing
degradation of service quality for other customers.  Especially the
lowcosters which even cut the costs of processing the abuse, and OVH is the
world-renowned cost cutter.

Look out for data centers with appropriate DDoS countermeasures.  Those
won't be cheap though.

--
Töma

Re: QUIC traffic throttled on AT residential

[Lukas Tribus]

Hello,


On Thu, 20 Feb 2020 at 21:30, Daniel Sterling  wrote:
> As has been continually noted, this issue goes away if you use v4 TCP or v6 
> UDP.

IPv6 UDP is currently not broken, that doesn't mean v6 is the solution
to this problem. It's just means the particular ISP did not yet deploy
the same policies or "mitigations" for v6 traffic. As v6 adoption
increases, so will abuse/misuse, especially when attackers notice that
their attack traffic is rate-limited on v4 but not on v6 and P2P
gaming switches from v4 to v6. And at some point this will lead to
"feature parity" in IPv6. So while v6 UDP currently works, I don't
think we can assume that's permanent.

I disagree this approach is necessary to keep the network running and
the pagers from buzzing. In a much smaller eyeball environment (with
much smaller chokepoints), we have mapped possibly amplificated
packets (ip frag, dns, ntp, memcached, et all) to a specific queue.
Unless the links are congested, this traffic passes just as any other
traffic and during congestion it only uses whatever bandwidth the
queue has - no static rate-limits. I'm not saying this will fix
whatever the policies discussed here are supposed to fix, but there is
always a way to improve and make the mitigations more nuanced.

Of course ISPs will protect the network and the customers. But whether
you apply a simple rate-limiting for some traffic or some AI-assisted
auto-mitigation for traffic misuse, you gotta be prepared to monitor
and update it, staying on top of at least the major false-positives,
short-term but long-term as well. After all, things tend to change
over time.



lukas

Re: [External] Re: QUIC traffic throttled on AT residential

[Hunter Fuller]

On Thu, Feb 20, 2020 at 3:45 PM Jared Mauch  wrote:
> I can think of many legitimate cases, but i think this is where you have 
> internet for everyone and internet for the tech-savvy/business split that 
> becomes interesting.
>
> I’ve generally been willing to pay more for a business class service for 
> support and improved response SLA.  The average user isn’t going to detect 
> that 10% of their UDP has gone missing, nor should they be expected to.

I really hope my constituents don't have to get business class
connections just to get decent performance out of our services, such
as UDP based tunnels. They barely care what a VPN is, much less what
UDP is. And if our VPN software detects that UDP is available, it will
use it, so I suspect it would be (or is being) affected by this.

TCP-AMP DDoS Attack - Fake abuse reports problem

[Octolus Development]

A very old attack method called TCP-AMP ( https://pastebin.com/jYhWdgHn 
[https://pastebin.com/jYhWdgHn] ) has been getting really popular recently. 

I've been a victim of it multiple times on many of my IP's and every time it 
happens - My IP's end up getting blacklisted in major big databases. We also 
receive tons of abuse reports for "Port Scanning".

Example of the reports we're getting:
tcp: 51.81.XX.XX:19342 -> 209.208.XX.XX:80 (SYN_RECV)
tcp: 51.81.XX.XX:14066 -> 209.208.XX.XX:80 (SYN_RECV)

OVH are threatening to kick us off their network, because we are victims of 
this attack. And requesting us to do something about it, despite the fact that 
there is nothing you can do when you are being victim of an DDoS Attack.

Anyone else had any problems with these kind of attacks?

The attack basically works like this;
- The attacker scans the internet for TCP Services, i.e port 80.
- The attacker then sends spoofed requests from our IP to these TCP Services, 
which makes the remote service attempt to connect to us to initiate the 
handshake.. This clearly fails.
... Which ends up with hundreds of request to these services, reporting us for 
"port flood".

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 4:53 PM, Blake Hudson  wrote:
> 
> 
As a network operator my goal was always to ensure customers receive
 the traffic they expected, high rates of UDP were often not what they 
 wanted.
 
Adusting the limits may be useful but I still think the question of
 what rate of UDP traffic is acceptable is a practical one for the future.
 
- Jared
>>> I think that's a fair statement Jared. How about this question: Would it be 
>>> reasonable for one to presume that someone purchasing a 25Mbps internet 
>>> connection might potentially want to send or receive 25Mbps of UDP traffic? 
>>> I can think of a few (not uncommon) applications where this would be the 
>>> case (VPNs, security cameras using RTP, teleconferencing, web browsers 
>>> implementing QUIC, DNS servers, hosted PBX, etc).
>> I can think of many legitimate cases, but i think this is where you have 
>> internet for everyone and internet for the tech-savvy/business split that 
>> becomes interesting.
>> 
>> I’ve generally been willing to pay more for a business class service for 
>> support and improved response SLA.  The average user isn’t going to detect 
>> that 10% of their UDP has gone missing, nor should they be expected to.
>> 
>> - Jared
> And here I think is where one crosses the threshold between providing an 
> "internet connection" and providing a connection "that can be used to access 
> specific applications or services" (as defined by your provider). This is one 
> step away from your ISP selling you a connection to access Facebook, if you 
> want to access Twitter that's available on their premium package. Oh, you 
> want to access Slack, sorry we don't offer that as a package yet. Call back 
> in a month. You need to esss-esss-achhh? I've never heard of that, why would 
> you want to do that?

AT has rarely offered internet service, their required devices for their 
U-Verse often munged traffic.  I recall when you could reboot their boxes by 
sending SIP packets to devices behind them and it would intercept them and 
think it was for itself for their POTS service.

If you have any NAT/ALG in there, it’s not pure internet, but most people want 
access to the “web” and aren’t running ftp/finger/ytalk/uucp/sip etc.. This is 
why SSL VPNs on 443 became a thing over time.

- jared

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

I think that's a fair statement Jared. How about this question: Would it be 
reasonable for one to presume that someone purchasing a 25Mbps internet 
connection might potentially want to send or receive 25Mbps of UDP traffic? I 
can think of a few (not uncommon) applications where this would be the case 
(VPNs, security cameras using RTP, teleconferencing, web browsers implementing 
QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have 
internet for everyone and internet for the tech-savvy/business split that 
becomes interesting.

I’ve generally been willing to pay more for a business class service for 
support and improved response SLA.  The average user isn’t going to detect that 
10% of their UDP has gone missing, nor should they be expected to.

- Jared
And here I think is where one crosses the threshold between providing an 
"internet connection" and providing a connection "that can be used to 
access specific applications or services" (as defined by your provider). 
This is one step away from your ISP selling you a connection to access 
Facebook, if you want to access Twitter that's available on their 
premium package. Oh, you want to access Slack, sorry we don't offer that 
as a package yet. Call back in a month. You need to esss-esss-achhh? 
I've never heard of that, why would you want to do that?

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 4:42 PM, Blake Hudson  wrote:
> 
> 
> 
> On 2/20/2020 1:10 PM, Jared Mauch wrote:
>> On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
>>> On 2/20/2020 10:34 AM, Ca By wrote:
 On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson >>> > wrote:
 Dropping udp is not from a “best practice” doc from a vendor, it is
 deployed by network ops folks that are trying to sleep at night.
>>> I get it Ca, I happen to be one of those network ops folks that likes to
>>> sleep at night. However, I've never thought it was a good practice to break
>>> applications in fun ways for my customers to discover on their own and I've
>>> never sold someone a 150Mbps package that actually only delivers 10Mbps for
>>> certain applications. Regardless of the intent, ATT and Cox's policies are
>>> not transparent, open, or neutral on this topic. This leaves us to speculate
>>> on what their intentions might have been and whether their actions are an
>>> appropriate response to any concerns they might have had.
>>  I was responsible for deploying such policies in the past, going back as
>> far as the UDP/1434 filters I was forced to deploy due to persistent network
>> congestion.  Rolling these back took some time.
>> 
>>  The same is true for UDP policers we ended up rolling out for NTP, 
>> chargen
>> and other activities.
>> 
>>  Extending these to consumer side where the traffic often originated 
>> makes
>> sense until the devices can be secured.  You can blame the providers for 
>> deploying
>> filters, or not disconnecting consumers that have devices that can be 
>> exploited
>> or whatever other reason you believe.
>> 
>>  As a network operator my goal was always to ensure customers receive
>> the traffic they expected, high rates of UDP were often not what they wanted.
>> 
>>  Adusting the limits may be useful but I still think the question of
>> what rate of UDP traffic is acceptable is a practical one for the future.
>> 
>>  - Jared
> 
> I think that's a fair statement Jared. How about this question: Would it be 
> reasonable for one to presume that someone purchasing a 25Mbps internet 
> connection might potentially want to send or receive 25Mbps of UDP traffic? I 
> can think of a few (not uncommon) applications where this would be the case 
> (VPNs, security cameras using RTP, teleconferencing, web browsers 
> implementing QUIC, DNS servers, hosted PBX, etc).

I can think of many legitimate cases, but i think this is where you have 
internet for everyone and internet for the tech-savvy/business split that 
becomes interesting.

I’ve generally been willing to pay more for a business class service for 
support and improved response SLA.  The average user isn’t going to detect that 
10% of their UDP has gone missing, nor should they be expected to.

- Jared

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 1:10 PM, Jared Mauch wrote:

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:

On 2/20/2020 10:34 AM, Ca By wrote:

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson mailto:bl...@ispn.net>> wrote:
Dropping udp is not from a “best practice” doc from a vendor, it is
deployed by network ops folks that are trying to sleep at night.

I get it Ca, I happen to be one of those network ops folks that likes to
sleep at night. However, I've never thought it was a good practice to break
applications in fun ways for my customers to discover on their own and I've
never sold someone a 150Mbps package that actually only delivers 10Mbps for
certain applications. Regardless of the intent, ATT and Cox's policies are
not transparent, open, or neutral on this topic. This leaves us to speculate
on what their intentions might have been and whether their actions are an
appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as
far as the UDP/1434 filters I was forced to deploy due to persistent network
congestion.  Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, 
chargen
and other activities.

Extending these to consumer side where the traffic often originated 
makes
sense until the devices can be secured.  You can blame the providers for 
deploying
filters, or not disconnecting consumers that have devices that can be exploited
or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared


I think that's a fair statement Jared. How about this question: Would it 
be reasonable for one to presume that someone purchasing a 25Mbps 
internet connection might potentially want to send or receive 25Mbps of 
UDP traffic? I can think of a few (not uncommon) applications where this 
would be the case (VPNs, security cameras using RTP, teleconferencing, 
web browsers implementing QUIC, DNS servers, hosted PBX, etc).

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

> On Feb 20, 2020, at 3:30 PM, Daniel Sterling  
> wrote:
> 
> On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch  wrote:
>> if the question is will the browser vendor (google) or the broadband 
>> provider (att)
>> move first, i can already predict the answer.  my experience (again) with 
>> the quic
>> wg is they seem to think there's many options and bad providers will be 
>> replaced
>> which seems disconnected from reality.
> 
> Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where
> there are multiple ISPs.
> 
> But not everyone is, and local-loop unbundling is no longer a thing :(
> 
> The average user is at the mercy of their local ISP. So the major
> providers should (but may not have much incentive to) provide decent
> service.
> 

Honestly, there’s part of me that wants to say be lucky you have a choice.

My options here are (1.5m DSL, fixed wireless ISP behind several layers of NAT,
Cellular data or dial-up).

“Watch this space” for my solution.  Several people know what it is, but
my solution isn’t for everyone.

- Jared

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Thu, Feb 20, 2020 at 2:57 PM Jared Mauch  wrote:
> if the question is will the browser vendor (google) or the broadband provider 
> (att)
> move first, i can already predict the answer.  my experience (again) with the 
> quic
> wg is they seem to think there's many options and bad providers will be 
> replaced
> which seems disconnected from reality.

Agreed. Since I'm in RTP, I *am* lucky enough to live in an area where
there are multiple ISPs.

But not everyone is, and local-loop unbundling is no longer a thing :(

The average user is at the mercy of their local ISP. So the major
providers should (but may not have much incentive to) provide decent
service.

As has been continually noted, this issue goes away if you use v4 TCP or v6 UDP.

And one may argue that most users *will* by default have v6 UDP
connectivity from their ISP -- to Google at least.

But HTTP/3 is coming -- and site operators may not realize deploying
HTTP/3 on v4 only will break connectivity for their users.

We're currently in a limbo where v4, which had been working fine, has
been *broken* -- and not everyone uses v6.

Site operators aren't used to deploying services that break when
served over ipv4. HTTP/3 will be the first widely-deployed service
that only properly works when served from v6.

So something is going to give:

* site operators will have to deploy v6 before they deploy HTTP/3, or

* network operators will have to make v4 UDP as reliable as v4 TCP (or
block HTTP/3)

If neither happens, then as more sites start pushing v4 UDP, more and
more users will start seeing issues, even if they *do* have v6 working
well at their house.

It may become common practice to block HTTP/3 on the client side to
improve connectivity to v4-only services.

Exciting times lay ahead, indeed

-- Dan

Re: QUIC traffic throttled on AT residential

[Tom Beecher]

>
> i don't think you've addressed the "replace your broken ISP" action that
> is clearly sane and would fix this, right?
>

The sanity presumes two things:

A: That he could do so without having to change addresses as well.
(Something that is still all too true for much of the US.)
B: The other provider is not doing anything on their network that might
similarly be impacting this specific traffic. (But could very well be doing
something to impact other traffic.)


On Thu, Feb 20, 2020 at 2:53 PM Todd Underwood  wrote:

> and just to check one thing...
>
> On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
> wrote:
>
>> I don't particularly *want* to block or advocate blocking QUIC, but if
>> I keep hitting the issue and can't help people troubleshoot, what
>> other sane option have I?
>>
>
> i don't think you've addressed the "replace your broken ISP" action that
> is clearly sane and would fix this, right?
>
> i'm assuming that this is not an option to get a functional IP layer?
>
> t
>
>
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

On Thu, Feb 20, 2020 at 02:50:58PM -0500, Todd Underwood wrote:
> and just to check one thing...
> 
> On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
> wrote:
> 
> > I don't particularly *want* to block or advocate blocking QUIC, but if
> > I keep hitting the issue and can't help people troubleshoot, what
> > other sane option have I?
> >
> 
> i don't think you've addressed the "replace your broken ISP" action that is
> clearly sane and would fix this, right?
> 
> i'm assuming that this is not an option to get a functional IP layer?

often one has to live in the world of reality.  if google is very worried about
the broadband experience they can continue to invest in this space and 
challenges.

it seems the corporate overlords has led it to not continue to persue this 
option.

it's hard to have it both ways, wanting unlimited UDP, low cost (free?) ports 
for
bits, etc..

there's natural dynamics at play here with business interests that haven't quite
balanced out yet.  perhaps the LTE/5G overlay will replace fixed broadband at 
home
with managed CPE from the cellular providers.

if the question is will the browser vendor (google) or the broadband provider 
(att)
move first, i can already predict the answer.  my experience (again) with the 
quic
wg is they seem to think there's many options and bad providers will be replaced
which seems disconnected from reality.

- jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: QUIC traffic throttled on AT residential

[Todd Underwood]

and just to check one thing...

On Thu, Feb 20, 2020 at 2:33 PM Daniel Sterling 
wrote:

> I don't particularly *want* to block or advocate blocking QUIC, but if
> I keep hitting the issue and can't help people troubleshoot, what
> other sane option have I?
>

i don't think you've addressed the "replace your broken ISP" action that is
clearly sane and would fix this, right?

i'm assuming that this is not an option to get a functional IP layer?

t


>
> -- Dan
>

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

On Thu, Feb 20, 2020 at 2:11 PM Jared Mauch  wrote:
> As a network operator my goal was always to ensure customers receive
> the traffic they expected, high rates of UDP were often not what they wanted.

Well, I wouldn't say I *want* UDP traffic, but if everyone is bound
and determined to send it to me for web / video traffic unless I block
it, I suppose we should all work together to improve my (average v4
end-user) experience.

I set up a rolling tcpdump to capture QUIC / HTTP/3 traffic.

tcpdump -C 100 -w quic -W 100 -i eth1 'udp and port 443'

I can make this dump available for inspection if (when) I organically
experience the issue again.

Is there anything else I can do to help Google or AT improve things?
Any magic debugging I can turn on on the client side?

Feel free to ping me off list...

I don't particularly *want* to block or advocate blocking QUIC, but if
I keep hitting the issue and can't help people troubleshoot, what
other sane option have I?

-- Dan

Re: QUIC traffic throttled on AT residential

[Jared Mauch]

On Thu, Feb 20, 2020 at 10:57:46AM -0600, Blake Hudson wrote:
> On 2/20/2020 10:34 AM, Ca By wrote:
> > 
> > On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson  > > wrote:
> > Dropping udp is not from a “best practice” doc from a vendor, it is
> > deployed by network ops folks that are trying to sleep at night.
> 
> I get it Ca, I happen to be one of those network ops folks that likes to
> sleep at night. However, I've never thought it was a good practice to break
> applications in fun ways for my customers to discover on their own and I've
> never sold someone a 150Mbps package that actually only delivers 10Mbps for
> certain applications. Regardless of the intent, ATT and Cox's policies are
> not transparent, open, or neutral on this topic. This leaves us to speculate
> on what their intentions might have been and whether their actions are an
> appropriate response to any concerns they might have had.

I was responsible for deploying such policies in the past, going back as
far as the UDP/1434 filters I was forced to deploy due to persistent network
congestion.  Rolling these back took some time.

The same is true for UDP policers we ended up rolling out for NTP, 
chargen
and other activities.

Extending these to consumer side where the traffic often originated 
makes
sense until the devices can be secured.  You can blame the providers for 
deploying
filters, or not disconnecting consumers that have devices that can be exploited
or whatever other reason you believe.

As a network operator my goal was always to ensure customers receive
the traffic they expected, high rates of UDP were often not what they wanted.

Adusting the limits may be useful but I still think the question of
what rate of UDP traffic is acceptable is a practical one for the future.

- Jared

-- 
Jared Mauch  | pgp key available via finger from ja...@puck.nether.net
clue++;  | http://puck.nether.net/~jared/  My statements are only mine.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 10:41 AM Dave Bell  wrote:

>
> Not indiscriminate.
>>
>
> Indiscriminate - done at random or without careful judgement.
>
> Considering that Daniel is complaining that QUIC is broken, it certainly
> seems like some network operators are subjecting all UDP traffic on their
> network to the same policers. This feels pretty indiscriminate to me.
>
> I'm all for policing the known baddies, such as CHARGEN and NTP, but to
> discard UDP for fun is like taking a sledgehammer where a scalpel will do.
>
>

For fun?

We are done here

Access networks need controls to maintain uptime against the non-stop
>> barrage of ddos attacks. I am sure you have seen the headlines and mails on
>> this list, ddos is hard to deal with. Access network will use whatever
>> tools are required to keep the pagers quiet and customers happy.
>>
>
> I operate an access network that does not blanket police UDP. Google give
> me a dashboard that tell me 45% of requests  were served happily by QUIC,
> and I have no customers complaining about things not working, and our
> pagers are silent.
>

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 10:41 AM, Dave Bell wrote:


Not indiscriminate.


Indiscriminate - done at random or without careful judgement.

Considering that Daniel is complaining that QUIC is broken, it 
certainly seems like some network operators are subjecting all UDP 
traffic on their network to the same policers. This feels pretty 
indiscriminate to me.


I'm all for policing the known baddies, such as CHARGEN and NTP, but 
to discard UDP for fun is like taking a sledgehammer where a scalpel 
will do.


Access networks need controls to maintain uptime against the
non-stop barrage of ddos attacks. I am sure you have seen the
headlines and mails on this list, ddos is hard to deal with.
Access network will use whatever tools are required to keep the
pagers quiet and customers happy.


I operate an access network that does not blanket police UDP. Google 
give me a dashboard that tell me 45% of requests  were served happily 
by QUIC, and I have no customers complaining about things not working, 
and our pagers are silent.



Dave, just wanted to say that I 100% agree with your comments. The bad 
actors are well known. I believe treating all UDP as bad is misguided. 
Like you, I assist in operation of several access networks that do not 
blanket police UDP and my pager remains relatively silent.

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/20/2020 10:34 AM, Ca By wrote:


On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson > wrote:



Your comments seem to differentiate IP4 vs IP6, but I don't
believe that
is relevant to the issue of an ISP throttling or breaking specific
applications. If you have evidence that UDP on IP4 is treated
differently than UDP on IP6 by your provider, without further
information I would suspect that this is simply an unintentional over
sight on their part.



This is your misunderstanding. The protections are to drop ipv4 udp 
because that is where the ddos / iot trash is , not v6 for now




Perhaps the attention you've generated on this topic, along with the
adoption of additional UDP based applications like QUIC, will
encourage
ISPs to treat UDP in a more neutral manner and not simply see UDP as
something that is "bad".


Dropping udp is not from a “best practice” doc from a vendor, it is 
deployed by network ops folks that are trying to sleep at night.




I get it Ca, I happen to be one of those network ops folks that likes to 
sleep at night. However, I've never thought it was a good practice to 
break applications in fun ways for my customers to discover on their own 
and I've never sold someone a 150Mbps package that actually only 
delivers 10Mbps for certain applications. Regardless of the intent, ATT 
and Cox's policies are not transparent, open, or neutral on this topic. 
This leaves us to speculate on what their intentions might have been and 
whether their actions are an appropriate response to any concerns they 
might have had.

Re: QUIC traffic throttled on AT residential

[Dave Bell]

> Not indiscriminate.
>

Indiscriminate - done at random or without careful judgement.

Considering that Daniel is complaining that QUIC is broken, it certainly
seems like some network operators are subjecting all UDP traffic on their
network to the same policers. This feels pretty indiscriminate to me.

I'm all for policing the known baddies, such as CHARGEN and NTP, but to
discard UDP for fun is like taking a sledgehammer where a scalpel will do.


> Access networks need controls to maintain uptime against the non-stop
> barrage of ddos attacks. I am sure you have seen the headlines and mails on
> this list, ddos is hard to deal with. Access network will use whatever
> tools are required to keep the pagers quiet and customers happy.
>

I operate an access network that does not blanket police UDP. Google give
me a dashboard that tell me 45% of requests  were served happily by QUIC,
and I have no customers complaining about things not working, and our
pagers are silent.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 10:19 AM Blake Hudson  wrote:

>
>
> On 2/19/2020 3:21 PM, Daniel Sterling wrote:
> > On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:
> >> Yeah, that was a nice surprise to find that my tethered LTE connection
> >> was out performing my wired cable modem service. Of course, I had
> >> already signed up for a year of service and there were early termination
> >> fees for cancelling... that and there are no other wireline providers
> >> available at my home (not even ATT).
> > So we're left with some questions:
> >
> > 1. It's clear I'm not the only one experiencing this issue. How
> > widespread is this problem, really? Has it gotten rather worse over
> > the past ~year?
> >
> > 2. Are customers of larger ISPs much more impacted than customers of
> > smaller ones that (assumedly) don't have to deprioritize UDP so much?
> > 2a. If users *are* impacted, as Blake notes, they may not be able to
> > switch ISPs to improve their lot.. will customers complain to their
> > ISP or to Google?
> >
> > 3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
> > only works on v6 (and if it in fact continues to actively BREAK
> > v4-only users), then is this v6's "killer app" that will drive
> > adoption?
> > 3a. Or will this issue hinder HTTP/3 deployment (or cause mass
> > blocking of UDP on clients)?
> >
> > 4. Will ISPs be willing to give UDP traffic higher priority to improve
> > user experience? Will that only happen once HTTP/3 is widely deployed?
> >
> > 5. We can only assume Google is aware of this issue; will Google work
> > to improve QUIC fallback to TCP, or will they work with ISPs to get
> > QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
> > actively encourage QUIC to break v4 at the expensive of current user
> > experience?
> > 5a. Will another company that wants HTTP/3 to succeed take the mantle
> > and work with ISPs to improve the situation? I'm reminded of when
> > Microsoft worked with ISPs to ensure xbox UDP traffic would transit
> > properly
> >
> > -- Dan
> Dan, my experience with Cox is that their standard cable internet
> package (advertised as 150Mbps) rate limits UDP to ~10Mbps. This appears
> to be controlled via the cable modem config file which is enforced by
> both the cable modem and the CMTS. I do not know if this is per flow or
> per circuit or affects IP4 differently than IP6. I suspect that someone
> at Cox decided that the only applications using UDP were VoIP and DNS
> and that those applications never needed more than 1Mbps so anything
> else must be "bad" and should be stopped. Whether "bad" means harmful to
> network operation, harmful to support costs, or harmful to profits, I do
> not know.
>
> Your comments seem to differentiate IP4 vs IP6, but I don't believe that
> is relevant to the issue of an ISP throttling or breaking specific
> applications. If you have evidence that UDP on IP4 is treated
> differently than UDP on IP6 by your provider, without further
> information I would suspect that this is simply an unintentional over
> sight on their part.



This is your misunderstanding. The protections are to drop ipv4 udp because
that is where the ddos / iot trash is , not v6 for now

>
>
> Perhaps the attention you've generated on this topic, along with the
> adoption of additional UDP based applications like QUIC, will encourage
> ISPs to treat UDP in a more neutral manner and not simply see UDP as
> something that is "bad".
>

Dropping udp is not from a “best practice” doc from a vendor, it is
deployed by network ops folks that are trying to sleep at night.


> --Blake
>

Re: QUIC traffic throttled on AT residential

[Blake Hudson]

On 2/19/2020 3:21 PM, Daniel Sterling wrote:

On Wed, Feb 19, 2020 at 3:34 PM Blake Hudson  wrote:

Yeah, that was a nice surprise to find that my tethered LTE connection
was out performing my wired cable modem service. Of course, I had
already signed up for a year of service and there were early termination
fees for cancelling... that and there are no other wireline providers
available at my home (not even ATT).

So we're left with some questions:

1. It's clear I'm not the only one experiencing this issue. How
widespread is this problem, really? Has it gotten rather worse over
the past ~year?

2. Are customers of larger ISPs much more impacted than customers of
smaller ones that (assumedly) don't have to deprioritize UDP so much?
2a. If users *are* impacted, as Blake notes, they may not be able to
switch ISPs to improve their lot.. will customers complain to their
ISP or to Google?

3. How much worse is the problem when using v4 UDP QUIC vs v6? If QUIC
only works on v6 (and if it in fact continues to actively BREAK
v4-only users), then is this v6's "killer app" that will drive
adoption?
3a. Or will this issue hinder HTTP/3 deployment (or cause mass
blocking of UDP on clients)?

4. Will ISPs be willing to give UDP traffic higher priority to improve
user experience? Will that only happen once HTTP/3 is widely deployed?

5. We can only assume Google is aware of this issue; will Google work
to improve QUIC fallback to TCP, or will they work with ISPs to get
QUIC (esp v4 QUIC) prioritized, or will they do nothing, or will they
actively encourage QUIC to break v4 at the expensive of current user
experience?
5a. Will another company that wants HTTP/3 to succeed take the mantle
and work with ISPs to improve the situation? I'm reminded of when
Microsoft worked with ISPs to ensure xbox UDP traffic would transit
properly

-- Dan
Dan, my experience with Cox is that their standard cable internet 
package (advertised as 150Mbps) rate limits UDP to ~10Mbps. This appears 
to be controlled via the cable modem config file which is enforced by 
both the cable modem and the CMTS. I do not know if this is per flow or 
per circuit or affects IP4 differently than IP6. I suspect that someone 
at Cox decided that the only applications using UDP were VoIP and DNS 
and that those applications never needed more than 1Mbps so anything 
else must be "bad" and should be stopped. Whether "bad" means harmful to 
network operation, harmful to support costs, or harmful to profits, I do 
not know.


Your comments seem to differentiate IP4 vs IP6, but I don't believe that 
is relevant to the issue of an ISP throttling or breaking specific 
applications. If you have evidence that UDP on IP4 is treated 
differently than UDP on IP6 by your provider, without further 
information I would suspect that this is simply an unintentional over 
sight on their part.


Perhaps the attention you've generated on this topic, along with the 
adoption of additional UDP based applications like QUIC, will encourage 
ISPs to treat UDP in a more neutral manner and not simply see UDP as 
something that is "bad".


--Blake

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 9:56 AM Dave Bell  wrote:

>
> On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:
>
>> UDP is broken
>>
>
> I would argue that UDP isn't broken. Networks which drop it
> indiscriminately are broken.
>
Not indiscriminate.

As Google was informed by network operators all along since 2014, ipv4 UDP
is a major uptime threat via DDoS to access networks.

Access networks need controls to maintain uptime against the non-stop
barrage of ddos attacks. I am sure you have seen the headlines and mails on
this list, ddos is hard to deal with. Access network will use whatever
tools are required to keep the pagers quiet and customers happy.

Google choose not to be sensitive to that, they were told where the
landmines were, they choose to go on anyhow.

Re: QUIC traffic throttled on AT residential

[Aled Morris via NANOG]

On Thu, 20 Feb 2020 at 15:57, Dave Bell  wrote:

>
> On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:
>
>> UDP is broken
>>
>
> I would argue that UDP isn't broken. Networks which drop it
> indiscriminately are broken.
>

Does this errant network behaviour not impact RTP applications like video
streams?

Aled

Re: Re: QUIC traffic throttled on AT residential {5403687}

[Dave Bell]

I didn't contact you. Fuck off.

On Thu, 20 Feb 2020 at 16:01, Dead.net Customer Service <
d...@wmgcustomerservice.com> wrote:

> Thank you for contacting Dead.net customer service.
>
> Our customer service team will reply to your email as soon as possible.
>
> Due to our current email volume, replies may take up to 2 business days.
>
>  We apologize for the inconvenience and thank you for your patience.
>
>  Sincerely,
>
>  The Dead.net Customer Service Team
>
>
>

Re: QUIC traffic throttled on AT residential

[Dave Bell]

On Thu, 20 Feb 2020 at 15:31, Ca By  wrote:

> UDP is broken
>

I would argue that UDP isn't broken. Networks which drop it
indiscriminately are broken.

RE: QUIC traffic throttled on AT residential

[Keith Medcalf]

On Thursday, 20 February, 2020 08:31, Ca By  wrote:

>On Thu, Feb 20, 2020 at 8:34 AM Tom Beecher  wrote:

>   I only wish I were insane; but from where I'm sitting, QUIC
>has broken
>   my internet, and the resolution is blocking QUIC.
>
>   The QUIC protocol itself isn't breaking anything ; some middlebox is
>breaking QUIC. It's likely collateral damage from honest attempts to
>mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
>degree, but the upstream issue still remains.

>   I recall reading a draft from the WG about having things open a
>parallel TCP connection in case UDP got horked for seamless fallback, but
>I don't remember if it ever moved past that, or if anyone actually
>implemented it.

>UDP is broken
>
>The Google devs said it would in fine in 2014
>
>They said it would be “exciting”

UDP is not "broken".  UDP is the UNRELIABLE DATAGRAM PROTOCOL and it is living 
up to its name!  What is broken is something pretending that the "U" in UDP 
means something other than UNRELIABLE and then pretending their use of an 
unreliable delivery mechanism makes it reliable when that clearly is not the 
case.

In other words, applying a coat of paint to a sows ear to make it look like a 
silk purse does not make the sows ear into a silk purse.  It is still just a 
sows ear painted to look like a silk purse, and still behaves like a sows ear.

--
The fact that there's a Highway to Hell but only a Stairway to Heaven says a 
lot about anticipated traffic volume.

Re: QUIC traffic throttled on AT residential

[Ca By]

On Thu, Feb 20, 2020 at 8:34 AM Tom Beecher  wrote:

> I only wish I were insane; but from where I'm sitting, QUIC has broken
>> my internet, and the resolution is blocking QUIC.
>>
>
> The QUIC protocol itself isn't breaking anything ; some middlebox is
> breaking QUIC. It's likely collateral damage from honest attempts to
> mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
> degree, but the upstream issue still remains.
>
> I recall reading a draft from the WG about having things open a
> parallel TCP connection in case UDP got horked for seamless fallback, but I
> don't remember if it ever moved past that, or if anyone actually
> implemented it.
>
>
UDP is broken

The Google devs said it would in fine in 2014

They said it would be “exciting”


https://groups.google.com/a/chromium.org/forum/m/#!msg/proto-quic/09L5YD2u5xU/dK6oU2UA67YJ



> On Wed, Feb 19, 2020 at 11:33 PM Daniel Sterling <
> sterling.dan...@gmail.com> wrote:
>
>> On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
>>  wrote:
>> > A problem of QUIC with NAT is that existing NAT can not detect
>> > graceful shutdown of QUIC and must depends on timeout.
>> >
>> > So, port numbers may be used up before timeout.
>>
>> Hmm, this is not what is happening.
>>
>> I managed to (fairly easily!) reproduce the issue earlier tonight. I
>> generated a fair bit of UDP traffic via xbox, a corporate VPN, and
>> youtube over quic.
>>
>> Sure enough, after about 45 minutes, the YouTube app on my iPad paused
>> and then auto-reset the stream quality to "144p" resolution.
>>
>> I was able to set it back to 720p60, but that only lasted about 2
>> minutes before the stream stopped playing. I waited several minutes
>> for it to resume; it did not. So, I then blocked UDP 443 on my router.
>> The video then *immediately* resumed at 720p60.
>>
>> I didn't run tcpdump but I did grab some screenshots of iftop. It
>> looks like my iPad connected to AS15169 with a single UDP connection.
>> I see one consistent source and dest IP / port combo for those 10s of
>> minutes, up until UDP is blocked. Local port 58053, remote port 443 on
>> the same IP for the whole time.
>>
>> At first the connection averages around 2mbps; when the issue occurs,
>> I see it has dropped to a rate of under 200kbps.
>>
>> I've no idea what to make of that. Surely Google isn't throttling
>> their traffic to me? If so why allow a fallback to TCP?
>>
>> When I originally discovered this issue, I was of course not trying to
>> do anything odd with my internet connection. And I didn't immediately
>> know QUIC was the issue.
>>
>> Only after it happened several times did I dig into the traffic and
>> then block QUIC, and I was shocked to see that both resolve the issue
>> and prevent its recurrence.
>>
>> So again -- I hit this issue repeatedly without trying to --
>>
>> And just now, I was able to reproduce it simply by generating a bit of
>> UDP traffic on purpose!
>>
>> I only wish I were insane; but from where I'm sitting, QUIC has broken
>> my internet, and the resolution is blocking QUIC.
>>
>> -- Dan
>>
>

Re: QUIC traffic throttled on AT residential

[Tom Beecher]

>
> I only wish I were insane; but from where I'm sitting, QUIC has broken
> my internet, and the resolution is blocking QUIC.
>

The QUIC protocol itself isn't breaking anything ; some middlebox is
breaking QUIC. It's likely collateral damage from honest attempts to
mitigate bad stuff. Blocking QUIC at your home edge surely helps to some
degree, but the upstream issue still remains.

I recall reading a draft from the WG about having things open a
parallel TCP connection in case UDP got horked for seamless fallback, but I
don't remember if it ever moved past that, or if anyone actually
implemented it.


On Wed, Feb 19, 2020 at 11:33 PM Daniel Sterling 
wrote:

> On Wed, Feb 19, 2020 at 8:27 PM Masataka Ohta
>  wrote:
> > A problem of QUIC with NAT is that existing NAT can not detect
> > graceful shutdown of QUIC and must depends on timeout.
> >
> > So, port numbers may be used up before timeout.
>
> Hmm, this is not what is happening.
>
> I managed to (fairly easily!) reproduce the issue earlier tonight. I
> generated a fair bit of UDP traffic via xbox, a corporate VPN, and
> youtube over quic.
>
> Sure enough, after about 45 minutes, the YouTube app on my iPad paused
> and then auto-reset the stream quality to "144p" resolution.
>
> I was able to set it back to 720p60, but that only lasted about 2
> minutes before the stream stopped playing. I waited several minutes
> for it to resume; it did not. So, I then blocked UDP 443 on my router.
> The video then *immediately* resumed at 720p60.
>
> I didn't run tcpdump but I did grab some screenshots of iftop. It
> looks like my iPad connected to AS15169 with a single UDP connection.
> I see one consistent source and dest IP / port combo for those 10s of
> minutes, up until UDP is blocked. Local port 58053, remote port 443 on
> the same IP for the whole time.
>
> At first the connection averages around 2mbps; when the issue occurs,
> I see it has dropped to a rate of under 200kbps.
>
> I've no idea what to make of that. Surely Google isn't throttling
> their traffic to me? If so why allow a fallback to TCP?
>
> When I originally discovered this issue, I was of course not trying to
> do anything odd with my internet connection. And I didn't immediately
> know QUIC was the issue.
>
> Only after it happened several times did I dig into the traffic and
> then block QUIC, and I was shocked to see that both resolve the issue
> and prevent its recurrence.
>
> So again -- I hit this issue repeatedly without trying to --
>
> And just now, I was able to reproduce it simply by generating a bit of
> UDP traffic on purpose!
>
> I only wish I were insane; but from where I'm sitting, QUIC has broken
> my internet, and the resolution is blocking QUIC.
>
> -- Dan
>

Re: QUIC traffic throttled on AT residential

[Jay Hennigan]

On 2/18/20 18:40, nanog-l...@contactdaniel.net wrote:

Growing prevalence of IPv6-only 
sites is probably the only thing that will get a lot of access networks 
to support v6.


I recall a similar idea called "The Great IPv6 Experiment" back in 2007. ;-)


--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV