Re: Google peering in LAX

[Matthew Petach]

It may be worthwhile for you to consider adding 15169 to your "Don't accept
$tier1 prefixes from other peers" policy in your inbound policy chain.

I've found that there's a set of $LARGE_ENOUGH networks that, even though
they're not literal $tier1 providers, benefit from that same level of
filtering.  You wouldn't want to try sending Level3  traffic through a
random peer, as the results would likely be catastrophic; so, make use of
that same filter rule in your inbound policy to filter out hearing 15169
prefixes from other peering sessions.

The caveat to that, of course, is that successful failover will mean
carrying traffic across your backbone when your 15169 prefixes in one
location disappear during an outage/maintenance window, so make sure your
backbone is correctly sized to handle those reroute situations.  It also
means that multi-homed downstream customers are likely to send less
upstream traffic through you to reach Google.

But that *will* mean that no amount of leaking more specific prefixes
through other paths will unexpectedly cause your traffic to shift.

Matt



On Mon, Mar 2, 2020 at 5:39 PM Seth Mattinen  wrote:

> On 3/2/20 4:32 PM, Patrick W. Gilmore wrote:
> > That said, I fear this is going to be a problem long term. A blind “no
> /24s” filter is dangerous, plus it might solve all traffic issues. It is
> going to take effort to be sure you don’t get bitten by the Law Of
> Unintended Consequences.
>
>
> As soon as Google un-freezes new peering requests so I can get a direct
> peering that includes appropriate /24's I've been told offlist I should
> get (instead of the route server subset) I'll happily remove the transit
> filters. But I can only work with what I'm given.
>
>

Re: Google peering in LAX

[Seth Mattinen]

On 3/2/20 4:32 PM, Patrick W. Gilmore wrote:

That said, I fear this is going to be a problem long term. A blind “no /24s” 
filter is dangerous, plus it might solve all traffic issues. It is going to 
take effort to be sure you don’t get bitten by the Law Of Unintended 
Consequences.



As soon as Google un-freezes new peering requests so I can get a direct 
peering that includes appropriate /24's I've been told offlist I should 
get (instead of the route server subset) I'll happily remove the transit 
filters. But I can only work with what I'm given.

Re: Google peering in LAX

[Patrick W. Gilmore]

On Mar 2, 2020, at 6:30 PM, Seth Mattinen  wrote:
> On 3/2/20 3:09 PM, Patrick W. Gilmore wrote:
>> Your routers, your decision.
>> But how much traffic are you sending TO Google? Most people get the vast 
>> majority of traffic FROM Google. They send you videos, you send them ACKs. 
>> Does it matter where the ACKs go?
> 
> 
> A customer is complaining that data they're sending is going over a higher 
> latency (longer) path. I don't know what they're doing I don't generally ask 
> why, but they claim it's a problem for whatever they're doing and I don't 
> have a reason to doubt them. It's not youtube.
> 
> I agree that it's an undesirable long term solution but if filtering select 
> transit-only /24's shifts the path to peering and reduces latency, if the 
> customer is happy then I'm happy and if/when Google starts accepting peering 
> requests again I'll revisit it.

Again, your routers, your decision. But if I had a customer who was 
complaining, I would take steps to fix it.

Google is sending you prefixes over the IX. You have every right to send them 
traffic over the IX to those prefixes.

That said, I fear this is going to be a problem long term. A blind “no /24s” 
filter is dangerous, plus it might solve all traffic issues. It is going to 
take effort to be sure you don’t get bitten by the Law Of Unintended 
Consequences.

Good luck.

-- 
TTFN,
patrick

Legacy Concentric/XO Web Services Blocking?

[James Breeden]

NANOG,

Looking for anyone from XO or Legacy Concentric web hosting services (now 
VDMS). I have a mutual customer that is getting caught at some form of Web App 
firewall coming from a specific IP range.

Thank you!


James W. Breeden
Managing Partner

[logo_transparent_background]
Arenal Group: Arenal Consulting Group | Atheral | Ceteris Coin | Acilis Telecom 
| Pines Events and Media | BlueNinja
Corporate: PO Box 1063 | Smithville, TX 78957
Email: ja...@arenalgroup.co | office 512.360. 
| www.arenalgroup.co

Re: Google peering in LAX

[Seth Mattinen]

On 3/2/20 3:09 PM, Patrick W. Gilmore wrote:


Your routers, your decision.

But how much traffic are you sending TO Google? Most people get the vast 
majority of traffic FROM Google. They send you videos, you send them 
ACKs. Does it matter where the ACKs go?



A customer is complaining that data they're sending is going over a 
higher latency (longer) path. I don't know what they're doing I don't 
generally ask why, but they claim it's a problem for whatever they're 
doing and I don't have a reason to doubt them. It's not youtube.


I agree that it's an undesirable long term solution but if filtering 
select transit-only /24's shifts the path to peering and reduces 
latency, if the customer is happy then I'm happy and if/when Google 
starts accepting peering requests again I'll revisit it.

Re: Google peering in LAX

[Justin Seabrook-Rocha]

You hit the nail on the head. Google only seems to announce a subset of their 
routes to the route servers, but does announce all routes (for some definition 
of “all”) to direct peers. I notice this every time I turn up a new IX and 
traffic heads off onto my backbone instead of the local IX.

I did a spot check and I get that /24 via my direct peering (along with the 
/16).

Justin Seabrook-Rocha
-- 
Xenith || xen...@xenith.org || http://xenith.org/



> On Mar 2, 2020, at 12:40, Seth Mattinen  wrote:
> 
> Anyone know why Google announces only aggregates via peering and disaggregate 
> prefixes over transit?
> 
> For example, I had a customer complaining about a path that was taking the 
> long way instead of via peering and when I looked I saw:
> 
> Only 172.217.0.0/16 over Any2 LAX
> 
> That plus 172.217.14.0/24 over transit
> 
> Any inquiries to Google just get a generic "we're not setting up any new 
> peering but we're on route servers" response for almost a year now. Or is it 
> because they don't send the /24's to route servers and I'm stuck until they 
> finish their forever improvement project to turn up a direct neighbor?

Re: Google peering in LAX

[Seth Mattinen]

On 3/2/20 3:02 PM, Randy Carpenter wrote:

I would say it would be best to see if you can get a direct peer with Google 
via the IX. I have done this with some of the ISPs I work with. It was no 
additional cost since the physical connections are already in place and 
actually was highly recommended when first turning up the IX circuits.



They won't; I just get a canned message that says they aren't doing any 
new IX peering "as we improve our automation systems".

Re: Google peering in LAX

[Patrick W. Gilmore]

On Mar 2, 2020, at 17:38, Seth Mattinen  wrote:
> On 3/2/20 2:20 PM, Hugo Slabbert wrote:
>> I believe Owen was referring here to Google's actions: that the disagg is 
>> the antisocial behaviour and that transit providers (the people they are 
>> paying) would be more tolerant of that antisocial behaviour than would be 
>> peers (the people they are not paying).
> 
> 
> I suppose that one went over my head.
> 
> To clarify I am the one with peering in LAX and I'm only seeing the big 
> aggregates via the Any2 Easy servers. At the moment I can only infer that 
> Google announces aggregates to the route servers and maybe one only gets the 
> /24's after you turn up a direct neighbor or PNI, but there's no way to do 
> that since Google isn't accepting new peering requests and steers such 
> requests back to what's available on route servers.
> 
> I suppose what I could do is filter /24's from 15169$ in the absence of being 
> able to see if a direct/PNI peering would include them where route servers do 
> not.

Your routers, your decision.

But how much traffic are you sending TO Google? Most people get the vast 
majority of traffic FROM Google. They send you videos, you send them ACKs. Does 
it matter where the ACKs go?

-- 
TTFN,
patrick

Re: Google peering in LAX

[Randy Carpenter]

- On Mar 2, 2020, at 5:37 PM, Seth Mattinen se...@rollernet.us wrote:

> I suppose that one went over my head.
> 
> To clarify I am the one with peering in LAX and I'm only seeing the big
> aggregates via the Any2 Easy servers. At the moment I can only infer
> that Google announces aggregates to the route servers and maybe one only
> gets the /24's after you turn up a direct neighbor or PNI, but there's
> no way to do that since Google isn't accepting new peering requests and
> steers such requests back to what's available on route servers.
> 
> I suppose what I could do is filter /24's from 15169$ in the absence of
> being able to see if a direct/PNI peering would include them where route
> servers do not.

I would say it would be best to see if you can get a direct peer with Google 
via the IX. I have done this with some of the ISPs I work with. It was no 
additional cost since the physical connections are already in place and 
actually was highly recommended when first turning up the IX circuits.


-Randy

Re: China’s Slow Transnational Network

[Tom Paseka via NANOG]

Most of the performance hit is because of commercial actions, not
censorship.

When there is a tri-opoly, with no opportunity of competition, its easily
possible to set prices which are very different than market conditions.
This is what is happening here.

Prices are set artificially high, so their interconnection partners wont
purchase enough capacity. additionally, the three don't purchase enough to
cover demand for their own network. Results in congestion.

On Mon, Mar 2, 2020 at 2:49 PM Pengxiong Zhu  wrote:

> You seem to be implying that you don't believe/can't see the GFW
>
>
> No, that's not what I meant. I thought mandatory content filtering at the
> border means traffic throttling at the border, deliberately or accidentally
> rate-limiting the traffic, now
> I think he was referring to GFW and the side effect of deep packet
> inspection.
>
> In fact, we designed a small experiment to locate the hops with GFW
> presence, and then try to match them with the bottleneck hops. We found
> only in 34.45% of the cases, the GFW hops match the bottleneck hops.
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>
>
> On Mon, Mar 2, 2020 at 1:13 PM Matt Corallo  wrote:
>
>> > find out direct evidence of mandatory content filtering at the border
>>
>> You seem to be implying that you don't believe/can't see the GFW, which
>> seems surprising. I've personally had issues with traffic crossing it
>> getting RST'd (luckily I was fortunate enough to cross through a GFW
>> instance which was easy to avoid with a simple iptables DROP), but its
>> also one of the most well-studied bits of opaque internet censorship
>> gear in the world. I'm not sure how you could possibly miss it.
>>
>> Matt
>>
>> On 3/2/20 2:55 PM, Pengxiong Zhu wrote:
>> > Yes, we agree. The poor transnational Internet performance effectively
>> > puts any foreign business that does not have a physical presence (i.e.,
>> > servers) in China at a disadvantage.
>> > The challenge is to find out direct evidence to prove mandatory content
>> > filtering at the border, if the government is actually doing it.
>> >
>> > Best,
>> > Pengxiong Zhu
>> > Department of Computer Science and Engineering
>> > University of California, Riverside
>> >
>> >
>> > On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo > > > wrote:
>> >
>> > It also gives local competitors a leg up by helping domestic apps
>> > perform better simply by being hosted domestically (or making
>> > foreign players host inside China).
>> >
>> >> On Mar 2, 2020, at 11:27, Ben Cannon > >> > wrote:
>> >>
>> >> 
>> >> It’s the Government doing mandatory content filtering at the
>> >> border.  Their hardware is either deliberately or accidentally
>> >> poor-performing.
>> >>
>> >> I believe providing limited and throttled external connectivity
>> >> may be deliberate; think of how that curtails for one thing;
>> >> streaming video?
>> >>
>> >> -Ben.
>> >>
>> >> -Ben Cannon
>> >> CEO 6x7 Networks & 6x7 Telecom, LLC
>> >> b...@6by7.net 
>> >>
>> >>
>> >>
>> >>> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu > >>> > wrote:
>> >>>
>> >>> Hi all,
>> >>>
>> >>> We are a group of researchers at University of California,
>> >>> Riverside who have been working on measuring the transnational
>> >>> network performance (and have previously asked questions on the
>> >>> mailing list). Our work has now led to a publication in
>> >>> Sigmetrics 2020 and we are eager to share some
>> >>> interesting findings.
>> >>>
>> >>> We find China's transnational networks have extremely poor
>> >>> performance when accessing foreign sites, where the throughput is
>> >>> often persistently
>> >>> low (e.g., for the majority of the daytime). Compared to other
>> >>> countries we measured including both developed and developing,
>> >>> China's transnational network performance is among the worst
>> >>> (comparable and even worse than some African countries).
>> >>>
>> >>> Measuring from more than 400 pairs of mainland China and foreign
>> >>> nodes over more than 53 days, our result shows when data
>> >>> transferring from foreign nodes to China, 79% of measured
>> >>> connections has throughput lower than the 1Mbps, sometimes it is
>> >>> even much lower. The slow speed occurs only during certain times
>> >>> and forms a diurnal pattern that resembles congestion
>> >>> (irrespective of network protocol and content), please see the
>> >>> following figure. The diurnal pattern is fairly stable, 80% to
>> >>> 95% of the transnational connections have a less than 3 hours
>> >>> standard deviation of the slowdown hours each day over the entire
>> >>> duration. However, the speed rises up from 1Mbps to 4Mbps in
>> >>>

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

DDoS traffic is coming from China to the outside world, which
should saturate the upstream link of China, however, what we observed is
that the upstream link has high and stable performance, while the
downstream link of China, which is traffic coming from the outside world to
China, is suffering from slow speed.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 8:11 AM Compton, Rich A 
wrote:

> My guess is that it’s all the DDoS traffic coming from China saturating
> the links.
>
>
>
> *From: *NANOG Email List  on behalf of Pengxiong
> Zhu 
> *Date: *Monday, March 2, 2020 at 8:58 AM
> *To: *NANOG list 
> *Cc: *Zhiyun Qian 
> *Subject: *China’s Slow Transnational Network
>
>
>
> Hi all,
>
>
>
> We are a group of researchers at University of California, Riverside who
> have been working on measuring the transnational network performance (and
> have previously asked questions on the mailing list). Our work has now led
> to a publication in Sigmetrics 2020 and we are eager to share some
>
> interesting findings.
>
>
>
> We find China's transnational networks have extremely poor performance
> when accessing foreign sites, where the throughput is often persistently
>
> low (e.g., for the majority of the daytime). Compared to other countries
> we measured including both developed and developing, China's transnational
> network performance is among the worst (comparable and even worse than some
> African countries).
>
>
>
> Measuring from more than 400 pairs of mainland China and foreign nodes
> over more than 53 days, our result shows when data transferring from
> foreign nodes to China, 79% of measured connections has throughput lower
> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
> during certain times and forms a diurnal pattern that resembles congestion
> (irrespective of network protocol and content), please see the following
> figure. The diurnal pattern is fairly stable, 80% to 95% of the
> transnational connections have a less than 3 hours standard deviation of
> the slowdown hours each day over the entire duration. However, the speed
> rises up from 1Mbps to 4Mbps in about half an hour.
>
>
>
> [image: blob:null/71cf5a6a-3841-41ce-a1d4-207b59182189]
>
>
>
> We are able to confirm that high packet loss rates and delays are incurred
> in the foreign-to-China direction only. Moreover, the end-to-end loss rate
> could rise up to 40% during the slow period, with ~15% on average.
>
>
>
> There are a few things noteworthy regarding the phenomenon. First of all,
> all traffic types are treated equally, HTTP(S), VPN, etc., which means it
> is discriminating or differentiating any specific kinds of traffic. Second,
> we found for 71% of connections, the bottleneck is located inside China
> (the second hop after entering China or further), which means that it is
> mostly unrelated to the transnational link itself (e.g., submarine cable).
> Yet we never observed any such domestic traffic slowdowns within China.
>
> Assuming this is due to congestion, it is unclear why the infrastructures
> within China that handles transnational traffic is not even capable to
> handle the capacity of transnational links, e.g., submarine cable, which
> maybe the most expensive investment themselves.
>
>
>
> Here is the link to our paper:
>
> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>
>
> We appreciate any comments or feedback.
>
> --
>
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
> The contents of this e-mail message and
> any attachments are intended solely for the
> addressee(s) and may contain confidential
> and/or legally privileged information. If you
> are not the intended recipient of this message
> or if this message has been addressed to you
> in error, please immediately alert the sender
> by reply e-mail and then delete this message
> and any attachments. If you are not the
> intended recipient, you are notified that
> any use, dissemination, distribution, copying,
> or storage of this message or any attachment
> is strictly prohibited.
>

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

Yes, CERNET has indeed smaller slowdown period(4 hours) than commodity
networks(12 hours), but still has slowdown.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 2:36 PM David Burns  wrote:

> Did you compare CERNET with commodity networks?  (My
> anecdotal observations from a couple years ago suggest that Internet2 to
> CERNET is very good when other paths are poor to unusable.)
>
> --David Burns
>
> On Mon, Mar 2, 2020 at 7:58 AM Pengxiong Zhu  wrote:
>
>> Hi all,
>>
>> We are a group of researchers at University of California, Riverside who
>> have been working on measuring the transnational network performance (and
>> have previously asked questions on the mailing list). Our work has now led
>> to a publication in Sigmetrics 2020 and we are eager to share some
>> interesting findings.
>>
>> We find China's transnational networks have extremely poor performance
>> when accessing foreign sites, where the throughput is often persistently
>> low (e.g., for the majority of the daytime). Compared to other countries
>> we measured including both developed and developing, China's transnational
>> network performance is among the worst (comparable and even worse than some
>> African countries).
>>
>> Measuring from more than 400 pairs of mainland China and foreign nodes
>> over more than 53 days, our result shows when data transferring from
>> foreign nodes to China, 79% of measured connections has throughput lower
>> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
>> during certain times and forms a diurnal pattern that resembles congestion
>> (irrespective of network protocol and content), please see the following
>> figure. The diurnal pattern is fairly stable, 80% to 95% of the
>> transnational connections have a less than 3 hours standard deviation of
>> the slowdown hours each day over the entire duration. However, the speed
>> rises up from 1Mbps to 4Mbps in about half an hour.
>>
>>
>> We are able to confirm that high packet loss rates and delays are
>> incurred in the foreign-to-China direction only. Moreover, the end-to-end
>> loss rate could rise up to 40% during the slow period, with ~15% on average.
>>
>> There are a few things noteworthy regarding the phenomenon. First of all,
>> all traffic types are treated equally, HTTP(S), VPN, etc., which means it
>> is discriminating or differentiating any specific kinds of traffic. Second,
>> we found for 71% of connections, the bottleneck is located inside China
>> (the second hop after entering China or further), which means that it is
>> mostly unrelated to the transnational link itself (e.g., submarine cable).
>> Yet we never observed any such domestic traffic slowdowns within China.
>> Assuming this is due to congestion, it is unclear why the infrastructures
>> within China that handles transnational traffic is not even capable to
>> handle the capacity of transnational links, e.g., submarine cable, which
>> maybe the most expensive investment themselves.
>>
>> Here is the link to our paper:
>> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>>
>> We appreciate any comments or feedback.
>> --
>>
>> Best,
>> Pengxiong Zhu
>> Department of Computer Science and Engineering
>> University of California, Riverside
>>
>

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

>
> You seem to be implying that you don't believe/can't see the GFW


No, that's not what I meant. I thought mandatory content filtering at the
border means traffic throttling at the border, deliberately or accidentally
rate-limiting the traffic, now
I think he was referring to GFW and the side effect of deep packet
inspection.

In fact, we designed a small experiment to locate the hops with GFW
presence, and then try to match them with the bottleneck hops. We found
only in 34.45% of the cases, the GFW hops match the bottleneck hops.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 1:13 PM Matt Corallo  wrote:

> > find out direct evidence of mandatory content filtering at the border
>
> You seem to be implying that you don't believe/can't see the GFW, which
> seems surprising. I've personally had issues with traffic crossing it
> getting RST'd (luckily I was fortunate enough to cross through a GFW
> instance which was easy to avoid with a simple iptables DROP), but its
> also one of the most well-studied bits of opaque internet censorship
> gear in the world. I'm not sure how you could possibly miss it.
>
> Matt
>
> On 3/2/20 2:55 PM, Pengxiong Zhu wrote:
> > Yes, we agree. The poor transnational Internet performance effectively
> > puts any foreign business that does not have a physical presence (i.e.,
> > servers) in China at a disadvantage.
> > The challenge is to find out direct evidence to prove mandatory content
> > filtering at the border, if the government is actually doing it.
> >
> > Best,
> > Pengxiong Zhu
> > Department of Computer Science and Engineering
> > University of California, Riverside
> >
> >
> > On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo  > > wrote:
> >
> > It also gives local competitors a leg up by helping domestic apps
> > perform better simply by being hosted domestically (or making
> > foreign players host inside China).
> >
> >> On Mar 2, 2020, at 11:27, Ben Cannon  >> > wrote:
> >>
> >> 
> >> It’s the Government doing mandatory content filtering at the
> >> border.  Their hardware is either deliberately or accidentally
> >> poor-performing.
> >>
> >> I believe providing limited and throttled external connectivity
> >> may be deliberate; think of how that curtails for one thing;
> >> streaming video?
> >>
> >> -Ben.
> >>
> >> -Ben Cannon
> >> CEO 6x7 Networks & 6x7 Telecom, LLC
> >> b...@6by7.net 
> >>
> >>
> >>
> >>> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu  >>> > wrote:
> >>>
> >>> Hi all,
> >>>
> >>> We are a group of researchers at University of California,
> >>> Riverside who have been working on measuring the transnational
> >>> network performance (and have previously asked questions on the
> >>> mailing list). Our work has now led to a publication in
> >>> Sigmetrics 2020 and we are eager to share some
> >>> interesting findings.
> >>>
> >>> We find China's transnational networks have extremely poor
> >>> performance when accessing foreign sites, where the throughput is
> >>> often persistently
> >>> low (e.g., for the majority of the daytime). Compared to other
> >>> countries we measured including both developed and developing,
> >>> China's transnational network performance is among the worst
> >>> (comparable and even worse than some African countries).
> >>>
> >>> Measuring from more than 400 pairs of mainland China and foreign
> >>> nodes over more than 53 days, our result shows when data
> >>> transferring from foreign nodes to China, 79% of measured
> >>> connections has throughput lower than the 1Mbps, sometimes it is
> >>> even much lower. The slow speed occurs only during certain times
> >>> and forms a diurnal pattern that resembles congestion
> >>> (irrespective of network protocol and content), please see the
> >>> following figure. The diurnal pattern is fairly stable, 80% to
> >>> 95% of the transnational connections have a less than 3 hours
> >>> standard deviation of the slowdown hours each day over the entire
> >>> duration. However, the speed rises up from 1Mbps to 4Mbps in
> >>> about half an hour.
> >>>
> >>>
> >>> We are able to confirm that high packet loss rates and delays are
> >>> incurred in the foreign-to-China direction only. Moreover, the
> >>> end-to-end loss rate could rise up to 40% during the slow period,
> >>> with ~15% on average.
> >>>
> >>> There are a few things noteworthy regarding the phenomenon. First
> >>> of all, all traffic types are treated equally, HTTP(S), VPN,
> >>> etc., which means it is discriminating or differentiating any
> >>> specific kinds of traffic. Second, we found for 71% of
> >>> connections, the bottleneck is located 

Re: China’s Slow Transnational Network

[David Burns]

Did you compare CERNET with commodity networks?  (My anecdotal observations
from a couple years ago suggest that Internet2 to CERNET is very good when
other paths are poor to unusable.)

--David Burns

On Mon, Mar 2, 2020 at 7:58 AM Pengxiong Zhu  wrote:

> Hi all,
>
> We are a group of researchers at University of California, Riverside who
> have been working on measuring the transnational network performance (and
> have previously asked questions on the mailing list). Our work has now led
> to a publication in Sigmetrics 2020 and we are eager to share some
> interesting findings.
>
> We find China's transnational networks have extremely poor performance
> when accessing foreign sites, where the throughput is often persistently
> low (e.g., for the majority of the daytime). Compared to other countries
> we measured including both developed and developing, China's transnational
> network performance is among the worst (comparable and even worse than some
> African countries).
>
> Measuring from more than 400 pairs of mainland China and foreign nodes
> over more than 53 days, our result shows when data transferring from
> foreign nodes to China, 79% of measured connections has throughput lower
> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
> during certain times and forms a diurnal pattern that resembles congestion
> (irrespective of network protocol and content), please see the following
> figure. The diurnal pattern is fairly stable, 80% to 95% of the
> transnational connections have a less than 3 hours standard deviation of
> the slowdown hours each day over the entire duration. However, the speed
> rises up from 1Mbps to 4Mbps in about half an hour.
>
>
> We are able to confirm that high packet loss rates and delays are incurred
> in the foreign-to-China direction only. Moreover, the end-to-end loss rate
> could rise up to 40% during the slow period, with ~15% on average.
>
> There are a few things noteworthy regarding the phenomenon. First of all,
> all traffic types are treated equally, HTTP(S), VPN, etc., which means it
> is discriminating or differentiating any specific kinds of traffic. Second,
> we found for 71% of connections, the bottleneck is located inside China
> (the second hop after entering China or further), which means that it is
> mostly unrelated to the transnational link itself (e.g., submarine cable).
> Yet we never observed any such domestic traffic slowdowns within China.
> Assuming this is due to congestion, it is unclear why the infrastructures
> within China that handles transnational traffic is not even capable to
> handle the capacity of transnational links, e.g., submarine cable, which
> maybe the most expensive investment themselves.
>
> Here is the link to our paper:
> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>
> We appreciate any comments or feedback.
> --
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>

Re: Google peering in LAX

[Seth Mattinen]

On 3/2/20 2:20 PM, Hugo Slabbert wrote:
I believe Owen was referring here to Google's actions: that the disagg 
is the antisocial behaviour and that transit providers (the people they 
are paying) would be more tolerant of that antisocial behaviour than 
would be peers (the people they are not paying).



I suppose that one went over my head.

To clarify I am the one with peering in LAX and I'm only seeing the big 
aggregates via the Any2 Easy servers. At the moment I can only infer 
that Google announces aggregates to the route servers and maybe one only 
gets the /24's after you turn up a direct neighbor or PNI, but there's 
no way to do that since Google isn't accepting new peering requests and 
steers such requests back to what's available on route servers.


I suppose what I could do is filter /24's from 15169$ in the absence of 
being able to see if a direct/PNI peering would include them where route 
servers do not.

Re: Google peering in LAX

[Owen DeLong]

Yes… That’s correct.

Owen


> On Mar 2, 2020, at 2:20 PM, Hugo Slabbert  wrote:
> 
> I believe Owen was referring here to Google's actions: that the disagg is the 
> antisocial behaviour and that transit providers (the people they are paying) 
> would be more tolerant of that antisocial behaviour than would be peers (the 
> people they are not paying). 
> 
> On Mon., Mar. 2, 2020, 13:19 Seth Mattinen  > wrote:
> On 3/2/20 12:44 PM, Owen DeLong wrote:
> > In part, it might be because people you’re not paying may be less tolerant 
> > of anti-social behavior than people you are paying.
> > 
> 
> I'm not sure how I was being offensive but OK.

Re: Google peering in LAX

[Hugo Slabbert]

I believe Owen was referring here to Google's actions: that the disagg is
the antisocial behaviour and that transit providers (the people they are
paying) would be more tolerant of that antisocial behaviour than would be
peers (the people they are not paying).

On Mon., Mar. 2, 2020, 13:19 Seth Mattinen  wrote:

> On 3/2/20 12:44 PM, Owen DeLong wrote:
> > In part, it might be because people you’re not paying may be less
> tolerant of anti-social behavior than people you are paying.
> >
>
> I'm not sure how I was being offensive but OK.
>

Re: China’s Slow Transnational Network

[Scott Weeks]

In fact, Great Canon (GC) [55] is such an in-path system. But it 
is known for intercepting a subset of traffic (based on protocol 
type) only. What’s more, GC has been activated only twice in 
history (the last one in 2015 [55]). 
---


AT security says otherwise:

https://cybersecurity.att.com/blogs/labs-research/the-great-cannon-has-been-deployed-again

The Great Cannon is a distributed denial of service tool (“DDoS”) 
that operates by injecting malicious Javascript into pages served 
from behind the Great Firewall."

"The Great Cannon was the subject of intense research after it was 
used to disrupt access to the website Github.com in 2015. Little 
has been seen of the Great Cannon since 2015. However, we’ve 
recently observed new attacks..."

"On August 31, 2019, the Great Cannon initiated an attack 
against a website (lihkg.com) used by members of the Hong 
Kong democracy movement to plan protests."

scott

Take-Two Interactive Software NOC Contact

[Tim Nowaczyk]

Network Engineer for a small ISP here. Our customers seem to be having 
connectivity issues with Take-Two Software, specifically NBA 2k20. Traceroute 
makes it to Akamai Prolexic before being dropped. Does anyone have contact info 
for someone at Take-Two?

Thanks,
Tim Nowaczyk

--  
Timothy Nowaczyk  |  Senior Network Manager
office  703.554.6622   |  mobile  571.318.9434 
  

 

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

In fact, the three large carriers provide 98.5% of China’s total
transnational bandwidth. We observe this across all the three large
carriers, as well as one smaller carrier, CERNET(China Education and
Research Network).

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 12:12 PM Ben Cannon  wrote:

>
> On Mar 2, 2020, at 11:38 AM, Pengxiong Zhu  wrote:
>
> Those are good insights. Our first guess is censorship too, and we
> discussed the possibilities of censorship side effects in Section 5.1
> *Censorship*.
>
> It’s the Government doing mandatory content filtering at the border.
>> Their hardware is either deliberately or accidentally poor-performing.
>>
>
> However, GFW operates as an on-path system [72], which only processes
> copies of existing packets without the ability to discard existing packets.
> Evidently, prior work has shown that GFW fails to inject RST packets during
> busy hours while the packets containing sensitive keywords are still
> delivered successfully [34]. However, we are unable to rule out the
> possibility that GFW has evolved to acquire the capability to discard
> packets.
>
> Maybe... I dunno get rid of the Great Firewall of China?
>
>
> We designed a small experiment to locate the hops with GFW presence, and
> then try to match them with the bottleneck hops. We found only in 34.45% of
> the cases, the GFW hops match the bottleneck hops.
>
> My guess is that it’s all the DDoS traffic coming from China saturating
>> the links.
>>
>
> In fact, Great Canon (GC) [55] is such an in-path system. But it is known
> for intercepting a subset of traffic (based on protocol type) only. What’s
> more, GC has been activated only twice in history (the last one in 2015
> [55]). However, it might be the case that the in-path capability is
> re-purposed to perform general traffic throttling. If that is the case,
> they have done a good job because the throttling resembles natural
> congestion from the loss rate and latency point of view.
>
>
> I believe this is what’s happening, and I believe they are rate-limiting
> and causing actual congestion, as opposed to simulating it. The losses
> would be real, actual saturation, on simply rate-limited flows.  Unclear if
> this is being done on a per-flow basis or per-source or what.  You might be
> able to find out.  I’m curious if you see this across all carriers or only
> the larger ones?
>
> -Ben.
>
> The asymmetric performance between downstream and upstream traffic can be
> explained by the natural imbalance of transnational traffic (where the
> upstream traffic from China to outside is not significant enough to
> throttle).
>
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>
>
> On Mon, Mar 2, 2020 at 8:11 AM Compton, Rich A 
> wrote:
>
>> My guess is that it’s all the DDoS traffic coming from China saturating
>> the links.
>>
>>
>>
>> *From: *NANOG Email List  on behalf of
>> Pengxiong Zhu 
>> *Date: *Monday, March 2, 2020 at 8:58 AM
>> *To: *NANOG list 
>> *Cc: *Zhiyun Qian 
>> *Subject: *China’s Slow Transnational Network
>>
>>
>>
>> Hi all,
>>
>>
>>
>> We are a group of researchers at University of California, Riverside who
>> have been working on measuring the transnational network performance (and
>> have previously asked questions on the mailing list). Our work has now led
>> to a publication in Sigmetrics 2020 and we are eager to share some
>>
>> interesting findings.
>>
>>
>>
>> We find China's transnational networks have extremely poor performance
>> when accessing foreign sites, where the throughput is often persistently
>>
>> low (e.g., for the majority of the daytime). Compared to other countries
>> we measured including both developed and developing, China's transnational
>> network performance is among the worst (comparable and even worse than some
>> African countries).
>>
>>
>>
>> Measuring from more than 400 pairs of mainland China and foreign nodes
>> over more than 53 days, our result shows when data transferring from
>> foreign nodes to China, 79% of measured connections has throughput lower
>> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
>> during certain times and forms a diurnal pattern that resembles congestion
>> (irrespective of network protocol and content), please see the following
>> figure. The diurnal pattern is fairly stable, 80% to 95% of the
>> transnational connections have a less than 3 hours standard deviation of
>> the slowdown hours each day over the entire duration. However, the speed
>> rises up from 1Mbps to 4Mbps in about half an hour.
>>
>>
>>
>> [image: blob:null/71cf5a6a-3841-41ce-a1d4-207b59182189]
>>
>>
>>
>> We are able to confirm that high packet loss rates and delays are
>> incurred in the foreign-to-China direction only. Moreover, the end-to-end
>> loss rate could rise up to 40% during the slow period, with ~15% on 

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

Yes, we agree. The poor transnational Internet performance effectively puts
any foreign business that does not have a physical presence (i.e., servers)
in China at a disadvantage.
The challenge is to find out direct evidence to prove mandatory content
filtering at the border, if the government is actually doing it.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo  wrote:

> It also gives local competitors a leg up by helping domestic apps perform
> better simply by being hosted domestically (or making foreign players host
> inside China).
>
> On Mar 2, 2020, at 11:27, Ben Cannon  wrote:
>
> 
> It’s the Government doing mandatory content filtering at the border.
> Their hardware is either deliberately or accidentally poor-performing.
>
> I believe providing limited and throttled external connectivity may be
> deliberate; think of how that curtails for one thing; streaming video?
>
> -Ben.
>
> -Ben Cannon
> CEO 6x7 Networks & 6x7 Telecom, LLC
> b...@6by7.net
>
>
>
> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu  wrote:
>
> Hi all,
>
> We are a group of researchers at University of California, Riverside who
> have been working on measuring the transnational network performance (and
> have previously asked questions on the mailing list). Our work has now led
> to a publication in Sigmetrics 2020 and we are eager to share some
> interesting findings.
>
> We find China's transnational networks have extremely poor performance
> when accessing foreign sites, where the throughput is often persistently
> low (e.g., for the majority of the daytime). Compared to other countries
> we measured including both developed and developing, China's transnational
> network performance is among the worst (comparable and even worse than some
> African countries).
>
> Measuring from more than 400 pairs of mainland China and foreign nodes
> over more than 53 days, our result shows when data transferring from
> foreign nodes to China, 79% of measured connections has throughput lower
> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
> during certain times and forms a diurnal pattern that resembles congestion
> (irrespective of network protocol and content), please see the following
> figure. The diurnal pattern is fairly stable, 80% to 95% of the
> transnational connections have a less than 3 hours standard deviation of
> the slowdown hours each day over the entire duration. However, the speed
> rises up from 1Mbps to 4Mbps in about half an hour.
>
>
> We are able to confirm that high packet loss rates and delays are incurred
> in the foreign-to-China direction only. Moreover, the end-to-end loss rate
> could rise up to 40% during the slow period, with ~15% on average.
>
> There are a few things noteworthy regarding the phenomenon. First of all,
> all traffic types are treated equally, HTTP(S), VPN, etc., which means it
> is discriminating or differentiating any specific kinds of traffic. Second,
> we found for 71% of connections, the bottleneck is located inside China
> (the second hop after entering China or further), which means that it is
> mostly unrelated to the transnational link itself (e.g., submarine cable).
> Yet we never observed any such domestic traffic slowdowns within China.
> Assuming this is due to congestion, it is unclear why the infrastructures
> within China that handles transnational traffic is not even capable to
> handle the capacity of transnational links, e.g., submarine cable, which
> maybe the most expensive investment themselves.
>
> Here is the link to our paper:
> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>
> We appreciate any comments or feedback.
> --
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>
>
>

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

Those are good insights. Our first guess is censorship too, and we
discussed the possibilities of censorship side effects in Section 5.1
*Censorship*.

My guess is that it’s all the DDoS traffic coming from China saturating the
> links.
>

In fact, Great Canon (GC) [55] is such an in-path system. But it is known
for intercepting a subset of traffic (based on protocol type) only. What’s
more, GC has been activated only twice in history (the last one in 2015
[55]). However, it might be the case that the in-path capability is
re-purposed to perform general traffic throttling. If that is the case,
they have done a good job because the throttling resembles natural
congestion from the loss rate and latency point of view. The asymmetric
performance between downstream and upstream traffic can be explained by the
natural imbalance of transnational traffic (where the upstream traffic from
China to outside is not significant enough to throttle).

Maybe... I dunno get rid of the Great Firewall of China?
>

What do you mean? Do you mean the slow traffic is to bypass the GFW or the
slow traffic is caused by GFW?

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 11:38 AM Pengxiong Zhu  wrote:

> Those are good insights. Our first guess is censorship too, and we
> discussed the possibilities of censorship side effects in Section 5.1
> *Censorship*.
>
> It’s the Government doing mandatory content filtering at the border.
>> Their hardware is either deliberately or accidentally poor-performing.
>>
>
> However, GFW operates as an on-path system [72], which only processes
> copies of existing packets without the ability to discard existing packets.
> Evidently, prior work has shown that GFW fails to inject RST packets during
> busy hours while the packets containing sensitive keywords are still
> delivered successfully [34]. However, we are unable to rule out the
> possibility that GFW has evolved to acquire the capability to discard
> packets.
>
> Maybe... I dunno get rid of the Great Firewall of China?
>
>
> We designed a small experiment to locate the hops with GFW presence, and
> then try to match them with the bottleneck hops. We found only in 34.45% of
> the cases, the GFW hops match the bottleneck hops.
>
> My guess is that it’s all the DDoS traffic coming from China saturating
>> the links.
>>
>
> In fact, Great Canon (GC) [55] is such an in-path system. But it is known
> for intercepting a subset of traffic (based on protocol type) only. What’s
> more, GC has been activated only twice in history (the last one in 2015
> [55]). However, it might be the case that the in-path capability is
> re-purposed to perform general traffic throttling. If that is the case,
> they have done a good job because the throttling resembles natural
> congestion from the loss rate and latency point of view. The asymmetric
> performance between downstream and upstream traffic can be explained by the
> natural imbalance of transnational traffic (where the upstream traffic from
> China to outside is not significant enough to throttle).
>
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>
>
> On Mon, Mar 2, 2020 at 8:11 AM Compton, Rich A 
> wrote:
>
>> My guess is that it’s all the DDoS traffic coming from China saturating
>> the links.
>>
>>
>>
>> *From: *NANOG Email List  on behalf of
>> Pengxiong Zhu 
>> *Date: *Monday, March 2, 2020 at 8:58 AM
>> *To: *NANOG list 
>> *Cc: *Zhiyun Qian 
>> *Subject: *China’s Slow Transnational Network
>>
>>
>>
>> Hi all,
>>
>>
>>
>> We are a group of researchers at University of California, Riverside who
>> have been working on measuring the transnational network performance (and
>> have previously asked questions on the mailing list). Our work has now led
>> to a publication in Sigmetrics 2020 and we are eager to share some
>>
>> interesting findings.
>>
>>
>>
>> We find China's transnational networks have extremely poor performance
>> when accessing foreign sites, where the throughput is often persistently
>>
>> low (e.g., for the majority of the daytime). Compared to other countries
>> we measured including both developed and developing, China's transnational
>> network performance is among the worst (comparable and even worse than some
>> African countries).
>>
>>
>>
>> Measuring from more than 400 pairs of mainland China and foreign nodes
>> over more than 53 days, our result shows when data transferring from
>> foreign nodes to China, 79% of measured connections has throughput lower
>> than the 1Mbps, sometimes it is even much lower. The slow speed occurs only
>> during certain times and forms a diurnal pattern that resembles congestion
>> (irrespective of network protocol and content), please see the following
>> figure. The diurnal pattern is fairly stable, 80% to 95% of the
>> transnational connections have a less than 3 hours standard deviation of

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

Yes, the sentence is missing a ‘not’. Sorry about that. It’s not
discriminating or differentiating any specific kinds of traffic.

On Mon, Mar 2, 2020 at 10:56 AM Valdis Klētnieks 
wrote:

> On Sun, 01 Mar 2020 21:00:05 -0800, Pengxiong Zhu said:
>
> > There are a few things noteworthy regarding the phenomenon. First of all,
> > all traffic types are treated equally, HTTP(S), VPN, etc., which means it
> > is discriminating or differentiating any specific kinds of traffic.
>
> This sentence is missing a 'not'.  However, I can't tell if it's "not
> treated equally"
> or "not discriminating"
>
-- 

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: QUIC traffic throttled on AT residential

[Daniel Sterling]

No voice service on my line, or TV. Just gigabit internet.

Also: I think ipv6 isn't working for me cuz it's being dropped by a switch
I'm using!

I will swap that out / remove that and try ipv6 again

-- Dan

On Thu, Feb 27, 2020, 9:10 AM Hiers, David  wrote:

> We find that they usually impose pretty harsh QOS on a link that has an
> ATT voice service.
>
> David
>
>
>
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jay Hennigan
> Sent: Thursday, February 20, 2020 12:13 AM
> To: nanog@nanog.org
> Subject: Re: QUIC traffic throttled on AT residential
>
> On 2/18/20 18:40, nanog-l...@contactdaniel.net wrote:
>
> > Growing prevalence of IPv6-only
> > sites is probably the only thing that will get a lot of access
> > networks to support v6.
>
> I recall a similar idea called "The Great IPv6 Experiment" back in 2007.
> ;-)
>
>
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
>
> --
> This message and any attachments are intended only for the use of the
> addressee and may contain information that is privileged and confidential.
> If the reader of the message is not the intended recipient or an authorized
> representative of the intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If you have
> received this communication in error, notify the sender immediately by
> return email and delete the message and any attachments from your system.
>

Re: Google peering in LAX

[Seth Mattinen]

On 3/2/20 12:44 PM, Owen DeLong wrote:

In part, it might be because people you’re not paying may be less tolerant of 
anti-social behavior than people you are paying.



I'm not sure how I was being offensive but OK.

Re: China’s Slow Transnational Network

[Matt Corallo]

> find out direct evidence of mandatory content filtering at the border

You seem to be implying that you don't believe/can't see the GFW, which
seems surprising. I've personally had issues with traffic crossing it
getting RST'd (luckily I was fortunate enough to cross through a GFW
instance which was easy to avoid with a simple iptables DROP), but its
also one of the most well-studied bits of opaque internet censorship
gear in the world. I'm not sure how you could possibly miss it.

Matt

On 3/2/20 2:55 PM, Pengxiong Zhu wrote:
> Yes, we agree. The poor transnational Internet performance effectively
> puts any foreign business that does not have a physical presence (i.e.,
> servers) in China at a disadvantage.
> The challenge is to find out direct evidence to prove mandatory content
> filtering at the border, if the government is actually doing it.
> 
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
> 
> 
> On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo  > wrote:
> 
> It also gives local competitors a leg up by helping domestic apps
> perform better simply by being hosted domestically (or making
> foreign players host inside China).
> 
>> On Mar 2, 2020, at 11:27, Ben Cannon > > wrote:
>>
>> 
>> It’s the Government doing mandatory content filtering at the
>> border.  Their hardware is either deliberately or accidentally
>> poor-performing.
>>
>> I believe providing limited and throttled external connectivity
>> may be deliberate; think of how that curtails for one thing;
>> streaming video? 
>>
>> -Ben.
>>
>> -Ben Cannon
>> CEO 6x7 Networks & 6x7 Telecom, LLC 
>> b...@6by7.net 
>>
>>
>>
>>> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu >> > wrote:
>>>
>>> Hi all,
>>>
>>> We are a group of researchers at University of California,
>>> Riverside who have been working on measuring the transnational
>>> network performance (and have previously asked questions on the
>>> mailing list). Our work has now led to a publication in
>>> Sigmetrics 2020 and we are eager to share some
>>> interesting findings. 
>>>
>>> We find China's transnational networks have extremely poor
>>> performance when accessing foreign sites, where the throughput is
>>> often persistently
>>> low (e.g., for the majority of the daytime). Compared to other
>>> countries we measured including both developed and developing,
>>> China's transnational network performance is among the worst
>>> (comparable and even worse than some African countries).
>>>
>>> Measuring from more than 400 pairs of mainland China and foreign
>>> nodes over more than 53 days, our result shows when data
>>> transferring from foreign nodes to China, 79% of measured
>>> connections has throughput lower than the 1Mbps, sometimes it is
>>> even much lower. The slow speed occurs only during certain times
>>> and forms a diurnal pattern that resembles congestion
>>> (irrespective of network protocol and content), please see the
>>> following figure. The diurnal pattern is fairly stable, 80% to
>>> 95% of the transnational connections have a less than 3 hours
>>> standard deviation of the slowdown hours each day over the entire
>>> duration. However, the speed rises up from 1Mbps to 4Mbps in
>>> about half an hour.
>>>
>>>
>>> We are able to confirm that high packet loss rates and delays are
>>> incurred in the foreign-to-China direction only. Moreover, the
>>> end-to-end loss rate could rise up to 40% during the slow period,
>>> with ~15% on average.
>>>
>>> There are a few things noteworthy regarding the phenomenon. First
>>> of all, all traffic types are treated equally, HTTP(S), VPN,
>>> etc., which means it is discriminating or differentiating any
>>> specific kinds of traffic. Second, we found for 71% of
>>> connections, the bottleneck is located inside China (the second
>>> hop after entering China or further), which means that it is
>>> mostly unrelated to the transnational link itself (e.g.,
>>> submarine cable). Yet we never observed any such domestic traffic
>>> slowdowns within China.
>>> Assuming this is due to congestion, it is unclear why the
>>> infrastructures within China that handles transnational traffic
>>> is not even capable to handle the capacity of transnational
>>> links, e.g., submarine cable, which maybe the most expensive
>>> investment themselves.
>>>
>>> Here is the link to our paper:
>>> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>>>
>>> We appreciate any comments or feedback. 
>>> -- 
>>>
>>> Best,
>>> Pengxiong Zhu
>>> Department of Computer Science and Engineering
>>> University of California, 

Re: Google peering in LAX

[Owen DeLong]

In part, it might be because people you’re not paying may be less tolerant of 
anti-social behavior than people you are paying.

It does seem rather odd that Google would prefer to receive their traffic over 
transit, but I’m not going to try and second guess that decision.

Owen


> On Mar 2, 2020, at 12:40 PM, Seth Mattinen  wrote:
> 
> Anyone know why Google announces only aggregates via peering and disaggregate 
> prefixes over transit?
> 
> For example, I had a customer complaining about a path that was taking the 
> long way instead of via peering and when I looked I saw:
> 
> Only 172.217.0.0/16 over Any2 LAX
> 
> That plus 172.217.14.0/24 over transit
> 
> Any inquiries to Google just get a generic "we're not setting up any new 
> peering but we're on route servers" response for almost a year now. Or is it 
> because they don't send the /24's to route servers and I'm stuck until they 
> finish their forever improvement project to turn up a direct neighbor?

Google peering in LAX

[Seth Mattinen]

Anyone know why Google announces only aggregates via peering and 
disaggregate prefixes over transit?


For example, I had a customer complaining about a path that was taking 
the long way instead of via peering and when I looked I saw:


Only 172.217.0.0/16 over Any2 LAX

That plus 172.217.14.0/24 over transit

Any inquiries to Google just get a generic "we're not setting up any new 
peering but we're on route servers" response for almost a year now. Or 
is it because they don't send the /24's to route servers and I'm stuck 
until they finish their forever improvement project to turn up a direct 
neighbor?

Re: China’s Slow Transnational Network

[Tom Beecher]

Poor network performance between the Chinese networks and the rest of the
world is not a bug ; it's an intentional feature. The government of China
has constructed these multiple systems to both control what information is
or is not received by their citizens, but also to ensure that domestic
internet companies and services face little to no competition from the
outside world.

As we've unfortunately seen domestically as well, it's a lot easier to
convince people to use YOUR service if performance to the other services
kinda sucks. This is the exact same thing, just at a national scale.


On Mon, Mar 2, 2020 at 3:06 PM Jeff Shultz  wrote:

>
> On Mon, Mar 2, 2020 at 11:46 AM Pengxiong Zhu  wrote:
>
>> Those are good insights. Our first guess is censorship too, and we
>> discussed the possibilities of censorship side effects in Section 5.1
>> *Censorship*.
>>
>> My guess is that it’s all the DDoS traffic coming from China saturating
>>> the links.
>>>
>>
>> In fact, Great Canon (GC) [55] is such an in-path system. But it is known
>> for intercepting a subset of traffic (based on protocol type) only. What’s
>> more, GC has been activated only twice in history (the last one in 2015
>> [55]). However, it might be the case that the in-path capability is
>> re-purposed to perform general traffic throttling. If that is the case,
>> they have done a good job because the throttling resembles natural
>> congestion from the loss rate and latency point of view. The asymmetric
>> performance between downstream and upstream traffic can be explained by the
>> natural imbalance of transnational traffic (where the upstream traffic from
>> China to outside is not significant enough to throttle).
>>
>> Maybe... I dunno get rid of the Great Firewall of China?
>>>
>>
>> What do you mean? Do you mean the slow traffic is to bypass the GFW or
>> the slow traffic is caused by GFW?
>>
>>
> You've pretty much determined there is nothing we can do on this side of
> the Chinese mainland to improve throughput - the bottlenecks are all inside
> China.
>
> As you noted, ~35% of the bottlenecks were GFW related.  I wonder how many
> retransmissions that results in, slowing everything down that much further?
> Until the mainland Chinese Government allows the free passage of
> information, there will be bottlenecks. And bottlenecks have a habit of
> affecting traffic flows outside of their own area.
>
> I doubt that any one thing is the source of the entire problem. But add
> them all together
>
> --
> Jeff Shultz
>
>
> Like us on Social Media for News, Promotions, and other information!!
>
>
> 
> 
> 
>
>
>
>
>
>
>
>  This message contains confidential information and is intended only
> for the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. E-mail transmission cannot be
> guaranteed to be secure or error-free as information could be intercepted,
> corrupted, lost, destroyed, arrive late or incomplete, or contain viruses.
> The sender therefore does not accept liability for any errors or omissions
> in the contents of this message, which arise as a result of e-mail
> transmission. 
>

Re: QUIC traffic throttled on AT residential

[Tom Hill]

On 21/02/2020 23:37, Owen DeLong wrote:
> What’s next? Why not simply eliminate port numbers altogether in favor
> of a single 16-bit client-side unique session identifier.


I see what you did there.

-- 
Tom

Re: China’s Slow Transnational Network

[Jeff Shultz]

On Mon, Mar 2, 2020 at 11:46 AM Pengxiong Zhu  wrote:

> Those are good insights. Our first guess is censorship too, and we
> discussed the possibilities of censorship side effects in Section 5.1
> *Censorship*.
>
> My guess is that it’s all the DDoS traffic coming from China saturating
>> the links.
>>
>
> In fact, Great Canon (GC) [55] is such an in-path system. But it is known
> for intercepting a subset of traffic (based on protocol type) only. What’s
> more, GC has been activated only twice in history (the last one in 2015
> [55]). However, it might be the case that the in-path capability is
> re-purposed to perform general traffic throttling. If that is the case,
> they have done a good job because the throttling resembles natural
> congestion from the loss rate and latency point of view. The asymmetric
> performance between downstream and upstream traffic can be explained by the
> natural imbalance of transnational traffic (where the upstream traffic from
> China to outside is not significant enough to throttle).
>
> Maybe... I dunno get rid of the Great Firewall of China?
>>
>
> What do you mean? Do you mean the slow traffic is to bypass the GFW or the
> slow traffic is caused by GFW?
>
>
You've pretty much determined there is nothing we can do on this side of
the Chinese mainland to improve throughput - the bottlenecks are all inside
China.

As you noted, ~35% of the bottlenecks were GFW related.  I wonder how many
retransmissions that results in, slowing everything down that much further?
Until the mainland Chinese Government allows the free passage of
information, there will be bottlenecks. And bottlenecks have a habit of
affecting traffic flows outside of their own area.

I doubt that any one thing is the source of the entire problem. But add
them all together

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: China’s Slow Transnational Network

[Valdis Klētnieks]

On Sun, 01 Mar 2020 21:00:05 -0800, Pengxiong Zhu said:

> There are a few things noteworthy regarding the phenomenon. First of all,
> all traffic types are treated equally, HTTP(S), VPN, etc., which means it
> is discriminating or differentiating any specific kinds of traffic.

This sentence is missing a 'not'.  However, I can't tell if it's "not treated 
equally"
or "not discriminating"


pgpfNu52qo1O3.pgp
Description: PGP signature

Re: China’s Slow Transnational Network

[Matt Corallo]

It also gives local competitors a leg up by helping domestic apps perform 
better simply by being hosted domestically (or making foreign players host 
inside China).

> On Mar 2, 2020, at 11:27, Ben Cannon  wrote:
> 
> 
> It’s the Government doing mandatory content filtering at the border.  Their 
> hardware is either deliberately or accidentally poor-performing.
> 
> I believe providing limited and throttled external connectivity may be 
> deliberate; think of how that curtails for one thing; streaming video? 
> 
> -Ben.
> 
> -Ben Cannon
> CEO 6x7 Networks & 6x7 Telecom, LLC 
> b...@6by7.net
> 
> 
> 
> 
>> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu  wrote:
>> 
>> Hi all,
>> 
>> We are a group of researchers at University of California, Riverside who 
>> have been working on measuring the transnational network performance (and 
>> have previously asked questions on the mailing list). Our work has now led 
>> to a publication in Sigmetrics 2020 and we are eager to share some
>> interesting findings. 
>> 
>> We find China's transnational networks have extremely poor performance when 
>> accessing foreign sites, where the throughput is often persistently
>> low (e.g., for the majority of the daytime). Compared to other countries we 
>> measured including both developed and developing, China's transnational 
>> network performance is among the worst (comparable and even worse than some 
>> African countries).
>> 
>> Measuring from more than 400 pairs of mainland China and foreign nodes over 
>> more than 53 days, our result shows when data transferring from foreign 
>> nodes to China, 79% of measured connections has throughput lower than the 
>> 1Mbps, sometimes it is even much lower. The slow speed occurs only during 
>> certain times and forms a diurnal pattern that resembles congestion 
>> (irrespective of network protocol and content), please see the following 
>> figure. The diurnal pattern is fairly stable, 80% to 95% of the 
>> transnational connections have a less than 3 hours standard deviation of the 
>> slowdown hours each day over the entire duration. However, the speed rises 
>> up from 1Mbps to 4Mbps in about half an hour.
>> 
>> 
>> 
>> We are able to confirm that high packet loss rates and delays are incurred 
>> in the foreign-to-China direction only. Moreover, the end-to-end loss rate 
>> could rise up to 40% during the slow period, with ~15% on average.
>> 
>> There are a few things noteworthy regarding the phenomenon. First of all, 
>> all traffic types are treated equally, HTTP(S), VPN, etc., which means it is 
>> discriminating or differentiating any specific kinds of traffic. Second, we 
>> found for 71% of connections, the bottleneck is located inside China (the 
>> second hop after entering China or further), which means that it is mostly 
>> unrelated to the transnational link itself (e.g., submarine cable). Yet we 
>> never observed any such domestic traffic slowdowns within China.
>> Assuming this is due to congestion, it is unclear why the infrastructures 
>> within China that handles transnational traffic is not even capable to 
>> handle the capacity of transnational links, e.g., submarine cable, which 
>> maybe the most expensive investment themselves.
>> 
>> Here is the link to our paper:
>> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf
>> 
>> We appreciate any comments or feedback. 
>> -- 
>> 
>> Best,
>> Pengxiong Zhu
>> Department of Computer Science and Engineering
>> University of California, Riverside
> 

Re: China’s Slow Transnational Network

[Ben Cannon]

It’s the Government doing mandatory content filtering at the border.  Their 
hardware is either deliberately or accidentally poor-performing.

I believe providing limited and throttled external connectivity may be 
deliberate; think of how that curtails for one thing; streaming video? 

-Ben.

-Ben Cannon
CEO 6x7 Networks & 6x7 Telecom, LLC 
b...@6by7.net 




> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu  wrote:
> 
> Hi all,
> 
> We are a group of researchers at University of California, Riverside who have 
> been working on measuring the transnational network performance (and have 
> previously asked questions on the mailing list). Our work has now led to a 
> publication in Sigmetrics 2020 and we are eager to share some
> interesting findings. 
> 
> We find China's transnational networks have extremely poor performance when 
> accessing foreign sites, where the throughput is often persistently
> low (e.g., for the majority of the daytime). Compared to other countries we 
> measured including both developed and developing, China's transnational 
> network performance is among the worst (comparable and even worse than some 
> African countries).
> 
> Measuring from more than 400 pairs of mainland China and foreign nodes over 
> more than 53 days, our result shows when data transferring from foreign nodes 
> to China, 79% of measured connections has throughput lower than the 1Mbps, 
> sometimes it is even much lower. The slow speed occurs only during certain 
> times and forms a diurnal pattern that resembles congestion (irrespective of 
> network protocol and content), please see the following figure. The diurnal 
> pattern is fairly stable, 80% to 95% of the transnational connections have a 
> less than 3 hours standard deviation of the slowdown hours each day over the 
> entire duration. However, the speed rises up from 1Mbps to 4Mbps in about 
> half an hour.
> 
> 
> 
> We are able to confirm that high packet loss rates and delays are incurred in 
> the foreign-to-China direction only. Moreover, the end-to-end loss rate could 
> rise up to 40% during the slow period, with ~15% on average.
> 
> There are a few things noteworthy regarding the phenomenon. First of all, all 
> traffic types are treated equally, HTTP(S), VPN, etc., which means it is 
> discriminating or differentiating any specific kinds of traffic. Second, we 
> found for 71% of connections, the bottleneck is located inside China (the 
> second hop after entering China or further), which means that it is mostly 
> unrelated to the transnational link itself (e.g., submarine cable). Yet we 
> never observed any such domestic traffic slowdowns within China.
> Assuming this is due to congestion, it is unclear why the infrastructures 
> within China that handles transnational traffic is not even capable to handle 
> the capacity of transnational links, e.g., submarine cable, which maybe the 
> most expensive investment themselves.
> 
> Here is the link to our paper:
> https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf 
> 
> 
> We appreciate any comments or feedback. 
> -- 
> 
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside

Re: China’s Slow Transnational Network

[Compton, Rich A]

My guess is that it’s all the DDoS traffic coming from China saturating the 
links.

From: NANOG Email List  on behalf of Pengxiong Zhu 

Date: Monday, March 2, 2020 at 8:58 AM
To: NANOG list 
Cc: Zhiyun Qian 
Subject: China’s Slow Transnational Network

Hi all,

We are a group of researchers at University of California, Riverside who have 
been working on measuring the transnational network performance (and have 
previously asked questions on the mailing list). Our work has now led to a 
publication in Sigmetrics 2020 and we are eager to share some
interesting findings.

We find China's transnational networks have extremely poor performance when 
accessing foreign sites, where the throughput is often persistently
low (e.g., for the majority of the daytime). Compared to other countries we 
measured including both developed and developing, China's transnational network 
performance is among the worst (comparable and even worse than some African 
countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over 
more than 53 days, our result shows when data transferring from foreign nodes 
to China, 79% of measured connections has throughput lower than the 1Mbps, 
sometimes it is even much lower. The slow speed occurs only during certain 
times and forms a diurnal pattern that resembles congestion (irrespective of 
network protocol and content), please see the following figure. The diurnal 
pattern is fairly stable, 80% to 95% of the transnational connections have a 
less than 3 hours standard deviation of the slowdown hours each day over the 
entire duration. However, the speed rises up from 1Mbps to 4Mbps in about half 
an hour.

[blob:null/71cf5a6a-3841-41ce-a1d4-207b59182189]

We are able to confirm that high packet loss rates and delays are incurred in 
the foreign-to-China direction only. Moreover, the end-to-end loss rate could 
rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all, all 
traffic types are treated equally, HTTP(S), VPN, etc., which means it is 
discriminating or differentiating any specific kinds of traffic. Second, we 
found for 71% of connections, the bottleneck is located inside China (the 
second hop after entering China or further), which means that it is mostly 
unrelated to the transnational link itself (e.g., submarine cable). Yet we 
never observed any such domestic traffic slowdowns within China.
Assuming this is due to congestion, it is unclear why the infrastructures 
within China that handles transnational traffic is not even capable to handle 
the capacity of transnational links, e.g., submarine cable, which maybe the 
most expensive investment themselves.

Here is the link to our paper:
https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf


We appreciate any comments or feedback.
--

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside
E-MAIL CONFIDENTIALITY NOTICE: 
The contents of this e-mail message and any attachments are intended solely for 
the addressee(s) and may contain confidential and/or legally privileged 
information. If you are not the intended recipient of this message or if this 
message has been addressed to you in error, please immediately alert the sender 
by reply e-mail and then delete this message and any attachments. If you are 
not the intended recipient, you are notified that any use, dissemination, 
distribution, copying, or storage of this message or any attachment is strictly 
prohibited.

Re: China’s Slow Transnational Network

[Jeff Shultz]

Maybe... I dunno get rid of the Great Firewall of China?
https://en.wikipedia.org/wiki/Great_Firewall

On Mon, Mar 2, 2020 at 7:59 AM Pengxiong Zhu  wrote:

> Hi all,
>
> We are a group of researchers at University of California, Riverside who
> have been working on measuring the transnational network performance (and
> have previously asked questions on the mailing list). Our work has now led
> to a publication in Sigmetrics 2020 and we are eager to share some
> interesting findings.
>
> We find China's transnational networks have extremely poor performance
> when accessing foreign sites, where the throughput is often persistently
> low (e.g., for the majority of the daytime). Compared to other countries
> we measured including both developed and developing, China's transnational
> network performance is among the worst (comparable and even worse than some
> African countries).
>
>
> --
>
> Best,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>


-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

China’s Slow Transnational Network

[Pengxiong Zhu]

Hi all,

We are a group of researchers at University of California, Riverside who
have been working on measuring the transnational network performance (and
have previously asked questions on the mailing list). Our work has now led
to a publication in Sigmetrics 2020 and we are eager to share some
interesting findings.

We find China's transnational networks have extremely poor performance when
accessing foreign sites, where the throughput is often persistently
low (e.g., for the majority of the daytime). Compared to other countries we
measured including both developed and developing, China's transnational
network performance is among the worst (comparable and even worse than some
African countries).

Measuring from more than 400 pairs of mainland China and foreign nodes over
more than 53 days, our result shows when data transferring from foreign
nodes to China, 79% of measured connections has throughput lower than the
1Mbps, sometimes it is even much lower. The slow speed occurs only during
certain times and forms a diurnal pattern that resembles congestion
(irrespective of network protocol and content), please see the following
figure. The diurnal pattern is fairly stable, 80% to 95% of the
transnational connections have a less than 3 hours standard deviation of
the slowdown hours each day over the entire duration. However, the speed
rises up from 1Mbps to 4Mbps in about half an hour.


We are able to confirm that high packet loss rates and delays are incurred
in the foreign-to-China direction only. Moreover, the end-to-end loss rate
could rise up to 40% during the slow period, with ~15% on average.

There are a few things noteworthy regarding the phenomenon. First of all,
all traffic types are treated equally, HTTP(S), VPN, etc., which means it
is discriminating or differentiating any specific kinds of traffic. Second,
we found for 71% of connections, the bottleneck is located inside China
(the second hop after entering China or further), which means that it is
mostly unrelated to the transnational link itself (e.g., submarine cable).
Yet we never observed any such domestic traffic slowdowns within China.
Assuming this is due to congestion, it is unclear why the infrastructures
within China that handles transnational traffic is not even capable to
handle the capacity of transnational links, e.g., submarine cable, which
maybe the most expensive investment themselves.

Here is the link to our paper:
https://www.cs.ucr.edu/~zhiyunq/pub/sigmetrics20_slowdown.pdf

We appreciate any comments or feedback.
-- 

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside