Re: [outages] Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Andras Toth
Seems like the destination is in Hetzner, they could also raise it with
Twelve99 or prepend routes to use an alternate path.


On Fri, Jan 26, 2024 at 7:46 PM Phil Lavin via Outages 
wrote:

> Thanks, ytti. I have raised a case with AWS but I expect it to be as
> unproductive as usual. In this type of issue, there is often a friendly
> Engineer keeping an eye on NANOG list or IRC channel who can mitigate it.
> Hoping that to be the case this time, also.
>
>
> > On 26 Jan 2024, at 08:40, Saku Ytti  wrote:
> >
> > On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG 
> wrote:
> >
> >
> >> 88.99.88.67 to 216.147.3.209:
> >> Host   Loss%   Snt   Last
>  Avg  Best  Wrst StDev
> >> 1. 10.88.10.254 0.0%   1760.2
>  0.1   0.1   0.3   0.1
> >> 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3
>  3.5   3.1  24.1   1.6
> >> 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9
> 18.9  18.7  19.2   0.1
> >> 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5
> 30.6  30.4  30.8   0.1
> >> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175
>  99.5  99.5  99.3 100.1   0.2
> >> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8
> 306.0 289.7 315.0   5.5
> >> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0
> 513.5 500.7 539.7   8.4
> >
> > This implies the problem is not on this path, because #10 is not
> > experiencing it, possibly because it happens to return a packet via
> > another option, but certainly shows the problem didn't happen in this
> > direction yet at #10, but because #8 and #9 saw it, they already saw
> > it on the other direction.
> >
> >
> >> 44.236.47.236 to 178.63.26.145:
> >> Host Loss%   Snt   Last
>  Avg  Best  Wrst StDev
> >> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2
>  0.2   0.2   0.4   0.0
> >> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8
>  5.9   5.6  11.8   0.5
> >> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1
> 21.5  21.0  58.4   3.1
> >> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4
> 22.7  21.3  86.2   6.5
> >> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7
> 422.7 407.2 438.5   6.5
> >> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6
> 485.4 481.6 491.1   3.9
> >> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1
> 499.8 489.8 510.1   5.9
> >> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5
> 519.6 504.4 561.7   7.6
> >> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4
> 519.2 506.0 545.5   6.9
> >> 19. core22.fsn1.hetzner.com   70.2%   266  521.7
> 519.2 498.5 531.7   6.6
> >> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4
> 375.4 364.9 396.5   4.1
> >> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9
> 518.4 506.9 531.3   6.1
> >
> > This suggests the congestion point is from sjo to nyk, in 1299, not AWS
> at all.
> >
> > You could try to fix SPORT/DPORT, and do several SPORT options, to see
> > if loss goes away with some, to determine if all LAG members are full
> > or just one.
> >
> >
> > At any rate, this seems business as usual, sometimes internet is very
> > lossy, you should contact your service provider, which I guess is AWS
> > here, so they can contact their service provider or 1299.
> >
> > --
> >  ++ytti
>
> ___
> Outages mailing list
> outa...@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


Re: Sling TV Geolocation

2024-01-26 Thread Tim Burke
Thanks for reminding me that I forgot to bump this.

I did use that contact form and got a response relatively quickly. It doesn't 
seem like they know what a geofeed is, but they were able to add my requested 
change to their data to take place in a couple of weeks.

Tim

From: Justin Krejci 
Sent: Friday, January 26, 2024 2:42 PM
To: nanog@nanog.org ; Tim Burke 
Subject: Re: Sling TV Geolocation

I have Digital Element in my own internal wiki page for managing/documenting IP 
geolocation services headaches.

Searching them up on my page I see noted they have a contact us form that 
specifically lists "IP Address Data Update" as a contact reason. Maybe that 
will give you or others some avenue into the proper eyeballs over there.
https://www.digitalelement.com/contact-us/

I appreciate the follow up and will add a note to my page that Sling TV uses 
Digital Element, at least at the moment.

As always, good luck on your endeavor.



-Original Message-
From: Tim Burke mailto:tim%20burke%20%3c...@mid.net%3e>>
To: nanog@nanog.org 
mailto:%22na...@nanog.org%22%20%3cna...@nanog.org%3e>>
Subject: Re: Sling TV Geolocation
Date: Wed, 24 Jan 2024 20:32:10 +

(long overdue) Follow up on this – after plenty of emails, phone calls, and 
research, and our poor customers having to watch the Packer game, I was able to 
find out that Sling is using Digital Envoy/DigitalElement for geolocation... I 
assume the info on 
https://thebrotherswisp.com/index.php/geo-and-vpn/
 should work for this, but I am waiting for hear back from said geolocation 
vendor with an answer.

Thanks,
Tim



From: Tim Burke
Sent: Thursday, December 7, 2023 11:36 AM
To: nanog@nanog.org 
Subject: Sling TV Geolocation

Yet another geolocation post, because content networks don't pay attention to 
geofeeds... :-)

Anyone know who Sling TV is using for geolocation, or have a contact at Sling 
that can help? We acquired a /19 in July that we just started pushing out to 
customers, it is still geolocating back to the previous owner on Sling TV, 
despite publishing the prefix in our geofeed. Checked the usual lists with no 
luck.

Thanks,
Tim



Re: Sling TV Geolocation

2024-01-26 Thread Justin Krejci
I have Digital Element in my own internal wiki page for managing/documenting IP 
geolocation services headaches.

Searching them up on my page I see noted they have a contact us form that 
specifically lists "IP Address Data Update" as a contact reason. Maybe that 
will give you or others some avenue into the proper eyeballs over there.
https://www.digitalelement.com/contact-us/

I appreciate the follow up and will add a note to my page that Sling TV uses 
Digital Element, at least at the moment.

As always, good luck on your endeavor.



-Original Message-
From: Tim Burke mailto:tim%20burke%20%3c...@mid.net%3e>>
To: nanog@nanog.org 
mailto:%22na...@nanog.org%22%20%3cna...@nanog.org%3e>>
Subject: Re: Sling TV Geolocation
Date: Wed, 24 Jan 2024 20:32:10 +

(long overdue) Follow up on this – after plenty of emails, phone calls, and 
research, and our poor customers having to watch the Packer game, I was able to 
find out that Sling is using Digital Envoy/DigitalElement for geolocation... I 
assume the info on 
https://thebrotherswisp.com/index.php/geo-and-vpn/
 should work for this, but I am waiting for hear back from said geolocation 
vendor with an answer.

Thanks,
Tim



From: Tim Burke
Sent: Thursday, December 7, 2023 11:36 AM
To: nanog@nanog.org 
Subject: Sling TV Geolocation

Yet another geolocation post, because content networks don't pay attention to 
geofeeds... :-)

Anyone know who Sling TV is using for geolocation, or have a contact at Sling 
that can help? We acquired a /19 in July that we just started pushing out to 
customers, it is still geolocating back to the previous owner on Sling TV, 
despite publishing the prefix in our geofeed. Checked the usual lists with no 
luck.

Thanks,
Tim



Weekly Global IPv4 Routing Table Report

2024-01-26 Thread Routing Table Analysis Role Account
This is an automated weekly mailing describing the state of the Global
IPv4 Routing Table as seen from APNIC's router in Japan.

The posting is sent to APOPS, NANOG, AfNOG, SANOG, PacNOG, SAFNOG
UKNOF, TZNOG, MENOG, BJNOG, SDNOG, CMNOG, LACNOG and the RIPE Routing WG.

Daily listings are sent to bgp-st...@lists.apnic.net.

For historical data, please see https://thyme.apnic.net.

If you have any comments please contact Philip Smith .

IPv4 Routing Table Report   04:00 +10GMT Sat 27 Jan, 2024

  BGP Table (Global) as seen in Japan.

Report Website: https://thyme.apnic.net
Detailed Analysis:  https://thyme.apnic.net/current/

Analysis Summary


BGP routing table entries examined:  940117
Prefixes after maximum aggregation (per Origin AS):  357735
Deaggregation factor:  2.63
Unique aggregates announced (without unneeded subnets):  456211
Total ASes present in the Internet Routing Table: 75349
Prefixes per ASN: 12.48
Origin-only ASes present in the Internet Routing Table:   64603
Origin ASes announcing only one prefix:   26556
Transit ASes present in the Internet Routing Table:   10746
Transit-only ASes present in the Internet Routing Table:499
Average AS path length visible in the Internet Routing Table:   4.3
Max AS path length visible: 121
Max AS path prepend of ASN (150315) 116
Prefixes from unregistered ASNs in the Routing Table:  1123
Number of instances of unregistered ASNs:  1125
Number of 32-bit ASNs allocated by the RIRs:  43482
Number of 32-bit ASNs visible in the Routing Table:   35776
Prefixes from 32-bit ASNs in the Routing Table:  180692
Number of bogon 32-bit ASNs visible in the Routing Table:34
Special use prefixes present in the Routing Table:1
Prefixes being announced from unallocated address space:499
Number of addresses announced to Internet:   3047946240
Equivalent to 181 /8s, 171 /16s and 248 /24s
Percentage of available address space announced:   82.3
Percentage of allocated address space announced:   82.3
Percentage of available address space allocated:  100.0
Percentage of address space in use by end-sites:   99.6
Total number of prefixes smaller than registry allocations:  310463

APNIC Region Analysis Summary
-

Prefixes being announced by APNIC Region ASes:   251719
Total APNIC prefixes after maximum aggregation:   72784
APNIC Deaggregation factor:3.46
Prefixes being announced from the APNIC address blocks:  244588
Unique aggregates announced from the APNIC address blocks:99925
APNIC Region origin ASes present in the Internet Routing Table:   13942
APNIC Prefixes per ASN:   17.54
APNIC Region origin ASes announcing only one prefix:   4210
APNIC Region transit ASes present in the Internet Routing Table:   1886
Average APNIC Region AS path length visible:4.4
Max APNIC Region AS path length visible:121
Number of APNIC region 32-bit ASNs visible in the Routing Table:   9307
Number of APNIC addresses announced to Internet:  774890112
Equivalent to 46 /8s, 47 /16s and 226 /24s
APNIC AS Blocks4608-4864, 7467-7722, 9216-10239, 17408-18431
(pre-ERX allocations)  23552-24575, 37888-38911, 45056-46079, 55296-56319,
   58368-59391, 63488-64098, 64297-64395, 131072-153913
APNIC Address Blocks 1/8,  14/8,  27/8,  36/8,  39/8,  42/8,  43/8,
49/8,  58/8,  59/8,  60/8,  61/8, 101/8, 103/8,
   106/8, 110/8, 111/8, 112/8, 113/8, 114/8, 115/8,
   116/8, 117/8, 118/8, 119/8, 120/8, 121/8, 122/8,
   123/8, 124/8, 125/8, 126/8, 133/8, 150/8, 153/8,
   163/8, 171/8, 175/8, 180/8, 182/8, 183/8, 202/8,
   203/8, 210/8, 211/8, 218/8, 219/8, 220/8, 221/8,
   222/8, 223/8,

ARIN Region Analysis Summary


Prefixes being announced by ARIN Region ASes:274037
Total ARIN prefixes after maximum aggregation:   123967
ARIN Deaggregation factor: 2.21
Prefixes being announced from the ARIN address blocks:   278953
Unique aggregates announced from the ARIN address blocks:132163
ARIN Region origin ASes present in the Internet Routing Table:19089
ARIN Prefixes per ASN:   

Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Bjoern Franke via NANOG

Hi,



Cross posting to outages and nanog.

Can anybody from Twelve99 / AWS investigate high loss and latency around SJO 
between Twelve99 and AWS?

Traces below:

88.99.88.67 to 216.147.3.209:



this seems not only AWS related - from AS50629 to AS36086 via AS1299 - 
also high latency and loss since sjo:



   9.|-- adm-bb2-link.ip.twelve99. 60.0%10   10.1  10.3  10.1  10.5
  0.2
  10.|-- ldn-bb2-link.ip.twelve99. 90.0%10   15.4  15.4  15.4  15.4
  0.0
  11.|-- nyk-bb1-link.ip.twelve99.  0.0%10   84.2  84.2  83.9  84.8
  0.3
  12.|-- sjo-b23-link.ip.twelve99. 20.0%10  296.5 297.0 294.9 298.3
  1.1
  13.|-- palo-b24-link.ip.twelve99 20.0%10  485.5 487.2 481.9 491.2
  3.2
  14.|-- digitalrealty-ic-360311.i 70.0%10  488.8 485.2 482.4 488.8
  3.3
  15.|-- irb-20-cr1-sfo1-dia.telxg 20.0%10  510.1 508.9 502.5 516.9
  5.0
  16.|-- 208.65.109.19060.0%10  506.1 501.3 498.3 506.1
  3.6
  17.|-- ice5.somafm.com   60.0%10  499.3 501.1 498.8 504.2
  2.5


Regards
Bjoern


Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
I see the AWS->me path has just changed to Cogent. me->AWS path still appears 
to be Twelve99. Loss and latency has now subsided.

Thanks


> On 26 Jan 2024, at 08:19, Phil Lavin  wrote:
> 
> Hi Folks,
> 
> Cross posting to outages and nanog.
> 
> Can anybody from Twelve99 / AWS investigate high loss and latency around SJO 
> between Twelve99 and AWS?
> 
> Traces below:
> 
> 88.99.88.67 to 216.147.3.209:
> 
>My traceroute  [v0.94]
> phil (10.88.10.10) -> 216.147.3.209   
> 2024-01-26T08:16:16+
> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>  Packets   
> Pings
> Host   Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. 10.88.10.254 0.0%   1760.2   0.1   
> 0.1   0.3   0.1
> 2. fe-fw-01.lavin.me.uk 0.0%   1760.4   0.4   
> 0.2   0.7   0.1
> 3. 10.69.69.1   0.0%   1760.5   0.4   
> 0.2   0.7   0.1
> 4. static.225.8.4.46.clients.your-server.de 0.0%   1760.8   1.0   
> 0.5   8.2   1.0
> 5. core22.fsn1.hetzner.com  0.6%   1760.8   2.0   
> 0.7  39.0   4.6
> 6. static.213-239-254-234.clients.your-server.de0.0%   1763.2   3.8   
> 3.1  36.5   3.3
> 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5   
> 3.1  24.1   1.6
> 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
> 18.7  19.2   0.1
> 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
> 30.4  30.8   0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5  
> 99.3 100.1   0.2
> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
> 289.7 315.0   5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
> 500.7 539.7   8.4
> 13. (waiting for reply)
> …...
> 26. 216.147.3.209   59.1%   150  534.2 534.4 
> 513.5 549.7   8.1
> 
> 44.236.47.236 to 178.63.26.145:
> 
> jcasc-rtc01 (10.96.54.240) -> hc-ping.com   
> 2024-01-26T08:10:24+
> Keys:  Help   Display mode   Restart statistics   Order of fields   quit
>Packets   Pings
> Host Loss%   Snt   Last   Avg  
> Best  Wrst StDev
> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
> 0.2   0.4   0.0
> 2. ec2-34-221-151-225.us-west-2.compute.amazonaw  0.0%   2676.8  15.9   
> 0.7 114.1  24.1
> 3. (waiting for reply)
> 4. (waiting for reply)
> 5. (waiting for reply)
> 6. 241.0.2.5  0.0%   2670.4   0.4   
> 0.3   1.2   0.1
> 7. 242.0.42.195   0.0%   2671.2   1.7   
> 0.5  23.9   3.0
> 8. 240.2.144.12   0.0%   2675.2   5.2   
> 5.2  27.4   1.4
> 9. amazon-ic-375418.ip.twelve99-cust.net  0.0%   2675.3   5.2   
> 5.2   6.1   0.1
> 10. port-b4-link.ip.twelve99.net   0.0%   2675.3   5.5   
> 4.9  35.5   3.0
> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
> 5.6  11.8   0.5
> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
> 21.0  58.4   3.1
> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
> 21.3  86.2   6.5
> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
> 407.2 438.5   6.5
> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
> 481.6 491.1   3.9
> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
> 489.8 510.1   5.9
> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
> 504.4 561.7   7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
> 506.0 545.5   6.9
> 19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
> 498.5 531.7   6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
> 364.9 396.5   4.1
> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
> 506.9 531.3   6.1
> 
> 
> Thanks



Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
Thanks, ytti. I have raised a case with AWS but I expect it to be as 
unproductive as usual. In this type of issue, there is often a friendly 
Engineer keeping an eye on NANOG list or IRC channel who can mitigate it. 
Hoping that to be the case this time, also.


> On 26 Jan 2024, at 08:40, Saku Ytti  wrote:
> 
> On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG  wrote:
> 
> 
>> 88.99.88.67 to 216.147.3.209:
>> Host   Loss%   Snt   Last   Avg  
>> Best  Wrst StDev
>> 1. 10.88.10.254 0.0%   1760.2   0.1  
>>  0.1   0.3   0.1
>> 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5  
>>  3.1  24.1   1.6
>> 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
>> 18.7  19.2   0.1
>> 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
>> 30.4  30.8   0.1
>> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5 
>>  99.3 100.1   0.2
>> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
>> 289.7 315.0   5.5
>> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
>> 500.7 539.7   8.4
> 
> This implies the problem is not on this path, because #10 is not
> experiencing it, possibly because it happens to return a packet via
> another option, but certainly shows the problem didn't happen in this
> direction yet at #10, but because #8 and #9 saw it, they already saw
> it on the other direction.
> 
> 
>> 44.236.47.236 to 178.63.26.145:
>> Host Loss%   Snt   Last   Avg  
>> Best  Wrst StDev
>> 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
>> 0.2   0.4   0.0
>> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
>> 5.6  11.8   0.5
>> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
>> 21.0  58.4   3.1
>> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
>> 21.3  86.2   6.5
>> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
>> 407.2 438.5   6.5
>> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
>> 481.6 491.1   3.9
>> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
>> 489.8 510.1   5.9
>> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
>> 504.4 561.7   7.6
>> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
>> 506.0 545.5   6.9
>> 19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
>> 498.5 531.7   6.6
>> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
>> 364.9 396.5   4.1
>> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
>> 506.9 531.3   6.1
> 
> This suggests the congestion point is from sjo to nyk, in 1299, not AWS at 
> all.
> 
> You could try to fix SPORT/DPORT, and do several SPORT options, to see
> if loss goes away with some, to determine if all LAG members are full
> or just one.
> 
> 
> At any rate, this seems business as usual, sometimes internet is very
> lossy, you should contact your service provider, which I guess is AWS
> here, so they can contact their service provider or 1299.
> 
> -- 
>  ++ytti



Re: Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Saku Ytti
On Fri, 26 Jan 2024 at 10:23, Phil Lavin via NANOG  wrote:


> 88.99.88.67 to 216.147.3.209:
>  Host   Loss%   Snt   Last   Avg  
> Best  Wrst StDev
>  1. 10.88.10.254 0.0%   1760.2   0.1  
>  0.1   0.3   0.1
>  7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5  
>  3.1  24.1   1.6
>  8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
> 18.7  19.2   0.1
>  9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
> 30.4  30.8   0.1
> 10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5  
> 99.3 100.1   0.2
> 11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
> 289.7 315.0   5.5
> 12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
> 500.7 539.7   8.4

This implies the problem is not on this path, because #10 is not
experiencing it, possibly because it happens to return a packet via
another option, but certainly shows the problem didn't happen in this
direction yet at #10, but because #8 and #9 saw it, they already saw
it on the other direction.


> 44.236.47.236 to 178.63.26.145:
>  Host Loss%   Snt   Last   Avg  
> Best  Wrst StDev
>  1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
> 0.2   0.4   0.0
> 11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
> 5.6  11.8   0.5
> 12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
> 21.0  58.4   3.1
> 13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
> 21.3  86.2   6.5
> 14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
> 407.2 438.5   6.5
> 15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
> 481.6 491.1   3.9
> 16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
> 489.8 510.1   5.9
> 17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
> 504.4 561.7   7.6
> 18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
> 506.0 545.5   6.9
> 19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
> 498.5 531.7   6.6
> 20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
> 364.9 396.5   4.1
> 21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
> 506.9 531.3   6.1

This suggests the congestion point is from sjo to nyk, in 1299, not AWS at all.

You could try to fix SPORT/DPORT, and do several SPORT options, to see
if loss goes away with some, to determine if all LAG members are full
or just one.


At any rate, this seems business as usual, sometimes internet is very
lossy, you should contact your service provider, which I guess is AWS
here, so they can contact their service provider or 1299.

-- 
  ++ytti


Twelve99 / AWS usw2 significant loss

2024-01-26 Thread Phil Lavin via NANOG
Hi Folks,

Cross posting to outages and nanog.

Can anybody from Twelve99 / AWS investigate high loss and latency around SJO 
between Twelve99 and AWS?

Traces below:

88.99.88.67 to 216.147.3.209:

My traceroute  [v0.94]
phil (10.88.10.10) -> 216.147.3.209   
2024-01-26T08:16:16+
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
  Packets   
Pings
 Host   Loss%   Snt   Last   Avg  
Best  Wrst StDev
 1. 10.88.10.254 0.0%   1760.2   0.1   
0.1   0.3   0.1
 2. fe-fw-01.lavin.me.uk 0.0%   1760.4   0.4   
0.2   0.7   0.1
 3. 10.69.69.1   0.0%   1760.5   0.4   
0.2   0.7   0.1
 4. static.225.8.4.46.clients.your-server.de 0.0%   1760.8   1.0   
0.5   8.2   1.0
 5. core22.fsn1.hetzner.com  0.6%   1760.8   2.0   
0.7  39.0   4.6
 6. static.213-239-254-234.clients.your-server.de0.0%   1763.2   3.8   
3.1  36.5   3.3
 7. nug-b1-link.ip.twelve99.net  0.0%   1763.3   3.5   
3.1  24.1   1.6
 8. hbg-bb2-link.ip.twelve99.net86.9%   175   18.9  18.9  
18.7  19.2   0.1
 9. ldn-bb2-link.ip.twelve99.net92.0%   175   30.5  30.6  
30.4  30.8   0.1
10. nyk-bb1-link.ip.twelve99.net 4.6%   175   99.5  99.5  
99.3 100.1   0.2
11. sjo-b23-link.ip.twelve99.net56.3%   175  296.8 306.0 
289.7 315.0   5.5
12. amazon-ic-366608.ip.twelve99-cust.net   80.5%   175  510.0 513.5 
500.7 539.7   8.4
13. (waiting for reply)
…...
26. 216.147.3.209   59.1%   150  534.2 534.4 
513.5 549.7   8.1

44.236.47.236 to 178.63.26.145:

jcasc-rtc01 (10.96.54.240) -> hc-ping.com   
2024-01-26T08:10:24+
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
Packets   Pings
 Host Loss%   Snt   Last   Avg  
Best  Wrst StDev
 1. ip-10-96-50-153.us-west-2.compute.internal 0.0%   2670.2   0.2   
0.2   0.4   0.0
 2. ec2-34-221-151-225.us-west-2.compute.amazonaw  0.0%   2676.8  15.9   
0.7 114.1  24.1
 3. (waiting for reply)
 4. (waiting for reply)
 5. (waiting for reply)
 6. 241.0.2.5  0.0%   2670.4   0.4   
0.3   1.2   0.1
 7. 242.0.42.195   0.0%   2671.2   1.7   
0.5  23.9   3.0
 8. 240.2.144.12   0.0%   2675.2   5.2   
5.2  27.4   1.4
 9. amazon-ic-375418.ip.twelve99-cust.net  0.0%   2675.3   5.2   
5.2   6.1   0.1
10. port-b4-link.ip.twelve99.net   0.0%   2675.3   5.5   
4.9  35.5   3.0
11. port-b3-link.ip.twelve99.net   0.0%   2675.8   5.9   
5.6  11.8   0.5
12. palo-b24-link.ip.twelve99.net  4.9%   267   21.1  21.5  
21.0  58.4   3.1
13. sjo-b23-link.ip.twelve99.net   0.0%   266   21.4  22.7  
21.3  86.2   6.5
14. nyk-bb1-link.ip.twelve99.net  58.1%   266  432.7 422.7 
407.2 438.5   6.5
15. ldn-bb2-link.ip.twelve99.net  98.1%   266  485.6 485.4 
481.6 491.1   3.9
16. hbg-bb2-link.ip.twelve99.net  92.5%   266  504.1 499.8 
489.8 510.1   5.9
17. nug-b1-link.ip.twelve99.net   55.5%   266  523.5 519.6 
504.4 561.7   7.6
18. hetzner-ic-340780.ip.twelve99-cust.net53.6%   266  524.4 519.2 
506.0 545.5   6.9
19. core22.fsn1.hetzner.com   70.2%   266  521.7 519.2 
498.5 531.7   6.6
20. static.213-239-254-150.clients.your-server.de 33.2%   266  382.4 375.4 
364.9 396.5   4.1
21. static.145.26.63.178.clients.your-server.de   62.0%   266  529.9 518.4 
506.9 531.3   6.1


Thanks