As a follow up, we're still working with Spectrum engineers on a workaround, IPv6 is still mostly down in the DFW area on Spectrum outside of the large CDNs (i.e. Quad9 and Telia-bound traffic is still offline) - their ETA is another 24-48 hours as a line card for Houston was still on order, according to an engineer I spoke to. Not a great time for us IPv6-only networks.
On Thu, Apr 20, 2023 at 11:18 AM Tom Strickx <[email protected]> wrote: > > Charter has acknowledged the issue, and is tracking this internally as > INC000026440632. Appears to be an issue with a linecard in Houston. > > We've rerouted IPv6 out of Dallas for now to mitigate impact. Sorry for the > trouble! > -- > Tom Strickx > Principal Network Engineer > AS13335 - Cloudflare > > > On Thu, Apr 20, 2023 at 2:05 PM Tom Strickx <[email protected]> wrote: >> >> Hey folks, >> >> Could either of you contact me off-list so I can investigate? A specific >> start-of-problem would be super helpful. >> -- >> Tom Strickx >> Principal Network Engineer >> AS13335 - Cloudflare >> >> >> On Thu, Apr 20, 2023 at 4:38 AM Daniel Marks via Outages >> <[email protected]> wrote: >>> >>> Have been dealing with this for 2 days, nobody at Spectrum can figure this >>> out. >>> >>> Daniel Marks >>> May Mobility, Inc >>> >>> > >>> > On Apr 19, 2023, at 19:26, Carl Perry via Outages <[email protected]> >>> > wrote: >>> > >>> > Things have been rather slow the last couple of days, just figured out >>> > why; No IPv6 connectivity to any CloudFlare services. >>> > >>> > > mtr -6 --report-wide cloudflare.com >>> > Start: 2023-04-19T23:20:42+0000 >>> > HOST: defiant Loss% Snt Last Avg Best Wrst StDev >>> > 1.|-- 2603-8081-2340-2868-0215-17ff-fe94-6457.res6.spectrum.com 0.0% >>> > 10 3.6 19.1 1.5 93.2 36.1 >>> > 2.|-- 2603-90c5-0003-0180-0000-0000-0000-0001.inf6.spectrum.com 0.0% >>> > 10 14.5 36.6 11.3 171.2 50.0 >>> > 3.|-- lag-63.hcr02rdrktxaz.netops.charter.com 0.0% 10 18.8 37.3 >>> > 12.0 143.9 45.3 >>> > 4.|-- lag-19.ausxtxir02r.netops.charter.com 60.0% 10 65.8 30.2 >>> > 11.7 65.8 24.9 >>> > 5.|-- lag-22.rcr01hstqtx02.netops.charter.com 80.0% 10 26.5 25.5 >>> > 24.5 26.5 1.4 >>> > 6.|-- lag-16.hstqtx0209w-bcr00.netops.charter.com 90.0% 10 25.7 >>> > 25.7 25.7 25.7 0.0 >>> > 7.|-- lag-12.dllstx976iw-bcr00.netops.charter.com 40.0% 10 25.4 >>> > 26.3 18.2 44.4 9.5 >>> > 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 >>> > >>> > IPv4 works however: >>> > >>> > > mtr -4 --report-wide cloudflare.com >>> > Start: 2023-04-19T23:20:21+0000 >>> > HOST: defiant Loss% Snt Last Avg >>> > Best Wrst StDev >>> > 1.|-- _gateway 0.0% 10 2.1 1.8 >>> > 1.4 2.1 0.2 >>> > 2.|-- cpe-70-114-160-1.austin.res.rr.com 0.0% 10 51.9 >>> > 34.4 14.3 149.7 42.1 >>> > 3.|-- lag-63.hcr01rdrktxaz.netops.charter.com 0.0% 10 13.1 >>> > 38.5 13.1 159.1 44.6 >>> > 4.|-- lag-19.ausutxla01r.netops.charter.com 0.0% 10 18.1 >>> > 21.0 14.3 34.8 6.0 >>> > 5.|-- lag-22.rcr01dllatx37.netops.charter.com 0.0% 10 23.4 >>> > 23.3 18.3 33.3 4.3 >>> > 6.|-- lag-14.dllstx976iw-bcr00.netops.charter.com 0.0% 10 19.8 >>> > 27.0 16.1 51.2 11.9 >>> > 7.|-- lag-0.pr3.dfw10.netops.charter.com 0.0% 10 39.3 >>> > 28.5 14.1 62.3 16.7 >>> > 8.|-- 141.101.74.104 40.0% 10 53.2 >>> > 44.3 19.4 120.2 39.2 >>> > 9.|-- 172.71.168.2 0.0% 10 28.3 >>> > 28.8 10.8 79.4 19.5 >>> > 10.|-- 104.16.133.229 0.0% 10 122.0 >>> > 42.8 16.9 122.0 34.4 >>> > >>> > It's kind of wild how many things don't fall back to IPv4 quickly (lots >>> > of Android TV apps) or at all (.NET). I tried reaching out to Spectrum >>> > support, but being a lowly consumer I was only told that "The modem >>> > doesn't have any filters" and that "the problem may be on the CloudFlare >>> > side", which was sadly what I predicted. So, I figured I would raise the >>> > issue here in the hopes that actual network people would see it. >>> > >>> > -Carl >>> > >>> > >>> > _______________________________________________ >>> > Outages mailing list >>> > [email protected] >>> > https://www.google.com/url?q=https://puck.nether.net/mailman/listinfo/outages&source=gmail-imap&ust=1682551587000000&usg=AOvVaw0omaxNXXnf3Ff4dHAQDhA8 >>> _______________________________________________ >>> Outages mailing list >>> [email protected] >>> https://puck.nether.net/mailman/listinfo/outages -- -Dan _______________________________________________ Outages mailing list [email protected] https://puck.nether.net/mailman/listinfo/outages
