Re: MA statewide 911 outage and resolved shortly afterwards?

[Tony Tauber]

Interestingly, I saw this post on social media showing some confusion
across state lines.

I believe this person lives around 100 miles from the nearest border of
Maine.  Note the first alert doesn’t specify which state is affected but
the second does.
Commenters on the post indicate they received the alert in NY and NH (both
of which border MA directly whereas Maine does not).

Tony



On Tue, Jun 18, 2024 at 10:39 PM Tony Tauber  wrote:

> Yes, got alerts on phones on two different carriers for outage and
> restoration.
>
> Also email from local police for both (I’m subscribed to those.)
>
> Tony
>
>
>
> On Tue, Jun 18, 2024 at 4:56 PM Eric Kuhnke  wrote:
>
>> Via the Outages list, info from the Boston police department followed
>> about an hour later by a "it's fixed" message.
>>
>> Did anyone in MA receive a push notification from their cellular carrier?
>>
>>
>> Quotation from original posts follows
>>
>> "I have a report this 3pm Tue, hour from the authenticated FB page of the
>> Boston PD,
>> that 9-1-1 service is out statewide in Mass; they're handling it as a
>> Code 2 AHOD;
>> flag down a cruiser if you need help.
>>
>> """
>> 9-11 is currently down statewide.
>>
>> If you are experiencing an emergency please pull a Fire Box.
>>
>> If you need assistance, please reach out to your local Boston Police
>> District station.
>>
>> Boston Police will be patrolling with their blue lights activated for
>> high visibility.
>> Please approach an officer if you need assistance:
>>
>> Downtown A-1: (617) 343-4240
>> Charlestown A-15: (617) 343-4888
>> East Boston A-7: (617) 343-4220
>> Roxbury B-2: (617) 343-4270
>> Mattapan B-3: (617) 343-4700
>> South Boston C-6: (617) 343-4730
>> Dorchester C-11: (617) 343-4330
>> South End D-4: (617) 343-4250
>> Allston/Brighton D-14: (617) 343-4260
>> Roslindale/ West Roxbury E-5: (617) 343-4560
>> Jamaica Plain E-13: (617) 343-5630
>> Hyde Park E-18: (617) 343-5600
>>
>> 
>>
>> 911 Operating System is back up and running
>>
>> CANCEL 9-11 is currently down statewide.
>>
>> If you are experiencing an emergency please pull a Fire Box.
>>
>> If you need assistance, please reach out to your local Boston Police
>> District station.
>>
>> Boston Police will be patrolling with their blue lights activated for
>> high visibility.
>>
>> Please approach an officer if you need assistance:
>>
>> Downtown A-1: (617) 343-4240
>>
>> Charlestown A-15: (617) 343-4888
>>
>> East Boston A-7: (617) 343-4220
>>
>> Roxbury B-2: (617) 343-4270
>>
>> Mattapan B-3: (617) 343-4700
>>
>> South Boston C-6: (617) 343-4730
>>
>> Dorchester C-11: (617) 343-4330
>>
>> South End D-4: (617) 343-4250
>>
>> Allston/Brighton D-14: (617) 343-4260
>>
>> Roslindale/ West Roxbury E-5: (617) 343-4560
>>
>> Jamaica Plain E-13: (617) 343-5630
>>
>> Hyde Park E-18: (617) 343-5600
>>
>

Re: MA statewide 911 outage and resolved shortly afterwards?

[Tony Tauber]

Yes, got alerts on phones on two different carriers for outage and
restoration.

Also email from local police for both (I’m subscribed to those.)

Tony



On Tue, Jun 18, 2024 at 4:56 PM Eric Kuhnke  wrote:

> Via the Outages list, info from the Boston police department followed
> about an hour later by a "it's fixed" message.
>
> Did anyone in MA receive a push notification from their cellular carrier?
>
>
> Quotation from original posts follows
>
> "I have a report this 3pm Tue, hour from the authenticated FB page of the
> Boston PD,
> that 9-1-1 service is out statewide in Mass; they're handling it as a Code
> 2 AHOD;
> flag down a cruiser if you need help.
>
> """
> 9-11 is currently down statewide.
>
> If you are experiencing an emergency please pull a Fire Box.
>
> If you need assistance, please reach out to your local Boston Police
> District station.
>
> Boston Police will be patrolling with their blue lights activated for high
> visibility.
> Please approach an officer if you need assistance:
>
> Downtown A-1: (617) 343-4240
> Charlestown A-15: (617) 343-4888
> East Boston A-7: (617) 343-4220
> Roxbury B-2: (617) 343-4270
> Mattapan B-3: (617) 343-4700
> South Boston C-6: (617) 343-4730
> Dorchester C-11: (617) 343-4330
> South End D-4: (617) 343-4250
> Allston/Brighton D-14: (617) 343-4260
> Roslindale/ West Roxbury E-5: (617) 343-4560
> Jamaica Plain E-13: (617) 343-5630
> Hyde Park E-18: (617) 343-5600
>
> 
>
> 911 Operating System is back up and running
>
> CANCEL 9-11 is currently down statewide.
>
> If you are experiencing an emergency please pull a Fire Box.
>
> If you need assistance, please reach out to your local Boston Police
> District station.
>
> Boston Police will be patrolling with their blue lights activated for high
> visibility.
>
> Please approach an officer if you need assistance:
>
> Downtown A-1: (617) 343-4240
>
> Charlestown A-15: (617) 343-4888
>
> East Boston A-7: (617) 343-4220
>
> Roxbury B-2: (617) 343-4270
>
> Mattapan B-3: (617) 343-4700
>
> South Boston C-6: (617) 343-4730
>
> Dorchester C-11: (617) 343-4330
>
> South End D-4: (617) 343-4250
>
> Allston/Brighton D-14: (617) 343-4260
>
> Roslindale/ West Roxbury E-5: (617) 343-4560
>
> Jamaica Plain E-13: (617) 343-5630
>
> Hyde Park E-18: (617) 343-5600
>

MA statewide 911 outage and resolved shortly afterwards?

[Eric Kuhnke]

Via the Outages list, info from the Boston police department followed about
an hour later by a "it's fixed" message.

Did anyone in MA receive a push notification from their cellular carrier?


Quotation from original posts follows

"I have a report this 3pm Tue, hour from the authenticated FB page of the
Boston PD,
that 9-1-1 service is out statewide in Mass; they're handling it as a Code
2 AHOD;
flag down a cruiser if you need help.

"""
9-11 is currently down statewide.

If you are experiencing an emergency please pull a Fire Box.

If you need assistance, please reach out to your local Boston Police
District station.

Boston Police will be patrolling with their blue lights activated for high
visibility.
Please approach an officer if you need assistance:

Downtown A-1: (617) 343-4240
Charlestown A-15: (617) 343-4888
East Boston A-7: (617) 343-4220
Roxbury B-2: (617) 343-4270
Mattapan B-3: (617) 343-4700
South Boston C-6: (617) 343-4730
Dorchester C-11: (617) 343-4330
South End D-4: (617) 343-4250
Allston/Brighton D-14: (617) 343-4260
Roslindale/ West Roxbury E-5: (617) 343-4560
Jamaica Plain E-13: (617) 343-5630
Hyde Park E-18: (617) 343-5600



911 Operating System is back up and running

CANCEL 9-11 is currently down statewide.

If you are experiencing an emergency please pull a Fire Box.

If you need assistance, please reach out to your local Boston Police
District station.

Boston Police will be patrolling with their blue lights activated for high
visibility.

Please approach an officer if you need assistance:

Downtown A-1: (617) 343-4240

Charlestown A-15: (617) 343-4888

East Boston A-7: (617) 343-4220

Roxbury B-2: (617) 343-4270

Mattapan B-3: (617) 343-4700

South Boston C-6: (617) 343-4730

Dorchester C-11: (617) 343-4330

South End D-4: (617) 343-4250

Allston/Brighton D-14: (617) 343-4260

Roslindale/ West Roxbury E-5: (617) 343-4560

Jamaica Plain E-13: (617) 343-5630

Hyde Park E-18: (617) 343-5600

Re: HE.net outage / status page ?

[Michael Brown via NANOG]

On 2024-06-14 13:49, Scott Q. wrote:
Do you know if they have a status page or some other place where they 
write post-mortems ?


I don't believe such a page exists, no.

Re: HE.net outage / status page ?

[Scott Q.]

Do you know if they have a status page or some other place where
they write post-mortems ?

On Friday, 14/06/2024 at 13:03 Michael Brown via NANOG wrote:



On 2024-06-14 11:57, John Von Essen wrote:
> Could be related or coincidence, but at the same time as this HE 
> issue, we've been seeing inbound connectivity issues to Azure US 
> Central. Azure’s ASN does have a peer to HE….

HE is one of our upstreams for our hosting in Seattle.

Looking back at some internal monitoring, the IPv4 connectivity
problems 
started around 09:42Z. Problems appear to have been limited to v4
only.

While we were announcing via them, our incoming traffic was 
significantly reduced (and we received complaints) from the Eastern NA

and a good chunk of Europe.

Our impression is they were starting to look at the problem around 
10:30Z when we got in touch.

Allegedly the problem was fixed around 12:15Z but we haven't yet
failed 
back to test.

Re: HE.net outage / status page ?

[Michael Brown via NANOG]

On 2024-06-14 11:57, John Von Essen wrote:
Could be related or coincidence, but at the same time as this HE 
issue, we've been seeing inbound connectivity issues to Azure US 
Central. Azure’s ASN does have a peer to HE….


HE is one of our upstreams for our hosting in Seattle.

Looking back at some internal monitoring, the IPv4 connectivity problems 
started around 09:42Z. Problems appear to have been limited to v4 only.


While we were announcing via them, our incoming traffic was 
significantly reduced (and we received complaints) from the Eastern NA 
and a good chunk of Europe.


Our impression is they were starting to look at the problem around 
10:30Z when we got in touch.


Allegedly the problem was fixed around 12:15Z but we haven't yet failed 
back to test.

Re: HE.net outage / status page ?

[John Von Essen]

Could be related or coincidence, but at the same time as this HE issue, we've 
been seeing inbound connectivity issues to Azure US Central. Azure’s ASN does 
have a peer to HE….

-John


> On Jun 14, 2024, at 7:23 AM, Michael Brown via NANOG  wrote:
> 
> On 2024-06-14 06:32, Scott Q. wrote:
>> Anyone knows if Hurricane Electric is having an outage ? I also can't seem 
>> to find a status page for them.
>> 
>> Tracerouting to 118.172.248.120 for example through their network is 
>> impossible, but many other blocks are affected as well.
> Yes they are, seems to be a regional problem that started ~9:42Z.
> We just had to deactivate our sessions with them in Seattle due to this, but 
> San Jose / Dublin / Toronto are fine
> 
> M.
> 

Re: HE.net outage / status page ?

[Michael Brown via NANOG]

On 2024-06-14 06:32, Scott Q. wrote:
Anyone knows if Hurricane Electric is having an outage ? I also can't 
seem to find a status page for them.


Tracerouting to 118.172.248.120 for example through their network is 
impossible, but many other blocks are affected as well.

Yes they are, seems to be a regional problem that started ~9:42Z.

We just had to deactivate our sessions with them in Seattle due to this, 
but San Jose / Dublin / Toronto are fine


M.

Re: HE.net outage / status page ?

[Scott Q.]

I'm not sure what the issue was, Anurag showed some packet traces
from their network that were reaching the destination, but it is
mostly fixed now ( still seeing ~5% packet loss ).

There was an issue for sure because a few blocks from
Thailand/Singapore/Indonesia were not reachable at all even through
random locations from HE's LG. 

>From our point of view, the issue lasted about 6-7h starting @ 1AM
EST.


On Friday, 14/06/2024 at 09:11 Andy Ringsmuth wrote:



It would be far more helpful to have the discussion on the list as I
assume others are affected as well, which is the whole point of this
list. 



Andy Ringsmuth
5609 Harding Drive [1]
Lincoln, NE 68521-5831 [1]
(402) 202-1230
a...@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas,
1863




On Jun 14, 2024, at 6:26 AM, Anurag Bhatia  wrote:







Hi Scott 


Anurag @ AS6939 here. Will take this discussion off list. 


Thanks. 


On Fri, Jun 14, 2024 at 4:03 PM Scott Q.  wrote:




Anyone knows if Hurricane Electric is having an outage ? I also can't
seem to find a status page for them.

Tracerouting to 118.172.248.120 for example through their network is
impossible, but many other blocks are affected as well.

Verified through: https://lg.he.net/  but there's no status page
anywhere...

Thanks!
 



-- 





Anurag Bhatia


anuragbhatia.com [2] 





Links:
--
[1] x-apple-data-detectors://0/1
[2] https://anuragbhatia.com

Re: HE.net outage / status page ?

[Andy Ringsmuth]

It would be far more helpful to have the discussion on the list as I assume others are affected as well, which is the whole point of this list. Andy Ringsmuth5609 Harding DriveLincoln, NE 68521-5831(402) 202-1230a...@andyring.com“Better even die free, than to live slaves.” - Frederick Douglas, 1863On Jun 14, 2024, at 6:26 AM, Anurag Bhatia  wrote:Hi Scott Anurag @ AS6939 here. Will take this discussion off list. Thanks. On Fri, Jun 14, 2024 at 4:03 PM Scott Q. <qm...@top-consulting.net> wrote:





Anyone knows if Hurricane Electric is having an outage ? I also can't seem to find a status page for them.Tracerouting to 118.172.248.120 for example through their network is impossible, but many other blocks are affected as well.Verified through: https://lg.he.net/  but there's no status page anywhere...Thanks!
-- Anurag Bhatiaanuragbhatia.com

Re: HE.net outage / status page ?

[Anurag Bhatia]

Hi Scott


Anurag @ AS6939 here. Will take this discussion off list.


Thanks.

On Fri, Jun 14, 2024 at 4:03 PM Scott Q.  wrote:

> Anyone knows if Hurricane Electric is having an outage ? I also can't seem
> to find a status page for them.
>
> Tracerouting to 118.172.248.120 for example through their network is
> impossible, but many other blocks are affected as well.
>
> Verified through: https://lg.he.net/  but there's no status page
> anywhere...
>
> Thanks!
>
>

-- 
Anurag Bhatia
anuragbhatia.com

HE.net outage / status page ?

[Scott Q.]

Anyone knows if Hurricane Electric is having an outage ? I also
can't seem to find a status page for them.

Tracerouting to 118.172.248.120 for example through their network is
impossible, but many other blocks are affected as well.

Verified through: https://lg.he.net/  but there's no status page
anywhere...

Thanks!

Re: Another outage somewhere in Asia ?

[Grant Taylor via NANOG]

On 5/30/24 6:56 AM, Mehmet wrote:

This is within LA metro


We got bit by this overnight.

Customers transiting our network on the west coast of North America may 
have experienced Latency and Packet Loss. This was due to a fiber cut in 
our vendors network around the Los Angeles area. The vendor has found 
the damaged fiber and have repaired the cables. Cogent NOC is now 
monitoring this event. Customers should have seen their services 
restored at 11:46 AM ET. The master case is HD303782528.


https://ecogent.cogentco.com/network-status




--
Grant. . . .
unix || die

Re: Another outage somewhere in Asia ?

[Mehmet]

This is within LA metro

Mehmet


On Thu, May 30, 2024 at 07:53 Scott Q.  wrote:

> Oh, ok. That explains why it stops in Lax within Cogent. It appears to be
> a submarine cable cut from their page:
>
> Customers Transiting our network from Europe to our Asian markets may be
> experiencing Latency and Packet Loss. Multiple vessels are currently being
> gathered and/or being dispatched to carry out the repair operations from
> the submarine faults currently being reported. There is NO ETR at this time.
>
>
> On Thursday, 30/05/2024 at 07:27 Mehmet wrote:
>
> I assume this is due to cogent having massive outage in California and
> traffic shifting away from Cogent congesting 1299. It’s my quick
> observation
>
>
> On Thu, May 30, 2024 at 07:24 Scott Q.  wrote:
>
>> Anyone else noticed some major latency / packet loss going to Asia
>> starting roughly 10 hours ago ?
>>
>> For example, tracerouting to 36.68.218.127 provides very mixed results.
>>
>> Traceroute from 192.168.121.126 to 36.68.218.127 (36.68.218.127):
>> 1 192.168.121.1 1.337ms 0.375ms 0.34ms
>> 2 lo0-0.bng4.wh-man.zen.net.uk (51.148.77.131) 5.744ms 4.817ms 4.139ms
>> 3 lag-8.p2.wh-man.zen.net.uk (51.148.73.118) 11.71ms 11.515ms 11.817ms
>> 4 lag-3.p1.thn-lon.zen.net.uk (51.148.73.136) 11.844ms 11.645ms 11.63ms
>> 5 ldn-bb1-link.ip.twelve99.net (62.115.120.74) 12.61ms 12.755ms
>> 12.758ms
>> 6 prs-bb1-link.ip.twelve99.net (62.115.135.25) 19.242ms 18.703ms *
>> 7 mei-b5-link.ip.twelve99.net (62.115.124.55) 28.8ms 28.702ms 28.593ms
>> 8 * prs-bb1-link.ip.twelve99.net (62.115.124.54) 563.155ms *
>> 9 * ldn-bb1-link.ip.twelve99.net (62.115.135.24) 637.818ms *
>> 10 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 364.331ms * *
>> 11 * * *
>> 12 port-b3-link.ip.twelve99.net (62.115.115.25) 1130.258ms 1155.178ms
>> 1149.024ms
>> 13 * tky-b3-link.ip.twelve99.net (62.115.127.11) 1109.197ms *
>> 14 * tky-b2-link.ip.twelve99.net (62.115.126.134) 1498.216ms *
>> 15 * * *
>> 16 * * *
>> 17 * * *
>> 18 180.240.191.166 901.364ms * *
>> 19 * 180.240.204.9 1164.257ms *
>> 20 * * *
>> 21 * * *
>> 22 * * *
>> 23 * * *
>> 24 * * *
>> 255 * * *
>>
>

Re: Another outage somewhere in Asia ?

[Scott Q.]

Oh, ok. That explains why it stops in Lax within Cogent. It
appears to be a submarine cable cut from their page:

Customers Transiting our network from Europe to our Asian markets may
be experiencing Latency and Packet Loss. Multiple vessels are
currently being gathered and/or being dispatched to carry out the
repair operations from the submarine faults currently being reported.
There is NO ETR at this time.

On Thursday, 30/05/2024 at 07:27 Mehmet wrote:




I assume this is due to cogent having massive outage in California and
traffic shifting away from Cogent congesting 1299. It’s my quick
observation 




On Thu, May 30, 2024 at 07:24 Scott Q.  wrote:


 
Anyone else noticed some major latency / packet loss going to Asia
starting roughly 10 hours ago ?

For example, tracerouting to 36.68.218.127 provides very mixed
results.


Traceroute from 192.168.121.126 to 36.68.218.127 (36.68.218.127):
1 192.168.121.1 1.337ms 0.375ms 0.34ms  
2 lo0-0.bng4.wh-man.zen.net.uk [1] (51.148.77.131) 5.744ms 4.817ms
4.139ms  
3 lag-8.p2.wh-man.zen.net.uk [2] (51.148.73.118) 11.71ms 11.515ms
11.817ms  
4 lag-3.p1.thn-lon.zen.net.uk [3] (51.148.73.136) 11.844ms 11.645ms
11.63ms  
5 ldn-bb1-link.ip.twelve99.net [4] (62.115.120.74) 12.61ms 12.755ms
12.758ms  
6 prs-bb1-link.ip.twelve99.net [5] (62.115.135.25) 19.242ms 18.703ms
*  
7 mei-b5-link.ip.twelve99.net [6] (62.115.124.55) 28.8ms 28.702ms
28.593ms  
8 * prs-bb1-link.ip.twelve99.net [5] (62.115.124.54) 563.155ms *  
9 * ldn-bb1-link.ip.twelve99.net [4] (62.115.135.24) 637.818ms *  
10 nyk-bb2-link.ip.twelve99.net [7] (62.115.113.20) 364.331ms * *  
11 * * *  
12 port-b3-link.ip.twelve99.net [8] (62.115.115.25) 1130.258ms
1155.178ms 1149.024ms  
13 * tky-b3-link.ip.twelve99.net [9] (62.115.127.11) 1109.197ms *  
14 * tky-b2-link.ip.twelve99.net [10] (62.115.126.134) 1498.216ms
*  
15 * * *  
16 * * *  
17 * * *  
18 180.240.191.166 901.364ms * *  
19 * 180.240.204.9 1164.257ms *  
20 * * *  
21 * * *  
22 * * *  
23 * * *  
24 * * *  
255 * * *   





Links:
--
[1] http://lo0-0.bng4.wh-man.zen.net.uk
[2] http://lag-8.p2.wh-man.zen.net.uk
[3] http://lag-3.p1.thn-lon.zen.net.uk
[4] http://ldn-bb1-link.ip.twelve99.net
[5] http://prs-bb1-link.ip.twelve99.net
[6] http://mei-b5-link.ip.twelve99.net
[7] http://nyk-bb2-link.ip.twelve99.net
[8] http://port-b3-link.ip.twelve99.net
[9] http://tky-b3-link.ip.twelve99.net
[10] http://tky-b2-link.ip.twelve99.net

Re: Another outage somewhere in Asia ?

[Mehmet]

I assume this is due to cogent having massive outage in California and
traffic shifting away from Cogent congesting 1299. It’s my quick
observation


On Thu, May 30, 2024 at 07:24 Scott Q.  wrote:

> Anyone else noticed some major latency / packet loss going to Asia
> starting roughly 10 hours ago ?
>
> For example, tracerouting to 36.68.218.127 provides very mixed results.
>
> Traceroute from 192.168.121.126 to 36.68.218.127 (36.68.218.127):
> 1 192.168.121.1 1.337ms 0.375ms 0.34ms
> 2 lo0-0.bng4.wh-man.zen.net.uk (51.148.77.131) 5.744ms 4.817ms 4.139ms
> 3 lag-8.p2.wh-man.zen.net.uk (51.148.73.118) 11.71ms 11.515ms 11.817ms
> 4 lag-3.p1.thn-lon.zen.net.uk (51.148.73.136) 11.844ms 11.645ms 11.63ms
> 5 ldn-bb1-link.ip.twelve99.net (62.115.120.74) 12.61ms 12.755ms 12.758ms
> 6 prs-bb1-link.ip.twelve99.net (62.115.135.25) 19.242ms 18.703ms *
> 7 mei-b5-link.ip.twelve99.net (62.115.124.55) 28.8ms 28.702ms 28.593ms
> 8 * prs-bb1-link.ip.twelve99.net (62.115.124.54) 563.155ms *
> 9 * ldn-bb1-link.ip.twelve99.net (62.115.135.24) 637.818ms *
> 10 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 364.331ms * *
> 11 * * *
> 12 port-b3-link.ip.twelve99.net (62.115.115.25) 1130.258ms 1155.178ms
> 1149.024ms
> 13 * tky-b3-link.ip.twelve99.net (62.115.127.11) 1109.197ms *
> 14 * tky-b2-link.ip.twelve99.net (62.115.126.134) 1498.216ms *
> 15 * * *
> 16 * * *
> 17 * * *
> 18 180.240.191.166 901.364ms * *
> 19 * 180.240.204.9 1164.257ms *
> 20 * * *
> 21 * * *
> 22 * * *
> 23 * * *
> 24 * * *
> 255 * * *
>

Another outage somewhere in Asia ?

[Scott Q.]

Anyone else noticed some major latency / packet loss going to
Asia starting roughly 10 hours ago ?

For example, tracerouting to 36.68.218.127 provides very mixed
results.


Traceroute from 192.168.121.126 to 36.68.218.127 (36.68.218.127):
1 192.168.121.1 1.337ms 0.375ms 0.34ms  
2 lo0-0.bng4.wh-man.zen.net.uk (51.148.77.131) 5.744ms 4.817ms
4.139ms  
3 lag-8.p2.wh-man.zen.net.uk (51.148.73.118) 11.71ms 11.515ms
11.817ms  
4 lag-3.p1.thn-lon.zen.net.uk (51.148.73.136) 11.844ms 11.645ms
11.63ms  
5 ldn-bb1-link.ip.twelve99.net (62.115.120.74) 12.61ms 12.755ms
12.758ms  
6 prs-bb1-link.ip.twelve99.net (62.115.135.25) 19.242ms 18.703ms *  
7 mei-b5-link.ip.twelve99.net (62.115.124.55) 28.8ms 28.702ms
28.593ms  
8 * prs-bb1-link.ip.twelve99.net (62.115.124.54) 563.155ms *  
9 * ldn-bb1-link.ip.twelve99.net (62.115.135.24) 637.818ms *  
10 nyk-bb2-link.ip.twelve99.net (62.115.113.20) 364.331ms * *  
11 * * *  
12 port-b3-link.ip.twelve99.net (62.115.115.25) 1130.258ms 1155.178ms
1149.024ms  
13 * tky-b3-link.ip.twelve99.net (62.115.127.11) 1109.197ms *  
14 * tky-b2-link.ip.twelve99.net (62.115.126.134) 1498.216ms *  
15 * * *  
16 * * *  
17 * * *  
18 180.240.191.166 901.364ms * *  
19 * 180.240.204.9 1164.257ms *  
20 * * *  
21 * * *  
22 * * *  
23 * * *  
24 * * *  
255 * * *  

RE: Frontier fiber outage in Tampa Bay?

[Greg Dickinson]

Folks on the outages list say it's a widespread problem. To quote:

"I just spoke to the NOC manager and they are aware of the issue relating to a 
specific BNG and are working on it."

Greg Dickinson, CCNA
Network Engineer

[mid:ac0798f5d04aec2c4c40f9c44056646c8ba72bfb332f7f64d451d99665886...@getboxer.com/image001.png@01D2DDE3.06E76B70]

From: NANOG  On Behalf 
Of David Hubbard
Sent: Tuesday, May 7, 2024 1:39 PM
To: nanog@nanog.org
Subject: Frontier fiber outage in Tampa Bay?




This message originates from outside Bryant Bank.   Please use caution when 
opening this correspondence, attachments or hyperlinks (URLs).  If you have 
questions, please contact IT Support.  Thank you.



Curious if anyone else is having issues with Frontier fiber service in Tampa 
Bay region?  I've got a couple locations down.  Their support could not 
possibly be worse, so figured I'd start asking around while I consider giving 
up lol

Thanks


NOTICE: This electronic mail message and any files transmitted with it are 
intended exclusively for the individual or entity to which it is addressed. The 
message, together with any attachment, may contain confidential and/or 
privileged information. Any unauthorized review, use, print, save, copy, 
disclosure or distribution is strictly prohibited. If you have received this 
message in error, please immediately advise the sender by reply email and 
delete copies.  Thank you.

Frontier fiber outage in Tampa Bay?

[David Hubbard]

Curious if anyone else is having issues with Frontier fiber service in Tampa 
Bay region?  I've got a couple locations down.  Their support could not 
possibly be worse, so figured I'd start asking around while I consider giving 
up lol

Thanks

RE: Meta outage

[Ray Van Dolson via NANOG]

On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun  wrote:

>> What I found intriguing was that I was logged out by Google Docs at 
>> the same moment FB logged me out.  Downdetector showed a number of 
>> other supposedly unrelated services with large outage report spikes 
>> at roughly the same time.

> I was logged out of Honeywell's Total Connect Comfort site (remote
> thermostat control) at the same time FB logged me out. I'm not using
> OAUTH logins anywhere.
>
> I had recently changed SSID and the thermostat wasn't accepting remote
> commands. I thought maybe the SSID change had broken it and had just
> deleted the device from the TCC site to try adding it back when I was
> logged out.
>
> It's funny timing because earlier today I was telling an employee how
> we don't like to have our maintenance overlap with vendor's
> maintenance/repair window because when something breaks while we are
> making changes we can't always tell who is at fault.

Interesting.  I have a script that makes a call to TCC every so often to read
temperature data, and it failed from 8:50AM PST to 10:30AM PST.

Ray

RE: Meta outage

[Mike Lewinski via NANOG]

On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun  wrote:

> What I found intriguing was that I was logged out by Google Docs at the same 
> moment FB logged me out.  Downdetector showed a number of other supposedly 
> unrelated services with large outage report spikes at roughly the same time.

I was logged out of Honeywell's Total Connect Comfort site (remote thermostat 
control) at the same time FB logged me out. I'm not using OAUTH logins anywhere.

I had recently changed SSID and the thermostat wasn't accepting remote 
commands. I thought maybe the SSID change had broken it and had just deleted 
the device from the TCC site to try adding it back when I was logged out.

It's funny timing because earlier today I was telling an employee how we don't 
like to have our maintenance overlap with vendor's maintenance/repair window 
because when something breaks while we are making changes we can't always tell 
who is at fault.

Re: Meta outage

[Michael Rathbun]

On Tue, 05 Mar 2024 11:06:11 -0500, Jay Ashworth  wrote:

>It's making the general press this hour so of course you already know about it 
>but my question is this: who peers with meta and have you seen BGP sessions 
>drop or the like? Do you operate meta CDN nodes in your network? Are they 
>screaming for help? 
>
>This doesn't sound like it's a network layer problem but I'm curious.

What I found intriguing was that I was logged out by Google Docs at the
same moment FB logged me out.  Downdetector showed a number of other
supposedly unrelated services with large outage report spikes at roughly
the same time.

mdr
-- 
 "There are no laws here, only agreements."  
-- Masahiko

Re: Meta outage

[Ross Tajvar]

That's from 2010. You can see if you look at the non-mobile version of the
page (requires login though):
https://www.facebook.com/notes/10158791436142200/
[image: image.png]

On Tue, Mar 5, 2024 at 1:32 PM Jorge Amodio  wrote:

>
> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F
>
> - Jorge (mobile)
>
>
> On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG 
> wrote:
>
> 
>
> Does meta keep a board somewhere to tell the world it’s down?
>
>
>
> *From:* NANOG  *On Behalf Of *Jay
> Ashworth
> *Sent:* Tuesday, March 05, 2024 11:06 AM
> *To:* nanog@nanog.org
> *Subject:* Meta outage
>
>
>
> WARNING: This message originated outside of Ford Motor Company. Use
> caution when opening attachments, clicking links, or responding.
>
>
>
> It's making the general press this hour so of course you already know
> about it but my question is this: who peers with meta and have you seen BGP
> sessions drop or the like? Do you operate meta CDN nodes in your network?
> Are they screaming for help?
>
> This doesn't sound like it's a network layer problem but I'm curious.
>
> Cheers,
> -- jra
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
>

Re: Meta outage

[Jorge Amodio]

https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F

- Jorge (mobile)


> On Mar 5, 2024, at 10:25, Kain, Becki (.) via NANOG  wrote:
> 
> 
> Does meta keep a board somewhere to tell the world it’s down?
>  
> From: NANOG  On Behalf Of Jay 
> Ashworth
> Sent: Tuesday, March 05, 2024 11:06 AM
> To: nanog@nanog.org
> Subject: Meta outage
>  
> WARNING: This message originated outside of Ford Motor Company. Use caution 
> when opening attachments, clicking links, or responding.
>  
> It's making the general press this hour so of course you already know about 
> it but my question is this: who peers with meta and have you seen BGP 
> sessions drop or the like? Do you operate meta CDN nodes in your network? Are 
> they screaming for help?
> 
> This doesn't sound like it's a network layer problem but I'm curious.
> 
> Cheers,
> -- jra
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [External] Re: Meta outage

[Hunter Fuller via NANOG]

On Tue, Mar 5, 2024 at 10:38 AM Chris K  wrote:
> see: Status and outages of Meta business products (metastatus.com)

Things seem to be returning now, aside from the status page, which
still appears blank. Interesting.

-- 
Hunter Fuller (they)
Router Jockey
VBH M-1C
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering

Re: Meta outage

[Stephane Bortzmeyer]

On Tue, Mar 05, 2024 at 04:23:42PM +,
 Kain, Becki (.) via NANOG  wrote 
https://metastatus.com/ a message of 210 lines which said:

> Does meta keep a board somewhere to tell the world it’s down?

https://metastatus.com/

Re: Meta outage

[Chris K]

see: Status and outages of Meta business products 
(metastatus.com)<https://metastatus.com/>

From: NANOG  on behalf of Kain, Becki 
(.) via NANOG 
Sent: Tuesday, March 5, 2024 11:23 AM
To: Jay Ashworth ; nanog@nanog.org 
Subject: RE: Meta outage


Does meta keep a board somewhere to tell the world it’s down?



From: NANOG  On Behalf Of Jay Ashworth
Sent: Tuesday, March 05, 2024 11:06 AM
To: nanog@nanog.org
Subject: Meta outage



WARNING: This message originated outside of Ford Motor Company. Use caution 
when opening attachments, clicking links, or responding.



It's making the general press this hour so of course you already know about it 
but my question is this: who peers with meta and have you seen BGP sessions 
drop or the like? Do you operate meta CDN nodes in your network? Are they 
screaming for help?

This doesn't sound like it's a network layer problem but I'm curious.

Cheers,
-- jra

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

RE: Meta outage

[Kain, Becki (.) via NANOG]

Thank you

From: Jay Ashworth 
Sent: Tuesday, March 05, 2024 11:25 AM
To: Kain, Becki (.) ; nanog@nanog.org
Subject: RE: Meta outage

Yes: metastatus.com

It isn't happy.

On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)" 
mailto:bka...@ford.com>> wrote:
Does meta keep a board somewhere to tell the world it’s down?

From: NANOG 
mailto:nanog-bounces+bkain1=ford@nanog.org>>
 On Behalf Of Jay Ashworth
Sent: Tuesday, March 05, 2024 11:06 AM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Meta outage

WARNING: This message originated outside of Ford Motor Company. Use caution 
when opening attachments, clicking links, or responding.

It's making the general press this hour so of course you already know about it 
but my question is this: who peers with meta and have you seen BGP sessions 
drop or the like? Do you operate meta CDN nodes in your network? Are they 
screaming for help?

This doesn't sound like it's a network layer problem but I'm curious.

Cheers,
-- jra
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Meta outage

[Dominik Dobrowolski]

This time it looks like AAA issue on Facebook side.

All peerings are up and cdns are reachable

Dominik Dobrowolski

W dniu wt., 5.03.2024 o 17:14 Stephane Bortzmeyer 
napisał(a):

> On Tue, Mar 05, 2024 at 11:06:11AM -0500,
>  Jay Ashworth  wrote
>  a message of 124 lines which said:
>
> > This doesn't sound like it's a network layer problem but I'm curious.
>
> We can see the start page, authentification fails with an error
> message. It does not look like a network issue.
>
>

RE: Meta outage

[Jay Ashworth]

Yes: metastatus.com

It isn't happy.

On March 5, 2024 11:23:42 AM EST, "Kain, Becki (.)"  wrote:
>Does meta keep a board somewhere to tell the world it’s down?
>
>From: NANOG  On Behalf Of Jay Ashworth
>Sent: Tuesday, March 05, 2024 11:06 AM
>To: nanog@nanog.org
>Subject: Meta outage
>
>WARNING: This message originated outside of Ford Motor Company. Use caution 
>when opening attachments, clicking links, or responding.
>
>It's making the general press this hour so of course you already know about it 
>but my question is this: who peers with meta and have you seen BGP sessions 
>drop or the like? Do you operate meta CDN nodes in your network? Are they 
>screaming for help?
>
>This doesn't sound like it's a network layer problem but I'm curious.
>
>Cheers,
>-- jra
>--
>Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

RE: Meta outage

[Kain, Becki (.) via NANOG]

Does meta keep a board somewhere to tell the world it’s down?

From: NANOG  On Behalf Of Jay Ashworth
Sent: Tuesday, March 05, 2024 11:06 AM
To: nanog@nanog.org
Subject: Meta outage

WARNING: This message originated outside of Ford Motor Company. Use caution 
when opening attachments, clicking links, or responding.

It's making the general press this hour so of course you already know about it 
but my question is this: who peers with meta and have you seen BGP sessions 
drop or the like? Do you operate meta CDN nodes in your network? Are they 
screaming for help?

This doesn't sound like it's a network layer problem but I'm curious.

Cheers,
-- jra
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Meta outage

[Dave Taht]

My first thought, I'll admit, is that they turned a LLM loose on their
configuration controls, and it "naturally" put itself in route 666.

I sure hope from whatever subbasement clueful admins are still in over
there, they can put it back together. Otherwise, productivity
might improve as users flee to Office 365 to get some work done.

I keep worrying about KEYTRAP

--
https://www.youtube.com/watch?v=N0Tmvv5jJKs Epik Mellon: Making the
Internet Better
Dave Täht CSO, LibreQos

Re: Meta outage

[Jorge Amodio]

They reported a major failure (fat finger source) on one of their DB
clusters.

-J

On Tue, Mar 5, 2024 at 10:07 AM Jay Ashworth  wrote:

> It's making the general press this hour so of course you already know
> about it but my question is this: who peers with meta and have you seen BGP
> sessions drop or the like? Do you operate meta CDN nodes in your network?
> Are they screaming for help?
>
> This doesn't sound like it's a network layer problem but I'm curious.
>
> Cheers,
> -- jra
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

Re: Meta outage

[Stephane Bortzmeyer]

On Tue, Mar 05, 2024 at 11:06:11AM -0500,
 Jay Ashworth  wrote 
 a message of 124 lines which said:

> This doesn't sound like it's a network layer problem but I'm curious.

We can see the start page, authentification fails with an error
message. It does not look like a network issue.

Meta outage

[Jay Ashworth]

It's making the general press this hour so of course you already know about it 
but my question is this: who peers with meta and have you seen BGP sessions 
drop or the like? Do you operate meta CDN nodes in your network? Are they 
screaming for help? 

This doesn't sound like it's a network layer problem but I'm curious.

Cheers,
-- jra
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: Any info on AT Wireless Outage?

[Joel Esler]

/me waves my hand dismissingly— Sent from my iPhoneOn Feb 29, 2024, at 14:55, Javier J  wrote:Where did you see this? Erik Prince was on the PBD podcast saying he has a 70% chance in his head it was China. I tend to learn towards human error from my experience in the IT biz.- JOn Wed, Feb 28, 2024 at 10:58 AM <j...@joelesler.net> wrote:I read it as “someone pushed an ACL that wasn’t properly reviewed and it really screwed things up."On Feb 27, 2024, at 21:41, Mark Seiden <m...@seiden.com> wrote:aside from the official pablum that was released about an “incorrect process used”(which says exactly nothing) does anyone actually know anything accurate and more specific about the root cause?(and why it took 11 hours to recover?)On Feb 22, 2024, at 11:15 AM, John Councilman <jcouncil...@gmail.com> wrote:From what I've read, they lost their database of SIM cards.  I could be wrong of course.On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel <d...@hetzel.org> wrote:As widespread as it seemed to be, it feels like it would be quite a trick if it were a single piece of hardware.  Firmware load that ended badly, I wonder?On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG <nanog@nanog.org> wrote:






Do you have the ability to expand on this at all? Do you mean a hardware failure of some kind IE router, optitcs, etc? 

From: NANOG advance-trading@nanog.org>
On Behalf Of R. Leigh Hennig
Sent: Thursday, February 22, 2024 8:17 AM
To: Robert DeVita <radev...@mejeticks.com>
Cc: nanog@nanog.org
Subject: Re: Any info on AT Wireless Outage?

 Word around the campfire is that it’s a Cisco issue.




On Feb 22, 2024, at 8:03 AM, Robert DeVita <radev...@mejeticks.com> wrote:
 

Reports have it starting at 4:30 a.m.. SOS on all phones..

 

 

 


















Robert DeVita









CEO and Founder

















t: (469) 581-2160



 | 



m: (469) 441-8864

















e: radev...@mejeticks.com



 | 



w: mejeticks.com













a: 







2323 N Akard Street



, 



Dallas



, 



75201




















































































 



The risk of trading futures and options can be substantial. All information, publications, and material used and distributed by Advance Trading Inc. shall be construed as a solicitation. ATI does not maintain an independent research department as defined in
 CFTC Regulation 1.71. Information obtained from third-party sources is believed to be reliable, but its accuracy is not guaranteed by Advance Trading Inc. Past performance is not necessarily indicative of future results.

Re: Any info on AT Wireless Outage?

[Tom Beecher]

>
> Erik Prince was on the PBD podcast saying he has a 70% chance in his head
> it was China.
>

Why do we care what his (almost certainly uninformed) opinion is?

On Thu, Feb 29, 2024 at 2:56 PM Javier J  wrote:

> Where did you see this? Erik Prince was on the PBD podcast saying he has a
> 70% chance in his head it was China. I tend to learn towards human error
> from my experience in the IT biz.
>
> - J
>
> On Wed, Feb 28, 2024 at 10:58 AM  wrote:
>
>> I read it as “someone pushed an ACL that wasn’t properly reviewed and it
>> really screwed things up."
>>
>> On Feb 27, 2024, at 21:41, Mark Seiden  wrote:
>>
>> aside from the official pablum that was released about an “incorrect
>> process used”
>> (which says exactly nothing) does anyone actually know anything accurate
>> and
>> more specific about the root cause?
>>
>> (and why it took 11 hours to recover?)
>>
>> On Feb 22, 2024, at 11:15 AM, John Councilman 
>> wrote:
>>
>> From what I've read, they lost their database of SIM cards.  I could be
>> wrong of course.
>>
>> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>>
>>> As widespread as it seemed to be, it feels like it would be quite a
>>> trick if it were a single piece of hardware.  Firmware load that ended
>>> badly, I wonder?
>>>
>>>
>>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>>> wrote:
>>>
>>>> Do you have the ability to expand on this at all? Do you mean a
>>>> hardware failure of some kind IE router, optitcs, etc?
>>>>
>>>>
>>>>
>>>> *From:* NANOG  *On
>>>> Behalf Of *R. Leigh Hennig
>>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>>> *To:* Robert DeVita 
>>>> *Cc:* nanog@nanog.org
>>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>>
>>>>
>>>>
>>>> Word around the campfire is that it’s a Cisco issue.
>>>>
>>>>
>>>>
>>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Robert DeVita**​**​**​**​*
>>>>
>>>> *CEO and Founder*
>>>>
>>>> t: (469) 581-2160 <(469)%20581-2160>
>>>>
>>>>  |
>>>>
>>>> m: (469) 441-8864 <(469)%20441-8864>
>>>>
>>>> e: radev...@mejeticks.com
>>>>
>>>>  |
>>>>
>>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>>
>>>> a:
>>>>
>>>> 2323 N Akard Street
>>>>
>>>> ,
>>>>
>>>> Dallas
>>>>
>>>> ,
>>>>
>>>> 75201
>>>>
>>>> <https://www.linkedin.com/company/mejeticks/>
>>>>
>>>> <https://twitter.com/mejeticks>
>>>>
>>>> <https://www.facebook.com/mejeticks>
>>>>
>>>> <https://linktr.ee/mejeticks>
>>>>
>>>>
>>>>
>>>>
>>>> The risk of trading futures and options can be substantial. All
>>>> information, publications, and material used and distributed by Advance
>>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>>> independent research department as defined in CFTC Regulation 1.71.
>>>> Information obtained from third-party sources is believed to be reliable,
>>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>>> is not necessarily indicative of future results.
>>>>
>>>
>>
>>

Re: Any info on AT Wireless Outage?

[Ryan Wilkins via NANOG]

No $5 received here.. Just a text message saying, "It's AT We apologize for 
Thursday's outage, which may have impacted you. As a valued customer, your 
connection matters and we are committed to doing better.” 

I had the thought the other day that maybe this was a hack and that they didn’t 
want to admit such.  I’m sure FirstNet being taken over by hackers isn’t 
exactly the message they want out there.  Regardless of what the actual cause 
was, their lack of communicating anything meaningful is saying something which 
boosts the chances that this was a hack in my mind.

Agreed on the human error thing, but maybe the human error was something the 
hackers did that they didn’t mean to.


> On Feb 29, 2024, at 2:55 PM, Javier J  wrote:
> 
> Where did you see this? Erik Prince was on the PBD podcast saying he has a 
> 70% chance in his head it was China. I tend to learn towards human error from 
> my experience in the IT biz.
> 
> - J
> 
> On Wed, Feb 28, 2024 at 10:58 AM  <mailto:j...@joelesler.net>> wrote:
>> I read it as “someone pushed an ACL that wasn’t properly reviewed and it 
>> really screwed things up."
>> 
>>> On Feb 27, 2024, at 21:41, Mark Seiden >> <mailto:m...@seiden.com>> wrote:
>>> 
>>> aside from the official pablum that was released about an “incorrect 
>>> process used”
>>> (which says exactly nothing) does anyone actually know anything accurate 
>>> and 
>>> more specific about the root cause?
>>> 
>>> (and why it took 11 hours to recover?)
>>> 
>>>> On Feb 22, 2024, at 11:15 AM, John Councilman >>> <mailto:jcouncil...@gmail.com>> wrote:
>>>> 
>>>> From what I've read, they lost their database of SIM cards.  I could be 
>>>> wrong of course.
>>>> 
>>>> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel >>> <mailto:d...@hetzel.org>> wrote:
>>>>> As widespread as it seemed to be, it feels like it would be quite a trick 
>>>>> if it were a single piece of hardware.  Firmware load that ended badly, I 
>>>>> wonder?
>>>>> 
>>>>> 
>>>>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG >>>> <mailto:nanog@nanog.org>> wrote:
>>>>>> Do you have the ability to expand on this at all? Do you mean a hardware 
>>>>>> failure of some kind IE router, optitcs, etc?
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> From: NANOG >>>>> <mailto:advance-trading@nanog.org>> On Behalf Of R. Leigh Hennig
>>>>>> Sent: Thursday, February 22, 2024 8:17 AM
>>>>>> To: Robert DeVita >>>>> <mailto:radev...@mejeticks.com>>
>>>>>> Cc: nanog@nanog.org <mailto:nanog@nanog.org>
>>>>>> Subject: Re: Any info on AT Wireless Outage?
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> Word around the campfire is that it’s a Cisco issue.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita >>>>> <mailto:radev...@mejeticks.com>> wrote:
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>>  
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Robert DeVita
>>>>>> 
>>>>>> CEO and Founder
>>>>>> 
>>>>>> t: (469) 581-2160  
>>>>>>  | 
>>>>>> 
>>>>>> m: (469) 441-8864 
>>>>>> e: radev...@mejeticks.com <mailto:radev...@mejeticks.com>
>>>>>>  | 
>>>>>> 
>>>>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>>>> a: 
>>>>>> 
>>>>>> 2323 N Akard Street
>>>>>> 
>>>>>> , 
>>>>>> 
>>>>>> Dallas
>>>>>> 
>>>>>> , 
>>>>>> 
>>>>>> 75201
>>>>>> 
>>>>>>  <https://www.linkedin.com/company/mejeticks/>   
>>>>>>  <https://twitter.com/mejeticks> 
>>>>>>  <https://www.facebook.com/mejeticks>
>>>>>>  <https://linktr.ee/mejeticks>
>>>>>>  
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> The risk of trading futures and options can be substantial. All 
>>>>>> information, publications, and material used and distributed by Advance 
>>>>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain 
>>>>>> an independent research department as defined in CFTC Regulation 1.71. 
>>>>>> Information obtained from third-party sources is believed to be 
>>>>>> reliable, but its accuracy is not guaranteed by Advance Trading Inc. 
>>>>>> Past performance is not necessarily indicative of future results.
>>> 
>> 

Re: Any info on AT Wireless Outage?

[Javier J]

Where did you see this? Erik Prince was on the PBD podcast saying he has a
70% chance in his head it was China. I tend to learn towards human error
from my experience in the IT biz.

- J

On Wed, Feb 28, 2024 at 10:58 AM  wrote:

> I read it as “someone pushed an ACL that wasn’t properly reviewed and it
> really screwed things up."
>
> On Feb 27, 2024, at 21:41, Mark Seiden  wrote:
>
> aside from the official pablum that was released about an “incorrect
> process used”
> (which says exactly nothing) does anyone actually know anything accurate
> and
> more specific about the root cause?
>
> (and why it took 11 hours to recover?)
>
> On Feb 22, 2024, at 11:15 AM, John Councilman 
> wrote:
>
> From what I've read, they lost their database of SIM cards.  I could be
> wrong of course.
>
> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>
>> As widespread as it seemed to be, it feels like it would be quite a trick
>> if it were a single piece of hardware.  Firmware load that ended badly, I
>> wonder?
>>
>>
>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>> wrote:
>>
>>> Do you have the ability to expand on this at all? Do you mean a hardware
>>> failure of some kind IE router, optitcs, etc?
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *R. Leigh Hennig
>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>> *To:* Robert DeVita 
>>> *Cc:* nanog@nanog.org
>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>
>>>
>>>
>>> Word around the campfire is that it’s a Cisco issue.
>>>
>>>
>>>
>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>> wrote:
>>>
>>>
>>>
>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Robert DeVita**​**​**​**​*
>>>
>>> *CEO and Founder*
>>>
>>> t: (469) 581-2160 <(469)%20581-2160>
>>>
>>>  |
>>>
>>> m: (469) 441-8864 <(469)%20441-8864>
>>>
>>> e: radev...@mejeticks.com
>>>
>>>  |
>>>
>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>
>>> a:
>>>
>>> 2323 N Akard Street
>>>
>>> ,
>>>
>>> Dallas
>>>
>>> ,
>>>
>>> 75201
>>>
>>> <https://www.linkedin.com/company/mejeticks/>
>>>
>>> <https://twitter.com/mejeticks>
>>>
>>> <https://www.facebook.com/mejeticks>
>>>
>>> <https://linktr.ee/mejeticks>
>>>
>>>
>>>
>>>
>>> The risk of trading futures and options can be substantial. All
>>> information, publications, and material used and distributed by Advance
>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>> independent research department as defined in CFTC Regulation 1.71.
>>> Information obtained from third-party sources is believed to be reliable,
>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>> is not necessarily indicative of future results.
>>>
>>
>
>

Re: Any info on AT Wireless Outage?

[Robert DeVita]

The $5 you got back should help alleviate any concerns going forward

Robert DeVita
Founder & CEO
Mejeticks
c. 469-441-8864
e. radev...@mejeticks.com

From: NANOG  on behalf of Mark 
Seiden 
Sent: Tuesday, February 27, 2024 8:41:49 PM
To: John Councilman 
Cc: nanog@nanog.org 
Subject: Re: Any info on AT Wireless Outage?

aside from the official pablum that was released about an “incorrect process 
used”
(which says exactly nothing) does anyone actually know anything accurate and
more specific about the root cause?

(and why it took 11 hours to recover?)

On Feb 22, 2024, at 11:15 AM, John Councilman  wrote:

From what I've read, they lost their database of SIM cards.  I could be wrong 
of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel 
mailto:d...@hetzel.org>> wrote:
As widespread as it seemed to be, it feels like it would be quite a trick if it 
were a single piece of hardware.  Firmware load that ended badly, I wonder?


On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
mailto:nanog@nanog.org>> wrote:

Do you have the ability to expand on this at all? Do you mean a hardware 
failure of some kind IE router, optitcs, etc?



From: NANOG 
mailto:advance-trading@nanog.org>>
 On Behalf Of R. Leigh Hennig
Sent: Thursday, February 22, 2024 8:17 AM
To: Robert DeVita mailto:radev...@mejeticks.com>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Any info on AT Wireless Outage?



Word around the campfire is that it’s a Cisco issue.



On Feb 22, 2024, at 8:03 AM, Robert DeVita 
mailto:radev...@mejeticks.com>> wrote:



Reports have it starting at 4:30 a.m.. SOS on all phones..







[X]

Robert DeVita

CEO and Founder

t: (469) 581-2160

 |

m: (469) 441-8864

e: radev...@mejeticks.com<mailto:radev...@mejeticks.com>

 |

w: mejeticks.com<https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

[X]<https://www.linkedin.com/company/mejeticks/>

[X]<https://twitter.com/mejeticks>

[X]<https://www.facebook.com/mejeticks>

[X]<https://linktr.ee/mejeticks>




The risk of trading futures and options can be substantial. All information, 
publications, and material used and distributed by Advance Trading Inc. shall 
be construed as a solicitation. ATI does not maintain an independent research 
department as defined in CFTC Regulation 1.71. Information obtained from 
third-party sources is believed to be reliable, but its accuracy is not 
guaranteed by Advance Trading Inc. Past performance is not necessarily 
indicative of future results.

Re: Any info on AT Wireless Outage?

[Paul Bradford]

missing an   permit ip any any?  classic



On Wed, Feb 28, 2024 at 10:56 AM  wrote:

> I read it as “someone pushed an ACL that wasn’t properly reviewed and it
> really screwed things up."
>
> On Feb 27, 2024, at 21:41, Mark Seiden  wrote:
>
> aside from the official pablum that was released about an “incorrect
> process used”
> (which says exactly nothing) does anyone actually know anything accurate
> and
> more specific about the root cause?
>
> (and why it took 11 hours to recover?)
>
> On Feb 22, 2024, at 11:15 AM, John Councilman 
> wrote:
>
> From what I've read, they lost their database of SIM cards.  I could be
> wrong of course.
>
> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>
>> As widespread as it seemed to be, it feels like it would be quite a trick
>> if it were a single piece of hardware.  Firmware load that ended badly, I
>> wonder?
>>
>>
>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>> wrote:
>>
>>> Do you have the ability to expand on this at all? Do you mean a hardware
>>> failure of some kind IE router, optitcs, etc?
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *R. Leigh Hennig
>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>> *To:* Robert DeVita 
>>> *Cc:* nanog@nanog.org
>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>
>>>
>>>
>>> Word around the campfire is that it’s a Cisco issue.
>>>
>>>
>>>
>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>> wrote:
>>>
>>>
>>>
>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Robert DeVita**​**​**​**​*
>>>
>>> *CEO and Founder*
>>>
>>> t: (469) 581-2160 <(469)%20581-2160>
>>>
>>>  |
>>>
>>> m: (469) 441-8864 <(469)%20441-8864>
>>>
>>> e: radev...@mejeticks.com
>>>
>>>  |
>>>
>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>
>>> a:
>>>
>>> 2323 N Akard Street
>>>
>>> ,
>>>
>>> Dallas
>>>
>>> ,
>>>
>>> 75201
>>>
>>> <https://www.linkedin.com/company/mejeticks/>
>>>
>>> <https://twitter.com/mejeticks>
>>>
>>> <https://www.facebook.com/mejeticks>
>>>
>>> <https://linktr.ee/mejeticks>
>>>
>>>
>>>
>>>
>>> The risk of trading futures and options can be substantial. All
>>> information, publications, and material used and distributed by Advance
>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>> independent research department as defined in CFTC Regulation 1.71.
>>> Information obtained from third-party sources is believed to be reliable,
>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>> is not necessarily indicative of future results.
>>>
>>
>
>

Re: Any info on AT Wireless Outage?

[joel]

I read it as “someone pushed an ACL that wasn’t properly reviewed and it really 
screwed things up."

> On Feb 27, 2024, at 21:41, Mark Seiden  wrote:
> 
> aside from the official pablum that was released about an “incorrect process 
> used”
> (which says exactly nothing) does anyone actually know anything accurate and 
> more specific about the root cause?
> 
> (and why it took 11 hours to recover?)
> 
>> On Feb 22, 2024, at 11:15 AM, John Councilman  wrote:
>> 
>> From what I've read, they lost their database of SIM cards.  I could be 
>> wrong of course.
>> 
>> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel > <mailto:d...@hetzel.org>> wrote:
>>> As widespread as it seemed to be, it feels like it would be quite a trick 
>>> if it were a single piece of hardware.  Firmware load that ended badly, I 
>>> wonder?
>>> 
>>> 
>>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG >> <mailto:nanog@nanog.org>> wrote:
>>>> Do you have the ability to expand on this at all? Do you mean a hardware 
>>>> failure of some kind IE router, optitcs, etc?
>>>> 
>>>>  
>>>> 
>>>> From: NANOG >>> <mailto:advance-trading@nanog.org>> On Behalf Of R. Leigh Hennig
>>>> Sent: Thursday, February 22, 2024 8:17 AM
>>>> To: Robert DeVita mailto:radev...@mejeticks.com>>
>>>> Cc: nanog@nanog.org <mailto:nanog@nanog.org>
>>>> Subject: Re: Any info on AT Wireless Outage?
>>>> 
>>>>  
>>>> 
>>>> Word around the campfire is that it’s a Cisco issue.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita >>> <mailto:radev...@mejeticks.com>> wrote:
>>>> 
>>>>  
>>>> 
>>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>> 
>>>>  
>>>> 
>>>>  
>>>> 
>>>>  
>>>> 
>>>> 
>>>> 
>>>> Robert DeVita
>>>> 
>>>> CEO and Founder
>>>> 
>>>> t: (469) 581-2160
>>>>  | 
>>>> 
>>>> m: (469) 441-8864 
>>>> e: radev...@mejeticks.com <mailto:radev...@mejeticks.com>  
>>>>  | 
>>>> 
>>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>> a: 
>>>> 
>>>> 2323 N Akard Street
>>>> 
>>>> , 
>>>> 
>>>> Dallas
>>>> 
>>>> , 
>>>> 
>>>> 75201
>>>> 
>>>>  <https://www.linkedin.com/company/mejeticks/> 
>>>>  <https://twitter.com/mejeticks>   
>>>>  <https://www.facebook.com/mejeticks>  
>>>>  <https://linktr.ee/mejeticks>
>>>>  
>>>> 
>>>> 
>>>> 
>>>> The risk of trading futures and options can be substantial. All 
>>>> information, publications, and material used and distributed by Advance 
>>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain 
>>>> an independent research department as defined in CFTC Regulation 1.71. 
>>>> Information obtained from third-party sources is believed to be reliable, 
>>>> but its accuracy is not guaranteed by Advance Trading Inc. Past 
>>>> performance is not necessarily indicative of future results.
> 

Re: Any info on AT Wireless Outage?

[Mark Seiden]

aside from the official pablum that was released about an “incorrect process 
used”
(which says exactly nothing) does anyone actually know anything accurate and 
more specific about the root cause?

(and why it took 11 hours to recover?)

> On Feb 22, 2024, at 11:15 AM, John Councilman  wrote:
> 
> From what I've read, they lost their database of SIM cards.  I could be wrong 
> of course.
> 
> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  <mailto:d...@hetzel.org>> wrote:
>> As widespread as it seemed to be, it feels like it would be quite a trick if 
>> it were a single piece of hardware.  Firmware load that ended badly, I 
>> wonder?
>> 
>> 
>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG > <mailto:nanog@nanog.org>> wrote:
>>> Do you have the ability to expand on this at all? Do you mean a hardware 
>>> failure of some kind IE router, optitcs, etc?
>>> 
>>>  
>>> 
>>> From: NANOG >> <mailto:advance-trading@nanog.org>> On Behalf Of R. Leigh Hennig
>>> Sent: Thursday, February 22, 2024 8:17 AM
>>> To: Robert DeVita mailto:radev...@mejeticks.com>>
>>> Cc: nanog@nanog.org <mailto:nanog@nanog.org>
>>> Subject: Re: Any info on AT Wireless Outage?
>>> 
>>>  
>>> 
>>> Word around the campfire is that it’s a Cisco issue.
>>> 
>>> 
>>> 
>>> 
>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita >> <mailto:radev...@mejeticks.com>> wrote:
>>> 
>>>  
>>> 
>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>> 
>>>  
>>> 
>>>  
>>> 
>>>  
>>> 
>>> 
>>> 
>>> Robert DeVita
>>> 
>>> CEO and Founder
>>> 
>>> t: (469) 581-2160 
>>>  | 
>>> 
>>> m: (469) 441-8864 
>>> e: radev...@mejeticks.com <mailto:radev...@mejeticks.com>   
>>>  | 
>>> 
>>> w: mejeticks.com <https://www.mejeticks.com/>
>>> a: 
>>> 
>>> 2323 N Akard Street
>>> 
>>> , 
>>> 
>>> Dallas
>>> 
>>> , 
>>> 
>>> 75201
>>> 
>>>  <https://www.linkedin.com/company/mejeticks/>  
>>>  <https://twitter.com/mejeticks>
>>>  <https://www.facebook.com/mejeticks>   
>>>  <https://linktr.ee/mejeticks>
>>>  
>>> 
>>> 
>>> 
>>> The risk of trading futures and options can be substantial. All 
>>> information, publications, and material used and distributed by Advance 
>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an 
>>> independent research department as defined in CFTC Regulation 1.71. 
>>> Information obtained from third-party sources is believed to be reliable, 
>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance 
>>> is not necessarily indicative of future results.

Re: Any info on AT Wireless Outage?

[Chris K]

Could it be someone fat fingered in preparation of this?

"Starting in March, FirstNet will be the first and only wireless network to 
provide America’s first responders with always-on priority and preemption 
across 5G, expanding to include all AT 5G commercial spectrum. "

https://www.firstnet.gov/newsroom/press-releases/firstnet-authority-att-announce-10-year-investment-transform-americas
FirstNet Authority, ATT Announce 10-Year Investment to Transform America’s 
Public Safety Broadband Network | First Responder Network 
Authority<https://www.firstnet.gov/newsroom/press-releases/firstnet-authority-att-announce-10-year-investment-transform-americas>
The First Responder Network Authority is launching the next phase of FirstNet 
through a series of strategic investments totaling more than $8 billion over 10 
years. The FirstNet Authority and its network partner, ATT, join to unveil 
the latest network investment of $6.3 billion, delivering full 5G capabilities 
on FirstNet, expanded mission-critical services, and enhanced coverage.
www.firstnet.gov


From: NANOG  on behalf of Brandon 
Jackson 
Sent: Thursday, February 22, 2024 8:06 PM
To: Patric Morgan 
Cc: NANOG 
Subject: Re: Any info on AT Wireless Outage?

If it's AT wireless Home broadband Service then it would be affected just the 
same.

It's still a cellular modem. It still has a SIM card. It's no different than a 
hotspot really, or just about any other cellular mobile device on the network.

If it's wireline service though, that had nothing to do with this and would 
have been some separate issue.

On Thu, Feb 22, 2024, 14:42 Patric Morgan 
mailto:patric.morg...@gmail.com>> wrote:
I have several friends in the Nashville area who did not have AT Home 
broadband service this morning, could be unrelated but would point to a 
different issue than the SIMs.

On Thu, Feb 22, 2024 at 1:16 PM John Councilman 
mailto:jcouncil...@gmail.com>> wrote:
From what I've read, they lost their database of SIM cards.  I could be wrong 
of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel 
mailto:d...@hetzel.org>> wrote:
As widespread as it seemed to be, it feels like it would be quite a trick if it 
were a single piece of hardware.  Firmware load that ended badly, I wonder?


On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
mailto:nanog@nanog.org>> wrote:

Do you have the ability to expand on this at all? Do you mean a hardware 
failure of some kind IE router, optitcs, etc?



From: NANOG 
mailto:advance-trading@nanog.org>>
 On Behalf Of R. Leigh Hennig
Sent: Thursday, February 22, 2024 8:17 AM
To: Robert DeVita mailto:radev...@mejeticks.com>>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Any info on AT Wireless Outage?



Word around the campfire is that it’s a Cisco issue.



On Feb 22, 2024, at 8:03 AM, Robert DeVita 
mailto:radev...@mejeticks.com>> wrote:



Reports have it starting at 4:30 a.m.. SOS on all phones..







[X]

Robert DeVita

CEO and Founder

t: (469) 581-2160

 |

m: (469) 441-8864

e: radev...@mejeticks.com<mailto:radev...@mejeticks.com>

 |

w: mejeticks.com<https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

[X]<https://www.linkedin.com/company/mejeticks/>

[X]<https://twitter.com/mejeticks>

[X]<https://www.facebook.com/mejeticks>

[X]<https://linktr.ee/mejeticks>




The risk of trading futures and options can be substantial. All information, 
publications, and material used and distributed by Advance Trading Inc. shall 
be construed as a solicitation. ATI does not maintain an independent research 
department as defined in CFTC Regulation 1.71. Information obtained from 
third-party sources is believed to be reliable, but its accuracy is not 
guaranteed by Advance Trading Inc. Past performance is not necessarily 
indicative of future results.

Re: Any info on AT Wireless Outage?

[Brandon Jackson]

If it's AT wireless Home broadband Service then it would be affected just
the same.

It's still a cellular modem. It still has a SIM card. It's no different
than a hotspot really, or just about any other cellular mobile device on
the network.

If it's wireline service though, that had nothing to do with this and would
have been some separate issue.

On Thu, Feb 22, 2024, 14:42 Patric Morgan  wrote:

> I have several friends in the Nashville area who did not have AT Home
> broadband service this morning, could be unrelated but would point to a
> different issue than the SIMs.
>
> On Thu, Feb 22, 2024 at 1:16 PM John Councilman 
> wrote:
>
>> From what I've read, they lost their database of SIM cards.  I could be
>> wrong of course.
>>
>> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>>
>>> As widespread as it seemed to be, it feels like it would be quite a
>>> trick if it were a single piece of hardware.  Firmware load that ended
>>> badly, I wonder?
>>>
>>>
>>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>>> wrote:
>>>
>>>> Do you have the ability to expand on this at all? Do you mean a
>>>> hardware failure of some kind IE router, optitcs, etc?
>>>>
>>>>
>>>>
>>>> *From:* NANOG  *On
>>>> Behalf Of *R. Leigh Hennig
>>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>>> *To:* Robert DeVita 
>>>> *Cc:* nanog@nanog.org
>>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>>
>>>>
>>>>
>>>> Word around the campfire is that it’s a Cisco issue.
>>>>
>>>>
>>>>
>>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>>> wrote:
>>>>
>>>>
>>>>
>>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Robert DeVita**​**​**​**​*
>>>>
>>>> *CEO and Founder*
>>>>
>>>> t: (469) 581-2160 <(469)%20581-2160>
>>>>
>>>>  |
>>>>
>>>> m: (469) 441-8864 <(469)%20441-8864>
>>>>
>>>> e: radev...@mejeticks.com
>>>>
>>>>  |
>>>>
>>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>>
>>>> a:
>>>>
>>>> 2323 N Akard Street
>>>>
>>>> ,
>>>>
>>>> Dallas
>>>>
>>>> ,
>>>>
>>>> 75201
>>>>
>>>> <https://www.linkedin.com/company/mejeticks/>
>>>>
>>>> <https://twitter.com/mejeticks>
>>>>
>>>> <https://www.facebook.com/mejeticks>
>>>>
>>>> <https://linktr.ee/mejeticks>
>>>>
>>>>
>>>>
>>>>
>>>> The risk of trading futures and options can be substantial. All
>>>> information, publications, and material used and distributed by Advance
>>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>>> independent research department as defined in CFTC Regulation 1.71.
>>>> Information obtained from third-party sources is believed to be reliable,
>>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>>> is not necessarily indicative of future results.
>>>>
>>>

Re: Any info on AT Wireless Outage?

[Patric Morgan]

I have several friends in the Nashville area who did not have AT Home
broadband service this morning, could be unrelated but would point to a
different issue than the SIMs.

On Thu, Feb 22, 2024 at 1:16 PM John Councilman 
wrote:

> From what I've read, they lost their database of SIM cards.  I could be
> wrong of course.
>
> On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:
>
>> As widespread as it seemed to be, it feels like it would be quite a trick
>> if it were a single piece of hardware.  Firmware load that ended badly, I
>> wonder?
>>
>>
>> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
>> wrote:
>>
>>> Do you have the ability to expand on this at all? Do you mean a hardware
>>> failure of some kind IE router, optitcs, etc?
>>>
>>>
>>>
>>> *From:* NANOG  *On
>>> Behalf Of *R. Leigh Hennig
>>> *Sent:* Thursday, February 22, 2024 8:17 AM
>>> *To:* Robert DeVita 
>>> *Cc:* nanog@nanog.org
>>> *Subject:* Re: Any info on AT Wireless Outage?
>>>
>>>
>>>
>>> Word around the campfire is that it’s a Cisco issue.
>>>
>>>
>>>
>>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>>> wrote:
>>>
>>>
>>>
>>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *Robert DeVita**​**​**​**​*
>>>
>>> *CEO and Founder*
>>>
>>> t: (469) 581-2160 <(469)%20581-2160>
>>>
>>>  |
>>>
>>> m: (469) 441-8864 <(469)%20441-8864>
>>>
>>> e: radev...@mejeticks.com
>>>
>>>  |
>>>
>>> w: mejeticks.com <https://www.mejeticks.com/>
>>>
>>> a:
>>>
>>> 2323 N Akard Street
>>>
>>> ,
>>>
>>> Dallas
>>>
>>> ,
>>>
>>> 75201
>>>
>>> <https://www.linkedin.com/company/mejeticks/>
>>>
>>> <https://twitter.com/mejeticks>
>>>
>>> <https://www.facebook.com/mejeticks>
>>>
>>> <https://linktr.ee/mejeticks>
>>>
>>>
>>>
>>>
>>> The risk of trading futures and options can be substantial. All
>>> information, publications, and material used and distributed by Advance
>>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>>> independent research department as defined in CFTC Regulation 1.71.
>>> Information obtained from third-party sources is believed to be reliable,
>>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>>> is not necessarily indicative of future results.
>>>
>>

Re: Any info on AT Wireless Outage?

[John Councilman]

>From what I've read, they lost their database of SIM cards.  I could be
wrong of course.

On Thu, Feb 22, 2024 at 2:02 PM Dorn Hetzel  wrote:

> As widespread as it seemed to be, it feels like it would be quite a trick
> if it were a single piece of hardware.  Firmware load that ended badly, I
> wonder?
>
>
> On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
> wrote:
>
>> Do you have the ability to expand on this at all? Do you mean a hardware
>> failure of some kind IE router, optitcs, etc?
>>
>>
>>
>> *From:* NANOG  *On
>> Behalf Of *R. Leigh Hennig
>> *Sent:* Thursday, February 22, 2024 8:17 AM
>> *To:* Robert DeVita 
>> *Cc:* nanog@nanog.org
>> *Subject:* Re: Any info on AT Wireless Outage?
>>
>>
>>
>> Word around the campfire is that it’s a Cisco issue.
>>
>>
>>
>> On Feb 22, 2024, at 8:03 AM, Robert DeVita 
>> wrote:
>>
>>
>>
>> Reports have it starting at 4:30 a.m.. SOS on all phones..
>>
>>
>>
>>
>>
>>
>>
>> *Robert DeVita**​**​**​**​*
>>
>> *CEO and Founder*
>>
>> t: (469) 581-2160 <(469)%20581-2160>
>>
>>  |
>>
>> m: (469) 441-8864 <(469)%20441-8864>
>>
>> e: radev...@mejeticks.com
>>
>>  |
>>
>> w: mejeticks.com <https://www.mejeticks.com/>
>>
>> a:
>>
>> 2323 N Akard Street
>>
>> ,
>>
>> Dallas
>>
>> ,
>>
>> 75201
>>
>> <https://www.linkedin.com/company/mejeticks/>
>>
>> <https://twitter.com/mejeticks>
>>
>> <https://www.facebook.com/mejeticks>
>>
>> <https://linktr.ee/mejeticks>
>>
>>
>>
>>
>> The risk of trading futures and options can be substantial. All
>> information, publications, and material used and distributed by Advance
>> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
>> independent research department as defined in CFTC Regulation 1.71.
>> Information obtained from third-party sources is believed to be reliable,
>> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
>> is not necessarily indicative of future results.
>>
>

Re: Any info on AT Wireless Outage?

[Dorn Hetzel]

As widespread as it seemed to be, it feels like it would be quite a trick
if it were a single piece of hardware.  Firmware load that ended badly, I
wonder?


On Thu, Feb 22, 2024 at 1:51 PM Leato, Gary via NANOG 
wrote:

> Do you have the ability to expand on this at all? Do you mean a hardware
> failure of some kind IE router, optitcs, etc?
>
>
>
> *From:* NANOG  *On
> Behalf Of *R. Leigh Hennig
> *Sent:* Thursday, February 22, 2024 8:17 AM
> *To:* Robert DeVita 
> *Cc:* nanog@nanog.org
> *Subject:* Re: Any info on AT Wireless Outage?
>
>
>
> Word around the campfire is that it’s a Cisco issue.
>
>
>
> On Feb 22, 2024, at 8:03 AM, Robert DeVita  wrote:
>
>
>
> Reports have it starting at 4:30 a.m.. SOS on all phones..
>
>
>
>
>
>
>
> *Robert DeVita**​**​**​**​*
>
> *CEO and Founder*
>
> t: (469) 581-2160 <(469)%20581-2160>
>
>  |
>
> m: (469) 441-8864 <(469)%20441-8864>
>
> e: radev...@mejeticks.com
>
>  |
>
> w: mejeticks.com <https://www.mejeticks.com/>
>
> a:
>
> 2323 N Akard Street
>
> ,
>
> Dallas
>
> ,
>
> 75201
>
> <https://www.linkedin.com/company/mejeticks/>
>
> <https://twitter.com/mejeticks>
>
> <https://www.facebook.com/mejeticks>
>
> <https://linktr.ee/mejeticks>
>
>
>
>
> The risk of trading futures and options can be substantial. All
> information, publications, and material used and distributed by Advance
> Trading Inc. shall be construed as a solicitation. ATI does not maintain an
> independent research department as defined in CFTC Regulation 1.71.
> Information obtained from third-party sources is believed to be reliable,
> but its accuracy is not guaranteed by Advance Trading Inc. Past performance
> is not necessarily indicative of future results.
>

RE: Any info on AT Wireless Outage?

[Leato, Gary via NANOG]

Do you have the ability to expand on this at all? Do you mean a hardware 
failure of some kind IE router, optitcs, etc?

From: NANOG  On Behalf Of 
R. Leigh Hennig
Sent: Thursday, February 22, 2024 8:17 AM
To: Robert DeVita 
Cc: nanog@nanog.org
Subject: Re: Any info on AT Wireless Outage?

Word around the campfire is that it’s a Cisco issue.


On Feb 22, 2024, at 8:03 AM, Robert DeVita 
mailto:radev...@mejeticks.com>> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..



[cid:image001.jpg@01DA655D.39E82510]
Robert DeVita
CEO and Founder
t: (469) 581-2160
 |
m: (469) 441-8864
e: radev...@mejeticks.com<mailto:radev...@mejeticks.com>
 |
w: mejeticks.com<https://www.mejeticks.com/>
a:
2323 N Akard Street
,
Dallas
,
75201
[cid:image002.png@01DA655D.39E82510]<https://www.linkedin.com/company/mejeticks/>
[cid:image003.png@01DA655D.39E82510]<https://twitter.com/mejeticks>
[cid:image004.png@01DA655D.39E82510]<https://www.facebook.com/mejeticks>
[cid:image005.png@01DA655D.39E82510]<https://linktr.ee/mejeticks>



The risk of trading futures and options can be substantial. All information, 
publications, and material used and distributed by Advance Trading Inc. shall 
be construed as a solicitation. ATI does not maintain an independent research 
department as defined in CFTC Regulation 1.71. Information obtained from 
third-party sources is believed to be reliable, but its accuracy is not 
guaranteed by Advance Trading Inc. Past performance is not necessarily 
indicative of future results.

Re: Any info on AT Wireless Outage?

[Tim Burke]

No issues on FirstNet here in Houston metro, but AT consumer core still 
appears to be non-functional.



From: NANOG on behalf of Ryan A. Krenzischek via NANOG
Sent: Thursday, February 22, 2024 8:03 AM
To: Ray Orsini
Cc: nanog@nanog.org
Subject: Re: Any info on AT Wireless Outage?

The same as well for FirstNet but am now able to make calls.  Others who are on 
AT are unable to receive or send calls.  Enabling wifi calling on a regular 
AT phone (android) results in a 502 bad gateway error message.

Ryan

On Feb 22, 2024, at 08:11, Ray Orsini via NANOG  wrote:


We're affected as well. Unable to dial out. I haven't found any official 
statement though.

RE: Any info on AT Wireless Outage?

[Christopher Trudeau]

I’ve got about 500 Cradlepoints with AT throughout the US. I saw about 20 of 
them go down, all different locations, all around the same time (around 5am 
CST).
FWIW these are across 3 different AT OU’s… AT IoT (with a FirstNet APN), 
AT 4G Business Broadband, and straight AT Business/Retail 4G.

The response from my account manager is slightly comical:
"Some of our customers are experiencing wireless service interruptions this 
morning. We are working urgently to restore service to them. We encourage the 
use of Wi-Fi calling until service is restored."

Let me get right on activating Wifi Calling on my Cradlepoints…

Chris Trudeau
Network Engineer, Nitel

From: NANOG  On Behalf Of R. 
Leigh Hennig
Sent: Thursday, February 22, 2024 8:17 AM
To: Robert DeVita 
Cc: nanog@nanog.org
Subject: Re: Any info on AT Wireless Outage?

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.
Word around the campfire is that it’s a Cisco issue.


On Feb 22, 2024, at 8:03 AM, Robert DeVita 
mailto:radev...@mejeticks.com>> wrote:

Reports have it starting at 4:30 a.m.. SOS on all phones..



[cid:image001.jpg@01DA655D.39E82510]
Robert DeVita
CEO and Founder
t: (469) 581-2160
 |
m: (469) 441-8864
e: radev...@mejeticks.com<mailto:radev...@mejeticks.com>
 |
w: mejeticks.com<https://www.mejeticks.com/>
a:
2323 N Akard Street
,
Dallas
,
75201
[cid:image002.png@01DA655D.39E82510]<https://www.linkedin.com/company/mejeticks/>
[cid:image003.png@01DA655D.39E82510]<https://twitter.com/mejeticks>
[cid:image004.png@01DA655D.39E82510]<https://www.facebook.com/mejeticks>
[cid:image005.png@01DA655D.39E82510]<https://linktr.ee/mejeticks>




CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or 
previous e-mail messages attached to it may contain confidential information 
that is legally privileged. If you are not the intended recipient, or a person 
responsible for delivering it to the intended recipient, you are hereby 
notified that any disclosure, copying, distribution or use of any of the 
information contained in or attached to this transmission is STRICTLY 
PROHIBITED. If you have received this transmission in error please notify the 
sender immediately by replying to this e-mail. You must destroy the original 
transmission and its attachments without reading or saving in any manner.
Thank you.

Re: Any info on AT Wireless Outage?

[R. Leigh Hennig]

Word around the campfire is that it’s a Cisco issue.

> On Feb 22, 2024, at 8:03 AM, Robert DeVita  wrote:
> 
> Reports have it starting at 4:30 a.m.. SOS on all phones..
>  
>  
>  
> 
> Robert DeVita
> CEO and Founder
> t: (469) 581-2160   
>  | 
> m: (469) 441-8864 
> e: radev...@mejeticks.com  
>  | 
> w: mejeticks.com 
> a: 
> 2323 N Akard Street
> , 
> Dallas
> , 
> 75201
>  
>    
>   
>  

Re: Any info on AT Wireless Outage?

[Ryan A. Krenzischek via NANOG]

The same as well for FirstNet but am now able to make calls.  Others who are on 
AT are unable to receive or send calls.  Enabling wifi calling on a regular 
AT phone (android) results in a 502 bad gateway error message.  

Ryan

> On Feb 22, 2024, at 08:11, Ray Orsini via NANOG  wrote:
> 
> 
> We're affected as well. Unable to dial out. I haven't found any official 
> statement though.
> 


smime.p7s
Description: S/MIME cryptographic signature

Re: Any info on AT Wireless Outage?

[Dorn Hetzel]

Appears to have affected at least some FirstNet SIMS as well.

On Thu, Feb 22, 2024, 08:10 Ray Orsini via NANOG  wrote:

> We're affected as well. Unable to dial out. I haven't found any official
> statement though.
>
> [image: OIT Website] <https://www.oit.co/>
> Ray Orsini
> Chief Executive Officer
> OIT, LLC
>  305.967.6756 x1009 <305.967.6756%20x1009>  |   305.571.6272
>  r...@oit.co  |  [image: https://www.oit.co] <https://www.oit.co/>
>  www.oit.co <https://www.oit.co/>
>  oit.co/ray
> [image: Facebook] <https://go.oit.co/facebook>
> [image: LinkedIn] <https://go.oit.co/linkedin>
> [image: Twitter] <https://go.oit.co/twitter>
> [image: YouTube] <https://go.oit.co/youtube>
> How are we doing? We'd love to hear your feedback.
> https://go.oit.co/review <https://go.oit.co/review>
> --
> *From:* NANOG  on behalf of Robert
> DeVita 
> *Sent:* Thursday, February 22, 2024 8:03 AM
> *To:* nanog@nanog.org 
> *Subject:* Any info on AT Wireless Outage?
>
>
> Reports have it starting at 4:30 a.m.. SOS on all phones..
>
>
>
>
>
>
>
> *Robert DeVita**​**​**​**​*
>
> *CEO and Founder*
>
> t: (469) 581-2160 <(469)%20581-2160>
>
>  |
>
> m: (469) 441-8864 <(469)%20441-8864>
>
> e: radev...@mejeticks.com
>
>  |
>
> w: mejeticks.com <https://www.mejeticks.com/>
>
> a:
>
> 2323 N Akard Street
>
> ,
>
> Dallas
>
> ,
>
> 75201
>
> <https://www.linkedin.com/company/mejeticks/>
>
> <https://twitter.com/mejeticks>
>
> <https://www.facebook.com/mejeticks>
>
> <https://linktr.ee/mejeticks>
>
>
>

Re: Any info on AT Wireless Outage?

[Ray Orsini via NANOG]

We're affected as well. Unable to dial out. I haven't found any official 
statement though.

[OIT Website]<https://www.oit.co/>
Ray Orsini
Chief Executive Officer
OIT, LLC
[cid:c8519a01-d6f7-40e7-8376-cceab8183f23]   305.967.6756 
x1009|  
[cid:4471b74c-1422-4b97-b14d-39a4abda62c8]   305.571.6272
[cid:0c583814-5618-4d1c-adef-47d8ae21bb43]   
r...@oit.co<mailto:r...@oit.co>   |  [https://www.oit.co] 
<https://www.oit.co/>   www.oit.co<https://www.oit.co/>
[cid:1c14adba-384b-4917-bca8-3410f5298a27]
 oit.co/ray
[Facebook]<https://go.oit.co/facebook>
[LinkedIn]<https://go.oit.co/linkedin>
[Twitter]<https://go.oit.co/twitter>
[YouTube]<https://go.oit.co/youtube>
How are we doing? We'd love to hear your feedback. 
https://go.oit.co/review<https://go.oit.co/review>

From: NANOG  on behalf of Robert DeVita 

Sent: Thursday, February 22, 2024 8:03 AM
To: nanog@nanog.org 
Subject: Any info on AT Wireless Outage?


Reports have it starting at 4:30 a.m.. SOS on all phones..







[cid:image001.jpg@01DA655D.39E82510]

Robert DeVita

CEO and Founder

t: (469) 581-2160

 |

m: (469) 441-8864

e: radev...@mejeticks.com<mailto:radev...@mejeticks.com>

 |

w: mejeticks.com<https://www.mejeticks.com/>

a:

2323 N Akard Street

,

Dallas

,

75201

[cid:image002.png@01DA655D.39E82510]<https://www.linkedin.com/company/mejeticks/>

[cid:image003.png@01DA655D.39E82510]<https://twitter.com/mejeticks>

[cid:image004.png@01DA655D.39E82510]<https://www.facebook.com/mejeticks>

[cid:image005.png@01DA655D.39E82510]<https://linktr.ee/mejeticks>

Any info on AT Wireless Outage?

[Robert DeVita]

Reports have it starting at 4:30 a.m.. SOS on all phones..



[cid:image001.jpg@01DA655D.39E82510]
Robert DeVita
CEO and Founder
t: (469) 581-2160
 |
m: (469) 441-8864
e: radev...@mejeticks.com
 |
w: mejeticks.com
a:
2323 N Akard Street
,
Dallas
,
75201
[cid:image002.png@01DA655D.39E82510]
[cid:image003.png@01DA655D.39E82510]
[cid:image004.png@01DA655D.39E82510]
[cid:image005.png@01DA655D.39E82510]

Re: Canadian regulator CRTC hires private company to investigate 2022 Rogers outage

[Mark Prosser]

The Canadian Radio-television and Telecommunications Commission (CRTC) 
confirmed in an email it hired engineering consultant Xona Partners in 
May to provide a report on the Rogers network and "help inform what 
further regulatory action is needed."


I think we all know the "regulatory action" we /want/. I just don't it's 
the action that CRTC will deliver -- if any.


--
Mark Prosser
// E:m...@zealnetworks.ca
// W:https://zealnetworks.ca

Canadian regulator CRTC hires private company to investigate 2022 Rogers outage

[Sean Donelan]

CRTC hires private company to investigate 2022 Rogers outage
Critics say regulator being too secretive about probe, too slow to force 
more transparency from big telecoms


https://www.cbc.ca/news/politics/crtc-rogers-outage-investigation-1.6963052


CBC News has learned Canada's telecommunications regulator has hired a 
private consulting firm to investigate the massive Rogers outage last 
summer that left more than 10 million customers without cellphone and 
internet access.


The Canadian Radio-television and Telecommunications Commission (CRTC) 
confirmed in an email it hired engineering consultant Xona Partners in May 
to provide a report on the Rogers network and "help inform what further 
regulatory action is needed."

Re: American Internet Services (AXSCN) outage?

[Antonio Querubin]

On Wed, 26 Jul 2023, Grant Taylor via NANOG wrote:


Subject: Re: American Internet Services (AXSCN) outage?

On 7/26/23 2:51 PM, Antonio Querubin wrote:

 Wondering if anyone can shed some light on American Internet Services
 going offline.


I saw a short (< 5 minute) blip with them earlier today.  I don't remember 
what time it was.


Are you still seeing problems?


 Their nameservers (ns1/2.axscn.net) aren't even reachable.


I question the veracity of that.

ns1.axscn.net responds to DNS queries just fine.  Though it doesn't respond 
to pings.  --  Maybe they filter ICMP.


ns2.axscn.net doesn't respond to DNS queries nor ping.

Can ~> will you share any more details about the problem that you're having 
that has prompted you to try talking to ns1.axscn.net / ns2.axscn.nett? 
Maybe someone can shed some light if you provide more details.


faxagent.com's portal addresses weren't resolving for over 53 hours 
beginning Monday morning.  Neither axscn.net NS were responding to queries 
nor were they pingable during that time.  Whatever the problem was 
eventually got fixed.


Antonio Querubin
e-mail:  t...@lavanauts.org

Re: American Internet Services (AXSCN) outage?

[Grant Taylor via NANOG]

On 7/26/23 2:51 PM, Antonio Querubin wrote:
Wondering if anyone can shed some light on American Internet Services 
going offline.


I saw a short (< 5 minute) blip with them earlier today.  I don't 
remember what time it was.


Are you still seeing problems?


Their nameservers (ns1/2.axscn.net) aren't even reachable.


I question the veracity of that.

ns1.axscn.net responds to DNS queries just fine.  Though it doesn't 
respond to pings.  --  Maybe they filter ICMP.


ns2.axscn.net doesn't respond to DNS queries nor ping.

Can ~> will you share any more details about the problem that you're 
having that has prompted you to try talking to ns1.axscn.net / 
ns2.axscn.nett?  Maybe someone can shed some light if you provide more 
details.




Grant. . . .

Re: American Internet Services (AXSCN) outage?

[Norman Jester]

On Jul 26, 2023, at 12:53 PM, Antonio Querubin  wrote:
> 
> Wondering if anyone can shed some light on American Internet Services going 
> offline.  Their nameservers (ns1/2.axscn.net) aren't even reachable.

If it’s the same American Internet Servcies that later became NFINIT, they were 
aquired by Lightedge.
Possibly that has something to do with it.

American Internet Services (AXSCN) outage?

[Antonio Querubin]

Wondering if anyone can shed some light on American Internet Services going 
offline.  Their nameservers (ns1/2.axscn.net) aren't even reachable.


Antonio Querubin
e-mail:  t...@lavanauts.org

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Justin Streiner]

Sounds like either the National Driver Register or NHTSA is single-homed to
Verizon, or the state DMVs each have a WAN circuit of some sort through
Verizon to where the National Driver Register system physically lives.  If
it's the latter, it sounds like a job that could be handled much more
effectively using site-to-site VPN tunnels, but I understand the realities
of being stuck in a long-term contract, policy mandates that dictate a
physical connection, or things along those lines.  Government agencies can
and sometimes do get funding for infrastructure upgrades and resiliency -
at least if they budget for it...

I'm sure the backstory here is both mundane and interesting.

Thank you
jms

On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:

> Verizon network maintenance will impact access to the “National Driver
> Register,” a system that motor vehicle offices around the country need to
> check before handing out a license.
>
> All 50 states and D.C. participate in the National Driver Register, a
> database maintained by the National Highway Traffic Safety Administration.
> The register contains information about drivers who have had their driving
> privileges revoked, suspended or denied due to serious traffic violations,
> such as driving under the influence of alcohol or drugs, reckless driving
> or excessive speeding.
>
>
> The scheduled maintenance should be finished by Monday, in case you needed
> to update your driver's license or planned to do some reckless driving
> this weekend.
>

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Brian Knight via NANOG]

It seems to say more about fluctuating funding and IT management.I seem to recall an issue with the FAA’s NOTAM / TFR database a few weeks back, one that grounded all flights one fine morning. Wasn’t network-related, but the articles I read about the application’s architecture and fault-tolerance make it seem… lightweight. :) And I would guess that outage was far more impactful than this one is likely to be.-BrianOn Mar 1, 2023, at 9:45 AM, Jason Leschnik  wrote:Says a lot about the architecture of the application and redundancy. I'd love to know what the failover looks like in a worst-case scenario  On Sun, 26 Feb 2023 at 10:51, Aaron de Bruyn via NANOG <nanog@nanog.org> wrote:If we have downtime, we lose revenue, customers, sleep, etc...If the government does it, what are you going to do?  Get your license somewhere else?-AOn Sat Feb 25, 2023, 11:39 PM GMT, Christopher Morrow wrote:On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan <s...@donelan.com> wrote:Verizon network maintenance will impact access to the “National DriverRegister,” a system that motor vehicle offices around the country need tocheck before handing out a license.Wait, what year is it?how is a network maintenance on what seems like a fairly critical system goingto cause a total outage of said system?I think we time traveled back to 1990 here...All 50 states and D.C. participate in the National Driver Register, adatabase maintained by the National Highway Traffic Safety Administration.The register contains information about drivers who have had their drivingprivileges revoked, suspended or denied due to serious traffic violations,such as driving under the influence of alcohol or drugs, reckless drivingor excessive speeding.The scheduled maintenance should be finished by Monday, in case you neededto update your driver's license or planned to do some reckless drivingthis weekend.

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Jason Leschnik]

Says a lot about the architecture of the application and redundancy. I'd
love to know what the failover looks like in a worst-case scenario

On Sun, 26 Feb 2023 at 10:51, Aaron de Bruyn via NANOG 
wrote:

> If we have downtime, we lose revenue, customers, sleep, etc...
>
> If the government does it, what are you going to do? Get your license
> somewhere else?
>
> -A
>
> On Sat Feb 25, 2023, 11:39 PM GMT, Christopher Morrow
>  wrote:
>
> On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:
>
>
> Verizon network maintenance will impact access to the “National Driver
> Register,” a system that motor vehicle offices around the country need to
> check before handing out a license.
>
>
> Wait, what year is it?
> how is a network maintenance on what seems like a fairly critical system
> going
> to cause a total outage of said system?
>
> I think we time traveled back to 1990 here...
>
>
> All 50 states and D.C. participate in the National Driver Register, a
> database maintained by the National Highway Traffic Safety Administration.
> The register contains information about drivers who have had their driving
> privileges revoked, suspended or denied due to serious traffic violations,
> such as driving under the influence of alcohol or drugs, reckless driving
> or excessive speeding.
>
>
> The scheduled maintenance should be finished by Monday, in case you needed
> to update your driver's license or planned to do some reckless driving
> this weekend.
>
>

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Henry Yen]

On Sat, Feb 25, 2023 at 06:11:55AM -0500, Sean Donelan wrote:
> [...] or planned to do some reckless driving this weekend.

wasn't planning to before, but surely will do so now, thanks!

-- 
Henry Yen   Aegis Information Systems, Inc.
Senior Systems Programmer   Hicksville, New York

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Aaron de Bruyn via NANOG]

If we have downtime, we lose revenue, customers, sleep, etc...

If the government does it, what are you going to do? Get your license somewhere 
else?

-A

On Sat Feb 25, 2023, 11:39 PM GMT, Christopher Morrow 
<mailto:morrowc.li...@gmail.com> wrote:
> On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:
>>
>> Verizon network maintenance will impact access to the “National Driver
>> Register,” a system that motor vehicle offices around the country need to
>> check before handing out a license.
>
> Wait, what year is it?
> how is a network maintenance on what seems like a fairly critical system going
> to cause a total outage of said system?
>
> I think we time traveled back to 1990 here...
>>
>> All 50 states and D.C. participate in the National Driver Register, a
>> database maintained by the National Highway Traffic Safety Administration.
>> The register contains information about drivers who have had their driving
>> privileges revoked, suspended or denied due to serious traffic violations,
>> such as driving under the influence of alcohol or drugs, reckless driving
>> or excessive speeding.
>>
>>
>> The scheduled maintenance should be finished by Monday, in case you needed
>> to update your driver's license or planned to do some reckless driving
>> this weekend.

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Christopher Morrow]

On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:
>
> Verizon network maintenance will impact access to the “National Driver
> Register,” a system that motor vehicle offices around the country need to
> check before handing out a license.

Wait, what year is it?
how is a network maintenance on what seems like a fairly critical system going
to cause a total outage of said system?

I think we time traveled back to 1990 here...

>
> All 50 states and D.C. participate in the National Driver Register, a
> database maintained by the National Highway Traffic Safety Administration.
> The register contains information about drivers who have had their driving
> privileges revoked, suspended or denied due to serious traffic violations,
> such as driving under the influence of alcohol or drugs, reckless driving
> or excessive speeding.
>
>
> The scheduled maintenance should be finished by Monday, in case you needed
> to update your driver's license or planned to do some reckless driving
> this weekend.

Scheduled outage -- Nationwide no driver license updates this weekend

[Sean Donelan]

Verizon network maintenance will impact access to the “National Driver 
Register,” a system that motor vehicle offices around the country need to 
check before handing out a license.


All 50 states and D.C. participate in the National Driver Register, a 
database maintained by the National Highway Traffic Safety Administration.
The register contains information about drivers who have had their driving 
privileges revoked, suspended or denied due to serious traffic violations, 
such as driving under the influence of alcohol or drugs, reckless driving 
or excessive speeding.



The scheduled maintenance should be finished by Monday, in case you needed 
to update your driver's license or planned to do some reckless driving 
this weekend.

Re: txt.att.net outage?

[John Levine]

It appears that Simmons, Jay via NANOG  said:
>-=-=-=-=-=-
>This may be the issue

Sorry, but no.

>Here are some details on this Government protocol implemented by all Telecom 
>Carriers.
>
>Why it is being done? To support FCC mandate for STIR/SHAKEN, an industry set 
>of rules designed to authenticate
>and validate CallerID information associated with phone calls using digital 
>signatures.
>
>SHAKEN/STIR ...

STIR/SHAKEN only affects voice calls.  It has nothing to do with SMS.

As several other people have said, if you're sending safety critical SMS 
messages, use
a real SMS service, not a carrier's courtesy low volume e-mail gateway.

SMS services are not free, but they are not expensive, typically about 1/2 cent 
per message.

R's,
John

RE: txt.att.net outage?

[Simmons, Jay via NANOG]

This may be the issue

Here are some details on this Government protocol implemented by all Telecom 
Carriers.

Why it is being done? To support FCC mandate for STIR/SHAKEN, an industry set 
of rules designed to authenticate and validate CallerID information associated 
with phone calls using digital signatures.

SHAKEN/STIR is a framework of interconnected standards. SHAKEN/STIR are 
acronyms for Signature-based Handling of Asserted Information Using tokens 
(SHAKEN) and the Secure Telephone Identity Revisited (STIR) standards.

What is the benefit to the clients being impacted (what do they get as a 
result)? Provide a means to mitigate/eliminate illegally spoofed calls from 
RoboCallers.






From: NANOG  On Behalf Of 
Tim Burke
Sent: Friday, January 20, 2023 8:21 AM
To: Dan Walters ; nanog@nanog.org list 
Subject: Re: txt.att.net outage?

EXTERNAL EMAIL
FWIW, AT does not suggest using the txt.att.net<http://txt.att.net> email for 
anything critical. In a previous role I handled public safety CAD, and AT 
(and Verizon, if I remember correctly) made it very clear that if you want to 
be able to send anything other than a miniscule volume of texts through the 
email-to-SMS gateway, you would need to pay for their enterprise messaging 
service.

If the public safety agency/agencies in question subscribe to the FirstNet 
service, they should be able to reach out to their principal consultant 
(account team) and get a good resolution.

V/r
Tim


From: NANOG 
mailto:nanog-bounces+tim=mid@nanog.org>>
 on behalf of Dan Walters via NANOG mailto:nanog@nanog.org>>
Sent: Thursday, January 19, 2023 10:09 PM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: txt.att.net<http://txt.att.net> outage?


Know this is a longshot, any chance anyone from the 
txt.att.net<http://txt.att.net> domain might be able to help us with what we 
believe is a blacklist block or possibly an outage?

We deal with 911 cad dispatching and is affecting first responders so looking 
to see if there is a faster way to resolution.



Thanks, in advance.





Daniel Walters

Sr. Systems Admin

Phone: 866.421.2374 ext.6213

daniel.walt...@omnigo.com<mailto:daniel.walt...@omnigo.com>

[cid:image001.png@01D92C52.BEDD12C0]

[cid:image002.png@01D92C52.BEDD12C0][cid:image003.png@01D92C52.BEDD12C0] 
[cid:image004.png@01D92C52.BEDD12C0]

Re: txt.att.net outage?

[Martin Hannigan]

On Fri, Jan 20, 2023 at 9:15 AM William Herrin  wrote:

> On Thu, Jan 19, 2023 at 8:09 PM Dan Walters via NANOG 
> wrote:
> > Know this is a longshot, any chance anyone from the txt.att.net domain
> might be able to help us with what we believe is a blacklist block or
> possibly an outage?
> > We deal with 911 cad dispatching and is affecting first responders so
> looking to see if there is a faster way to resolution.
>
> Hi Dan,
>
> As I understand it, txt.att.net is a low-volume courtesy service not
> intended for important communications. A paid service like Twilio can
> handle production-grade SMS delivery.
>
>
>
I think ChatGPT agrees with you:

txt.att.net is a domain owned by AT, a large American telecommunications
company. It is typically used to send and receive text messages, and it is
generally considered a reliable resource for this purpose. However, as with
any online service, it is possible for issues to arise, such as delays in
delivery or errors in sending messages. It is also worth noting that
txt.att.net is a free service, so it might not have the same level of
security or reliability as a paid service. If you have any specific
concerns or experience issues with txt.att.net, it's best to contact AT
customer service for assistance.

;-)

Warm regards,

-M<

Re: txt.att.net outage?

[Aaron de Bruyn via NANOG]

txt.att.net <http://txt.att.net> is returning MX records and those machines 
don't have port 444 open...

Wouldn't you want to be sending something like a SNPP message instead? It's a 
much less convoluted delivery process and is almost real-time (no queuing).

I guess it's been a decade or so since I've dealt with emergency services and 
paging...is SNPP even a thing anymore?

I looked at some old code I wrote 
(https://github.com/darkpixel/snppsend/blob/master/more-providers 
<https://github.com/darkpixel/snppsend/blob/master/more-providers>), and it 
doesn't look like snpp.attws.net <http://snpp.attws.net> exists.

-A

On Fri Jan 20, 2023, 02:12 PM GMT, William Herrin <mailto:b...@herrin.us> wrote:
> On Thu, Jan 19, 2023 at 8:09 PM Dan Walters via NANOG  wrote:
>> Know this is a longshot, any chance anyone from the txt.att.net domain might 
>> be able to help us with what we believe is a blacklist block or possibly an 
>> outage?
>> We deal with 911 cad dispatching and is affecting first responders so 
>> looking to see if there is a faster way to resolution.
>
> Hi Dan,
>
> As I understand it, txt.att.net is a low-volume courtesy service not
> intended for important communications. A paid service like Twilio can
> handle production-grade SMS delivery.
>
> Regards,
> Bill Herrin
>
> --
> For hire. https://bill.herrin.us/resume/

Re: txt.att.net outage?

[Tim Burke]

FWIW, AT does not suggest using the txt.att.net email for anything critical. 
In a previous role I handled public safety CAD, and AT (and Verizon, if I 
remember correctly) made it very clear that if you want to be able to send 
anything other than a miniscule volume of texts through the email-to-SMS 
gateway, you would need to pay for their enterprise messaging service.

If the public safety agency/agencies in question subscribe to the FirstNet 
service, they should be able to reach out to their principal consultant 
(account team) and get a good resolution.

V/r
Tim


From: NANOG  on behalf of Dan Walters via 
NANOG 
Sent: Thursday, January 19, 2023 10:09 PM
To: nanog@nanog.org
Subject: txt.att.net outage?


Know this is a longshot, any chance anyone from the txt.att.net domain might be 
able to help us with what we believe is a blacklist block or possibly an outage?

We deal with 911 cad dispatching and is affecting first responders so looking 
to see if there is a faster way to resolution.



Thanks, in advance.





Daniel Walters

Sr. Systems Admin

Phone: 866.421.2374 ext.6213

daniel.walt...@omnigo.com<mailto:daniel.walt...@omnigo.com>

[cid:image001.png@01D92C52.BEDD12C0]

[cid:image002.png@01D92C52.BEDD12C0][cid:image003.png@01D92C52.BEDD12C0] 
[cid:image004.png@01D92C52.BEDD12C0]

Re: txt.att.net outage?

[William Herrin]

On Thu, Jan 19, 2023 at 8:09 PM Dan Walters via NANOG  wrote:
> Know this is a longshot, any chance anyone from the txt.att.net domain might 
> be able to help us with what we believe is a blacklist block or possibly an 
> outage?
> We deal with 911 cad dispatching and is affecting first responders so looking 
> to see if there is a faster way to resolution.

Hi Dan,

As I understand it, txt.att.net is a low-volume courtesy service not
intended for important communications. A paid service like Twilio can
handle production-grade SMS delivery.

Regards,
Bill Herrin

-- 
For hire. https://bill.herrin.us/resume/

txt.att.net outage?

[Dan Walters via NANOG]

Know this is a longshot, any chance anyone from the txt.att.net domain might be 
able to help us with what we believe is a blacklist block or possibly an outage?
We deal with 911 cad dispatching and is affecting first responders so looking 
to see if there is a faster way to resolution.

Thanks, in advance.


Daniel Walters
Sr. Systems Admin
Phone: 866.421.2374 ext.6213
daniel.walt...@omnigo.com<mailto:daniel.walt...@omnigo.com>
[cid:image001.png@01D92C52.BEDD12C0]
[cid:image002.png@01D92C52.BEDD12C0][cid:image003.png@01D92C52.BEDD12C0] 
[cid:image004.png@01D92C52.BEDD12C0]

Cloudflare/Internet2 Peering Outage?

[Norman Elton]

Any Cloudflare contacts lurking?

We observed a spike in DNS timeouts beginning around 7:20 PM EST. We
eventually determined that DNS queries to Cloudflare-managed DNS
servers, traversing our Internet2 connection, were consistently
failing. Cloudflare servers reachable via our Cogent internet
connection were fine.

We applied a band-aid by shutting down our Internet2 connection, at
which point all queries began to flow properly again via our Cogent
connection

We'll continue investigating in the morning, but might need assistance
from Cloudflare.

Thanks,

Norman Elton
William & Mary (wm.edu, AS 11975)

Cuba island-wide power outage after hurricane Ian

[Sean Donelan]

Lazaro Guerra, the technical director of the Electric Union of Cuba, the 
country’s state power company, said Cuba had "zero electric generation" 
after the hurricane caused problems in links in the west, center and east 
of the island. Mr. Guerra said crews would be working all night and 
Wednesday morning to re-establish power. He said transmission lines in 
western Cuba were down.


from WSJ

IODA shows Internet accessibility at approximately 80%

https://ioda.inetintel.cc.gatech.edu/country/CU

Kentik is also reporting a traffic drop in Cuba, but BGP routes are still 
announced.




The National Hurricane Center said the eye will come ashore in Florida on 
Wednesday afternoon.

Re: Rogers Outage: What do we Know After Two Months?

[Clayton Zekelman]

It was mentioned:

 It shouldn’t take a parliamentary inquiry or 
pointed attention from Canada’s telecoms 
regulator, or the 
<https://www.cbc.ca/news/business/rogers-outage-relevant-to-shaw-deal-1.6565709>fear 
of having its corporate merger with another 
enormous IP network backbone blocked, to bring 
those technical lessons into the light.


But not by name.

At 07:21 PM 12/09/2022, Eric Kuhnke wrote:

I did a ctrl-f for "Shaw" in that article and there's zero mention of it.

I realize that the Internet Society is meant to 
remain neutral and not comment subjectively on 
matters of market competition and conglomeration of telecoms.


It's very concerning to me that the Rogers/Shaw 
acquisition-merger will likely be allowed to 
proceed, even further reducing competition, and 
increasing centralization (the opposite of the 
decentralization mentioned in the article). From 
the point of view of a Canadian who works 
primarily for US-based ISPs these days, at least 
in the US there's seven or eight gargantuan 
multi-billion-dollar sized last mile cable 
operators. In many parts of Canada it's just 
Rogers or Shaw. It's not a good situation at all for the consumer.






On Mon, 12 Sept 2022 at 07:42, Sean Donelan 
<<mailto:s...@donelan.com>s...@donelan.com> wrote:

Article by Internet Society's Resident Advisor Jim Cowie.


Rogers Outage: What do we Know After Two Months?

<https://pulse.internetsociety.org/blog/rogers-outage-what-do-we-know-after-two-months>https://pulse.internetsociety.org/blog/rogers-outage-what-do-we-know-after-two-months

September 9, 2022

It’s now been a full two months since Rogers Telecom suffered a nationwide
Internet outage, leaving tens of millions of Canadians without
telecommunications services.


--

Clayton Zekelman
Managed Network Systems Inc. (MNSi)
3363 Tecumseh Rd. E
Windsor, Ontario
N8W 1H4

tel. 519-985-8410
fax. 519-985-8409

Re: Rogers Outage: What do we Know After Two Months?

[Eric Kuhnke]

I did a ctrl-f for "Shaw" in that article and there's zero mention of it.

I realize that the Internet Society is meant to remain neutral and not
comment subjectively on matters of market competition and conglomeration of
telecoms.

It's very concerning to me that the Rogers/Shaw acquisition-merger will
likely be allowed to proceed, even further reducing competition, and
increasing centralization (the opposite of the *decentralization* mentioned
in the article). From the point of view of a Canadian who works primarily
for US-based ISPs these days, at least in the US there's seven or eight
gargantuan multi-billion-dollar sized last mile cable operators. In many
parts of Canada it's just Rogers or Shaw. It's not a good situation at all
for the consumer.





On Mon, 12 Sept 2022 at 07:42, Sean Donelan  wrote:

> Article by Internet Society's Resident Advisor Jim Cowie.
>
>
> Rogers Outage: What do we Know After Two Months?
>
>
> https://pulse.internetsociety.org/blog/rogers-outage-what-do-we-know-after-two-months
>
> September 9, 2022
>
> It’s now been a full two months since Rogers Telecom suffered a nationwide
> Internet outage, leaving tens of millions of Canadians without
> telecommunications services.
>

Rogers Outage: What do we Know After Two Months?

[Sean Donelan]

Article by Internet Society's Resident Advisor Jim Cowie.


Rogers Outage: What do we Know After Two Months?

https://pulse.internetsociety.org/blog/rogers-outage-what-do-we-know-after-two-months

September 9, 2022

It’s now been a full two months since Rogers Telecom suffered a nationwide 
Internet outage, leaving tens of millions of Canadians without 
telecommunications services.

Frontier Temecula CA outage

[James Laszko via NANOG]

Anyone able to give any insight into Frontier FIOS outage in the Temecula area? 
 Our team can’t get past level 1 support with unknown ETTR times for hundreds 
of customers.  Over 16 hours down so far.  Any info appreciated!


Thanks,


James Laszko
P 951-813-2674
F 951-252-6210
Helpdesk 951-813-2685
jam...@mythostech.com
mythostech.com

Re: Rogers outage - House of Commons INDU hearing

[Victor Kuarsingh]

Let's see how this goes.  A political hearing will (IMO) likely not resolve
in the identification of core technical issues nor meaningful changes to
avoid what was experienced on July 8th.   A third party conducting a real
technical investigation is likely the way to find a RCA and set of
recommendations.

regards,

Victor K

On Mon, Jul 25, 2022 at 8:56 PM Sean Donelan  wrote:

> Canada is part of North America, and may be of interest to NANOG.
>
> Rogers had a nationwide outage on July 8, 2022.  The House of Commons
> held a hearing today (July 25, 2022) on the outage and invited the Rogers
> CEO, other company CEOs, regulators and academics to testify.
>
> Recording of the House of Commons INDU hearing
>
> https://www.ourcommons.ca/Committees/en/INDU
>
> Rogers CEO Tony Staffieri said the company now plans to physically
> separate its wireless and internet networks to ensure more redundancy in
> the future. Company estimates it will cost at least $250 million.
>
> Rogers still plans to complete the proposed $26-billion acquisition of
> Shaw.  The CEO claimed the greater scale of the merged companies would
> allow Rogers to create seperate wireless and internet networks faster.
>
> CRTC chairperson said that reliability and competition are separate
> issues.
>
>
> Department of Industry
> • Simon Kennedy, Deputy Minister
> • Éric Dagenais, Senior Assistant Deputy Minister, Spectrum and
> Telecommunications Sector
> • Mark Schaan, Senior Assistant Deputy Minister, Strategy and Innovation
> Policy Sector
>
> Rogers Communications Inc.
> • Tony Staffieri, President and Chief Executive Officer
> • Ron McKenzie, Chief Technology and Information Officer
> • Ted Woodhead, Chief Regulatory Officer and Government Affairs
>
> Canadian Radio-television and Telecommunications Commission
> • Ian Scott, Chairperson and Chief Executive Officer
> • Fiona Gilfillan, Executive Director, Telecommunications
> • Michel Murray, Director, Dispute Resolution and Regulatory
> Implementation, Telecommunications
>
> As an individual
> • Michael Geist, Canada Research Chair in Internet and E-Commerce Law,
> Faculty of Law, University of Ottawa
> • Ben Klass, PhD Candidate, Carleton University, Senior Research
> Associate, Canadian Media Concentration Research Project
> • Dwayne Winseck, Professor, Carleton University, Director, Global Media
> and Internet Concentration Project
>
> Public Interest Advocacy Centre
> • John Lawford, Executive Director and General Counsel
>
>

Rogers outage - House of Commons INDU hearing

[Sean Donelan]

Canada is part of North America, and may be of interest to NANOG.

Rogers had a nationwide outage on July 8, 2022.  The House of Commons 
held a hearing today (July 25, 2022) on the outage and invited the Rogers 
CEO, other company CEOs, regulators and academics to testify.


Recording of the House of Commons INDU hearing

https://www.ourcommons.ca/Committees/en/INDU

Rogers CEO Tony Staffieri said the company now plans to physically 
separate its wireless and internet networks to ensure more redundancy in 
the future. Company estimates it will cost at least $250 million.


Rogers still plans to complete the proposed $26-billion acquisition of 
Shaw.  The CEO claimed the greater scale of the merged companies would 
allow Rogers to create seperate wireless and internet networks faster.


CRTC chairperson said that reliability and competition are separate 
issues.



Department of Industry
• Simon Kennedy, Deputy Minister
• Éric Dagenais, Senior Assistant Deputy Minister, Spectrum and 
Telecommunications Sector
• Mark Schaan, Senior Assistant Deputy Minister, Strategy and Innovation 
Policy Sector


Rogers Communications Inc.
• Tony Staffieri, President and Chief Executive Officer
• Ron McKenzie, Chief Technology and Information Officer
• Ted Woodhead, Chief Regulatory Officer and Government Affairs

Canadian Radio-television and Telecommunications Commission
• Ian Scott, Chairperson and Chief Executive Officer
• Fiona Gilfillan, Executive Director, Telecommunications
• Michel Murray, Director, Dispute Resolution and Regulatory 
Implementation, Telecommunications


As an individual
• Michael Geist, Canada Research Chair in Internet and E-Commerce Law, 
Faculty of Law, University of Ottawa
• Ben Klass, PhD Candidate, Carleton University, Senior Research 
Associate, Canadian Media Concentration Research Project
• Dwayne Winseck, Professor, Carleton University, Director, Global Media 
and Internet Concentration Project


Public Interest Advocacy Centre
• John Lawford, Executive Director and General Counsel

Re: Rogers Outage Canada

[Snowmobile2004]

Looks like new agreements might be put in place to prevent this from
happening again in the future.


https://www.ctvnews.ca/business/champagne-directs-major-telecoms-to-come-up-with-agreement-on-future-outages-1.5981800

-Josh
On Mon, Jul 11, 2022 at 7:35 PM Andrew Paolucci via NANOG 
wrote:

>
> Class action lawsuit has been filled in Quebec as well.
>
> https://lpclex.com/rogers/
>
>
> --- Original Message ---
>
> On Monday, July 11th, 2022 at 3:45 PM, L F  wrote:
>
> 341,296 have signed. Let’s get to 500,000!
> At 500,000 signatures, this petition becomes one of the top signed on
> Change.org!
>
> On Mon, Jul 11, 2022 at 3:44 PM L F  wrote:
>
>> FYI --- this is now making email rounds and is available to view and sign
>> on CHANGE.ORG
>>
>>
>>
>> On Friday, Rogers experienced its second nationwide outage in as many
>> years. Hundreds of thousands of Canadians were unable to access internet
>> and cell services for hours, despite paying some of the highest fees in the
>> world. Petition starter Celina is calling on the CRTC to protect Canadians
>> from exorbitant cell and internet fees and more than 275,000 frustrated
>> Canadians have joined her. You can show your support by signing the
>> petition now.
>>
>> <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>
>> *Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone
>> Service!
>> <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>*
>>
>> *276,541 have signed* Celina London’s petition. Let’s get to 300,000!
>> Sign now with a click
>> <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAg~~>
>>
>> CANADIANS ARE BEING SCAMMED!
>>
>> Here in Canada we pay extremely high prices for our cell phones and cable
>> in comparison to countries like Australia, United Kingdom and United States
>> of America.
>>
>> As of August 2019, a plan with 30GB data from Australia’s Vodafone
>> network is $45. In Canada, major providers like Rogers and Telus charge $95
>> for 20GB at max speed.
>>
>> This is unethical and not okay. It’s time we stand up for our hard-earned
>> dollars and demand limitations be put in place to stop the wallet gouging
>> of hard-working Canadians. We must force our telecommunications companies
>> to provide Canadians better service for the astronomically high price
>> they’re forced to pay for what is a necessity to function in our high-tech
>> society in 2019.
>>
>> Please act now and sign your name below to show the CRTC (Canadian
>> Radio-Television and Telecommunications Commission) that we’ve had ENOUGH
>> and demand a change to our current telecommunications laws and regulations
>> to implement a limit on price and an immediate increase in service pr

Re: Rogers Outage Canada

[Andrew Paolucci via NANOG]

Class action lawsuit has been filled in Quebec as well.

https://lpclex.com/rogers/

--- Original Message ---
On Monday, July 11th, 2022 at 3:45 PM, L F  wrote:

> 341,296 have signed. Let’s get to 500,000!
>
> At 500,000 signatures, this petition becomes one of the top signed on 
> Change.org!
>
> On Mon, Jul 11, 2022 at 3:44 PM L F  wrote:
>
>> FYI --- this is now making email rounds and is available to view and sign on 
>> CHANGE.ORG
>>
>> On Friday, Rogers experienced its second nationwide outage in as many years. 
>> Hundreds of thousands of Canadians were unable to access internet and cell 
>> services for hours, despite paying some of the highest fees in the world. 
>> Petition starter Celina is calling on the CRTC to protect Canadians from 
>> exorbitant cell and internet fees and more than 275,000 frustrated Canadians 
>> have joined her. You can show your support by signing the petition now.
>>
>> https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC
>>
>> [Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone 
>> Service!](https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC)
>>
>> 276,541 have signed Celina London’s petition. Let’s get to 300,000!
>>
>> [Sign now with a 
>> click](https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAg~~)
>>
>> CANADIANS ARE BEING SCAMMED!
>>
>> Here in Canada we pay extremely high prices for our cell phones and cable in 
>> comparison to countries like Australia, United Kingdom and United States of 
>> America.
>>
>> As of August 2019, a plan with 30GB data from Australia’s Vodafone network 
>> is $45. In Canada, major providers like Rogers and Telus charge $95 for 20GB 
>> at max speed.
>>
>> This is unethical and not okay. It’s time we stand up for our hard-earned 
>> dollars and demand limitations be put in place to stop the wallet gouging of 
>> hard-working Canadians. We must force our telecommunications companies to 
>> provide Canadians better service for the astronomically high price they’re 
>> forced to pay for what is a necessity to function in our high-tech society 
>> in 2019.
>>
>> Please act now and sign your name below to show the CRTC (Canadian 
>> Radio-Television and Telecommunications Commission) that we’ve had ENOUGH 
>> and demand a change to our current telecommunications laws and regulations 
>> to implement a limit on price and an immediate increase in service provided 
>> for both existing and new customers.
>>
>> Once you’ve signed please take the time to share

Re: Rogers Outage Canada

[L F]

341,296 have signed. Let’s get to 500,000!
At 500,000 signatures, this petition becomes one of the top signed on
Change.org!

On Mon, Jul 11, 2022 at 3:44 PM L F  wrote:

> FYI --- this is now making email rounds and is available to view and sign
> on CHANGE.ORG
>
>
>
> On Friday, Rogers experienced its second nationwide outage in as many
> years. Hundreds of thousands of Canadians were unable to access internet
> and cell services for hours, despite paying some of the highest fees in the
> world. Petition starter Celina is calling on the CRTC to protect Canadians
> from exorbitant cell and internet fees and more than 275,000 frustrated
> Canadians have joined her. You can show your support by signing the
> petition now.
>
> <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>
> *Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone
> Service!
> <https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>*
>
> *276,541 have signed* Celina London’s petition. Let’s get to 300,000!
> Sign now with a click
> <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAg~~>
>
> CANADIANS ARE BEING SCAMMED!
>
> Here in Canada we pay extremely high prices for our cell phones and cable
> in comparison to countries like Australia, United Kingdom and United States
> of America.
>
> As of August 2019, a plan with 30GB data from Australia’s Vodafone network
> is $45. In Canada, major providers like Rogers and Telus charge $95 for
> 20GB at max speed.
>
> This is unethical and not okay. It’s time we stand up for our hard-earned
> dollars and demand limitations be put in place to stop the wallet gouging
> of hard-working Canadians. We must force our telecommunications companies
> to provide Canadians better service for the astronomically high price
> they’re forced to pay for what is a necessity to function in our high-tech
> society in 2019.
>
> Please act now and sign your name below to show the CRTC (Canadian
> Radio-Television and Telecommunications Commission) that we’ve had ENOUGH
> and demand a change to our current telecommunications laws and regulations
> to implement a limit on price and an immediate increase in service provided
> for both existing and new customers.
>
> Once you’ve signed please take the time to share with your friends and
> family as this affects us all and together we can make a big difference and
> be heard!
>
> Thank you!
>
> Celina London - concerned and annoyed Canadian.
>
>
> Sign now with a click
> <https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2

Re: Rogers Outage Canada

[L F]

FYI --- this is now making email rounds and is available to view and sign
on CHANGE.ORG



On Friday, Rogers experienced its second nationwide outage in as many
years. Hundreds of thousands of Canadians were unable to access internet
and cell services for hours, despite paying some of the highest fees in the
world. Petition starter Celina is calling on the CRTC to protect Canadians
from exorbitant cell and internet fees and more than 275,000 frustrated
Canadians have joined her. You can show your support by signing the
petition now.
<https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>
*Canadians Are Being SCAMMED - Tell the CRTC We Demand Better Cell Phone
Service!
<https://click.e.change.org/f/a/apTbTnAR7N3DW6NYQSfVMw~~/AANj1QA~/RgRkruHwP4StAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24_Y3NfdGs9QWp0bEtmZjYtX3hBU215WDFXSUFBWGljeXl2TnlRRUFCRjhCdkdKalZUd09fUklzdEFXSnBOV19PMVklM0QmcHQ9QVZCbGRHbDBhVzl1QU5WTDJRQUFBQUFBWXN4YzdJRG5PWWRqWkRkaU9URm1OQSUyNTNEJTI1M0Qmc291cmNlX2xvY2F0aW9uPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fY2FtcGFpZ249NjJiOWYzZDM2MzA0NGVhMWJhYWNmYmFkZDM2MjFiOTgmdXRtX2NvbnRlbnQ9aW5pdGlhbF92MF8wXzImdXRtX21lZGl1bT1lbWFpbCZ1dG1fc291cmNlPWFhX2F1Z21lbnRlZF9zaWduX2FzayZ1dG1fdGVybT1jc1cDc3BjQgpiu_VczGKYxTR0UhBkbnBjb25AZ21haWwuY29tWAQC>*

*276,541 have signed* Celina London’s petition. Let’s get to 300,000!
Sign now with a click
<https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAg~~>

CANADIANS ARE BEING SCAMMED!

Here in Canada we pay extremely high prices for our cell phones and cable
in comparison to countries like Australia, United Kingdom and United States
of America.

As of August 2019, a plan with 30GB data from Australia’s Vodafone network
is $45. In Canada, major providers like Rogers and Telus charge $95 for
20GB at max speed.

This is unethical and not okay. It’s time we stand up for our hard-earned
dollars and demand limitations be put in place to stop the wallet gouging
of hard-working Canadians. We must force our telecommunications companies
to provide Canadians better service for the astronomically high price
they’re forced to pay for what is a necessity to function in our high-tech
society in 2019.

Please act now and sign your name below to show the CRTC (Canadian
Radio-Television and Telecommunications Commission) that we’ve had ENOUGH
and demand a change to our current telecommunications laws and regulations
to implement a limit on price and an immediate increase in service provided
for both existing and new customers.

Once you’ve signed please take the time to share with your friends and
family as this affects us all and together we can make a big difference and
be heard!

Thank you!

Celina London - concerned and annoyed Canadian.


Sign now with a click
<https://click.e.change.org/f/a/p3sYjweM2cnYRV3CmS3wvA~~/AANj1QA~/RgRkruHwP4SxAWh0dHBzOi8vd3d3LmNoYW5nZS5vcmcvcC9jdXRjZWxsY29zdHMtY2FuYWRpYW5zLWFyZS1iZWluZy1zY2FtbWVkLXRlbGwtdGhlLWNydGMtd2UtZGVtYW5kLWxvd2VyLWNlbGwtcGhvbmUtc2VydmljZS1jcnRjZW5nL3NpZ24tbm93P2NzX3RrPUFqdGxLZmY2LV94QVNteVgxV0lBQVhpY3l5dk55UUVBQkY4QnZHSmpWVHdPX1JJc3RBV0pwTldfTzFZJTNEJnB0PUFWQmxkR2wwYVc5dUFOVkwyUUFBQUFBQVlzeGM3SURuT1lkalpEZGlPVEZtTkElMjUzRCUyNTNEJnNvdXJjZV9sb2NhdGlvbj1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX2NhbXBhaWduPTYyYjlmM2QzNjMwNDRlYTFiYWFjZmJhZGQzNjIxYjk4JnV0bV9jb250ZW50PWluaXRpYWxfdjBfMF8yJnV0bV9tZWRpdW09ZW1haWwmdXRtX3NvdXJjZT1hYV9hdWdtZW50ZWRfc2lnbl9hc2smdXRtX3Rlcm09Y3NXA3NwY0IKYrv1XMximMU0dFIQZG5wY29uQGdtYWlsLmNvbVgEAg~~>

Visit petition page
<https://click.e.change.org/f/a/apTbT

Re: Rogers Outage Canada

[Andrey Kostin]

It's hard to believe that a same time maintenance affecting so many 
devices in the core network could be approved. Core networks are build 
with redundancy, so that failures can't completely destroy the whole 
network. If it would be just "particular devices" they could be isolated 
much faster. In this case the outage was so huge that it affected mostly 
all network gear, maybe all devices from one vendor. And even after most 
of the services restored we still observed partial connectivity in some 
areas that means the impact was mush deeper, in some places down to 
access layer.


Kind regards,
Andrey

Shane Ronan писал(а) 2022-07-11 10:09:

What in depth analysis have you seen? Seems to me, this was a failure
in a known maintenance activity, and they simply disconnected the
devices under maintenance from the network.

Shane

Re: Rogers Outage Canada

[Niels Bakker]

* vic...@jvknet.com (Victor Kuarsingh) [Mon 11 Jul 2022, 17:17 CEST]:
This is the most they can and will say.  For liabilities reasons, 
specifics are likely not in the cards.


I doubt it. Given that emergency services were impacted you can count 
on a proper assessment being made available to a parliamentary 
inquiry. Too much broke to cover this up any further.



-- Niels.

Re: Rogers Outage Canada

[Jared Mauch]

> On Jul 11, 2022, at 11:15 AM, Victor Kuarsingh  wrote:
> 
> This is the most they can and will say.  For liabilities reasons, specifics 
> are likely not in the cards.  As most services ride over common service 
> networks, its quite possible that a network substrate failure can have a 
> number of upstream service impacts.  The point here is that the CEO is 
> directly addressing the customer base, which is needed here.
> 

Yes, unless you get a side-leak from someone in-the-know, I expect either 
direct or CRTC will be the best source.  I know a few of us are concerned about 
if we have any similar risk of impact to our environments, so we will be 
watching close for any hints.

- Jared

Re: Rogers Outage Canada

[Keith Stokes]

As a Rogers data center customer I received this on Saturday:


You have received this bulletin because you are an official contact for your 
Rogers Data Centres Server Colocation service

The nationwide network outage has now been resolved.

Rogers Data Centres is reporting all network and dependent services as 
operating normally. If you are experiencing any ongoing issues please reach out 
to the Data Centres help desk

An official Incident Report will be shared in 5 to 7 Business days.

We thank you for your patience during the restoration.


--

Keith Stokes
SalonBiz, Inc



On Jul 11, 2022, at 10:16 AM, Victor Kuarsingh 
mailto:vic...@jvknet.com>> wrote:

This is the most they can and will say.  For liabilities reasons, specifics are 
likely not in the cards.  As most services ride over common service networks, 
its quite possible that a network substrate failure can have a number of 
upstream service impacts.  The point here is that the CEO is directly 
addressing the customer base, which is needed here.

regards,

Victor K

On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan 
mailto:sh...@ronan-online.com>> wrote:
What in depth analysis have you seen? Seems to me, this was a failure in a 
known maintenance activity, and they simply disconnected the devices under 
maintenance from the network.

Shane

On Mon, Jul 11, 2022 at 5:41 AM Jon Sands 
mailto:fohdee...@gmail.com>> wrote:
Given the outage was so bad it was disrupting select E911 services nationwide 
for something like 24+ hours, it's great to see such an in depth analysis and 
plan of action to prevent such things in the future. bravo rogers

On 7/10/2022 2:55 PM, L F wrote:
fyi - see BOLD.

A Message from Rogers President and CEO



Dear Valued Customer,

As you know, we experienced a service outage across the Rogers, Fido, Chatr and 
Cityfone wireless networks on Friday.

I am reaching out to share that our services have been restored, and our 
networks and systems are close to fully operational. Our technical teams are 
continuing to monitor for any remaining intermittent issues. I also want to 
outline an action plan we are putting in place to address what happened.

I want to share what we know about what happened on Friday. We now believe 
we’ve narrowed the cause to a network system failure following a maintenance 
update in our core network, which caused some of our routers to malfunction. We 
disconnected the specific equipment and redirected traffic, which allowed our 
network and services to come back online over time as we managed traffic 
volumes returning to normal levels.



On Sat, Jul 9, 2022 at 9:09 PM L F 
mailto:liz.faze...@gmail.com>> wrote:

Lest we ALL MOVE ON….


Yes he said

“RETARD” =

The delay in processing the current status of a Situation…


Lets move on to bigger n scarier Post mortems:


Was This or Was this NOT

A CYBERATTACK ???

Done.

Lets deal with REAL threats not PC references.

Time to evaluate n re eval Our NTS!!

Veni vidi vici 2022




On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:
Can we have a discussion with the list admins about a list member appending a 
threat of violence to their outbound emails?  Whether serious or not.

Does this person need directions to some local mental healthcare resources?


On Sat, 9 Jul 2022 at 08:48, Keith Medcalf 
mailto:kmedc...@dessus.com>> wrote:

>I can't either, but the reality right now seems to be that 911 calls are
>failing for anyone on a Rogers cellphone.

This is par for the course.  These people chose to deal with Rogers despite 
knowing the consequences.  It is like if you bought a Rogers Snowblower and it 
did not work.  That would mean that people who bought the Rogers Snowblower 
will not be using it to get rid of the snow that is preventing them from 
leaving their house.

Mutatis mutandis when Rogers is down things that are Rogers dependent will not 
work.

Some people are so retarded it is astonishing!

--
(CAUTION) You are advised that if you attack my person or property, you will be 
put down in accordance with the provisions of section 34 & 35 of the Criminal 
Code respectively.  If you are brandishing (or in possession) of a weapon then 
lethal force will be applied to your person in accordance with the law.  This 
means that your misadventures may end in your death.  Consider yourself 
cautioned and govern your actions appropriately.

>-Original Message-
>From: NANOG 
>mailto:dessus@nanog.org>> On 
>Behalf Of
>Eric Kuhnke
>Sent: Friday, 8 July, 2022 13:34
>To: jim deleskie mailto:deles...@gmail.com>>
>Cc: NANOG list mailto:nanog@nanog.org>>
>Subject: Re: Rogers Outage Canada
>
>
>I have seen anecdotal reports that the mobile network is in a half broken
>state that phones remain registered to, so a 911 call will attempt and
>then fail.
>
>
>This is unlike what would happen if you had a US/

Re: Rogers Outage Canada

[Victor Kuarsingh]

This is the most they can and will say.  For liabilities reasons, specifics
are likely not in the cards.  As most services ride over common service
networks, its quite possible that a network substrate failure can have a
number of upstream service impacts.  The point here is that the CEO is
directly addressing the customer base, which is needed here.

regards,

Victor K

On Mon, Jul 11, 2022 at 10:11 AM Shane Ronan  wrote:

> What in depth analysis have you seen? Seems to me, this was a failure in a
> known maintenance activity, and they simply disconnected the devices under
> maintenance from the network.
>
> Shane
>
> On Mon, Jul 11, 2022 at 5:41 AM Jon Sands  wrote:
>
>> Given the outage was so bad it was disrupting select E911 services
>> nationwide for something like 24+ hours, it's great to see such an in depth
>> analysis and plan of action to prevent such things in the future. bravo
>> rogers
>>
>> On 7/10/2022 2:55 PM, L F wrote:
>>
>> fyi - see BOLD.
>>
>> A Message from Rogers President and CEO
>>
>>
>> Dear Valued Customer,
>>
>> As you know, we experienced a service outage across the Rogers, Fido,
>> Chatr and Cityfone wireless networks on Friday.
>>
>> I am reaching out to share that our services have been restored, and our
>> networks and systems are close to fully operational. Our technical teams
>> are continuing to monitor for any remaining intermittent issues. I also
>> want to outline an action plan we are putting in place to address what
>> happened.
>>
>> I want to share what we know about what happened on Friday. We now
>> believe *we’ve narrowed the cause to a network system failure following
>> a maintenance update in our core network, which caused some of our routers
>> to malfunction. We disconnected the specific equipment and redirected
>> traffic,* which allowed our network and services to come back online
>> over time as we managed traffic volumes returning to normal levels.
>>
>>
>>
>> On Sat, Jul 9, 2022 at 9:09 PM L F  wrote:
>>
>>>
>>> Lest we ALL MOVE ON….
>>>
>>>
>>> Yes he said
>>>
>>> “RETARD” =
>>>
>>> The delay in processing the current status of a Situation…
>>>
>>>
>>> Lets move on to bigger n scarier Post mortems:
>>>
>>>
>>> Was This or Was this NOT
>>>
>>> A CYBERATTACK ???
>>>
>>> Done.
>>>
>>> Lets deal with REAL threats not PC references.
>>>
>>> Time to evaluate n re eval Our NTS!!
>>>
>>> Veni vidi vici 2022
>>>
>>>
>>>
>>>
>>> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke 
>>> wrote:
>>>
>>>> Can we have a discussion with the list admins about a list member
>>>> appending a threat of violence to their outbound emails?  Whether serious
>>>> or not.
>>>>
>>>> Does this person need directions to some local mental healthcare
>>>> resources?
>>>>
>>>>
>>>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:
>>>>
>>>>>
>>>>> >I can't either, but the reality right now seems to be that 911 calls
>>>>> are
>>>>> >failing for anyone on a Rogers cellphone.
>>>>>
>>>>> This is par for the course.  These people chose to deal with Rogers
>>>>> despite knowing the consequences.  It is like if you bought a Rogers
>>>>> Snowblower and it did not work.  That would mean that people who bought 
>>>>> the
>>>>> Rogers Snowblower will not be using it to get rid of the snow that is
>>>>> preventing them from leaving their house.
>>>>>
>>>>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>>>>> will not work.
>>>>>
>>>>> Some people are so retarded it is astonishing!
>>>>>
>>>>> --
>>>>> (CAUTION) You are advised that if you attack my person or property,
>>>>> you will be put down in accordance with the provisions of section 34 & 35
>>>>> of the Criminal Code respectively.  If you are brandishing (or in
>>>>> possession) of a weapon then lethal force will be applied to your person 
>>>>> in
>>>>> accordance with the law.  This means that your misadventures may end in
>>>>> your death.  Consider yourself cautioned and govern your actions
>>>>&

Re: Rogers Outage Canada

[Shane Ronan]

What in depth analysis have you seen? Seems to me, this was a failure in a
known maintenance activity, and they simply disconnected the devices under
maintenance from the network.

Shane

On Mon, Jul 11, 2022 at 5:41 AM Jon Sands  wrote:

> Given the outage was so bad it was disrupting select E911 services
> nationwide for something like 24+ hours, it's great to see such an in depth
> analysis and plan of action to prevent such things in the future. bravo
> rogers
>
> On 7/10/2022 2:55 PM, L F wrote:
>
> fyi - see BOLD.
>
> A Message from Rogers President and CEO
>
>
> Dear Valued Customer,
>
> As you know, we experienced a service outage across the Rogers, Fido,
> Chatr and Cityfone wireless networks on Friday.
>
> I am reaching out to share that our services have been restored, and our
> networks and systems are close to fully operational. Our technical teams
> are continuing to monitor for any remaining intermittent issues. I also
> want to outline an action plan we are putting in place to address what
> happened.
>
> I want to share what we know about what happened on Friday. We now believe 
> *we’ve
> narrowed the cause to a network system failure following a maintenance
> update in our core network, which caused some of our routers to
> malfunction. We disconnected the specific equipment and redirected traffic,*
> which allowed our network and services to come back online over time as we
> managed traffic volumes returning to normal levels.
>
>
>
> On Sat, Jul 9, 2022 at 9:09 PM L F  wrote:
>
>>
>> Lest we ALL MOVE ON….
>>
>>
>> Yes he said
>>
>> “RETARD” =
>>
>> The delay in processing the current status of a Situation…
>>
>>
>> Lets move on to bigger n scarier Post mortems:
>>
>>
>> Was This or Was this NOT
>>
>> A CYBERATTACK ???
>>
>> Done.
>>
>> Lets deal with REAL threats not PC references.
>>
>> Time to evaluate n re eval Our NTS!!
>>
>> Veni vidi vici 2022
>>
>>
>>
>>
>> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke  wrote:
>>
>>> Can we have a discussion with the list admins about a list member
>>> appending a threat of violence to their outbound emails?  Whether serious
>>> or not.
>>>
>>> Does this person need directions to some local mental healthcare
>>> resources?
>>>
>>>
>>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:
>>>
>>>>
>>>> >I can't either, but the reality right now seems to be that 911 calls
>>>> are
>>>> >failing for anyone on a Rogers cellphone.
>>>>
>>>> This is par for the course.  These people chose to deal with Rogers
>>>> despite knowing the consequences.  It is like if you bought a Rogers
>>>> Snowblower and it did not work.  That would mean that people who bought the
>>>> Rogers Snowblower will not be using it to get rid of the snow that is
>>>> preventing them from leaving their house.
>>>>
>>>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>>>> will not work.
>>>>
>>>> Some people are so retarded it is astonishing!
>>>>
>>>> --
>>>> (CAUTION) You are advised that if you attack my person or property, you
>>>> will be put down in accordance with the provisions of section 34 & 35 of
>>>> the Criminal Code respectively.  If you are brandishing (or in possession)
>>>> of a weapon then lethal force will be applied to your person in accordance
>>>> with the law.  This means that your misadventures may end in your death.
>>>> Consider yourself cautioned and govern your actions appropriately.
>>>>
>>>> >-Original Message-
>>>> >From: NANOG  On Behalf Of
>>>> >Eric Kuhnke
>>>> >Sent: Friday, 8 July, 2022 13:34
>>>> >To: jim deleskie 
>>>> >Cc: NANOG list 
>>>> >Subject: Re: Rogers Outage Canada
>>>> >
>>>> >
>>>> >I have seen anecdotal reports that the mobile network is in a half
>>>> broken
>>>> >state that phones remain registered to, so a 911 call will attempt and
>>>> >then fail.
>>>> >
>>>> >
>>>> >This is unlike what would happen if you had a US/Canada cellphone with
>>>> >battery power but no SIM card in it that would search for any available
>>>> >network in RF range for a 911 call if needed.
>>

Re: Rogers Outage Canada

[Alain Hebert]

    Seeing the little of that thread.

    I never understood why someone thinks their experience with Level 1 
thru 3 staff reflect the actual situation.


    That was clearly an issue with network engineering and their 
processes.  Because a 14h downtime for, as Rogers says, "a specific 
equipment" is too big of a spin.


-
Alain hebertaheb...@pubnix.net
PubNIX Inc.

50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911http://www.pubnix.net Fax: 514-990-9443

On 7/9/22 21:18, Victor Kuarsingh wrote:
We should move on and focus on constructive comments vs. kicking an 
operator when they are down.


That said, demeaning comments, no matter what they are or whom they 
are targeted at, is not healthy and is not helpful.

regards,

Victor K

On Sat, Jul 9, 2022 at 9:11 PM L F  wrote:


Lest we ALL MOVE ON….


Yes he said

“RETARD” =

The delay in processing the current status of a Situation…


Lets move on to bigger n scarier Post mortems:


Was This or Was this NOT

A CYBERATTACK ???

Done.

Lets deal with REAL threats not PC references.

Time to evaluate n re eval Our NTS!!

Veni vidi vici 2022




On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke 
wrote:

Can we have a discussion with the list admins about a list
member appending a threat of violence to their outbound
emails?  Whether serious or not.

Does this person need directions to some local mental
healthcare resources?


On Sat, 9 Jul 2022 at 08:48, Keith Medcalf
 wrote:


>I can't either, but the reality right now seems to be
that 911 calls are
>failing for anyone on a Rogers cellphone.

This is par for the course.  These people chose to deal
with Rogers despite knowing the consequences.  It is like
if you bought a Rogers Snowblower and it did not work. 
That would mean that people who bought the Rogers
Snowblower will not be using it to get rid of the snow
that is preventing them from leaving their house.

Mutatis mutandis when Rogers is down things that are
Rogers dependent will not work.

Some people are so retarded it is astonishing!

-- 
(CAUTION) You are advised that if you attack my person or

property, you will be put down in accordance with the
provisions of section 34 & 35 of the Criminal Code
respectively.  If you are brandishing (or in possession)
of a weapon then lethal force will be applied to your
person in accordance with the law.  This means that your
misadventures may end in your death. Consider yourself
cautioned and govern your actions appropriately.

>-Original Message-
>From: NANOG 
On Behalf Of
>Eric Kuhnke
>Sent: Friday, 8 July, 2022 13:34
>To: jim deleskie 
>Cc: NANOG list 
    >Subject: Re: Rogers Outage Canada
>
>
>I have seen anecdotal reports that the mobile network is
in a half broken
>state that phones remain registered to, so a 911 call
will attempt and
>then fail.
>
>
>This is unlike what would happen if you had a US/Canada
cellphone with
>battery power but no SIM card in it that would search for
any available
>network in RF range for a 911 call if needed.
>
>
>On Fri, 8 Jul 2022 at 12:31, jim deleskie <mailto:deles...@gmail.com> > wrote:
>
>
>       i cant see BGP taking out SS7.
>
>       -jim
>
>       On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>mailto:greenjosh6...@gmail.com>
> wrote:
>
>
>               According to Cloudflare Radar
><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>
, Rogers
>BGP announcements spiked massively to levels 536,777%
higher than normal
>(343,601 vs 64 normally) just minutes before the outage.
I would not be
>surprised if this happened to be the culprit.
>
>               Regards,
>               Josh Green
>
>               On Fri, Jul 8, 2022 at 2:19 PM Andrew
Paolucci via NANOG
>mailto:nanog@nanog.org> > wrote:
>
>
>                       In the ea

Re: Rogers Outage Canada

[Jon Sands]

Given the outage was so bad it was disrupting select E911 services 
nationwide for something like 24+ hours, it's great to see such an in 
depth analysis and plan of action to prevent such things in the future. 
bravo rogers


On 7/10/2022 2:55 PM, L F wrote:

fyi - see BOLD.

A Message from Rogers President and CEO



Dear Valued Customer,

As you know, we experienced a service outage across the Rogers, Fido, 
Chatr and Cityfone wireless networks on Friday.


I am reaching out to share that our services have been restored, and 
our networks and systems are close to fully operational. Our technical 
teams are continuing to monitor for any remaining intermittent issues. 
I also want to outline an action plan we are putting in place to 
address what happened.


I want to share what we know about what happened on Friday. We now 
believe *we’ve narrowed the cause to a network system failure 
following a maintenance update in our core network, which caused some 
of our routers to malfunction. We disconnected the specific equipment 
and redirected traffic,*which allowed our network and services to come 
back online over time as we managed traffic volumes returning to 
normal levels.





On Sat, Jul 9, 2022 at 9:09 PM L F  wrote:


Lest we ALL MOVE ON….


Yes he said

“RETARD” =

The delay in processing the current status of a Situation…


Lets move on to bigger n scarier Post mortems:


Was This or Was this NOT

A CYBERATTACK ???

Done.

Lets deal with REAL threats not PC references.

Time to evaluate n re eval Our NTS!!

Veni vidi vici 2022




On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke 
wrote:

Can we have a discussion with the list admins about a list
member appending a threat of violence to their outbound
emails?  Whether serious or not.

Does this person need directions to some local mental
healthcare resources?


On Sat, 9 Jul 2022 at 08:48, Keith Medcalf
 wrote:


>I can't either, but the reality right now seems to be
that 911 calls are
>failing for anyone on a Rogers cellphone.

This is par for the course.  These people chose to deal
with Rogers despite knowing the consequences.  It is like
if you bought a Rogers Snowblower and it did not work. 
That would mean that people who bought the Rogers
Snowblower will not be using it to get rid of the snow
that is preventing them from leaving their house.

Mutatis mutandis when Rogers is down things that are
Rogers dependent will not work.

Some people are so retarded it is astonishing!

-- 
(CAUTION) You are advised that if you attack my person or

property, you will be put down in accordance with the
provisions of section 34 & 35 of the Criminal Code
respectively.  If you are brandishing (or in possession)
of a weapon then lethal force will be applied to your
person in accordance with the law.  This means that your
misadventures may end in your death. Consider yourself
cautioned and govern your actions appropriately.

>-Original Message-
>From: NANOG 
On Behalf Of
>Eric Kuhnke
>Sent: Friday, 8 July, 2022 13:34
>To: jim deleskie 
>Cc: NANOG list 
    >Subject: Re: Rogers Outage Canada
>
>
>I have seen anecdotal reports that the mobile network is
in a half broken
>state that phones remain registered to, so a 911 call
will attempt and
>then fail.
>
>
>This is unlike what would happen if you had a US/Canada
cellphone with
>battery power but no SIM card in it that would search for
any available
>network in RF range for a 911 call if needed.
>
>
>On Fri, 8 Jul 2022 at 12:31, jim deleskie <mailto:deles...@gmail.com> > wrote:
>
>
>       i cant see BGP taking out SS7.
>
>       -jim
>
>       On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>mailto:greenjosh6...@gmail.com>
> wrote:
>
>
>               According to Cloudflare Radar
><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours>
, Rogers
>BGP announcements spiked massively to levels 536,777%
higher than normal
>(343,601 vs 64 normally) just minutes before the outage.
I would not be
>

Re: Rogers Outage Canada

[L F]

fyi - see BOLD.

A Message from Rogers President and CEO

Dear Valued Customer,

As you know, we experienced a service outage across the Rogers, Fido, Chatr
and Cityfone wireless networks on Friday.

I am reaching out to share that our services have been restored, and our
networks and systems are close to fully operational. Our technical teams
are continuing to monitor for any remaining intermittent issues. I also
want to outline an action plan we are putting in place to address what
happened.

I want to share what we know about what happened on Friday. We now
believe *we’ve
narrowed the cause to a network system failure following a maintenance
update in our core network, which caused some of our routers to
malfunction. We disconnected the specific equipment and redirected traffic,*
which allowed our network and services to come back online over time as we
managed traffic volumes returning to normal levels.



On Sat, Jul 9, 2022 at 9:09 PM L F  wrote:

>
> Lest we ALL MOVE ON….
>
>
> Yes he said
>
> “RETARD” =
>
> The delay in processing the current status of a Situation…
>
>
> Lets move on to bigger n scarier Post mortems:
>
>
> Was This or Was this NOT
>
> A CYBERATTACK ???
>
> Done.
>
> Lets deal with REAL threats not PC references.
>
> Time to evaluate n re eval Our NTS!!
>
> Veni vidi vici 2022
>
>
>
>
> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke  wrote:
>
>> Can we have a discussion with the list admins about a list member
>> appending a threat of violence to their outbound emails?  Whether serious
>> or not.
>>
>> Does this person need directions to some local mental healthcare
>> resources?
>>
>>
>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:
>>
>>>
>>> >I can't either, but the reality right now seems to be that 911 calls are
>>> >failing for anyone on a Rogers cellphone.
>>>
>>> This is par for the course.  These people chose to deal with Rogers
>>> despite knowing the consequences.  It is like if you bought a Rogers
>>> Snowblower and it did not work.  That would mean that people who bought the
>>> Rogers Snowblower will not be using it to get rid of the snow that is
>>> preventing them from leaving their house.
>>>
>>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>>> will not work.
>>>
>>> Some people are so retarded it is astonishing!
>>>
>>> --
>>> (CAUTION) You are advised that if you attack my person or property, you
>>> will be put down in accordance with the provisions of section 34 & 35 of
>>> the Criminal Code respectively.  If you are brandishing (or in possession)
>>> of a weapon then lethal force will be applied to your person in accordance
>>> with the law.  This means that your misadventures may end in your death.
>>> Consider yourself cautioned and govern your actions appropriately.
>>>
>>> >-Original Message-
>>> >From: NANOG  On Behalf Of
>>> >Eric Kuhnke
>>> >Sent: Friday, 8 July, 2022 13:34
>>> >To: jim deleskie 
>>> >Cc: NANOG list 
>>> >Subject: Re: Rogers Outage Canada
>>> >
>>> >
>>> >I have seen anecdotal reports that the mobile network is in a half
>>> broken
>>> >state that phones remain registered to, so a 911 call will attempt and
>>> >then fail.
>>> >
>>> >
>>> >This is unlike what would happen if you had a US/Canada cellphone with
>>> >battery power but no SIM card in it that would search for any available
>>> >network in RF range for a 911 call if needed.
>>> >
>>> >
>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie >> ><mailto:deles...@gmail.com> > wrote:
>>> >
>>> >
>>> >   i cant see BGP taking out SS7.
>>> >
>>> >   -jim
>>> >
>>> >   On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>>> >mailto:greenjosh6...@gmail.com> > wrote:
>>> >
>>> >
>>> >   According to Cloudflare Radar
>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> ,
>>> Rogers
>>> >BGP announcements spiked massively to levels 536,777% higher than normal
>>> >(343,601 vs 64 normally) just minutes before the outage. I would not be
>>> >surprised if this happened to be the culprit.
>>> >
>>> >   Regards,
>>> >   Josh Green
>>> >
&

Re: Rogers Outage Canada

[Victor Kuarsingh]

We should move on and focus on constructive comments vs. kicking an
operator when they are down.

That said, demeaning comments, no matter what they are or whom they are
targeted at, is not healthy and is not helpful.

regards,

Victor K

On Sat, Jul 9, 2022 at 9:11 PM L F  wrote:

>
> Lest we ALL MOVE ON….
>
>
> Yes he said
>
> “RETARD” =
>
> The delay in processing the current status of a Situation…
>
>
> Lets move on to bigger n scarier Post mortems:
>
>
> Was This or Was this NOT
>
> A CYBERATTACK ???
>
> Done.
>
> Lets deal with REAL threats not PC references.
>
> Time to evaluate n re eval Our NTS!!
>
> Veni vidi vici 2022
>
>
>
>
> On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke  wrote:
>
>> Can we have a discussion with the list admins about a list member
>> appending a threat of violence to their outbound emails?  Whether serious
>> or not.
>>
>> Does this person need directions to some local mental healthcare
>> resources?
>>
>>
>> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:
>>
>>>
>>> >I can't either, but the reality right now seems to be that 911 calls are
>>> >failing for anyone on a Rogers cellphone.
>>>
>>> This is par for the course.  These people chose to deal with Rogers
>>> despite knowing the consequences.  It is like if you bought a Rogers
>>> Snowblower and it did not work.  That would mean that people who bought the
>>> Rogers Snowblower will not be using it to get rid of the snow that is
>>> preventing them from leaving their house.
>>>
>>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>>> will not work.
>>>
>>> Some people are so retarded it is astonishing!
>>>
>>> --
>>> (CAUTION) You are advised that if you attack my person or property, you
>>> will be put down in accordance with the provisions of section 34 & 35 of
>>> the Criminal Code respectively.  If you are brandishing (or in possession)
>>> of a weapon then lethal force will be applied to your person in accordance
>>> with the law.  This means that your misadventures may end in your death.
>>> Consider yourself cautioned and govern your actions appropriately.
>>>
>>> >-Original Message-
>>> >From: NANOG  On Behalf Of
>>> >Eric Kuhnke
>>> >Sent: Friday, 8 July, 2022 13:34
>>> >To: jim deleskie 
>>> >Cc: NANOG list 
>>> >Subject: Re: Rogers Outage Canada
>>> >
>>> >
>>> >I have seen anecdotal reports that the mobile network is in a half
>>> broken
>>> >state that phones remain registered to, so a 911 call will attempt and
>>> >then fail.
>>> >
>>> >
>>> >This is unlike what would happen if you had a US/Canada cellphone with
>>> >battery power but no SIM card in it that would search for any available
>>> >network in RF range for a 911 call if needed.
>>> >
>>> >
>>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie >> ><mailto:deles...@gmail.com> > wrote:
>>> >
>>> >
>>> >   i cant see BGP taking out SS7.
>>> >
>>> >   -jim
>>> >
>>> >   On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>>> >mailto:greenjosh6...@gmail.com> > wrote:
>>> >
>>> >
>>> >   According to Cloudflare Radar
>>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> ,
>>> Rogers
>>> >BGP announcements spiked massively to levels 536,777% higher than normal
>>> >(343,601 vs 64 normally) just minutes before the outage. I would not be
>>> >surprised if this happened to be the culprit.
>>> >
>>> >   Regards,
>>> >   Josh Green
>>> >
>>> >   On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG
>>> >mailto:nanog@nanog.org> > wrote:
>>> >
>>> >
>>> >   In the early hours of the morning around 2-3am
>>> my modem
>>> >got hit with a configuration update that caused a DHCP release that
>>> >wasn't renewed for about two hours, after rollback the connection was
>>> >fine for 3 hours before this network wide outage.
>>> >
>>> >
>>> >   Maybe a failed night time update was attempted
>>> again
>>> &

Re: Rogers Outage Canada

[L F]

Lest we ALL MOVE ON….


Yes he said

“RETARD” =

The delay in processing the current status of a Situation…


Lets move on to bigger n scarier Post mortems:


Was This or Was this NOT

A CYBERATTACK ???

Done.

Lets deal with REAL threats not PC references.

Time to evaluate n re eval Our NTS!!

Veni vidi vici 2022




On Sat, Jul 9, 2022 at 5:01 PM Eric Kuhnke  wrote:

> Can we have a discussion with the list admins about a list member
> appending a threat of violence to their outbound emails?  Whether serious
> or not.
>
> Does this person need directions to some local mental healthcare resources?
>
>
> On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:
>
>>
>> >I can't either, but the reality right now seems to be that 911 calls are
>> >failing for anyone on a Rogers cellphone.
>>
>> This is par for the course.  These people chose to deal with Rogers
>> despite knowing the consequences.  It is like if you bought a Rogers
>> Snowblower and it did not work.  That would mean that people who bought the
>> Rogers Snowblower will not be using it to get rid of the snow that is
>> preventing them from leaving their house.
>>
>> Mutatis mutandis when Rogers is down things that are Rogers dependent
>> will not work.
>>
>> Some people are so retarded it is astonishing!
>>
>> --
>> (CAUTION) You are advised that if you attack my person or property, you
>> will be put down in accordance with the provisions of section 34 & 35 of
>> the Criminal Code respectively.  If you are brandishing (or in possession)
>> of a weapon then lethal force will be applied to your person in accordance
>> with the law.  This means that your misadventures may end in your death.
>> Consider yourself cautioned and govern your actions appropriately.
>>
>> >-Original Message-
>> >From: NANOG  On Behalf Of
>> >Eric Kuhnke
>> >Sent: Friday, 8 July, 2022 13:34
>> >To: jim deleskie 
>> >Cc: NANOG list 
>> >Subject: Re: Rogers Outage Canada
>> >
>> >
>> >I have seen anecdotal reports that the mobile network is in a half broken
>> >state that phones remain registered to, so a 911 call will attempt and
>> >then fail.
>> >
>> >
>> >This is unlike what would happen if you had a US/Canada cellphone with
>> >battery power but no SIM card in it that would search for any available
>> >network in RF range for a 911 call if needed.
>> >
>> >
>> >On Fri, 8 Jul 2022 at 12:31, jim deleskie > ><mailto:deles...@gmail.com> > wrote:
>> >
>> >
>> >   i cant see BGP taking out SS7.
>> >
>> >   -jim
>> >
>> >   On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
>> >mailto:greenjosh6...@gmail.com> > wrote:
>> >
>> >
>> >   According to Cloudflare Radar
>> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> ,
>> Rogers
>> >BGP announcements spiked massively to levels 536,777% higher than normal
>> >(343,601 vs 64 normally) just minutes before the outage. I would not be
>> >surprised if this happened to be the culprit.
>> >
>> >   Regards,
>> >   Josh Green
>> >
>> >   On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG
>> >mailto:nanog@nanog.org> > wrote:
>> >
>> >
>> >   In the early hours of the morning around 2-3am my
>> modem
>> >got hit with a configuration update that caused a DHCP release that
>> >wasn't renewed for about two hours, after rollback the connection was
>> >fine for 3 hours before this network wide outage.
>> >
>> >
>> >   Maybe a failed night time update was attempted
>> again
>> >during office hours, I've heard daytime guys are still WFH and night
>> >shift is in building.
>> >
>> >
>> >   I expect we'll never get a real explanation.
>> Rogers is
>> >notorious for withholding any type of helpful or technical information.
>> >
>> >
>> >   Sent from my inoperable Rogers Mobile via
>> emergency eSIM.
>> >
>> >
>> >   Regards,
>> >
>> >   Andrew Paolucci
>> >    Original Message 
>> >   On Jul. 8, 2022, 1:48 p.m., Jay Hennigan <
>> j...@west.net
>> ><mailto:j...@west.net> > wrote:
>> >
>> >
>> >   On 7/8/22 07:44, Robert DeVita wrote: >
>> Does anyone
>> >have information on a widespread Rogers outage in Canada. I > have
>> >customers with multiple sites down. There's discussion on the Outages
>> >mailing list. Seems widespread, affecting all services, mobile, voice,
>> >Internet. No cause or ETR posted yet. -- Jay Hennigan - j...@west.net
>> ><mailto:j...@west.net>  Network Engineering - CCIE #7880 503 897-8550 -
>> >WB6RDV
>> >
>> >
>> >
>> >   --
>> >
>> >   Josh Green.
>>
>>
>>
>>
>> --
  Liz ***
416.660.5456

Re: Rogers Outage Canada

[Eric Kuhnke]

Can we have a discussion with the list admins about a list member appending
a threat of violence to their outbound emails?  Whether serious or not.

Does this person need directions to some local mental healthcare resources?


On Sat, 9 Jul 2022 at 08:48, Keith Medcalf  wrote:

>
> >I can't either, but the reality right now seems to be that 911 calls are
> >failing for anyone on a Rogers cellphone.
>
> This is par for the course.  These people chose to deal with Rogers
> despite knowing the consequences.  It is like if you bought a Rogers
> Snowblower and it did not work.  That would mean that people who bought the
> Rogers Snowblower will not be using it to get rid of the snow that is
> preventing them from leaving their house.
>
> Mutatis mutandis when Rogers is down things that are Rogers dependent will
> not work.
>
> Some people are so retarded it is astonishing!
>
> --
> (CAUTION) You are advised that if you attack my person or property, you
> will be put down in accordance with the provisions of section 34 & 35 of
> the Criminal Code respectively.  If you are brandishing (or in possession)
> of a weapon then lethal force will be applied to your person in accordance
> with the law.  This means that your misadventures may end in your death.
> Consider yourself cautioned and govern your actions appropriately.
>
> >-Original Message-
> >From: NANOG  On Behalf Of
> >Eric Kuhnke
> >Sent: Friday, 8 July, 2022 13:34
> >To: jim deleskie 
> >Cc: NANOG list 
> >Subject: Re: Rogers Outage Canada
> >
> >
> >I have seen anecdotal reports that the mobile network is in a half broken
> >state that phones remain registered to, so a 911 call will attempt and
> >then fail.
> >
> >
> >This is unlike what would happen if you had a US/Canada cellphone with
> >battery power but no SIM card in it that would search for any available
> >network in RF range for a 911 call if needed.
> >
> >
> >On Fri, 8 Jul 2022 at 12:31, jim deleskie  ><mailto:deles...@gmail.com> > wrote:
> >
> >
> >   i cant see BGP taking out SS7.
> >
> >   -jim
> >
> >   On Fri, Jul 8, 2022 at 2:45 PM Snowmobile2004
> >mailto:greenjosh6...@gmail.com> > wrote:
> >
> >
> >   According to Cloudflare Radar
> ><https://radar.cloudflare.com/asn/812?date_filter=last_24_hours> , Rogers
> >BGP announcements spiked massively to levels 536,777% higher than normal
> >(343,601 vs 64 normally) just minutes before the outage. I would not be
> >surprised if this happened to be the culprit.
> >
> >   Regards,
> >   Josh Green
> >
> >   On Fri, Jul 8, 2022 at 2:19 PM Andrew Paolucci via NANOG
> >mailto:nanog@nanog.org> > wrote:
> >
> >
> >   In the early hours of the morning around 2-3am my
> modem
> >got hit with a configuration update that caused a DHCP release that
> >wasn't renewed for about two hours, after rollback the connection was
> >fine for 3 hours before this network wide outage.
> >
> >
> >   Maybe a failed night time update was attempted
> again
> >during office hours, I've heard daytime guys are still WFH and night
> >shift is in building.
> >
> >
> >   I expect we'll never get a real explanation.
> Rogers is
> >notorious for withholding any type of helpful or technical information.
> >
> >
> >   Sent from my inoperable Rogers Mobile via
> emergency eSIM.
> >
> >
> >   Regards,
> >
> >   Andrew Paolucci
> >    Original Message 
> >   On Jul. 8, 2022, 1:48 p.m., Jay Hennigan <
> j...@west.net
> ><mailto:j...@west.net> > wrote:
> >
> >
> >   On 7/8/22 07:44, Robert DeVita wrote: >
> Does anyone
> >have information on a widespread Rogers outage in Canada. I > have
> >customers with multiple sites down. There's discussion on the Outages
> >mailing list. Seems widespread, affecting all services, mobile, voice,
> >Internet. No cause or ETR posted yet. -- Jay Hennigan - j...@west.net
> ><mailto:j...@west.net>  Network Engineering - CCIE #7880 503 897-8550 -
> >WB6RDV
> >
> >
> >
> >   --
> >
> >   Josh Green.
>
>
>
>
>