Wow -
For the record: Inner West Council(NSW) went out and this from NSW police
UPDATE: Optus network disruption
/Wednesday, 08 November 2023 03:41:31 PM/
The Optus Network outage has caused a major communications disruption
to millions around the country today (Wednesday 8 November 2023).
In NSW, mobile and landline services were impacted, including an
impact on calling Triple Zero (000), especially from fixed landlines.
Service restoration is progressing across all domains - Regional and
remote service restoration will be concurrent with capital city services.
Community members who are unable to contact emergency services due to
the outage are urged to try another device, or Telstra payphones until
their service is restored.
Family, friends and neighbours of vulnerable community members are
urged to check in on them and alert emergency services if they need
assistance.
There are no major disruptions to the operations of NSW Police Force,
Rural Fire Service, Fire and Rescue NSW, NSW Ambulance, Transport for
NSW or the Department of Education.
Optus customers will not be able to reach the NSW SES on 132500 on
mobile phones or landlines until the outage is resolved.
Mobile phones will connect to Triple Zero 000, which people should
call for life-threatening situations, provided there is an alternative
telecommunications network available.
Service NSW can be reached on 13 77 88. There are likely to be longer
customer wait times.
A statement from Optus is included below.
OPTUS STATEMENT:
https://www.optus.com.au/content/optus/en/notices/outage.html
We're aware of an issue impacting Optus Mobile, Fixed Internet and
Phone services
Latest updates:
Some internet and phone services are gradually being restored.
This may take a few hours for all services to come back online and
different services may restore at different sites over that time. No
action is required, and services will be automatically restored.
We are aware of some mobile phones having issues connecting to 000. If
Optus customers need to call emergency services, we suggest finding a
family member or neighbour with an alternative device.
We apologise to customers for the nationwide service outage that has
occurred this morning.
We will continue to provide updates as we have information available.
https://www.police.nsw.gov.au/news/news_article?sq_content_src=%2BdXJsPWh0dHBzJTNBJTJGJTJGZWJpenByZC5wb2xpY2UubnN3Lmdvdi5hdSUyRm1lZGlhJTJGMTA5MDkxLmh0bWwmYWxsPTE%3D
Media Statement - Triple Zero unaffected by Optus outage
/Wednesday, 08 November 2023 10:05:31 AM/
*/Due to the ongoing Optus Network outage, the NSW Police would like
to reassure the community that there are no issues affecting Triple
Zero (‘000’) phone calls./*
*/Police urge anyone requiring urgent assistance to contract Triple
Zero. /*
*/For all other non-emergencies please continue to contact Crime
Stoppers on 1800 333 000, the Police Assistance line 131 444 or your
local police station./*
https://www.police.nsw.gov.au/news/news_article?sq_content_src=%2BdXJsPWh0dHBzJTNBJTJGJTJGZWJpenByZC5wb2xpY2UubnN3Lmdvdi5hdSUyRm1lZGlhJTJGMTA5MDg2Lmh0bWwmYWxsPTE%3D
On 9/11/23 11:33, Kate Lance wrote:
Hi Narelle,
An interesting post on Mastodon from Rob Thomas, supporting the idea
it was
a route reflector overload -
https://mastodon.au/@xrobau/111376847362633903
The problem yesterday started at about 4am, when Optus told the world
'I no
longer have any internet connectivity', and 'Do not send any internet
traffic
to me, at all'. The technical description is that they withdrew ALL of
their
routes from the #DFZ (Which is "The Internet", as seen by all the core
routers
that ACTUALLY control the internet).
However, as a precursor at about 3am there was a hint that things weren't
perfect, as there was a flurry of changes from Optus to the outside world
saying, roughly, 'Something has changed inside my network, but you can
still
keep sending me stuff'.
Now, as two final bits of possibly relevant information, the default for
maximum-prefix on #Cisco #ASR9000 is 1048576 (this number is 'the
number of
routes that can be accepted by this router'), and MOST IMPORTANTLY the DFZ
("the internet") has about 980,000 routes in it at the moment. That's
only 90k
odd routes LESS than the default maximum.
I'd be amazed if Optus has less than 100k internal routes that aren't
visible
to the internet, but are visible internally.
So here's what I think happened. The at 3am, the first core #router was
upgraded, and a new config was put in place. This did not join the network
correctly, and things were half broken. What SHOULD have happened is
that all
the changes should have stopped, and either rolled back, or waited for
further
investigation (the cause being that more than 1mil routes were
visible, causing
it to shut down)
However, someone decided 'Well, maybe if we upgrade the SECOND one,
that'll fix
the first one' at 4am. That broke the SECOND one, and took Optus
completely off
the internet.
(Continued, see next for why this is far worse than it should have been)
.....
Regards, Kate
On Wed, Nov 08, 2023 at 05:33:43PM +1100, Narelle Clark wrote:
Rumour has it was a BGP update from an external source that wasn't
filtered
properly with which the BGP route reflectors then overloaded the internal
routers. Persistently.
It was clearly an internal transport problem arising from an
underlying IP
protocol. BGP fits that bill completely as it would be redistributed, and
clearly their management network isn't sufficiently out of band. Once a
network of that scale goes down like that, you can't just turn it back on
and expect it to all work fine - millions of devices all want to
re-register at once, and all those state changes across the network
have to
converge...
Narelle
On Wed, 8 Nov 2023 at 10:30, Alex (Maxious) Sadleir <[email protected]>
wrote:
Around 4am, Optus networks re-announced all their BGP routes at once
https://radar.cloudflare.com/routing/as7474
https://radar.cloudflare.com/routing/as4804
This is indicative of a change management malfunction, akin to IBM's
2016 eCensus routers restarting with no routes
https://www.itnews.com.au/news/ibm-treasury-in-settlement-talks-over-census-failure-440066
The VoWifi infrastructure seems to be online but unable to connect any
calls
https://goughlui.com/2023/11/08/breaking-optus-nationwide-outage-08-11-2023/
More alarmingly 000 doesn't work on landlines
https://twitter.com/lucethoughts/status/1722029287727825124 contrary
to advice from emergency services
https://twitter.com/nswpolice/status/1722028862161449151
On Wed, Nov 8, 2023 at 10:08 AM Tom Worthington
<[email protected]> wrote:
Any more news on what caused the Optus network outage? On ABC Canberra
Radio this morning I suggested it was most likely a software upgrade
which went wrong, and would be fixed by 6pm.
Is VoWiFi working?
I use Telstra, but when COVID-19 struck, I purchased an Optus 4G modem,
with an Optus SIM. This was in case Telstra went down.
--
Tom Worthington http://www.tomw.net.au
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
--
Narelle
[email protected]
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
--
Marghanita da Cruz
Telephone: 0414-869202
Email: [email protected]
Website: http://ramin.com.au
_______________________________________________
Link mailing list
[email protected]
https://mailman.anu.edu.au/mailman/listinfo/link
