>> https://www.abc.net.au/news/2023-11-13/optus-identifies-cause-of-nationwide-outage-software-upgrade/103099902On 13/11/23 17:06, Roger Clarke wrote:This seems to indicate that the interpolations made early by linkers and others were pretty close to what actually happened ...
On 13/11/23 6:11 pm, Tom Worthington wrote:
Yes, no surprises there. Similar to the problem which took out the Australian Population Census in 2016: https://www.abc.net.au/news/2016-08-09/abs-website-inaccessible-on-census-night/7711652 One question which needs to be asked is if Optus is implementing the two man rule. That is, can one person make a change to the system on their own? There needs to be one person input the change and another check it.
One? Onion rings / defence in depth used to mean plurality. (Okay, accountancy debased that to dual-signatory approaches, but ...).
On Sky News I was asked if Optus should have redundancy. It would be possible to replicate all the hardware, but that would double to cost of services to customers and would not stop a systematic failure of this sort.
Redundancy as a general principle is fine. But if one limits their thinking to just one of the elements, even "hardware", they're done.
One surprising outcome is that, in this case, mobile phones were more reliable than land lines for emergency calls. The mobile phone standards have provision for using any company's network to make an emergency call. So phones automatically switched from Optus to Telstra, or Vodaphone.
Shhhhh! Corporate strategists don't know that. If they realise that's the case, they'll strangle the people they sent to Standards Committees, and debase the Standards. (I'm joking of course. Corporate strategists don't read things like link, and wouldn't understand if they did).
The Australian Government is already working on mobile roaming between carriers during natural disasters. https://minister.infrastructure.gov.au/rowland/media-release/government-scope-emergency-mobile-roaming-capability-during-natural-disasters This could be extended to cover other network outages. On a trip to India, I used one telco in Goa, and when they were not available in Bangalore, the phone automatically switched to another network. This is a commercial arrangement between carriers. It would require some difficult commercial and regulatory negotiations to implement in Australia.
ACCC is authorised to, and does, grant requests for approvals of apparent complicity between competitors. But I don't think they receive all that many requests ...
For government, business, and domestic users of internet and phone services there are some clear lessons from the Optus outage. Don't have all your phones and Internet provided by the one company. If you are providing safety critical services, have connections to multiple networks.
Two-SIM handhelds? Rolling, cheap, year-long contracts with someone you can switch to if/when needed? But watch out for unexpected pointc-of-failure / dependencies, non-scalability (Vodefone cope with large-scale and quick switching from a Telstra meltdown??), and the takeover game.
-- Roger Clarke mailto:[email protected] T: +61 2 6288 6916 http://www.xamax.com.au http://www.rogerclarke.comXamax Consultancy Pty Ltd 78 Sidaway St, Chapman ACT 2611 AUSTRALIA
Visiting Professor in the Faculty of Law University of N.S.W. Visiting Professor in Computer Science Australian National University
OpenPGP_signature
Description: OpenPGP digital signature
_______________________________________________ Link mailing list [email protected] https://mailman.anu.edu.au/mailman/listinfo/link
