+1800 more stupid points to me. Guess I will just drink a big ol cup of STFU and nurse my bump from the clue stick.
The math still holds, you need more contractors but still save a ton compared to 1/2 day downtime for everyone. Wasn't the downtime more likely 1-2 days in this case if you were hit anyway.
37,500 systems need touching
6 * 30 = 180 days ( Yeah, that thump hurts my head )
180 - ( 6*8 + 2 ) = 130
500 sites / 6 sites pp = 83 p
6 sites * 75 systems = 450 systems per person
450 pp / 130 days = 3.5 systems per day per person
130 days at .5 days patching = 65 days of lost productivity
65 days at 8 hrs day = 520 hrs
520 hrs at $30 hr burden = $15,600
$15,600 * 83 = $1.3 million in lost time patching
Compared to the very conservative 4 million lost otherwise?
Add another million to the 1.3 mil to hire contractors and you still save almost 2 million.
Jason wrote:
Lets get real with the numbers here. I think these are on the generous side mind you.
Background statement.
** If you have systems worth protecting you hire people capable of protecting them **
I think I will add a new one
** If you have 150,000 systems then you most likely have someone with the knowledge to make it all work and that someone probably understands staffing, budgets, planning, building a business case, presenting it... **
Paul Schmehl wrote:
When you have 150,000 machines worldwide, having 1% of those unpatched (which is a 99% *success* rate) means you have 1500! vulnerable machines. Most situations that I'm familiar with were in the tens - not even the hundreds - but it only took 10 or 15 machines to take down the entire network due to the nature of that worm. 10 or 15 boxes represents 1/100th of a percent of the total, yet that small number could completely destablize a network and cause untold hours of work for the admins and networking staff.
ok, when you have 150,000 machines worldwide, having 100% of them running some version of SQL that is exposed to the network, any network, is just plain _stupid_. I am referring to even a basic understanding of best practice, with that you can knock out at least 10% of the vulnerable system population... there went 15,000 systems.
Lets use these numbers anyway. Even if there is a 50% penetration rate of systems with situations that require SQL and cannot be mitigated we have 75,000 systems.
Now assuming a 50% automatic patch penetration rate on those 75,000 systems you have 37,500 systems still exposed after 5 days of the release of the patch.
6 months * 30 days = 1800 days to patch
1800 days - 6 months * 8 weekend days per month = 48 days.
add 2 days grace for holiday or whatever and you have 1750 days in which to manually touch all of these systems and patch them.
37,500 systems / ~75 systems per site = 500 sites.
NOTE: systems per site should be much higher in a global org with 150k systems and the ability to patch these without touching every one should be fairly high, at least as high as 20%.
500 sites / ~6 sites per person = ~83 people to patch.
6 sites * 75 systems = 450 systems per person
1750 days / 450 systems = ~4 days per system per person to patch.
On the generous side there are still 3.5 days per 4 day cycle to do normal work.
6 months at .5 days every 4 days for 83 people is only 2.7 days productivity per person lost. 2.7 days equals ~21.6 manhours per person.
If we wanted to maintain that productivity we could hire 4 contractors to patch for 3 months and get 2688 manhours to apply to the problem. At the outrageous rate of $150 an hour that is $403,200 to address the problem in the time available. Add in another outrageous $600,000 for misc expenses and travel and we are out $1,000,000
Given a conservative half a day downtime for only 100,000 of the more likely 150,000 employees at a very conservative average burden of $10 per hour you have spent $4,000,000 in productivity losses alone. This completely ignores costs like lost data, lost confidence, work that has to be redone...
Now anybody who wants to tell me that a 0.01% failure rate in a patching program proves the admins are incompetent is simply ignorant of the issues. I guess it's just impossible for people who don't actually run a large network to grasp the nature of the issues.
I think that failure rate in patching when you reasonably have 4 days per system that needs to be touched proves incompetence.
It proves an inability to manage a network of that size. It proves that project management is lacking. It proves a lack of defined and accepted response plans.
It proves there is no one place that things break down and when several of these places take the same attitude that the problem is insurmountable we end up with slammer causing major problems.
** Oh my gawd, I think I need a bigger clue stick. **
Sorry to mix mails together, it is the same concept...
John Airey wrote:
Imagine a company where a user is told by the IT department that such and such a computer can't be used. He then goes and buys it on his own credit card and claims it back on expenses (this happens more than you realise). Said IT department now has to support the machine that he was told he couldn't have, probably because someone higher up in the organisation says that it has to. This computer will probably consume a disproportionate amount of support time. The irony is that the purchaser will probably then tell you it was a bargain (yeah, right!).
Imagine the same company where the user has requirements, these requirements are fed into the defined and accepted policy, the management understands this policy, and the IT group understands that the world has needs.
Now IT needs to take into account these needs and present a viable solution in a timely manner. The problem here is that IT thinks that they can hide behind security because they "do not have the time" to make these needs a reality. The reality is they do not have time because they do not want to solve the problem, this end up costing them more time and forcing them to do things they do not want to do.
The bottom line is that these days, the IT departments do not have enough power to enforce any radical suggestions. I'd be surprised if any organisation exists (outside of the military) that insists on knowing the MAC addresses of machines before they get connected to the network. (In our case we monitor MAC addresses instead as we can then spot network
problems).
:-)
I know of several, the largest one has 13,000 _client_ systems, nearly 5000 of them laptops.
Yes there are times when the fire drill has to be run to meet the needs of some business critical project however these are the exception not the rule.
Then you say, what happens when these laptops are roaming on the company network or at a different office? This is pretty easy to handle actually, roaming gets handled by designating conference rooms and temporary office space as hostile networks and forcing having these connections use a VPN to get to _anything_ they need, otherwise it is just like connecting to your home cable modem only with better security for the clients and tighter auditing. You hop onto a wireless segment you need to
* auth to the AP * auth to the firewall * VPN to use corp services
Failure to VPN only gets you authenticated web browsing.
The truth is that all sysadmins are all involved in damage limitation,
which
is why we subscribe to this list. We do our utmost to prevent damage, but recent history shows us just one user clicking on a dodgy email attachment can bring down major networks. In other cases not knowing what a firewall should and shouldn't do has caused other outages (even affecting
Microsoft).
Only because best practices were not implemented. Had that been done our damage would have been significantly reduced and exposure nearly completely eliminated.
Not knowing what a firewall should do... Sorry, no sympathy here.
After all, if what has been suggested is true and has been implemented, why bother to subscribe to this list?
* new vulns can be evaluated against our environment and a mitigation/resolution can be implemented * things change, we need to stay up to date * Someone can say I had that problem once, here is how it is mitigated * we learn these things we did not know about that can be implemented today
_______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.netsys.com/full-disclosure-charter.html
_______________________________________________ Full-Disclosure - We believe in it. Charter: http://lists.netsys.com/full-disclosure-charter.html
