Okay, I'll respond generally about DREAD. The issue comes up when people say "We'll treat a DREAD rating of >= 8 as critical." Then someone looks at your discoverability of 7, and says "hmm, if this were a 6, then DREAD would be 7.9...can we change it?" Lacking any guidance on the difference, it's hard to say no.
Really, it's often "You're being unreasonable by making discoverability a 7 here. It's not that high!" Adam On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote: > I probably shouldn't have brought it up - I'm not involved much on the > pentesting side. I know we've discussed replacing it, but finding little out > there to replace it with. > > In my own work, I find most of my pentesting results come down to a binary > value (hackable, not hackable) and some sense of likelihood of it getting > exploited by a malicious party. Highs/mediums/lows all seem pointless when > emulating the attacker perspective. > > Looking at DREAD, I honestly can't say I find anything fatally wrong with it. > Perhaps it's because I've never known pentesting to be terribly consistent > across tests or consultants in my career, so the bar is set pretty low in my > mind? > > --Adrian > > On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <[email protected]> wrote: > > On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote: > > Our pentesters use DREAD, which I think most people have moved on from, > but at > > least the scoring is clear and consistent. > > I'm sorry, but I need to rant a little. > > A decade back, I wrote a "DREAD is DEAD, please stop" blog post for > Microsoft. If you are getting consistent scoring out of DREAD, you > are not using DREAD (as described in Writing Secure Code 1, which I > think is the first public description). > > You are using some derivitive that adds tools to provide for > that consistency. Those tools may be as simple as a set of examples > of each of the constiuents and what constitutes a 7 or a 3. > > I care about this because I one of the biggest things that I see > making threat modeling hard is everyone calls their specific > collection of techniques 'lightweight threat modeling' or 'agile > threat modeling' and people trying to learn get confused because > there's 6 contradictory descriptions that have been labeled "agile > tm". People writing down process so their engineers can do it > consistently get confused in the same way they'd get confused if we > all said "oh yeah. we're writing code, and you can assign variables > with either = or <=". We name our languages, we version them. We need > to start doing the same for threat modeling constructs. > > If you say "We're using DREADNOP 1.0" that's cool. Alternately, maybe > you're using DREAD 1.0 in its raw form, in which case, how are you > getting consistent scores? > > Adam > > > > In addition to CVE being wrong on critical details, I've found that most > of > > ExploitDB isn't exploits. Many are vulnerability checks and almost all > are > > incorrectly entered. PrivEsc will be labeled RCE and RCE will be labeled > DoS. > > It's all a mess. If I had the resources to burn it all down and start > from > > scratch, I'd do it. > > > > --Adrian > > > > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <[email protected] wrote: > > > > > They use a ton of big words in the paper to call CVSS out and give > it a > > shellacking. Like most of you, we have extensive use of CVSS in our > > consulting practice and I've seen this stuff first hand. CVSS is of > course > > just a buggy compression algorithm for taking complex qualitative > data and > > then putting it on a number line. > > > > > > Over the years I've worked at a few different consultancies and at > least > > originally basically no one used any sort of standardized metric, > the > > reports were generally humorous from a technical standpoint as the > numbers > > were basically just made up and didn't adhere to even basic > statistics > > methodologies-- we take the X and multiple it by Y and add the Z and > > there's your score! Some even plotted them along cartoon looking > graphs and > > plots and my suspicion was that they were really included to give > depth to > > the reports and break up the monotony of text on text on text. > Oddly, > I've > > never even worked at a place that described the methodology as > outlined in > > their reports to their employees leaving some question as to how a > > methodology was to be implemented if only the client ever heard > about > it. > > > > In that sense, CVSS et al make some amount of sense, if nothing else > it > > standardizes to a common metric and isn't what the sales guy or > operations > > manager made up. Additionally, what a strange word-- shellacking, > there is > > no 'k' in the word until its made into a present participle. > > > > > The paper has three angles here: > > > Qualitative mappings into quantitative numbers are a silly thing > to > do, > > like people trying to do "social science" by using SurveyMonkey. > > > > Which is what most people are or were selling. > > > > > It's fine to have a lossy compression algorithm that emphasizes > certain > > aspects of the input signal over others, of course, but an > additional > CERT/ > > CC critique is we have no reason to think CVSS does this in any > useful way. > > > > Well there 's a missing line here, you can see it from the way that > > client-side attacks perverted the concept of remote and so they made > them > > remote also instead of adding the new line to the plot. Because of > stuff > > like this. everything is remote now which limits its usefulness. > This > > doesn't even touch on the fact that most of the CVE database is > basically > > wrong from submissions including very limited data, id est "memory > > corruption results in a DoS". > > > > Nathaniel > > > > 在 2019-01-09 00:14:00，"Dave Aitel" <[email protected]> 写道： > > > > > > I wanted to take a few minutes and do a quick highlight of a > paper from > > CMU-CERT which I think most people have missed out on: https:// > > resources.sei.cmu.edu/asset_files/WhitePaper/ > 2018_019_001_538372.pdf > > > > Towards Improving CVSS - resources.sei.cmu.edu > > resources.sei.cmu.edu > > SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY > > REV-03.18.2016.0 Distribution Statement A: Approved for Public > Release; > > Distribution Is Unlimited TOWARDS IMPROVING CVSS > > It's almost as funny a read as their previous best work on how " > > clientless HTTPS VPNs are insanely dumb what were you thinking > omg?" > > > > They use a ton of big words in the paper to call CVSS out and > give it a > > shellacking. Like most of you, we have extensive use of CVSS in > our > > consulting practice and I've seen this stuff first hand. CVSS is > of > > course just a buggy compression algorithm for taking complex > > qualitative data and then putting it on a number line. The paper > has > > three angles here: > > 1. Qualitative mappings into quantitative numbers are a silly > thing to > > do, like people trying to do "social science" by using > > SurveyMonkey. > > 2. We're pretty sure that the compression algorithm is not, in > fact, > > putting higher risk items as bigger numbers, which is the > whole > > point of the thing. > > 3. Nobody is applying this in any sort of consistent way (which > is > > probably impossible) which is ALSO the whole point of the > thing. > > > > It's fine to have a lossy compression algorithm that emphasizes > certain > > aspects of the input signal over others, of course, but an > additional > > CERT/CC critique is we have no reason to think CVSS does this in > any > > useful way. > > > > > > There's definitely people in the CVSS process (who I will avoid > calling > > out by name) who think ANY quantization is good. But read the > paper and > > decide for yourself - because these are probably serious issues > that > > are turning your entire risk org into a Garbage-In-Garbage-Out > org... > > > > > > -dave > > > > > > > > > > > > > > > > _______________________________________________ > > Dailydave mailing list > > [email protected] > > https://lists.immunityinc.com/mailman/listinfo/dailydave > > > > > _______________________________________________ > > Dailydave mailing list > > [email protected] > > https://lists.immunityinc.com/mailman/listinfo/dailydave > > > -- > Adam Shostack > President, Shostack & Associates > https://associates.shostack.org • +1 917 391 2168 > > Join my very quiet annnoucement list: https://adam.shostack.org/newthing > > -- Adam Shostack President, Shostack & Associates https://associates.shostack.org • +1 917 391 2168 Join my very quiet annnoucement list: https://adam.shostack.org/newthing _______________________________________________ Dailydave mailing list [email protected] https://lists.immunityinc.com/mailman/listinfo/dailydave
