Everywhere I've ever pentested, we've used a low/medium/high or low/medium/high/critical scale - this is my first encounter with DREAD. What you describe though - clients attempting to negotiate down the severity of vulns on the report - was common regardless of the scoring system used. I don't see DREAD being unique in that respect.
Reflecting, it's probably what pushed me towards the binary system I ended up using. No score - just exploitable or not. Then, if it's someone that just refuses to have a critical finding on their report, they'll argue over the likelihood of it being exploited. Or over whether the finding was in scope. Then I consider firing that client. I try to explain - when you're hiring a pentester, you're paying for their opinion as they perform a point-in-time assessment. The client is free to debate the findings and they don't have to agree with the consultant, but the client can't change/rewrite the report without destroying the integrity of the report. --Adrian On Thu, Jan 10, 2019 at 5:21 PM Adam Shostack <[email protected]> wrote: > Okay, I'll respond generally about DREAD. The issue comes up when > people say "We'll treat a DREAD rating of >= 8 as critical." Then > someone looks at your discoverability of 7, and says "hmm, if this > were a 6, then DREAD would be 7.9...can we change it?" Lacking any > guidance on the difference, it's hard to say no. > > Really, it's often "You're being unreasonable by making > discoverability a 7 here. It's not that high!" > > Adam > > > > On Thu, Jan 10, 2019 at 04:38:30PM -0500, Adrian Sanabria wrote: > > I probably shouldn't have brought it up - I'm not involved much on the > > pentesting side. I know we've discussed replacing it, but finding little > out > > there to replace it with. > > > > In my own work, I find most of my pentesting results come down to a > binary > > value (hackable, not hackable) and some sense of likelihood of it getting > > exploited by a malicious party. Highs/mediums/lows all seem pointless > when > > emulating the attacker perspective. > > > > Looking at DREAD, I honestly can't say I find anything fatally wrong > with it. > > Perhaps it's because I've never known pentesting to be terribly > consistent > > across tests or consultants in my career, so the bar is set pretty low > in my > > mind? > > > > --Adrian > > > > On Thu, Jan 10, 2019 at 2:12 PM Adam Shostack <[email protected]> wrote: > > > > On Wed, Jan 09, 2019 at 08:18:48AM -0500, Adrian Sanabria wrote: > > > Our pentesters use DREAD, which I think most people have moved on > from, > > but at > > > least the scoring is clear and consistent. > > > > I'm sorry, but I need to rant a little. > > > > A decade back, I wrote a "DREAD is DEAD, please stop" blog post for > > Microsoft. If you are getting consistent scoring out of DREAD, you > > are not using DREAD (as described in Writing Secure Code 1, which I > > think is the first public description). > > > > You are using some derivitive that adds tools to provide for > > that consistency. Those tools may be as simple as a set of examples > > of each of the constiuents and what constitutes a 7 or a 3. > > > > I care about this because I one of the biggest things that I see > > making threat modeling hard is everyone calls their specific > > collection of techniques 'lightweight threat modeling' or 'agile > > threat modeling' and people trying to learn get confused because > > there's 6 contradictory descriptions that have been labeled "agile > > tm". People writing down process so their engineers can do it > > consistently get confused in the same way they'd get confused if we > > all said "oh yeah. we're writing code, and you can assign variables > > with either = or <=". We name our languages, we version them. We > need > > to start doing the same for threat modeling constructs. > > > > If you say "We're using DREADNOP 1.0" that's cool. Alternately, > maybe > > you're using DREAD 1.0 in its raw form, in which case, how are you > > getting consistent scores? > > > > Adam > > > > > > > In addition to CVE being wrong on critical details, I've found > that most > > of > > > ExploitDB isn't exploits. Many are vulnerability checks and almost > all > > are > > > incorrectly entered. PrivEsc will be labeled RCE and RCE will be > labeled > > DoS. > > > It's all a mess. If I had the resources to burn it all down and > start > > from > > > scratch, I'd do it. > > > > > > --Adrian > > > > > > On Tue, Jan 8, 2019, 17:47 Nathaniel Ferguson <[email protected] > wrote: > > > > > > > They use a ton of big words in the paper to call CVSS out > and give > > it a > > > shellacking. Like most of you, we have extensive use of CVSS > in our > > > consulting practice and I've seen this stuff first hand. CVSS > is of > > course > > > just a buggy compression algorithm for taking complex > qualitative > > data and > > > then putting it on a number line. > > > > > > > > > Over the years I've worked at a few different consultancies > and at > > least > > > originally basically no one used any sort of standardized > metric, the > > > reports were generally humorous from a technical standpoint as > the > > numbers > > > were basically just made up and didn't adhere to even basic > > statistics > > > methodologies-- we take the X and multiple it by Y and add the > Z and > > > there's your score! Some even plotted them along cartoon > looking > > graphs and > > > plots and my suspicion was that they were really included to > give > > depth to > > > the reports and break up the monotony of text on text on text. > Oddly, > > I've > > > never even worked at a place that described the methodology as > > outlined in > > > their reports to their employees leaving some question as to > how a > > > methodology was to be implemented if only the client ever > heard about > > it. > > > > > > In that sense, CVSS et al make some amount of sense, if > nothing else > > it > > > standardizes to a common metric and isn't what the sales guy or > > operations > > > manager made up. Additionally, what a strange word-- > shellacking, > > there is > > > no 'k' in the word until its made into a present participle. > > > > > > > The paper has three angles here: > > > > Qualitative mappings into quantitative numbers are a silly > thing to > > do, > > > like people trying to do "social science" by using > SurveyMonkey. > > > > > > Which is what most people are or were selling. > > > > > > > It's fine to have a lossy compression algorithm that > emphasizes > > certain > > > aspects of the input signal over others, of course, but an > additional > > CERT/ > > > CC critique is we have no reason to think CVSS does this in any > > useful way. > > > > > > Well there 's a missing line here, you can see it from the way > that > > > client-side attacks perverted the concept of remote and so > they made > > them > > > remote also instead of adding the new line to the plot. > Because of > > stuff > > > like this. everything is remote now which limits its > usefulness. This > > > doesn't even touch on the fact that most of the CVE database is > > basically > > > wrong from submissions including very limited data, id est > "memory > > > corruption results in a DoS". > > > > > > Nathaniel > > > > > > 在 2019-01-09 00:14:00，"Dave Aitel" <[email protected]> > 写道： > > > > > > > > > I wanted to take a few minutes and do a quick highlight of > a > > paper from > > > CMU-CERT which I think most people have missed out > on: https:// > > > resources.sei.cmu.edu/asset_files/WhitePaper/ > > 2018_019_001_538372.pdf > > > > > > Towards Improving CVSS - resources.sei.cmu.edu > > > resources.sei.cmu.edu > > > SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY > > > REV-03.18.2016.0 Distribution Statement A: Approved for > Public > > Release; > > > Distribution Is Unlimited TOWARDS IMPROVING CVSS > > > It's almost as funny a read as their previous best work on > how " > > > clientless HTTPS VPNs are insanely dumb what were you > thinking > > omg?" > > > > > > They use a ton of big words in the paper to call CVSS out > and > > give it a > > > shellacking. Like most of you, we have extensive use of > CVSS in > > our > > > consulting practice and I've seen this stuff first hand. > CVSS is > > of > > > course just a buggy compression algorithm for taking > complex > > > qualitative data and then putting it on a number line. The > paper > > has > > > three angles here: > > > 1. Qualitative mappings into quantitative numbers are a > silly > > thing to > > > do, like people trying to do "social science" by using > > > SurveyMonkey. > > > 2. We're pretty sure that the compression algorithm is > not, in > > fact, > > > putting higher risk items as bigger numbers, which is > the > > whole > > > point of the thing. > > > 3. Nobody is applying this in any sort of consistent way > (which > > is > > > probably impossible) which is ALSO the whole point of > the > > thing. > > > > > > It's fine to have a lossy compression algorithm that > emphasizes > > certain > > > aspects of the input signal over others, of course, but an > > additional > > > CERT/CC critique is we have no reason to think CVSS does > this in > > any > > > useful way. > > > > > > > > > There's definitely people in the CVSS process (who I will > avoid > > calling > > > out by name) who think ANY quantization is good. But read > the > > paper and > > > decide for yourself - because these are probably serious > issues > > that > > > are turning your entire risk org into a > Garbage-In-Garbage-Out > > org... > > > > > > > > > -dave > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Dailydave mailing list > > > [email protected] > > > https://lists.immunityinc.com/mailman/listinfo/dailydave > > > > > > > > _______________________________________________ > > > Dailydave mailing list > > > [email protected] > > > https://lists.immunityinc.com/mailman/listinfo/dailydave > > > > > > -- > > Adam Shostack > > President, Shostack & Associates > > https://associates.shostack.org • +1 917 391 2168 > > > > Join my very quiet annnoucement list: > https://adam.shostack.org/newthing > > > > > > -- > Adam Shostack > President, Shostack & Associates > https://associates.shostack.org • +1 917 391 2168 > > Join my very quiet annnoucement list: https://adam.shostack.org/newthing > >
_______________________________________________ Dailydave mailing list [email protected] https://lists.immunityinc.com/mailman/listinfo/dailydave
