First, without commenting on the relevance to the original discussion, I want to address the CTF topic. As something I know[1] a bit[2] about[3], I would mention that PPP is the "real deal" in regard to CTF performance. Having won defcon multiple times and won and played in many many others, I assure you that the Korean CTFs are extremely challenging.
PPP is one of the more well-known and respected up and coming CTF teams amongst the folks I know who play a lot of such games. Now, as to whether it's relevant to the AEG work, I leave up for others to debate. Additionally, I would add that I don't know much about the composition of PPP and how it relates specifically to Brumley's group--whether there's total overlap, significant overlap, or only some overlap, but I assume that's an easy thing to discover. Full Disclosure: I just ran a warm-up event[4] for a CTF some friends and I are putting on at ShmooCon and a member of PPP won the competition, so I might have some incentive to speak well of their CTF performance. ;-) Speaking of that performance, check out Andrew Wesie's writeup[5] for the binary exploit he solved in just a few hours for that CTF. That was very solid work. Indeed, read through the writeups for many of the challenges solved on their blog, there's some impressive solutions there. [1] http://nopsr.us/ctf2006/ [2] http://nopsr.us/ctf2007/ [3] http://capture.thefl.ag/2009/DefconCTF/ [4] http://ghostintheshellcode.com/2011/ [5] http://ppp.cylab.cmu.edu/wordpress/?p=410 On Mon, Jan 10, 2011 at 5:04 PM, Kevin Miller <[email protected]> wrote: > Now that the holiday rush has subsided I have taken some time to read this > thread and do some investigation beyond the email content, as I expect some > others have. Several points that are made in the response*, supposedly from > David, seem unwarranted and/or out of place. What purpose does mentioning > highly public names such as Kevin Mitnick and Robert Morris serve? > Likewise, what purpose does mentioning smpCTF serve? smpCTF, WTF is that? > Google easily helps, but really? Why even mention it, simply to mislead > those of us that have only seen CTF at DEFCON?** > > One rebuttal comment brought up the longstanding argument that academics > typically don't solve real world problems and those of us that work in the > real world are typically shunned by academia. This paper specifically > states that they target a practical problem space. If this is indeed true, > why does the response not even address Dave's simple classroom example? > They have acknowledged existence of the thread - why are they not addressing > any of the concerns raised? > > Two whole 0-days! Wow! Are they even remotely useful? Are they ... never > mind, not even worth enumerating. > > Forward symbolic execution? Really? At some point you really need to stop > regurgitating your own thesis work, and increasing your publication and > citation count by having students re-tool said work for conferences[1]. > > Finally, releases are mentioned. Where is this software? Are these > releases in the common academic vernacular: as in a commit exists in some > SVN repo somewhere with a the comment containing the string "release", and > the public will likely never see them. > > I applaud Dave, Sean and others that have similarly called BS on this > paper. Now if only academia would accept some of you into their circles > maybe we would see a truly technical program committee that would reject > such a paper. Alas, this publication and later presentation will surely be > a large win for the students who will use the publication as a requirement > for degrees and for the junior faculty member who is certainly targeting a > quantity over quality approach toward tenure. > > Do list members think that this research group[2] is ill intentioned, or do > they honestly believe that they are making a positive impact ... or even > forward progress? > > Gave up on academic research long ago, > > Kevin > > > * Calling it a response is fairly gracious. If the authors of the paper are > acknowledging questions that have arisen on this mailing list, surely CMU > faculty and students are capable of figuring out how to join the list, if > needed, and reply instead of creating a small html page that will never have > a decent page rank and essentially has no public audit. Offering to address > concerns in person in Pittsburgh? Really? I am certain this is some sort > of humor, I can't figure out exactly how it is supposed to be funny, but it > surely isn't how a CMU professor cops out of addressing public criticism. > > ** at first glance the group seems to have a solid track record in CTF > competitions. Until you dig a little deeper and find out that the ones they > have one are basically substandard and many of them are small competitions > run in Korea. Is it mere coincidence that David's group includes several > Korean exchange students? To be fair, the group apparently did recently win > iCTF in December which I gather actually is an accomplishment. > > [1] http://oakland31.cs.virginia.edu/slides/thanassis_oakland10.pdf > [2] http://security.ece.cmu.edu/people.html > > On Thu, Dec 16, 2010 at 5:02 PM, Sean Heelan <[email protected]> wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I think 'lets create a list of real world problems for academics to >> consider' is missing the point somewhat. The problem here isn't that AEG >> isn't a worthy or difficult problem, it is. The problem is that in order >> to work realistically on AEG (and follow up with claims of changing >> threat models etc) you need a good working knowledge of real world >> exploit writing. The general impression I get from some of the research >> groups working on the problem is they are unwilling to invest the time >> required to gain this knowledge. (To kick a dead horse some more, one >> should feel free to play around in a sandbox of vulnerabilities and >> protection mechanisms from the past decade. It's obviously necessary for >> tool development and trying out new ideas. The problem starts when you >> forget half-way through that you're playing within a sandbox and pretend >> it's the real world in your paper). >> >> Instead of a list of problems to attack how about list of real world >> tasks that the authors should be able to complete manually before >> deciding to automate the process? I would have thought this was a pretty >> sensible thing that most people would do but apparently not. Trying to >> automate a process that you only have a vague idea of doesn't sound like >> something that is every going to go too well. >> >> Its very easy to run off down the path of an under-explored research >> problem (I should know, I've done it :P), it's a lot less glamorous in >> academia to spend weeks/months sitting in front of a debugger in order >> to figure out the subtleties of the problem you are actually addressing. >> This can hardly be considered an excessive request if one is working at >> a well funded research group at a respected university though. Without >> putting in this effort then the research output and paper quality have a >> ceiling in terms of real-world applicability that will never be broken >> through and the disparity between claims and facts will continue to >> induce ong-winded and 'venomous' (chuckle) blog posts. >> >> (Btw, at some point during the discussion a few people began to assume I >> was criticising all of academia. This isn't the case. I was pretty >> specific in my initial blog post (http://bit.ly/ikvR0y) where my issues >> lay and they are with small proportion of the overall research output of >> academia and industry. There is no 'us vs them' here. I would expect >> anyone writing a paper to at least have a cursory understanding of >> everything they discuss. Furthermore, I wasn't criticising the people >> cited in the paper when I suggested less nepotism in citations would be >> useful. I was suggesting that the papers authors perhaps read something >> like Phrack, or Uninformed a little more extensively than 'Smashing the >> Stack for Fun and Profit'. >> >> My general opinion is that academia is both necessary and useful. That's >> partially why I wrote the blog post to begin with - the paper is a >> perfect example of the stereotype many have of academics as people with >> their heads in the clouds dictating to those with their feet on the >> ground. I know this isn't true for many so it's annoying when someone >> comes along and proves it correct. >> >> It's also worth mentioning that CMU's response "If Mr. Heelan feels >> there are real scientific issues to discuss, he is welcome to call or >> visit us at CMU to discuss them." conveniently ignores the fact that I >> did send them an email outlining (yet again) both my issues with the >> paper and some technical issues. I received one reply requesting a phone >> call instead of email, which I declined as real-time conversations on >> technical matters tend to miss a lot IMO, and then never heard anything >> back. No feedback on their heap claims, nothing on their stack fix-ups, >> nothing on their plans to scale to modern bugs/exploits and no response >> to any of the valid complaints raised here. The only point of their >> response.html [1] seems to be to foster the image that my complaints >> stem from an anti-academia sentiment instead of engaging on the issues >> raised here and elsewhere. Hardly the most common way for a research >> group to deal with questions and comments from a pretty sizeable >> proportion of their target audience.) >> >> In hopeful expectation of a productive discussion (or failing that a >> link or two to some funny cat videos), >> >> Sean >> >> [1] http://security.ece.cmu.edu/aeg/response.html >> >> On 12/15/2010 08:54 AM, Miles Fidelman wrote: >> > Anton Chuvakin wrote: >> >>> I would love to see a resource for real-world problems that the >> >>> academic >> >>> community could consider... or even a resource for other up-and-coming >> >>> researchers to examine at for ideas. Such a site might not be >> >>> relevant >> >>> enough for PhD thesis work (which thrives on originality as I >> >>> understand >> >>> it?) >> >>> >> >> Well, if it is created by the industry, the academics will ignore it. >> >> And if created by academics, well, see discussion in this thread. >> >> >> > Call me cynical but.... >> > >> > If it has serious commercial potential, academics may be doing the >> > research, but saving the results for their side/spinout companies. >> > >> > The really interesting research (or least the well funded-research) gets >> > funded by DoD, with classified results, and never gets published. >> > >> > And folks who have serious countermeasures to large spambot networks >> > might just not want their names visible to the unsavory characters who >> > run large spambot networks. >> > >> > Now a list of relevant problems to research would be interesting, but I >> > expect there will be little feedback as to which problems people end up >> > taking on. >> > >> -----BEGIN PGP SIGNATURE----- >> Version: GnuPG v1.4.11 (GNU/Linux) >> Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/ >> >> iQEcBAEBAgAGBQJNCkYoAAoJEMW6jFWLazyQbSMH/3azWDftzsCwVs3H3xvO8YW9 >> OII+v+fa20Jpkqh/KtSx9g4AjvootsxahTXv5e0pqqOIsRwQkP+eemC9xcDs/Kk/ >> BhGnIyvz54tANy2/TgKQZwLTPvkbICfbtyP7gQCr9rKk9DJaC7SyEcKjBDdaDEGF >> jBFXFufjQZqpcF8kYOE7c5sLqYp2Lsfy/Kzroa4lKeoQFDyp5MjMTWzLqzULcRFl >> zfdt8jNbZR3iAGYJdzbhPSFRsfseI69UOKsXLuZwGJUvNThDkyOpvlguqjkYwJ8J >> THQ6ULvNuPLiSLbFLYKPI2KDYMWgjF1DXjswjmfSN8/Zv9RF10K5+n0hBlfp3r8= >> =Wfg6 >> -----END PGP SIGNATURE----- >> _______________________________________________ >> Dailydave mailing list >> [email protected] >> https://lists.immunityinc.com/mailman/listinfo/dailydave > > > _______________________________________________ > Dailydave mailing list > [email protected] > https://lists.immunityinc.com/mailman/listinfo/dailydave > > _______________________________________________ Dailydave mailing list [email protected] https://lists.immunityinc.com/mailman/listinfo/dailydave
