On Sun, Aug 02, 2015 at 07:53:43PM +0100, Greg Wilson wrote: > (Sorry to be so slow replying to this thread...) > > On 2015-07-23 5:50 PM, Neil Chue Hong (SSI) wrote: >> I've talked to the SoftwareX editors previously, and I think we agree >> that actually the tricky thing here is providing the right tools to >> make reviewing software easier, and that's something where publishers >> can certainly make improvements. > > The most important finding of the studies Marian Petre and I did in > 2013-14 of code review in the sciences is that doing a review when the > software is "done" is neither useful nor affordable: > > * It's not useful because it's too late for the author to act on issues > that the reviewers find. > > * It's not affordable because the time required to review a piece of > code grows very quickly with the size of the code being reviewed. > > I believe that if we want scientists to start doing code reviews, we > have to persuade them to do those reviews *as the code is being > written*, in the same way that most open source projects do it - i.e., > we have to get them to review small incremental patches as they're > written, so that (a) authors can fix problems before they waste time > using the code, and (b) the effort required is as small as possible. If > this is right, changes to tooling alone aren't going to help - instead, > publishers should focus their efforts on changing the review process so > that it runs in parallel with coding and analysis, rather than afterward.
I agree with everything up until the last sentence, which I don't see possibly working ever no way no how are you kidding what? But, rather than leaving things at that unhelpful statement, here's a helpful suggestion :) What about drawing an analogy to the pregistered study model -- https://osf.io/8mpji/wiki/home/?_ga=1.189671019.1900172679.1438548591 and basically saying that software publications are virtually guaranteed if the *methodology* of software development is reviewed as part of an initial submission? Then sometime later (after the software has reached most of its specified milestones) the publication can happen with only grammatical review of the writeup. It doesn't necessarily work for early stage software projects where the success or failure of the basic idea is in question, but I'd certainly do that for my lab's current software effort (khmer & screed) and my next project (tentatively named buoy). cheers, --titus _______________________________________________ Discuss mailing list [email protected] http://lists.software-carpentry.org/mailman/listinfo/discuss_lists.software-carpentry.org
