5. SCIENCE AS CRAFT
Woodward and Goodstein question the general validity of the following principle: Scientists must report what they have done so fully that any other scientist can reproduce the experiment or calculation. They claim that science has a large "skill" or "craft" component, and that Conducting an experiment in a way that produces reliable results is not a matter of following algorithmic rules that specify exactly what is to be done at each step. This may be true of some areas in the biological sciences and other experimental sciences in which the behavior of living organisms or the functioning of complicated instrumentation may not be well understood, but this does not apply to computer simulation experiments. We can and must insist on exact reproducibility of simulation experiments; and this should, in fact, be a matter of following precisely stated, fully documented algorithms. There is of course a large "craft" component in building and using simulation models. Different individuals presented with the same system to be modeled neither build identical simulations nor apply those models in precisely the same way, just as different researchers in any other scientific discipline will neither build the same experimental apparatus nor carry out exactly the same experimental protocol to study a given effect. Nevertheless in these situations different simulationists should be able to reproduce each other's results in order to judge the significance and limitations of the conclusions based on the experiments in question. More generally, there is a large "craft" component in doing simulation research just as there is a large "craft" component in doing other types of scientific research--but this state of affairs does not mitigate the need for reproducibility of the main experiments associated with such research. 6. PEERS AND PUBLICATION 6.1 Is the Scientific Paper a Fraud? Woodward and Goodstein cite Peter Medawar's (1991) paper entitled "Is the Scientific Paper a Fraud?" to argue that because most archival papers in the scientific literature do not accurately portray the way scientific research is actually done, these papers fail to measure up to Feynman's ideal of "leaning over backwards." It is certainly true that primary journal articles in the scientific literature do not document all of the mistakes, dead ends, and backtracking that are an inevitable part of virtually every successful scientific investigation. Medawar (1982, p. 92) himself admitted that I reckon that for all the use it has been to science about four-fifths of my time has been wasted, and I believe this to be the common lot of people who are not merely playing follow-my-leader in research. In my view, the fundamental issue here is that there simply is not enough space in all the scientific journals to document the way that science is actually done; moreover no one has the time to absorb all the final results even in a relatively narrow area of specialization, much less to read the associated background material. Nowadays many high school students are sufficiently sophisticated to realize that primary journal articles are vehicles for efficiently communicating significant discoveries rather than for documenting the processes by which those discoveries were made. Moreover, this issue is rapidly becoming moot because of current trends toward complementing the printed version of a primary journal article with comprehensive supporting documentation (such as appendices containing lengthy proofs or detailed descriptions of experimental protocols) archived on a World Wide Web server that is maintained by the journal's sponsoring organization. 6.2 Problems with the Peer Review System Finally Woodward and Goodstein examine the peer review system for evaluation of research proposals and primary journal articles, concluding that the conflict of interest inherent in asking competitors to evaluate each other's work has inflicted genuine distress on the system. In my own experience, by far the most common form of misconduct by peer reviewers has nothing to do with conflicts of interest; instead the problem is simple dereliction of duty by reviewers who cannot be bothered to read and evaluate carefully the work of other researchers. Although this remark applies to evaluation of research proposals as well as refereeing of primary journal articles, I am most concerned with problems in refereeing. In my judgment, the problem of nonperformance by referees has reached epidemic proportions, and I believe it is urgently necessary for the scientific community to address this scandalous state of affairs. In preparing these remarks I solicited comments from numerous colleagues not only in the simulation community but also in the "hard" scientific disciplines, and I have been startled by the vehemence of their agreement with my evaluation of the current state of the refereeing system. Based on numerous conversations with colleagues in biology, electrical engineering, industrial engineering, mathematics, and statistics, I have a sense that problems with refereeing are much worse in these fields than in the simulation community. Perhaps the most egregious failure of the refereeing system in recent years was the publication of the initial paper on cold fusion by Fleischmann and Pons (1989a). This paper was published in the Journal of Electroanalytical Chemistry in just four weeks; and a long list of errata soon followed (Fleischmann and Pons 1989b)--including the name of M. Hawkins, a coauthor who was somehow omitted from the original paper. A detailed account of this infamous episode can be found on pp. 218-220 of Huizenga (1993). 6.3 Refereeing Remedies The two main reasons for breakdowns in the operation of the refereeing system are (a) misconceptions by referees about the job they are supposed to do, and (b) lack of incentives for doing a good job of refereeing. As Gleser (1986) points out, many referees think that a manuscript must be checked line by line for errors; and seeing that this will be extremely time-consuming, they continually put off the task. On the contrary, the referee's main responsibility is to serve the editor as an "expert witness" in answering certain key questions about the manuscript--and most of these questions can be answered under the assumption that the manuscript is error-free. These key questions are given in Table 2 and are elaborated in Forscher (1965), Gleser (1986), and Macrina (1995, pp. 84-89) along with general guidelines for refereeing that should be required reading for every research worker in the field of computer simulation. Table 2: Key Questions to be Answered in a Referee's Report _____ 1. Are the problems discussed in the paper of substantial interest? Would solutions of these problems materially advance knowledge of theory, methods, or applications? 2. Does the author either solve these problems or else make a contribution toward a solution that improves substantially upon previous work? 3. Are the methods of solution new? Can the proposed solution methods be used to solve other problems of interest? 4. Does the exposition of the paper help to clarify our understanding of this area of research or application? Does the paper hold our interest and make us want to give the paper the careful reading that we give to important papers in our area of specialization? 5. Are the topic and nature of this paper appropriate for this journal? Are the abstract and introduction accessible to a general reader of this journal? Is the rest of the paper accessible to a readily identified group of readers of this journal? 6. Are the clarity and readability of the manuscript acceptable? Is the writing grammatically correct? 7. Does the manuscript contain an adequate set of references? Is adequate credit given to prior work in the field upon which the present paper is built? 8. Is the material appropriately organized into an effective mix of text, figures and tables? Are data given in tables better presented in figures or in the text? 9. Is the work technically correct? Are the main conclusions justified by the experimental data and by logically valid arguments? Are the theorems stated and proved correctly given the assumptions? In practical applications of the theoretical results, do the authors check the validity of the underlying assumptions? 10. Are there gaps in the discussion of the experimental methods or results? If there are such gaps, can the closing of these gaps be considered (i) essential, (ii) desirable, or (iii) interesting? Are the experimental methods described in sufficient detail so that other investigators can reproduce the experiments? _____ If a paper passes the initial screening that consists of answering questions 1-8 in Table 2, then it is necessary to undertake the verification of technical correctness required to answer questions 9 and 10. If competent referees had scrutinized the initial paper on cold fusion by Fleischmann and Pons (1989a) with the objective of answering questions 9 and 10 in Table 2, then the fatal flaws in this work would have been uncovered immediately. In my view it is imperative that we protect the simulation literature against the long-lasting stigma that results from permitting the publication of technically incorrect work. If everyone in the simulation community followed the guidelines in Table 2 for preparing referee's reports, then I believe our problems with peer review would largely disappear. Additional tips on effective refereeing are given by Waser, Price, and Grosberg (1992). A set of questions similar to those given in Table 2 can be found on the home page of the ACM Transactions on Modeling and Computer Simulation by using the URL http://www.acm.org/pubs/tomacs/review/review.html. There remains the question of adequate incentives for good refereeing. In reviewing preliminary versions of these remarks, several individuals complained about general lack of editorial feedback on (a) the strengths and weaknesses of their reviews, and (b) the issues identified in other referees' reports on the same paper. As a routine professional courtesy, editors should include such feedback with their letters of appreciation to referees. Moreover, editors should strive to ensure that individuals who provide prompt and thorough refereeing will receive comparable service when those individuals submit their own papers for review. Ultimately refereeing is one of the professional responsibilities that each of us must fulfill to ensure the vitality of our chosen field, but doing this job well should be a source of pride and satisfaction commensurate with that of our other professional contributions to the field.
