Paul, Here are a couple of passages that I have found cause problems:
In particular, when an analysis uses methods to address missing data such as regression imputation, maximum likelihood, or non-response weights, the review process described in the last subsection of Section C (Analyses with Missing Data) should be followed instead, which includes an assessment of potential bias from using imputed data instead of actual data. When the outcome measure is imputed for some subjects in the analytic sample, in addition to (c) and (d), the following data are required: (e) the means of the outcome measure for the subjects in the analytic sample with observed outcome data, separately for the intervention and comparison groups; (f) the means of the outcome measure for the subjects in the analytic sample with observed baseline and outcome data, separately for the intervention and comparison groups; (g) the standard deviations of the outcome measure for either the sample of subjects in the analytic sample with observed outcome data or the sample with observed baseline and outcome data; and (h) the number of subjects with observed outcome data in the analytic sample by condition. If these data are not reported in the study, the WWC will request them from the authors. In practice, this has been interpreted as requiring a complete-case analysis in addition to the researcher's preferred approach. This allows WWC reviewers to essentially override the researcher's approach when they review the study. I worked on a training program trial that used local college records for outcome measurement. The outcome Y was defined =1 if the local records indicated credential receipt and =0 otherwise. It was known that a subset of the sample attended some other college and that the subset was considerable larger in the control sample than in the treatment sample. So, we imputed credential receipt for people who attended other colleges. In this case, a complete case analysis would, I trust you agree, be strongly biased. I was unwilling to prepare impact estimates that treated everyone not attending the local college as failing to have earned a credential. I failed to see how such alternate impact estimates could have been relevant to determining the quality of our impact estimates. --Dave From: Impute -- Imputations in Data Analysis [mailto:[email protected]] On Behalf Of Allison, Paul D Sent: Wednesday, September 2, 2020 7:45 AM To: [email protected] Subject: Re: WWC, Imputation, and JSM Warning from Abt: External email. Be careful opening links and attachments. Hi Dave: I'm not familiar with the WWC, but your email prompted me to take a look at their standards handbook at https://urldefense.com/v3/__https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf__;!!Dq0X2DkFhyF93HkjWTBQKhk!FK42zAwx6nEiF-EG441HwbntI195rgGIYVJYLfiwuz1CXoX1-WS8M9snyXJ1Yc9geltSho0B8BupBQ$ <https://urldefense.com/v3/__https:/ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf__;!!Dq0X2DkFhyF93HkjWTBQKhk!AckH5BhOYbPDmObE0ynScTETKdcEKvZFJF8XFlC6ZoE8vD1nraio173bGltqUL4cSMMuCxA1DROD4Q$> The section on missing data seems to take a more balanced view than your email suggests. Am I missing something? Paul Paul D. Allison, Professor Emeritus Department of Sociology University of Pennsylvania 362 McNeil Building 3718 Locust Walk Philadelphia, PA 19104-6299 215-898-6717 ________________________________ From: Impute -- Imputations in Data Analysis <[email protected]<mailto:[email protected]>> on behalf of David Judkins <[email protected]<mailto:[email protected]>> Sent: Tuesday, September 1, 2020 2:50 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: WWC, Imputation, and JSM Dear ListServe Members, Many of you probably know this, but the What Works Clearinghouse (WWC) in the United States has a strongly anti-imputation set of standards. I have been told my studies will get failing grades unless I redo the analyses without imputed data. I am trying to pull together a draft invited paper/panel session for JSM 2021 on this topic. Anyone interested? --Dave Judkins Abt Associates ________________________________ This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system. ________________________________ This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.
