http://slate.msn.com/id/2086617/
hey, wait a minute The conventional wisdom debunked.
Faith-Based Fudging
How a Bush-promoted Christian prison program fakes success by massaging
data.
By Mark A.R. Kleiman
Posted Tuesday, August 5, 2003, at 9:35 AM PT
The White House, the Wall Street Journal, and Christian conservatives
have been crowing since June over news that President George W. Bush's
favorite faith-based initiative is a smashing success.
When he was governor of Texas, Bush invited Charles Colson's Prison
Fellowship to start InnerChange Freedom Initiative, a fundamentalist
prison-within-a-prison where inmates undergo vigorous evangelizing,
prayer sessions, and intensive counseling. Now comes a study from the
University of Pennsylvania's Center for Research on Religion and Urban
Civil Society reporting that InnerChange graduates have been rearrested
and reimprisoned at dramatically lower rates than a matched control
group.
For those who know how hard it is to reduce recidivism, the reported
results were impressive. Colson celebrated the report by visiting the
White House for a photo op with the president. House Majority Leader Tom
DeLay issued a triumphal press release. The Journal smacked critics of
faith-based programs for turning a blind eye to science by opposing
InnerChange. The report heartened officials in the four states that have
InnerChange programs and buttressed President Bush's plan to introduce
the Christian program in federal prisons.
You don't have to believe in faith-healing to think that an intensive
16-month program, with post-release follow-up, run by deeply caring
people might be the occasion for some inmates to turn their lives around.
The report seemed to present liberal secularists with an unpleasant
choice: Would you rather have people saved by Colson, or would you
rather have them commit more crimes and go back to prison?
But when you look carefully at the Penn study, it's clear that the
program didn't work. The InnerChange participants did somewhat worse than
the controls: They were slightly more likely to be rearrested and
noticeably more likely (24 percent versus 20 percent) to be reimprisoned.
If faith is, as Paul told the Hebrews, the evidence of things not seen,
then InnerChange is an opportunity to cultivate faith; we certainly
haven't seen any results.
So, how did the Penn study get perverted into evidence that InnerChange
worked? Through one of the oldest tricks in the book, one almost
guaranteed to make a success of any program: counting the winners and
ignoring the losers. The technical term for this in statistics is
selection bias; program managers know it as creaming. Harvard public
policy professor Anne Piehl, who reviewed the study before it was
published, calls this instance of it cooking the books.
Here's how the study got adulterated.
InnerChange started with 177 volunteer prisoners but only 75 of them
graduated. Graduation involved sticking with the program, not only in
prison but after release. No one counted as a graduate, for example,
unless he got a job. Naturally, the graduates did better than the control
group. Anything that selects out from a group of ex-inmates those who
hold jobs is going to look like a miracle cure, because getting a job is
among the very best predictors of staying out of trouble. And inmates who
stick with a demanding program of self-improvement through 16 months
probably have more inner resources, and a stronger determination to turn
their lives around, than the average inmate.
The InnerChange cheerleaders simply ignored the other 102 participants
who dropped out, were kicked out, or got early parole and didn't finish.
Naturally, the non-graduates did worse than the control group. If you
select out the winners, you leave mostly losers.
Overall, the 177 entrants did a little bit worse than the controls. That
result ought to discourage InnerChange's advocates, but it doesn't
because they have just ignored the failure of the failures and focused on
the success of the successes.
The Penn study doesn't conceal the actual poor outcome: All the facts
reported above come straight from that report. But the study goes out of
its way to put a happy face on the sad results, leading with the
graduates-only figures before getting to the grim facts. Apparently, the
Prison Fellowship press office simply wrote a press release off the spin,
and the White House worked off the press release. Probably no one was
actually lying; they were just believing, and repeating as fact, what
they wanted to believe. It's hard to know for sure what those involved
were thinking: Study author Byron Johnson canceled a scheduled interview
at the last moment. The White House didn't respond to requests for
comment.
InnerChange program manager Jerry Wilger says he doesn't know much about
research, but he doesn't think it's fair to count the performance of the
people who dropped out of his program against him, a fair-sounding
objection that
