P.S. Apologies for the second message. My sentence below should instead have read "See their Table 1, which yielded larger effect sizes for psychotherapy for indicators that were more objective (e.g., arrest data) than for self-reports or observer (not objective, as I initially wrote) ratings." All the best...Scott
Scott O. Lilienfeld, Ph.D. Department of Psychology, Room 473 Emory University 36 Eagle Row, Atlanta, Georgia 30322 [email protected]; 404-727-1125 -----Original Message----- From: Lilienfeld, Scott O [mailto:[email protected]] Sent: Tuesday, September 13, 2011 10:22 AM To: Teaching in the Psychological Sciences (TIPS) Subject: RE: RE:[tips] Clinical training: Boulder and Denver Mike Williams wrote that " Since all the dependent measures involve a judgement by the patient or the investigator that the disorder got better or worse, they are all influenced by the expectation bias that the treatment worked." The assertion ("all of the dependent measures...") in the first half of Mike's sentence is false. Many psychotherapy outcome studies in the child and adult literature have used observer ratings, including ratings by family members and friends (what Kazdin termed "social validation"), measures of out-of-session behaviors, or both, and found positive and clinically robust results. See just one example in the child literature: http://www.springerlink.com/content/u1mg416h25n29405/fulltext.pdf See their Table 1, which yielded larger effect sizes for psychotherapy for indicators that were more objective (e.g., arrest data) than for self-reports or objective ratings. See also meta-analyses of the literature on psychotherapy for anger problems: http://onlinelibrary.wiley.com/doi/10.1093/clipsy.10.1.70/pdf which shows approximately equal effect sizes for self-versus observer reports, as well as marked effects on physiological indicators. There are many other examples. Again, there are plenty of interesting issues to raise here (e.g., degree of blindness of observers to treatment assignment, susceptibility of physiological indicators to expectancy effects), but Mike's hypothesis that the results are due simply to subject expectancies and/or demand characteristics does not square with an enormous body of meta-analytic data on psychotherapy outcome showing positive effects on observer ratings (which are often conducted blindly in RCTs of psychotherapy outcome) and largely objective indicators of real-world behavior. Mike W. is right to raise useful questions regarding the internal validity of psychotherapy designs, but I agree with Mike P. that he is wrong to categorically dismiss all of them simply as "invalid." Surely, no study is perfect, but many of them yield highly useful inferences. In addition to Mike P.'s endorsement of Don Campbell's writings on internal validity, I'd like to add Campbell's helpful principle of the heterogeneity of irrelevancies. The most helpful inferences derive the consilience of multiple independent studies, all with largely offsetting flaws. Scott O. Lilienfeld, Ph.D. Department of Psychology, Room 473 Emory University 36 Eagle Row, Atlanta, Georgia 30322 [email protected]; 404-727-1125 -----Original Message----- From: Mike Palij [mailto:[email protected]] Sent: Tuesday, September 13, 2011 9:26 AM To: Teaching in the Psychological Sciences (TIPS) Cc: Mike Palij Subject: RE:[tips] Clinical training: Boulder and Denver Just a few points: (1) I guess that before going to bed folks should read Campbell and Stanley and/or Cook and Campbell and/or Shadish's updating of their material because the issues raised by Mike Williams fall under the topic of "threats to internal validity". Specifically, (a) diffusion or imitation of treatments and (b) compensatory rivalry or resentful demoralization. In both of these situations, knowledge of the different treatments used in a between-subjects design, can affect how subjects respond or fail to respond to treatments. It should be noted that drug treatment studies can be conducted with within-subject designs such as crossover designs where one group receives a drug treatment first and, after a washout period, receives a placebo treatment. Another group has placebo first and drug later. In any event, a competent researcher will make sure that the design they use addresses threats to the different types of validity involved in the study and try to make sure that their effect is negated or minimized. Here is link to Pedhazur & Schmelkin's treatment of validity issues in their text "Measurement, Design, and Analysis": http://tinyurl.com/PedhazurIntValid (2) It might be a somewhat useful to follow the research heuristic that "all treatment/medication studies involving human are invalid" but, as with all heuristics, there will be situations where it fails and situations where it is right but for the wrong reasons. In the example that Williams uses below, it is clear that in an inpatient/ward study, one has to guard against the threats I identify above. But if one uses an outpatient population where the participants have no contact with each other, it is hard to see the merit in Williams' critique. There are many different ways to conduct a drug/treatment/intervention study and the quality of that research is dependent upon what the person in charge knows about research design, the different types of validities involved as well as threats to them, and the degree of control one has over the research situation. (3) A minor point: I would assert that though one's own personal experience is, perhaps, a useful guide to think about things, it does not necessarily constitute a valid guide. It is helpful to remember that "anecdotes do not constitute data" (unless, of course, one is doing a qualitative study where subject's statements are the data). Williams does not provide the results of the study he was involved which is odd because he suggests that the study should have produced statistically significant results. If the study did, the logical follow-up question would be was the effect obtained with other types of designs (e.g., out patient studies where subjects did not have contact with each other). The objection that Williams makes is limited to a particular research situation and does not hold for other designs where the threats are not relevant. This one point invalidates the "universality" of his claims that all drug studies are all invalid. (4) I have conducted the statistical analysis for a few drug studies as represented in the following publications: (a) The effectiveness of antianxiety medications (after participating receiving a sodium lactate infusion to see if it provoked a panic attack): http://tinyurl.com/Liebowitz1984 (b) Whether fenfluramine positively affects behavior and attention in young children with autism: http://www.springerlink.com/content/t152543684578671/ (c) The use of bromocriptine in the treatment of cocaine addiction: http://onlinelibrary.wiley.com/doi/10.1111/j.1521-0391.1997.tb00392.x/abstract One of the things that one learns from analyzing the data from such studies is that subjects can have very different responses to treatment, sometimes contrary to what "commonsense" would lead one to expect, such as finding that the placebo condition has MORE side effects than the drug condition. One should also keep in mind that though one can engage in a "guessing game" about which treatment one is receiving, people are not 100% accurate in making such predictions and there probably are drugs where the effects are not perceptible or the effects build up slowly over time and the day to day changes are not noticeable (think of the frog in pot on the stove where the heat is low but eventually comes to a boil). In summary, many research studies are flawed, some fatally flawed and useless while the flaws in others are not fatal. One can always check the systematic reviews on the Cochrane Collaboration website (www.cochrane.org ) to see how many crappy treatment/intervention studies have been published because the researchers did not address the different types of threats to the validity of the study. Trying to claim that all studies are invalid or all studies are valid is logically invalid from an inductive perspective -- it is as foolish as claiming that "All swans are white". Those without experience with black swans will swear the "all swans are white" if that has been their lifelong experience. -Mike Palij New York University [email protected] ------------------ Original Messages ------------------------------ On Mon, 12 Sep 2011 23:24:29 -0700, Mike Williams wrote: Hello All. When I was a grad student, we were conducting a clinical trial of Imipramine vs Xanex in the treatment of severe depression. The study was conducted on an inpatient research unit in the hospital. The patients lived there and I noticed that they would sit in the day room in the evenings and discuss their treatment. Although the medications were assigned randomly and the researchers did not know the assignment, the patients with dry mouth and constipation knew they were taking the medications. Those given placebo knew this because they did not suffer constipation and dry mouth (the anticholinergic side effects). The patients knew which treatment they were receiving and they communicated this to the investigators because the investigators constantly monitored the side effects. The constant monitoring of side effects unblinds the study. This happens in every clinical trial of psychotropic medications. This problem is even more obvious in every clinical trial of psychotherapy. All these studies are invalid. I could explain why they are invalidated by referring to the gigantic literature on expectation biases. Since all the dependent measures involve a judgement by the patient or the investigator that the disorder got better or worse, they are all influenced by the expectation bias that the treatment worked. I think many subjects want to help the researchers and they endorse small positive changes on the dependent measures. The people who get placebo behave consistent with this because they know they never got treatment. All the investigators have to do is anonymously survey the subjects. The results will blow their minds. To my knowledge, this obvious, simple assessment has never been made. Now you may be able to understand why the treatment effect size today for antidepressants is the same as the placebo effect for some studies in the past - its all noise. Mike Williams ______________________________________________________ Hi Mike: This is a very interesting point but I am not sure that I follow the argument completely. Please expand your argument, dotting the 'i's and crossing the 't's. Ken On 9/12/2011 3:00 AM, Mike Wiliams wrote: Clinical Psychology psychotherapy and psychotropic medication therapies will never have sufficient empirical support simply because the subjects are never blind to the treatment condition. ************************* All the investigators are doing is training the subjects to endorse change on the dependent measures. ************************** That's why the meta-analyses conclude that any therapy is effective. I have never seen an analysis that addressed this research problem. It's similar to the obesity researchers who never notice that their entire field is based on the dieting behavior of young women. Mike Williams Drexel University --------------------------------------------------------------- --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13509.d0999cebc8f4ed4eb54d5317367e9b2f&n=T&l=tips&o=12615 or send a blank email to leave-12615-13509.d0999cebc8f4ed4eb54d5317367e9...@fsulist.frostburg.edu ________________________________ This e-mail message (including any attachments) is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this message (including any attachments) is strictly prohibited. If you have received this message in error, please contact the sender by reply e-mail message and destroy all copies of the original message (including attachments). --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13509.d0999cebc8f4ed4eb54d5317367e9b2f&n=T&l=tips&o=12619 or send a blank email to leave-12619-13509.d0999cebc8f4ed4eb54d5317367e9...@fsulist.frostburg.edu --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=12620 or send a blank email to leave-12620-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
