It's hard to know what I would conclude. Are there other significant effects found in the study with the small sample size? Power is not an issue if you have statistical significance. I have a published paper in which one study has an N of 8, four in each of the two groups. It was significant with a huge effect size (naturally, with such a small sample). People may not believe it (is it robust?) but power is not the reason to doubt it.
With a marginally significant effect (I've never heard the term tendentially used in this context) it is essentially the same problem. It is what it is. They don't have statistical significance. They have a small, but not absurdly small, sample size (how many studies with df = 50 are published out there?). I would be less concerned with the stats on this and more concerned with the claims they make in their discussion about the finding. Do they write their discussion with scant or no real accounting for the fact that it is marginally significant? They need to describe their finding as tentative and suggestive that a future study needs greater control over the IV and possibly increased sample size. If they try to act like they have a big discovery, I'd be requesting a rewrite. In fact, an important part of the interpretation is the degree of surprise in the finding. Is it consistent with other findings in the domain? If so, then they can speak more strongly (not a lot, just a little) about the meaning of their findings. If it is surprising and contrary to other findings in the literature, then I'd be prone to rejection of the article due to lack of a sufficient finding to change my prior view. Paul On Aug 27, 2013, at 9:59 AM, Michael Britt wrote: Also helpful. So, to answer my own previous question, based on what they found in the correlational study and what one might guess from previous research, I'm going to assume that the effect size here, if it exists, is probably small. So I used .3 in G*Power. The result? G*Power suggests that I get 242 subjects per group. These researchers had 26 subjects in each group. So: if you were the reviewer what would you conclude? The researchers found: "...the results revealed that participants in the anthropomorphism condition were tendentially less willing to help the victims of the natural disaster (M = 4.39, SD = 1.02) than participants in the control condition (M = 4.89, SD = 0.87), t(50) = –1.91, p = .06, d = 0.53. Would you recommend that they get more subjects? Michael Michael A. Britt, Ph.D. [email protected]<mailto:[email protected]> http://www.ThePsychFiles.com Twitter: @mbritt On Aug 27, 2013, at 8:59 AM, Stuart McKelvie <[email protected]<mailto:[email protected]>> wrote: Dear Tipsters, There are various ways to plan sample size. When teaching this in research methods, I divide the issues into two parts: 1. Estimation of population values. Here, more is better but there are diminishing returns. Think of the fact that we rarely see more than 1500 people in national polls and surveys. The formula is based on minimizing standard error. Of course, sampling is critical. 2. Conducting studies with variables: experimental, subject or correlational. There are four interconnected concepts: effect size, alpha, power and sample size. When any three are known, the fourth is determined. You can decide where to set alpha and power. For effect size (d), you can be guided by Cohen's guidelines for small, medium and large (.3, .5, .8) and choose the value you are looking for. This may come from past research or, in its absence, what you think is interesting theoretically or practically. Cohen's book on power analysis gives tables where you can look up the sample size needed after specifying the values you choose. There is also this webiste: http://homepage.stat.uiowa.edu/~rlenth/Power/ Sincerely, Stuart _____________________________________________________ Sent via Web Access "Floreat Labore" "Recti cultus pectora roborant" Stuart J. McKelvie, Ph.D., Phone: 819 822 9600 x 2402 Department of Psychology, Fax: 819 822 9661 Bishop's University, 2600 rue College, Sherbrooke, Québec J1M 1Z7, Canada. E-mail: [email protected]<mailto:[email protected]> (or [email protected]<mailto:[email protected]>) Bishop's University Psychology Department Web Page: http://www.ubishops.ca/ccc/div/soc/psy " Floreat Labore" _______________________________________________________ ________________________________ From: Paul C Bernhardt [[email protected]<mailto:[email protected]>] Sent: 27 August 2013 08:41 To: Teaching in the Psychological Sciences (TIPS) Subject: Re: [tips] Sample Size: How to Determine it? There is software to determine this. One excellent and free app is G*Power. http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3/ I would use the correlational study to give me an estimate of effect size. As you describe, I would use that in the software to estimate my number of participants to attain the desired power. Practicality constraints on number of available participants usually limits things. I did such an estimate using G*Power a few weeks ago for a study we are planning. We will need to collect data over two semesters because the anticipated number of participants available from one semester's worth of students would only give us power of about .66, whereas two semester's worth would bump us up over .90. Paul On Aug 27, 2013, at 8:18 AM, Michael Britt wrote: I'm reading an interesting piece of research on anthropomorphism which essentially states after a natural disaster if we use the term "mother nature" when describing it, people will be less willing to contribute to relief efforts ("Humanizing nature could help the perceiver to conceive natural events as imbued with intentionality and significance rather than considering them merely random and meaningless phenomena"). They did two studies. Here's the issue/question: * Study 1 was correlational and involved 96 students. The results were supportive at <.001 * Study 2 was an experiment (no need to go into the details) involving 56 students. The results were, in the authors words, "tangentially" supportive with p<.06 I think the study was well conducted so I don't mean to slight the researchers. My guess is that if they used more subjects they probably would have reached p<.05 - but would that have been an example of "selective stopping"? I assume it would be. So how exactly does a researcher determine beforehand - as we are suggesting they do - the number of subjects they ought to try to get for the study? I'm just not familiar with the process. Does one look at the effect sizes of previous related studies to determine if the effect is large or small and then make a decision? But let's say the effect is assumed to be small, so do you use 100 subjects? 500? How is this number determined? Appreciate the insight in this. Michael Michael A. Britt, Ph.D. [email protected]<mailto:[email protected]> http://www.ThePsychFiles.com<http://www.ThePsychFiles.com/> Twitter: @mbritt --- You are currently subscribed to tips as: [email protected]<mailto:[email protected]>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13441.4e79e96ebb5671bdb50111f18f263003&n=T&l=tips&o=27372 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-27372-13441.4e79e96ebb5671bdb50111f18f263...@fsulist.frostburg.edu<mailto:leave-27372-13441.4e79e96ebb5671bdb50111f18f263...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: [email protected]<mailto:[email protected]>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13510.2cc18398df2e6692fffc29a610cb72e3&n=T&l=tips&o=27373 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-27373-13510.2cc18398df2e6692fffc29a610cb7...@fsulist.frostburg.edu<mailto:leave-27373-13510.2cc18398df2e6692fffc29a610cb7...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: [email protected]<mailto:[email protected]>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13405.0125141592fa9ededc665c55d9958f69&n=T&l=tips&o=27374 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-27374-13405.0125141592fa9ededc665c55d9958...@fsulist.frostburg.edu<mailto:leave-27374-13405.0125141592fa9ededc665c55d9958...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: [email protected]<mailto:[email protected]>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13441.4e79e96ebb5671bdb50111f18f263003&n=T&l=tips&o=27379 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-27379-13441.4e79e96ebb5671bdb50111f18f263...@fsulist.frostburg.edu<mailto:leave-27379-13441.4e79e96ebb5671bdb50111f18f263...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=27396 or send a blank email to leave-27396-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
