Donald Burrill wrote:
Hi, Chuck. Did you ever get a satisfactory reply to your question? (I've a vague recollection of seeing one or two replies, but seem not to have saved them.) Had a thought or two that might be helpful.
In the first place, the b's for both younger and older adults are of the same sign (not surprisingly), and for both examples that you cited, were weaker for older persons. They were significant (i.e., different from zero, I presume?) for one age group, but not for the other.
That they were not distinguishably different from each other is also not very surprising: apparently (guessing from the data you cited) the threshold for significance (at whatever significance level you were using - you didn't say) is around 0.2, so that 0.31 and 0.24 are significant, but 0.15 and 0.19 are not; for the conditions (including sample sizes) of your study.
Since the noise level (e.g., standard error) around a difference is larger than that around a single estimate by about half (actually by a factor of <square root of 2> for independent samples with the same N), you'd need _differences_ on the order of 0.3 for them to be significant and the differences you got (0.16 and 0.05) just don't cut it.
Next question (I can't tell): are the raw differences you found (laying aside issues of significance for the moment) in directions that are consistent with respect to whatever underlying theory(ies) may inform this research? If not, that the differences sometimes seem to favor such a theory and sometimes oppose it would suggest that there may not be anything going on, at least not in accordance with theory. However, if they all (or most of them, for the several measures you took) are in a consistent direction, your evidence is somewhat stronger with respect to the theory, only the effect is on the average too small to detect with a t-test and your sample size. (As you are undoubtedly aware, ANY effect can be found to be significant if one throws enough observations at it: if N=50 isn't large enough, try N=400 or 2,000 or 40 million.)
But it occurs to me that your categories "younger adults" and "older adults" aren't very precise, and may well include a range of ages in each category. If you have enough data, you might be able to obtain more than 2 b's as a function of age groups defined more narrowly than you had done: in that case, you could plot b vs age (or average age) and eyeball the plot to see if it appears to wander consistently upward (or downward) as age increases. (And if your "younger" and "older" groups are separated by a range of ages for which you have no Ss, be alert for a corresponding gap in the apparent progression of b values.)
I wouldn't be surprised if the subgroups so obtained had subsample sizes so small that none of these b's were significantly different from zero; but if there are enough of them, and the changes with age are consistent, you could attack the problem with a sign test, thus: Suppose you had been able to ten age groups. Then there are nine differences from one age group to the next older one. By the null hypothesis of no effect due to age, one would expect half of these differences to be positive and half negative. If they're ALL in the same direction, that's impressive evidence that SOMEthing is there, even if the "something" is too subtle to be detected by a t-test (and given the rather large standard errors of the regression slopes, arising from the small subsample sizes). If not all, but most of the differences are consistent in this sense, you still may have interesting evidence (and in any case evidence to which you can attach a significance level, and for which a larger level (0.10, say) than your usual standard might well be appropriate); especially if the number of different age groups is smallish due to constraints imposed by the original selection process.
Of course, a sign test does have the disadvantage that there isn't really an "effect size" that one can readily compute. But sometimes the logical priority is first "Is there an effect?", and after that one can be more relaxed in acknowledging that the effect appears to be more subtle than can readily be estimated with the tools at hand...
HTH -- but I did want to suggest that approach, just in case it might have been helpful. Good luck! -- Don Burrill.
On Sun, 30 Nov 2003, Chuck Robertson wrote:
I have a question about power analysis related to a t-test of some weights in multiple regression equations. I'm afraid I'm a cognitive psychologist, so if I give you too much info below about my experiment please accept my apology.
I've taken memory measures from younger and older adults at several different encoding times and I also have created constructs of popular cognitive resources as predictors (perceptual speed, working memory, vocabulary,..) I am interested in how the regression equations change in each age group as memory performance increases (as I give people more time to study). I ran the regression equations for younger and older adults at a given memory measure and computed a t-test for difference between the predictor weights for what one might expect to be different equations. As an example, at a 1 second encoding time the older adults have a significant beta weight for speed as a predictor (b = .31) but the young adults do not (b = .15), in the same equation young adults have a significant beta weight for vocabulary (b = .24) but older adults do not (b = .19). This casually looks like a difference when eyeballing the equations, speed is the only significant predictor for older adults and vocabulary is the only significant predictor for young adults at a 1 second encoding time for the memory material. Interestingly, when I look at the t-test for difference between the B weights, I fail to reject the null of no significant difference between the weights. The t-test for difference between the speed predictors is t = -.51 and t = .14 for the vocabulary predictors. Of course I do not want to accept the null, I am really curious as to how much power I had to detect a significant difference with my sample size of 80 younger and 80 older adults in each equation.
I need help computing the effect size given my SPSS output from the regressions. I assume that I need to compute f as an effect size since this isn't teh typical t-test. Can anyone tell me how to do that given the weights and the standard errors for the weights from the output? I must be doing something wrong as my f's are way to large.
Thanks, Chuck
. . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
----------------------------------------------------------------------- Donald F. Burrill [EMAIL PROTECTED] 56 Sebbins Pond Drive, Bedford, NH 03110 (603) 626-0816 . . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
-- Bob Wheeler --- http://www.bobwheeler.com/ ECHIP, Inc. --- Randomness comes in bunches.
. . ================================================================= Instructions for joining and leaving this list, remarks about the problem of INAPPROPRIATE MESSAGES, and archives are available at: . http://jse.stat.ncsu.edu/ . =================================================================
