Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
Dear Michael and other readers, Please see below for my answers to your questions about my data. On 07/06/2013 02:56 PM, Michael Dewey wrote: [..] Because everything was randomized, I can only calculate the total number of times a certain response was used under each type of trial. There is no pairing of trials under two treatments, so I am forced to use the marginal totals from your table. But presumably you could calculate some statistic suitable for summarising the relevant features here? Difference in proportions, odds ratio, ... Using the totals, it is indeed easy to calculate the difference in proportions or odds ratio based on these totals. However, I am not sure how I should calculate a study-level statistic suitable for meta-analysis on the basis of these participant-level proportion differences. So, for instance, I have the following table; pp proportion_difference 1 0.1 2 0.05 3 0.08 4 0.02 .. N .. Can I just calculate the mean and standard deviation of these proportion differences -- mean(proportion_difference) and sd(proportion_difference) -- and use these for meta-analysis? If yes, what escalc measure should I use? [..] One alternative that I have tried over the last few days, is to use the b parameter of interest and it's corresponding standard error from the lme4 regression output that I use to analyse the individual experiments. Then, I use rma(yi, sei) to do a meta-analysis on these parameters. I am not sure this is correct though, since it takes into account between-subjects variance (through a random effect for subject), and it is sensitive to the covariates/moderators I include in the models that I get the b parameters from. So you end up with 5 values of b? The fact that they adjust for different moderators does not seem an issue to me, indeed it could be argued to be an advantage of the meta-analytic approach here. OK, thank you for your comment on this one. I think the results of a meta-analysis using these 5 b values are indeed more or less sensible, which is encouraging. I think I will go this way if it turns out I cannot find a simpler approach, as a simpler approach would be easier to sell to potential reviewers. [.. I think we are all assuming you have different participants in each experiment but I thought I would raise that as a question. You are right in assuming this, I have different participants in all 5 experiments. Thanks all for the help so far, your suggestions are highly appreciated! Regards, Marc __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
It seems to me that the most straightforward approach would be to pool the b coefficients directly. I assume you used some kind of mixed-effects logistic model to analyze those data for each study with random effects for subjects and a dummy variable for treatment (besides possibly some other covariates) and the coefficient of interest is the one corresponding to the dummy for treatment. Given those 5 coefficients and the corresponding standard errors, you can then pool them with rma(b, sei=se), where 'b' is the vector with the coefficients and 'se' is the vector with the corresponding standard errors. Careful: rma(b, se) would not be correct, since the second argument is assumed to represent the *sampling variances*: library(metafor) args(rma) function (yi, vi, sei, [SNIP]) So, due to positional matching of arguments, rma(b, se) would be treated as rma(yi=b, vi=se), which is not what you want (you would not be the first person to fall into that trap). And yes, if you include additional covariates in the mixed-effects logistic model, then of course the b's and se's will change, but I see no issue here. It's just a question of whether you want to pool raw or adjusted coefficients (if you do adjust though, it would be best to adjust for the same covariates, so that the adjusted coefficients are directly comparable). Aside from all of that, there is another approach you could take. Since you have the raw data from all 5 studies, you could just as well pool the raw data into one dataset and analyze that. This is essentially an individual person (or patient) data meta-analysis (IPDMA). You would then include either fixed or random effects for studies and you probably would also want the treatment effect to vary randomly across studies. That approach should give you similar results as pooling the 5 coefficients (which is a sort of two-step approach). Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Psychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com -Original Message- From: Marc Heerdink [mailto:m.w.heerd...@uva.nl] Sent: Sunday, July 07, 2013 23:51 To: Michael Dewey Cc: Viechtbauer Wolfgang (STAT); r-help@r-project.org Subject: Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor Dear Michael and other readers, Please see below for my answers to your questions about my data. On 07/06/2013 02:56 PM, Michael Dewey wrote: [..] Because everything was randomized, I can only calculate the total number of times a certain response was used under each type of trial. There is no pairing of trials under two treatments, so I am forced to use the marginal totals from your table. But presumably you could calculate some statistic suitable for summarising the relevant features here? Difference in proportions, odds ratio, ... Using the totals, it is indeed easy to calculate the difference in proportions or odds ratio based on these totals. However, I am not sure how I should calculate a study-level statistic suitable for meta-analysis on the basis of these participant-level proportion differences. So, for instance, I have the following table; ppproportion_difference 1 0.1 2 0.05 3 0.08 4 0.02 .. N .. Can I just calculate the mean and standard deviation of these proportion differences -- mean(proportion_difference) and sd(proportion_difference) -- and use these for meta-analysis? If yes, what escalc measure should I use? [..] One alternative that I have tried over the last few days, is to use the b parameter of interest and it's corresponding standard error from the lme4 regression output that I use to analyse the individual experiments. Then, I use rma(yi, sei) to do a meta-analysis on these parameters. I am not sure this is correct though, since it takes into account between-subjects variance (through a random effect for subject), and it is sensitive to the covariates/moderators I include in the models that I get the b parameters from. So you end up with 5 values of b? The fact that they adjust for different moderators does not seem an issue to me, indeed it could be argued to be an advantage of the meta-analytic approach here. OK, thank you for your comment on this one. I think the results of a meta-analysis using these 5 b values are indeed more or less sensible, which is encouraging. I think I will go this way if it turns out I cannot find a simpler approach, as a simpler approach would be easier to sell to potential reviewers. [.. I think we are all assuming you have different participants in each experiment but I thought I would raise that as a question. You are right in assuming
Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
., Statistician Department of Psychiatry and Psychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com From: r-help-boun...@r-project.org [r-help-boun...@r-project.org] On Behalf Of Marc Heerdink [m.w.heerd...@uva.nl] Sent: Wednesday, July 03, 2013 2:15 PM To: r-help@r-project.org Subject: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor Hi all, I am currently attempting to compile a summary of a series of five psychological experiments, and I am trying to do this using the metafor package. However, I am quite unsure which of the scenarios described in the metafor help pages applies to these data, because it is a repeated measures design, with multiple trials in each condition. Assume that for every participant, I have a basic contingency table such as this one: treatment 1 2 response 1 10 20 2 20 10 (if this ASCII version does not work, I have 30 trials in each treatment, and participants give either response 1 or 2; the exact numbers don't matter) The problem that I am trying to solve is how to convert these numbers to an effect size estimate that I can use with metafor. As far as I understand it, I can only use it to get an effect size for outcomes that are dichotomous; i.e., either 1 or 0 for any subject. However, I have proportion data for every participant. I have considered and tried these strategies: 1. Base the effect size on within-participant proportion differences. That is, in the table above, the treatment effect would be (20/30)-(10/30) = 1/3; and I would take the M and SD of these values to estimate a study-level effect (MN measure in metafor). 2. Use the overall treatment * response contingency table, ignoring the fact that these counts come from different participants (PHI or OR measures in metafor). In a study with 10 participants, I would get cell counts around 150. However, from the research I've done into this topic, I know that 1) is not applicable to (as far as I understand) an odds ratio, and I suspect 2) overestimates the effect. A third method would be to use the regression coefficients, that I can easily obtain since I have all the raw data that I need. However, it is unclear to me whether and if yes, how I can use these in the metafor package. From my understanding of another message about this topic I found on this list (1), I understand that having access to the raw data is an advantage, but I am not sure whether the scenario mentioned applies to my situation. 1: http://r.789695.n4.nabble.com/meta-analysis-with-repeated-measure-designs-td2252644.html I would very much appreciate any suggestions or hints on this topic. Regards, Marc __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- Marc Heerdink, MSc. (PhD. candidate) Dept. of Social Psychology University of Amsterdam http://home.medewerker.uva.nl/m.w.heerdink/ http://www.easi-lab.nl/ Michael Dewey i...@aghmed.fsnet.co.uk http://www.aghmed.fsnet.co.uk/home.html __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
Dear Wolfgang and other readers of the r-help list, Thank you very much for your suggestion. Unfortunately, the data that I have can not be described with a table such as the one you have made, because there's no identical trial under both treatment 1 and treatment 2. To explain, let me explain a bit more about the experiments: * All subjects were presented with the same number of trials * Half of these trials were preceded by a prime from category 1 (treatment 1) and half of these trials with a prime from category 2 (treatment 2) * Subjects were asked to respond to these trials (a unique stimulus for each trial) by pressing one of two keys on the keyboard. Because everything was randomized, I can only calculate the total number of times a certain response was used under each type of trial. There is no pairing of trials under two treatments, so I am forced to use the marginal totals from your table. I have uploaded a simplified version of the data for one experiment to illustrate this (the actual experiments have five treatments and some have moderators): https://www.dropbox.com/s/rhgo12cm1asl6x8/exampledata.csv This is the script that I used to generate the data: https://www.dropbox.com/s/7uyeaexhnqiiy55/exampledata.R The problem thus appears to lie mainly in estimating the variance of the proportion difference from only the marginal totals, is that correct? Is there a way to calculate it from only the marginal totals? One alternative that I have tried over the last few days, is to use the b parameter of interest and it's corresponding standard error from the lme4 regression output that I use to analyse the individual experiments. Then, I use rma(yi, sei) to do a meta-analysis on these parameters. I am not sure this is correct though, since it takes into account between-subjects variance (through a random effect for subject), and it is sensitive to the covariates/moderators I include in the models that I get the b parameters from. Thanks again for your help, and for any suggestions for solving this problem! Regards, Marc On 07/04/2013 11:21 PM, Viechtbauer Wolfgang (STAT) wrote: Dear Marc, Let me see if I understand the type of data you have. You say that you have 5 experiments. And within each experiment, you have n subjects and for each subject, you have data in the form described in your post. Now for each subject, you want to calculate some kind of measure that quantifies how much more likely it was that subjects gave/chose response 2 under treatment 2 versus treatment 1. So, you would have n such values. And then you want to pool those values over the n subjects within a particular experiment and then ultimately over the 5 experiments. Is that correct so far? Assuming I got this right, let me ask you about those data that you have for each subject. In particular, are these paired data? In other words, is there are 1:1 relationship between the 30 trials under treatment 1 versus treatment 2? Or phrased yet another way, can you construct a table like this for every subject: trt 2 resp1 resp2 trt 1 resp1 a b 10 resp2 c d 20 2010 30 Note that I added the marginal counts based on your example data, but this is not sufficient to reconstruct how often response 1 was chosen for the same trial under both treatment 1 and treatment 2 (cell a). And so on for the other 3 cells. If all of this applies, then essentially you are dealing with dependent proportions and you can calculate the difference y = (20/30)-(10/30) as you have done. The corresponding sampling variance can be estimated with v = var(y) = (a+b)*(c+d)/t^3 + (a+c)*(b+d)/t^3 - 2*(a*d/t^3 - b*c/t^3) (where t is the number of trials, i.e., 30 in the example above). See, for example, section 10.1.1. in Agresti (2002) (Categorical data analysis, 2nd ed.). So, ultimately, you will have n values of y and v for a particular experiment and then the same thing for all 5 experiments. You can then pool those values with rma(yi, vi) in metafor (yi and vi being the vectors of the y and v values). You probably want to add a factor to the model that indicates which experiment those values came from. So, something like: rma(yi, vi, mods = ~ factor(experiment)). Well, I hope that I understood your data correctly. Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Psychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com From: r-help-boun...@r-project.org [r-help-boun...@r-project.org] On Behalf Of Marc Heerdink [m.w.heerd...@uva.nl] Sent: Wednesday, July 03, 2013 2:15 PM To: r-help@r-project.org Subject: [R] Meta-analysis on a repeated measures design
Re: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
Dear Marc, Let me see if I understand the type of data you have. You say that you have 5 experiments. And within each experiment, you have n subjects and for each subject, you have data in the form described in your post. Now for each subject, you want to calculate some kind of measure that quantifies how much more likely it was that subjects gave/chose response 2 under treatment 2 versus treatment 1. So, you would have n such values. And then you want to pool those values over the n subjects within a particular experiment and then ultimately over the 5 experiments. Is that correct so far? Assuming I got this right, let me ask you about those data that you have for each subject. In particular, are these paired data? In other words, is there are 1:1 relationship between the 30 trials under treatment 1 versus treatment 2? Or phrased yet another way, can you construct a table like this for every subject: trt 2 resp1 resp2 trt 1 resp1 a b 10 resp2 c d 20 2010 30 Note that I added the marginal counts based on your example data, but this is not sufficient to reconstruct how often response 1 was chosen for the same trial under both treatment 1 and treatment 2 (cell a). And so on for the other 3 cells. If all of this applies, then essentially you are dealing with dependent proportions and you can calculate the difference y = (20/30)-(10/30) as you have done. The corresponding sampling variance can be estimated with v = var(y) = (a+b)*(c+d)/t^3 + (a+c)*(b+d)/t^3 - 2*(a*d/t^3 - b*c/t^3) (where t is the number of trials, i.e., 30 in the example above). See, for example, section 10.1.1. in Agresti (2002) (Categorical data analysis, 2nd ed.). So, ultimately, you will have n values of y and v for a particular experiment and then the same thing for all 5 experiments. You can then pool those values with rma(yi, vi) in metafor (yi and vi being the vectors of the y and v values). You probably want to add a factor to the model that indicates which experiment those values came from. So, something like: rma(yi, vi, mods = ~ factor(experiment)). Well, I hope that I understood your data correctly. Best, Wolfgang -- Wolfgang Viechtbauer, Ph.D., Statistician Department of Psychiatry and Psychology School for Mental Health and Neuroscience Faculty of Health, Medicine, and Life Sciences Maastricht University, P.O. Box 616 (VIJV1) 6200 MD Maastricht, The Netherlands +31 (43) 388-4170 | http://www.wvbauer.com From: r-help-boun...@r-project.org [r-help-boun...@r-project.org] On Behalf Of Marc Heerdink [m.w.heerd...@uva.nl] Sent: Wednesday, July 03, 2013 2:15 PM To: r-help@r-project.org Subject: [R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor Hi all, I am currently attempting to compile a summary of a series of five psychological experiments, and I am trying to do this using the metafor package. However, I am quite unsure which of the scenarios described in the metafor help pages applies to these data, because it is a repeated measures design, with multiple trials in each condition. Assume that for every participant, I have a basic contingency table such as this one: treatment 1 2 response 1 10 20 2 20 10 (if this ASCII version does not work, I have 30 trials in each treatment, and participants give either response 1 or 2; the exact numbers don't matter) The problem that I am trying to solve is how to convert these numbers to an effect size estimate that I can use with metafor. As far as I understand it, I can only use it to get an effect size for outcomes that are dichotomous; i.e., either 1 or 0 for any subject. However, I have proportion data for every participant. I have considered and tried these strategies: 1. Base the effect size on within-participant proportion differences. That is, in the table above, the treatment effect would be (20/30)-(10/30) = 1/3; and I would take the M and SD of these values to estimate a study-level effect (MN measure in metafor). 2. Use the overall treatment * response contingency table, ignoring the fact that these counts come from different participants (PHI or OR measures in metafor). In a study with 10 participants, I would get cell counts around 150. However, from the research I've done into this topic, I know that 1) is not applicable to (as far as I understand) an odds ratio, and I suspect 2) overestimates the effect. A third method would be to use the regression coefficients, that I can easily obtain since I have all the raw data that I need. However, it is unclear to me whether and if yes, how I can use these in the metafor package. From my understanding of another message about this topic I found on this list (1), I understand that having access to the raw data
[R] Meta-analysis on a repeated measures design with multiple trials per subject using metafor
Hi all, I am currently attempting to compile a summary of a series of five psychological experiments, and I am trying to do this using the metafor package. However, I am quite unsure which of the scenarios described in the metafor help pages applies to these data, because it is a repeated measures design, with multiple trials in each condition. Assume that for every participant, I have a basic contingency table such as this one: treatment 1 2 response 1 10 20 2 20 10 (if this ASCII version does not work, I have 30 trials in each treatment, and participants give either response 1 or 2; the exact numbers don't matter) The problem that I am trying to solve is how to convert these numbers to an effect size estimate that I can use with metafor. As far as I understand it, I can only use it to get an effect size for outcomes that are dichotomous; i.e., either 1 or 0 for any subject. However, I have proportion data for every participant. I have considered and tried these strategies: 1. Base the effect size on within-participant proportion differences. That is, in the table above, the treatment effect would be (20/30)-(10/30) = 1/3; and I would take the M and SD of these values to estimate a study-level effect (MN measure in metafor). 2. Use the overall treatment * response contingency table, ignoring the fact that these counts come from different participants (PHI or OR measures in metafor). In a study with 10 participants, I would get cell counts around 150. However, from the research I've done into this topic, I know that 1) is not applicable to (as far as I understand) an odds ratio, and I suspect 2) overestimates the effect. A third method would be to use the regression coefficients, that I can easily obtain since I have all the raw data that I need. However, it is unclear to me whether and if yes, how I can use these in the metafor package. From my understanding of another message about this topic I found on this list (1), I understand that having access to the raw data is an advantage, but I am not sure whether the scenario mentioned applies to my situation. 1: http://r.789695.n4.nabble.com/meta-analysis-with-repeated-measure-designs-td2252644.html I would very much appreciate any suggestions or hints on this topic. Regards, Marc __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
