I work with a system that handles a lot of complicated problems very simply but I've recently run into a difficult problem, the solution of which would be how do we re-construct the overall standard deviation of a number of series, to which we would rather not refer to in detail, when we have only the sample size, the mean, and the standard deviation for each group.
How do we combine these? There's a lot of stuff on the web but most of what I've found is wrong - unless I'm wrong and the numerous different answers are all right - but for this reference - almost: http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart. I say "almost" because, on the website, the formula is correct but the arithmetic is wrong! The site shows an example worked out in painstaking detail - given the summary stats for this example - but the final answer is shown as 26.1952 when the correct answer is 26.4226. However, if I properly reproduce the formula and do my own math, I get the correct answer, to whit - here's the formula, as given, with the math, and following is my own work in J, reproducing the formulas but getting the math right. Combined Mean = 64.8236 Combined Standard deviation : σ12=√(N1-1)⋅σ21+(N2-1)⋅σ22+N1⋅N2N1+N2⋅(ˉx21+ˉx22-2ˉx1ˉx2)N1+N2-1 ]nn=. (5?@$100);12?@$100 +--------------+-----------------------------------+ |23 81 52 67 34|97 72 53 22 85 39 97 94 82 35 79 90| +--------------+-----------------------------------+ stddev ;nn NB. Correct answer 26.4226 (*:@:stddev)&>nn 557.3 696.811 var&>nn 557.3 696.811 (#,mean,stddev)&>nn 5 51.4 23.6072 12 70.4167 26.3972 NB. Following based on formula here: http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart combineSDs=: [: %: ((([: <: #&>) +/ .* var&>) + ([: (*/ % +/) #&>) * ([: +/ [: *: mean&>) - [: +: [: */ mean&>) % [: <: [: +/ #&> combineSDs nn NB. This works based on the mean, variance, and counts of nn. 26.4226 NB. This version makes it clear we can do this using only the summary statistics: 'n1 n2'=. #&>nn [ 's21 s22'=. var&>nn [ 'm1 m2'=. mean&>nn [ m=. mean ;nn %:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2 26.4226 combineSDsScalars=: 3 : 0 'n1 n2 m1 m2 m s21 s22'=. y %:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2 ) combineSDsScalars n1,n2,m1,m2,m,s21,s22 26.4226 But how do I extend this for more than two series? -- Devon McCormick, CFA Quantitative Consultant ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
