I think the approach should extend to multiple series if you build a
combine routine which works in a reduce context.
In other words, you need to build all the summary statistics when
combining two series. Or, something like this (warning: untested
code):
combineSum=: +&{.
combineMean=: +/&(*/)&}: % combineSum
combine=: combineSum, combineMean, combineSDsS@,@,.
I *think* combineSDsS would be like combineSDsScalars, but without the
m argument (which doesn't seem to be relevant). If that's right, you'd
use it like this:
combine&;/summary1;summary2;summary3
I hope this helps,
--
Raul
On Thu, Sep 20, 2018 at 11:18 AM Devon McCormick <[email protected]>
wrote:
I work with a system that handles a lot of complicated problems very
simply
but I've recently run into a difficult problem, the solution of which
would
be how do we re-construct the overall standard deviation of a number of
series, to which we would rather not refer to in detail, when we have
only
the sample size, the mean, and the standard deviation for each group.
How do we combine these?
There's a lot of stuff on the web but most of what I've found is wrong -
unless I'm wrong and the numerous different answers are all right - but
for
this reference - almost:
http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart
.
I say "almost" because, on the website, the formula is correct but the
arithmetic is wrong!
The site shows an example worked out in painstaking detail - given the
summary stats for this example - but the final answer is shown as 26.1952
when the correct answer is 26.4226.
However, if I properly reproduce the formula and do my own math, I get
the
correct answer, to whit - here's the formula, as given, with the math,
and
following is my own work in J, reproducing the formulas but getting the
math right.
Combined Mean = 64.8236
Combined Standard deviation :
σ12=√(N1-1)⋅σ21+(N2-1)⋅σ22+N1⋅N2N1+N2⋅(ˉx21+ˉx22-2ˉx1ˉx2)N1+N2-1
]nn=. (5?@$100);12?@$100
+--------------+-----------------------------------+
|23 81 52 67 34|97 72 53 22 85 39 97 94 82 35 79 90|
+--------------+-----------------------------------+
stddev ;nn NB. Correct answer
26.4226
(*:@:stddev)&>nn
557.3 696.811
var&>nn
557.3 696.811
(#,mean,stddev)&>nn
5 51.4 23.6072
12 70.4167 26.3972
NB. Following based on formula here:
http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart
combineSDs=: [: %: ((([: <: #&>) +/ .* var&>) + ([: (*/ % +/) #&>) *
([:
+/ [: *: mean&>) - [: +: [: */ mean&>) % [: <: [: +/ #&>
combineSDs nn NB. This works based on the mean, variance, and
counts of nn.
26.4226
NB. This version makes it clear we can do this using only the summary
statistics:
'n1 n2'=. #&>nn [ 's21 s22'=. var&>nn [ 'm1 m2'=. mean&>nn [ m=. mean
;nn
%:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2
26.4226
combineSDsScalars=: 3 : 0
'n1 n2 m1 m2 m s21 s22'=. y
%:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2
)
combineSDsScalars n1,n2,m1,m2,m,s21,s22
26.4226
But how do I extend this for more than two series?
--
Devon McCormick, CFA
Quantitative Consultant
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
