I think the approach should extend to multiple series if you build a
combine routine which works in a reduce context.
In other words, you need to build all the summary statistics when
combining two series. Or, something like this (warning: untested
code):
combineSum=: +&{.
combineMean=: +/&(*/)&}: % combineSum
combine=: combineSum, combineMean, combineSDsS@,@,.
I *think* combineSDsS would be like combineSDsScalars, but without the
m argument (which doesn't seem to be relevant). If that's right, you'd
use it like this:
combine&;/summary1;summary2;summary3
I hope this helps,
--
Raul
On Thu, Sep 20, 2018 at 11:18 AM Devon McCormick <[email protected]> wrote:
>
> I work with a system that handles a lot of complicated problems very simply
> but I've recently run into a difficult problem, the solution of which would
> be how do we re-construct the overall standard deviation of a number of
> series, to which we would rather not refer to in detail, when we have only
> the sample size, the mean, and the standard deviation for each group.
>
> How do we combine these?
>
> There's a lot of stuff on the web but most of what I've found is wrong -
> unless I'm wrong and the numerous different answers are all right - but for
> this reference - almost:
> http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart.
> I say "almost" because, on the website, the formula is correct but the
> arithmetic is wrong!
>
> The site shows an example worked out in painstaking detail - given the
> summary stats for this example - but the final answer is shown as 26.1952
> when the correct answer is 26.4226.
>
> However, if I properly reproduce the formula and do my own math, I get the
> correct answer, to whit - here's the formula, as given, with the math, and
> following is my own work in J, reproducing the formulas but getting the
> math right.
> Combined Mean = 64.8236
> Combined Standard deviation :
> σ12=√(N1-1)⋅σ21+(N2-1)⋅σ22+N1⋅N2N1+N2⋅(ˉx21+ˉx22-2ˉx1ˉx2)N1+N2-1
> ]nn=. (5?@$100);12?@$100
> +--------------+-----------------------------------+
> |23 81 52 67 34|97 72 53 22 85 39 97 94 82 35 79 90|
> +--------------+-----------------------------------+
> stddev ;nn NB. Correct answer
> 26.4226
> (*:@:stddev)&>nn
> 557.3 696.811
> var&>nn
> 557.3 696.811
>
> (#,mean,stddev)&>nn
> 5 51.4 23.6072
> 12 70.4167 26.3972
>
> NB. Following based on formula here:
> http://atozmath.com/CONM/Ch2_CombinedSD.aspx?q1=1.1%605%2c12%6051.4%2c70.4167%6023.6072%2c26.3972%60SD2#PrevPart
>
> combineSDs=: [: %: ((([: <: #&>) +/ .* var&>) + ([: (*/ % +/) #&>) * ([:
> +/ [: *: mean&>) - [: +: [: */ mean&>) % [: <: [: +/ #&>
> combineSDs nn NB. This works based on the mean, variance, and
> counts of nn.
> 26.4226
>
> NB. This version makes it clear we can do this using only the summary
> statistics:
> 'n1 n2'=. #&>nn [ 's21 s22'=. var&>nn [ 'm1 m2'=. mean&>nn [ m=. mean ;nn
>
> %:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2
> 26.4226
> combineSDsScalars=: 3 : 0
> 'n1 n2 m1 m2 m s21 s22'=. y
>
> %:(((n1-1)*s21)+((n2-1)*s22)+((n1*n2)%n1+n2)*(*:m1)+(*:m2)-+:m1*m2)%<:n1+n2
> )
> combineSDsScalars n1,n2,m1,m2,m,s21,s22
> 26.4226
>
> But how do I extend this for more than two series?
>
> --
>
> Devon McCormick, CFA
>
> Quantitative Consultant
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
