That's why you need to be careful with median and quartiles. You must know how it is calculated before you interpret the result.
On Fri, Oct 16, 2009 at 6:38 PM, Sherlock, Ric <[email protected]>wrote: > The following is based on Keith Similie's stats companion. > > NB. Median and quartiles > midpt=: -:@<:@# > median=: -:@(+/)@((<.,>.)@midpt { /:~) > Q1=: [: median ] #~ median > ] > Q3=: [: median ] #~ median < ] > quartiles=: Q1 , median , Q3 > > Another definition of median where the domain is integers. > > median=: ~.@((<.,>.)@midpt { /:~) > > > From: Devon McCormick > > > > Don - I like yours better than the one I have now, though I'll probably > > generalize it into an "Ntiler". > > > > Part of the problem is that there are multiple correct answers if we > > define > > quartile numbers as those which divide the set as evenly as possible > > into > > four groups, e.g. > > > > quartileCt=: 4 : '+/"1 (y>:/~x,_) *. y< /~__,x' NB. Count > > elements/quartile > > NB. All these different answers work correctly: > > (52.75 61 70.25) quartileCt scrs NB. Excel > > 5 5 5 5 > > (52.5 61 70.5) quartileCt scrs NB. web site > > 5 5 5 5 > > (52.1 61.1 70.1) quartileCt scrs NB. another answer... > > 5 5 5 5 > > > > One way to test, as you suggest is to look at the behavior when we have > > an > > odd number of elements, i.e. "odd" with respect to four: > > > > NB. Two different ways of counting number of elements/quartile: > > quartileCt=: 4 : '+/"1 (y>:/~x,_) *. y< /~__,x' > > quartileCt2=: 4 : '+/"1 (y> /~x,_) *. y<:/~__,x' > > NB. Two different quartilers: > > test0=: 1 : '(3{.4 ntilebps y) u y' NB. Mine > > test1=: 1 : '(qr y) u y' NB. Don's > > > > NB. Both work OK for even and odd cases counted one way... > > quartileCt test0&>0 1 2 3 4}.&.><scrs > > 5 5 5 5 > > 4 5 5 5 > > 4 5 4 5 > > 4 4 4 5 > > 4 4 4 4 > > quartileCt test1&>0 1 2 3 4}.&.><scrs > > 5 5 5 5 > > 5 5 5 4 > > 5 4 5 4 > > 5 4 4 4 > > 4 4 4 4 > > > > NB. Mine falls down for a couple of cases counted the other way: > > quartileCt2 test0&>0 1 2 3 4}.&.><scrs > > 4 5 5 6 > > 4 5 5 5 > > 4 4 5 5 > > 4 4 4 5 > > 3 4 4 5 > > NB. but Don's works OK under different counting method as well: > > quartileCt2 test1&>0 1 2 3 4}.&.><scrs > > 5 5 5 5 > > 5 5 5 4 > > 5 4 5 4 > > 5 4 4 4 > > 4 4 4 4 > > > > Thanks for your suggestions. > > > > Regards, > > > > Devon > > > > On Fri, Oct 16, 2009 at 3:47 PM, Don Guinn <[email protected]> wrote: > > > > > Looked up the definition of "median" and it appears that there are > > several > > > definitions of "median". And, according to > > > http://en.wikipedia.org/wiki/Median median and quartiles can be messy > > with > > > badly skewed data. Best I can tell this is a measurement that should > > be > > > used > > > with care. > > > I wrote a quick verb which gives the same answers as the site you > > > referenced > > > and it does strange things, depending on the data. If the count of > > the set > > > is odd, which group should have the extra number? What if the data is > > > really > > > skewed? > > > > > > qr=.([:([:(+/%#)]{~[:(<:,:])[:>.0.25 0.5 0.75"_*#)]/:]) NB. Needs > > > cleaning up. > > > qr scrs > > > 52.5 61 70.5 > > > qr i.4 > > > 0.5 1.5 2.5 > > > qr i.5 > > > 1.5 2.5 3.5 > > > qr i.12 > > > 2.5 5.5 8.5 > > > qr i.11 > > > 2.5 5.5 8.5 > > > qr i.13 > > > 3.5 6.5 9.5 > > > -~/0 2{qr scrs > > > 18 > > > qr 1 1 1 1 1 2 3 4 > > > 1 1 2.5 > > > > > > > > > On Fri, Oct 16, 2009 at 1:21 PM, Devon McCormick <[email protected]> > > > wrote: > > > > > > > Members of the forum - > > > > > > > > while looking up some statistical definitions, I came across this > > example > > > > > > http://www2.le.ac.uk/offices/ssds/sd/ld/resources/numeracy/variability > > > > in which the calculation of the median disagrees with the result of > > the > > > one > > > > listed as "m0=: median=: <....@-:@# { /:~" in "MathStats" on the J > > wiki. > > > > > > > > I was actually looking at the definition of quartiles when I > > noticed > > > this. > > > > > > > > For the series > > > > > > > > #scrs=. 43 48 50 50 52 53 56 58 59 60 62 65 66 68 70 71 74 76 78 > > 80 > > > > 20 > > > > m0=: <....@-:@# { /:~ > > > > m0 scrs > > > > 62 > > > > median scrs NB. my own definition > > > > 61 > > > > median > > > > -:@(+/)@((<. , >.)@midpt { /:~) > > > > midpt > > > > -:@<:@# > > > > > > > > Also, this site's answers disagree with Excel and with my own > > quartile > > > > function, applied to "scrs" above, but I think the site is correct: > > > > NB. Quartiles 1-3 according to Excel: > > > > 52.75 61 70.25 > > > > > > > > NB. According to > > > > > > http://www2.le.ac.uk/offices/ssds/sd/ld/resources/numeracy/variability: > > > > 52.5 61 70.5 > > > > > > > > 0 1 2 quartile&><scrs > > > > 52 60 70 > > > > > > > > NB. My "quartile" disagrees with my "median": the middle quartile > > should > > > be > > > > the same as the median. > > > > quartile > > > > 4 : 'x{4 ntilebps y' > > > > ntilebps > > > > 4 : 0 > > > > NB.* ntilebps: return breakpoint values of x-tiles of y; e.g. 4 > > ntilebps > > > y > > > > NB. -> quartiles; 0-based so "1st" quartile is 0{4 ntilebps y. > > > > quant=. x > > > > y=. /:~y > > > > wh=. 0 1#:(i.quant)*quant%~#y NB. Where partition points are > > exactly > > > > 'n f'=. |:wh NB. whole and fractional part of > > > > partitions > > > > 1|.+/"1 ((1-f),.f)*(n+/_1 0){y NB. "1|." moves top quantile to > > end. > > > > ) > > > > > > > > Anyone care to weigh in on this? > > > > > > > > Regards, > > > > > > > > Devon > > > > > > > > > > > > -- > > > > Devon McCormick, CFA > > > > ^me^ at acm. > > > > org is my > > > > preferred e-mail > > > > ------------------------------------------------------------------- > > --- > > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > > > --------------------------------------------------------------------- > > - > > > For information about J forums see > > http://www.jsoftware.com/forums.htm > > > > > > > > > > > -- > > Devon McCormick, CFA > > ^me^ at acm. > > org is my > > preferred e-mail > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
