Re: Disadvantage of Non-parametric vs. Parametric Test

1999-12-08 Thread Rich Ulrich

 - I have a comment on an offhand remark of Glen's, at the start of
his interesting posting -

On Tue, 07 Dec 1999 15:58:11 +1100, Glen Barnett
[EMAIL PROTECTED] wrote:

 Alex Yu wrote:
  
  Disadvantages of non-parametric tests:
  
  Losing precision: Edgington (1995) asserted that when more precise
  measurements are available, it is unwise to degrade the precision by
  transforming the measurements into ranked data.
 
 So this is an argument against rank-based nonparametric tests
 rather than nonparametric tests in general. In fact, I think
 you'll find Edgington highly supportive of randomization procedures,
 which are nonparametric.
 
 - In my vocabulary, these days, "nonparametric"  starts out with data
being ranked, or otherwise being placed into categories -- it is the
infinite parameters involved in that sort of non-reversible re-scoring
which earns the label, nonparametric.  (I am still trying to get my
definition to be complete and concise.)

I know that when *nonparametric*  and  *distribution-free*  were the
two alternatives to ANOVAs, either of the two labels was slapped onto
people's pet procedures, fairly  indiscriminately;  and a lack of
discrimination seems to have widened to encompass  *robust*,  later
on.  Okay, I see that exact evaluation by randomization of a fixed
sample does not use a t or F distribution for its p-levels.   Okay, I
see that it is not ANOVA.   But, I'm sorry,  I don't regard a test as
nonparametric which *does*  preserve and use the original metric and
means.  Comparison of means is parametric, and that contrasts to
nonparametric.

Similarly, bootstrapping is a method of "robust variance estimation"
but it does not change the metric like a power transformation does, or
abandon the metric like a rank-order transformation does.  If it were
proper  terminology to say randomization is nonparametric, you would
probably want to say bootstrapping is nonparametric, too.  (I think
some people have done so; but it is not widespread.)

-- 
Rich Ulrich, [EMAIL PROTECTED]
http://www.pitt.edu/~wpilib/index.html



Re: Disadvantage of Non-parametric vs. Parametric Test

1999-12-08 Thread Glen Barnett

Frank E Harrell Jr wrote:
 
   Alex Yu wrote:
   
Disadvantages of non-parametric tests:
   
Losing precision: Edgington (1995) asserted that when more precise
measurements are available, it is unwise to degrade the precision by
transforming the measurements into ranked data.
 
 Edgington's comment is off the mark in most cases.  The efficiency of the
 Wilcoxon-Mann-Whitney test is 3/pi (0.96) with respect to the t-test
 IF THE DATA ARE NORMAL.  If they are non-normal, the relative
 efficiency of the Wilcoxon test can be arbitrarily better than the t-test.
 Likewise, Spearman's correlation test is quite efficient (I think the
 efficiency is 9/pi^2) relative to the Pearson r test if the data are
 bivariate normal.
 
 Where you lose efficiency with nonparametric methods is with estimation
 of absolute quantities, not with comparing groups or testing correlations.
 The sample median has efficiency of only 2/pi against the sample mean
 if the data are from a normal distribution.

Yes, the median is inefficient at the normal. This is the
location estimator corresponding to the sign test in the one-sample
case. But if you use the location estimator corresponding to the 
signed-rank test (say) instead, the efficiency improves substantially.

Glen



Re: Disadvantage of Non-parametric vs. Parametric Test

1999-12-08 Thread Glen Barnett

Rich Ulrich wrote:
  - In my vocabulary, these days, "nonparametric"  starts out with data
 being ranked, or otherwise being placed into categories -- it is the
 infinite parameters involved in that sort of non-reversible re-scoring
 which earns the label, nonparametric.  (I am still trying to get my
 definition to be complete and concise.)

Well, I am happy for you to use this definition of nonparametric now 
that you've said what you want it to mean, but it isn't exactly
what most statisticians - including those of us that distinguish
between the terms "distribution-free" and "nonparametric" - mean 
by "nonparametric", so you'll have to excuse my earlier ignorance 
of your definition.

If my recollection is correct, a parametric procedure is where the
entire distribution is specified up to a finite number of parameters,
whereas a nonparametric procedure is one where the distribution 
can't be/isn't specified with only a finite number of unspecified
parameters. This typically includes the usual distribution-free 
procedures, including many rank-based procedures, but it also 
includes many other things - including some that don't transform 
the data in any way, and even some based on means.

So, for example, ordinary simple linear regression is parametric,
because the distribution of y|x is specified, up to the value of 
the parameters specifying the intercept and slope of the line, and
the variance about the line.

Nonparametric regression (as the term is typically  
used in the literature), by contrast, is effectively
infinite-parametric, because the distribution of y|x
doesn't depend only on a finite number of parameters 
(often the distribution *about* E[y|x] is parametric 
- typically gaussian - but E[y|x] itself is where the 
infinite-parametric part comes from).

Nonparametric regression would not seem to fit your definition 
of "nonparametric", since your usage seems to require some
loss of information through ranking or categorisation. 

Once we start using the same terminology, we tend to find the
disagreements die down a bit. 

Glen



Re: Disadvantage of Non-parametric vs. Parametric Test

1999-12-07 Thread Glen Barnett

Alex Yu wrote:
 
 Disadvantages of non-parametric tests:
 
 Losing precision: Edgington (1995) asserted that when more precise
 measurements are available, it is unwise to degrade the precision by
 transforming the measurements into ranked data.

So this is an argument against rank-based nonparametric tests
rather than nonparametric tests in general. In fact, I think
you'll find Edgington highly supportive of randomization procedures,
which are nonparametric.

In fact, surprising as it may seem, a lot of the location 
information in a two sample problem is in the ranks. Where
you really start to lose information is in ignoring ordering
when it is present.
 
 Low power: Generally speaking, the statistical power of non-parametric
 tests are lower than that of their parametric counterpart except on a few
 occasions (Hodges  Lehmann, 1956; Tanizaki, 1997).

When the parametric assumptions hold, yes. e.g. if you assume normality
and the data really *are* normal. When the parametric assumptions are
violated, it isn't hard to beat the standard parametric techniques.

However, frequently that loss is remarkably small when the parametric
assumption holds exactly. In cases where they both do badly, the
parametric may outperform the nonparametric by a more substantial
margin (that is, when you should use something else anyway - for
example, a t-test outperforms a WMW when the distributions are
uniform).

 Inaccuracy in multiple violations: Non-parametric tests tend to produce
 biased results when multiple assumptions are violated (Glass, 1996;
 Zimmerman, 1998).

Sometimes you only need one violation:
Some nonparametric procedures are even more badly affected by
some forms of non-independence than their parametric equivalents.
 
 Testing distributions only: Further, non-parametric tests are criticized
 for being incapable of answering the focused question. For example, the
 WMW procedure tests whether the two distributions are different in some
 way but does not show how they differ in mean, variance, or shape. Based
 on this limitation, Johnson (1995) preferred robust procedures and data
 transformation to non-parametric tests.

But since WMW is completely insensitive to a change in spread without
a change in location, if either were possible, a rejection would 
imply that there was indeed a location difference of some kind. This
objection strikes me as strange indeed. Does Johnson not understand
what WMW is doing? Why on earth does he think that a t-test suffers
any less from these problems than WMW?
 
Similarly, a change in shape sufficient to get a rejection of a WMW
test would imply a change in location (in the sense that the "middle"
had moved, though the term 'location' becomes somewhat harder to pin
down precisely in this case).  e.g. (use a monospaced font to see this):

:. .:
::.   =  .::
...   ...
a b   a b
 
would imply a different 'location' in some sense, which WMW will
pick up. I don't understand the problem - a t-test will also reject
in this case; it suffers from this drawback as well (i.e. they are
*both* tests that are sensitive to location differences, insensitive
to spread differences without a corresponding location change, and
both pick up a shape change that moves the "middle" of the data).

However, if such a change in shape were anticipated, simply testing
for a location difference (whether by t-test or not) would be silly. 

Nonparametric (notably rank-based) tests do have some problems,
but making progress on understanding just what they are is 
difficult when such seemingly spurious objections are thrown in.

His preference for robust procedures makes some sense, but the
preference for (presumably monotonic) transformation I would 
see as an argument for a rank-based procedure. e.g. lets say
we are in a two-sample situation, and we decide to use a t-test
after taking logs, because the data are then reasonably normal...
in that situation, the WMW procedure gives the same p-value as 
for the untransformed data. However, let's assume that the 
log-transform wasn't quite right... maybe not strong enough. When 
you finally find the "right" transformation to normality, there
you finally get an extra 5% (roughly) efficiency over the WMW you
started with. Except of course, you never know you have the right
transformation - and if the distribution the data are from are
still skewed/heavy-tailed after transformation (maybe they were
log-gamma to begin with or something), then you still may be better
off using WMW.

Do you have a full reference for Johnson? I'd like to read what
the reference actually says.

Glen