On Date: Fri, 29 Jan 2016 12:57:50 -0600, Douglas Peterson wrote:
No argument here. Just me not being clear.
No problem. Just me being argumentative. ;-)
A' and AUC are valid measures comparing two systems
and much more interpretable than other SDT measures
given the parameters as Mike explains but they are not
direct measures of SDT parameters as typically explains.
I'm not sure why being "direct measure of SDT" is more
important thant the usefulness of the measure (e.g., A'
being more useful than d'). Although defining d' as the
difference between Z-hit rate - Z-false alarm is one way
to calculate d', this form seems to depend upon the
assumption that the probability dietributions are normal
in which case the means and standard deviations are
independent which is not the case in some other
distributions.
True story: I learned about SDT from Sheila Chase with
whom I took experiment psych lab at Hunter College as
an undergraduate. She did psychophysics work with
pigeons and we used Michael D'Amato's (an NYU Ph.D. ;-)
textbook on experimental psychology which covered
classical psychophysics and SDT. When I got to grad
school in experimental psych at Stony Brook, I got
additional coverage of SDT in my graduate S&P class.
However, I had to take a year off from grad school to
deal with some family matters and enrolled part-time
at NYU's graduate experimental psych program so I
could keep up my studies. One of the courses involved
George Sperling who taught the "sensation" and psychophysics
part of a course called "Basic Processes I" that all first
year NYU students had to take (Lloyd Kaufman taught
the "perception" part of the course). Speriling took a
heavily mathematical approach and when he started to
cover SDT I thought "Cool, I know this stuff."
Well, unfornately, Sperling started out with an example of
SDT that used two exponential distributions instead of normal
distributions. Now, I had some idea of what an exponential
distribution was but that was never covered in any class I took
(I read about them in my readings on SDT). He went into the math
for doing SDT with exponential distributions and I was lost --
a major problem was that Sperling didn't provide a reading list
on the topics, so you either followed what he said in class or
you had to find sources on your own. I thouught I was a total
idiot until I talked to my classmates who also had no idea
what an exponential distribution was. This, of course, raised
the question why Sperling assumed we would know about
exponential distributions (later we learned that he just expected
a certain level of math stat knowledge in his students and if
they didn't have it, they could just flunk out).
Unltimately, Sperling had to point out to us idiots that the
signficance of exponential distributions is that the variance
of such distributions is the square of the mean of the distribution
which poses problems for SDT because (a) the mean and SD
are not independent, and (b) as the mean increased, so did the
variance. This violated traditional SDT based on normal
distributions but early work by Egan and others showed how
other distributions could be used -- one just needed either a
background in electrical engineering (Sperling was working
at Bell Labs at the time) or a masters in math stat.
Needless to say, I bombed this part of the course -- it didn't make
me feel any better that the rest of the class also bombed out.
However, I did ace Kaufman's part of the course. ;-)
Morale: the form of SDT that relies upon normal distributions is
only one form of SDT analysis and other forms (as we'll see shortly)
may make the analysis more complex and rely upon other aspects
of the analysis (e.g., A' actually is better than d' in certain
situations).
Pastore, Crawley, Berens and Skelly (2003) present a good
discussion of the issues including the advantages and
disadvantages of A'.
*smacks forehead* How did I forget Pastore et al (2003)? I looked
at the article after you mentioned it and realized that I had in fact
read
but a while ago. However, turns out the situation may be more
complex than they present. More on this shortly.
Specifically, A' is not independent from bias and is actually a
poorer estimate when performance is nearer to perfect in terns
of hits or false alarms. For the 3 of us who care about this issue,
I think we will soon reach N <1 of Tipsters who care about this
issue. ;-)
estimates of d' aren't much good in those extremes either.
You mean like how Fechner's law breaks down at very low and very
high intensities because the Weber ratio is not constant for
all stimulus values; e.g., see:
https://books.google.com/books?id=ALsP3Rv3fFgC&pg=PA288&dq=%22fechner%27s+law%22+extremes&hl=en&sa=X&ved=0ahUKEwjtzK7n3dHKAhXLWT4KHTnBBkoQ6AEIKDAC#v=onepage&q=%22fechner%27s%20law%22%20extremes&f=false
Macmillan and Creelman (1991) suggest adjusting hit rates of
100% to 1-(1/2n) an false alarm rates to 1/2n and I don't have
any reason to doubt that I just don't see it used very often.
Hit rates of 100% indicate a ceiling effect which means that
(a) in psychophysics, stimuli are clearly discriminable, and
(b) in recognition memory research, the material being learned
is too easy. These and similar situations are really experimental
deisgn problems and requie a change in materials or procedure.
For the record, in a recognition memory experiment I once
conducted, some people did have hit rates of 1.00 or 100%
and I reduced these to .95 in order to calculate d-prime.
The use of the sensitivity/specificity reporting doesn't capture
both the sensitive and response bias as explained in SDT
examples (i.e.g, estimate of the distance between the two
distributions (I believe this the reason that this entry is clear
to distinguish the sensitivity index, called d' as something
different from sensitivity as true positives). The two approaches
might be considered two sides of the same coin but they
are not the same side of the same coin.
The tradition of mathematical analysis of diagnositic testing
and SDT both have common roots, I believe, in Wald's statistical
decision theory, so they have common origins. However,
the difergence in the use of terms used over time reflect
the interests of the two disciplines (i.e., diagnostic testing
vx psychophysics). Sensitivity means different things in the
two discrplins but once one knows the terms, one should be
able to make the appropraite translations (e.g., in diagnostic
test "sensitivity" is equivalent to the "Hit rate" in SDT).
Macmillan, N.A., & Creelman, C.D. (1991). Detection Theory:
A User’s Guide. NY: Cambridge University Press.
NOTE: the 2004 (2nd ed) is published by Psychology Press
which gobbling up publishers in psychology.
Pastore, R.E., Crawley, E.J., Berens, M.S., & Skelley, M.A.
(2003). "Nonparametirc" A' and other modern misconceptions
about signal detection theory. Psychonomic Builletin and
Review, 10(3), 556-569.
If we're playing poker here, I see your Pastore et al (2003) and
raise you a Diana Kornbrut (2006); see:
Kornbrot, D. E. (2006). Signal detection theory, the approach
of choice: Model-based and distribution-free measures and
evaluation. Perception & psychophysics, 68(3), 393-414.
http://link.springer.com/article/10.3758%2FBF03193685#/page-1
In this paper Kornbrot distinguishes between what has been
traditionally referred to a "parametric" SDT which she calls
"model-based" because arguments that measures like A' are
"non-parametric" turn out to be false, that is, they actually rely
upon logistic distributions instead of normal distributions. Both
of these analyses have to assume a particular probability model
for a valid analysis, hence "model based". If one works directly
from the Hit and False Alarm rates and use the area under the
curve (AuC) directly (though AuC appears to be calculated differntly
from traditional formulas for A' which also appears to have different
forms), one does not have to make assumptions about underlying
distributions, hence "distribution-free". Kornbrut provides a
historical
and theoretical review as well as analyzes the data from Balakrishnan
(1999) in order to compare different measures of sensitivity and
bias. Much to her surprise, the form of A' she uses perform better
than the traditional d-prime. Quoting from her discussion:
|On the basis of the analyses presented here, as well as
|on the vast body of existing literature going back to the
|1950s, the signal detection approach provides a useful
|framework for describing discrimination. A distribution-free
|version provides reliable measures of sensitivity, A-prime,
|and bias, ln(Beta-prime sub K), for simple two-choice
|experiments, even without confidence ratings. Given rating
|data, one can compare models based on different distributions.
(p406)
Several measures of sensitivity and bias are used (including
ones based on Luce's decision axioms) and an important
consideration is whether one has a single pair of Hit-False
Alarm rates or rating scale data where a Hit-False Alarm
pair is provided for each point on the scale (an additional
distinction is made between scales that have, say, 5 levels
vesus 100 points [using a value between 0 and 100] which
provides some benefits). It is interesting stuff but it does
show that the application of SDT is not as simple as is usally
presented (just like Weber's ratio and Fechner's law and
Stevens power law).
The scene from the movie "Constantine" comes to mind
where John Constantine attends to his dying Kramer (the
angel Gabriel has murdered him) and Kramer says
"It's not like it says in the books." which Constantine repeats
in sorrow.
I believe that now we have officially reached N< 0 people
interested in this topic. ;-)
-Mike Palij
New York University
[email protected]
P.S. Sidenote: Diana Kornbrot pops up from time to time on
the SPSS mailing list and her posts might be characterized as
being somewhat "eccentric". Not that she is obviously wrong
in how she presents problems or issues, it is just clear that she
is thinking about things in terms that appear to be different from
that of other people.
---
You are currently subscribed to tips as: [email protected].
To unsubscribe click here:
http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=48020
or send a blank email to
leave-48020-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
