Carol, E-mail in three parts 1. The activity I use to demonstrate SDT 2. Why SDT is useful and applicable 3. Why ROC curves are better in application
PART 1 I use "the dice game" activity when teaching SDT and ROC curves and find that it helps students really grasp how shifting the criterion has no effect on estimated d' but does change estimated beta. How the game is played. I role 3 six-sided dice. Two of the dice are normal ranging from 1-6 and the third die (called the signal) is either 0 (1-3) or 1 (4-6). The goal of the game is to determine based on the total number of all three dice whether the signal die is a 0 or a 1. The regular dice produce the noise in which the signal is either hidden or not. You can play the game a few times and then ask students how they decide when to say signal or no signal, most will develop a natural criterion point and which totals above some number result in saying signal (you may need an aside on the gambler's fallacy too). You can manipulate signal strength by making the value of the signal die larger (e.g., 0,3 or 0,6) and play again. They will see that the stronger the signal, the easier it is to be accurate. You can also introduce pay-off matrices in terms of points for hits vs. correct rejections and watch their criterion shift in one direction or the other. This is all fun but the real power of the game is in the next step. You can create the probability function for both outcomes for every dice total (and it isn't overwhelming because there are only 36 possible noise totals and 36 possible signal+noise totals). For example: a total of 2 must be (1-1-0; with the last number representing the signal die value). A total of 4 can be (1-3-0, 3-1-0, 2-2-0 for the no signal combinations and 1-2-1, 2-1-1 for the signal present combinations). I have my students draw these on graph paper and the patterns of number of combinations becomes obvious. Further, if they draw the distributions for two different signal strengths and they will see the s+n curve shift to the right. Once you have these distributions you can choose any criterion (let's say 8 or higher total I say signal) and calculate the hit and false alarm rate. Hit rate will be 21/36 or 58.3% (there are 21 combinations of the two regular dice plus 1 that produce a total of 8 or higher) and the false alarm rate will be 15/36 or 41.6%. With these two values students can use a computational estimate for d' (d'=z(hit)-z(FA)). I have a spreadsheet that does this OR use this website by Ian Neath (http://memory.psych.mun.ca/models/dprime/). Thus for the example d' is .422 and beta (is 1 which isn't computed on the website). Students can chose different criterion and should note that d' changes only slightly (because it is an estimate) but beta will shift (the website uses C which is easier to interpret because no bais is zero with values being either positive for conservative - less accepting of a Type 1 error - criterion point and negative for less conservative - less accepting of a Type 2 error). PART 2 The primary value of SDT is for comparison of two circumstances where there is bias toward one type of error or the other and you wish to compare the two situations. For example, lets say we are designing a severe weather indicator for small aircraft. One display results in 97% hits (correctly recognizing severe weather when it is present) but also produces (65% false alarms). Is that display better or worse than one that produces only 80% hits and 9% false alarms? Based on SDT estimates of d' the second display is better (d' of 2.18 for the latter and 1.49 for the former). The big difference is that the two displays produce different bias in responding and if we were to adopt the same level of bias in the second display resulting in 97% hit rate we would find that the associated FA rate would be 38%. Gee, wouldn't it be nice if we could somehow visualize how that works? You can with an ROC curve. But the even more important question is do you want a weather display that encourages MORE risky decisions even if it is better in terms of absolute signal detection? I use SDT analysis all the time in Human Factors applications, you'll find it (or a derivative) in medial research and anyone who has been to the eye doctor should be able to appreciate that comparing two images repeatedly until you can't tell a difference could be considered a process of driving d' between the two option to zero (I'll have to think about this one a little more). PART 3 I take the example I explained in PART 2 and plot it with hit rate on the y-axis and FA rate on the X-axis. Two points are difficult to compare because one has a much better hit rate but the other has a better FA rate. Assuming we can manipulate bias in our observers you can use instructions or incentives to generate more points and start to estimate the curve associated with each system. Recognizing that a system with no ability to distinguish will produce a straight line with a slope of 1 (that is FA rate and Hit rate rise and fall together) we have a representation of what a system with d'=0 would look like. The more the curve bows away from that straight line the stronger the signal strength, responses in that system will fall along that curve depending on the bias with the neutral bias falling along a line perpendicular to the d'=0 line and extending to the upper left corner (not many examples using google images have this line but you can find one). From here you can talk about fuzzy signal detection theory with three outcome states (no signal, not sure, signal). The simplest use is to treat the "not sure" as no signal in one computation and as a signal response in a second and you get two points from the same system and now you can estimate the curve. I realize this is long and I just tried to explain in e-mail what I spend an entire day of class talking about but I hope it helps. I'd be happy to make another attempt at explanation or maybe making a short video/screen capture explanations. SDT continues to be applicable in a number of settings, particularly medical tests, many use a the AUC that Mike mentions and while this isn't technically SDT (no z transforms) the ROC method is identical (here is a short and good example http://www.nature.com/nmeth/journal/v12/n9/fig_tab/nmeth.3482_SF9.html) All the best, Doug Doug Peterson, PhD Associate Professor of Psychology The University of South Dakota Vermillion SD 57069 605.677.5295 ________________________________________ From: Carol DeVolder [[email protected]] Sent: Thursday, January 28, 2016 10:06 PM To: Teaching in the Psychological Sciences (TIPS) Subject: [tips] signal detection and ROC curves Dear TIPSters, I am currently teaching about the Theory of Signal Detectability, Stevens's Power Law, and ROC curves in my Sensation and Perception course. Do any of you have any examples that you work on in class or use to illustrate how to implement them? I want to do several things. First, I want to be able to explain the logic of SDT, the power law, and ROCs. Second, I want to be able to make the topics relevant and convince the students that these concepts are active in their daily lives. And third, I want to give them some opportunities to practice. I've already talked about hits, misses, false alarms, and correct rejections in class, and using payoffs to manipulate response criteria, now I want to make it all applicable.I welcome any and all ideas. Thank you very much. Carol Carol DeVolder, Ph.D. Professor of Psychology St. Ambrose University 518 West Locust Street Davenport, Iowa 52803 563-333-6482 --- You are currently subscribed to tips as: [email protected]<mailto:[email protected]>. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=12991.6a54289b29ceb58cb7609cc50e0dc1c8&n=T&l=tips&o=48000 (It may be necessary to cut and paste the above URL if the line is broken) or send a blank email to leave-48000-12991.6a54289b29ceb58cb7609cc50e0dc...@fsulist.frostburg.edu<mailto:leave-48000-12991.6a54289b29ceb58cb7609cc50e0dc...@fsulist.frostburg.edu> --- You are currently subscribed to tips as: [email protected]. To unsubscribe click here: http://fsulist.frostburg.edu/u?id=13090.68da6e6e5325aa33287ff385b70df5d5&n=T&l=tips&o=48010 or send a blank email to leave-48010-13090.68da6e6e5325aa33287ff385b70df...@fsulist.frostburg.edu
