[silk] UK judge bans Bayesian logic
Fascinating. Will the next ruling ban inductive logic as well? Udhay http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage A formula for justice Bayes' theorem is a mathematical equation used in court cases to analyse statistical evidence. But a judge has ruled it can no longer be used. Will it result in more miscarriages of justice? Angela Saini guardian.co.uk, Sunday 2 October 2011 21.30 BST It's not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law. At its heart, this is a story about chance. And it begins with a convicted killer, T, who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed. But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future. It's not the first time that judges have shown hostility to using formulae. But the real worry, say forensic experts, is that the ruling could lead to miscarriages of justice. The impact will be quite shattering, says Professor Norman Fenton, a mathematician at Queen Mary, University of London. In the last four years he has been an expert witness in six cases, including the 2007 trial of Levi Bellfield for the murders of Marsha McDonnell and Amelie Delagrange. He claims that the decision in the shoeprint case threatens to damage trials now coming to court because experts like him can no longer use the maths they need. Specifically, he means a statistical tool called Bayes' theorem. Invented by an 18th-century English mathematician, Thomas Bayes, this calculates the odds of one event happening given the odds of other related events. Some mathematicians refer to it simply as logical thinking, because Bayesian reasoning is something we do naturally. If a husband tells his wife he didn't eat the leftover cake in the fridge, but she spots chocolate on his face, her estimate of his guilt goes up. But when lots of factors are involved, a Bayesian calculation is a more precise way for forensic scientists to measure the shift in guilt or innocence. In the shoeprint murder case, for example, it meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect's house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn't mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant. The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates. And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are firm. The decision could affect drug traces and fibre-matching from clothes, as well as footwear evidence, although not DNA. We hope the court of appeal will reconsider this ruling, says Colin Aitken, professor of forensic statistics at the University of Edinburgh, and the chairman of the Royal Statistical Society's working group on statistics and the law. It's usual, he explains, for forensic experts to use Bayes' theorem even when data is limited, by making assumptions and then drawing up reasonable estimates of what the numbers might be. Being unable to do this, he says, could risk miscarriages of justice. From being quite precise and being able to quantify your uncertainty, you've got to give a completely bland statement as an expert, which says 'maybe' or 'maybe not'. No numbers, explains Fenton. It's potentially very damaging, agrees University College London psychologist, Dr David Lagnado. Research has shown that people frequently make mistakes when crunching probabilities in their heads. We like a good story to explain the evidence and this makes us use statistics inappropriately, he says. When Sally Clark was convicted in 1999 of smothering her two children, jurors and judges bought into the claim that the odds of siblings dying by cot
[silk] Introducing myself
Hi everyone, Some of you already know me from the other lists. For the rest of you, here's a quick intro: I'm a 20something entrepreneur based in Singapore. I run a really small, but growing, digital games business. In a previous life I was a writer and photographer, traipsing around South Asia and the Middle East. Udhay's just added me so I'll be pleased to make your acquaintances and to take part in some of these interesting discussions. Adrianna
Re: [silk] UK judge bans Bayesian logic
Not read the judgment so take this with that caveat (although the Guardian's legal reporting is usually pretty decent). Don't see the problem with this - the CA (Court of Appeal) hasn't banned Bayesian logic or said it is wrong or anything like that. All they've said is that it shouldn't be used as evidence in (criminal?) legal proceedings (actually, they may not have gone this far and only spoken about the weight to be attached to it, but I'd have to read the judgment to check that) unless the underlying numbers used in the calculation are sound. Basically the CA's point is that if there is no confidence in the underlying numbers, then it doesn't matter how good Bayes theorem (or any other calculation) is - it's a GIGO situation, and can actually cause harm, because you get a seemingly precise number which isn't really precise at all. In that situation, it's better not to have that false certainty.. I think the broad thrust of the article - that courts should understand statistics better - is a great idea. But I'm not sure the CA has made a mistake in this instance. Badri On 3 Oct 2011, at 12:21, Udhay Shankar N wrote: Fascinating. Will the next ruling ban inductive logic as well? Udhay http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage A formula for justice Bayes' theorem is a mathematical equation used in court cases to analyse statistical evidence. But a judge has ruled it can no longer be used. Will it result in more miscarriages of justice? Angela Saini guardian.co.uk, Sunday 2 October 2011 21.30 BST It's not often that the quiet world of mathematics is rocked by a murder case. But last summer saw a trial that sent academics into a tailspin, and has since swollen into a fevered clash between science and the law. At its heart, this is a story about chance. And it begins with a convicted killer, T, who took his case to the court of appeal in 2010. Among the evidence against him was a shoeprint from a pair of Nike trainers, which seemed to match a pair found at his home. While appeals often unmask shaky evidence, this was different. This time, a mathematical formula was thrown out of court. The footwear expert made what the judge believed were poor calculations about the likelihood of the match, compounded by a bad explanation of how he reached his opinion. The conviction was quashed. But more importantly, as far as mathematicians are concerned, the judge also ruled against using similar statistical analysis in the courts in future. It's not the first time that judges have shown hostility to using formulae. But the real worry, say forensic experts, is that the ruling could lead to miscarriages of justice. The impact will be quite shattering, says Professor Norman Fenton, a mathematician at Queen Mary, University of London. In the last four years he has been an expert witness in six cases, including the 2007 trial of Levi Bellfield for the murders of Marsha McDonnell and Amelie Delagrange. He claims that the decision in the shoeprint case threatens to damage trials now coming to court because experts like him can no longer use the maths they need. Specifically, he means a statistical tool called Bayes' theorem. Invented by an 18th-century English mathematician, Thomas Bayes, this calculates the odds of one event happening given the odds of other related events. Some mathematicians refer to it simply as logical thinking, because Bayesian reasoning is something we do naturally. If a husband tells his wife he didn't eat the leftover cake in the fridge, but she spots chocolate on his face, her estimate of his guilt goes up. But when lots of factors are involved, a Bayesian calculation is a more precise way for forensic scientists to measure the shift in guilt or innocence. In the shoeprint murder case, for example, it meant figuring out the chance that the print at the crime scene came from the same pair of Nike trainers as those found at the suspect's house, given how common those kinds of shoes are, the size of the shoe, how the sole had been worn down and any damage to it. Between 1996 and 2006, for example, Nike distributed 786,000 pairs of trainers. This might suggest a match doesn't mean very much. But if you take into account that there are 1,200 different sole patterns of Nike trainers and around 42 million pairs of sports shoes sold every year, a matching pair becomes more significant. The data needed to run these kinds of calculations, though, isn't always available. And this is where the expert in this case came under fire. The judge complained that he couldn't say exactly how many of one particular type of Nike trainer there are in the country. National sales figures for sports shoes are just rough estimates. And so he decided that Bayes' theorem shouldn't again be used unless the underlying statistics are firm. The decision could affect drug
Re: [silk] Introducing myself
Welcome to Silk, Adrianna. Digital games? Interesting... some game we all know about, perhaps? I used to live in S'pore a little over a decade ago. I understand it has undergone some significant changes now. Google maps tells me that the apartment bldg I used to live is now a commercial complex. Let the thread drifting begin. :) Cheers Venkat -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Adrianna Tan skinnyla...@gmail.com wrote: Hi everyone, Some of you already know me from the other lists. For the rest of you, here's a quick intro: I'm a 20something entrepreneur based in Singapore. I run a really small, but growing, digital games business. In a previous life I was a writer and photographer, traipsing around South Asia and the Middle East. Udhay's just added me so I'll be pleased to make your acquaintances and to take part in some of these interesting discussions. Adrianna
Re: [silk] UK judge bans Bayesian logic
On Mon, Oct 3, 2011 at 4:51 PM, Udhay Shankar N ud...@pobox.com wrote: Fascinating. Will the next ruling ban inductive logic as well? Udhay http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage A formula for justice The judgment's logic is highly malleable. I don't think it means the death of Bayesian logic. If the underlying factors for a problem are based on firm data (what appears to have been decisive here is that shoe sale data in relation to each and every pair is imprecise since it isn't otherwise calculated or estimated). So if either through the use of data or through mathematical logic it can be established in another case that a higher level of precision is possible, then I don't think the court will come in the way of the use of such a method. I don't think this judgment is a criticism of Bayesian techniques - rather it's a reinforcement of the reasonable doubt standard. If there is reasonable doubt, the court must acquit. And if the mathematical formula that determines the likelihood of a crime based on circumstantial evidence is inherently based on imprecise data, then the mathematical formula does not obviate reasonable doubt and can't be reliably employed. Regards, Nikhil Mehra Advocate, Supreme Court of India Tel: (+91) 9810776904 Res: C-I/10, AIIMS Campus, Ansari Nagar (East) New Delhi - 110029.
Re: [silk] Introducing myself
Welcome to Silk, Adrianna. Digital games? Interesting... some game we all know about, perhaps? I used to live in S'pore a little over a decade ago. I understand it has undergone some significant changes now. Google maps tells me that the apartment bldg I used to live is now a commercial complex. Venkat, we just got started, so we're actually finishing up our first title. Unfortunately we have been mostly working on client projects right now as we need the capital before we go into building our own. But soon enough you'll hear about something we've done :) Singapore... ah yes the chances are insane. And it hasn't let up. I spend 3/4 my time in Kuala Lumpur, and when I go home I do feel everything's changed. This week they're launching the Circle line, for example, so when before one could not get from my house to MacRitchie reservoir in less than 2 hours by public transport, it's now down to 20 minutes or so. Some of the changes remarkable, others not so. Adrianna
Re: [silk] Introducing myself
Venkat, we just got started, so we're actually finishing up our first title. Unfortunately we have been mostly working on client projects right now as we need the capital before we go into building our own. But soon enough you'll hear about something we've done :) Welcome… Nice to see another game geek around. Casey
Re: [silk] UK judge bans Bayesian logic
On Monday 03 Oct 2011 4:51:12 pm Udhay Shankar N wrote: Fascinating. Will the next ruling ban inductive logic as well? I don't know what inductive logic means, but here are three views (at the bottom of this post) from the article. I disagree with the mathematician and agree with the psychologist and lawyer. The judge IMO is right. Statistics is abouut probability. The law is all about being innocent until proven guilty. If it was one's own ass in the firing line - or that of a dear one such as a wife or a husband one would certainly support the innocent until proven guilty attitude. We are taught to expect precision from mathematics. 2 x 2 = 4 . Period. We would never condone an airline pilot for landing in Karachi instad of Mumbai because he was unable to feed in the coordinates for navigation. 27 x 67 is exactly 1809. It is not approximately 1800, or nearly 2000 Humans continuously make guesstimates like nearly 2000 or approximately 1800. Pilots can navigate by dead reckoning and the position of the sun, but they frequently get lost. If you use mathematics the lay person expects certainty, not a propability. Any fool and his uncle would be able to come up with guesswork, hunches and probabilities. Now if a mathematician tries to convince me that his method probability and how he derives his gut feeling is mathematical and better than mine it is bullsh1t if it is not exact. I can do guesswork too. A whole lot of astrology is based on probabilities. We just don't want to believe it. It's too far fetched. But statistics is astrology based on information that we can relate to and ends up being more credible than astrology. It still deals only with probabilities and likelihoods. If you want to gamble your money, use statistics over astrology if you like. But when it comes to putting someone in jail, neither astrology nor hi funda statistics cut it. Mathematician: The impact will be quite shattering, says Professor Norman Fenton, a mathematician at Queen Mary, University of London. In the last four years he has been an expert witness in six cases, including the 2007 trial of Levi Bellfield for the murders of Marsha McDonnell and Amelie Delagrange. He claims that the decision in the shoeprint case threatens to damage trials now coming to court because experts like him can no longer use the maths they need. Psychiologist: It's potentially very damaging, agrees University College London psychologist, Dr David Lagnado. Research has shown that people frequently make mistakes when crunching probabilities in their heads. We like a good story to explain the evidence and this makes us use statistics inappropriately, he says. When Sally Clark was convicted in 1999 of smothering her two children, jurors and judges bought into the claim that the odds of siblings dying by cot death was too unlikely for her to be innocent. In fact, it was statistically more rare for a mother to kill both her children. Clark was finally freed in 2003. Lawyer: Lawyers call this type of mistake the prosecutor's fallacy, when people confuse the odds associated with a piece of evidence with the odds of guilt. Recognising this is also what eventually quashed the 1991 conviction for rape of Andrew Deen in Manchester. The courts realised at appeal that a one-in-three-million chance of a random DNA match for a semen stain from the crime scene did not mean there was only a one-in-three-million chance that anyone other than Deen could have been a match – those odds actually depend on the pool of potential suspects. In a population of 20 million adult men, for example, there could be as many as six other matches.
[silk] Fwd: UK judge bans Bayesian logic
Sorry for breaking this thread, sending along a couple of emails forwarded on with permission from another list where this was being discussed. Email #1 below --srs(iPad) Begin forwarded message: - Forwarded by Suresh Ramasubramanian1/India/IBM on 10/04/2011 07:47 AM - From:Joe St Sauver j...@oregon.uoregon.edu To:Suresh Ramasubramanian1/India/IBM@IBMIN, Date:10/03/2011 06:53 PM Subject:Re: Fw: [silk] UK judge bans Bayesian logic Hi Suresh, Thanks for passing along the pointer to the Guardian article # http://www.guardian.co.uk/law/2011/oct/02/formula-justice-bayes-theorem-miscarriage The funny thing is that the bench in this case has actually weighed in, perhaps unintentionally, on a long-standing debate in the statistical community -- folks may or may not know this, but there are different camps in the statistical community, much as there often is in many different academic disciplines. For example, in the Decision Sciences Department at UO, the chair (and my dissertation advisor) was an adherent of Fisher; another faculty member, very well regarded, was a passionate and well published advocate of Bayes. If folks are curious what all the fuss is about, I'd recommend the article, Why Isn't Everyone a Bayesian? by B. Efron, from the February 1986 American Statistician. A copy of that article is available online at http://www.isye.gatech.edu/~brani/isyebayes/bank/EfronWhyEveryone.pdf If that article is too opaque, and you'd just like to see if you yourself are a latent Bayesian, consider the classic Monty Hall game show -- for those of you who might never have seen it, Monty would select a member of the audience and offer them the opportunity to pick one of three doors. Behind one door, there might be a terrific prize, such as a new car. Behind another door, there might be a gag prize such as a lifesize crude ceramic billy goat, the perfect kitsch addition for your living room, eh? And then, behind the third door, there's some other prize, which might be pretty cool, or pretty lame, it would vary, but usually be something like a major appliance. Then again, sometimes Monty might have two lame prizes. The contestant gets to pick one door. At that point, what is his or her chance of winning the high dollar value good prize, e.g., the car? (most folks would say, 1-in-3) To make things more interesting, Monty would remind the contestant that while there's a terrific prize behind one of the doors, they might not have picked it. He'd then offer them cash-in-hand, if they want to take the money and run. To help the contestant, Monty would also open one door. Since Monty knew what door actually has the top prize, he'd never open that one. You might see, instead, a nice washer and dryer set, or maybe the goat. You'd never see the car (if there was a car). And now we come to the question that determines if you're a Fisherian or a Bayesian at heart: *what's the probability that the contestant will win the car NOW that Monty has opened one door?* Remember, there are two choices left, one of which has the car, one of which does not. Fisher and his fans would say, obviously, 1-in-2, or 50%. Bayes and his adherents would say, no, the correct answer is 2-in-3, or 66%. If you find yourself leaning toward Bayes, let me ask you an additional question: assume the audience member is given the chance to *switch* their choice, and pick the other unopened door. Should they? Would it matter? If Fisher is right, both doors have an equal chance of being right, and there's no reason why the person should switch. What would Bayesians say? :-; http://en.wikipedia.org/wiki/Monty_Hall_problem#Bayes.27_theorem Regards, Joe
[silk] Fwd: UK judge bans Bayesian logic
--srs(iPad) Begin forwarded message: - Forwarded by Suresh Ramasubramanian1/India/IBM on 10/04/2011 07:48 AM - From:Dave CROCKER dcroc...@bbiw.net To:Joe St Sauver j...@oregon.uoregon.edu, Cc:Suresh Ramasubramanian1/India/IBM@IBMIN Date:10/03/2011 08:18 PM Subject:Re: Fw: [silk] UK judge bans Bayesian logic On 10/3/2011 5:26 AM, Joe St Sauver wrote: If that article is too opaque, and you'd just like to see if you yourself are a latent Bayesian, consider the classic Monty Hall game show -- for those of you who might never have seen it, Monty would select a member of the audience and offer them the opportunity to pick one of three doors. I've long held two views that diverge from much of what is used for behavior-related research: 1. Sophisticated statistics are appropriate only when there is massively good data that is extremely well understood. Since that's rare, most use of statistics should be simple and obvious and use algorithms that are relatively IN sensitive. 2. The framework or methodology for approaching an analysis is far more important than the statistical algorithm. For example, from the Guardian article: When Sally Clark was convicted in 1999 of smothering her two children, jurors and judges bought into the claim that the odds of siblings dying by cot death was too unlikely for her to be innocent. In fact, it was statistically more rare for a mother to kill both her children. That's highlights a methodology error in the original work and it's one that is fundamental. The original trial took a statistic in isolation rather than asking about comparable choices and /their/ numbers. (One of the engineers who worked on the original HP hand caclulator in the early 70s wrote an article about its impact. He cited an experience with a banker, when he and some friends were trying to get a loan for an airplane purchase and they haggled with the banker over some of the numbers. The engineer pulled out his brand new (and extremely rare) calculator, pushed a few buttons, showed the result to the banker and the banker caved on the negotiation, without question any of the underlying details.) For most behavioral analysis, we simply do not know enough about the surrounding environment or the population to be as precise as many statistics tools imply. And too frequently that surrounding analytic framework has a deep flaw that isn't even within site of those deciding whether to accept the statistical numbers. Two anecdotes in this vein... Back when I was still in school mode, I twice got into quite a bit of trouble for my simplistic attitude. Just after dropping out of undergrad, I interviewed with the folks at Engelbart's SRI project, for a kind of user support job. (These are the folks that invented the mouse, office automatic, and otherwise laid the foundation for the work that was done at Xerox Parc and then Apple.) I had been dealing with them for a couple of years, so this was a friendly interview, until... at lunch with the guy I knew, and the guy who worked for him who would be my boss, the latter described the challenges of developing a good survey instrument to assess user 'needs'. In a fit of massive political stupidity, I noted that I had been told that such things were indeed hard to do well but that in the interim, couldn't he just /ask/ users what they wanted? He immediately stiffened and -- I swear he started looking down his nose at me -- he said that that would be methodologically naive. I looked at his boss who shrugged with an obvious meaning that this meant he knew the guy would not tolerate my working for him. We were done. On the other hand, it was my first taste of Anchor Steam Beer. And then when working at Rand, there was some spectacularly good information processing / cognitive psychology work being done by 3 very hot researchers. (The term cognitive psych was not yet in vogue for info proc work; these guys were trailblazers on the psych side and were /very/ well regarded in the field with an impressive publication record.) To get a raise at Rand, you needed to publish Rand Reports, no matter what outside publications you had. So they assembled their hottest published papers into a compendium. Rand Reports are refereed and they asked me to be a reviewer. There were few folk at Rand with a psych and computer science background, especially with any background on info processing psych. Unfortunately I was back in school by then and taking a multivarate stat course an dthe prof had just made us do an 'error' paper, where the term was not about the error part of stats algorithms but about methodological errors. In assigning the task -- we had to find an example in the literature of our field, in my case that was Human/Mass
