Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Ryan Hayward]

no

On Thu, Mar 31, 2016 at 8:33 AM, Lucas, Simon M <s...@essex.ac.uk> wrote:

> Thanks Ryan,
>
> Nice paper – did you follow up on any of the future work?
>
>   Simon
>
>
>
> From: Computer-go <computer-go-boun...@computer-go.org> on behalf of Ryan
> Hayward <hayw...@ualberta.ca>
> Reply-To: "computer-go@computer-go.org" <computer-go@computer-go.org>
> Date: Wednesday, 30 March 2016 at 18:59
> To: "computer-go@computer-go.org" <computer-go@computer-go.org>
> Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical
> significance of results)
>
> Hey Simon,
>
> I only now remembered:
>
> we actually experimented on the effect
> of making 1 blunder (random move instead of learned/searched move)
> in Go and Hex
>
> "Blunder Cost in Go and Hex"
>
> so this might be a starting point for your question
> of measuring player strength by measuring
> all move strengths...
>
> https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf
>
> On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M <s...@essex.ac.uk> wrote:
>
>> In my original post I put a link to
>> the relevant section of the MacKay
>> book that shows exactly how to calculate
>> the probability of superiority
>> assuming the game outcome is modelled as
>> a biased coin toss:
>>
>> http://www.inference.phy.cam.ac.uk/itila/
>>
>>
>> I was making the point that for this
>>
>> and for other outcomes of skill-based games
>> we can do so much more (and as humans we intuitively
>> DO do so much more) than just look at the event
>> outcome - and maybe as a community we should do that more
>> routinely and more quantitatively (e.g.
>> by analysing the quality of each move / action)
>>
>> Best wishes,
>>
>>   Simon
>>
>>
>>
>> On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." <
>> computer-go-boun...@computer-go.org on behalf of djhbr...@gmail.com>
>> wrote:
>>
>> >Simon wrote: "I was discussing the results with a colleague outside
>> >of the Game AI area the other day when he raised
>> >the question (which applies to nearly all sporting events,
>> >given the small sample size involved)
>> >of statistical significance - suggesting that on another week
>> >the result might have been 4-1 to Lee Sedol."
>> >
>> >call me naive, but perhaps you could ask your colleague to calculate
>> >the probability one of side winning 4 games out of 5, and then say
>> >whether that is within 2 standard deviations of the norm.
>> >
>> >his suggestion is complete nonsense, regardless of the small sample
>> >size.  perhaps you could ask a statistician next time.
>> >
>> >--
>> >patient: "whenever i open my mouth, i get a shooting pain in my foot"
>> >doctor: "fire!"
>> >http://sites.google.com/site/djhbrown2/home
>> >https://www.youtube.com/user/djhbrown
>> >___
>> >Computer-go mailing list
>> >Computer-go@computer-go.org
>> >http://computer-go.org/mailman/listinfo/computer-go
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
>
>
> --
> Ryan B Hayward
> Professor and Director (Outreach+Diversity)
> Computing Science,  UAlberta
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>



-- 
Ryan B Hayward
Professor and Director (Outreach+Diversity)
Computing Science,  UAlberta
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Lucas, Simon M]

Thanks Ryan,

Nice paper – did you follow up on any of the future work?

  Simon



From: Computer-go 
<computer-go-boun...@computer-go.org<mailto:computer-go-boun...@computer-go.org>>
 on behalf of Ryan Hayward <hayw...@ualberta.ca<mailto:hayw...@ualberta.ca>>
Reply-To: "computer-go@computer-go.org<mailto:computer-go@computer-go.org>" 
<computer-go@computer-go.org<mailto:computer-go@computer-go.org>>
Date: Wednesday, 30 March 2016 at 18:59
To: "computer-go@computer-go.org<mailto:computer-go@computer-go.org>" 
<computer-go@computer-go.org<mailto:computer-go@computer-go.org>>
Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance 
of results)

Hey Simon,

I only now remembered:

we actually experimented on the effect
of making 1 blunder (random move instead of learned/searched move)
in Go and Hex

"Blunder Cost in Go and Hex"

so this might be a starting point for your question
of measuring player strength by measuring
all move strengths...

https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf

On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M 
<s...@essex.ac.uk<mailto:s...@essex.ac.uk>> wrote:
In my original post I put a link to
the relevant section of the MacKay
book that shows exactly how to calculate
the probability of superiority
assuming the game outcome is modelled as
a biased coin toss:

http://www.inference.phy.cam.ac.uk/itila/


I was making the point that for this

and for other outcomes of skill-based games
we can do so much more (and as humans we intuitively
DO do so much more) than just look at the event
outcome - and maybe as a community we should do that more
routinely and more quantitatively (e.g.
by analysing the quality of each move / action)

Best wishes,

  Simon



On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." 
<computer-go-boun...@computer-go.org<mailto:computer-go-boun...@computer-go.org>
 on behalf of djhbr...@gmail.com<mailto:djhbr...@gmail.com>> wrote:

>Simon wrote: "I was discussing the results with a colleague outside
>of the Game AI area the other day when he raised
>the question (which applies to nearly all sporting events,
>given the small sample size involved)
>of statistical significance - suggesting that on another week
>the result might have been 4-1 to Lee Sedol."
>
>call me naive, but perhaps you could ask your colleague to calculate
>the probability one of side winning 4 games out of 5, and then say
>whether that is within 2 standard deviations of the norm.
>
>his suggestion is complete nonsense, regardless of the small sample
>size.  perhaps you could ask a statistician next time.
>
>--
>patient: "whenever i open my mouth, i get a shooting pain in my foot"
>doctor: "fire!"
>http://sites.google.com/site/djhbrown2/home
>https://www.youtube.com/user/djhbrown
>___
>Computer-go mailing list
>Computer-go@computer-go.org<mailto:Computer-go@computer-go.org>
>http://computer-go.org/mailman/listinfo/computer-go
___
Computer-go mailing list
Computer-go@computer-go.org<mailto:Computer-go@computer-go.org>
http://computer-go.org/mailman/listinfo/computer-go



--
Ryan B Hayward
Professor and Director (Outreach+Diversity)
Computing Science,  UAlberta
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[uurtamo .]

Or, if it's lopsided far from 1/2, Wilson's is just as good, in my
experience.
On Mar 30, 2016 10:29 AM, "Olivier Teytaud"  wrote:

> don't use asymptotic normality with a sample size 5, use Fisher's exact
> test
>
> the p-value for the rejection of
> "P(alpha-Go wins a given game against Lee Sedol)<.5"
> might be something like 3/16
> (under the "independent coin" assumption!)
>
> this is not 0.05, but still quite an impressive result :-)
>
> with 5-0 it would have been <0.05.
>
>
>
> On Wed, Mar 30, 2016 at 6:59 PM, Ryan Hayward  wrote:
>
>> Hey Simon,
>>
>> I only now remembered:
>>
>> we actually experimented on the effect
>> of making 1 blunder (random move instead of learned/searched move)
>> in Go and Hex
>>
>> "Blunder Cost in Go and Hex"
>>
>> so this might be a starting point for your question
>> of measuring player strength by measuring
>> all move strengths...
>>
>> https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf
>>
>> On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M  wrote:
>>
>>> In my original post I put a link to
>>> the relevant section of the MacKay
>>> book that shows exactly how to calculate
>>> the probability of superiority
>>> assuming the game outcome is modelled as
>>> a biased coin toss:
>>>
>>> http://www.inference.phy.cam.ac.uk/itila/
>>>
>>>
>>> I was making the point that for this
>>>
>>> and for other outcomes of skill-based games
>>> we can do so much more (and as humans we intuitively
>>> DO do so much more) than just look at the event
>>> outcome - and maybe as a community we should do that more
>>> routinely and more quantitatively (e.g.
>>> by analysing the quality of each move / action)
>>>
>>> Best wishes,
>>>
>>>   Simon
>>>
>>>
>>>
>>> On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." <
>>> computer-go-boun...@computer-go.org on behalf of djhbr...@gmail.com>
>>> wrote:
>>>
>>> >Simon wrote: "I was discussing the results with a colleague outside
>>> >of the Game AI area the other day when he raised
>>> >the question (which applies to nearly all sporting events,
>>> >given the small sample size involved)
>>> >of statistical significance - suggesting that on another week
>>> >the result might have been 4-1 to Lee Sedol."
>>> >
>>> >call me naive, but perhaps you could ask your colleague to calculate
>>> >the probability one of side winning 4 games out of 5, and then say
>>> >whether that is within 2 standard deviations of the norm.
>>> >
>>> >his suggestion is complete nonsense, regardless of the small sample
>>> >size.  perhaps you could ask a statistician next time.
>>> >
>>> >--
>>> >patient: "whenever i open my mouth, i get a shooting pain in my foot"
>>> >doctor: "fire!"
>>> >http://sites.google.com/site/djhbrown2/home
>>> >https://www.youtube.com/user/djhbrown
>>> >___
>>> >Computer-go mailing list
>>> >Computer-go@computer-go.org
>>> >http://computer-go.org/mailman/listinfo/computer-go
>>> ___
>>> Computer-go mailing list
>>> Computer-go@computer-go.org
>>> http://computer-go.org/mailman/listinfo/computer-go
>>>
>>
>>
>>
>> --
>> Ryan B Hayward
>> Professor and Director (Outreach+Diversity)
>> Computing Science,  UAlberta
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
>
>
> --
> =
> Olivier Teytaud, olivier.teyt...@inria.fr, TAO, LRI, UMR 8623(CNRS -
> Univ. Paris-Sud),
> bat 490 Univ. Paris-Sud F-91405 Orsay Cedex France
> http://www.slideshare.net/teytaud
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Olivier Teytaud]

don't use asymptotic normality with a sample size 5, use Fisher's exact test

the p-value for the rejection of
"P(alpha-Go wins a given game against Lee Sedol)<.5"
might be something like 3/16
(under the "independent coin" assumption!)

this is not 0.05, but still quite an impressive result :-)

with 5-0 it would have been <0.05.



On Wed, Mar 30, 2016 at 6:59 PM, Ryan Hayward  wrote:

> Hey Simon,
>
> I only now remembered:
>
> we actually experimented on the effect
> of making 1 blunder (random move instead of learned/searched move)
> in Go and Hex
>
> "Blunder Cost in Go and Hex"
>
> so this might be a starting point for your question
> of measuring player strength by measuring
> all move strengths...
>
> https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf
>
> On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M  wrote:
>
>> In my original post I put a link to
>> the relevant section of the MacKay
>> book that shows exactly how to calculate
>> the probability of superiority
>> assuming the game outcome is modelled as
>> a biased coin toss:
>>
>> http://www.inference.phy.cam.ac.uk/itila/
>>
>>
>> I was making the point that for this
>>
>> and for other outcomes of skill-based games
>> we can do so much more (and as humans we intuitively
>> DO do so much more) than just look at the event
>> outcome - and maybe as a community we should do that more
>> routinely and more quantitatively (e.g.
>> by analysing the quality of each move / action)
>>
>> Best wishes,
>>
>>   Simon
>>
>>
>>
>> On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." <
>> computer-go-boun...@computer-go.org on behalf of djhbr...@gmail.com>
>> wrote:
>>
>> >Simon wrote: "I was discussing the results with a colleague outside
>> >of the Game AI area the other day when he raised
>> >the question (which applies to nearly all sporting events,
>> >given the small sample size involved)
>> >of statistical significance - suggesting that on another week
>> >the result might have been 4-1 to Lee Sedol."
>> >
>> >call me naive, but perhaps you could ask your colleague to calculate
>> >the probability one of side winning 4 games out of 5, and then say
>> >whether that is within 2 standard deviations of the norm.
>> >
>> >his suggestion is complete nonsense, regardless of the small sample
>> >size.  perhaps you could ask a statistician next time.
>> >
>> >--
>> >patient: "whenever i open my mouth, i get a shooting pain in my foot"
>> >doctor: "fire!"
>> >http://sites.google.com/site/djhbrown2/home
>> >https://www.youtube.com/user/djhbrown
>> >___
>> >Computer-go mailing list
>> >Computer-go@computer-go.org
>> >http://computer-go.org/mailman/listinfo/computer-go
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
>
>
> --
> Ryan B Hayward
> Professor and Director (Outreach+Diversity)
> Computing Science,  UAlberta
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>



-- 
=
Olivier Teytaud, olivier.teyt...@inria.fr, TAO, LRI, UMR 8623(CNRS - Univ.
Paris-Sud),
bat 490 Univ. Paris-Sud F-91405 Orsay Cedex France
http://www.slideshare.net/teytaud
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Ryan Hayward]

Hey Simon,

I only now remembered:

we actually experimented on the effect
of making 1 blunder (random move instead of learned/searched move)
in Go and Hex

"Blunder Cost in Go and Hex"

so this might be a starting point for your question
of measuring player strength by measuring
all move strengths...

https://webdocs.cs.ualberta.ca/~hayward/papers/blunder.pdf

On Wed, Mar 30, 2016 at 5:29 AM, Lucas, Simon M  wrote:

> In my original post I put a link to
> the relevant section of the MacKay
> book that shows exactly how to calculate
> the probability of superiority
> assuming the game outcome is modelled as
> a biased coin toss:
>
> http://www.inference.phy.cam.ac.uk/itila/
>
>
> I was making the point that for this
>
> and for other outcomes of skill-based games
> we can do so much more (and as humans we intuitively
> DO do so much more) than just look at the event
> outcome - and maybe as a community we should do that more
> routinely and more quantitatively (e.g.
> by analysing the quality of each move / action)
>
> Best wishes,
>
>   Simon
>
>
>
> On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." <
> computer-go-boun...@computer-go.org on behalf of djhbr...@gmail.com>
> wrote:
>
> >Simon wrote: "I was discussing the results with a colleague outside
> >of the Game AI area the other day when he raised
> >the question (which applies to nearly all sporting events,
> >given the small sample size involved)
> >of statistical significance - suggesting that on another week
> >the result might have been 4-1 to Lee Sedol."
> >
> >call me naive, but perhaps you could ask your colleague to calculate
> >the probability one of side winning 4 games out of 5, and then say
> >whether that is within 2 standard deviations of the norm.
> >
> >his suggestion is complete nonsense, regardless of the small sample
> >size.  perhaps you could ask a statistician next time.
> >
> >--
> >patient: "whenever i open my mouth, i get a shooting pain in my foot"
> >doctor: "fire!"
> >http://sites.google.com/site/djhbrown2/home
> >https://www.youtube.com/user/djhbrown
> >___
> >Computer-go mailing list
> >Computer-go@computer-go.org
> >http://computer-go.org/mailman/listinfo/computer-go
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>



-- 
Ryan B Hayward
Professor and Director (Outreach+Diversity)
Computing Science,  UAlberta
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Lucas, Simon M]

In my original post I put a link to
the relevant section of the MacKay 
book that shows exactly how to calculate
the probability of superiority 
assuming the game outcome is modelled as 
a biased coin toss:

http://www.inference.phy.cam.ac.uk/itila/


I was making the point that for this

and for other outcomes of skill-based games
we can do so much more (and as humans we intuitively
DO do so much more) than just look at the event
outcome - and maybe as a community we should do that more
routinely and more quantitatively (e.g.
by analysing the quality of each move / action)

Best wishes,

  Simon



On 30/03/2016, 11:57, "Computer-go on behalf of djhbrown ." 
 wrote:

>Simon wrote: "I was discussing the results with a colleague outside
>of the Game AI area the other day when he raised
>the question (which applies to nearly all sporting events,
>given the small sample size involved)
>of statistical significance - suggesting that on another week
>the result might have been 4-1 to Lee Sedol."
>
>call me naive, but perhaps you could ask your colleague to calculate
>the probability one of side winning 4 games out of 5, and then say
>whether that is within 2 standard deviations of the norm.
>
>his suggestion is complete nonsense, regardless of the small sample
>size.  perhaps you could ask a statistician next time.
>
>-- 
>patient: "whenever i open my mouth, i get a shooting pain in my foot"
>doctor: "fire!"
>http://sites.google.com/site/djhbrown2/home
>https://www.youtube.com/user/djhbrown
>___
>Computer-go mailing list
>Computer-go@computer-go.org
>http://computer-go.org/mailman/listinfo/computer-go
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[djhbrown .]

"do they have positive or negative correlation?"  intriguing question,
Petri.  Intuitively, we might arbitrarily divide the human population
into two groups; one which is discouraged by failure, and the other
which takes the Lady MacBeth attitude of "screw your courage to the
sticking point, and we'll not fail !"  Again, intuitively, we might
expect the two groups to be roughly equal in size, whereupon it
follows that a series of results involving randomly chosen players
would not be markedly different from statistical independence.

these intuitions are corroborated by one experimental study i came
across with Google Scholar's help:
http://www.tandfonline.com/doi/abs/10.1198/016214501753168217

On 30/03/2016, Petri Pitkanen  wrote:
> Since there are only two possible outcomes it pretty much normal. Actually
> binomial which will converge to normal given enough samples
>
> Only thing that cans distort is that consecutive games are not
> independent (which
> is probably the case but do they have positive or negative correlation?)
>
> 2016-03-30 13:06 GMT+03:00 Рождественский Дмитрий :
>
>> I think the error here is that the game outcome is not a normaly
>> distributed random value.
>>
>> Dmitry
>>
>> 30.03.2016, 12:57, "djhbrown ." :
>> > Simon wrote: "I was discussing the results with a colleague outside
>> > of the Game AI area the other day when he raised
>> > the question (which applies to nearly all sporting events,
>> > given the small sample size involved)
>> > of statistical significance - suggesting that on another week
>> > the result might have been 4-1 to Lee Sedol."
>> >
>> > call me naive, but perhaps you could ask your colleague to calculate
>> > the probability one of side winning 4 games out of 5, and then say
>> > whether that is within 2 standard deviations of the norm.
>> >
>> > his suggestion is complete nonsense, regardless of the small sample
>> > size. perhaps you could ask a statistician next time.
>> >
>> > --
>> > patient: "whenever i open my mouth, i get a shooting pain in my foot"
>> > doctor: "fire!"
>> > http://sites.google.com/site/djhbrown2/home
>> > https://www.youtube.com/user/djhbrown
>> > ___
>> > Computer-go mailing list
>> > Computer-go@computer-go.org
>> > http://computer-go.org/mailman/listinfo/computer-go
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>


-- 
patient: "whenever i open my mouth, i get a shooting pain in my foot"
doctor: "fire!"
http://sites.google.com/site/djhbrown2/home
https://www.youtube.com/user/djhbrown
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Petri Pitkanen]

Since there are only two possible outcomes it pretty much normal. Actually
binomial which will converge to normal given enough samples

Only thing that cans distort is that consecutive games are not
independent (which
is probably the case but do they have positive or negative correlation?)

2016-03-30 13:06 GMT+03:00 Рождественский Дмитрий :

> I think the error here is that the game outcome is not a normaly
> distributed random value.
>
> Dmitry
>
> 30.03.2016, 12:57, "djhbrown ." :
> > Simon wrote: "I was discussing the results with a colleague outside
> > of the Game AI area the other day when he raised
> > the question (which applies to nearly all sporting events,
> > given the small sample size involved)
> > of statistical significance - suggesting that on another week
> > the result might have been 4-1 to Lee Sedol."
> >
> > call me naive, but perhaps you could ask your colleague to calculate
> > the probability one of side winning 4 games out of 5, and then say
> > whether that is within 2 standard deviations of the norm.
> >
> > his suggestion is complete nonsense, regardless of the small sample
> > size. perhaps you could ask a statistician next time.
> >
> > --
> > patient: "whenever i open my mouth, i get a shooting pain in my foot"
> > doctor: "fire!"
> > http://sites.google.com/site/djhbrown2/home
> > https://www.youtube.com/user/djhbrown
> > ___
> > Computer-go mailing list
> > Computer-go@computer-go.org
> > http://computer-go.org/mailman/listinfo/computer-go
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Рождественский Дмитрий]

I think the error here is that the game outcome is not a normaly distributed 
random value.

Dmitry

30.03.2016, 12:57, "djhbrown ." :
> Simon wrote: "I was discussing the results with a colleague outside
> of the Game AI area the other day when he raised
> the question (which applies to nearly all sporting events,
> given the small sample size involved)
> of statistical significance - suggesting that on another week
> the result might have been 4-1 to Lee Sedol."
>
> call me naive, but perhaps you could ask your colleague to calculate
> the probability one of side winning 4 games out of 5, and then say
> whether that is within 2 standard deviations of the norm.
>
> his suggestion is complete nonsense, regardless of the small sample
> size. perhaps you could ask a statistician next time.
>
> --
> patient: "whenever i open my mouth, i get a shooting pain in my foot"
> doctor: "fire!"
> http://sites.google.com/site/djhbrown2/home
> https://www.youtube.com/user/djhbrown
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Nick Wedd]

On 22 March 2016 at 21:43, Darren Cook  wrote:

< snip >

>
> C'mon DeepMind, put that same version on KGS, set to only play 9p
> players, with the same time controls, and let's get 40 games to give it
> a proper ranking. (If 5 games against Lee Sedol are useful, 40 games
> against a range of players with little to lose, who are systematically
> trying to find its weaknesses, are going to be amazing.)


I don't think DeepMind's real objective was to create a world Go champion.
I think it was to build an engine that could learn to perform difficult
intellectual tasks better than any human; and they chose Go for their first
test because there was a clear objective measure of success.  If I am
right, we won't see much more of their "Go engine", instead they will
direct their resources to training their general-purpose engine to do other
more useful things.

Nick
-- 
Nick Wedd  mapr...@gmail.com
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Robert Jasiek]

On 23.03.2016 15:32, Petr Baudis wrote:

these are beautiful posts.

https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-alphagos-weakness/


Before you become too excited, also read my comments on the commentary:
http://www.lifein19x19.com/forum/viewtopic.php?p=200539#p200539

--
robert jasiek
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Petr Baudis]

  Thank you, these are beautiful posts.  I enjoyed very much reading
a writeup by Go professional who also took the effort to understand the
principles behind MCTS programs as well as develop a basic intution of
the gameplay artifacts, strengths and weaknesses of MCTS.  It also
nicely describes the reason and consequences for Lee Sedol's different
approaches during the match.

  I think these are really great posts for Go programmers to share with
their Go-playing friends if they are curious about AlphaGo's strength
and what went on in these games!

On Tue, Mar 22, 2016 at 06:26:45PM -0400, Chun Sun wrote:
> FYI. We have translated 3 posts by Li Zhe 6p into English.
> 
> https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-alphagos-weakness/
> https://massgoblog.wordpress.com/2016/03/11/game-2-a-nobody-could-have-done-a-better-job-than-lee-sedol/
> https://massgoblog.wordpress.com/2016/03/15/before-game-5/
> 
> These may provide a slightly different perspective than many other pros.
> 
> 
> On Tue, Mar 22, 2016 at 6:21 PM, Darren Cook  wrote:
> 
> > > ...
> > > Pro players who are not familiar with MCTS bot behavior will not see
> > this.
> >
> > I stand by this:
> >
> > >> If you want to argue that "their opinion" was wrong because they don't
> > >> understand the game at the level AlphaGo was playing at, then you can't
> > >> use their opinion in a positive way either.
> >
> > Darren
> >
> > ___
> > Computer-go mailing list
> > Computer-go@computer-go.org
> > http://computer-go.org/mailman/listinfo/computer-go
> >

> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go


-- 
Petr Baudis
If you have good ideas, good data and fast computers,
you can do almost anything. -- Geoffrey Hinton
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Chun Sun]

FYI. We have translated 3 posts by Li Zhe 6p into English.

https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-alphagos-weakness/
https://massgoblog.wordpress.com/2016/03/11/game-2-a-nobody-could-have-done-a-better-job-than-lee-sedol/
https://massgoblog.wordpress.com/2016/03/15/before-game-5/

These may provide a slightly different perspective than many other pros.


On Tue, Mar 22, 2016 at 6:21 PM, Darren Cook  wrote:

> > ...
> > Pro players who are not familiar with MCTS bot behavior will not see
> this.
>
> I stand by this:
>
> >> If you want to argue that "their opinion" was wrong because they don't
> >> understand the game at the level AlphaGo was playing at, then you can't
> >> use their opinion in a positive way either.
>
> Darren
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Darren Cook]

> ...
> Pro players who are not familiar with MCTS bot behavior will not see this.

I stand by this:

>> If you want to argue that "their opinion" was wrong because they don't
>> understand the game at the level AlphaGo was playing at, then you can't
>> use their opinion in a positive way either.

Darren

___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Ingo Althöfer]

Hi Darren,

"Darren Cook" 
> ... But, there were also numerous moves where
> the 9-dan pros said, that in *their* opinion, the moves were weak/wrong.
> E.g. wasting ko threats for no reason. Moves even a 1p would never make.
>
> If you want to argue that "their opinion" was wrong because they don't
> understand the game at the level AlphaGo was playing at, then you can't
> use their opinion in a positive way either.

For these situations there is a very natural explanation, at least for
computer go insiders: The seemingly weak moves happened bcause AlphaGo simply 
tried to maximize the winning probability and not the expected score. 

Pro players who are not familiar with MCTS bot behavior will not see this.


Ingo.
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Ingo Althöfer]

"Lucas, Simon M" 
> my point is that I *think* we can say more (for example
> by not treating the outcome as a black-box event,
> but by appreciating the skill of the individual moves)


* Human professional players were full of praise for some of
AlphaGo's moves, for instance move 37 in game 2.


* Although the bots Zen and Crazy are not independent witnesses:
they both saw AlphaGo on the winning path early on in all four won games.


* The score order 1-0, 2-0, 3-0, 3-1, 4-1 with the Sedol win in the
second match half is an indicator that he may have learned something
about the opponent during the early games. 

Ingo.
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Darren Cook]

> ... we witnessed hundreds of moves vetted by 9dan players, especially
> Michael Redmond's, where each move was vetted. 

This is a promising approach. But, there were also numerous moves where
the 9-dan pros said, that in *their* opinion, the moves were weak/wrong.
E.g. wasting ko threats for no reason. Moves even a 1p would never make.

If you want to argue that "their opinion" was wrong because they don't
understand the game at the level AlphaGo was playing at, then you can't
use their opinion in a positive way either.

> nearly all sporting events, given the small sample size involved) of
> statistical significance - suggesting that on another week the result
> might have been 4-1 to Lee Sedol.

If his 2nd game had been the one where he created vaguely alive/dead
groups and forced a mistake, and given that we were told the computer
was not being changed during the match, he might have created 2 wins
just by playing exactly the same.

And if he had known this in advance he might then have realized that
creating multiple weak groups and some large complicated kos are the way
to beat it, and so it could well have gone 4-1 to Lee Sedol in "another
week".

C'mon DeepMind, put that same version on KGS, set to only play 9p
players, with the same time controls, and let's get 40 games to give it
a proper ranking. (If 5 games against Lee Sedol are useful, 40 games
against a range of players with little to lose, who are systematically
trying to find its weaknesses, are going to be amazing.)

Darren
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[uurtamo .]

This is somewhat moot - if any moves had been significantly and obviously
weak to any observers, the results wouldn't have been 4-1.

I.e. One bad move out of 5 games would give roughly the same strength
information as one loss out of 5 games; consider that the kibitzing was
being done in real time.

s.
On Mar 22, 2016 11:08 AM, "Jim O'Flaherty" 
wrote:

> I think you are reinforcing Simon's original point; i.e. using a more fine
> grained approach to statically approximate AlphaGo's ELO where fine grained
> is degree of vetting per move and/or a series of moves. That is a
> substantially larger sample size and each sample will have a pretty high
> degree of quality (given the vetting is being done by top level
> professionals).
> On Mar 22, 2016 1:04 PM, "Jeffrey Greenberg" 
> wrote:
>
>> Given the minimal sample size, bothering over this question won't amount
>> to much. I think the proper response is that no one thought we'd see this
>> level of play at this point in our AI efforts and point to the fact that we
>> witnessed hundreds of moves vetted by 9dan players, especially Michael
>> Redmond's, where each move was vetted. In other words "was the level of
>> play very high?" versus the question "have we beat all humans". The answer
>> is more or less, yes.
>>
>> On Tuesday, March 22, 2016, Lucas, Simon M  wrote:
>>
>>> Hi all,
>>>
>>> I was discussing the results with a colleague outside
>>> of the Game AI area the other day when he raised
>>> the question (which applies to nearly all sporting events,
>>> given the small sample size involved)
>>> of statistical significance - suggesting that on another week
>>> the result might have been 4-1 to Lee Sedol.
>>>
>>> I pointed out that in games of skill there's much more to judge than
>>> just the final
>>> outcome of each game, but wondered if anyone had any better (or worse :)
>>> arguments - or had even engaged in the same type of
>>> conversation.
>>>
>>> With AlphaGo winning 4 games to 1, from a simplistic
>>> stats point of view (with the prior assumption of a fair
>>> coin toss) you'd not be able to claim much statistical
>>> significance, yet most (me included) believe that
>>> AlphaGo is a genuinely better Go player than Lee Sedol.
>>>
>>> From a stats viewpoint you can use this approach:
>>> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
>>> (see section 3.2 on page 51)
>>>
>>> but given even priors it won't tell you much.
>>>
>>> Anyone know any good references for refuting this
>>> type of argument - the fact is of course that a game of Go
>>> is nothing like a coin toss.  Games of skill tend to base their
>>> outcomes on the result of many (in the case of Go many hundreds of)
>>> individual actions.
>>>
>>> Best wishes,
>>>
>>>   Simon
>>>
>>>
>>> ___
>>> Computer-go mailing list
>>> Computer-go@computer-go.org
>>> http://computer-go.org/mailman/listinfo/computer-go
>>
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Jim O'Flaherty]

I think you are reinforcing Simon's original point; i.e. using a more fine
grained approach to statically approximate AlphaGo's ELO where fine grained
is degree of vetting per move and/or a series of moves. That is a
substantially larger sample size and each sample will have a pretty high
degree of quality (given the vetting is being done by top level
professionals).
On Mar 22, 2016 1:04 PM, "Jeffrey Greenberg"  wrote:

> Given the minimal sample size, bothering over this question won't amount
> to much. I think the proper response is that no one thought we'd see this
> level of play at this point in our AI efforts and point to the fact that we
> witnessed hundreds of moves vetted by 9dan players, especially Michael
> Redmond's, where each move was vetted. In other words "was the level of
> play very high?" versus the question "have we beat all humans". The answer
> is more or less, yes.
>
> On Tuesday, March 22, 2016, Lucas, Simon M  wrote:
>
>> Hi all,
>>
>> I was discussing the results with a colleague outside
>> of the Game AI area the other day when he raised
>> the question (which applies to nearly all sporting events,
>> given the small sample size involved)
>> of statistical significance - suggesting that on another week
>> the result might have been 4-1 to Lee Sedol.
>>
>> I pointed out that in games of skill there's much more to judge than just
>> the final
>> outcome of each game, but wondered if anyone had any better (or worse :)
>> arguments - or had even engaged in the same type of
>> conversation.
>>
>> With AlphaGo winning 4 games to 1, from a simplistic
>> stats point of view (with the prior assumption of a fair
>> coin toss) you'd not be able to claim much statistical
>> significance, yet most (me included) believe that
>> AlphaGo is a genuinely better Go player than Lee Sedol.
>>
>> From a stats viewpoint you can use this approach:
>> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
>> (see section 3.2 on page 51)
>>
>> but given even priors it won't tell you much.
>>
>> Anyone know any good references for refuting this
>> type of argument - the fact is of course that a game of Go
>> is nothing like a coin toss.  Games of skill tend to base their
>> outcomes on the result of many (in the case of Go many hundreds of)
>> individual actions.
>>
>> Best wishes,
>>
>>   Simon
>>
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Thomas Wolf]

I am sorry, but I think this discussion is a bit pointless.
While I write these 3 lines and you read them, AlphGo got 20 ELO 
points stronger. :-)


Thomas

On Tue, 22 Mar 2016, Lucas, Simon M wrote:



Still an interesting question is how one could make

more powerful inferences by observing the skill of

the players in each action they take rather than just

the final outcome of each game.

 

If you saw me play a single game of tennis against Federer

you’d have no doubt as to which way the next 100 games would go.

 

From: Computer-go [mailto:computer-go-boun...@computer-go.org] On Behalf Of 
Álvaro Begué
Sent: 22 March 2016 17:21
To: computer-go <computer-go@computer-go.org>
Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance 
of results)

 

A very simple-minded analysis is that, if the null hypothesis is that AlphaGo 
and Lee Sedol are
equally strong, AlphaGo would do as well as we observed or better 15.625% of 
the time. That's a
p-value that even social scientists don't get excited about. :)

Álvaro.

 

On Tue, Mar 22, 2016 at 12:48 PM, Jason House <jason.james.ho...@gmail.com> 
wrote:

  Statistical significance requires a null hypothesis... I think it's 
probably easiest to
  ask the question of if I assume an ELO difference of x, how likely it's a 
4-1 result?
  Turns out that 220 to 270 ELO has a 41% chance of that result.
  >= 10% is -50 to 670 ELO
  >= 1% is -250 to 1190 ELO
  My numbers may be slightly off from eyeballing things in a simple excel 
sheet. The idea
  and ranges should be clear though

  On Mar 22, 2016 12:00 PM, "Lucas, Simon M" <s...@essex.ac.uk> wrote:

Hi all,

I was discussing the results with a colleague outside
of the Game AI area the other day when he raised
the question (which applies to nearly all sporting events,
given the small sample size involved)
of statistical significance - suggesting that on another week
the result might have been 4-1 to Lee Sedol.

I pointed out that in games of skill there's much more to judge 
than just the
final
outcome of each game, but wondered if anyone had any better (or 
worse :)
arguments - or had even engaged in the same type of
conversation.

With AlphaGo winning 4 games to 1, from a simplistic
stats point of view (with the prior assumption of a fair
coin toss) you'd not be able to claim much statistical
significance, yet most (me included) believe that
AlphaGo is a genuinely better Go player than Lee Sedol.

From a stats viewpoint you can use this approach:
http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
(see section 3.2 on page 51)

but given even priors it won't tell you much.

Anyone know any good references for refuting this
type of argument - the fact is of course that a game of Go
is nothing like a coin toss.  Games of skill tend to base their
outcomes on the result of many (in the case of Go many hundreds of)
individual actions.

Best wishes,

  Simon


___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go


___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

 


___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Jeffrey Greenberg]

Given the minimal sample size, bothering over this question won't amount to
much. I think the proper response is that no one thought we'd see this
level of play at this point in our AI efforts and point to the fact that we
witnessed hundreds of moves vetted by 9dan players, especially Michael
Redmond's, where each move was vetted. In other words "was the level of
play very high?" versus the question "have we beat all humans". The answer
is more or less, yes.

On Tuesday, March 22, 2016, Lucas, Simon M  wrote:

> Hi all,
>
> I was discussing the results with a colleague outside
> of the Game AI area the other day when he raised
> the question (which applies to nearly all sporting events,
> given the small sample size involved)
> of statistical significance - suggesting that on another week
> the result might have been 4-1 to Lee Sedol.
>
> I pointed out that in games of skill there's much more to judge than just
> the final
> outcome of each game, but wondered if anyone had any better (or worse :)
> arguments - or had even engaged in the same type of
> conversation.
>
> With AlphaGo winning 4 games to 1, from a simplistic
> stats point of view (with the prior assumption of a fair
> coin toss) you'd not be able to claim much statistical
> significance, yet most (me included) believe that
> AlphaGo is a genuinely better Go player than Lee Sedol.
>
> From a stats viewpoint you can use this approach:
> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
> (see section 3.2 on page 51)
>
> but given even priors it won't tell you much.
>
> Anyone know any good references for refuting this
> type of argument - the fact is of course that a game of Go
> is nothing like a coin toss.  Games of skill tend to base their
> outcomes on the result of many (in the case of Go many hundreds of)
> individual actions.
>
> Best wishes,
>
>   Simon
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org 
> http://computer-go.org/mailman/listinfo/computer-go
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Ryan Hayward]

another interesting question is to judge the bot's strength
by watching the facial gestures and body language of Lee Sedol
with each move...

On Tue, Mar 22, 2016 at 11:46 AM, Álvaro Begué 
wrote:

>
>
> On Tue, Mar 22, 2016 at 1:40 PM, Nick Wedd  wrote:
>
>> On 22 March 2016 at 17:20, Álvaro Begué  wrote:
>>
>>> A very simple-minded analysis is that, if the null hypothesis is that
>>> AlphaGo and Lee Sedol are equally strong, AlphaGo would do as well as we
>>> observed or better 15.625% of the time. That's a p-value that even social
>>> scientists don't get excited about. :)
>>>
>>>
>> "For "as well ... or better", I make it 18.75%.
>>
>
> I obviously can't count. :)
>
> Thanks for the correction.
>
> Álvaro.
>
>
>
>
>>
>> Nick
>>
>>
>>
>>> Álvaro.
>>>
>>>
>>> On Tue, Mar 22, 2016 at 12:48 PM, Jason House <
>>> jason.james.ho...@gmail.com> wrote:
>>>
 Statistical significance requires a null hypothesis... I think it's
 probably easiest to ask the question of if I assume an ELO difference of x,
 how likely it's a 4-1 result?
 Turns out that 220 to 270 ELO has a 41% chance of that result.
 >= 10% is -50 to 670 ELO
 >= 1% is -250 to 1190 ELO
 My numbers may be slightly off from eyeballing things in a simple excel
 sheet. The idea and ranges should be clear though
 On Mar 22, 2016 12:00 PM, "Lucas, Simon M"  wrote:

> Hi all,
>
> I was discussing the results with a colleague outside
> of the Game AI area the other day when he raised
> the question (which applies to nearly all sporting events,
> given the small sample size involved)
> of statistical significance - suggesting that on another week
> the result might have been 4-1 to Lee Sedol.
>
> I pointed out that in games of skill there's much more to judge than
> just the final
> outcome of each game, but wondered if anyone had any better (or worse
> :)
> arguments - or had even engaged in the same type of
> conversation.
>
> With AlphaGo winning 4 games to 1, from a simplistic
> stats point of view (with the prior assumption of a fair
> coin toss) you'd not be able to claim much statistical
> significance, yet most (me included) believe that
> AlphaGo is a genuinely better Go player than Lee Sedol.
>
> From a stats viewpoint you can use this approach:
> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
> (see section 3.2 on page 51)
>
> but given even priors it won't tell you much.
>
> Anyone know any good references for refuting this
> type of argument - the fact is of course that a game of Go
> is nothing like a coin toss.  Games of skill tend to base their
> outcomes on the result of many (in the case of Go many hundreds of)
> individual actions.
>
> Best wishes,
>
>   Simon
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go


 ___
 Computer-go mailing list
 Computer-go@computer-go.org
 http://computer-go.org/mailman/listinfo/computer-go

>>>
>>>
>>> ___
>>> Computer-go mailing list
>>> Computer-go@computer-go.org
>>> http://computer-go.org/mailman/listinfo/computer-go
>>>
>>
>>
>>
>> --
>> Nick Wedd  mapr...@gmail.com
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>



-- 
Ryan B Hayward
Professor and Director (Outreach+Diversity)
Computing Science,  UAlberta
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Álvaro Begué]

On Tue, Mar 22, 2016 at 1:40 PM, Nick Wedd  wrote:

> On 22 March 2016 at 17:20, Álvaro Begué  wrote:
>
>> A very simple-minded analysis is that, if the null hypothesis is that
>> AlphaGo and Lee Sedol are equally strong, AlphaGo would do as well as we
>> observed or better 15.625% of the time. That's a p-value that even social
>> scientists don't get excited about. :)
>>
>>
> "For "as well ... or better", I make it 18.75%.
>

I obviously can't count. :)

Thanks for the correction.

Álvaro.




>
> Nick
>
>
>
>> Álvaro.
>>
>>
>> On Tue, Mar 22, 2016 at 12:48 PM, Jason House <
>> jason.james.ho...@gmail.com> wrote:
>>
>>> Statistical significance requires a null hypothesis... I think it's
>>> probably easiest to ask the question of if I assume an ELO difference of x,
>>> how likely it's a 4-1 result?
>>> Turns out that 220 to 270 ELO has a 41% chance of that result.
>>> >= 10% is -50 to 670 ELO
>>> >= 1% is -250 to 1190 ELO
>>> My numbers may be slightly off from eyeballing things in a simple excel
>>> sheet. The idea and ranges should be clear though
>>> On Mar 22, 2016 12:00 PM, "Lucas, Simon M"  wrote:
>>>
 Hi all,

 I was discussing the results with a colleague outside
 of the Game AI area the other day when he raised
 the question (which applies to nearly all sporting events,
 given the small sample size involved)
 of statistical significance - suggesting that on another week
 the result might have been 4-1 to Lee Sedol.

 I pointed out that in games of skill there's much more to judge than
 just the final
 outcome of each game, but wondered if anyone had any better (or worse :)
 arguments - or had even engaged in the same type of
 conversation.

 With AlphaGo winning 4 games to 1, from a simplistic
 stats point of view (with the prior assumption of a fair
 coin toss) you'd not be able to claim much statistical
 significance, yet most (me included) believe that
 AlphaGo is a genuinely better Go player than Lee Sedol.

 From a stats viewpoint you can use this approach:
 http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
 (see section 3.2 on page 51)

 but given even priors it won't tell you much.

 Anyone know any good references for refuting this
 type of argument - the fact is of course that a game of Go
 is nothing like a coin toss.  Games of skill tend to base their
 outcomes on the result of many (in the case of Go many hundreds of)
 individual actions.

 Best wishes,

   Simon


 ___
 Computer-go mailing list
 Computer-go@computer-go.org
 http://computer-go.org/mailman/listinfo/computer-go
>>>
>>>
>>> ___
>>> Computer-go mailing list
>>> Computer-go@computer-go.org
>>> http://computer-go.org/mailman/listinfo/computer-go
>>>
>>
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>>
>
>
>
> --
> Nick Wedd  mapr...@gmail.com
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Lucas, Simon M]

Still an interesting question is how one could make
more powerful inferences by observing the skill of
the players in each action they take rather than just
the final outcome of each game.

If you saw me play a single game of tennis against Federer
you’d have no doubt as to which way the next 100 games would go.

From: Computer-go [mailto:computer-go-boun...@computer-go.org] On Behalf Of 
Álvaro Begué
Sent: 22 March 2016 17:21
To: computer-go <computer-go@computer-go.org>
Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance 
of results)

A very simple-minded analysis is that, if the null hypothesis is that AlphaGo 
and Lee Sedol are equally strong, AlphaGo would do as well as we observed or 
better 15.625% of the time. That's a p-value that even social scientists don't 
get excited about. :)

Álvaro.

On Tue, Mar 22, 2016 at 12:48 PM, Jason House 
<jason.james.ho...@gmail.com<mailto:jason.james.ho...@gmail.com>> wrote:

Statistical significance requires a null hypothesis... I think it's probably 
easiest to ask the question of if I assume an ELO difference of x, how likely 
it's a 4-1 result?
Turns out that 220 to 270 ELO has a 41% chance of that result.
>= 10% is -50 to 670 ELO
>= 1% is -250 to 1190 ELO
My numbers may be slightly off from eyeballing things in a simple excel sheet. 
The idea and ranges should be clear though
On Mar 22, 2016 12:00 PM, "Lucas, Simon M" 
<s...@essex.ac.uk<mailto:s...@essex.ac.uk>> wrote:
Hi all,

I was discussing the results with a colleague outside
of the Game AI area the other day when he raised
the question (which applies to nearly all sporting events,
given the small sample size involved)
of statistical significance - suggesting that on another week
the result might have been 4-1 to Lee Sedol.

I pointed out that in games of skill there's much more to judge than just the 
final
outcome of each game, but wondered if anyone had any better (or worse :)
arguments - or had even engaged in the same type of
conversation.

With AlphaGo winning 4 games to 1, from a simplistic
stats point of view (with the prior assumption of a fair
coin toss) you'd not be able to claim much statistical
significance, yet most (me included) believe that
AlphaGo is a genuinely better Go player than Lee Sedol.

From a stats viewpoint you can use this approach:
http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
(see section 3.2 on page 51)

but given even priors it won't tell you much.

Anyone know any good references for refuting this
type of argument - the fact is of course that a game of Go
is nothing like a coin toss.  Games of skill tend to base their
outcomes on the result of many (in the case of Go many hundreds of)
individual actions.

Best wishes,

  Simon


___
Computer-go mailing list
Computer-go@computer-go.org<mailto:Computer-go@computer-go.org>
http://computer-go.org/mailman/listinfo/computer-go

___
Computer-go mailing list
Computer-go@computer-go.org<mailto:Computer-go@computer-go.org>
http://computer-go.org/mailman/listinfo/computer-go

___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Álvaro Begué]

A very simple-minded analysis is that, if the null hypothesis is that
AlphaGo and Lee Sedol are equally strong, AlphaGo would do as well as we
observed or better 15.625% of the time. That's a p-value that even social
scientists don't get excited about. :)

Álvaro.


On Tue, Mar 22, 2016 at 12:48 PM, Jason House 
wrote:

> Statistical significance requires a null hypothesis... I think it's
> probably easiest to ask the question of if I assume an ELO difference of x,
> how likely it's a 4-1 result?
> Turns out that 220 to 270 ELO has a 41% chance of that result.
> >= 10% is -50 to 670 ELO
> >= 1% is -250 to 1190 ELO
> My numbers may be slightly off from eyeballing things in a simple excel
> sheet. The idea and ranges should be clear though
> On Mar 22, 2016 12:00 PM, "Lucas, Simon M"  wrote:
>
>> Hi all,
>>
>> I was discussing the results with a colleague outside
>> of the Game AI area the other day when he raised
>> the question (which applies to nearly all sporting events,
>> given the small sample size involved)
>> of statistical significance - suggesting that on another week
>> the result might have been 4-1 to Lee Sedol.
>>
>> I pointed out that in games of skill there's much more to judge than just
>> the final
>> outcome of each game, but wondered if anyone had any better (or worse :)
>> arguments - or had even engaged in the same type of
>> conversation.
>>
>> With AlphaGo winning 4 games to 1, from a simplistic
>> stats point of view (with the prior assumption of a fair
>> coin toss) you'd not be able to claim much statistical
>> significance, yet most (me included) believe that
>> AlphaGo is a genuinely better Go player than Lee Sedol.
>>
>> From a stats viewpoint you can use this approach:
>> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
>> (see section 3.2 on page 51)
>>
>> but given even priors it won't tell you much.
>>
>> Anyone know any good references for refuting this
>> type of argument - the fact is of course that a game of Go
>> is nothing like a coin toss.  Games of skill tend to base their
>> outcomes on the result of many (in the case of Go many hundreds of)
>> individual actions.
>>
>> Best wishes,
>>
>>   Simon
>>
>>
>> ___
>> Computer-go mailing list
>> Computer-go@computer-go.org
>> http://computer-go.org/mailman/listinfo/computer-go
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
>
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Lucas, Simon M]

my point is that I *think* we can say more (for example
by not treating the outcome as a black-box event,
but by appreciating the skill of the individual moves)

From: Computer-go [mailto:computer-go-boun...@computer-go.org] On Behalf Of 
uurtamo .
Sent: 22 March 2016 16:25
To: computer-go <computer-go@computer-go.org>
Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance 
of results)


> I'm not sure if we can say with certainty that AlphaGo is significantly
> better Go player than Lee Sedol at this point.  What we can say with
> certainty is that AlphaGo is in the same ballpark and at least roughly
> as strong as Lee Sedol.  To me, that's enough to be really huge on its
> own accord!

Agreed, and exactly what I'm telling my friends who have asked the same 
question.

s.
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Petr Baudis]

On Tue, Mar 22, 2016 at 04:00:41PM +, Lucas, Simon M wrote:
> With AlphaGo winning 4 games to 1, from a simplistic
> stats point of view (with the prior assumption of a fair
> coin toss) you'd not be able to claim much statistical 
> significance, yet most (me included) believe that
> AlphaGo is a genuinely better Go player than Lee Sedol.
> 
> From a stats viewpoint you can use this approach:
> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
> (see section 3.2 on page 51)
> 
> but given even priors it won't tell you much.

What complicates things further is that the coin distribution is
non-stationary; definitely from the point of the human performance
against a fixed program (how much AlphaGo with its novel RL component
is fixed is of course another matter).  In fact, as anyone watching bots
playing on KGS knows, initially the non-stationarity is actually very
extreme as the human gets "used to" the computer's style and soon are
able to beat even a program that's formally quite stronger than the
human player.  At least that's the case for the weaker programs.

I'm not sure if we can say with certainty that AlphaGo is significantly
better Go player than Lee Sedol at this point.  What we can say with
certainty is that AlphaGo is in the same ballpark and at least roughly
as strong as Lee Sedol.  To me, that's enough to be really huge on its
own accord!

-- 
Petr Baudis
If you have good ideas, good data and fast computers,
you can do almost anything. -- Geoffrey Hinton
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[uurtamo .]

Simon,

There's no argument better than evidence, and no evidence available to us
other than *all* of the games that alphago has played publicly.

Among two humans, a 4-1 result wouldn't indicate any more or less than this
4-1 result, but we'd already have very strong elo-type information about
both humans because they both would have publicly played hundreds of games
to get to such a match.

I believe alphago played another match earlier in public, correct?  Then we
now have double the evidence, or a slight (50% or so) improvement in our
confidence bounds.

s.
On Mar 22, 2016 9:00 AM, "Lucas, Simon M"  wrote:

> Hi all,
>
> I was discussing the results with a colleague outside
> of the Game AI area the other day when he raised
> the question (which applies to nearly all sporting events,
> given the small sample size involved)
> of statistical significance - suggesting that on another week
> the result might have been 4-1 to Lee Sedol.
>
> I pointed out that in games of skill there's much more to judge than just
> the final
> outcome of each game, but wondered if anyone had any better (or worse :)
> arguments - or had even engaged in the same type of
> conversation.
>
> With AlphaGo winning 4 games to 1, from a simplistic
> stats point of view (with the prior assumption of a fair
> coin toss) you'd not be able to claim much statistical
> significance, yet most (me included) believe that
> AlphaGo is a genuinely better Go player than Lee Sedol.
>
> From a stats viewpoint you can use this approach:
> http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
> (see section 3.2 on page 51)
>
> but given even priors it won't tell you much.
>
> Anyone know any good references for refuting this
> type of argument - the fact is of course that a game of Go
> is nothing like a coin toss.  Games of skill tend to base their
> outcomes on the result of many (in the case of Go many hundreds of)
> individual actions.
>
> Best wishes,
>
>   Simon
>
>
> ___
> Computer-go mailing list
> Computer-go@computer-go.org
> http://computer-go.org/mailman/listinfo/computer-go
___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

[Lucas, Simon M]

Hi all,

I was discussing the results with a colleague outside
of the Game AI area the other day when he raised
the question (which applies to nearly all sporting events,
given the small sample size involved)
of statistical significance - suggesting that on another week
the result might have been 4-1 to Lee Sedol.

I pointed out that in games of skill there's much more to judge than just the 
final
outcome of each game, but wondered if anyone had any better (or worse :) 
arguments - or had even engaged in the same type of
conversation.

With AlphaGo winning 4 games to 1, from a simplistic
stats point of view (with the prior assumption of a fair
coin toss) you'd not be able to claim much statistical 
significance, yet most (me included) believe that
AlphaGo is a genuinely better Go player than Lee Sedol.

From a stats viewpoint you can use this approach:
http://www.inference.phy.cam.ac.uk/itprnn/book.pdf
(see section 3.2 on page 51)

but given even priors it won't tell you much.

Anyone know any good references for refuting this
type of argument - the fact is of course that a game of Go
is nothing like a coin toss.  Games of skill tend to base their
outcomes on the result of many (in the case of Go many hundreds of)
individual actions.

Best wishes,

  Simon


___
Computer-go mailing list
Computer-go@computer-go.org
http://computer-go.org/mailman/listinfo/computer-go