Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

> Hayward <hayw...@ualberta.ca> > Reply-To: "computer-go@computer-go.org" <computer-go@computer-go.org> > Date: Wednesday, 30 March 2016 at 18:59 > To: "computer-go@computer-go.org" <computer-go@computer-go.org> > Subject: Re: [Computer-go] Congratulations

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

ter-go@computer-go.org<mailto:computer-go@computer-go.org>> Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results) Hey Simon, I only now remembered: we actually experimented on the effect of making 1 blunder (random move instead of learned/searched move

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Or, if it's lopsided far from 1/2, Wilson's is just as good, in my experience. On Mar 30, 2016 10:29 AM, "Olivier Teytaud" wrote: > don't use asymptotic normality with a sample size 5, use Fisher's exact > test > > the p-value for the rejection of > "P(alpha-Go wins a given game

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

don't use asymptotic normality with a sample size 5, use Fisher's exact test the p-value for the rejection of "P(alpha-Go wins a given game against Lee Sedol)<.5" might be something like 3/16 (under the "independent coin" assumption!) this is not 0.05, but still quite an impressive result :-)

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Hey Simon, I only now remembered: we actually experimented on the effect of making 1 blunder (random move instead of learned/searched move) in Go and Hex "Blunder Cost in Go and Hex" so this might be a starting point for your question of measuring player strength by measuring all move

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

In my original post I put a link to the relevant section of the MacKay book that shows exactly how to calculate the probability of superiority assuming the game outcome is modelled as a biased coin toss: http://www.inference.phy.cam.ac.uk/itila/ I was making the point that for this and for

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

"do they have positive or negative correlation?" intriguing question, Petri. Intuitively, we might arbitrarily divide the human population into two groups; one which is discouraged by failure, and the other which takes the Lady MacBeth attitude of "screw your courage to the sticking point, and

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Since there are only two possible outcomes it pretty much normal. Actually binomial which will converge to normal given enough samples Only thing that cans distort is that consecutive games are not independent (which is probably the case but do they have positive or negative correlation?)

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

I think the error here is that the game outcome is not a normaly distributed random value. Dmitry 30.03.2016, 12:57, "djhbrown ." : > Simon wrote: "I was discussing the results with a colleague outside > of the Game AI area the other day when he raised > the question (which

[Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Simon wrote: "I was discussing the results with a colleague outside of the Game AI area the other day when he raised the question (which applies to nearly all sporting events, given the small sample size involved) of statistical significance - suggesting that on another week the result might have

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

On 22 March 2016 at 21:43, Darren Cook wrote: < snip > > > C'mon DeepMind, put that same version on KGS, set to only play 9p > players, with the same time controls, and let's get 40 games to give it > a proper ranking. (If 5 games against Lee Sedol are useful, 40 games >

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

On 23.03.2016 15:32, Petr Baudis wrote: these are beautiful posts. https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-alphagos-weakness/ Before you become too excited, also read my comments on the commentary: http://www.lifein19x19.com/forum/viewtopic.php?p=200539#p200539 --

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Thank you, these are beautiful posts. I enjoyed very much reading a writeup by Go professional who also took the effort to understand the principles behind MCTS programs as well as develop a basic intution of the gameplay artifacts, strengths and weaknesses of MCTS. It also nicely describes

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

FYI. We have translated 3 posts by Li Zhe 6p into English. https://massgoblog.wordpress.com/2016/03/11/lee-sedols-strategy-and-alphagos-weakness/ https://massgoblog.wordpress.com/2016/03/11/game-2-a-nobody-could-have-done-a-better-job-than-lee-sedol/

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

> ... > Pro players who are not familiar with MCTS bot behavior will not see this. I stand by this: >> If you want to argue that "their opinion" was wrong because they don't >> understand the game at the level AlphaGo was playing at, then you can't >> use their opinion in a positive way either.

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Hi Darren, "Darren Cook" > ... But, there were also numerous moves where > the 9-dan pros said, that in *their* opinion, the moves were weak/wrong. > E.g. wasting ko threats for no reason. Moves even a 1p would never make. > > If you want to argue that "their opinion" was wrong

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

"Lucas, Simon M" > my point is that I *think* we can say more (for example > by not treating the outcome as a black-box event, > but by appreciating the skill of the individual moves) * Human professional players were full of praise for some of AlphaGo's moves, for instance

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

> ... we witnessed hundreds of moves vetted by 9dan players, especially > Michael Redmond's, where each move was vetted. This is a promising approach. But, there were also numerous moves where the 9-dan pros said, that in *their* opinion, the moves were weak/wrong. E.g. wasting ko threats for no

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

This is somewhat moot - if any moves had been significantly and obviously weak to any observers, the results wouldn't have been 4-1. I.e. One bad move out of 5 games would give roughly the same strength information as one loss out of 5 games; consider that the kibitzing was being done in real

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

I think you are reinforcing Simon's original point; i.e. using a more fine grained approach to statically approximate AlphaGo's ELO where fine grained is degree of vetting per move and/or a series of moves. That is a substantially larger sample size and each sample will have a pretty high degree

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

] On Behalf Of Álvaro Begué Sent: 22 March 2016 17:21 To: computer-go <computer-go@computer-go.org> Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)   A very simple-minded analysis is that, if the null hypothesis is that AlphaGo and Lee Sedol are e

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Given the minimal sample size, bothering over this question won't amount to much. I think the proper response is that no one thought we'd see this level of play at this point in our AI efforts and point to the fact that we witnessed hundreds of moves vetted by 9dan players, especially Michael

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

another interesting question is to judge the bot's strength by watching the facial gestures and body language of Lee Sedol with each move... On Tue, Mar 22, 2016 at 11:46 AM, Álvaro Begué wrote: > > > On Tue, Mar 22, 2016 at 1:40 PM, Nick Wedd wrote:

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

On Tue, Mar 22, 2016 at 1:40 PM, Nick Wedd wrote: > On 22 March 2016 at 17:20, Álvaro Begué wrote: > >> A very simple-minded analysis is that, if the null hypothesis is that >> AlphaGo and Lee Sedol are equally strong, AlphaGo would do as well as we >>

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

games would go. From: Computer-go [mailto:computer-go-boun...@computer-go.org] On Behalf Of Álvaro Begué Sent: 22 March 2016 17:21 To: computer-go <computer-go@computer-go.org> Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results) A very simple-

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

A very simple-minded analysis is that, if the null hypothesis is that AlphaGo and Lee Sedol are equally strong, AlphaGo would do as well as we observed or better 15.625% of the time. That's a p-value that even social scientists don't get excited about. :) Álvaro. On Tue, Mar 22, 2016 at 12:48

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

<computer-go@computer-go.org> Subject: Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results) > I'm not sure if we can say with certainty that AlphaGo is significantly > better Go player than Lee Sedol at this point. What we can say with > certainty

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

On Tue, Mar 22, 2016 at 04:00:41PM +, Lucas, Simon M wrote: > With AlphaGo winning 4 games to 1, from a simplistic > stats point of view (with the prior assumption of a fair > coin toss) you'd not be able to claim much statistical > significance, yet most (me included) believe that > AlphaGo

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Simon, There's no argument better than evidence, and no evidence available to us other than *all* of the games that alphago has played publicly. Among two humans, a 4-1 result wouldn't indicate any more or less than this 4-1 result, but we'd already have very strong elo-type information about

Re: [Computer-go] Congratulations to AlphaGo (Statistical significance of results)

Hi all, I was discussing the results with a colleague outside of the Game AI area the other day when he raised the question (which applies to nearly all sporting events, given the small sample size involved) of statistical significance - suggesting that on another week the result might have been

Re: [Computer-go] Congratulations to AlphaGo

Here is a tinyURL link to a panel discussion of things AlphaGo that included: • Oliver Roeder: Senior writer at FiveThirtyEight. All too human. • David Doshay: Archivist for the American Go Association, co-creator of SlugGo, a Go-playing computer program. • Matt Ginsberg:

[Computer-go] Congratulations to AlphaGo

Congratulations to AlphaGo and its team! You have done what many of us could only dream to do and in such short time I may add. This is a truly historical moment and an amazing achievement for AI research! I hope this is not the end of Go and only sparks more interest in this beautiful game. What

Re: [Computer-go] Congratulations to AlphaGo

On 12.03.2016 22:03, Thomas Wolf wrote: We currently have no measure at all to judge how safe a winor loss is at any stage of the game. We have my theory according to my books for assessing the territorial and the dynamic aspects (development directions, neutral stones, statuses (incl. those

Re: [Computer-go] Congratulations to AlphaGo

The single machine version and the 2000 machine version is apparently a difference in 150 ELO, so maybe a 32 core instance in Amazon would be close enough, costing about 1200$ a month. Maybe it's doable, or for a Tokyo U. or BGA or AGA to set up something like this with Google permission. On

Re: [Computer-go] Congratulations to AlphaGo

I'd expect this achievement by AlphaGo is very similar to when the first human ran a 4 minute mile. No one had done it prior. However, right after Roger Bannister did it, suddenly there were people all over the planet doing it. Roger Bannister essentially made the possibility real, and then the

Re: [Computer-go] Congratulations to AlphaGo

At this point I don't doubt that the single machine version is professional strength which is enough to be used as a tool to analyze games... On 2016-03-12 16:24, Lukas van de Wiel wrote: This would be many thousands of dollars per day. A single game would be more than a thousand dollars in

Re: [Computer-go] Congratulations to AlphaGo

This would be many thousands of dollars per day. A single game would be more than a thousand dollars in total costs. I do not think a kickstarter project or so would be successful, as the go community is simply not *that* big... On Sun, Mar 13, 2016 at 1:17 PM, Gonçalo Mendes Ferreira

Re: [Computer-go] Congratulations to AlphaGo

It does seem unlikely for DeepMind not to move on to "bigger" things, but maybe the Go community can make some kind of fundraiser to keep an instance of AlphaGo playing 24/7? I think there are some websites for this kind of thing. Someone would be in charge of scheduling time for it to play pros,

Re: [Computer-go] Congratulations to AlphaGo

Oh, I did not say that it would not be beneficial, to AlphaGo, and to the people playing it, and to the Go community as a whole, but still, it will have to come from somewhere. Just the electricity bill alone would be hair-raising. And the big-scale benefits in prestige and marketing are over,

Re: [Computer-go] Congratulations to AlphaGo

On Sat, 12 Mar 2016, Lukas van de Wiel wrote: And the hardware available for this tournament was tremendous. It remains to be seen whether the hardware and the people maintaining it would be available for a longer period. The costs of this are not to be underestimated. Who would pay it?

Re: [Computer-go] Congratulations to AlphaGo

And the hardware available for this tournament was tremendous. It remains to be seen whether the hardware and the people maintaining it would be available for a longer period. The costs of this are not to be underestimated. Who would pay it? Lukas On Sun, Mar 13, 2016 at 12:20 PM, Clark B.

Re: [Computer-go] Congratulations to AlphaGo

On Sat, Mar 12, 2016 at 5:05 PM, Thomas Wolf wrote: > Having AlphaGo playing exclusively on KGS would be such a boost to KGS! > > For sure. The other Go servers might have their own opinion on that. Clark ___ Computer-go mailing list

Re: [Computer-go] Congratulations to AlphaGo

O'Flaherty Sent: Saturday, March 12, 2016 4:56 PM To: computer-go@computer-go.org Subject: Re: [Computer-go] Congratulations to AlphaGo   I think you're correct, Thomas. The challenge is going to be getting ANY professional to be the one who "takes handicap stones" for the first tim

Re: [Computer-go] Congratulations to AlphaGo

oo curious to see. >>> >>> Thomas >>> >>> On Sat, 12 Mar 2016, Aja Huang wrote: >>> >>> Thanks all. AlphaGo has won the match against Lee >>> Sedol. But there are still 2 games to play. >>>

Re: [Computer-go] Congratulations to AlphaGo

ory being made! Awesome! On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com <mailto:fotl...@smart-games.com> > wrote: Tremendous games by AlphaGo. Congratulations! From: Comp

Re: [Computer-go] Congratulations to AlphaGo

g wrote: >> >> Thanks all. AlphaGo has won the match against Lee >> Sedol. But there are still 2 games to play. >> Aja >> >> On Sat, Mar 12, 2016 at 5:49 PM, Jim O'Flaherty >> <jim.oflaherty...@gmail.com> wrote

Re: [Computer-go] Congratulations to AlphaGo

Of Thomas Wolf Sent: Saturday, March 12, 2016 4:04 PM To: computer-go@computer-go.org Subject: Re: [Computer-go] Congratulations to AlphaGo Chris, Prompted from a discussion on the computer go email list (and my last email today) : We currently have no measure at all to judge how safe a winor loss

Re: [Computer-go] Congratulations to AlphaGo

to witness history being made! Awesome! On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> wrote:       Tremendous games by AlphaGo.  Congratulations!               From: Computer-go

Re: [Computer-go] Congratulations to AlphaGo

ll 2 games to play. >> Aja >> >> On Sat, Mar 12, 2016 at 5:49 PM, Jim O'Flaherty < >> jim.oflaherty...@gmail.com> wrote: >> It was exhilerating to witness history being made! Awesome! >> >> On Sat, Mar 12, 2016 at 2:17 AM, David Fotland &l

Re: [Computer-go] Congratulations to AlphaGo

Hi Ingo, I have the manuscript of 2 books with each 100 computer generated problems which 1st class insei Yutae Seo (Korea) picked out of 20,000 computer generated problems, working for 5 months on this selection. Many have a tricky ko status. I am happy to provide them. Simpler even and more

Re: [Computer-go] Congratulations to AlphaGo

Give it the hatsuyo-ron :) On Mar 12, 2016, at 11:12 AM, Ingo Althöfer wrote: > Hi Thomas, > > Von: "Thomas Wolf" >> A suggestion for possible future games to be arranged between AlphaGo and >> strong players: >> >> Whoever lost shall be given 1 stone or the equivalent of 1/2

Re: [Computer-go] Congratulations to AlphaGo

Hi Thomas, Von: "Thomas Wolf" > A suggestion for possible future games to be arranged between AlphaGo and > strong players: > > Whoever lost shall be given 1 stone or the equivalent of 1/2 stone handcap in > the > next game. Games should continue until each side has won at

Re: [Computer-go] Congratulations to AlphaGo

e! Awesome! >> >> On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> >> wrote: >> >> Tremendous games by AlphaGo. Congratulations! >> >> >> >> From: Computer-go [mailto:computer-go-boun...@computer-go.

Re: [Computer-go] Congratulations to AlphaGo

com> wrote: It was exhilerating to witness history being made! Awesome! On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> wrote: Tremendous games by AlphaGo.  Congratulations!   From: Computer-go [mailto:computer-go-boun...@computer-go.or

Re: [Computer-go] Congratulations to AlphaGo

Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> >> wrote: >> >>> Tremendous games by AlphaGo. Congratulations! >>> >>> >>> >>> *From:* Computer-go [mailto:computer-go-boun...@computer-go.org] *On >>> Beha

Re: [Computer-go] Congratulations to AlphaGo

Dear Aja, congratulations to you and the whole AlphaGo team for this awesome performance of your bot! I fully understand that you take also the last two games seriously. But, please, do it in such a way that Lee Sedol will be willing to support you in further matches to come. Enjoy the day,

Re: [Computer-go] Congratulations to AlphaGo

;> >>> It was exhilerating to witness history being made! Awesome! >>> >>> On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> >>> wrote: >>> >>>> Tremendous games by AlphaGo. Congratulations! >>>> >

Re: [Computer-go] Congratulations to AlphaGo

tory being made! Awesome! >> >> On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> >> wrote: >> >>> Tremendous games by AlphaGo. Congratulations! >>> >>> >>> >>> *From:* Computer-go [mailt

Re: [Computer-go] Congratulations to AlphaGo

Well done, Aja and all the DeepMind team (including all the "backroom boys" who've given the reliability on the hardware side). BTW, I've gained great pleasure seeing you sitting there with the union jack, representing queen and country; you'll probably receive a knighthood. :-) > Thanks all.

Re: [Computer-go] Congratulations to AlphaGo

Congratulations to DeepMind team! What an excellent full-board perspective. Hideki Lukas van de Wiel:

Re: [Computer-go] Congratulations to AlphaGo

It was exhilerating to witness history being made! Awesome! On Sat, Mar 12, 2016 at 2:17 AM, David Fotland <fotl...@smart-games.com> wrote: > Tremendous games by AlphaGo. Congratulations! > > > > *From:* Computer-go [mailto:computer-go-boun...@computer-go.org] *On > Beha

Re: [Computer-go] Congratulations to AlphaGo

Tremendous games by AlphaGo. Congratulations! From: Computer-go [mailto:computer-go-boun...@computer-go.org] On Behalf Of Lukas van de Wiel Sent: Saturday, March 12, 2016 12:14 AM To: computer-go@computer-go.org Subject: [Computer-go] Congratulations to AlphaGo Whoa, what a fight! Well

[Computer-go] Congratulations to AlphaGo

Whoa, what a fight! Well fought, and well won! Lukas ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go