Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
I analyzed these positions with Valkyria, and it has no problems seeing what is happening in these positions. Also going back some moves valkyria for example proposes Ws8 instead of Wp12 for move 218, which clearly kills the black group and reduces it no eyes, and showes that Valkyria does know what it is doing (it is not perfect but play acceptable with a few seconds of thinking timew most of time). My point here is just that playing strongly in one type of semeai does not mean it will crush you. But you are right in that analyzing positions like this can identify systematic weaknesse of programs. The problem is that at least for how valkyria works there is not one single fix, the problem is often that a special situation in the playout has to be safely identified and a particulrly bad move need to be pruned, or a deterministic response has to be made to some special kind of threat. But these changes rarely apply to more than a small proportion of the positions encountered in games. The main playing strength come from efficient search in general, and the knowledge that apply to common shapes rather than special tricky situations. Nethertheless I discoverd a nice way of finding tricky situations where evaluations goes very wrong. I run long tests whith 50-500 playouts per move. I also allow Valkyria to resign in these tests. This makes testing go faster and as side effect one sometimes get very long games because the losing colour did not understand it was losing. So I play a 1000 such test games and look for unusually long games. Often it was just a complicated even fight but sometimes one finds some huge misevaluation to fix. Still fixing those things rarely give a measurable boost to playing strength. But I hope they will add up in the long run. Best Magnus Quoting terry mcintyre terrymcint...@yahoo.com: I haven't got a ladder example at the moment, but here's an instance where Leela does not realize it is in terrible trouble. I ( with my 8 kyu AGA rating) know with certainty by move 223 (T5) that Black has captured a large white group. A stronger player could read this out sooner than I. This fight is too big to lose for either side; nothing else on the board matters. ( anyone? how early is this outcome pre-ordained? ) Based on the results of its analysis mode, Leela does not recognize the outcome of this semeai until the large white group in the bottom right is down to two liberties. The problem is even more stark in example2 -- similar board, black has foolishly played one of his own liberties for illustrative purposes. It is black's play, black has three liberties, white has three. Black must take away a liberty from white to win the capturing race, or make two eyes at T8. Black has only four playable moves; any other choice fails. Leela proposes - even after several minutes of analysis and a million nodes - that Black should tennuki at H14. That would snatch defeat from the jaws of certain victory; White would dive into T8 and win the race. I started this thread with the contention that analysis mode can help developers find problems, I hope this example explains why. My theory is that if a program could reliably recognize the outcome of such capturing races five or ten moves sooner, it could crush the likes of me. :D Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy From: Michael Williams michaelwilliam...@gmail.com To: computer-go computer-go@computer-go.org Sent: Tuesday, April 21, 2009 1:57:54 PM Subject: Re: [computer-go] Reply to Lukasz and Don + Roadmap 2020 Mention the program so that the author can either refute your claim or fix the bug. terry mcintyre wrote: Is it reasonable to expect pro players to use 6-dan programs as a tool for analysis? The pro players are markedly better - at a rough guess, a pro player could give a 6 dan amateur human or program a 3 stone handicap. On the other end of the scale, beginning players and mid kyu players could indeed make good use of an analysis mode by a program which is better than themselves. Lastly, an analysis mode would be helpful to developers, methinks. After winning a game, I like to back up a few moves and find out when the program realized that it was behind. This often happens several moves after the fatal blow has already been struck. I know the feeling too well, when stronger players deftly skewer my group and I only discover the problem five moves later. What do they know that I don't? What do they know that the program doesn't? We have a saying, you learn the most from reviewing games which you have lost. An analysis mode can help developers to discover when their pride and joy first begins to miss the target. Lately, I have been playing quite
Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
I appreciate that Go programs are complex and not easy to tune. Thinking over Magnus' excellent automated method ( play many games, allow early resignation, inspect long games ), and my own experiences, I'd like to suggest an additional method: when a game ends with a large loss, determine retroactively a) how far back the program was doomed but deluded about the outcome, and b) what the correct earlier plays would have been. Tune and test with a regression suite of difficult positions. This is like troubleshooting any other large and complex program; a subtle error in one portion may only reveal itself when a perfect storm of circumstances arises - but the bug is there all the time. As Magnus and Valkyria pointed out, proper play at an earlier point in my example game would have destroyed Black's position. I don't feel so proud now, lol. At my level of play, it is distressingly common to mis-read capturing races -- it's possible that a good understanding of that topic would improve Go programs. The difficulty cannot be understated - others have indicated that Hunter's Counting Liberties and Winning Capturing Races book, valuable as it may be, misses some cases. As Hunter observes, even dan-level players sometimes make mistakes in capturing races. Programs which get semeai and seki right every time might be a few stones stronger. They'd certainly be more valuable as teaching tools. In the game above, a stronger program would have exploited my earlier weakness; this would have encouraged me to make better moves. Back to Roadmap: 2020, I'd love a status map showing groups which are certainly alive, groups which are unstable, and groups which are certainly dead ( assuming proper play ). That would be quite a feedback tool. When an approach move or throw-in threatens the status of a group, the group marker would change from green to blinking yellow. When the attack succeeds, the group marker would change to red. To be useful, this tool would have to be accurate. If not 100% accurate, it should at least give some indication of its level of confidence. Even better, especially for double-digit-kyu players, would be an exposition of why a group is live, dead, seki, unstable, etc. Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] Roadmap 2020 - using analysis mode to improve programs
Many faces will show group status, but with letters on the stones, not colors. From: computer-go-boun...@computer-go.org [mailto:computer-go-boun...@computer-go.org] On Behalf Of terry mcintyre Sent: Thursday, April 23, 2009 7:04 AM To: computer-go Subject: Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs Back to Roadmap: 2020, I'd love a status map showing groups which are certainly alive, groups which are unstable, and groups which are certainly dead ( assuming proper play ). That would be quite a feedback tool. When an approach move or throw-in threatens the status of a group, the group marker would change from green to blinking yellow. When the attack succeeds, the group marker would change to red. To be useful, this tool would have to be accurate. If not 100% accurate, it should at least give some indication of its level of confidence. Even better, especially for double-digit-kyu players, would be an exposition of why a group is live, dead, seki, unstable, etc. Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
RE: [computer-go] Roadmap 2020 - using analysis mode to improve programs
For example1, Many Faces' Game Score Graph shows the fight is over around move 208. From: computer-go-boun...@computer-go.org [mailto:computer-go-boun...@computer-go.org] On Behalf Of terry mcintyre Sent: Wednesday, April 22, 2009 6:27 PM To: computer-go Subject: Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs I haven't got a ladder example at the moment, but here's an instance where Leela does not realize it is in terrible trouble. I ( with my 8 kyu AGA rating) know with certainty by move 223 (T5) that Black has captured a large white group. A stronger player could read this out sooner than I. This fight is too big to lose for either side; nothing else on the board matters. ( anyone? how early is this outcome pre-ordained? ) Based on the results of its analysis mode, Leela does not recognize the outcome of this semeai until the large white group in the bottom right is down to two liberties. The problem is even more stark in example2 -- similar board, black has foolishly played one of his own liberties for illustrative purposes. It is black's play, black has three liberties, white has three. Black must take away a liberty from white to win the capturing race, or make two eyes at T8. Black has only four playable moves; any other choice fails. Leela proposes - even after several minutes of analysis and a million nodes - that Black should tennuki at H14. That would snatch defeat from the jaws of certain victory; White would dive into T8 and win the race. I started this thread with the contention that analysis mode can help developers find problems, I hope this example explains why. My theory is that if a program could reliably recognize the outcome of such capturing races five or ten moves sooner, it could crush the likes of me. :D Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy _ From: Michael Williams michaelwilliam...@gmail.com To: computer-go computer-go@computer-go.org Sent: Tuesday, April 21, 2009 1:57:54 PM Subject: Re: [computer-go] Reply to Lukasz and Don + Roadmap 2020 Mention the program so that the author can either refute your claim or fix the bug. terry mcintyre wrote: Is it reasonable to expect pro players to use 6-dan programs as a tool for analysis? The pro players are markedly better - at a rough guess, a pro player could give a 6 dan amateur human or program a 3 stone handicap. On the other end of the scale, beginning players and mid kyu players could indeed make good use of an analysis mode by a program which is better than themselves. Lastly, an analysis mode would be helpful to developers, methinks. After winning a game, I like to back up a few moves and find out when the program realized that it was behind. This often happens several moves after the fatal blow has already been struck. I know the feeling too well, when stronger players deftly skewer my group and I only discover the problem five moves later. What do they know that I don't? What do they know that the program doesn't? We have a saying, you learn the most from reviewing games which you have lost. An analysis mode can help developers to discover when their pride and joy first begins to miss the target. Lately, I have been playing quite a bit with a commercially available program. An almost-ladder which has an extra liberty will apparently be evaluated the same as a true ladder, and the program can be tricked into trying to capture my ladder-like position. This sort of predictable flaw might provide a clue to improve the next version. Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
2009/4/23 terry mcintyre terrymcint...@yahoo.com: Programs which get semeai and seki right every time might be a few stones stronger. They'd certainly be more valuable as teaching tools. In the game above, a stronger program would have exploited my earlier weakness; this would have encouraged me to make better moves. ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ Actually not. It seems so to a human who every now and then avoids loss by being better at these. But close semeais are rare. Sekis are rarer and program do not fail on theses every time. Go is a game where you can excel by making steady progress throughout the game without any brilliant moves. Also it is quite okay to compensate with other skill, I just played Mogo in KGS and got slaughtered after a careless cut. Well Killing and almost killing a group is where MC programs excel (relative to their strength) and those situations occur in almost every game.. I think semeai problem is easier to solve with: - Preanalysis by a classical go-algorithm. To my understanding this is what MFOG does - When we have even more CPU we can have even heavier playouts. Still an open issue whether smarter playout or more playouts is way to go. Although as I remember there were some mailing were it was mentioned cases where a smart playout could even hurt. -- Petri Pitkänen e-mail: petri.t.pitka...@gmail.com ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
I haven't got a ladder example at the moment, but here's an instance where Leela does not realize it is in terrible trouble. I ( with my 8 kyu AGA rating) know with certainty by move 223 (T5) that Black has captured a large white group. A stronger player could read this out sooner than I. This fight is too big to lose for either side; nothing else on the board matters. ( anyone? how early is this outcome pre-ordained? ) Based on the results of its analysis mode, Leela does not recognize the outcome of this semeai until the large white group in the bottom right is down to two liberties. The problem is even more stark in example2 -- similar board, black has foolishly played one of his own liberties for illustrative purposes. It is black's play, black has three liberties, white has three. Black must take away a liberty from white to win the capturing race, or make two eyes at T8. Black has only four playable moves; any other choice fails. Leela proposes - even after several minutes of analysis and a million nodes - that Black should tennuki at H14. That would snatch defeat from the jaws of certain victory; White would dive into T8 and win the race. I started this thread with the contention that analysis mode can help developers find problems, I hope this example explains why. My theory is that if a program could reliably recognize the outcome of such capturing races five or ten moves sooner, it could crush the likes of me. :D Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy From: Michael Williams michaelwilliam...@gmail.com To: computer-go computer-go@computer-go.org Sent: Tuesday, April 21, 2009 1:57:54 PM Subject: Re: [computer-go] Reply to Lukasz and Don + Roadmap 2020 Mention the program so that the author can either refute your claim or fix the bug. terry mcintyre wrote: Is it reasonable to expect pro players to use 6-dan programs as a tool for analysis? The pro players are markedly better - at a rough guess, a pro player could give a 6 dan amateur human or program a 3 stone handicap. On the other end of the scale, beginning players and mid kyu players could indeed make good use of an analysis mode by a program which is better than themselves. Lastly, an analysis mode would be helpful to developers, methinks. After winning a game, I like to back up a few moves and find out when the program realized that it was behind. This often happens several moves after the fatal blow has already been struck. I know the feeling too well, when stronger players deftly skewer my group and I only discover the problem five moves later. What do they know that I don't? What do they know that the program doesn't? We have a saying, you learn the most from reviewing games which you have lost. An analysis mode can help developers to discover when their pride and joy first begins to miss the target. Lately, I have been playing quite a bit with a commercially available program. An almost-ladder which has an extra liberty will apparently be evaluated the same as a true ladder, and the program can be tricked into trying to capture my ladder-like position. This sort of predictable flaw might provide a clue to improve the next version. Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ example.sgf Description: Binary data example2.sgf Description: Binary data ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
I ( with my 8 kyu AGA rating) know with certainty by move 223 (T5) that Black has captured a large white group. A stronger player could read this out sooner than I. This fight is too big to lose for either side; nothing else on the board matters. Yes! I use Many Faces and GoGui/Mogo for analyzing games [1]. The ability to be able to weight/suppress moves would be great. E.g. click a couple of white stones and say Stop trying to save these stones, they are dead! Or click all the stones in a group and say: Nothing else matters for either side except the live/death of this group. Or, click a bunch of points and say These are the current hotspots [2] Weighting moves should be relatively easy for programs to implement. The hardest part would be the GUI code. Agreeing on gtp commands for it would be a big first step. BTW, I appreciate that any move filter could lower strength of a program. The two dead white stones both Many Faces and Mogo keep trying to save is a classic example. What they are actually doing is making a dead group slightly stronger, strong enough so that nearby endgame moves switch from being gote to sente. But I'm trying to analyze the difference between two moves in the opposite corner and it is very frustrating when the cursed things keep playing tenuki! Darren [1]: I'm analyzing 9x9 games, and they are generally pro strength from the midgame onwards (bearing in mind their weaknesses of seki, and life/death situations where delicate play is only required by one player). [2]: Where a program could give hotspots 10 times more playouts compared to other moves, at root, with the 10:1 ratio deteriorating to 1:1 after N moves. (E.g. N could be 10, or some function of remaining empty points on board). -- Darren Cook, Software Researcher/Developer http://dcook.org/mlsn/ (English-Japanese-German-Chinese-Arabic open source dictionary/semantic network) http://dcook.org/work/ (About me and my work) http://dcook.org/blogs.html (My blogs and articles) ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] Roadmap 2020 - using analysis mode to improve programs
I've only looked at the first game, but it does seem very interesting to analyze. The white group around H2 is near death as well and I think Leela's evaluation considered that group to be threatened. Once that is solidly alive, it does switch to the semeai on the right. I probably would not have tenuki'd on move 206, but it appears safe when I read out how to respond after move 211. Moves 218 and 220 are reasonable tesuji's when played in a semeai (even if wrong for that semeai). I think you're right that there's probably plenty of analysis to do on how leela went wrong in the game. It may even be that weaker bots read out how to kill the S6 stones on the right and instead of focusing on a liberties race with a one eyed group. On Wed, 2009-04-22 at 18:27 -0700, terry mcintyre wrote: I haven't got a ladder example at the moment, but here's an instance where Leela does not realize it is in terrible trouble. I ( with my 8 kyu AGA rating) know with certainty by move 223 (T5) that Black has captured a large white group. A stronger player could read this out sooner than I. This fight is too big to lose for either side; nothing else on the board matters. ( anyone? how early is this outcome pre-ordained? ) Based on the results of its analysis mode, Leela does not recognize the outcome of this semeai until the large white group in the bottom right is down to two liberties. The problem is even more stark in example2 -- similar board, black has foolishly played one of his own liberties for illustrative purposes. It is black's play, black has three liberties, white has three. Black must take away a liberty from white to win the capturing race, or make two eyes at T8. Black has only four playable moves; any other choice fails. Leela proposes - even after several minutes of analysis and a million nodes - that Black should tennuki at H14. That would snatch defeat from the jaws of certain victory; White would dive into T8 and win the race. I started this thread with the contention that analysis mode can help developers find problems, I hope this example explains why. My theory is that if a program could reliably recognize the outcome of such capturing races five or ten moves sooner, it could crush the likes of me. :D Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy __ From: Michael Williams michaelwilliam...@gmail.com To: computer-go computer-go@computer-go.org Sent: Tuesday, April 21, 2009 1:57:54 PM Subject: Re: [computer-go] Reply to Lukasz and Don + Roadmap 2020 Mention the program so that the author can either refute your claim or fix the bug. terry mcintyre wrote: Is it reasonable to expect pro players to use 6-dan programs as a tool for analysis? The pro players are markedly better - at a rough guess, a pro player could give a 6 dan amateur human or program a 3 stone handicap. On the other end of the scale, beginning players and mid kyu players could indeed make good use of an analysis mode by a program which is better than themselves. Lastly, an analysis mode would be helpful to developers, methinks. After winning a game, I like to back up a few moves and find out when the program realized that it was behind. This often happens several moves after the fatal blow has already been struck. I know the feeling too well, when stronger players deftly skewer my group and I only discover the problem five moves later. What do they know that I don't? What do they know that the program doesn't? We have a saying, you learn the most from reviewing games which you have lost. An analysis mode can help developers to discover when their pride and joy first begins to miss the target. Lately, I have been playing quite a bit with a commercially available program. An almost-ladder which has an extra liberty will apparently be evaluated the same as a true ladder, and the program can be tricked into trying to capture my ladder-like position. This sort of predictable flaw might provide a clue to improve the next version. Terry McIntyre terrymcint...@yahoo.com Government is an association of men who do violence to the rest of us. - Leo Tolstoy ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___
