Re: [Computer-go] CGOS source on github
also frankly not a problem for a rating system to handle. a rating system shouldn't be tweaked to handle eccentricities of its players other than the general assumptions of how a game's result is determined (like, does it allow for "win" and "draw" and "undetermined" or just "win"). s. On Fri, Jan 22, 2021 at 6:29 AM David Wu wrote: > On Fri, Jan 22, 2021 at 8:08 AM Rémi Coulom wrote: > >> You are right that non-determinism and bot blind spots are a source of >> problems with Elo ratings. I add randomness to the openings, but it is >> still difficult to avoid repeating some patterns. I have just noticed that >> the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by >> very similar ladders in the opening: >> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf >> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf >> Such a huge blind spot in such a strong engine is likely to cause rating >> compression. >> Rémi >> > > I agree, ladders are definitely the other most noticeable way that Elo > model assumptions may be broken, since pure-zero bots have a hard time with > them, and can easily cause difference(A,B) + difference(B,C) to be very > inconsistent with difference(A,C). If some of A,B,C always handle ladders > very well and some are blind to them, then you are right that probably no > amount of opening randomization can smooth it out. > > ___ > Computer-go mailing list > Computer-go@computer-go.org > http://computer-go.org/mailman/listinfo/computer-go > ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
@Claude - Oh, sorry, I misread your message, you were also asking about ladders, not just liberties. In that case, yes! If you outright tell the neural net as an input whether each ladder works or not (doing a short tactical search to determine this), or something equivalent to it, then the net will definitely make use of that information, There are some bad side effects even to doing this, but it helps the most common case. This is something the first version of AlphaGo did (before they tried to make it "zero") and something that many other bots do as well. But Leela Zero and ELF do not do this, because of attempting to remain "zero", i.e. free as much as possible from expert human knowledge or specialized feature crafting. On Fri, Jan 22, 2021 at 9:26 AM David Wu wrote: > Hi Claude - no, generally feeding liberty counts to neural networks > doesn't help as much as one would hope with ladders and sekis and large > capturing races. > > The thing that is hard about ladders has nothing to do with liberties - a > trained net is perfectly capable of recognizing the atari, this is > extremely easy. The hard part is predicting if the ladder will work without > playing it out, because whether it works depends extremely sensitively on > the exact position of stones all the way on the other side of the board. A > net that fails to predict this well might prematurely reject a working > ladder (which is very hard for the search to correct), or be highly > overoptimistic about a nonworking ladder (which takes the search thousands > of playouts to correct in every single branch of the tree that it happens > in). > > For large sekis and capturing races, liberties usually don't help as much > as you would think. This is because approach liberties, ko liberties, big > eye liberties, shared liberties versus unshared liberties, throwin > possibilities all affect the "effective" liberty count significantly. Also > very commonly you have bamboo joints, simple diagonal or hanging > connections and other shapes where the whole group is not physically > connected, also making the raw liberty count not so useful. The neural net > still ultimately has to scan over the entire group anyways, computing these > things. > > On Fri, Jan 22, 2021 at 8:31 AM Claude Brisson via Computer-go < > computer-go@computer-go.org> wrote: > >> Hi. Maybe it's a newbie question, but since the ladders are part of the >> well defined topology of the goban (as well as the number of current >> liberties of each chain of stone), can't feeding those values to the >> networks (from the very start of the self teaching course) help with large >> shichos and sekis? >> >> Regards, >> >> Claude >> On 21-01-22 13 h 59, Rémi Coulom wrote: >> >> Hi David, >> >> You are right that non-determinism and bot blind spots are a source of >> problems with Elo ratings. I add randomness to the openings, but it is >> still difficult to avoid repeating some patterns. I have just noticed that >> the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by >> very similar ladders in the opening: >> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf >> http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf >> Such a huge blind spot in such a strong engine is likely to cause rating >> compression. >> >> Rémi >> >> ___ >> Computer-go mailing >> listComputer-go@computer-go.orghttp://computer-go.org/mailman/listinfo/computer-go >> >> ___ >> Computer-go mailing list >> Computer-go@computer-go.org >> http://computer-go.org/mailman/listinfo/computer-go >> > ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
Hi Claude - no, generally feeding liberty counts to neural networks doesn't help as much as one would hope with ladders and sekis and large capturing races. The thing that is hard about ladders has nothing to do with liberties - a trained net is perfectly capable of recognizing the atari, this is extremely easy. The hard part is predicting if the ladder will work without playing it out, because whether it works depends extremely sensitively on the exact position of stones all the way on the other side of the board. A net that fails to predict this well might prematurely reject a working ladder (which is very hard for the search to correct), or be highly overoptimistic about a nonworking ladder (which takes the search thousands of playouts to correct in every single branch of the tree that it happens in). For large sekis and capturing races, liberties usually don't help as much as you would think. This is because approach liberties, ko liberties, big eye liberties, shared liberties versus unshared liberties, throwin possibilities all affect the "effective" liberty count significantly. Also very commonly you have bamboo joints, simple diagonal or hanging connections and other shapes where the whole group is not physically connected, also making the raw liberty count not so useful. The neural net still ultimately has to scan over the entire group anyways, computing these things. On Fri, Jan 22, 2021 at 8:31 AM Claude Brisson via Computer-go < computer-go@computer-go.org> wrote: > Hi. Maybe it's a newbie question, but since the ladders are part of the > well defined topology of the goban (as well as the number of current > liberties of each chain of stone), can't feeding those values to the > networks (from the very start of the self teaching course) help with large > shichos and sekis? > > Regards, > > Claude > On 21-01-22 13 h 59, Rémi Coulom wrote: > > Hi David, > > You are right that non-determinism and bot blind spots are a source of > problems with Elo ratings. I add randomness to the openings, but it is > still difficult to avoid repeating some patterns. I have just noticed that > the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by > very similar ladders in the opening: > http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf > http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf > Such a huge blind spot in such a strong engine is likely to cause rating > compression. > > Rémi > > ___ > Computer-go mailing > listComputer-go@computer-go.orghttp://computer-go.org/mailman/listinfo/computer-go > > ___ > Computer-go mailing list > Computer-go@computer-go.org > http://computer-go.org/mailman/listinfo/computer-go > ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
On Fri, Jan 22, 2021 at 3:45 AM Hiroshi Yamashita wrote: > This kind of joseki is not good for Zero type. Ladder and capturing > race are intricately combined. In AlphaGo(both version of AlphaGoZero > and Master) published self-matches, this joseki is rare. > - > > I found this joseki in kata1_b40s575v100 (black) vs LZ_286_e6e2_p400 > (white). > http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/22/733340.sgf > Hi Hiroshi - yep. This is indeed a joseki that was partly popularized by AI and jointly explored with humans. It is probably fair to say that it is by far the most complicated common joseki known right now, and more complicated than either of the avalanche or the taisha. Some zero-trained bots will find and enter into this joseki, some won't. The ones that don't play this joseki in self-play will have a significant chance to be vulnerable to it if an opponent plays it against them, because there are a large number of traps and blind spots that cannot be solved if the net doesn't have experience with the position. And even having some experience is not always enough. For example, ELF and Leela Zero have learned some lines, but are far from perfect. There is a good chance that AlphaGoZero or Master would have been vulnerable to it as well. KataGo at the time of 1.3.5 was also vulnerable to it too - it only rarely came up in self-play, and therefore was never learned and correctly evaluated, so from the 3-3 invader's side the joseki could be forced and KataGo would likely mess up the joseki and be losing the game right at the start. (The most recent KataGo nets are much less vulnerable now though). The example you found is one where this has happened to Leela Zero. In the game you linked, move 34 is a big mistake. Leela Zero underweights the possibility of move 35, and then is blind to the seeming-bad-shape move of 37, and as a result, is in a bad position now. The current Leela Zero nets consistently makes this mistake, *and* consistently prefer playing down this line, so against an opponent happy to play it with them, Leela Zero will lose many games right in the opening all the same way. Anyways, the reason this joseki is responsible for more such distortions than other joseki seems to be because it is so sharp, and unlike most other common joseki, contains at least 5-6 enormous blind spots in different variations that zero-trained nets variously have trouble to learn on their own. > a very large sampling of positions from a wide range > > of human professional games, from say, move 20, and have bots play > starting > > from these sampled positions, in pairs once with each color. > > This sounds interesting. > I will think about another CGOS that handle this. I'm glad you're interested. I don't know if move 20 is a good number (I just threw it out there), maybe it should be varied, it might take some experimentation. And I'm not sure it's worth doing, since it's still probably only the smaller part of the problem in general - as Remi pointed out, likely ladder handling will be a thing that always continues to introduce Elo-nontransitivity, and probably all of this is less important than generally having a variety of long-running bots to help stabilize the system over time. ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
On Fri, Jan 22, 2021 at 8:08 AM Rémi Coulom wrote: > You are right that non-determinism and bot blind spots are a source of > problems with Elo ratings. I add randomness to the openings, but it is > still difficult to avoid repeating some patterns. I have just noticed that > the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by > very similar ladders in the opening: > http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf > http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf > Such a huge blind spot in such a strong engine is likely to cause rating > compression. > Rémi > I agree, ladders are definitely the other most noticeable way that Elo model assumptions may be broken, since pure-zero bots have a hard time with them, and can easily cause difference(A,B) + difference(B,C) to be very inconsistent with difference(A,C). If some of A,B,C always handle ladders very well and some are blind to them, then you are right that probably no amount of opening randomization can smooth it out. ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
Hi. Maybe it's a newbie question, but since the ladders are part of the well defined topology of the goban (as well as the number of current liberties of each chain of stone), can't feeding those values to the networks (from the very start of the self teaching course) help with large shichos and sekis? Regards, Claude On 21-01-22 13 h 59, Rémi Coulom wrote: Hi David, You are right that non-determinism and bot blind spots are a source of problems with Elo ratings. I add randomness to the openings, but it is still difficult to avoid repeating some patterns. I have just noticed that the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by very similar ladders in the opening: http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf Such a huge blind spot in such a strong engine is likely to cause rating compression. Rémi ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
Hi David, You are right that non-determinism and bot blind spots are a source of problems with Elo ratings. I add randomness to the openings, but it is still difficult to avoid repeating some patterns. I have just noticed that the two wins of CrazyStone-81-15po against LZ_286_e6e2_p400 were caused by very similar ladders in the opening: http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/73.sgf http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/21/733301.sgf Such a huge blind spot in such a strong engine is likely to cause rating compression. Rémi ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
Re: [Computer-go] CGOS source on github
Hi, The most noticeable case of this is with Mi Yuting's flying dagger joseki. I'm not familiar with this. I found Hirofumi Ohashi 6d pro's explanation half year ago in HCCL ML. The following is a quote. - https://gokifu.net/t2.php?s=3591591539793593 It seems that it is called a flying dagger joseki in China. This shape, direct 33 to lower tsuke (black 9th move B6) is researched jointly with humans and AI, but still inconclusive. After kiri (black 15th move E4), mainstream is white A, but depending on the version of KataGo, white B may be recommended. By the way, KataGo I'm using now is 1.3.5, which is just a short time ago. This kind of joseki is not good for Zero type. Ladder and capturing race are intricately combined. In AlphaGo(both version of AlphaGoZero and Master) published self-matches, this joseki is rare. - I found this joseki in kata1_b40s575v100 (black) vs LZ_286_e6e2_p400 (white). http://www.yss-aya.com/cgos/viewer.cgi?19x19/SGF/2021/01/22/733340.sgf Mi Yuting wiki has this joseki. https://zh.wikipedia.org/wiki/%E8%8A%88%E6%98%B1%E5%BB%B7 KataGo has special option. https://github.com/lightvector/KataGo/blob/4a79cde56e81209ce4e2fd231b0f2cbee3a8354b/cpp/neuralnet/nneval.cpp#L499 a very large sampling of positions from a wide range of human professional games, from say, move 20, and have bots play starting from these sampled positions, in pairs once with each color. This sounds interesting. I will think about another CGOS that handle this. Thanks, Hiroshi Yamashita ___ Computer-go mailing list Computer-go@computer-go.org http://computer-go.org/mailman/listinfo/computer-go
