Re: [computer-go] speeding up testing of computer go programs
Hi Jim, There is no black and white when it comes to right or wrong thinking. Each of us is guilty of it. You get an idea, try it out, test it thoroughly and find that it isn't useful. There is nothing wrong with this process itself - I view it as a device that trains your intuition. The more paths you go down, good and bad, the better your judgment the next time around. However, I have seen people get stuck on the wrong pathway for far too long. They get some idea that they are emotionally attached to, wrap their teeth around it, and won't let go. You see this is science all the time - scientists waiting too long to abandon their pet theory because of the emotional investment they have in it. This is a big fat gray area, so there is no standard method to clearly distinguish right thinking except to do the hard work. However, some common sense is required. Wrong thinking starts to come into play when you stubbornly refuse to acknowledge defeat in some pathway you are pursuing.Or when you fail to discern that you are being lazy in some area. If you are doing more coding than anything else you are probably guilty of wrong thinking (or perhaps not thinking is a better way to put it.) Probably every one of us has bad habits that hold us back to a certain extent. We also have prejudices that don't serve us well. I know that my testing methods are frowned on by some people and I frown on theirs.I know some people are convinced that self-testing is evil, some including myself are not. Writing code in a slow high level language because you can write it slightly faster is one of those things that seem like a good idea because you can try things more quickly but in fact it slows down your development time (as measured by the whole development cycle, not just the keying in the program part) and in addition you get a slower and weaker finished product. You pay for that decision forever and ever.Is that really a good trade off? This may not apply to other kinds of development where performance isn't the prime issue. Often such a case exists in corporate environments.I wrote CGOS in a scripting language because it was easier and performance is a minor issue. Sometimes you hear that someone stumbled on a great idea and that made them rich and/or famous. Some people have a knack for stumbling on the right ideas - I don't think that is an accident.They are just more methodical about eliminated bad thinking. So something that is wrong thinking for you may be a learning experience for me.It's still wrong thinking technically, but it's useful as a learning experience for a short period of time - then it becomes an Albatross. - Don Jim O'Flaherty, Jr. wrote: Don, I think we have a semantic problem. Some things don't work as expected but provide the genesis for further creativity. Other things work, but not with sufficient additional value for the disproportionate effort invested. Some things don't end up having any enduring value except as one of the many infinite possible paths eliminated. Maybe we are being too abstract here. If you have the time and motivation, would you please tell me in a couple of sentences what your standard(s) is upon which you can clearly distinguish right thinking from wrong thinking? And how that standard accommodates active random walk experimentation and creativity such that an experimenter can know PRIOR to doing the experiment if he is headed off in a right or wrong path? Jim Don Dailey wrote: Jim O'Flaherty, Jr. wrote: Heikki, I'm with you. There is no wrong thinking at the present time. Of course there is wrong thinking. Why do you think they call it the trial and error approach? - Don There are too many differing agendas, with building the strongest program immediately being only one, to claim any approach is futile, inefficient or erred. Once there are approaches that actually come near playing low dan levels against humans, I can see how narrowing approaches and thinking will become useful. Until then, lots of chaotic and random path experimentation is desirable, including other languages, specialized languages, techniques, etc. Jim Heikki Levanto wrote: On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new
Re: [computer-go] speeding up testing of computer go programs
However, perhaps there are ways to make testing a Go program use less clock time? This is the right idea. Chess programmers use massive automated testing - playing games. To measure a small ELO improvement in your program requires tens of thousands of games.I think it's something like 100,000 to measure 1 or 2 ELO with high confidence. It might make sense to test your scalable go program at levels where a game can be completed within 2 or 3 seconds. In this way, if a changes hurts your program significantly you can measure this within minutes. If it's a 5-10 ELO improvement you can measure it in a few hours if you have a few computers. At the level my program is at, I would be happy just to measure 20 ELO improvements so on the 3 CPU's I have I could do this relatively quickly if I speed up the tests to a couple of seconds per game. One must seek a compromise. Ideally, you would prefer to test at levels similar to what you will actually be playing. But this is not practical if you want high confidence in small improvements. I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. - Don Brian Slesinsky wrote: Thought I'd emphasize this point: Don Dailey: I spend a great deal of time waiting on the computer, because I have no clue what will work and I must test it. This makes Go programming somewhat unusual; for a lot of programs, you can arrange so that compiling and running your tests only takes a few seconds. But if waiting for test results takes hours or days, the highest-performance language is also the the most productive one to develop in, because you can do more iterations in any given amount of time. However, perhaps there are ways to make testing a Go program use less clock time? Could testing on smaller problems give results similar to testing on larger ones? Or suppose tests could be run in parallel on something like Amazon EC2? - Brian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] speeding up testing of computer go programs
On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new insight or theory on how best to approach the problem. A mere amateur like me could complain about the time invested in those small improvements, that I did not gain any new knowlege for myself, it was just routine programming. I humbly suggest that none of us (including you :-) is guilty of wrong thinking. Regards Heikki -- Heikki Levanto In Murphy We Turst heikki (at) lsd (dot) dk ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] speeding up testing of computer go programs
Heikki, I'm with you. There is no wrong thinking at the present time. There are too many differing agendas, with building the strongest program immediately being only one, to claim any approach is futile, inefficient or erred. Once there are approaches that actually come near playing low dan levels against humans, I can see how narrowing approaches and thinking will become useful. Until then, lots of chaotic and random path experimentation is desirable, including other languages, specialized languages, techniques, etc. Jim Heikki Levanto wrote: On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new insight or theory on how best to approach the problem. A mere amateur like me could complain about the time invested in those small improvements, that I did not gain any new knowlege for myself, it was just routine programming. I humbly suggest that none of us (including you :-) is guilty of wrong thinking. Regards Heikki ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] speeding up testing of computer go programs
Heikki Levanto wrote: On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. I'm not addressing the people that do this just for fun and don't care how strong their program is.I'm addressing (see my quote above) the people that are looking for breakthrough's. These people are doing this to advance the state of the art - they want to see strong programs and a few are trying to do it with hand tools when they should be using power tools. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new insight or theory on how best to approach the problem. I personally believe this is a fallacy. Having an efficient language doesn't inhibit breakthrough's.But high level language enthusiasts want you to believe that. At any rate, Java certainly isn' t much of a choice if a great deal of expressiveness is your criteria. You should probably be reaching for a high level language if that is you primary consideration.Java is a low level language in comparison to quite a few really cool expressive languages now available. A mere amateur like me could complain about the time invested in those small improvements, that I did not gain any new knowlege for myself, it was just routine programming. I humbly suggest that none of us (including you :-) is guilty of wrong thinking. I haven't found the language choice to be very much of an issue even when it comes to coding time.The biggest investment is the initial learning curve of learning a new language and once that goes away the amazing extra productivity over all other languages that every language claims is just hype. So you might as well use one that makes you productive in other ways. You can try new ideas much more quickly in C. - Don Regards Heikki ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] speeding up testing of computer go programs
Jim O'Flaherty, Jr. wrote: Heikki, I'm with you. There is no wrong thinking at the present time. Of course there is wrong thinking. Why do you think they call it the trial and error approach? - Don There are too many differing agendas, with building the strongest program immediately being only one, to claim any approach is futile, inefficient or erred. Once there are approaches that actually come near playing low dan levels against humans, I can see how narrowing approaches and thinking will become useful. Until then, lots of chaotic and random path experimentation is desirable, including other languages, specialized languages, techniques, etc. Jim Heikki Levanto wrote: On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new insight or theory on how best to approach the problem. A mere amateur like me could complain about the time invested in those small improvements, that I did not gain any new knowlege for myself, it was just routine programming. I humbly suggest that none of us (including you :-) is guilty of wrong thinking. Regards Heikki ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] speeding up testing of computer go programs
Don, I think we have a semantic problem. Some things don't work as expected but provide the genesis for further creativity. Other things work, but not with sufficient additional value for the disproportionate effort invested. Some things don't end up having any enduring value except as one of the many infinite possible paths eliminated. Maybe we are being too abstract here. If you have the time and motivation, would you please tell me in a couple of sentences what your standard(s) is upon which you can clearly distinguish right thinking from wrong thinking? And how that standard accommodates active random walk experimentation and creativity such that an experimenter can know PRIOR to doing the experiment if he is headed off in a right or wrong path? Jim Don Dailey wrote: Jim O'Flaherty, Jr. wrote: Heikki, I'm with you. There is no wrong thinking at the present time. Of course there is wrong thinking. Why do you think they call it the trial and error approach? - Don There are too many differing agendas, with building the strongest program immediately being only one, to claim any approach is futile, inefficient or erred. Once there are approaches that actually come near playing low dan levels against humans, I can see how narrowing approaches and thinking will become useful. Until then, lots of chaotic and random path experimentation is desirable, including other languages, specialized languages, techniques, etc. Jim Heikki Levanto wrote: On Sun, Nov 25, 2007 at 11:52:14AM -0500, Don Dailey wrote: I know that most go programmers don't concern themselves with small improvements because of the sense that there is bigger fish to fry. But this is wrong thinking. If you can get 10 small improvements it can be equivalent to one very large improvement. This is frong thinking *for you*. Wasn't it yourself who said that people program go for various reasons, only one of them being making the strongest possible program. A person with a more theoretical approach might lament that all that speed optimizing indeed improved the play considerably, but has not produced any new insight or theory on how best to approach the problem. A mere amateur like me could complain about the time invested in those small improvements, that I did not gain any new knowlege for myself, it was just routine programming. I humbly suggest that none of us (including you :-) is guilty of wrong thinking. Regards Heikki ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] speeding up testing of computer go programs
Thought I'd emphasize this point: Don Dailey: I spend a great deal of time waiting on the computer, because I have no clue what will work and I must test it. This makes Go programming somewhat unusual; for a lot of programs, you can arrange so that compiling and running your tests only takes a few seconds. But if waiting for test results takes hours or days, the highest-performance language is also the the most productive one to develop in, because you can do more iterations in any given amount of time. However, perhaps there are ways to make testing a Go program use less clock time? Could testing on smaller problems give results similar to testing on larger ones? Or suppose tests could be run in parallel on something like Amazon EC2? - Brian ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
