On Fri, Dec 28, 2012 at 2:35 PM, Ben Goertzel <[email protected]> wrote: > I'm not opposed to testing systems and their components, of course. It's > important.... I'm just skeptical of the quantitative-metric-focused, > "bake-off" mentality one sees e.g. in the machine learning community...
If you create your test cases before the software is complete, then there is a bias to design the software to pass the tests. If you create your test cases after the software is complete, then there is a bias to design the tests so that they can be passed. How do you plan to avoid the second kind of bias? I recall a while back there was some discussion of testing MOSES on the 4 bit parity problem. Sometimes it would run for 30 minutes and fail to solve it. I pointed out that the problem is trivially solved by several data compression programs (predicting the output given the input as context in the previous byte) in milliseconds. The discussion led to improvements in breaking out of local minima to solve the problem. There are probably problems that MOSES could solve that would defeat even the best compressors, and probably some other machine learning algorithms. But what? -- -- Matt Mahoney, [email protected] ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
