--- On Mon, 9/22/08, Ben Goertzel <[EMAIL PROTECTED]> wrote: > http://goertzel.org/agiq.pdf
Some of the problems you describe in intelligence testing also apply to data compression testing. For example, an AI could "cheat" by being tuned with the knowledge needed to pass a specific test. This is a common problem with compression programs, which are often tuned to do well on popular benchmarks. I once wrote as a joke a program that compresses the Calgary corpus files to 1 byte each. http://cs.fit.edu/~mmahoney/compression/barf.html Usually cheating is not so overt as hiding a copy of the benchmark (i.e. the test answers) in the program, but developers do test their programs on popular benchmarks. But it is hard to avoid unintentional bias in the form of shared information, such as external English dictionaries used to tokenize the input text. In earlier tests, I would distinguish such programs, but there are other ways around this. A common solution is to make the test data private, which many benchmarks do. However, since I run a benchmark and also write compression software, it would be unfair not to give competing software the same advantage I would have. You mentioned the "Nintendog" effect, where an AI could pretend to learn what it already knows. Your solution is to start with a base test, give different copies different training data, and test them on the newly learned data. In effect, we do this when we test a compressor with many different text files. It is much harder to hide knowledge about all of the input files in the model than about just one file. In compression testing, there is another option, which is to include the size of the decompression program. Any pre-programmed knowledge has to take up space which can't be compressed beyond its algorithmic complexity. This is the technique used in two benchmarks with prize money: Calgary challenge and the Hutter prize, and also in my large text benchmark. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=114414975-3c8e69 Powered by Listbox: http://www.listbox.com
