Matt:It is like the way evolution works, except that there is a human in the
loop to make the process a little more intelligent.
IOW this is like AGI, except that it's narrow AI. That's the whole point - you
have to remove the human from the loop. In fact, it also sounds like a
misconceived and rather literal idea of evolution as opposed to the reality.
From: Matt Mahoney
Sent: Monday, June 21, 2010 3:01 AM
To: agi
Subject: Re: [agi] An alternative plan to discover self-organization theory
Steve Richfield wrote:
> He suggested that I construct a "simple" NN that couldn't work without self
> organizing, and make dozens/hundreds of different neuron and synapse
> operational characteristics selectable ala genetic programming, put it on the
> fastest computer I could get my hands on, turn it loose trying arbitrary
> combinations of characteristics, and see what the "winning" combination turns
> out to be. Then, armed with that knowledge, refine the genetic
> characteristics and do it again, and iterate until it efficiently self
> organizes. This might go on for months, but self-organization theory might
> just emerge from such an effort.
Well, that is the process that created human intelligence, no? But months? It
actually took 3 billion years on a planet sized molecular computer.
That doesn't mean it won't work. It just means you have to narrow your search
space and lower your goals.
I can give you an example of a similar process. Look at the code for
PAQ8HP12ANY and LPAQ9M data compressors by Alexander Ratushnyak, which are the
basis of winning Hutter prize submissions. The basic principle is that you have
a model that receives a stream of bits from an unknown source and it uses a
complex hierarchy of models to predict the next bit. It is sort of like a
neural network because it averages together the results of lots of adaptive
pattern recognizers by processes that are themselves adaptive. But I would
describe the code as inscrutable, kind of like your DNA. There are lots of
parameters to tweak, such as how to preprocess the data, arrange the
dictionary, compute various contexts, arrange the order of prediction flows,
adjust various learning rates and storage capacities, and make various
tradeoffs sacrificing compression to meet memory and speed requirements. It is
simple to describe the process of writing the code. You make random changes and
keep the ones that work. It is like the way evolution works, except that there
is a human in the loop to make the process a little more intelligent.
There are also fully automated optimizers for compression algorithms, but they
are more limited in their search space. For example, the experimental PPM based
EPM by Serge Osnach includes a program EPMOPT that adjusts 20 numeric
parameters up or down using a hill climbing search to find the best
compression. It can be very slow. Another program, M1X2 by Christopher Mattern,
uses a context mixing (PAQ like) algorithm in which the contexts are selected
by using a hill climbing genetic algorithm to select a set of 64-bit masks. One
version was run for 3 days to find the best options to compress a file that
normally takes 45 seconds.
-- Matt Mahoney, matmaho...@yahoo.com
--------------------------------------------------------------------------------
From: Steve Richfield <steve.richfi...@gmail.com>
To: agi <agi@v2.listbox.com>
Sent: Sun, June 20, 2010 2:06:55 AM
Subject: [agi] An alternative plan to discover self-organization theory
No, I haven't been smokin' any wacky tobacy. Instead, I was having a long talk
with my son Eddie, about self-organization theory. This is his proposal:
He suggested that I construct a "simple" NN that couldn't work without self
organizing, and make dozens/hundreds of different neuron and synapse
operational characteristics selectable ala genetic programming, put it on the
fastest computer I could get my hands on, turn it loose trying arbitrary
combinations of characteristics, and see what the "winning" combination turns
out to be. Then, armed with that knowledge, refine the genetic characteristics
and do it again, and iterate until it efficiently self organizes. This might go
on for months, but self-organization theory might just emerge from such an
effort. I had a bunch of objections to his approach, e.g.
Q. What if it needs something REALLY strange to work?
A. Who better than you to come up with a long list of really strange
functionality?
Q. There are at least hundreds of bits in the "genome".
A. Try combinations in pseudo-random order, with each bit getting asserted in
~half of the tests. If/when you stumble onto a combination that sort of works,
switch to varying the bits one-at-a-time, and iterate in this way until the
best combination is found.
Q. Where are we if this just burns electricity for a few months and finds
nothing?
A. Print out the best combination, break out the wacky tobacy, and come up
with even better/crazier parameters to test.
I have never written a line of genetic programming, but I know that others here
have. Perhaps you could bring some rationality to this discussion?
What would be a "simple" NN that needs self-organization? Maybe a small "pot"
of neurons that could only work if they were organized into layers, e.g. a
simple 64-neuron system that would work as a 4x4x4-layer visual recognition
system, given the input that I fed it?
Any thoughts on how to "score" partial successes?
Has anyone tried anything like this in the past?
Is anyone here crazy enough to want to help with such an effort?
This Monte Carlo approach might just be simple enough to work, and simple
enough that it just HAS to be tried.
All thoughts, stones, and rotten fruit will be gratefully appreciated.
Thanks in advance.
Steve
agi | Archives | Modify Your Subscription
agi | Archives | Modify Your Subscription
-------------------------------------------
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c
Powered by Listbox: http://www.listbox.com
