David, What Matt is trying to explain is all right, but I think a better way of answering your question would be to invoke the mighty mysterious Bayes' Law.
I had an epiphany similar to yours (the one that started this thread) about 5 years ago now. At the time I did not know that it had all been done before. I think many people feel this way about MDL. Looking into the MDL (minimum description length) literature would be a good starting point. In brief, the answer to your question is: we formalize the description length heuristic by assigning lower probabilities to longer hypotheses, and we apply Bayes law to update these probabilities given the data we observe. This updating captures the idea that we should reward theories which explain/expect more of the observations; it also provides a natural way to balance simplicity vs explanatory power, so that we can compare any two theories with a single scoring mechanism. Bayes Law automatically places the right amount of pressure to avoid overly elegant explanations which don't get much right, and to avoid overly complex explanations which fit the observations perfectly but which probably won't generalize to new data. Bayes' Law and MDL have strong connections, though sometimes they part ways. There are deep theorems here. For me it's good enough to note that if we're using a maximally efficient code for our knowledge representation, they are equivalent. (This in itself involves some deep math; I can explain if you're interested, though I believe I've already posted a writeup to this list in the past.) Bayesian updating is essentially equivalent to scoring hypotheses as: hypothesis size + size of data's description using hypothesis. Lower scores are better (as the score is approximately -log(probability)). If you go down this path, you will eventually come to understand (and, probably, accept) algorithmic information theory. Matt may be tring to force it on you too soon. :) --Abram On Tue, Jun 29, 2010 at 10:44 AM, David Jones <davidher...@gmail.com> wrote: > Thanks Matt, > > Right. But Occam's Razor is not complete. It says simpler is better, but 1) > this only applies when two hypotheses have the same explanatory power and 2) > what defines simpler? > > So, maybe what I want to know from the state of the art in research is: > > 1) how precisely do other people define "simpler" > and > 2) More importantly, how do you compare competing explanations/hypotheses > that have more or less explanatory power. Simpler does not apply unless you > are comparing equally explanatory hypotheses. > > For example, the simplest hypothesis for all visual interpretation is that > everything in the first image is gone in the second image, and everything in > the second image is a new object. Simple. Done. Solved :) right? Well, > clearly a more complicated explanation is warranted because a more > complicated explanation is more *explanatory* and a better explanation. So, > why is it better? Can it be defined as better in a precise way so that you > can compare arbitrary hypotheses or explanations? That is what I'm trying to > learn about. I don't think much progress has been made in this area, but I'd > like to know what other people have done and any successes they've had. > > Dave > > > On Tue, Jun 29, 2010 at 10:29 AM, Matt Mahoney <matmaho...@yahoo.com>wrote: > >> David Jones wrote: >> > If anyone has any knowledge of or references to the state of the art in >> explanation-based reasoning, can you send me keywords or links? >> >> The simplest explanation of the past is the best predictor of the future. >> http://en.wikipedia.org/wiki/Occam's_razor<http://en.wikipedia.org/wiki/Occam%27s_razor> >> <http://en.wikipedia.org/wiki/Occam%27s_razor> >> http://www.scholarpedia.org/article/Algorithmic_probability >> <http://www.scholarpedia.org/article/Algorithmic_probability> >> >> -- Matt Mahoney, matmaho...@yahoo.com >> >> >> ------------------------------ >> *From:* David Jones <davidher...@gmail.com> >> >> *To:* agi <agi@v2.listbox.com> >> *Sent:* Tue, June 29, 2010 9:05:45 AM >> *Subject:* [agi] Re: Huge Progress on the Core of AGI >> >> If anyone has any knowledge of or references to the state of the art in >> explanation-based reasoning, can you send me keywords or links? I've read >> some through google, but I'm not really satisfied with anything I've found. >> >> Thanks, >> >> Dave >> >> On Sun, Jun 27, 2010 at 1:31 AM, David Jones <davidher...@gmail.com>wrote: >> >>> A method for comparing hypotheses in explanatory-based reasoning: * >>> >>> We prefer the hypothesis or explanation that ***expects* more >>> observations. If both explanations expect the same observations, then the >>> simpler of the two is preferred (because the unnecessary terms of the more >>> complicated explanation do not add to the predictive power).* >>> >>> *Why are expected events so important?* They are a measure of 1) >>> explanatory power and 2) predictive power. The more predictive and the more >>> explanatory a hypothesis is, the more likely the hypothesis is when compared >>> to a competing hypothesis. >>> >>> Here are two case studies I've been analyzing from sensory perception of >>> simplified visual input: >>> The goal of the case studies is to answer the following: How do you >>> generate the most likely motion hypothesis in a way that is general and >>> applicable to AGI? >>> *Case Study 1)* Here is a link to an example: animated gif of two black >>> squares move from left to >>> right<http://practicalai.org/images/CaseStudy1.gif>. >>> *Description: *Two black squares are moving in unison from left to right >>> across a white screen. In each frame the black squares shift to the right so >>> that square 1 steals square 2's original position and square two moves an >>> equal distance to the right. >>> *Case Study 2) *Here is a link to an example: the interrupted >>> square<http://practicalai.org/images/CaseStudy2.gif>. >>> *Description:* A single square is moving from left to right. Suddenly in >>> the third frame, a single black square is added in the middle of the >>> expected path of the original black square. This second square just stays >>> there. So, what happened? Did the square moving from left to right keep >>> moving? Or did it stop and then another square suddenly appeared and moved >>> from left to right? >>> >>> *Here is a simplified version of how we solve case study 1: >>> *The important hypotheses to consider are: >>> 1) the square from frame 1 of the video that has a very close position to >>> the square from frame 2 should be matched (we hypothesize that they are the >>> same square and that any difference in position is motion). So, what >>> happens is that in each two frames of the video, we only match one square. >>> The other square goes unmatched. >>> 2) We do the same thing as in hypothesis #1, but this time we also match >>> the remaining squares and hypothesize motion as follows: the first square >>> jumps over the second square from left to right. We hypothesize that this >>> happens over and over in each frame of the video. Square 2 stops and square >>> 1 jumps over it.... over and over again. >>> 3) We hypothesize that both squares move to the right in unison. This is >>> the correct hypothesis. >>> >>> So, why should we prefer the correct hypothesis, #3 over the other two? >>> >>> Well, first of all, #3 is correct because it has the most explanatory >>> power of the three and is the simplest of the three. Simpler is better >>> because, with the given evidence and information, there is no reason to >>> desire a more complicated hypothesis such as #2. >>> >>> So, the answer to the question is because explanation #3 expects the most >>> observations, such as: >>> 1) the consistent relative positions of the squares in each frame are >>> expected. >>> 2) It also expects their new positions in each from based on velocity >>> calculations. >>> 3) It expects both squares to occur in each frame. >>> >>> Explanation 1 ignores 1 square from each frame of the video, because it >>> can't match it. Hypothesis #1 doesn't have a reason for why the a new square >>> appears in each frame and why one disappears. It doesn't expect these >>> observations. In fact, explanation 1 doesn't expect anything that happens >>> because something new happens in each frame, which doesn't give it a chance >>> to confirm its hypotheses in subsequent frames. >>> >>> The power of this method is immediately clear. It is general and it >>> solves the problem very cleanly. >>> >>> *Here is a simplified version of how we solve case study 2:* >>> We expect the original square to move at a similar velocity from left to >>> right because we hypothesized that it did move from left to right and we >>> calculated its velocity. If this expectation is confirmed, then it is more >>> likely than saying that the square suddenly stopped and another started >>> moving. Such a change would be unexpected and such a conclusion would be >>> unjustifiable. >>> >>> I also believe that explanations which generate fewer incorrect >>> expectations should be preferred over those that more incorrect >>> expectations. >>> >>> The idea I came up with earlier this month regarding high frame rates to >>> reduce uncertainty is still applicable. It is important that all generated >>> hypotheses have as low uncertainty as possible given our constraints and >>> resources available. >>> >>> I thought I'd share my progress with you all. I'll be testing the ideas >>> on test cases such as the ones I mentioned in the coming days and weeks. >>> >>> Dave >>> >> >> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >> <https://www.listbox.com/member/archive/rss/303/> | >> Modify<https://www.listbox.com/member/?&;>Your Subscription >> <http://www.listbox.com> >> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >> <https://www.listbox.com/member/archive/rss/303/> | >> Modify<https://www.listbox.com/member/?&;>Your Subscription >> <http://www.listbox.com> >> > > *agi* | Archives <https://www.listbox.com/member/archive/303/=now> > <https://www.listbox.com/member/archive/rss/303/> | > Modify<https://www.listbox.com/member/?&;>Your Subscription > <http://www.listbox.com> > -- Abram Demski http://lo-tho.blogspot.com/ http://groups.google.com/group/one-logic ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=8660244-6e7fb59c Powered by Listbox: http://www.listbox.com
