Ben, For prediction of real-valued time series, one uses a variety of different > accuracy metrics, often custom ones based on what one is trying to use the > predictions for...
Ah, yes, that is a good point... prediction<->compression makes a very specific claim about the accuracy metric. While I feel this is a *good*metric, certainly better than the (more common?) difference-based metric, I can equally well see that it is not appropriate for all situations. --Abram On Tue, Sep 4, 2012 at 4:50 PM, Ben Goertzel <[email protected]> wrote: > > Well it's a subtle issue... > > For prediction of real-valued time series, one uses a variety of different > accuracy metrics, often custom ones based on what one is trying to use the > predictions for... > > Presumably predictions according to custom accuracy metrics, could be > mapped into lossy compressors with different metrics for lossiness.... > But the formulation of an approximate predictor in terms of lossy > compression may sometimes be a lot uglier... > > -- Ben > > On Tue, Sep 4, 2012 at 7:44 PM, Abram Demski <[email protected]>wrote: > >> Ben, >> >> That's completely fair. While any good predictor can be turned into a >> similarly good compressor by plugging it into standard arithmetic >> compression (so long as we can get probabilities out of the predictor), >> there is not really a need to do so in order to measure predictive >> accuracy... >> >> In other words, I would not argue that there is any great *need* to think >> about compression when working on prediction algorithms. I merely think the >> connection is rigorous. >> >> On the other hand, I do find it useful to think about at times (for >> example, when trying to intuitively think about KL-divergence, or as you >> mention, Occam bias). >> >> You could argue that everyone in the time series prediction field is >>> doing it wrong, and they should be using compressors instead of their >>> current algorithms. It's possible. But I'm not so sure... >> >> >> Definitely not claiming this. My presumption would sooner be that some of >> the best sequence prediction algorithms have not been tried out as >> compressors. Speed considerations are the first reason that comes to mind. >> To the extent that I would argue that PAQ should serve as an inspiration >> for AGI techniques, I would argue that it is an experimentally validated >> technique for fast predictions based on large amounts of data. >> >> --Abram >> >> On Tue, Sep 4, 2012 at 4:26 PM, Ben Goertzel <[email protected]> wrote: >> >>> >>> Abram, >>> >>> There is a large literature on the topic of time series prediction (an >>> area I've been doing some work in lately). There are many different >>> methods, applicable in various different contexts. I've been working >>> fairly short, noisy numerical time series... >>> >>> The methods one ends up using, in practice, when one wants to do >>> effective time series prediction of various sorts, are generally NOT based >>> on compression.... You could maybe theoretically interpret them in terms >>> of weird ways to do compression, with a lot of hand-waving, but that's not >>> really the convenient way to think about them... >>> >>> I guess this is the sort of thing Russell means by the disconnect >>> between theory and engineering/science practice, regarding the connection >>> between compression and prediction. >>> >>> You could argue that everyone in the time series prediction field is >>> doing it wrong, and they should be using compressors instead of their >>> current algorithms. It's possible. But I'm not so sure... >>> >>> In my own experience, for instance, if you're using an automated program >>> learning algorithm to do time series prediction, the Occam's Razor bias in >>> the program learning algorithm's fitness function is just one among many >>> aspects to tune to get good results. In this case, compression is one >>> ingredient to pay attention to in order to get prediction to work well, but >>> focusing solely on compression isn't the best path.... practically >>> speaking... >>> >>> -- Ben G >>> >>> >>> >>> >>> On Tue, Sep 4, 2012 at 7:07 PM, Abram Demski <[email protected]>wrote: >>> >>>> Matt tried precisely that experiment awhile ago, using a very good, >>>>> state-of-the-art text compression program. The fact that it could do text >>>>> prediction at all was an interesting validation of the mathematical >>>>> theory, >>>>> but its performance as a predictor was unsurprisingly abysmal. >>>> >>>> >>>> Russel, >>>> >>>> What are you referring to here? PAQ? PAQ uses arithmetic compression, >>>> so the compression performance varies precisely with the predictive >>>> ability; the only way to get it to compress better is to make it predict >>>> better, and the only reason it compresses well right now is because it >>>> predicts well. Furthermore, if we had some better prediction method, we >>>> could directly convert it to a better compression method. >>>> >>>> This is why I feel the compression<->prediction link is quite solid: >>>> because there is already an engineering-level solution here. >>>> >>>> There is one weakness here: predictions should be given as >>>> probabilities. If you give me a module which generates hard predictions, I >>>> can't so easily plug it into an arithmetic compressor. >>>> >>>> If we take a real compression program running on real hardware and try >>>>> to use it as a predictor let alone a general intelligence, does it perform >>>>> well? >>>> >>>> >>>> This paper: >>>> >>>> http://arxiv.org/abs/0909.0801/ >>>> >>>> uses a prediction framework similar (but, somewhat different) from PAQ, >>>> plugs it into a generic planner, and gets decent results. I find it >>>> unlikely that PAQ would do much worse in the same experimental set-up. (I >>>> find it likely that PAQ would do just a bit better... could be wrong, >>>> though.) >>>> >>>> So, the experimental answer seems to be "yes" to me. What experiments >>>> to the contrary are you thinking of? >>>> >>>> On Tue, Sep 4, 2012 at 3:06 PM, Russell Wallace < >>>> [email protected]> wrote: >>>> >>>>> For my part, the link I question is the implicit one between >>>>> mathematical principle and engineering practice. >>>>> >>>>> If we had an infinitely powerful computer on which we could run a >>>>> Kolmogorov compression oracle, would that give us AGI? Yes. >>>>> >>>>> If we take a real compression program running on real hardware and try >>>>> to use it as a predictor let alone a general intelligence, does it perform >>>>> well? No. Matt tried precisely that experiment awhile ago, using a very >>>>> good, state-of-the-art text compression program. The fact that it could do >>>>> text prediction at all was an interesting validation of the mathematical >>>>> theory, but its performance as a predictor was unsurprisingly abysmal. >>>>> >>>>> Even the reverse doesn't hold. We haven't got AGI to test it with, but >>>>> we can test it with NGI which we do have. Are humans good at compression? >>>>> No. Matt observes one reason for this is the lack of deterministic reset >>>>> capability, but if we imagine solving that problem by running an uploaded >>>>> human on digital hardware, thus enabling the mind in question to perform >>>>> lossless compression, it's clear the result would be very inefficient >>>>> compared to a specialized compression program. >>>>> >>>>> >>>>> On Tue, Sep 4, 2012 at 9:06 PM, Abram Demski <[email protected]>wrote: >>>>> >>>>>> Jim, >>>>>> >>>>>> What about the connection between prediction and compression? This is >>>>>> tightly argued. The basic argument for compression as an AGI test is: >>>>>> >>>>>> compression<-prediction<-AGI >>>>>> >>>>>> That is, AGI should be able to do prediction well, and predicting >>>>>> well allows compression. A stronger version goes like this: >>>>>> >>>>>> compression<->prediction<->AGI >>>>>> >>>>>> That is, compressing well also allows prediction, and prediction is >>>>>> central enough to AGI that we can reasonably say, predicting well would >>>>>> allow AGI. >>>>>> >>>>>> The various links and directions in this argument have different >>>>>> strengths and weaknesses. My question to you is: which links are you >>>>>> questioning here? >>>>>> >>>>>> Best, >>>>>> >>>>>> Abram >>>>>> >>>>>> On Mon, Sep 3, 2012 at 9:08 AM, Jim Bromer <[email protected]>wrote: >>>>>> >>>>>>> I would agree that any practical AGI program that I could imagine >>>>>>> would tend to compress some data. However, that does not mean that any >>>>>>> good compressor is practical AGI program. >>>>>>> >>>>>>> Now do all AGI programs have to be compressors? I would say that on >>>>>>> a theoretical basis no. For instance, if there was an effective AGI >>>>>>> program that compressed some data through common references, I could >>>>>>> offer >>>>>>> another AGI program that worked as well but simply duplicated all the >>>>>>> common references as they were needed to be used in some specific >>>>>>> circumstance. This shows that, on a theoretical basis, some AGI >>>>>>> programs >>>>>>> would not need to act as compressors. >>>>>>> >>>>>>> Now, let's see if all good effective AGI algorithms would >>>>>>> effectively compress data in all cases. We know that search time is >>>>>>> one of >>>>>>> the main contemporary problems and so this suggests that duplication - >>>>>>> at >>>>>>> least in some circumstances- might be more effective than compression. >>>>>>> >>>>>>> So to say that compression is the same as AGI is an over >>>>>>> generalization. >>>>>>> To say that any theoretical AGI program would have to be a >>>>>>> compressor is an over generalization. >>>>>>> To say that all effective AGI algorithms will effectively compress >>>>>>> data in all circumstances (as it learns) is an over generalization. >>>>>>> >>>>>>> Although I agree that any practical AGI program that I could imagine >>>>>>> will tend to compress some data, the imagined theoretical correlation >>>>>>> between compression and AGI is just not strong enough to argue that >>>>>>> compression is a good test of knowing. >>>>>>> >>>>>>> Jim Bromer >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mon, Sep 3, 2012 at 11:23 AM, Jim Bromer <[email protected]>wrote: >>>>>>> >>>>>>>> On Sat, Sep 1, 2012 at 1:35 PM, Abram Demski <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> This is *why* compression is a good test! It is a harsh test for >>>>>>>>> "knowing". >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Abram, >>>>>>>> That doesn't make any sense. You would have to substantiate that >>>>>>>> there is a high correlation between compression with "knowing" in >>>>>>>> order to >>>>>>>> make that statement. So far, the only correlation is of the nature of >>>>>>>> statements like those which say that calculators can do something that >>>>>>>> had >>>>>>>> previously only been possible for human beings to do. Although that >>>>>>>> kind >>>>>>>> of achievement can be taken as evidence that computers are able to do >>>>>>>> some >>>>>>>> kinds of thinking, that kind of thinking has been repeatedly >>>>>>>> repudiated in >>>>>>>> this group as being "narrow AI" and not AGI. So your claim that >>>>>>>> compression is a good harsh test for knowing has to be absolutely >>>>>>>> repudiated as well. >>>>>>>> >>>>>>>> I am glad that I am able to categorize the >>>>>>>> compression-is-intelligence approach as "narrow AI" even though I am >>>>>>>> not >>>>>>>> totally happy with that label. Because now, for the first time, I >>>>>>>> will be >>>>>>>> able to chip away at the basis of the approach (at least in this group) >>>>>>>> without having to go through some long winded reasoning that will be >>>>>>>> mostly >>>>>>>> ignored in this group (as it obviously has been in the past.) >>>>>>>> >>>>>>>> Compression is not a test for "knowing", except possibly for some >>>>>>>> special cases of narrow AI methods. >>>>>>>> >>>>>>>> Jim Bromer >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Sep 1, 2012 at 1:35 PM, Abram Demski <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Jim, >>>>>>>>> >>>>>>>>> This is *why* compression is a good test! It is a harsh test for >>>>>>>>> "knowing". >>>>>>>>> >>>>>>>>> --Abram >>>>>>>>> >>>>>>>>> On Fri, Aug 31, 2012 at 6:26 PM, Jim Bromer >>>>>>>>> <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> One of the mistakes that the compressors-is-intelligence guys >>>>>>>>>> make is >>>>>>>>>> that an ideal compression of knowledge cannot be made before >>>>>>>>>> knowing >>>>>>>>>> occurs. This is so obvious that some people will go to some >>>>>>>>>> lengths >>>>>>>>>> to point it out. But some of the implications are also really >>>>>>>>>> obvious >>>>>>>>>> as well. >>>>>>>>>> >>>>>>>>>> If an ideal compression cannot be made before knowing has occurred >>>>>>>>>> then an ideal compression can never be made for a program that is >>>>>>>>>> capable of learning because it will always be able to learn >>>>>>>>>> something >>>>>>>>>> new. Isn't this obvious? >>>>>>>>>> Jim Bromer >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ------------------------------------------- >>>>>>>>>> AGI >>>>>>>>>> Archives: https://www.listbox.com/member/archive/303/=now >>>>>>>>>> RSS Feed: >>>>>>>>>> https://www.listbox.com/member/archive/rss/303/7190161-766c6f07 >>>>>>>>>> Modify Your Subscription: https://www.listbox.com/member/?&; >>>>>>>>>> Powered by Listbox: http://www.listbox.com >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Abram Demski >>>>>>>>> http://lo-tho.blogspot.com/ >>>>>>>>> >>>>>>>>> *AGI* | Archives<https://www.listbox.com/member/archive/303/=now> >>>>>>>>> <https://www.listbox.com/member/archive/rss/303/10561250-164650b2>| >>>>>>>>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>>>>>>>> <http://www.listbox.com> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> *AGI* | Archives<https://www.listbox.com/member/archive/303/=now> >>>>>>> <https://www.listbox.com/member/archive/rss/303/7190161-766c6f07> | >>>>>>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>>>>>> <http://www.listbox.com> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Abram Demski >>>>>> http://lo-tho.blogspot.com/ >>>>>> >>>>>> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> >>>>>> <https://www.listbox.com/member/archive/rss/303/1658954-f53d1a3f> | >>>>>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>>>>> <http://www.listbox.com> >>>>>> >>>>> >>>>> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> >>>>> <https://www.listbox.com/member/archive/rss/303/7190161-766c6f07> | >>>>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>>>> <http://www.listbox.com> >>>>> >>>> >>>> >>>> >>>> -- >>>> Abram Demski >>>> http://lo-tho.blogspot.com/ >>>> >>>> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> >>>> <https://www.listbox.com/member/archive/rss/303/212726-11ac2389> | >>>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>>> <http://www.listbox.com> >>>> >>> >>> >>> >>> -- >>> Ben Goertzel, PhD >>> http://goertzel.org >>> >>> "My humanity is a constant self-overcoming" -- Friedrich Nietzsche >>> >>> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> >>> <https://www.listbox.com/member/archive/rss/303/7190161-766c6f07> | >>> Modify <https://www.listbox.com/member/?&;> Your Subscription >>> <http://www.listbox.com> >>> >> >> >> >> -- >> Abram Demski >> http://lo-tho.blogspot.com/ >> >> *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> >> <https://www.listbox.com/member/archive/rss/303/212726-11ac2389> | >> Modify<https://www.listbox.com/member/?&;>Your Subscription >> <http://www.listbox.com> >> > > > > -- > Ben Goertzel, PhD > http://goertzel.org > > "My humanity is a constant self-overcoming" -- Friedrich Nietzsche > > *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> > <https://www.listbox.com/member/archive/rss/303/7190161-766c6f07> | > Modify<https://www.listbox.com/member/?&;>Your Subscription > <http://www.listbox.com> > -- Abram Demski http://lo-tho.blogspot.com/ ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-c97d2393 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-2484a968 Powered by Listbox: http://www.listbox.com
