On Thu, Nov 21, 2019 at 3:33 PM Matt Mahoney <[email protected]> wrote:
> That's a good example of the weakness of the lossy compression model. We > only use it because it's better than nothing. > Ah, but my conjecture is that selecting the executable archive _is_ better than _any_ lossy compression model. Moreover, I believe I've provided a succinct and convincing case for it being relatively stronger: One model's "noise" is another model's cyphertext, or, to put it in terms more directly related to compression: Almost _any_ , even mediocre, compression scheme will output an incompressible stream of bits -- that is unless that stream is first decompressed and then recompressed with a superior compression scheme. > Find a function that approximates the data. Choose the function that > minimizes the description length of the function plus the prediction errors. > The operative word being "plus", which implies both the model and the error must be brought into commensurability, ie: the same units of information: e.g. bits. You don't get that out of ANY information criterion other than the executable archive (which may be thought of as MDL constrained to UTM equivalent symbols). > > Another example. Suppose you have 10 sample points of the form (x, e^x). > You can search the space of polynomials for the best fit. A 9th order > polynomial will fit exactly, but a lower order polynomial approximation > might get a better score. Which will make better predictions for new x? > Neither, of course, but we can't search all possible functions either. > Which will make better predictions for a new x? The answer is not "neither" let alone "neither, of course". The answer is the one producing the shortest executable archive of the data is more likely than the other to produce better predictions. Nor is this all mere pedantry. Since, as I've said repeatedly, we're facing a war that could kill enormous numbers of people -- a probabiity that may be ameliorated by getting a consensus on this. When even "The Atlantic" is carrying a full issue dedicated to finding ways of averting the looming "civil war" over social theories <https://www.theatlantic.com/magazine/toc/2012/02/>, and when at least one side is skeptical of claims by social scientists that they are "unbiased", finding *meta*-*consensus* on a *minimally* biased model selection criterion could significantly reduce the distrust of social theories so-selected. Since the Federal government has arrogated to itself powers of social policy well beyond those envisioned by the architects of the laboratory of the States, it is incumbent upon those who despair at giving priority to consent over consensus, due to the Federal government's hubris, to, at the very least, promote a *meta*-consensus so that whatever model selection criterion in used to reach consensus, has, itself, a consensus. If a quasi-religious war were to break out simply because people who should know better can't recognize the fact that the only "model selection criterion" that brings errors into commensurability (ie: same units of measure) with models is lossless compression (executable archive length), I wouldn't want to be in their shoes when the rivers of blood start flowing -- not even if I believed in a universe without justice. > > On Thu, Nov 21, 2019, 11:47 AM James Bowery <[email protected]> wrote: > >> I, quite deliberately, did not mention "Solomonoff Induction" as an >> information criterion for model selection, precisely because it is not >> computable. The point of my conjecture is that there is a very good >> reason to select "the smallest executable archive of the data" as your >> information criterion over the other information criteria -- and it has to >> do with the weakness of "lossy compression" as model selection. >> >> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + delivery > options <https://agi.topicbox.com/groups/agi/subscription> Permalink > <https://agi.topicbox.com/groups/agi/T0fc0d7591fcf61c5-M1030ad8f00ba56ae47106a3a> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T0fc0d7591fcf61c5-M7a6b224deb8a9a6a2e1bafcc Delivery options: https://agi.topicbox.com/groups/agi/subscription
