On Mon, Jun 29, 2020 at 9:43 AM Matt Mahoney <[email protected]> wrote:
> The problem with Occam's Razor or algorithmic information theory is > that simplicity is language dependent. In practice, that's for time complexity _only_. In practice, the data founding our models are vastly in excess of the size of the smallest UTM interpreter. This is not to minimize the practical importance of time complexity -- but "fair" model selection should be purely a matter of size under the same resource (such as time) constraints (as obviously is the case with the Hutter Prize). Any object can be described > using one bit if the language is complex enough. We should prefer > simple languages to avoid this problem, but defining simple languages > just leads to a circular definition. It seems there is no getting around picking a language arbitrarily. I have my doubts about this, which is why I asked the AGI conference panel: "Is there some notion of a UTM's complexity?" ...and got referred to Goertzel's paper. > And as long as we are doing so, why not pick one that is easy to compute > in our observed universe, and > penalizes run time. This is what Ben did. There is a long history of attempts to come up with a way of folding in time to algorithmic information measures. But, until there is a physical theory that defines informational "space-time" as a scalar quantity in the same sense that "mass-energy" is, the quantities are incommensurable hence lead to an "arbitrary" choice from among an infinite number of Pareto frontiers. (And of course Pareto frontiers being only partial orders, can't do model selection.) I'm all ears about efforts in this direction and hope it can be accomplished in a principled manner. Speaking of which, I received a notification from Chaitin Academia account last week when he uploaded "THE PERFECT LANGUAGE (Inference: International Review of Science, 2015) <https://www.academia.edu/14013024/_THE_PERFECT_LANGUAGE_Inference_International_Review_of_Science_2015_>" which contains this passage: "These self-delimiting binary languages are the ones that the study of program-size complexity has led us to discriminate as the ideal languages, the most perfect languages. We got to them in two stages, 1960s AIT and 1970s AIT. These are languages for computation, for expressing algorithms, not for mathematical reasoning. They are universal programming languages that are maximally expressive, maximally concise. We already knew how to do that in the 1960s, but in the 1970s we realized that programs should be self-delimiting, which made it possible to define the halting probability Ω." As opposed to e.g. Wolfram's > 3 color 2 state Turing machine, where even the proof of Turing > completeness was far from trivial. > > In spite of its problems, Occam's Razor is well established in every > branch of science and is the foundation of every application of > machine learning. I would really like to see a definition of > simplicity that is grounded in experimental data, but I understand the > difficulties of doing so. In addition to covering such a broad area of > science, it is difficult to test any model that isn't easy to compute. > For example, there may be a description of the observable universe > that requires only a few hundred bits, but there is obviously no way > to simulate a universe (estimated at 10^120 quantum operations) on any > computer contained within it. > > On Sun, Jun 28, 2020 at 1:28 AM Ben Goertzel <[email protected]> wrote: > > > > Hi James, etc., > > > > That paper sat on my hard drive for about a decade because I wasn't so > > happy with the way I'd phrased things in the introduction... but > > finally I decided to just post it on Arxiv anyway because I felt the > > basic formalization of simplicity measures was OK, and I wanted to use > > it in some other papers I was going to publish or post... > > > > Anyway the general framing discussion at the start of that paper is > > probably not how I'd choose to frame things today, but the key point > > there is that I wanted to have a characterization of "what is a > > simplicity measure" that was more abstract and axiomatic rather than > > committing intrinsically to a particular measure (such as algorithmic > > information, or mixes of runtime and program length as in > > Schmidhuber's frontier search, etc.)... > > > > This was useful to me in thinking about combinatorial decision dags > > (which I'm looking at as a potential representational underpinning for > > Atomese 2.0 language) and also in thinking about Occam's Razor in the > > context of hypercomputation and nonwellfounded sets (which are beyond > > the Turing level of computation), which is relevant to my recent blog > > post on preservation of goal systems under self-modification.... (in > > that case I am looking at goal systems defined in terms of > > nonwellfounded sets as a potential way of thinking about computable > > goal systems, in teame way that we can look at real number math as a > > way of thinking about practical calculations involving > > finite-precision numbers) ... > > > > I don't expect you to read through it all, but this formalization of > > simplicity for me is part of an overall attempt to come to a > > fundamental theoretical understanding of what is general intelligence, > > which has been written up in bits and pieces in various papers over > > the years, as roughly listed out in > > > > > http://multiverseaccordingtoben.blogspot.com/2020/05/gtgi-general-theory-of-general.html > > > > This theoretical work has been only off-and-on correlated w/ practical > > work I've done w/ openCog, SingularityNET etc. but is playing a > > slightly greater role in recent work aimed at formulating an effective > > meta-representational and programming language framework for a new > > majorly improved/different version of openCog some colleagues and I > > are now working on... > > > > ben > > > > On Sat, Jun 27, 2020 at 7:09 AM James Bowery <[email protected]> wrote: > > > > > > I don't hold that against Goertzel. Solomonoff's 2 seminal papers on > algorithmic induction are "complex" as well. It's just that I'm not very > motivated by a complaint that universal computation is "very specialized" > without a "general" context stated in an incisive, concise and intuitive > manner. The complaint is absurd on its face. > > > > > > On Sat, Jun 27, 2020 at 7:15 AM stefan.reich.maker.of.eye via AGI < > [email protected]> wrote: > > >> > > >> It's a little funny when a paper on defining simplicity is a highly > complex read... :) > > > > > > Artificial General Intelligence List / AGI / see discussions + > participants + delivery options Permalink > > > > > > -- > > Ben Goertzel, PhD > > http://goertzel.org > > > > “The only people for me are the mad ones, the ones who are mad to > > live, mad to talk, mad to be saved, desirous of everything at the same > > time, the ones who never yawn or say a commonplace thing, but burn, > > burn, burn like fabulous yellow roman candles exploding like spiders > > across the stars.” -- Jack Kerouac > > > -- > -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T37756381803ac879-Mb38b0f6ec5d16aefcb4f9470 Delivery options: https://agi.topicbox.com/groups/agi/subscription
