I agree. The top ranked text compressors don't model grammar at all.

On Fri, May 24, 2024, 11:47 PM Rob Freeman <chaotic.langu...@gmail.com>
wrote:

> Ah, I see. Yes, I saw that reference. But I interpreted it only to
> mean the general forms of a grammar. Do you think he means the
> mechanism must actually be a grammar?
>
> In the earlier papers I interpret him to be saying, if language is a
> grammar, what kind of a grammar must it be? And, yes, it seemed he was
> toying with actual physical mechanisms relating to levels of brain
> structure. Thalamo-cortical loops?
>
> The problem with that is, language doesn't actually seem to be any
> kind of grammar at all.
>
> It's like saying if the brain had to be an internal combustion engine,
> it might be a Mazda rotary. BFD. It's not an engine at all.
>
> I don't know if the authors realized that. But surely that's the point
> of the HNet paper. That something can generate the general forms of a
> grammar, without actually being a grammar.
>
> I guess this goes back to your assertion in our prior thread that
> "learning" needs to be constrained by "physical priors" of some kind
> (was it?) Are there physical "objects" constraining the "learning", or
> does the "learning" vaguely resolve as physical objects, but not
> quite?
>
> I don't think vague resemblance to objects means the objects must exist,
> at all.
>
> Take Kepler and the planets. If the orbits of planets are epicycles,
> which epicycles would they be? The trouble is, it turns out they are
> not epicycles.
>
> And at least epicycles work! That's the thing for natural language.
> Formal grammar doesn't even work. None of them. Nested stacks, context
> free, Chomsky hierarchy up, down, and sideways. They don't work. So
> figuring out which formal grammar is best, is a pointless exercise.
> None of them work.
>
> Yes, broadly human language seems to resolve itself into forms which
> resemble formal grammar (it's probably designed to do that, so that it
> can usefully represent the world.) And it might be generally useful to
> decide which formal grammar it best (vaguely) resembles.
>
> But in detail it turns out human language does not obey the rules of
> any formal grammar at all.
>
> It seems to be a bit like the way the output of a TV screen looks like
> objects moving around in space. Yes, it looks like objects moving in
> space. You might even generate a physics based on the objects which
> appear to be there. It might work quite well until you came to Road
> Runner cartoons. That doesn't mean the output of a TV screen is
> actually objects moving around in space. If you insist on implementing
> a TV screen as objects moving around in space, well, it might be a
> puppet show similar enough to amuse the kids. But you won't make a TV
> screen. You will always fail. And fail in ways very reminiscent of the
> way formal grammars almost succeed... but fail, to represent human
> language.
>
> Same thing with a movie. Also looks a lot like objects moving around
> on a screen. But is it objects moving on a screen? Different again.
>
> Superficial forms do not always equate to mechanisms.
>
> That's what's good about the HNet paper for me. It discusses how those
> general forms might emerge from something else.
>
> The history of AI in general, and natural language processing in
> particular, has been a search for those elusive "grammars" we see
> chasing around on the TV screens of our minds. And they all failed.
> What has succeeded has been breaking the world into bits (pixels?) and
> allowing them to come together in different ways. Then the game became
> how to bring them together. Supervised "learning" spoon fed the
> "objects" and bound the pixels together explicitly. Unsupervised
> learning tried to resolve "objects" as some kind of similarity between
> pixels. AI got a bump when, by surprise, letting the "objects" go
> entirely turned out to generate text that was more natural than ever!
> Who'd a thunk it? Letting "objects" go entirely works best! If it
> hadn't been for the particular circumstances of language, pushing you
> to a "prediction" conception of the problem, how long would it have
> taken us to stumble on that? The downside to that was, letting
> "objects" go entirely also doesn't totally fit with what we
> experience. We do experience the world as "objects". And without those
> "objects" at all, LLMs are kind of unhinged babblers.
>
> So where's the right balance? Is the solution as LeCun, and perhaps
> you, suggest (or Ben, looking for "semantic primitives" two years
> ago...), to forget about the success LLMs had by letting go of objects
> entirely. To repeat our earlier failures and seek the "objects"
> elsewhere. Some other data. Physics? I see the objects, dammit! Look!
> There's a coyote, and there's a road runner, and... Oh, my physics
> didn't allow for that...
>
> Or could it be the right balance is, yes, to ignore the exact
> structure of the objects as LLMs have done, but no, not to do it as
> LLMs do by totally ignoring "objects", but to ignore only the internal
> structure of the "objects", by focusing on relations defining objects
> in ways which allow their internal "pattern" to vary.
>
> That's what I see being presented in the HNet paper. Maybe I'm getting
> ahead of its authors. Because that is the solution I'm presenting
> myself. But I interpret the HNet paper to present that option also.
> Cognitive objects, including "grammar", can emerge with a freedom
> which resembles the LLM freedom of totally ignoring "objects" (which
> seems to be necessary, both by the success of LLMs at generating text,
> and by the observed failure of formal grammars historically) if you
> specify them in terms of external relations.
>
> Maybe the paper authors don't see it. But the way they talk about
> generating grammars based on external relations, opens the door to it.
>
> On Fri, May 24, 2024 at 10:12 PM James Bowery <jabow...@gmail.com> wrote:
> >
> >
> >
> > On Thu, May 23, 2024 at 9:19 PM Rob Freeman <chaotic.langu...@gmail.com>
> wrote:
> >>
> >> ...(Regarding the HNet paper)
> >> The ideas of relational category in that paper might really shift the
> >> needle for current language models.
> >>
> >> That as distinct from the older "grammar of mammalian brain capacity"
> >> paper, which I frankly think is likely a dead end.
> >
> >
> > Quoting the HNet paper:
> >>
> >> We conjecture that ongoing hierarchical construction of
> >> such entities can enable increasingly “symbol-like” repre-
> >> sentations, arising from lower-level “statistic-like” repre-
> >> sentations. Figure 9 illustrates construction of simple “face”
> >> configuration representations, from exemplars constructed
> >> within the CLEVR system consisting of very simple eyes,
> >> nose, mouth features. Categories (¢) and sequential rela-
> >> tions ($) exhibit full compositionality into sequential rela-
> >> tions of categories of sequential relations, etc.; these define
> >> formal grammars (Rodriguez & Granger 2016; Granger
> >> 2020). Exemplars (a,b) and near misses (c,d) are presented,
> >> initially yielding just instances, which are then greatly re-
> >> duced via abductive steps (see Supplemental Figure 13).
> >
> > Artificial General Intelligence List / AGI / see discussions +
> participants + delivery options Permalink

------------------------------------------
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T682a307a763c1ced-Mca3eb6ef6f8a4b6ebcbad2b5
Delivery options: https://agi.topicbox.com/groups/agi/subscription

Reply via email to