Rob, have you looked at the examples cited in my article, that I linked here? Seeing this particular sort of stupidity from them, it's hard to see how these networks would be learning the same sorts of "causal invariants" as humans are...
Transformers clearly ARE a full grammar learning architecture, but in a non-AGI-ish sense. They are learning the grammar of the language underlying their training corpus, but mixed up in a weird and non-human-like way with so many particulars of the corpus. Humans also learn the grammar of their natural languages mixed up with the particulars of the linguistic constructs they've encountered -- but the "subtle" point (which obviously you are extremely capable to grok) is that the mixing-up of abstract grammatical patterns with concrete usage patterns in human minds is of a different nature than the mixing-up of abstract grammatical patterns with concrete usage patterns in GPT3 and other transformer networks. The human form of mixing-up is more amenable to appropriate generalization. ben On Fri, Jul 31, 2020 at 6:33 PM Rob Freeman <[email protected]> wrote: > > On Sat, Aug 1, 2020 at 3:52 AM <[email protected]> wrote: >> >> ... >> Semantics: >> If 'cat' and 'dog' both share 50% of the same contexts, then maybe the ones >> they don't share are shared as well. So you see cat ate, cat ran, cat ran, >> cat jumped, cat jumped, cat licked......and dog ate, dog ran, dog ran. >> Therefore, probably the predictions not shared could be shared as well, so >> maybe 'dog jumped' is a good prediction. > > > Agree with you on this. This is the basis of all grammar learning. > > I now believe it may equate to "causal invariants" (same contexts), and so > possibly learned by position "transforms" or permutations. > > So with "transformers" the RNN guys may have stumbled on a true sense of > meaning. > > Added to that, the fact that abandoning the original RNN model made it > possible to learn hierarchy, may mean GPT-3 is now learning "grammar", with > hierarchies and all. > > So transformers may equate to a full grammar learning architecture. > > It's even possible that because they have no guiding theory, they may be > allowing contradictions in their parameters in some way. That would be a big > thing from my point of view. I think the doctrinal rejection of contradiction > is what is holding back those who formally attempt to learn grammar. Like > OpenCog's own historical grammar learning projects. > > If GPT-3 is learning grammatical forms which contradict according to context, > the only remaining problem from the point of view of my dogma would be that > it is trying to learn, not generate, grammar using this (causal invariant, > transformer) principle. I think it needs to generate these 175+ parameters on > the fly, not try to enumerate them all beforehand at a cost of $12M (Geoff > Hinton suggests end the search at 4.398 trillion = 2^42 :-) > > -Rob > Artificial General Intelligence List / AGI / see discussions + participants + > delivery options Permalink -- Ben Goertzel, PhD http://goertzel.org “The only people for me are the mad ones, the ones who are mad to live, mad to talk, mad to be saved, desirous of everything at the same time, the ones who never yawn or say a commonplace thing, but burn, burn, burn like fabulous yellow roman candles exploding like spiders across the stars.” -- Jack Kerouac ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T21c073d3fe3faef0-M9843aab216f1ab32192d5101 Delivery options: https://agi.topicbox.com/groups/agi/subscription
