On Sat, Aug 1, 2020 at 3:52 AM <[email protected]> wrote:
> ... > Semantics: > If 'cat' and 'dog' both share 50% of the same contexts, then maybe the > ones they don't share are shared as well. So you see cat ate, cat ran, cat > ran, cat jumped, cat jumped, cat licked......and dog ate, dog ran, dog ran. > Therefore, probably the predictions not shared could be shared as well, so > maybe 'dog jumped' is a good prediction. > Agree with you on this. This is the basis of all grammar learning. I now believe it may equate to "causal invariants" (same contexts), and so possibly learned by position "transforms" or permutations. So with "transformers" the RNN guys may have stumbled on a true sense of meaning. Added to that, the fact that abandoning the original RNN model made it possible to learn hierarchy, may mean GPT-3 is now learning "grammar", with hierarchies and all. So transformers may equate to a full grammar learning architecture. It's even possible that because they have no guiding theory, they may be allowing contradictions in their parameters in some way. That would be a big thing from my point of view. I think the doctrinal rejection of contradiction is what is holding back those who formally attempt to learn grammar. Like OpenCog's own historical grammar learning projects. If GPT-3 is learning grammatical forms which contradict according to context, the only remaining problem from the point of view of my dogma would be that it is trying to learn, not generate, grammar using this (causal invariant, transformer) principle. I think it needs to generate these 175+ parameters on the fly, not try to enumerate them all beforehand at a cost of $12M (Geoff Hinton suggests end the search at 4.398 trillion = 2^42 :-) -Rob ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T21c073d3fe3faef0-Mee5f7150dd41c23eb94f5620 Delivery options: https://agi.topicbox.com/groups/agi/subscription
