Linas, Ooh, Nice.
This is different to what I saw in the links Ben posted. If you are really deconstructing your grammar like this then Ben could be right, it might be a good fit with me. Everything can reduce to graphs. If you are visiting there from Link Grammar rather than embedding vectors which was my path, that does not matter. So long as you travel fully along to the destination of a raw network we can get the same power. An aside. You mention sheaf theory as a way to get around the linearity of vector spaces. Is this influenced in any way by what Coecke, Sadrzadeh, and Clark proposed for compositional distributional models in the '00s? E.g. Category-Theoretic Quantitative Compositional Distributional Models of Natural Language Semantics Edward Grefenstette https://arxiv.org/abs/1311.1539 I see you cite Coecke in your 2017 "Sheaves: A Topological Approach to Big Data" paper. Personally I followed their work when I came across it in the '00s. It was the first other work in a compositional distributional vein I had come across so I was delighted to find it. There was precious little about distributed models in the '00s, let alone compositional distributional. But I decided that the formalisms of both category theory as a response to the subjectivity of maths, and QM as a model for the subjectivity of physics, may well apply, but that in practice it will be easier to build structures which manifest these properties, rather than to formally describe them. Anyway, perhaps Ben is right, you may be doing the first two steps of my suggested solution: 1) coding only a sequence net of observed sequences, and 2) projecting out latent "invariants" by clustering according to shared contexts. But then if you are doing all this, why are you using BERT type training "to guide the numerical weightings of symbolic language-patterns"? That will still trap you in the limitations of learned representations. The whole point of a network is that, like a distributed representation, it can handle multiplicity of interpretation. Once you fix it by "learning" you have lost this. Perhaps the high current state of development of these learning algorithms may help in the short term, but it seems like a misstep. The solution I came is to forget all thought of training or "learning" representations. Not least because you get contradictions. And I believe the best way to do that will be to set the network oscillating and varying inhibition, to get the resolution of groupings we want dynamically. -Rob On Tue, Feb 19, 2019 at 6:45 PM Linas Vepstas <[email protected]> wrote: > Hi Rob, > > On Mon, Feb 18, 2019 at 4:40 PM Rob Freeman <[email protected]> > wrote: > >> Ben, >> >> That's what I thought. You're still working with Link Grammar. >> >> But since last year working on informing your links with stats from >> deep-NN type, learned, embedding vector based predictive models? You're >> trying to span the weakness of each formalism with the strengths of the >> other?? >> > > Yes but no. I've been trying to explain what exactly is good, and what, > exactly is bad with NN vector-space models. There is a long tract written > on this here. > https://github.com/opencog/opencog/raw/master/opencog/nlp/learn/learn-lang-diary/skippy.pdf > > > >> >> There's a lot to say about all of that. >> >> Your grammar will be learned, with only the resolution you bake in from >> the beginning. >> > No. > > >> Your embedding vectors will be learned, >> > > The point of the long PDF is to explain why NN-vectors are bad. It > attempts to first explain *why* neural nets work for language, and why > vectors are *almost* the right thing, and then it tries to explain why NN > vectors don't actually do everything you actually want. I've noticed that, > in the middle of all these explanations, I lose my audience; haven't > figured out how to keep them, yet. > > >> and the dependency decisions they can inform on learned, and thus finite, >> too. Plus you need to keep two formalisms and marry them together... Large >> teams for all of that... >> > > No. I've already got 75% of it coded up. It actually works, I've got long > diary entries and notes with detailed stats on it all. Unfortunately, I > have not been able to carve out the time to finish the work, its been > stalled since the fall of last year. > > It would be wonderful if I could get someone else interested in this work. > > --linas > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T581199cf280badd7-Mf8d91ef7fb9013cf13f130c7 Delivery options: https://agi.topicbox.com/groups/agi/subscription
