*** But then if you are doing all this, why are you using BERT type training "to guide the numerical weightings of symbolic language-patterns"? That will still trap you in the limitations of learned representations. The whole point of a network is that, like a distributed representation, it can handle multiplicity of interpretation. Once you fix it by "learning" you have lost this. Perhaps the high current state of development of these learning algorithms may help in the short term, but it seems like a misstep. ***
Yes, it's intended to help in the short term by providing better data-driven weightings for particular interpretations... I agree in the medium term we don't need to use deep NNs to tweak the weights of various patterns, but in the short term I believe it will help considerably... The unfortunate fact is we can't currently feed as much data into our OpenCog self-adapting graph as we can into a BERT type model, given available resources... thus using the latter to help tweak weights in the former may have significant tactical advantage... ben On Tue, Feb 19, 2019 at 5:23 PM Rob Freeman <[email protected]> wrote: > > Linas, > > Ooh, Nice. > > This is different to what I saw in the links Ben posted. If you are really > deconstructing your grammar like this then Ben could be right, it might be a > good fit with me. Everything can reduce to graphs. If you are visiting there > from Link Grammar rather than embedding vectors which was my path, that does > not matter. So long as you travel fully along to the destination of a raw > network we can get the same power. > > An aside. You mention sheaf theory as a way to get around the linearity of > vector spaces. Is this influenced in any way by what Coecke, Sadrzadeh, and > Clark proposed for compositional distributional models in the '00s? > > E.g. > Category-Theoretic Quantitative Compositional Distributional Models of > Natural Language Semantics > Edward Grefenstette > https://arxiv.org/abs/1311.1539 > > I see you cite Coecke in your 2017 "Sheaves: A Topological Approach to Big > Data" paper. > > Personally I followed their work when I came across it in the '00s. It was > the first other work in a compositional distributional vein I had come across > so I was delighted to find it. There was precious little about distributed > models in the '00s, let alone compositional distributional. But I decided > that the formalisms of both category theory as a response to the subjectivity > of maths, and QM as a model for the subjectivity of physics, may well apply, > but that in practice it will be easier to build structures which manifest > these properties, rather than to formally describe them. > > Anyway, perhaps Ben is right, you may be doing the first two steps of my > suggested solution: 1) coding only a sequence net of observed sequences, and > 2) projecting out latent "invariants" by clustering according to shared > contexts. > > But then if you are doing all this, why are you using BERT type training "to > guide the numerical weightings of symbolic language-patterns"? That will > still trap you in the limitations of learned representations. The whole point > of a network is that, like a distributed representation, it can handle > multiplicity of interpretation. Once you fix it by "learning" you have lost > this. Perhaps the high current state of development of these learning > algorithms may help in the short term, but it seems like a misstep. > > The solution I came is to forget all thought of training or "learning" > representations. Not least because you get contradictions. > > And I believe the best way to do that will be to set the network oscillating > and varying inhibition, to get the resolution of groupings we want > dynamically. > > -Rob > > On Tue, Feb 19, 2019 at 6:45 PM Linas Vepstas <[email protected]> wrote: >> >> Hi Rob, >> >> On Mon, Feb 18, 2019 at 4:40 PM Rob Freeman <[email protected]> >> wrote: >>> >>> Ben, >>> >>> That's what I thought. You're still working with Link Grammar. >>> >>> But since last year working on informing your links with stats from deep-NN >>> type, learned, embedding vector based predictive models? You're trying to >>> span the weakness of each formalism with the strengths of the other?? >> >> >> Yes but no. I've been trying to explain what exactly is good, and what, >> exactly is bad with NN vector-space models. There is a long tract written on >> this here. >> https://github.com/opencog/opencog/raw/master/opencog/nlp/learn/learn-lang-diary/skippy.pdf >> >>> >>> >>> There's a lot to say about all of that. >>> >>> Your grammar will be learned, with only the resolution you bake in from the >>> beginning. >> >> No. >> >>> >>> Your embedding vectors will be learned, >> >> >> The point of the long PDF is to explain why NN-vectors are bad. It attempts >> to first explain *why* neural nets work for language, and why vectors are >> *almost* the right thing, and then it tries to explain why NN vectors don't >> actually do everything you actually want. I've noticed that, in the middle >> of all these explanations, I lose my audience; haven't figured out how to >> keep them, yet. >> >>> >>> and the dependency decisions they can inform on learned, and thus finite, >>> too. Plus you need to keep two formalisms and marry them together... Large >>> teams for all of that... >> >> >> No. I've already got 75% of it coded up. It actually works, I've got long >> diary entries and notes with detailed stats on it all. Unfortunately, I >> have not been able to carve out the time to finish the work, its been >> stalled since the fall of last year. >> >> It would be wonderful if I could get someone else interested in this work. >> >> --linas > > Artificial General Intelligence List / AGI / see discussions + participants + > delivery options Permalink -- Ben Goertzel, PhD http://goertzel.org "Listen: This world is the lunatic's sphere, / Don't always agree it's real. / Even with my feet upon it / And the postman knowing my door / My address is somewhere else." -- Hafiz ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T581199cf280badd7-M20f940db3fad8c9f623cf564 Delivery options: https://agi.topicbox.com/groups/agi/subscription
