Short answer only: I've been busy distracted with other things, but this remark:
> The problem with sparse representations is that ANNs work better with dense tensors. is something I have been trying to say for what, five, seven years? I've written two or three very long papers explaining this in gory detail, and they've been received like a fish-flop. It's been very discouraging, its pushing a rock uphill. Mostly I've been told that "its obvious" and one fellow even had the gall to suggest that I go back and finish college. I'll try to get back to these topics at some point "real soon now"... -- Linas On Tue, May 19, 2020 at 6:10 AM Adrian Borucki <[email protected]> wrote: > > > On Friday, 1 May 2020 16:31:33 UTC+2, linas wrote: >> >> Hi Debbie, >> >> Let me take a step back. My interest in this thread is to define generic >> API's for the AtomSpace that can be used in a variety of different ways, by >> different people, for different projects. Thus, I want to understand the >> embedding itself, and understand how to create an abstract API for it. I >> want to give you two examples of what I'm talking about, and it would be >> great if you could look them over. Now, I'm NOT expecting you to write any >> new code, nor to modify your existing code. Instead, I'm looking for a >> discussion on creating the kind of API's that might have made the embedding >> easier to do, in the first place. By "API", I usually mean "Atomese", but >> not always... the goal is to create a proposal, a sketch of what the API >> might look like; the code would only be written later (by whomever, I don't >> expect it to be you. It would be great if it wasn't me, but someone I could >> tutor...) >> >> So, two examples to ponder. One is a blog entry about "value flows" and >> it gives a hint of how one can apply "formulas" to change "values" attached >> to a graph. I'm not saying that this is useful for your work -- rather, its >> just an example of what I mean, when I say "Atomese" -- so here: >> https://blog.opencog.org/2020/04/08/value-flows/ >> >> Another example is the existing API for extracting vectors from the >> AtomSpace. It's surely very different from what you did, but it does show >> how to obtain AtomSpace contents as a vector -- an extremely sparse, very >> high-dimensional vector. >> > > That's interesting, I've been looking into Vector Symbolic Architectures > recently (for example A comparison of Vector Symbolic Architectures > <https://www.semanticscholar.org/paper/A-comparison-of-Vector-Symbolic-Architectures-Schlegel-Neubert/ba0caaa0817379c76ca06f7b53fb7c38a59663c1#extracted>). > Is the aforementioned sparse representation of the same family of > techniques? I've been wondering for a long time how to integrate neural and > OpenCog techniques - there are mainstream ways of doing that nowadays, with > dense embeddings that you can get from algorithms like DeepWalk. The > problem with sparse representations is that ANNs work better with dense > tensors. The alternative is to go in the direction of more neuromorphic > approaches - like what Numenta does (How Can We Be So Dense? The Benefits > of Using Highly Sparse Representations > <https://www.semanticscholar.org/paper/How-Can-We-Be-So-Dense-The-Benefits-of-Using-Highly-Ahmad-Scheinkman/3055e31b007c93fb9291b3ecd5b3c5f94fe9225f>, > I played around with their code but it kept crashing on larger problems), this > Ogma.ai guy <https://ogma.ai/2020/05/a-tutorial-on-ogmaneo2/> or > "cortical computing" that Janusz Marecki (a researcher from Deep Mind) > talks about (I can send some slides but they are not to be circulated > widely). > > One thing that one could achieve with this representation is to have a > network that learns to directly manipulate and read AtomSpace contents. It > seems that dense embeddings are mostly used to extract information from > graphs but not to modify it - a notable exception here is Learning to > Reason with Third-Order Tensor Products > <https://www.semanticscholar.org/paper/Learning-to-Reason-with-Third-Order-Tensor-Products-Schlag-Schmidhuber/f6fa98b0d1f6bff608a056acd7b82e2a9dd0a68e>, > they use Tensor Product Representations which have the unfortunate > disadvantage of growing quadratically in size with the size of the > structure they model (they are not "reduced"). > > >> I used this very heavily in the language work. I imagine its completely >> different from what you did, but I suspect that it might be a very good >> direction to move in for vector (generically, tensor) mappings. Basically, >> the AtomSpace can be thought of "very naturally" as a high-dimensional, >> ultra-sparse tensor. *Everything* we need for that is already present. What >> is absent is a clear, easy-to-understand, easy-to-use API for it. So, >> here: >> https://github.com/opencog/atomspace/blob/master/opencog/matrix/README.md >> >> Please read both of these, think about them, and see if you can make >> sense of them, and then see if there is some sensible way of doing >> something similar that would have made your original embedding simpler or >> easier. >> >> --linas >> >> >> >> On Thu, Apr 30, 2020 at 8:32 AM Deborah Duong <[email protected]> >> wrote: >> >>> >>> Hi Linas, >>> Hope you and your family are doing well! Long time no see! >>> I wrote some deepwalk embeddings of the atomspace in a Python notebook >>> (too big to attach), and Mike mentioned to me this morning that we could >>> add them to your vision and integrate them into the Opencog codebase. If >>> you are interested please tell me how I should modify the code or otherwise >>> help. >>> Debbie >>> >>> On Wednesday, April 29, 2020 at 8:31:10 AM UTC-7, linas wrote: >>>> >>>> Well, let me rephrase that: I was hoping to provoke a discussion >>>> regarding new features or capabilities or code that could be added to the >>>> atomspace, or tuned/improved, or the creation of an adjunct project (i.e. >>>> additional git repos where the implementation would sit). >>>> >>>> --linas >>>> >>>> On Wed, Apr 29, 2020 at 2:22 AM Ben Goertzel <[email protected]> wrote: >>>> >>>>> Well DeepWalk for embedding graph nodes into vectors already has been >>>>> popularized and has fairly broad applicability, what is new in what >>>>> Debbie Duong did (which I'll say more about soon) is applying it to a >>>>> labeled hypergraph rather than just a graph... >>>>> >>>>> >>>>> On Tue, Apr 28, 2020 at 11:57 PM Linas Vepstas <[email protected]> >>>>> wrote: >>>>> > >>>>> > Ben, as you know, I'm more interested in code and API's that are >>>>> generic and useful. So I'm looking at the creation of things that have >>>>> broad applicability, are easy to use, and can be popularized. Things that >>>>> can be demoed. Demos that can be tinkered with by ordinary people. >>>>> > >>>>> > --linas >>>>> > >>>>> > On Wed, Apr 29, 2020 at 12:39 AM Ben Goertzel <[email protected]> >>>>> wrote: >>>>> >> >>>>> >> Re vector embeddings, we have been using DeepWalk to create vector >>>>> >> embeddings from Atomspace, and have achieved some interesting >>>>> mappings >>>>> >> btw Atom logical relationships and corresponding vector >>>>> relationships >>>>> >> ... a draft paper on this will be posted soon... >>>>> >> >>>>> >> On Tue, Apr 28, 2020 at 8:55 PM Linas Vepstas <[email protected]> >>>>> wrote: >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > On Tue, Apr 28, 2020 at 9:58 PM Michael Duncan <[email protected]> >>>>> wrote: >>>>> >> >> >>>>> >> >> >>>>> >> >> and some more detailed thoughts on automated hypothesis >>>>> formation regarding COVID-19 therapies are at >>>>> >> >> >>>>> https://docs.google.com/document/d/1QAP00zGlpn2Bnrsug622DbrB8s6l3RzlfYUw14rszQw/edit >>>>> >> > >>>>> >> > >>>>> >> > The above document mentions "vector embeddings". The current >>>>> vector embedding API is in the (opencog matrix) module, and there's >>>>> actually quite a lot of stuff there. I suspect no one beside me has used >>>>> that code ... >>>>> >> > >>>>> >> > A writeup of that code is here: >>>>> >> > https://github.com/opencog/atomspace/tree/master/opencog/matrix >>>>> >> > >>>>> >> > I also suspect that whoever wrote the words "vector embedding" in >>>>> the google doc is probably thinking along a different set of lines. >>>>> (whoever => normally iso-9000 says documents should always have an author, >>>>> a date, and a contact point) >>>>> >> > >>>>> >> > Anyway -- vector embeddings ... lets hear the ideas... >>>>> >> > >>>>> >> > --linas >>>>> >> > >>>>> >> > -- >>>>> >> > cassette tapes - analog TV - film cameras - you >>>>> >> > >>>>> >> > -- >>>>> >> > You received this message because you are subscribed to the >>>>> Google Groups "opencog" group. >>>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to [email protected]. >>>>> >> > To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/opencog/CAHrUA37n%3D76LBMHzucTT12AMGKqrc5KEkGTspVc6z6QVfipLDg%40mail.gmail.com >>>>> . >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> Ben Goertzel, PhD >>>>> >> http://goertzel.org >>>>> >> >>>>> >> “The only people for me are the mad ones, the ones who are mad to >>>>> >> live, mad to talk, mad to be saved, desirous of everything at the >>>>> same >>>>> >> time, the ones who never yawn or say a commonplace thing, but burn, >>>>> >> burn, burn like fabulous yellow roman candles exploding like spiders >>>>> >> across the stars.” -- Jack Kerouac >>>>> >> >>>>> >> -- >>>>> >> You received this message because you are subscribed to the Google >>>>> Groups "opencog" group. >>>>> >> To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to [email protected]. >>>>> >> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/opencog/CACYTDBdbgsPf_rnct5g17sKi%3DZ91%3Dm5ByQfvOBMscYbT%2BZ50ig%40mail.gmail.com >>>>> . >>>>> > >>>>> > >>>>> > >>>>> > -- >>>>> > cassette tapes - analog TV - film cameras - you >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> Groups "opencog" group. >>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>> send an email to [email protected]. >>>>> > To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/opencog/CAHrUA34Kd8ESp29CWiZ003DcJpYfJORtJtnGVYeBApmFNz8TrQ%40mail.gmail.com >>>>> . >>>>> >>>>> >>>>> >>>>> -- >>>>> Ben Goertzel, PhD >>>>> http://goertzel.org >>>>> >>>>> “The only people for me are the mad ones, the ones who are mad to >>>>> live, mad to talk, mad to be saved, desirous of everything at the same >>>>> time, the ones who never yawn or say a commonplace thing, but burn, >>>>> burn, burn like fabulous yellow roman candles exploding like spiders >>>>> across the stars.” -- Jack Kerouac >>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "opencog" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/opencog/CACYTDBfJx9noV6-B-4Y75%2BerLp7ZF1BpsWquE%3DUqvwiiuTYcNw%40mail.gmail.com >>>>> . >>>>> >>>> >>>> >>>> -- >>>> cassette tapes - analog TV - film cameras - you >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "opencog" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/opencog/5615eb7c-d1a2-4bfc-94d7-5817fcb53e33%40googlegroups.com >>> <https://groups.google.com/d/msgid/opencog/5615eb7c-d1a2-4bfc-94d7-5817fcb53e33%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> cassette tapes - analog TV - film cameras - you >> > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/36bafe60-d803-4f66-89b5-f77a811b71dc%40googlegroups.com > <https://groups.google.com/d/msgid/opencog/36bafe60-d803-4f66-89b5-f77a811b71dc%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- cassette tapes - analog TV - film cameras - you -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAHrUA34Y%2BrQphBmJKqop3BaVJx2aZxmn1GKuVxp1SiFLV0J-dw%40mail.gmail.com.
