Thanks Ian, for your ideas. Now I'm curious as to what would happen if you trained two different HTM networks with the same data, and with a lot of it. Since they're randomly initialized, I imagine that the positions of patterns in the encodings would differ, but the relative existence of those patterns would be analogous between the networks. If this is true, then the problem becomes, can you identify and transfer these patterns between networks?
I agree it sounds like a very challenging problem :) On Fri, Sep 13, 2013 at 4:49 PM, Ian Danforth <[email protected]>wrote: > Chetan, > > This is a really hard problem, but there are a couple of datapoints that > give me hope. > > 1. Neural networks trained on natural image scenes end up with gabor > filters > > Many many different techniques for doing autoencoders end up with gabor > like filters. And if you use the same techniques on different classes of > natural images you still get gabor like filters, in similar if not > perfectly aligned shapes / proportions. > > 2. There is a great deal of similarity in activations areas in the human > brain > > The general map of activity for certain perceptions and actions in the > brain is very similar between people. There is a lot of variation around > the edges of regions, but you can rely on some consistency in nearly every > brain area. > > So what do we do with this information? Well my suspicion is that the > statement "the internal connections between neurons don't translate > between models" will turn out to be practically false. > > If the general characteristics of the experience are shared between two > models then every layer of their representations will be analogous. > > In CLA especially if the SP states are shared between two networks > (pretrained) I think updating the TP weights from one to the other could > work quite well. > > I'd love to see a tool that can 'diff' two networks so that assumptions > like this could be evaluated. > > Ian > > > On Fri, Sep 13, 2013 at 4:18 PM, Chetan Surpur <[email protected]> wrote: > >> Anyone have any ideas about this? It's been something I've been curious >> about for a while now, and just keeps popping into my head :) >> >> >> On Thursday, August 29, 2013, Chetan Surpur wrote: >> >>> Hello everyone, >>> >>> I've been wondering if it's possible to transfer knowledge from one >>> trained HTM network to another. >>> >>> For instance, let's say there's a trained language model on every user's >>> phone, and there's a global language model on the cloud. The distributed >>> client models were initially copies of the cloud model but further trained >>> on the user's own data, thus personalizing them. Then, you train the cloud >>> model with more public textual training data, and it learns new patterns >>> (new vocabulary, new phrases, etc.). What would be the best way to transfer >>> the new knowledge from the cloud model to the client models? >>> >>> Since the internal connections between neurons don't translate between >>> models, I imagine that only the externally facing layers (the input and >>> output layers) are useful in transferring data. So then one way would be to >>> have the cloud model generate patterns at the output layer, and feed that >>> to the client model's input layer. Kind of like the cloud model is >>> "talking", and the client model is "listening". After all, this is the only >>> effective way to transfer knowledge between humans, since we can't connect >>> our brains to each other directly. But it's at least faster than training >>> the client models directly on the raw training data, because the cloud >>> model can compress the patterns and transfer them more efficiently. >>> >>> That's just one idea, and I'm not even sure how exactly that would work. >>> I pretty much just thought of it analogous to human communication. Are >>> there better ways with HTMs? >>> >>> Thanks, >>> Chetan >>> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
