Hi Kevin, Thanks for the thoughtful note. I was just about to reply with the same observation you mentioned. Sparse representations are not mathematically orthogonal in general. Some of the main work done in this area from a mathematical standpoint were a set of Olshausen and Field papers on sparse coding. (See below for one of them.)
Kanerva also has a great set of papers on high dimensional sparse representations and their properties. One of the key properties exploited by the temporal pooler is the ability to combine multiple bit patterns into the same fixed vector. You can, with very high probability, tell if any of the original patterns are contained or not in the combined vector. As Fergal pointed out in the other email, you must have a high enough dimensionality for this to work. Bloom filters also rely on something similar. I don't think this property is possible with orthogonal representations. (I don't have a proof, this is just a conjecture.) BTW, I'm pretty sure there is a large body of evidence supporting sparse representations everywhere in the cortex. --Subutai Sparse Coding with an Overcomplete Basis Set: A Strategy Employed by V1? Olshausen and Field, 1997 Vision Res., Vol 37, No 23. pp 3311-3325 PDF: http://www.chaos.gwdg.de/~michael/CNS_course_2004/papers_max/OlshausenFieldVR97.pdf On Wed, Oct 16, 2013 at 7:50 AM, Archie, Kevin <[email protected]>wrote: > On reflection, I think that the sort of compact representations I'm > talking about aren't really orthogonal in the algebraic sense so much as > sufficiently distinct (and I can elaborate on that if it's still unclear). > I suspect that any strictly orthogonal representation is isomorphic to a > sparse representation and so there's no space advantage to strictly > orthogonal vs. sparse. I've heard the word "orthogonalize" applied to the > process of building sufficiently-distinct representations, and so that's > the word I pulled out. Sorry for any confusion I may have caused by being > imprecise. > > - Kevin > > On Oct 16, 2013, at 9:04 AM, Archie, Kevin wrote: > > > I've been sitting on this question for a while, and it came to mind > again a couple of days ago when I heard Jack Gallant talk about some work > by his student Alex Huth. He was showing multiple simultaneous recordings > from prefrontal cortex (I think) and each neuron was carrying several > signals, that (paraphrasing roughly) couldn't be extracted by looking at > individual neurons but could be teased out by extracting components from > the network activity. (John Maunsell and Bill Newsome also gave talks that > similarly showed single neurons firing in response to lots of things, and > pulling out the meaning required the context of the network.) The sense I > was getting: this is not sparse coding. > > > > In traditional neural network models (Hopfield-ish associative memories, > perceptron networks and the like), generally what you need is not really > sparseness but orthogonality. Sparseness is one way to get that, but it's a > space-time tradeoff: you can often build a sparse representation quickly if > you have plenty of space. There are other ways to get orthogonality, and a > dense representation would be making a different tradeoff -- and big brains > being metabolically expensive, space is a nontrivial constraint. A > speculation I heard some years ago (and I wish I remember from whom; Google > yields some echoes but no clear origin) is that the hippocampus and > entorhinal cx are busy during sleep building more compact orthogonal > representations of the day's input for use by higher association areas. > > > > Pretty clearly the sensory periphery uses sparse representations, and > similarly for areas with really-motor motor outputs. (Extreme example: V1 > certainly uses sparse representation. V1 is really freakin' big.) Probably > some sparse representations persist in, say, anterior temporal, parietal, > and frontal cx, but I would suspect that compact orthogonal representations > would be important in higher (and smaller) areas. Of course, my suspicions > are not evidence, I'm ten years mostly away from the neurophysiology > literature, and data beats my speculations. Is there direct evidence that > higher cortical areas traffic exclusively or primarily in sparse > representations? > > > > That's the brain theory side. On the more immediately practical side: > has anyone tried using compact orthogonal representations with NuPIC? Any > success (or failure) stories? I don't even have a guess to what extent SDR > is necessary versus just customary. > > > > Thanks, > > > > - Kevin > > > > p.s. Apologies for the theoretical bent of this question. Too many years > hanging out in universities have left me tending to think too much rather > than just getting started. > > > > ________________________________ > > > > The material in this message is private and may contain Protected > Healthcare Information (PHI). If you are not the intended recipient, be > advised that any unauthorized use, disclosure, copying or the taking of any > action in reliance on the contents of this information is strictly > prohibited. If you have received this email in error, please immediately > notify the sender via telephone or return mail. > > > > _______________________________________________ > > nupic mailing list > > [email protected] > > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
