On reflection, I think that the sort of compact representations I'm talking about aren't really orthogonal in the algebraic sense so much as sufficiently distinct (and I can elaborate on that if it's still unclear). I suspect that any strictly orthogonal representation is isomorphic to a sparse representation and so there's no space advantage to strictly orthogonal vs. sparse. I've heard the word "orthogonalize" applied to the process of building sufficiently-distinct representations, and so that's the word I pulled out. Sorry for any confusion I may have caused by being imprecise.
- Kevin On Oct 16, 2013, at 9:04 AM, Archie, Kevin wrote: > I've been sitting on this question for a while, and it came to mind again a > couple of days ago when I heard Jack Gallant talk about some work by his > student Alex Huth. He was showing multiple simultaneous recordings from > prefrontal cortex (I think) and each neuron was carrying several signals, > that (paraphrasing roughly) couldn't be extracted by looking at individual > neurons but could be teased out by extracting components from the network > activity. (John Maunsell and Bill Newsome also gave talks that similarly > showed single neurons firing in response to lots of things, and pulling out > the meaning required the context of the network.) The sense I was getting: > this is not sparse coding. > > In traditional neural network models (Hopfield-ish associative memories, > perceptron networks and the like), generally what you need is not really > sparseness but orthogonality. Sparseness is one way to get that, but it's a > space-time tradeoff: you can often build a sparse representation quickly if > you have plenty of space. There are other ways to get orthogonality, and a > dense representation would be making a different tradeoff -- and big brains > being metabolically expensive, space is a nontrivial constraint. A > speculation I heard some years ago (and I wish I remember from whom; Google > yields some echoes but no clear origin) is that the hippocampus and > entorhinal cx are busy during sleep building more compact orthogonal > representations of the day's input for use by higher association areas. > > Pretty clearly the sensory periphery uses sparse representations, and > similarly for areas with really-motor motor outputs. (Extreme example: V1 > certainly uses sparse representation. V1 is really freakin' big.) Probably > some sparse representations persist in, say, anterior temporal, parietal, and > frontal cx, but I would suspect that compact orthogonal representations would > be important in higher (and smaller) areas. Of course, my suspicions are not > evidence, I'm ten years mostly away from the neurophysiology literature, and > data beats my speculations. Is there direct evidence that higher cortical > areas traffic exclusively or primarily in sparse representations? > > That's the brain theory side. On the more immediately practical side: has > anyone tried using compact orthogonal representations with NuPIC? Any success > (or failure) stories? I don't even have a guess to what extent SDR is > necessary versus just customary. > > Thanks, > > - Kevin > > p.s. Apologies for the theoretical bent of this question. Too many years > hanging out in universities have left me tending to think too much rather > than just getting started. > > ________________________________ > > The material in this message is private and may contain Protected Healthcare > Information (PHI). If you are not the intended recipient, be advised that any > unauthorized use, disclosure, copying or the taking of any action in reliance > on the contents of this information is strictly prohibited. If you have > received this email in error, please immediately notify the sender via > telephone or return mail. > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
