I've been sitting on this question for a while, and it came to mind again a couple of days ago when I heard Jack Gallant talk about some work by his student Alex Huth. He was showing multiple simultaneous recordings from prefrontal cortex (I think) and each neuron was carrying several signals, that (paraphrasing roughly) couldn't be extracted by looking at individual neurons but could be teased out by extracting components from the network activity. (John Maunsell and Bill Newsome also gave talks that similarly showed single neurons firing in response to lots of things, and pulling out the meaning required the context of the network.) The sense I was getting: this is not sparse coding.
In traditional neural network models (Hopfield-ish associative memories, perceptron networks and the like), generally what you need is not really sparseness but orthogonality. Sparseness is one way to get that, but it's a space-time tradeoff: you can often build a sparse representation quickly if you have plenty of space. There are other ways to get orthogonality, and a dense representation would be making a different tradeoff -- and big brains being metabolically expensive, space is a nontrivial constraint. A speculation I heard some years ago (and I wish I remember from whom; Google yields some echoes but no clear origin) is that the hippocampus and entorhinal cx are busy during sleep building more compact orthogonal representations of the day's input for use by higher association areas. Pretty clearly the sensory periphery uses sparse representations, and similarly for areas with really-motor motor outputs. (Extreme example: V1 certainly uses sparse representation. V1 is really freakin' big.) Probably some sparse representations persist in, say, anterior temporal, parietal, and frontal cx, but I would suspect that compact orthogonal representations would be important in higher (and smaller) areas. Of course, my suspicions are not evidence, I'm ten years mostly away from the neurophysiology literature, and data beats my speculations. Is there direct evidence that higher cortical areas traffic exclusively or primarily in sparse representations? That's the brain theory side. On the more immediately practical side: has anyone tried using compact orthogonal representations with NuPIC? Any success (or failure) stories? I don't even have a guess to what extent SDR is necessary versus just customary. Thanks, - Kevin p.s. Apologies for the theoretical bent of this question. Too many years hanging out in universities have left me tending to think too much rather than just getting started. ________________________________ The material in this message is private and may contain Protected Healthcare Information (PHI). If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
