This is a really good observation. Theoretical and computational models of neuroscience are hampered by the limitations in the measurement techniques available. These inability to measure fine temporal detail across a cortical area make it almost impossible to measure how neurons are learning and representing temporal sequences.
--Subutai On Wed, Oct 16, 2013 at 7:48 AM, Fergal Byrne <[email protected]>wrote: > Hi Kevin, > > Good question. I'm pretty sure that SDR is a crucial central idea in > Jeff's theory, but let's leave that aside for now. > > There's another way of looking at your question, inverting it so to speak. > Perhaps we have big brains because we need the sparseness in order to > process information the way we do. Certainly in NuPIC there seems to be a > threshold of 5-600 columns (of the otherwise typical size) in order to have > the Spatial Pooler and the Temporal Pooler work really well. Below this > size the sparseness is hard to establish, and the TP hasn't enough active > connections to operate well. Above this size the capacity is soon so high > that it's hard to bang up against it. > > The model we use in NuPIC is binary (active or not, connected or not, > etc), and timestep-based. These are simplifications of real neurons, which > have many signalling styles and which operate asynchronously. Given the > limitations of measurement of living neocortex, it's unlikely that you > could capture the true neuron-by-neuron, millisecond-by-millisecond > "network traffic" so it's hard to claim that the signalling is either very > sparse or very dense. > > The observation of neurons carrying "several signals" may be explained by > a high rate of change of input, inhibition and intra-region activity, which > could cause some overlap in the apparent state of each neuron at any given > time. The evolving shape of the signal in this case could be said to encode > part of the data. The analogue of this in HTM/CLA is a specific sequence of > SDR's, each of which could be regarded as a freeze-frame of the pattern of > activity across the region. In the CLA (unlike most other networks), the > sequences are just as important as the individual patterns. > > Looked at in this way, any fast-changing series of SDR's would appear > "dense" if measured at intervals significantly longer than the "timestep" > of the sequence. So, it's possible that these observations are not in > contradiction to the HTM/CLA theories, but are the result of the method of > measurement. > > We do know that inhibitory interneurons act to sparsify activation > patterns, and we have empirical evidence that (the computational analogue > of) this is key to getting NuPIC to work. This is why we believe the > representations are sparse. Any argument in favour of non-sparseness should > therefore have one or both of a neuroscientific and computational basis. > > Regarding orthogonality, you could view an SDR as being a multi-bit (or > fuzzy) "dense(r) orthogonal" representation, if you observe that > closely-related inputs give rise to closely-matching outputs. If you > "wiggle" the inputs one input field at a time, and combine the active bits > in the SDR's, you can see that each such set of bits represents a multi-bit > representation of some aspect of the input. Any higher-level columns > "seeing" these bits will be detecting a similar value of a similar feature. > Unlike with the other networks you mention, the "orthogonality" is going to > be learned by the SP process, rather than imposed by the structure of the > inputs. > > Regards, > > Fergal Byrne > > > > > On Wed, Oct 16, 2013 at 3:04 PM, Archie, Kevin <[email protected]>wrote: > >> I've been sitting on this question for a while, and it came to mind again >> a couple of days ago when I heard Jack Gallant talk about some work by his >> student Alex Huth. He was showing multiple simultaneous recordings from >> prefrontal cortex (I think) and each neuron was carrying several signals, >> that (paraphrasing roughly) couldn't be extracted by looking at individual >> neurons but could be teased out by extracting components from the network >> activity. (John Maunsell and Bill Newsome also gave talks that similarly >> showed single neurons firing in response to lots of things, and pulling out >> the meaning required the context of the network.) The sense I was getting: >> this is not sparse coding. >> >> In traditional neural network models (Hopfield-ish associative memories, >> perceptron networks and the like), generally what you need is not really >> sparseness but orthogonality. Sparseness is one way to get that, but it's a >> space-time tradeoff: you can often build a sparse representation quickly if >> you have plenty of space. There are other ways to get orthogonality, and a >> dense representation would be making a different tradeoff -- and big brains >> being metabolically expensive, space is a nontrivial constraint. A >> speculation I heard some years ago (and I wish I remember from whom; Google >> yields some echoes but no clear origin) is that the hippocampus and >> entorhinal cx are busy during sleep building more compact orthogonal >> representations of the day's input for use by higher association areas. >> >> Pretty clearly the sensory periphery uses sparse representations, and >> similarly for areas with really-motor motor outputs. (Extreme example: V1 >> certainly uses sparse representation. V1 is really freakin' big.) Probably >> some sparse representations persist in, say, anterior temporal, parietal, >> and frontal cx, but I would suspect that compact orthogonal representations >> would be important in higher (and smaller) areas. Of course, my suspicions >> are not evidence, I'm ten years mostly away from the neurophysiology >> literature, and data beats my speculations. Is there direct evidence that >> higher cortical areas traffic exclusively or primarily in sparse >> representations? >> >> That's the brain theory side. On the more immediately practical side: has >> anyone tried using compact orthogonal representations with NuPIC? Any >> success (or failure) stories? I don't even have a guess to what extent SDR >> is necessary versus just customary. >> >> Thanks, >> >> - Kevin >> >> p.s. Apologies for the theoretical bent of this question. Too many years >> hanging out in universities have left me tending to think too much rather >> than just getting started. >> >> ________________________________ >> >> The material in this message is private and may contain Protected >> Healthcare Information (PHI). If you are not the intended recipient, be >> advised that any unauthorized use, disclosure, copying or the taking of any >> action in reliance on the contents of this information is strictly >> prohibited. If you have received this email in error, please immediately >> notify the sender via telephone or return mail. >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> > > > > -- > > Fergal Byrne > > <http://www.examsupport.ie>Brenter IT > [email protected] +353 83 4214179 > Formerly of Adnet [email protected] http://www.adnet.ie > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
