Dean. Your comment made me smile! This is the first time anyone has said I work at the 10,000 foot level of neuroscience! My problem is always the opposite.
The theory and models we work on are by most standards very biological. For example, I know of no other cortical theories/models that incorporate any of the following, active dendrites, differences between proximal and distal dendrites, synapse growth and decay, potential synapses, dendrite growth, depolarization as mode of prediction, mini-columns, multiple types of inhibition and their corresponding inhibitory neurons, etc. The new temporal pooling mechanism we are working on requires metabotropic receptors in the locations they are, and are not, found. Again, I don't know of any theories that have been reduced to practice that incorporate any, let alone all of these concepts. The machine learning community in general is ignorant of any of these kinds of details. One of the things I struggle with is how to describe the algorithms without the neuroscience. Most people are unfamiliar with the biological terms and concepts. But I am very happy to discuss them on this forum. Jeff From: nupic [mailto:[email protected]] On Behalf Of Dean Horak Sent: Thursday, February 06, 2014 11:31 AM To: NuPIC general mailing list. Subject: Re: [nupic-discuss] Tests for spatial pooling Great thanks. I'll do just that. Dean On Thu, Feb 6, 2014 at 1:28 PM, Matthew Taylor <[email protected]> wrote: Dean, This mailing list is entirely appropriate for those types of discussions. If you have a specific topic or question, please feel free to start a new conversation. --------- Matt Taylor OS Community Flag-Bearer Numenta On Thu, Feb 6, 2014 at 11:25 AM, Dean Horak <[email protected]> wrote: This may be a little off topic for this thread, but is there a proper place for discussions regarding neuroscience and how the CLA does or does not approach problems the way biology does? I bring this up here because while Jeff's comments are indeed correct, they are at a 10,000 ft level. It seems to me there are many insights that might well prove useful within the CLA that can be derived from discussions of neuroscience at a much deeper level. Thanks, Dean On Thu, Feb 6, 2014 at 8:30 AM, Christian Cleber Masdeval Braz <[email protected]> wrote: Kevin, i think i understood better the relation between inhibition and topology. If topology is taking into account, so the position of the input bits matter. In this case, the sub set of bits each column will "look" are not random but organized somehow. For an image, the first column will be connected to the first n bits (or some 2D portion in the beginning of the image), the second will be connected to the adjacent bits and so on. In this case, local activity and the job of inhibition makes sense to guarantee sparsity and distribution. On the other hand, if no topology exist, there is no difference in use inhibition or not. Will be ok in choosing the columns with the highest overlap values because each of them will be looking an aleatory sub set of bits spread all over the input. So, even if happens what you described, the resulting columns will have a good distribution. Regards, Christian 2014-02-06 Kevin Martin <[email protected]>: Thanks everyone, The term topology is very clear to me now. But in Fergal's explanation, he had said that "In the absence of topology, it's more efficient to just pick the highest activation potentials globally" But this means a distributed representation is not guaranteed right? Suppose we had 1000 columns and 200 input bits and desired sparsity is 1% (10 columns). Each column is potentially connected to say 75% of the input bits. Let the inhibition radius be 20 and desiredlocalactivity 5. Even though it is very unlikely, let us assume that 10 columns with the highest overlap values occur within a particular inhibition radius. That is, if the highest 11 overlap values are 125,124,122,....115 (for the sake of simplicity) in the entire layer, and the columns that have these values are ordered like this : x x 125 120 x 119 115 x 118 123 Y x x 121 122 x 116 x 117 x 124 Where x and Y are unknown overlaps. In this case, the inhibition radius of Y contains all the top 11 overlaps. Without inhibition, the columns with overlaps 125,124,123,122,121...116 will be the winners to give us the 1% sparsity. They all occur together. But with inhibition only 5 columns will be winners here and the other 5 will come from a different part of the layer. So even though just getting the columns with the highest overlaps is way more easier and efficient, will it not result in a higher chance of getting a representation that is not very distributed? Thanks, Kevin Martin Jose On Thu, Feb 6, 2014 at 6:33 AM, Fergal Byrne <[email protected]> wrote: Thanks Jeff, Perhaps the difficulty arises from the disparity between cortex (which just is as it is) and a realisation in software. Neurons can only operate locally, and their computation depends on the topology of the axons passing by them. In NuPIC, we can design both the topology of the neurons' inputs and also the inhibition strategy. I think Kevin is asking about the semantics of topology (and sparseness/distribution) in that sense. Regards, Fergal Byrne - Sent from Mailbox <https://www.dropbox.com/mailbox> for iPhone On Thu, Feb 6, 2014 at 12:52 AM, Jeff Hawkins <[email protected]> wrote: I hope to clarify what topology is. I agree with some of what Fergal wrote but disagree with other parts, so maybe we are thinking different things. In a brain, the cortex is a sheet of tissue. Everywhere you look, it is about 2mm thick and consists of 4 or 5 layers of cells (depends on how you count). The total area of human neocortex is about 1000 cm2. Cortical regions are 2D areas. Some of the regions, such as the primary visual or touch regions are large. The retina sends about a million axons to the primarily visual cortex, or V1. Each axon represents a small part of the visual space, sort of like a pixel in an image. When this bundle fibers reaches V1 the 2D topology of the axons is maintained. It is as if a 1M pixel image were projected on to the area of V1. (The "image" is inverted, split, and highly distorted, but in all cases any two adjacent areas of V1 receive input from adjacent areas of the retina.) Scientists would say V1 has a "retinotopic" orientation. Now comes the interesting part. The cells in V1 learn common patterns coming from the retina. The way the SP works is each bit in the SP tries to learn the largest spatial pattern it can. And each bit competes with the other bits. The number of patterns coming from the retina is so large that the SP bits are forced to learn only small spatial patterns that are localized on the retina. So cells on the right side of V1 learn small patterns on the left side of the retina (remember the image is inverted). In biological V1 all the learning rules are local. Inhibition between columns is restricted in radius. The axons that a cell can possibly connect to are restricted to a radius. When we say a region has topology it means that inhibition between columns is limited to a local radius and not the entire region. It means that the set of potential connections that a cell or column can make are also within a radius and not all possible input bits or all possible cells in the region. This is the way real neocortex works. When we were applying the CLA to vision-like problems we implemented it with topology. It worked as expected but it is more complicated to test. The current implementation of the CLA has no topology. This means that each column competes with all the other columns and the set of potential connections between cells is a subset of all the cells in the region. There is no ordering of the columns, nothing is "adjacent" to anything else. For applications that are small no topology works great. If the application had a large 2D sensory array then topology would likely be required. I hope that helped. Jeff From: nupic [mailto:[email protected]] On Behalf Of Fergal Byrne Sent: Wednesday, February 05, 2014 5:54 AM To: NuPIC general mailing list. Subject: Re: [nupic-discuss] Tests for spatial pooling HI Kevin, >> The following ideas are not things I think relate to "topology". Topology is the jargon we use to describe the semantic meaning (or lack of it) of a column (or input bit) position, and for connections between them. For example, a typical encoding of category data randomly picks 21 bits on out of 128. Each bit position has no real connection with the input category it encodes. This representation has no topology at all. Conversely, the sliding window of on-bits in a scalar encoding has one-dimensional topology. In this case, bit 0 might mean the input is < 10, while bit 23 might mean an input between 11 and 33. >> These ideas are things I think relate to "topology". An example of a 2d topology in an SDR is a retina-like sensor. In this case the light levels or presence of particular features in a certain spatial position would be represented by bits in the corresponding position. Topology in the SP relates to how each column position is connected with its input bits. An SP which is learning to recognise features in an SDR coming in from a retina would have columns "looking at" a set of input bits directly "below it" in the retina. The V1 primary visual cortex has exactly this property. If you have topology, you'll usually use local inhibition to pick winning columns, where local is defined according to the topology (1-d, 2-d distance). In the absence of topology, it's more efficient to just pick the highest activation potentials globally. Regards Fergal Byrne On Wed, Feb 5, 2014 at 11:25 AM, Kevin Martin <[email protected]> wrote: Hi all, Thanks for the help. But I'm not sure if I understand what you mean by topology. As far as I understand, the spatial pooler has 2 functions : Return a set of columns that are 1)sparse and 2)distributed. Sparsity can be maintained by making sure that the pooler returns only n% of columns and this criterion can be made into a test. And distribution can be tested by computing the average distance between winning columns. What is topology? Is it another way of referring to a distributed set of columns? I might need some help with local learning too. I'm afraid I have never heard the term before (its in the white paper? A quick search on the pdf did not see any hits for "local learning") Thanks, Kevin Martin On Wed, Feb 5, 2014 at 7:37 AM, Scott Purdy <[email protected]> wrote: I would go over the properties of the SP and create tests accordingly. The output, for instance, should be tolerant to spatial noise. So show an instance a set of very different spatial patterns over and over a number of times. Then show one of the patterns with a single bit flipped and make sure the output is very close (possibly identical if columns are saturated) to the same. There are a bunch of similar things you could check. Ian had a hackathon demo showing the connected bits for each column. He was able to validate different aspects of the SP visually by using different numbers of columns and % overlap. You could do something similar to make sure you see the right behavior but I don't know if you could make an automated test out of it. http://numenta.org/blog/2013/11/06/2013-fall-hackathon-outcome.html#sp_viewe r On Feb 4, 2014 11:23 AM, "Jeff Hawkins" <[email protected]> wrote: Kevin, Adding on to what Fergal said.. Here is what we did. We started with a fixed number of columns and no topology. In this situation we picked the n% most innervated columns as the output of the SP. Thus we were guaranteed to get the correct sparsity of active columns. With this setup we did a lot of other tests to verify that the SP was forming good representations. The tricky part is if you want to do this with topology and local learning rules. Is that your concern? With topology we didn't try to guarantee the entire layer of cells will have exactly n% active columns. Exact numbers are not essential. We tried several different local learning rules, all of them produced about the right level of sparsity. We than ran the other tests to make sure that local learning rules didn't cause the SP to form poor representations. You won't get exactly the same results with topology and without topology but my recollection was we were able to come up with a set of local inhibition rules that produced good results. Jeff From: nupic [mailto:[email protected]] On Behalf Of Fergal Byrne Sent: Tuesday, February 04, 2014 10:43 AM To: NuPIC general mailing list. Subject: Re: [nupic-discuss] Tests for spatial pooling Hi Kevin, If you're using no topology (and no local inhibition) then the fraction of active columns (2% in NuPIC) is the only important factor. With topology, you should choose winners relative to their neighbours. In either case, the first test is to count the fraction of active columns. In the second case, measure the average distance between active columns. Regards, Fergal Byrne On Tue, Feb 4, 2014 at 6:05 PM, Kevin Martin <[email protected]> wrote: Hi, I'm writing my own version of the CLA. I have not reached anywhere significant and I have been playing with a very small number of columns until now. I am starting by implementing a spatial pooler and spatial pooling requires a large number of columns. So far it has been easy to manually check if the synapses are distributed throughout the input, if the winning columns are really sparse etc. But it is going to be impossible when, say,I use 1000 columns. Have somebody used a test to check if the spatial pooler outputs a sparse distribution? Any suggestions on how to write a test? Thanks, Kevin Martin Jose _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org -- Fergal Byrne, Brenter IT http://inbits.com - Better Living through Thoughtful Technology e:[email protected] t:+353 83 4214179 <tel:%2B353%2083%204214179> Formerly of Adnet [email protected] http://www.adnet.ie <http://www.adnet.ie/> _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org -- Fergal Byrne, Brenter IT http://inbits.com - Better Living through Thoughtful Technology e:[email protected] t:+353 83 4214179 <tel:%2B353%2083%204214179> Formerly of Adnet [email protected] http://www.adnet.ie <http://www.adnet.ie/> _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
