Thanks everyone, The term topology is very clear to me now. But in Fergal's explanation, he had said that
"In the absence of topology, it's more efficient to just pick the highest activation potentials globally" But this means a distributed representation is not guaranteed right? Suppose we had 1000 columns and 200 input bits and desired sparsity is 1% (10 columns). Each column is potentially connected to say 75% of the input bits. Let the inhibition radius be 20 and desiredlocalactivity 5. Even though it is very unlikely, let us assume that 10 columns with the highest overlap values occur within a particular inhibition radius. That is, if the highest 11 overlap values are 125,124,122,....115 (for the sake of simplicity) in the entire layer, and the columns that have these values are ordered like this : x x 125 120 x 119 115 x 118 123 Y x x 121 122 x 116 x 117 x 124 Where x and Y are unknown overlaps. In this case, the inhibition radius of Y contains all the top 11 overlaps. Without inhibition, the columns with overlaps 125,124,123,122,121...116 will be the winners to give us the 1% sparsity. They all occur together. But with inhibition only 5 columns will be winners here and the other 5 will come from a different part of the layer. So even though just getting the columns with the highest overlaps is way more easier and efficient, will it not result in a higher chance of getting a representation that is not very distributed? Thanks, Kevin Martin Jose On Thu, Feb 6, 2014 at 6:33 AM, Fergal Byrne <[email protected]>wrote: > Thanks Jeff, > > Perhaps the difficulty arises from the disparity between cortex (which > just is as it is) and a realisation in software. Neurons can only operate > locally, and their computation depends on the topology of the axons passing > by them. > > In NuPIC, we can design both the topology of the neurons' inputs and also > the inhibition strategy. > > I think Kevin is asking about the semantics of topology (and > sparseness/distribution) in that sense. > > Regards, > > Fergal Byrne > -- > Sent from Mailbox <https://www.dropbox.com/mailbox> for iPhone > > > On Thu, Feb 6, 2014 at 12:52 AM, Jeff Hawkins <[email protected]>wrote: > >> I hope to clarify what topology is. I agree with some of what Fergal >> wrote but disagree with other parts, so maybe we are thinking different >> things. >> >> >> >> In a brain, the cortex is a sheet of tissue. Everywhere you look, it is >> about 2mm thick and consists of 4 or 5 layers of cells (depends on how you >> count). The total area of human neocortex is about 1000 cm2. Cortical >> regions are 2D areas. Some of the regions, such as the primary visual or >> touch regions are large. The retina sends about a million axons to the >> primarily visual cortex, or V1. Each axon represents a small part of the >> visual space, sort of like a pixel in an image. When this bundle fibers >> reaches V1 the 2D topology of the axons is maintained. It is as if a 1M >> pixel image were projected on to the area of V1. (The "image" is inverted, >> split, and highly distorted, but in all cases any two adjacent areas of V1 >> receive input from adjacent areas of the retina.) Scientists would say V1 >> has a "retinotopic" orientation. >> >> >> >> Now comes the interesting part. The cells in V1 learn common patterns >> coming from the retina. The way the SP works is each bit in the SP tries >> to learn the largest spatial pattern it can. And each bit competes with >> the other bits. The number of patterns coming from the retina is so large >> that the SP bits are forced to learn only small spatial patterns that are >> localized on the retina. So cells on the right side of V1 learn small >> patterns on the left side of the retina (remember the image is inverted). >> In biological V1 all the learning rules are local. Inhibition between >> columns is restricted in radius. The axons that a cell can possibly >> connect to are restricted to a radius. >> >> >> >> When we say a region has topology it means that inhibition between >> columns is limited to a local radius and not the entire region. It means >> that the set of potential connections that a cell or column can make are >> also within a radius and not all possible input bits or all possible cells >> in the region. This is the way real neocortex works. When we were >> applying the CLA to vision-like problems we implemented it with topology. >> It worked as expected but it is more complicated to test. The current >> implementation of the CLA has no topology. This means that each column >> competes with all the other columns and the set of potential connections >> between cells is a subset of all the cells in the region. There is no >> ordering of the columns, nothing is "adjacent" to anything else. For >> applications that are small no topology works great. If the application >> had a large 2D sensory array then topology would likely be required. >> >> >> >> I hope that helped. >> >> Jeff >> >> >> >> *From:* nupic [mailto:[email protected]] *On Behalf Of *Fergal >> Byrne >> *Sent:* Wednesday, February 05, 2014 5:54 AM >> *To:* NuPIC general mailing list. >> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >> >> >> >> HI Kevin, >> >> >> >> >> The following ideas are not things I think relate to "topology". >> >> >> >> Topology is the jargon we use to describe the semantic meaning (or lack >> of it) of a column (or input bit) position, and for connections between >> them. >> >> For example, a typical encoding of category data randomly picks 21 bits >> on out of 128. Each bit position has no real connection with the input >> category it encodes. This representation has no topology at all. >> >> Conversely, the sliding window of on-bits in a scalar encoding has >> one-dimensional topology. In this case, bit 0 might mean the input is < 10, >> while bit 23 might mean an input between 11 and 33. >> >> >> >> >> These ideas are things I think relate to "topology". >> >> >> >> An example of a 2d topology in an SDR is a retina-like sensor. In this >> case the light levels or presence of particular features in a certain >> spatial position would be represented by bits in the corresponding position. >> >> Topology in the SP relates to how each column position is connected >> with its input bits. An SP which is learning to recognise features in an >> SDR coming in from a retina would have columns "looking at" a set of input >> bits directly "below it" in the retina. The V1 primary visual cortex has >> exactly this property. >> >> If you have topology, you'll usually use local inhibition to pick >> winning columns, where local is defined according to the topology (1-d, 2-d >> distance). In the absence of topology, it's more efficient to just pick the >> highest activation potentials globally. >> >> >> >> Regards >> >> >> >> Fergal Byrne >> >> >> >> >> >> >> >> On Wed, Feb 5, 2014 at 11:25 AM, Kevin Martin < >> [email protected]> wrote: >> >> Hi all, >> >> Thanks for the help. But I'm not sure if I understand what you mean by >> topology. As far as I understand, the spatial pooler has 2 functions : >> Return a set of columns that are 1)sparse and 2)distributed. >> >> Sparsity can be maintained by making sure that the pooler returns only >> n% of columns and this criterion can be made into a test. And distribution >> can be tested by computing the average distance between winning columns. >> What is topology? Is it another way of referring to a distributed set of >> columns? I might need some help with local learning too. I'm afraid I have >> never heard the term before (its in the white paper? A quick search on the >> pdf did not see any hits for "local learning") >> >> Thanks, >> >> Kevin Martin >> >> >> >> On Wed, Feb 5, 2014 at 7:37 AM, Scott Purdy <[email protected]> wrote: >> >> I would go over the properties of the SP and create tests accordingly. >> The output, for instance, should be tolerant to spatial noise. So show an >> instance a set of very different spatial patterns over and over a number of >> times. Then show one of the patterns with a single bit flipped and make >> sure the output is very close (possibly identical if columns are saturated) >> to the same. >> >> There are a bunch of similar things you could check. >> >> Ian had a hackathon demo showing the connected bits for each column. He >> was able to validate different aspects of the SP visually by using >> different numbers of columns and % overlap. >> >> You could do something similar to make sure you see the right behavior >> but I don't know if you could make an automated test out of it. >> >> >> http://numenta.org/blog/2013/11/06/2013-fall-hackathon-outcome.html#sp_viewer >> >> On Feb 4, 2014 11:23 AM, "Jeff Hawkins" <[email protected]> wrote: >> >> Kevin, >> >> Adding on to what Fergal said.... >> >> >> >> Here is what we did. We started with a fixed number of columns and no >> topology. In this situation we picked the n% most innervated columns as >> the output of the SP. Thus we were *guaranteed* to get the correct >> sparsity of active columns. With this setup we did a lot of other tests to >> verify that the SP was forming good representations. >> >> >> >> The tricky part is if you want to do this with topology and local >> learning rules. Is that your concern? >> >> >> >> With topology we didn't try to guarantee the entire layer of cells will >> have exactly n% active columns. Exact numbers are not essential. We tried >> several different local learning rules, all of them produced about the >> right level of sparsity. We than ran the other tests to make sure that >> local learning rules didn't cause the SP to form poor representations. You >> won't get exactly the same results with topology and without topology but >> my recollection was we were able to come up with a set of local inhibition >> rules that produced good results. >> >> Jeff >> >> >> >> *From:* nupic [mailto:[email protected]] *On Behalf Of *Fergal >> Byrne >> *Sent:* Tuesday, February 04, 2014 10:43 AM >> *To:* NuPIC general mailing list. >> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >> >> >> >> Hi Kevin, >> >> >> >> If you're using no topology (and no local inhibition) then the fraction >> of active columns (2% in NuPIC) is the only important factor. With >> topology, you should choose winners relative to their neighbours. In either >> case, the first test is to count the fraction of active columns. In the >> second case, measure the average distance between active columns. >> >> >> >> Regards, >> >> >> >> Fergal Byrne >> >> >> >> On Tue, Feb 4, 2014 at 6:05 PM, Kevin Martin < >> [email protected]> wrote: >> >> Hi, >> >> I'm writing my own version of the CLA. I have not reached anywhere >> significant and I have been playing with a very small number of columns >> until now. I am starting by implementing a spatial pooler and spatial >> pooling requires a large number of columns. So far it has been easy to >> manually check if the synapses are distributed throughout the input, if the >> winning columns are really sparse etc. But it is going to be impossible >> when, say,I use 1000 columns. Have somebody used a test to check if the >> spatial pooler outputs a sparse distribution? Any suggestions on how to >> write a test? >> >> Thanks, >> >> Kevin Martin Jose >> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >> >> >> >> -- >> >> >> Fergal Byrne, Brenter IT >> >> >> >> http://inbits.com - Better Living through Thoughtful Technology >> >> >> >> e:[email protected] t:+353 83 4214179 >> >> Formerly of Adnet [email protected] http://www.adnet.ie >> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> >> >> >> >> -- >> >> >> Fergal Byrne, Brenter IT >> >> >> >> http://inbits.com - Better Living through Thoughtful Technology >> >> >> >> e:[email protected] t:+353 83 4214179 >> >> Formerly of Adnet [email protected] http://www.adnet.ie >> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
