This may be a little off topic for this thread, but is there a proper place for discussions regarding neuroscience and how the CLA does or does not approach problems the way biology does? I bring this up here because while Jeff's comments are indeed correct, they are at a 10,000 ft level. It seems to me there are many insights that might well prove useful within the CLA that can be derived from discussions of neuroscience at a much deeper level.
Thanks, Dean On Thu, Feb 6, 2014 at 8:30 AM, Christian Cleber Masdeval Braz < [email protected]> wrote: > > Kevin, i think i understood better the relation between inhibition and > topology. If topology is taking into account, so the position of the input > bits matter. In this case, the sub set of bits each column will "look" are > not random but organized somehow. For an image, the first column will be > connected to the first n bits (or some 2D portion in the beginning of the > image), the second will be connected to the adjacent bits and so on. In > this case, local activity and the job of inhibition makes sense to > guarantee sparsity and distribution. > > On the other hand, if no topology exist, there is no difference in use > inhibition or not. Will be ok in choosing the columns with the highest > overlap values because each of them will be looking an aleatory sub set of > bits spread all over the input. So, even if happens what you described, the > resulting columns will have a good distribution. > > > Regards, > > Christian > > > > > 2014-02-06 Kevin Martin <[email protected]>: > > Thanks everyone, >> >> The term topology is very clear to me now. But in Fergal's explanation, >> he had said that >> >> "In the absence of topology, it's more efficient to just pick the highest >> activation potentials globally" >> >> But this means a distributed representation is not guaranteed right? >> Suppose we had 1000 columns and 200 input bits and desired sparsity is 1% >> (10 columns). Each column is potentially connected to say 75% of the input >> bits. Let the inhibition radius be 20 and desiredlocalactivity 5. Even >> though it is very unlikely, let us assume that 10 columns with the highest >> overlap values occur within a particular inhibition radius. >> >> That is, if the highest 11 overlap values are 125,124,122,....115 (for >> the sake of simplicity) in the entire layer, and the columns that have >> these values are ordered like this : >> >> x x 125 120 x 119 115 x 118 123 Y x x 121 122 x 116 x 117 x 124 >> >> Where x and Y are unknown overlaps. In this case, the inhibition radius >> of Y contains all the top 11 overlaps. Without inhibition, the columns with >> overlaps 125,124,123,122,121...116 will be the winners to give us the 1% >> sparsity. They all occur together. But with inhibition only 5 columns will >> be winners here and the other 5 will come from a different part of the >> layer. So even though just getting the columns with the highest overlaps is >> way more easier and efficient, will it not result in a higher chance of >> getting a representation that is not very distributed? >> >> Thanks, >> >> Kevin Martin Jose >> >> >> On Thu, Feb 6, 2014 at 6:33 AM, Fergal Byrne <[email protected] >> > wrote: >> >>> Thanks Jeff, >>> >>> Perhaps the difficulty arises from the disparity between cortex (which >>> just is as it is) and a realisation in software. Neurons can only operate >>> locally, and their computation depends on the topology of the axons passing >>> by them. >>> >>> In NuPIC, we can design both the topology of the neurons' inputs and >>> also the inhibition strategy. >>> >>> I think Kevin is asking about the semantics of topology (and >>> sparseness/distribution) in that sense. >>> >>> Regards, >>> >>> Fergal Byrne >>> -- >>> Sent from Mailbox <https://www.dropbox.com/mailbox> for iPhone >>> >>> >>> On Thu, Feb 6, 2014 at 12:52 AM, Jeff Hawkins <[email protected]>wrote: >>> >>>> I hope to clarify what topology is. I agree with some of what Fergal >>>> wrote but disagree with other parts, so maybe we are thinking different >>>> things. >>>> >>>> >>>> >>>> In a brain, the cortex is a sheet of tissue. Everywhere you look, it is >>>> about 2mm thick and consists of 4 or 5 layers of cells (depends on how you >>>> count). The total area of human neocortex is about 1000 cm2. Cortical >>>> regions are 2D areas. Some of the regions, such as the primary visual or >>>> touch regions are large. The retina sends about a million axons to the >>>> primarily visual cortex, or V1. Each axon represents a small part of the >>>> visual space, sort of like a pixel in an image. When this bundle fibers >>>> reaches V1 the 2D topology of the axons is maintained. It is as if a 1M >>>> pixel image were projected on to the area of V1. (The "image" is inverted, >>>> split, and highly distorted, but in all cases any two adjacent areas of V1 >>>> receive input from adjacent areas of the retina.) Scientists would say V1 >>>> has a "retinotopic" orientation. >>>> >>>> >>>> >>>> Now comes the interesting part. The cells in V1 learn common patterns >>>> coming from the retina. The way the SP works is each bit in the SP tries >>>> to learn the largest spatial pattern it can. And each bit competes with >>>> the other bits. The number of patterns coming from the retina is so large >>>> that the SP bits are forced to learn only small spatial patterns that are >>>> localized on the retina. So cells on the right side of V1 learn small >>>> patterns on the left side of the retina (remember the image is inverted). >>>> In biological V1 all the learning rules are local. Inhibition between >>>> columns is restricted in radius. The axons that a cell can possibly >>>> connect to are restricted to a radius. >>>> >>>> >>>> >>>> When we say a region has topology it means that inhibition between >>>> columns is limited to a local radius and not the entire region. It means >>>> that the set of potential connections that a cell or column can make are >>>> also within a radius and not all possible input bits or all possible cells >>>> in the region. This is the way real neocortex works. When we were >>>> applying the CLA to vision-like problems we implemented it with topology. >>>> It worked as expected but it is more complicated to test. The current >>>> implementation of the CLA has no topology. This means that each column >>>> competes with all the other columns and the set of potential connections >>>> between cells is a subset of all the cells in the region. There is no >>>> ordering of the columns, nothing is "adjacent" to anything else. For >>>> applications that are small no topology works great. If the application >>>> had a large 2D sensory array then topology would likely be required. >>>> >>>> >>>> >>>> I hope that helped. >>>> >>>> Jeff >>>> >>>> >>>> >>>> *From:* nupic [mailto:[email protected]] *On Behalf Of >>>> *Fergal >>>> Byrne >>>> *Sent:* Wednesday, February 05, 2014 5:54 AM >>>> *To:* NuPIC general mailing list. >>>> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >>>> >>>> >>>> >>>> HI Kevin, >>>> >>>> >>>> >>>> >> The following ideas are not things I think relate to "topology". >>>> >>>> >>>> >>>> Topology is the jargon we use to describe the semantic meaning (or lack >>>> of it) of a column (or input bit) position, and for connections between >>>> them. >>>> >>>> For example, a typical encoding of category data randomly picks 21 >>>> bits on out of 128. Each bit position has no real connection with the input >>>> category it encodes. This representation has no topology at all. >>>> >>>> Conversely, the sliding window of on-bits in a scalar encoding has >>>> one-dimensional topology. In this case, bit 0 might mean the input is < 10, >>>> while bit 23 might mean an input between 11 and 33. >>>> >>>> >>>> >>>> >> These ideas are things I think relate to "topology". >>>> >>>> >>>> >>>> An example of a 2d topology in an SDR is a retina-like sensor. In this >>>> case the light levels or presence of particular features in a certain >>>> spatial position would be represented by bits in the corresponding >>>> position. >>>> >>>> Topology in the SP relates to how each column position is connected >>>> with its input bits. An SP which is learning to recognise features in an >>>> SDR coming in from a retina would have columns "looking at" a set of input >>>> bits directly "below it" in the retina. The V1 primary visual cortex has >>>> exactly this property. >>>> >>>> If you have topology, you'll usually use local inhibition to pick >>>> winning columns, where local is defined according to the topology (1-d, 2-d >>>> distance). In the absence of topology, it's more efficient to just pick the >>>> highest activation potentials globally. >>>> >>>> >>>> >>>> Regards >>>> >>>> >>>> >>>> Fergal Byrne >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, Feb 5, 2014 at 11:25 AM, Kevin Martin < >>>> [email protected]> wrote: >>>> >>>> Hi all, >>>> >>>> Thanks for the help. But I'm not sure if I understand what you mean >>>> by topology. As far as I understand, the spatial pooler has 2 functions : >>>> Return a set of columns that are 1)sparse and 2)distributed. >>>> >>>> Sparsity can be maintained by making sure that the pooler returns >>>> only n% of columns and this criterion can be made into a test. And >>>> distribution can be tested by computing the average distance between >>>> winning columns. What is topology? Is it another way of referring to a >>>> distributed set of columns? I might need some help with local learning too. >>>> I'm afraid I have never heard the term before (its in the white paper? A >>>> quick search on the pdf did not see any hits for "local learning") >>>> >>>> Thanks, >>>> >>>> Kevin Martin >>>> >>>> >>>> >>>> On Wed, Feb 5, 2014 at 7:37 AM, Scott Purdy <[email protected]> wrote: >>>> >>>> I would go over the properties of the SP and create tests accordingly. >>>> The output, for instance, should be tolerant to spatial noise. So show an >>>> instance a set of very different spatial patterns over and over a number of >>>> times. Then show one of the patterns with a single bit flipped and make >>>> sure the output is very close (possibly identical if columns are saturated) >>>> to the same. >>>> >>>> There are a bunch of similar things you could check. >>>> >>>> Ian had a hackathon demo showing the connected bits for each column. He >>>> was able to validate different aspects of the SP visually by using >>>> different numbers of columns and % overlap. >>>> >>>> You could do something similar to make sure you see the right behavior >>>> but I don't know if you could make an automated test out of it. >>>> >>>> >>>> http://numenta.org/blog/2013/11/06/2013-fall-hackathon-outcome.html#sp_viewer >>>> >>>> On Feb 4, 2014 11:23 AM, "Jeff Hawkins" <[email protected]> wrote: >>>> >>>> Kevin, >>>> >>>> Adding on to what Fergal said.... >>>> >>>> >>>> >>>> Here is what we did. We started with a fixed number of columns and no >>>> topology. In this situation we picked the n% most innervated columns as >>>> the output of the SP. Thus we were *guaranteed* to get the correct >>>> sparsity of active columns. With this setup we did a lot of other tests to >>>> verify that the SP was forming good representations. >>>> >>>> >>>> >>>> The tricky part is if you want to do this with topology and local >>>> learning rules. Is that your concern? >>>> >>>> >>>> >>>> With topology we didn't try to guarantee the entire layer of cells will >>>> have exactly n% active columns. Exact numbers are not essential. We tried >>>> several different local learning rules, all of them produced about the >>>> right level of sparsity. We than ran the other tests to make sure that >>>> local learning rules didn't cause the SP to form poor representations. You >>>> won't get exactly the same results with topology and without topology but >>>> my recollection was we were able to come up with a set of local inhibition >>>> rules that produced good results. >>>> >>>> Jeff >>>> >>>> >>>> >>>> *From:* nupic [mailto:[email protected]] *On Behalf Of >>>> *Fergal >>>> Byrne >>>> *Sent:* Tuesday, February 04, 2014 10:43 AM >>>> *To:* NuPIC general mailing list. >>>> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >>>> >>>> >>>> >>>> Hi Kevin, >>>> >>>> >>>> >>>> If you're using no topology (and no local inhibition) then the >>>> fraction of active columns (2% in NuPIC) is the only important factor. With >>>> topology, you should choose winners relative to their neighbours. In either >>>> case, the first test is to count the fraction of active columns. In the >>>> second case, measure the average distance between active columns. >>>> >>>> >>>> >>>> Regards, >>>> >>>> >>>> >>>> Fergal Byrne >>>> >>>> >>>> >>>> On Tue, Feb 4, 2014 at 6:05 PM, Kevin Martin < >>>> [email protected]> wrote: >>>> >>>> Hi, >>>> >>>> I'm writing my own version of the CLA. I have not reached anywhere >>>> significant and I have been playing with a very small number of columns >>>> until now. I am starting by implementing a spatial pooler and spatial >>>> pooling requires a large number of columns. So far it has been easy to >>>> manually check if the synapses are distributed throughout the input, if the >>>> winning columns are really sparse etc. But it is going to be impossible >>>> when, say,I use 1000 columns. Have somebody used a test to check if the >>>> spatial pooler outputs a sparse distribution? Any suggestions on how to >>>> write a test? >>>> >>>> Thanks, >>>> >>>> Kevin Martin Jose >>>> >>>> >>>> _______________________________________________ >>>> nupic mailing list >>>> [email protected] >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> Fergal Byrne, Brenter IT >>>> >>>> >>>> >>>> http://inbits.com - Better Living through Thoughtful Technology >>>> >>>> >>>> >>>> e:[email protected] t:+353 83 4214179 >>>> >>>> Formerly of Adnet [email protected] http://www.adnet.ie >>>> >>>> >>>> _______________________________________________ >>>> nupic mailing list >>>> [email protected] >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>> >>>> >>>> _______________________________________________ >>>> nupic mailing list >>>> [email protected] >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> nupic mailing list >>>> [email protected] >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> Fergal Byrne, Brenter IT >>>> >>>> >>>> >>>> http://inbits.com - Better Living through Thoughtful Technology >>>> >>>> >>>> >>>> e:[email protected] t:+353 83 4214179 >>>> >>>> Formerly of Adnet [email protected] http://www.adnet.ie >>>> >>>> >>> >>> _______________________________________________ >>> nupic mailing list >>> [email protected] >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
