Great thanks. I'll do just that. Dean
On Thu, Feb 6, 2014 at 1:28 PM, Matthew Taylor <[email protected]> wrote: > Dean, > > This mailing list is entirely appropriate for those types of discussions. > If you have a specific topic or question, please feel free to start a new > conversation. > > --------- > Matt Taylor > OS Community Flag-Bearer > Numenta > > > On Thu, Feb 6, 2014 at 11:25 AM, Dean Horak <[email protected]> wrote: > >> This may be a little off topic for this thread, but is there a proper >> place for discussions regarding neuroscience and how the CLA does or does >> not approach problems the way biology does? I bring this up here because >> while Jeff's comments are indeed correct, they are at a 10,000 ft level. It >> seems to me there are many insights that might well prove useful within the >> CLA that can be derived from discussions of neuroscience at a much deeper >> level. >> >> Thanks, >> Dean >> >> >> On Thu, Feb 6, 2014 at 8:30 AM, Christian Cleber Masdeval Braz < >> [email protected]> wrote: >> >>> >>> Kevin, i think i understood better the relation between inhibition and >>> topology. If topology is taking into account, so the position of the input >>> bits matter. In this case, the sub set of bits each column will "look" are >>> not random but organized somehow. For an image, the first column will be >>> connected to the first n bits (or some 2D portion in the beginning of the >>> image), the second will be connected to the adjacent bits and so on. In >>> this case, local activity and the job of inhibition makes sense to >>> guarantee sparsity and distribution. >>> >>> On the other hand, if no topology exist, there is no difference in use >>> inhibition or not. Will be ok in choosing the columns with the highest >>> overlap values because each of them will be looking an aleatory sub set of >>> bits spread all over the input. So, even if happens what you described, the >>> resulting columns will have a good distribution. >>> >>> >>> Regards, >>> >>> Christian >>> >>> >>> >>> >>> 2014-02-06 Kevin Martin <[email protected]>: >>> >>> Thanks everyone, >>>> >>>> The term topology is very clear to me now. But in Fergal's explanation, >>>> he had said that >>>> >>>> "In the absence of topology, it's more efficient to just pick the >>>> highest activation potentials globally" >>>> >>>> But this means a distributed representation is not guaranteed right? >>>> Suppose we had 1000 columns and 200 input bits and desired sparsity is 1% >>>> (10 columns). Each column is potentially connected to say 75% of the input >>>> bits. Let the inhibition radius be 20 and desiredlocalactivity 5. Even >>>> though it is very unlikely, let us assume that 10 columns with the highest >>>> overlap values occur within a particular inhibition radius. >>>> >>>> That is, if the highest 11 overlap values are 125,124,122,....115 (for >>>> the sake of simplicity) in the entire layer, and the columns that have >>>> these values are ordered like this : >>>> >>>> x x 125 120 x 119 115 x 118 123 Y x x 121 122 x 116 x 117 x 124 >>>> >>>> Where x and Y are unknown overlaps. In this case, the inhibition radius >>>> of Y contains all the top 11 overlaps. Without inhibition, the columns with >>>> overlaps 125,124,123,122,121...116 will be the winners to give us the 1% >>>> sparsity. They all occur together. But with inhibition only 5 columns will >>>> be winners here and the other 5 will come from a different part of the >>>> layer. So even though just getting the columns with the highest overlaps is >>>> way more easier and efficient, will it not result in a higher chance of >>>> getting a representation that is not very distributed? >>>> >>>> Thanks, >>>> >>>> Kevin Martin Jose >>>> >>>> >>>> On Thu, Feb 6, 2014 at 6:33 AM, Fergal Byrne < >>>> [email protected]> wrote: >>>> >>>>> Thanks Jeff, >>>>> >>>>> Perhaps the difficulty arises from the disparity between cortex (which >>>>> just is as it is) and a realisation in software. Neurons can only operate >>>>> locally, and their computation depends on the topology of the axons >>>>> passing >>>>> by them. >>>>> >>>>> In NuPIC, we can design both the topology of the neurons' inputs and >>>>> also the inhibition strategy. >>>>> >>>>> I think Kevin is asking about the semantics of topology (and >>>>> sparseness/distribution) in that sense. >>>>> >>>>> Regards, >>>>> >>>>> Fergal Byrne >>>>> -- >>>>> Sent from Mailbox <https://www.dropbox.com/mailbox> for iPhone >>>>> >>>>> >>>>> On Thu, Feb 6, 2014 at 12:52 AM, Jeff Hawkins <[email protected]>wrote: >>>>> >>>>>> I hope to clarify what topology is. I agree with some of what >>>>>> Fergal wrote but disagree with other parts, so maybe we are thinking >>>>>> different things. >>>>>> >>>>>> >>>>>> >>>>>> In a brain, the cortex is a sheet of tissue. Everywhere you look, it >>>>>> is about 2mm thick and consists of 4 or 5 layers of cells (depends on how >>>>>> you count). The total area of human neocortex is about 1000 cm2. >>>>>> Cortical regions are 2D areas. Some of the regions, such as the primary >>>>>> visual or touch regions are large. The retina sends about a million >>>>>> axons >>>>>> to the primarily visual cortex, or V1. Each axon represents a small part >>>>>> of the visual space, sort of like a pixel in an image. When this bundle >>>>>> fibers reaches V1 the 2D topology of the axons is maintained. It is as >>>>>> if >>>>>> a 1M pixel image were projected on to the area of V1. (The "image" is >>>>>> inverted, split, and highly distorted, but in all cases any two adjacent >>>>>> areas of V1 receive input from adjacent areas of the retina.) Scientists >>>>>> would say V1 has a "retinotopic" orientation. >>>>>> >>>>>> >>>>>> >>>>>> Now comes the interesting part. The cells in V1 learn common >>>>>> patterns coming from the retina. The way the SP works is each bit in the >>>>>> SP tries to learn the largest spatial pattern it can. And each bit >>>>>> competes with the other bits. The number of patterns coming from the >>>>>> retina is so large that the SP bits are forced to learn only small >>>>>> spatial >>>>>> patterns that are localized on the retina. So cells on the right side of >>>>>> V1 learn small patterns on the left side of the retina (remember the >>>>>> image >>>>>> is inverted). In biological V1 all the learning rules are local. >>>>>> Inhibition between columns is restricted in radius. The axons that a >>>>>> cell >>>>>> can possibly connect to are restricted to a radius. >>>>>> >>>>>> >>>>>> >>>>>> When we say a region has topology it means that inhibition between >>>>>> columns is limited to a local radius and not the entire region. It means >>>>>> that the set of potential connections that a cell or column can make are >>>>>> also within a radius and not all possible input bits or all possible >>>>>> cells >>>>>> in the region. This is the way real neocortex works. When we were >>>>>> applying the CLA to vision-like problems we implemented it with topology. >>>>>> It worked as expected but it is more complicated to test. The current >>>>>> implementation of the CLA has no topology. This means that each column >>>>>> competes with all the other columns and the set of potential connections >>>>>> between cells is a subset of all the cells in the region. There is no >>>>>> ordering of the columns, nothing is "adjacent" to anything else. For >>>>>> applications that are small no topology works great. If the application >>>>>> had a large 2D sensory array then topology would likely be required. >>>>>> >>>>>> >>>>>> >>>>>> I hope that helped. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> >>>>>> *From:* nupic [mailto:[email protected]] *On Behalf Of >>>>>> *Fergal Byrne >>>>>> *Sent:* Wednesday, February 05, 2014 5:54 AM >>>>>> *To:* NuPIC general mailing list. >>>>>> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >>>>>> >>>>>> >>>>>> >>>>>> HI Kevin, >>>>>> >>>>>> >>>>>> >>>>>> >> The following ideas are not things I think relate to "topology". >>>>>> >>>>>> >>>>>> >>>>>> Topology is the jargon we use to describe the semantic meaning (or >>>>>> lack of it) of a column (or input bit) position, and for connections >>>>>> between them. >>>>>> >>>>>> For example, a typical encoding of category data randomly picks 21 >>>>>> bits on out of 128. Each bit position has no real connection with the >>>>>> input >>>>>> category it encodes. This representation has no topology at all. >>>>>> >>>>>> Conversely, the sliding window of on-bits in a scalar encoding has >>>>>> one-dimensional topology. In this case, bit 0 might mean the input is < >>>>>> 10, >>>>>> while bit 23 might mean an input between 11 and 33. >>>>>> >>>>>> >>>>>> >>>>>> >> These ideas are things I think relate to "topology". >>>>>> >>>>>> >>>>>> >>>>>> An example of a 2d topology in an SDR is a retina-like sensor. In >>>>>> this case the light levels or presence of particular features in a >>>>>> certain >>>>>> spatial position would be represented by bits in the corresponding >>>>>> position. >>>>>> >>>>>> Topology in the SP relates to how each column position is connected >>>>>> with its input bits. An SP which is learning to recognise features in an >>>>>> SDR coming in from a retina would have columns "looking at" a set of >>>>>> input >>>>>> bits directly "below it" in the retina. The V1 primary visual cortex has >>>>>> exactly this property. >>>>>> >>>>>> If you have topology, you'll usually use local inhibition to pick >>>>>> winning columns, where local is defined according to the topology (1-d, >>>>>> 2-d >>>>>> distance). In the absence of topology, it's more efficient to just pick >>>>>> the >>>>>> highest activation potentials globally. >>>>>> >>>>>> >>>>>> >>>>>> Regards >>>>>> >>>>>> >>>>>> >>>>>> Fergal Byrne >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 5, 2014 at 11:25 AM, Kevin Martin < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Hi all, >>>>>> >>>>>> Thanks for the help. But I'm not sure if I understand what you >>>>>> mean by topology. As far as I understand, the spatial pooler has 2 >>>>>> functions : Return a set of columns that are 1)sparse and 2)distributed. >>>>>> >>>>>> Sparsity can be maintained by making sure that the pooler returns >>>>>> only n% of columns and this criterion can be made into a test. And >>>>>> distribution can be tested by computing the average distance between >>>>>> winning columns. What is topology? Is it another way of referring to a >>>>>> distributed set of columns? I might need some help with local learning >>>>>> too. >>>>>> I'm afraid I have never heard the term before (its in the white paper? A >>>>>> quick search on the pdf did not see any hits for "local learning") >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kevin Martin >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 5, 2014 at 7:37 AM, Scott Purdy <[email protected]> >>>>>> wrote: >>>>>> >>>>>> I would go over the properties of the SP and create tests >>>>>> accordingly. The output, for instance, should be tolerant to spatial >>>>>> noise. >>>>>> So show an instance a set of very different spatial patterns over and >>>>>> over >>>>>> a number of times. Then show one of the patterns with a single bit >>>>>> flipped >>>>>> and make sure the output is very close (possibly identical if columns are >>>>>> saturated) to the same. >>>>>> >>>>>> There are a bunch of similar things you could check. >>>>>> >>>>>> Ian had a hackathon demo showing the connected bits for each column. >>>>>> He was able to validate different aspects of the SP visually by using >>>>>> different numbers of columns and % overlap. >>>>>> >>>>>> You could do something similar to make sure you see the right >>>>>> behavior but I don't know if you could make an automated test out of it. >>>>>> >>>>>> >>>>>> http://numenta.org/blog/2013/11/06/2013-fall-hackathon-outcome.html#sp_viewer >>>>>> >>>>>> On Feb 4, 2014 11:23 AM, "Jeff Hawkins" <[email protected]> >>>>>> wrote: >>>>>> >>>>>> Kevin, >>>>>> >>>>>> Adding on to what Fergal said.... >>>>>> >>>>>> >>>>>> >>>>>> Here is what we did. We started with a fixed number of columns and >>>>>> no topology. In this situation we picked the n% most innervated columns >>>>>> as >>>>>> the output of the SP. Thus we were *guaranteed* to get the correct >>>>>> sparsity of active columns. With this setup we did a lot of other tests >>>>>> to >>>>>> verify that the SP was forming good representations. >>>>>> >>>>>> >>>>>> >>>>>> The tricky part is if you want to do this with topology and local >>>>>> learning rules. Is that your concern? >>>>>> >>>>>> >>>>>> >>>>>> With topology we didn't try to guarantee the entire layer of cells >>>>>> will have exactly n% active columns. Exact numbers are not essential. >>>>>> We >>>>>> tried several different local learning rules, all of them produced about >>>>>> the right level of sparsity. We than ran the other tests to make sure >>>>>> that >>>>>> local learning rules didn't cause the SP to form poor representations. >>>>>> You >>>>>> won't get exactly the same results with topology and without topology but >>>>>> my recollection was we were able to come up with a set of local >>>>>> inhibition >>>>>> rules that produced good results. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> >>>>>> *From:* nupic [mailto:[email protected]] *On Behalf Of >>>>>> *Fergal Byrne >>>>>> *Sent:* Tuesday, February 04, 2014 10:43 AM >>>>>> *To:* NuPIC general mailing list. >>>>>> *Subject:* Re: [nupic-discuss] Tests for spatial pooling >>>>>> >>>>>> >>>>>> >>>>>> Hi Kevin, >>>>>> >>>>>> >>>>>> >>>>>> If you're using no topology (and no local inhibition) then the >>>>>> fraction of active columns (2% in NuPIC) is the only important factor. >>>>>> With >>>>>> topology, you should choose winners relative to their neighbours. In >>>>>> either >>>>>> case, the first test is to count the fraction of active columns. In the >>>>>> second case, measure the average distance between active columns. >>>>>> >>>>>> >>>>>> >>>>>> Regards, >>>>>> >>>>>> >>>>>> >>>>>> Fergal Byrne >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Feb 4, 2014 at 6:05 PM, Kevin Martin < >>>>>> [email protected]> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I'm writing my own version of the CLA. I have not reached anywhere >>>>>> significant and I have been playing with a very small number of columns >>>>>> until now. I am starting by implementing a spatial pooler and spatial >>>>>> pooling requires a large number of columns. So far it has been easy to >>>>>> manually check if the synapses are distributed throughout the input, if >>>>>> the >>>>>> winning columns are really sparse etc. But it is going to be impossible >>>>>> when, say,I use 1000 columns. Have somebody used a test to check if the >>>>>> spatial pooler outputs a sparse distribution? Any suggestions on how to >>>>>> write a test? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Kevin Martin Jose >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> nupic mailing list >>>>>> [email protected] >>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> Fergal Byrne, Brenter IT >>>>>> >>>>>> >>>>>> >>>>>> http://inbits.com - Better Living through Thoughtful Technology >>>>>> >>>>>> >>>>>> >>>>>> e:[email protected] t:+353 83 4214179 >>>>>> >>>>>> Formerly of Adnet [email protected] http://www.adnet.ie >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> nupic mailing list >>>>>> [email protected] >>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> nupic mailing list >>>>>> [email protected] >>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> nupic mailing list >>>>>> [email protected] >>>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> >>>>>> Fergal Byrne, Brenter IT >>>>>> >>>>>> >>>>>> >>>>>> http://inbits.com - Better Living through Thoughtful Technology >>>>>> >>>>>> >>>>>> >>>>>> e:[email protected] t:+353 83 4214179 >>>>>> >>>>>> Formerly of Adnet [email protected] http://www.adnet.ie >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> nupic mailing list >>>>> [email protected] >>>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> nupic mailing list >>>> [email protected] >>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>>> >>>> >>> >>> _______________________________________________ >>> nupic mailing list >>> [email protected] >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
