>> 8. A friend of mine I mentioned the project to said that he couldn't find a crisp definition of "sparse distributed representation" anywhere. I'd indeed figured it was one of those things I'd pick up intuitively with enough reading, and did to a >> degree. Can I check my understanding with you? Specifically, it seems like the high-order feature is that you have a lot of bits, but only a small percentage are active.
In the academic literature I find that the term "sparse coding" seems to be used more than "sparse distributed representation", but they're the same thing. Here are a few papers/pages: This paper from the early 80s made a strong case for distributed representations: http://www-psych.stanford.edu/~jlm/papers/PDP/Chapter3.pdf This paper compares sparse vs dense distributed representations: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16.1720&rep=rep1&type=pdf This is a briefer discussion of sparse coding: http://www.scholarpedia.org/article/Sparse_coding -Mike _____________ Michael Ferrier Department of Cognitive, Linguistic and Psychological Sciences, Brown University [email protected] On Wed, Aug 21, 2013 at 5:01 PM, Gil Shotan <[email protected]>wrote: > Hi Tim and Hideaki, > > My name is Gil, and I'm currently an intern at Grok Solutions working on > revamping the Spatial Pooler. I've written a new implementation for the > spatial pooler in Python, and currently working on converting it to C++. > You are welcome to check out the new spatial pooler implementation on my > fork (There's an open pull request to that branch on the NuPIC repo): > > > https://github.com/gilsho/nupic/blob/newsp/py/nupic/research/spatial_pooler.py > > In designing the new spatial pooler implementation we tried to make it > as easy as possible for people to understand the code and experiment with > it. As such we took out some functionality such as input borders (although > as Scott mentioned it is pretty simple to add and I can show guide you into > how to add it) , but we did include support for topology. I actually > implemented exactly what Tim suggested, namely that topology is > conceptually separate. It is easy to override a function or two > (mapPotential and initPermanence) and implement any arbitrary topology and > neighboring relationships between columns as you'd like. If you get a > chance – please take a look. I'd appreciate feedback! > > And to address your concern Tim, the algorithms in its current state is > used in production, but that doesn't mean it can't be improved, or make > modifications to it to apply it to other domains. > > Gil. > > > From: Scott Purdy <[email protected]> > Reply-To: "NuPIC general mailing list." <[email protected]> > Date: Wednesday, August 21, 2013 1:43 PM > To: "NuPIC general mailing list." <[email protected]> > Subject: Re: [nupic-dev] A few questions about NuPIC > > Also, Gil is finishing up work on a new Python implementation of the > spatial pooler that includes support for wrapping input (Left of 0,0 is > 5,5). And we were going to leave the "input border" out of the > implementation for simplicity but it is only a single line to add back in. > The input border would leave space around the edges that you don't center > columns in -- the bits would still be covered but it would reduce or > eliminate the overhang of a column's input space outside the image. > > > On Wed, Aug 21, 2013 at 1:19 PM, Scott Purdy <[email protected]> wrote: > >> Haven't had time to thoroughly read through this but just want to mention >> that topology already exists in the codebase. By default, the CLA uses a >> spatial pooler that has global inhibition (every column is a neighbor of >> every other column) but you can specify in the constructor something else, >> including a 2D topology that might make sense for vision. >> >> You can take a look at the "inputShape" parameter in the spatial pooler >> constructor. >> >> >> On Wed, Aug 21, 2013 at 12:02 PM, Tim Boudreau <[email protected]>wrote: >> >>> Hi, Hideaki-san, >>> >>> Thanks for your response! Some comments inline: >>> >>> On Wed, Aug 21, 2013 at 1:55 PM, Hideaki Suzuki <[email protected]>wrote: >>> >>>> If you say "I have to solve it at least once myself" to mean >>>> implementing by yourself, >>>> I have the same feeling and have been working for three months of my >>>> part time. >>>> >>> >>> :-) >>> >>> >>>> As you can see it in the mail archive (http://nupic.markmail.org/), >>>> the current implementation of >>>> Nupic focuses on 1D topology for the proof of concept in production >>>> level, which means a set >>>> of values are put to CLA, and CLA predicts for future values. Not much >>>> "neighbors" and >>>> "hierarchies" exist yet. >>>> >>> >>> I figure that topology really should be conceptually separate - if >>> you've got an array of stuff, you can represent that as any number of >>> dimensions; it helps if n is an n-root of the number of elements, but if >>> you control the number of elements, that's easy. So I'd think you'd have >>> an abstraction for "topology" and when you want to find neighbors of a >>> cell/column, you'd "ask the topology". Otherwise you'd end up hard-coding >>> assumptions that might be difficult to rip out later. I find it easier to >>> think regions in terms of 2D layers, but that's what makes sense for a >>> human, not necessarily a computer. >>> >>> "proof of concept in production level" worries me a bit - I thought >>> this thing was already in production use; did I misunderstand and most of >>> this is untested theory? >>> >>> 1. In some places, the white paper implies that input bits are >>>>> wired directly to columns; elsewhere it suggests that there is a network >>>>> of synapses between input bits and columns in the layer that directly >>>>> receives input. >>>>> >>>> >>>> Yes. >>>> Those synapses are the links between input bits and your columns. >>>> Each column will have some number of synapses. Say, 256 or 1024. >>>> Each synapse is either connected or disconnected to its input bit >>>> location according to its permanence value. >>>> >>> >>> So, I guess my question is: Does a *proximal* dendrite connect to >>> more than one bit? >>> >>> >>>> If you have 1024 columns with 1024 synapses in your region, and all >>>> synapses are connected, you need to >>>> check 1 million links to see how strong each column is activated in >>>> SP. >>>> >>> >>> Yeah, this seems to be where the combinatorics go through the roof, >>> and where hardware could help; probably some clever optimizations are >>> possible to pre-determine a set that you don't need to check. >>> >>> >>>> 3. It's mentioned several times that "strongly activated" columns >>>>> inhibit nearby columns, to achieve sparseness. But it's also implied that >>>>> the input to a cell is a single bit, which has no concept of strong or >>>>> not-so-strong. Is "strongly activated" the count of cells in a column >>>>> which are active due to feed-forward input, or is the input actually >>>>> scalar >>>>> and I just misunderstood that? >>>>> >>>> >>>> As mentioned above, one column has many synapses. >>>> >>> >>> This is the sort of statement that makes me wonder if I'm >>> understanding things right. One *column* has many syapses, or one cell's >>> dendrite has many synapses? Or we are summing >>> column.cells.activeSynapseCount? >>> >>> >>>> 4. I may be misunderstanding the nature of distal dendrite >>>>> segments; I'm also trying to model the data efficiently. Is it sane to >>>>> model a distal dendrite segment as >>>>> - A column coordinate in a 2D topology - x,y coordinates >>>>> - A list of path-traversal instructions - (i.e. up-left, down, down, >>>>> right) in which no coordinate is traversed more than once >>>>> - Permanence values mapped to instructions, to model synapses >>>>> >>>> >>>> I may be misunderstanding your query..., but yes a column can be >>>> arranged in 2D. >>>> >>> >>> I was thinking of a 2D topology of columns, so you could refer to them >>> as 0,0 ... 0,1 ... 0,2 ... 1,0 ... etc. In that case, say with 4 cells per >>> column, the coordinates of one cell would be something like 0,1:3. >>> >>> >>>> Distal dendrite segment is used for TP. >>>> Cardinality is like... >>>> >>>> [Region] 1 --- 1..n [Column] >>>> [Column] 1 --- 1 [Proximal Segment] >>>> [Column] 1 --- 1..n [Cell] >>>> [Cell] 1 --- 1..n [Distal Segment] >>>> >>>> [Proximal Segment] 1 --- 1..n [Proximal Synapse] >>>> [Distal Segment] 1 --- 1..n [Distal Synapse] >>>> >>> >>> That helps, thanks! >>> >>> >>>> Potential synapses of a distal segment are setup virtually. >>>> You will pickup any cell at random to make literal connections. >>>> >>> >>> So basically, when creating a new distal segment, just randomly pick >>> some cells "near" (in whatever topology) the cell it connects to? >>> >>> It seems like the choice of connections should profoundly effect the >>> behavior of the system to the point of determining all of its future >>> behavior. >>> >>> >>>> >>>>> 7. It's not clear how the initial state of the system (before >>>>> any input) is established - i.e. you have to have some distal dendrite >>>>> segments and potential synapses >>>>> >>>> >>>> I believe distal segments and their potential synapses can be setup >>>> on the fly. >>>> >>> >>> Yes; the question is, what should be the algorithm to decide what >>> cells/columns the distal segments connect to. Say that we have a grid of >>> columns with 4-cells-per-column like >>> >>> ooooo >>> ooxoo >>> ooooo >>> >>> and we need to define the distal segment for x. Should we: >>> - Pick a random path among the columns? Or some sort of gaussian >>> distribution of neighboring columns? Or something else? >>> - Have synapses with *every* cell in the columns intersected? Or just >>> some? Or just one? >>> >>> >>>> For edge and corner detection, I'm not so sure how well SP can >>>> handle. >>>> At least, SP seems to work well for the roll of feature detection (V1) >>>> in the video above. >>>> >>> >>> I wasn't clear - I wasn't talking about edge and corner detection. >>> Rather, say we have a grid of cells: >>> xooox >>> ooooo >>> xooox >>> >>> The white paper states that columns are inhibited by other columns >>> near them. In this grid, most cells have 8 neighbors which could inhibit >>> or be inhibited by them. But the corner columns have only 3 neighbors and >>> the edge cells have 5 neighbors. Which means a corner column will be less >>> inhibited. So even if the signal activating it is less relevant, it has a >>> higher probability of being active. This seems like it ought to be a >>> problem; you could solve it by: >>> - Ignoring predictions from perimeter columns >>> - Wrapping around, so that above and to the left of 0,0 is 5,5 >>> - Having such a large number of columns that erroneous values from >>> (literally!) edge-cases have minimal effect >>> >>> Thanks, >>> >>> -Tim >>> >>> -- >>> http://timboudreau.com >>> >>> _______________________________________________ >>> nupic mailing list >>> [email protected] >>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >>> >>> >> > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
