Hi Gil san,
Thank you for the information. I like it very much; especially a lot of
comments in the code and the part that it's written in a single file.
synPermConnected is connectedPerm in the white paper, right? ;)
I'm thinking that it might be interesting for some people if we can run it
on silverlight + IronPython.
Best Regards,
Hideaki Suzuki.
2013/8/22 Gil Shotan <[email protected]>
> Hi Tim and Hideaki,
>
> My name is Gil, and I'm currently an intern at Grok Solutions working on
> revamping the Spatial Pooler. I've written a new implementation for the
> spatial pooler in Python, and currently working on converting it to C++.
> You are welcome to check out the new spatial pooler implementation on my
> fork (There's an open pull request to that branch on the NuPIC repo):
>
>
> https://github.com/gilsho/nupic/blob/newsp/py/nupic/research/spatial_pooler.py
>
> In designing the new spatial pooler implementation we tried to make it
> as easy as possible for people to understand the code and experiment with
> it. As such we took out some functionality such as input borders (although
> as Scott mentioned it is pretty simple to add and I can show guide you into
> how to add it) , but we did include support for topology. I actually
> implemented exactly what Tim suggested, namely that topology is
> conceptually separate. It is easy to override a function or two
> (mapPotential and initPermanence) and implement any arbitrary topology and
> neighboring relationships between columns as you'd like. If you get a
> chance – please take a look. I'd appreciate feedback!
>
> And to address your concern Tim, the algorithms in its current state is
> used in production, but that doesn't mean it can't be improved, or make
> modifications to it to apply it to other domains.
>
> Gil.
>
>
> From: Scott Purdy <[email protected]>
> Reply-To: "NuPIC general mailing list." <[email protected]>
> Date: Wednesday, August 21, 2013 1:43 PM
> To: "NuPIC general mailing list." <[email protected]>
> Subject: Re: [nupic-dev] A few questions about NuPIC
>
> Also, Gil is finishing up work on a new Python implementation of the
> spatial pooler that includes support for wrapping input (Left of 0,0 is
> 5,5). And we were going to leave the "input border" out of the
> implementation for simplicity but it is only a single line to add back in.
> The input border would leave space around the edges that you don't center
> columns in -- the bits would still be covered but it would reduce or
> eliminate the overhang of a column's input space outside the image.
>
>
> On Wed, Aug 21, 2013 at 1:19 PM, Scott Purdy <[email protected]> wrote:
>
>> Haven't had time to thoroughly read through this but just want to mention
>> that topology already exists in the codebase. By default, the CLA uses a
>> spatial pooler that has global inhibition (every column is a neighbor of
>> every other column) but you can specify in the constructor something else,
>> including a 2D topology that might make sense for vision.
>>
>> You can take a look at the "inputShape" parameter in the spatial pooler
>> constructor.
>>
>>
>> On Wed, Aug 21, 2013 at 12:02 PM, Tim Boudreau <[email protected]>wrote:
>>
>>> Hi, Hideaki-san,
>>>
>>> Thanks for your response! Some comments inline:
>>>
>>> On Wed, Aug 21, 2013 at 1:55 PM, Hideaki Suzuki <[email protected]>wrote:
>>>
>>>> If you say "I have to solve it at least once myself" to mean
>>>> implementing by yourself,
>>>> I have the same feeling and have been working for three months of my
>>>> part time.
>>>>
>>>
>>> :-)
>>>
>>>
>>>> As you can see it in the mail archive (http://nupic.markmail.org/),
>>>> the current implementation of
>>>> Nupic focuses on 1D topology for the proof of concept in production
>>>> level, which means a set
>>>> of values are put to CLA, and CLA predicts for future values. Not much
>>>> "neighbors" and
>>>> "hierarchies" exist yet.
>>>>
>>>
>>> I figure that topology really should be conceptually separate - if
>>> you've got an array of stuff, you can represent that as any number of
>>> dimensions; it helps if n is an n-root of the number of elements, but if
>>> you control the number of elements, that's easy. So I'd think you'd have
>>> an abstraction for "topology" and when you want to find neighbors of a
>>> cell/column, you'd "ask the topology". Otherwise you'd end up hard-coding
>>> assumptions that might be difficult to rip out later. I find it easier to
>>> think regions in terms of 2D layers, but that's what makes sense for a
>>> human, not necessarily a computer.
>>>
>>> "proof of concept in production level" worries me a bit - I thought
>>> this thing was already in production use; did I misunderstand and most of
>>> this is untested theory?
>>>
>>> 1. In some places, the white paper implies that input bits are
>>>>> wired directly to columns; elsewhere it suggests that there is a network
>>>>> of synapses between input bits and columns in the layer that directly
>>>>> receives input.
>>>>>
>>>>
>>>> Yes.
>>>> Those synapses are the links between input bits and your columns.
>>>> Each column will have some number of synapses. Say, 256 or 1024.
>>>> Each synapse is either connected or disconnected to its input bit
>>>> location according to its permanence value.
>>>>
>>>
>>> So, I guess my question is: Does a *proximal* dendrite connect to
>>> more than one bit?
>>>
>>>
>>>> If you have 1024 columns with 1024 synapses in your region, and all
>>>> synapses are connected, you need to
>>>> check 1 million links to see how strong each column is activated in
>>>> SP.
>>>>
>>>
>>> Yeah, this seems to be where the combinatorics go through the roof,
>>> and where hardware could help; probably some clever optimizations are
>>> possible to pre-determine a set that you don't need to check.
>>>
>>>
>>>> 3. It's mentioned several times that "strongly activated" columns
>>>>> inhibit nearby columns, to achieve sparseness. But it's also implied that
>>>>> the input to a cell is a single bit, which has no concept of strong or
>>>>> not-so-strong. Is "strongly activated" the count of cells in a column
>>>>> which are active due to feed-forward input, or is the input actually
>>>>> scalar
>>>>> and I just misunderstood that?
>>>>>
>>>>
>>>> As mentioned above, one column has many synapses.
>>>>
>>>
>>> This is the sort of statement that makes me wonder if I'm
>>> understanding things right. One *column* has many syapses, or one cell's
>>> dendrite has many synapses? Or we are summing
>>> column.cells.activeSynapseCount?
>>>
>>>
>>>> 4. I may be misunderstanding the nature of distal dendrite
>>>>> segments; I'm also trying to model the data efficiently. Is it sane to
>>>>> model a distal dendrite segment as
>>>>> - A column coordinate in a 2D topology - x,y coordinates
>>>>> - A list of path-traversal instructions - (i.e. up-left, down, down,
>>>>> right) in which no coordinate is traversed more than once
>>>>> - Permanence values mapped to instructions, to model synapses
>>>>>
>>>>
>>>> I may be misunderstanding your query..., but yes a column can be
>>>> arranged in 2D.
>>>>
>>>
>>> I was thinking of a 2D topology of columns, so you could refer to them
>>> as 0,0 ... 0,1 ... 0,2 ... 1,0 ... etc. In that case, say with 4 cells per
>>> column, the coordinates of one cell would be something like 0,1:3.
>>>
>>>
>>>> Distal dendrite segment is used for TP.
>>>> Cardinality is like...
>>>>
>>>> [Region] 1 --- 1..n [Column]
>>>> [Column] 1 --- 1 [Proximal Segment]
>>>> [Column] 1 --- 1..n [Cell]
>>>> [Cell] 1 --- 1..n [Distal Segment]
>>>>
>>>> [Proximal Segment] 1 --- 1..n [Proximal Synapse]
>>>> [Distal Segment] 1 --- 1..n [Distal Synapse]
>>>>
>>>
>>> That helps, thanks!
>>>
>>>
>>>> Potential synapses of a distal segment are setup virtually.
>>>> You will pickup any cell at random to make literal connections.
>>>>
>>>
>>> So basically, when creating a new distal segment, just randomly pick
>>> some cells "near" (in whatever topology) the cell it connects to?
>>>
>>> It seems like the choice of connections should profoundly effect the
>>> behavior of the system to the point of determining all of its future
>>> behavior.
>>>
>>>
>>>>
>>>>> 7. It's not clear how the initial state of the system (before
>>>>> any input) is established - i.e. you have to have some distal dendrite
>>>>> segments and potential synapses
>>>>>
>>>>
>>>> I believe distal segments and their potential synapses can be setup
>>>> on the fly.
>>>>
>>>
>>> Yes; the question is, what should be the algorithm to decide what
>>> cells/columns the distal segments connect to. Say that we have a grid of
>>> columns with 4-cells-per-column like
>>>
>>> ooooo
>>> ooxoo
>>> ooooo
>>>
>>> and we need to define the distal segment for x. Should we:
>>> - Pick a random path among the columns? Or some sort of gaussian
>>> distribution of neighboring columns? Or something else?
>>> - Have synapses with *every* cell in the columns intersected? Or just
>>> some? Or just one?
>>>
>>>
>>>> For edge and corner detection, I'm not so sure how well SP can
>>>> handle.
>>>> At least, SP seems to work well for the roll of feature detection (V1)
>>>> in the video above.
>>>>
>>>
>>> I wasn't clear - I wasn't talking about edge and corner detection.
>>> Rather, say we have a grid of cells:
>>> xooox
>>> ooooo
>>> xooox
>>>
>>> The white paper states that columns are inhibited by other columns
>>> near them. In this grid, most cells have 8 neighbors which could inhibit
>>> or be inhibited by them. But the corner columns have only 3 neighbors and
>>> the edge cells have 5 neighbors. Which means a corner column will be less
>>> inhibited. So even if the signal activating it is less relevant, it has a
>>> higher probability of being active. This seems like it ought to be a
>>> problem; you could solve it by:
>>> - Ignoring predictions from perimeter columns
>>> - Wrapping around, so that above and to the left of 0,0 is 5,5
>>> - Having such a large number of columns that erroneous values from
>>> (literally!) edge-cases have minimal effect
>>>
>>> Thanks,
>>>
>>> -Tim
>>>
>>> --
>>> http://timboudreau.com
>>>
>>> _______________________________________________
>>> nupic mailing list
>>> [email protected]
>>> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
>>>
>>>
>>
>
> _______________________________________________
> nupic mailing list
> [email protected]
> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
>
>
_______________________________________________
nupic mailing list
[email protected]
http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
