Hi, If you say "I have to solve it at least once myself" to mean implementing by yourself, I have the same feeling and have been working for three months of my part time.
I'm almost done with Spatial Pooler (SP), and I found that the basic logic was beautiful. I hope I can finish my learning of Temporal Pooler (TP) after another one or two months. As you can see it in the mail archive (http://nupic.markmail.org/), the current implementation of Nupic focuses on 1D topology for the proof of concept in production level, which means a set of values are put to CLA, and CLA predicts for future values. Not much "neighbors" and "hierarchies" exist yet. There is room of improvement for 2D vision and other "Domains". The topic of domain specific optimization is referred in JIRA ( https://issues.numenta.org/browse/NPC-293) To suggest improvement in those areas, publishing your code early and showing how better your code can work for some specific data set will help, because this is an open source project. You can refer to https://issues.numenta.org/browse/NPC-309. The above is my current understanding. I hope Numenta people will correct me when I have some misconception. I try to answer your queries inline. 2013/8/21 Tim Boudreau <[email protected]> > I'm new to this list and to nupic - it's a fascinating project that > confirms a few intuitions I've had, and might let me try out a few ideas > I've had in the back of my mind for a couple of decades, so I'm very > excited to play with it. > > I find that to understand a problem deeply enough to *really* get what's > going on, I have to solve it at least once myself. So I've been poking > around the nupic source code, but also read through the white paper, took > copious notes, and from that have been attempting to implement some of the > data structures and algorithms in Java (yes, I found the GitHub project > where someone else did that). It helps to get my head around the realistic > memory requirements of such a system (plus my C skills are 18 years > atrophied :-/), what's really going on under the hood, and understand its > limitations. > > So if you don't mind, can I ask a few questions about data structures, to > resolve a couple of ambiguity and figure out if I'm modelling it right? > Specifically: > > 1. In some places, the white paper implies that input bits are wired > directly to columns; elsewhere it suggests that there is a network of > synapses between input bits and columns in the layer that directly receives > input. > Yes. Those synapses are the links between input bits and your columns. Each column will have some number of synapses. Say, 256 or 1024. Each synapse is either connected or disconnected to its input bit location according to its permanence value. If you have 1024 columns with 1024 synapses in your region, and all synapses are connected, you need to check 1 million links to see how strong each column is activated in SP. As learning progresses, I observe the number of connected synapses decreases. SP will work more efficiently with your input patterns as time goes by. > > 2. It's implied several times that layers feed *back* as well as forward, > but no mechanism is really made explicit. Is that correct or am I > misunderstanding it? > There seem a lot of talks in this ML about hierarchy, attention, feedback, kind of things. People believe that some problems (not all) require "feedback" to get resolved. For the implementation perspective, deep learning seems more advanced than CLA, for now. I definitely want to see CLA's version of hierarchy construction in the near future. You or I could work on it, too, since this is open sourced. ;) > 3. It's mentioned several times that "strongly activated" columns inhibit > nearby columns, to achieve sparseness. But it's also implied that the > input to a cell is a single bit, which has no concept of strong or > not-so-strong. Is "strongly activated" the count of cells in a column > which are active due to feed-forward input, or is the input actually scalar > and I just misunderstood that? > As mentioned above, one column has many synapses. Having a pattern that overlaps with the pattern of connected synapses of a column will activate the column stronger. > > 4. I may be misunderstanding the nature of distal dendrite segments; I'm > also trying to model the data efficiently. Is it sane to model a distal > dendrite segment as > - A column coordinate in a 2D topology - x,y coordinates > - A list of path-traversal instructions - (i.e. up-left, down, down, > right) in which no coordinate is traversed more than once > - Permanence values mapped to instructions, to model synapses > I may be misunderstanding your query..., but yes a column can be arranged in 2D. Distal dendrite segment is used for TP. Cardinality is like... [Region] 1 --- 1..n [Column] [Column] 1 --- 1 [Proximal Segment] [Column] 1 --- 1..n [Cell] [Cell] 1 --- 1..n [Distal Segment] [Proximal Segment] 1 --- 1..n [Proximal Synapse] [Distal Segment] 1 --- 1..n [Distal Synapse] > > or am I misunderstanding something, and the set of potential synapses > should be *all* adjacent and near-adjacent cells? Or is randomly > generating some paths sufficient? > Potential synapses of a proximal segment are setup statically. A column has its fan-out area on the input bits. On the other way around, one input bit has its receptive field of columns on the region. The concept of receptive field, i.e. locality, is not precisely defined anywhere yet (several definitions are possible), probably because there is no locality in the current implementation. Potential synapses of a distal segment are setup virtually. You will pickup any cell at random to make literal connections. > > 5. It's not clear how distal dendrites relate to columns - right now I'm > treating all cells within a column a dendrite's "path" intercects with as > potential synapses. Not clear if that's the correct approach, or if the > "instructions" should include a spec for which cell in the intersected > column should be a potential synapse. > I hope the above cardinality can help. > > 6. Boost factor - this seems like something that could be modelled as a > single byte, scaled to 0.0-1.0 - or is that likely to be insufficient > resolution? > Right. The white paper clarifies the concept only. The meaning of boosting is, IIUC, we want all columns to be more-equally eligible to represent the input patterns; I would say, habitat segregation for the input patterns. Once I asked in the ML whether I can use addition rather than multiplication; is there a special meaning for multiplication? Jeff san answered, not a specific reason. In my C# implementation, I tried the following evaluation function for boosting, which seems to convey the similar effect but without involving neighbors (less cost). b = (# of inhibition the column c experienced) / (# of activation the column c experienced) if (b < threshold, say 100) boost(c) = 0 else boost(c) = b - threshold. I don't think the above function is best. I can explore better options, too. > 7. It's not clear how the initial state of the system (before any input) > is established - i.e. you have to have some distal dendrite segments and > potential synapses > I believe distal segments and their potential synapses can be setup on the fly. Proximal segments should be setup before feeding the input. > > 8. A friend of mine I mentioned the project to said that he couldn't find > a crisp definition of "sparse distributed representation" anywhere. I'd > indeed figured it was one of those things I'd pick up intuitively with > enough reading, and did to a degree. Can I check my understanding with > you? Specifically, it seems like the high-order feature is that you have a > lot of bits, but only a small percentage are active. > SDR seems a common concept in AI field recently. The following video (talking about Deep learning) looks like a good introduction for SDR. http://www.youtube.com/watch?v=n1ViNeWhC24&feature=em-hot-vrecs The time is around 0:21:30 -. I hope it helps. > 9. A number of things, such as column inhibition, are affected by what is > adjacent. This brings up a problem that also exists in image processing - > edges and corners are special, and you either a. simulate a continuous > topology by wrapping around, or b. interpolate values outside the bounds, > c. ignore values from some inner perimiter, or d. just live with it. So, > for example, a column in a corner should be less inhibited by neighboring > columns because it has fewer neighbors (it also has more limited choices > for potential synapses). Is this a problem, and if so, how do you deal > with it? > Right. We have currently the following layered structure in Nupic. Input Stream v [ encoder ] ... it corresponds to organs like retina. it converts input data to binary array. [ the spatial pooler ] ... it converts input patters to SDR by making the best use of columns [ the temporal pooler ] ... it converts SDR time-change patterns to the literal network connections [ classifier ] ... Numenta's artifact. it converts SDR predictions back to the usable data. v Predicted Data Stream For edge and corner detection, I'm not so sure how well SP can handle. At least, SP seems to work well for the roll of feature detection (V1) in the video above. I personally feel that I can explore various image filters in the encoder layer, and how well SP reacts against them. Another point is that, we will have many columns looking at a corner of your concern. One of those may have many connected synapses for a large pattern. One may have a few connected synapses for dot-like pattern. One may have a set of connected synapses with different angle of a corner. The one with the connected synapses over the best overlap will be the winner, unless you have a little less overlap but boosted one. I hope your queries could be solved by some extent. Best Regards, Hideaki Suzuki. > > > I realize that's a lot of questions. Any suggestions appreciated, or feel > free to tell me to just go read the code :-) > > Thanks, > > Tim Boudreau > -- > http://timboudreau.com > > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
