Here is a piece of Trivia..... In the write up below Gil says "My aim is to refactor the spatial pooler class (which for some reason is called FDRCSpatial2.py)"
Why the name "FDR"? One of the insights that led to the CLA was keeping the sparseness of activations at a relatively fixed level. The whole system relies on this. If we let the sparseness of activations vary a lot then the spatial pooler and temporal pooler wouldn't work well. So the original internal name for the CLA was FDR which stood for "Fixed-sparsity Distributed Representations" to differentiate it from other algorithms we were looking at that didn't require a relatively fixed sparseness. We later abandoned that term, but apparently it is still in the code. ;-) Jeff -----Original Message----- From: nupic [mailto:[email protected]] On Behalf Of Gil Shotan Sent: Friday, July 12, 2013 12:49 PM To: [email protected] Subject: [nupic-dev] Restructuring Spatial Pooler Hey Everyone, For those of you who had a chance to check out the spatial pooler implementation (it's located in nupic/py/nupic/research/FDRCSpatial2.py) you'll notice that its a very long file containing some obsolete code. I took it upon myself to rewrite it to make it more readable and accessible. To understand the changes I am proposing, a little background is in order (this will require having read the white paper): Basically, the spatial pooler uses a "sparse matrix" implementation under the hood to keep track of the permanences of each column (referred to as "coincidence" in the code). Each row in the matrix represents a cortical column, and each column in the matrix represents an input bit. Since a column is only connected to a subset of the inputs, using sparse matrix implementation makes operations such as computing the overlap more efficient while at the same time reducing the memory footprint. In the past, Numenta was experimenting with vision related problems, so each region had a 2D-topology of columns, as shown in the white paper. This topology determined a columns' neighbors for the purpose of inhibition, as well as the "natural center" over the inputs that each column is biased towards. With vision related inputs, each sub-region in lower regions might learn to represent the same patterns. For example, in the V1 region there are multiple "edge detectors" responsible for detecting edges at different areas of an eye's the field of vision. Since these "edge detectors" are very similar, there is an optimization in the code that allows for training a small number of columns in a region over a subset of the inputs and "cloning" the learned permanences across the entire region. Currently all this functionality is combined into one giant class (3000+) lines that is very difficult to understand. Furthermore, as Grok shifted away from dealing with vision, most of this functionality is not being used. For time-series data, Grok uses a subset of the aforementioned functionality that doesn't assume any topology or structure to the columns. Basically, each column can be connected to every input and the "winning" columns after the overlap has been computed are selected to be columns with highest overlap of the entire region (This is referred to as "global inhibition"). My aim is to refactor the spatial pooler class (which for some reason is called FDRCSpatial2.py) to make it more manageable and easier to change, while also supporting the aforementioned functionality, that even though outdated, might serve some of you who are interested in applying the NuPIC framework to interesting and diverse problems.Therefore I am planning on breaking down the spatial pooler class into several classes, with the following inheritance structure: +-------------------+ | SpatialPooler | +---+------------+--+ | | v v +--------------------+ +---------------------+ | FlatSpatialPooler | | VisionSpatialPooler | +--------------------+ +--------+------------+ | v +-----------------------+ | ClonedSpatialPooler | +-----------------------+ Where the "SpatialPooler" class will contain the underlying implementation and all the sparse matrix operations. The "FlatSpatialPooler" will contain only the minimal adjustments needed to work for streaming time series data (which is what is currently being used). The "VisionSpatialPooler" will contain additional structure data, allowing for local inhibition in a 2D region, and the "ClonedSpatialPooler" class will allow for training only a subset of a region and copying the learned permanence values across the rest of the region. In doing so, I will aim to rename some of the variables and methods to conform to the terminology used in the white paper. Please feel free to share any comments or suggestions you may have! Gil Shotan Algorithms Intern, Grok Solutions [email protected] _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org _______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
