That's an interesting way to think about hierarchy, trying to determine a formulaic way to structure the hierarchy. This kind of thought experiment can be helpful.
However, there is a lot of evidence to suggest real cortex isn't working on such a principle. One piece of evidence is that different mammals such as dogs, cats, monkeys, and humans have a similar set of sensors but different number of regions and levels in the hierarchy. Another piece of evidence is that most mammals have primary and secondary sensory regions, such as V1 and V2, where the secondary region is receiving input almost exclusively from the primary one. The convergence onto V2 is coming from parts of V1, not two different regions. A monkey has dozens of cortical regions. I don't think of the retina as a single sense. I think of it as an array of 1M sensors, that is the number of axons in the optic nerve. The auditory nerve is an array of 30K sensors. There are about 1M somatic sensors. The input to the cortex is not just a handful of senses but millions of sensors arranged in topological arrays. With this view a simple calculus doesn't work. The goal of the cortex is to form representations of its inputs. We want those representations to be independent of what sensors are sensing the object(s) and where on the sensor arrays the object is being sensed. In theory, if we had one region of almost unlimited capacity (very large number of mini-columns) and we had an almost unlimited amount of training data then a single region without hierarchy could learn everything. Hierarchy is needed to make a practical system. Consider V1. It receives the input from the optic nerve. The spatial pooler wants to form representations of the spatial patterns coming from the retina. There is a limited capacity in V1 and a lot of patterns on the optic nerve so each V1 SP bit learns to represent only a small part of the retina. As the number of spatial patterns seen by the animal declines then each bit in the SP will represent larger patterns, representing larger parts of the input area. Similarly, if we increased the number of SP bits (number of minicolumns in cortex) then each SP bit would learn to recognize larger input patterns. There is an equilibrium point where the SP forms good representations over some part of the input space. For humans and monkeys V1 neurons represent about 1 degree of visual field. If we severely restricted the number of patterns that the eye could see then neurons in V1 would represent larger areas of the retina. The same sort of give and take between resources and input pattern complexity exists in the sequence memory part of the CLA. Hierarchy allows the system to get to representations that are independent of sensor location in practical time with practical resources. Each level in the hierarchy builds on the previous. How many levels in the hierarchy are required? It depends on how many columns and cells exist at each level and how complex are the patterns in the input. There is a tradeoff of using a hierarchy. The cortex gives up the ability to learn many possible patterns such as a visual pattern where pixels on the left side of an image are tightly correlated with pixels on the right side of the image. However, practically these kinds of patterns don't occur. I hope that helped and didn't make things worse! Jeff From: nupic [mailto:[email protected]] On Behalf Of mariolakakis . Sent: Sunday, February 16, 2014 2:01 PM To: [email protected] Subject: [nupic-discuss] What dictates the exact formation of the cortical hierarchy? I know that the goal is efficiency in training and storage but how is the hierarchy in the neocortex done exactly? Is it a result of a mathematical equation? Or is it the throw it in the wall and see if it sticks process of evolution? My mathematical theory is very simple and it's based on binomial coefficients. Let's say that the human body consist of a K number of sensors. The number of regions X in the hierarchy should be equal to X = !K / (2! * (K - 2)!) + K (for each sensor). That's the number of all possible duplets of sensors with no repetitions. This equation creates a pool of the highest variety but least density that we can use for representations. And it also explains the tree like shape of the hierarchy and why it converges and diverges and you up and down. For example, if the human body consisted of just three sensors: S1 = Optic, S2 = Acoustic, S3 = Touch The number of regions would be X = 3 * 2 / (2 * 1) + 3 = 3 + 3 = 6 1. R1 = S1 (Optic) 2. R2 = S2 (Acoustic) 3. R3 = S3 (Touch) 4. R4 = R1, R2 (Optic + Acoustic) 5. R5 = R2, R3 (Acoustic + Touch) 6. R6 = R4, R5 (Optic + Acoustic + Touch) Let's consider the part of the neocortex that handles language. The number of characters in the alphabet is much smaller than the number of words and the number of words is tiny compared to the number of phrases. This simple observation makes me assume that the hierarchy in the brain is like this: 1. Letters (Highest level concepts) 2. Words 3. Phrases Using a single region we would have to assign columns to letters and sequences of cells to words. For example, the words "god" and "dog" would share the same spatial pattern but different temporal patterns. Since, the higher regions get only spatial patterns from below how does the distinction of those two gets communicated above? What happens if the word has multiple identical letters? Do the cells in a column connect to other cells in the same column? For example, the word "good" has two "o"s. To summarise, if one region wasn't enough and I wanted to reconstruct the human neocortex based on a K number of sensors, how would I know how many regions I would need, and in what way should I connect them to make it all work? Thats a problem you will face in the future. One day, one region won't be enough. I 've implemented a huge part of the CLA in Xcode and got it running on an iPhone, I've seen a dozen videos of Jeff Hawkins' presentations and I 've also bought the book On Intelligence but I haven't found any answers to these questions. I'm counting on you guys. :)
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
