--- Vladimir Nesov <[EMAIL PROTECTED]> wrote: > Say, each functional concept (a bit in total amount of memory) is > represented by R synapses and M neurons. When certain pattern of concepts is > observed, it creates a repeatable sequence of events. Say, pattern is one > concept being followed by another with a fixed delay. Observation of each > concept corresponds to approximately M events of firing of neurons > comprising a concept. Let's assume that received spikes can contribute to > the same spiking event if their arrival differs no more than 1/L of average > delay time. Each of M events in the sequence creates F spikes through F > outgoing synapses (F=10^4). So, this pattern will be learned if there is a > neuron that receives at least, say, 2 spikes from different concepts within > 1/L of average axonal delay time. At each moment there are M*F/L randomly > sampled neurons for each concept, with total of X neurons in the network, so > probability of two of them arriving in the same neuron is on the order of 1 > when (M*F/L)^2 is on the order of X. Calculating: X=10^11, L=(10^2)/2 (I'm > not sure about this one: average delays are up to about 50ms, I assume 1ms > as coincidence time), F=10^4, so M=(3*10^5)*((10^2)/2)/(10^4)=10^3, or > X/M=10^8 neurons representing unique concepts (where each neuron can > represent multiple groups of incoming events), with 1% of active synapses it > corresponds to 10^10 synapses without redundancy, add or take an order of > magnitude. > > So, this estimate gives a number close to 10^9, even though all 10^15 > synapses are required...
In most neural models, the important signal is the average rate of firing, so L = 1. I realize there are exceptions, for example the timing of firings is significant for the transmission of phase information for stereoscopic sound perception. But we know from lots of research in the 1980's that the simpler model works pretty well for most scaled down AI tasks. We also know that a 2 layer, fully connected symmetric n by n network can store 0.15 bits per synapse (0.15n vectors) [1], whether we use 2-state neurons or graded response neurons in our models [2]. This is consistent with cognitive language models where the number of concepts (10^4.5 words) is on the order of the square root of the entropy of the model (10^9 bits). The estimate of 10^9 bits comes from: 1. Turing's 1950 estimate, which he did not explain [3]. 2. Long term recall tests, such as those by Landauer [4]. 3. The fact that humans read and hear about 1 GB of language by adulthood, which is 10^9 bits assuming Shannon's estimate of 1 bit per character [5]. Of course we can make many qualitative arguments which would narrow the gap. For example, the brain has a lot of redundancy for fault tolerance, the constraint of parallelism (duplicate weights in different parts of the visual field), and the need to generate smooth signals from pulses. There are also an unknown number of fixed weight synapses that simply transmit signals but don't learn. At the other end of the gap, we probably remember more than is testable in a Turing test or a recall test. For example, Standing [6] had subjects look at 10,000 pictures at the rate of one every 5.6 seconds for 2 days, then tested their recall accuracy, which was something like 79%. You could get this result if you compressed each picture to a 16 bit feature vector and tested for a match. From this result, you would conclude that visual perception has a bandwidth of 16/5.6 = 2.8 bits per second. Tests with random words (spoken or written) or music clips would give you similar numbers. But this analysis underestimate memory. For example, vivid pictures (e.g. a dog smoking a pipe) are recalled more accurately than ordinary pictures (a dog or a pipe). A model in which pictures have variable length representations would require more memory to achieve the same error rate. Another problem is that not all memory can be measured by recall tests. Learning to walk or catch a ball clearly requires long term memory. How much? How would you measure it? Without a number, you could argue that the vast majority of synapses store subconscious (non recallable) memories. But I can still argue otherwise. Humans are not significantly superior to other large animals with smaller brains (such as a bear or a deer) in skills that don't involve language, such as running over rough terrain or discriminating various plants and animals. Why does language require such a large brain? References 1. Hopfield, J. J. (1982), "Neural networks and physical systems with emergent collective computational abilities", Proceedings of the National Academy of Sciences (79) 2554-2558. 2. Hopfield, J. J. (1984), "Neurons with Graded Response have Collective Properties like those of Two-State Neurons", Proceedings of the National Academy of Sciences, USA (81) pp. 3088-3092. 3. Turing, A. M., (1950) "Computing Machinery and Intelligence", Mind, 59:433-460. 4. Landauer, Tom (1986), "How much do people remember? Some estimates of the quantity of learned information in long term memory", Cognitive Science (10) pp. 477-493. 5. Shannon, Claude E. (1950), "Prediction and Entropy of Printed English", Bell Sys. Tech. J (3) p. 50-64. 6. Standing, L. (1973), "Learning 10,000 Pictures", Quarterly Journal of Experimental Psychology (25) pp. 207-222. -- Matt Mahoney, [EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=55636038-76824e
