Matt, As acknowledged before, your notion of using a single matrix to represent a neural net with say 10^5 neurons, each with up to 10^4 connections, has the benefit of allowing for much more efficient processing, since the rows could be read serially, which allows much faster memory access, and the output from the processing along each row that is contributed to column values, could be stored largely in cache, allowing for relatively rapid storage, even if the representations of contacts are relatively sparse along each row, and since the equivalent of address representations of contacts could be represented much more efficiently. I had known neural nets were often represented by matrices, but It is still not totally clear to me that it would work well with the type of system I am interested in. But it might.
-----Original Message----- From: Matt Mahoney [mailto:[EMAIL PROTECTED] Sent: Friday, June 13, 2008 2:30 PM To: [email protected] Subject: RE: [agi] IBM, Los Alamos scientists claim fastest computer With regard to representing different types of synapses (various time delays, strength bounds, learning rates, etc), this information can be recorded as characteristics of the input and output neurons and derived as needed to save space. [Ed Porter] I still think you are going to need multi-bit weights at row-column element in the matrix -- since most all representations of synapses I have seen have assumed a weight having at least 6 bits of information, and there is reason to think you need to store both a short term and a long term value, since it seems to me it is necessary for the temporal correlation component of Hebbian learning, and to represent the state information that is short-term memory. [Ed Porter] I don't understand how this information could be stored or accessed more efficiently than in row-column matrix elements, so they would be available when computing a row's contribution to each successive column. This would include any information such as whether excitatory or inhibitory, delays, strength bounds, learning rates, unless you assume such information is uniform in each row or each column. (In the brain, this may be the case for some attributes --such as, I think, having separate neurons for inhibition and excitation, but it would appears to require added complexity for many learning tasks). Minimizing inter-processor communication is a harder problem. This can be done by mapping the neural network into a hierarchical organization so that groups of co-located neurons are forced to communicate with other groups through narrow channels using a small number of neurons. We know that many problems can be solve this way. For example, a semantic language model made of a 20K by 20K word association matrix can be represented using singular value decomposition as a 3 layer neural network with about 100 to 200 hidden neurons [1,2]. The two weight matrices could then be implemented on separate processors which communicate through the hidden layer neurons. [Ed Porter] Having matrices for connecting the matrices makes sense. But my understanding is the SVD is often a form of lossy compression. [Ed Porter] As I understand it, it is the non-rectangular equivalent of finding the largest eigenvectors of matrix, and it involves discounting the information contained in the less important (smaller eigenvalue) dimensions. I assume an AGO could afford a certain amount of such error due to loss of detail, for tasks similar to the 20K word example where the goal was to provide a simplified representation of bi-grams, which are just statistical correlations in ordered word pairs, and might not be appropriate for representing episodic memory when more crisp, and less statistical info is of importance. [Ed Porter] Also, it would seem that dynamically learning new patterns would present problems with a system in which the interconnects between matrices have been determined by such an SVD process. Perhaps, there would be a separate system for learning new information, and information that represented important exceptions to the SVD compression. More generally, we know from chaos theory that complex systems must limit the number of interconnections to be stable [3], which suggests that many AI problems in general can be decomposed this way. Remember we need not model the human brain in precise detail, since our goal is to solve AGI by any means. We are allowed to use more efficient algorithms if we discover them. I ran some benchmarks on my PC (2.2 GHz Athlon-64, 3500+). It copies large arrays at 1 GB per second using MMX or SSE2, which is not quite fast enough for a 10^5 by 10^5 neural network simulation. [Ed Porter] I assume this is on an uncompressed matrix. I don't know what the overhead would be to compress the matrix, such as by representing all the elements that are empty with run-length encoding, and then trying to process it. Presumably to use mmx or sse2 you would have to load some of the compressed matrix into cache, de-compress it, then run it as a block through the mmx or sse2. 1. Bellegarda, Jerome R., John W. Butzberger, Yen-Lu Chow, Noah B. Coccaro, Devang Naik (1996), "A novel word clustering algorithm based on latent semantic analysis", Proc. IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing, vol. 1, 172-175. 2. Gorrell, Genevieve (2006), "Generalized Hebbian Algorithm for Incremental Singular Value Decomposition in Natural Language Processing", Proceedings of EACL 2006, Trento, Italy. http://www.aclweb.org/anthology-new/E/E06/E06-1013.pdf 3. Kauffman, Stuart A. (1991), "Antichaos and Adaptation", Scientific American, Aug. 1991, p. 64. -- Matt Mahoney, [EMAIL PROTECTED] ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?&; Powered by Listbox: http://www.listbox.com ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=106510220-47b225 Powered by Listbox: http://www.listbox.com
<<attachment: winmail.dat>>
