However, it's noteworthy that Hopfield nets and other ANN models generally have memory capacity far below what error-correcting-code theory would suggest is possible.
So, these bounds are not really that useful, because they don't seem to correspond to realistic incremental learning methods.... A Hopfield net or biological memory has to make up its coding as it goes along, it can't use a mathematically optimal codebook, because it doesn't know in advance what are the length-m sequences it's actually going to need to code in its length-n sequences... ben On Thu, Oct 16, 2008 at 6:43 PM, Ben Goertzel <[EMAIL PROTECTED]> wrote: > > They also note that according to their experiments, bounded-weight codes > don't offer much improvement over constant-weight codes, for which > analytical results *are* available... and for which lower bounds are given > at > > http://www.research.att.com/~njas/codes/Andw/<http://www.research.att.com/%7Enjas/codes/Andw/> > > ben > > > > > On Thu, Oct 16, 2008 at 6:40 PM, Ben Goertzel <[EMAIL PROTECTED]> wrote: > >> One more addition... >> >> Actually the Hamming-code problem is not exactly the same as your problem >> because it does not place an arbitrary limit on the size of the cell >> assembly... oops >> >> But I'm not sure why this limit is relevant, since cell assemblies in the >> brain could be very large >> >> Anyway, it seems that to answer your precise question, one must look at >> **bounded weight codes**, as is done in >> >> http://www.jucs.org/jucs_5_12/a_note_on_bounded/Bent_R.html >> >> They do not give analytical formulas there, but they give some >> computational results they found using some nice graph theory software. >> >> The final paragraph of their paper is interestingly relevant: >> >> *** >> However, as a final comment, we mention that maximum-sized codes may have >> potential future uses in alternate models of computation/communication. >> Though this is currently hypothetical, it is not entirely implausible. >> Consider for example some future alternate model of information (storage or) >> transmission - perhaps biological, perhaps electrical, perhaps something >> else - in which each (stored or) transmitted "word" has *n* binary "bits" >> (which might be represented via genetic material, or via charged particles >> in a given location, or so on) but such that, due to constraints of the >> (storage or) transmission medium, if more than *w* of the bits are "on" >> there is the possibility that the information in the word will degrade, or >> that the computer or transmission lines will incur physical damage. Possible >> reasons might include power limitations, heat dissipation, or attraction >> between biological components. In this admittedly extremely hypothetical >> setting, bounded-weight codes might play a valuable role, as their >> limitation would be exactly suited to the physical constraints imposed by >> the (storage or) transmission medium. >> *** >> >> ;-) >> >> -- Ben >> >> >> >> On Thu, Oct 16, 2008 at 6:24 PM, Ben Goertzel <[EMAIL PROTECTED]> wrote: >> >>> >>> Ed, >>> >>> After a little more thought, it occurred to me that this problem was >>> already solved in coding theory ... just take the bound given here, with >>> q=2: >>> >>> http://en.wikipedia.org/wiki/Hamming_bound >>> >>> The bound is achievable using Hamming codes (linked to from that page), >>> so it's realizable. >>> >>> What they do there is, look at q-ary code strings of length n (in the >>> binary case, q=2). >>> >>> Then, they say that d errors in transmission will occur while the code is >>> being sent, so they want any two code strings to differ in at least d >>> places. >>> >>> The bound tells how many different messages can be encoded in this way. >>> >>> I think this is the same as your problem. Assume you have n neurons. >>> Then each potential cell assembly may be represented as an length-n bit >>> string, where the i'th neuron in the network corresponds to the i'th bit in >>> the bit string, and an assembly A has a 1 in position i iff it contains the >>> i'th neuron. >>> >>> Then, what we want is for no two assemblies to have more than d neurons >>> overlap -- because there may be some noise in memory retrieval. >>> >>> So, the optimal way to do this would be to assign memories to cell >>> assemblies according to a perfect code, such as a Hamming code. >>> >>> Of course this is not how the brain works, though. >>> >>> But anyway, that seems to resolve your combinatorics problem... >>> >>> -- Ben >>> >>> >>> >>> On Thu, Oct 16, 2008 at 3:51 PM, Ed Porter <[EMAIL PROTECTED]> wrote: >>> >>>> Thanks, for offering to look into this.. The bounds I saw were upper >>>> bounds (such as just the number of possible combinations), and I was more >>>> interested in lower bounds. >>>> >>>> >>>> >>>> -----Original Message----- >>>> *From:* Ben Goertzel [mailto:[EMAIL PROTECTED] >>>> *Sent:* Thursday, October 16, 2008 2:45 PM >>>> *To:* [email protected] >>>> *Subject:* Re: [agi] Who is smart enough to answer this question? >>>> >>>> >>>> >>>> >>>> I am pretty sure their formulas give bounds on the number you want, but >>>> not an exact calculation... >>>> >>>> Sorry the terminology is a pain! At some later time I can dig into >>>> this for you but this week I'm swamped w/ practical stuff... >>>> >>>> On Thu, Oct 16, 2008 at 2:35 PM, Ed Porter <[EMAIL PROTECTED]> wrote: >>>> >>>> Ben, >>>> >>>> >>>> >>>> Thanks. I spent about an hour trying to understand this paper, and, >>>> from my limited reading and understanding, it was not clear it would answer >>>> my question, even if I took the time that would be necessary to understand >>>> it, although it clearly was in the same field of inquiry. >>>> >>>> >>>> >>>> After about an hour I gave up, since it uses a lot of terminology I do >>>> not know and since it is sufficiently deep that without the help of someone >>>> to guide me, I am not even clear that I would be able to understand it >>>> enough to extract from it guidance as to how to solve the problem I posed. >>>> >>>> >>>> >>>> Ed Porter >>>> >>>> >>>> >>>> -----Original Message----- >>>> *From:* Ben Goertzel [mailto:[EMAIL PROTECTED] >>>> *Sent:* Thursday, October 16, 2008 11:32 AM >>>> *To:* [email protected] >>>> *Subject:* Re: [agi] Who is smart enough to answer this question? >>>> >>>> >>>> >>>> >>>> OK, I see what you're asking now >>>> >>>> I think some bounds on the number you're looking for, are given by some >>>> classical combinatorial theorems, such as you may find in >>>> >>>> *http://www.math.ucla.edu/~bsudakov/cross-<http://www.math.ucla.edu/%7Ebsudakov/cross-> >>>> intersections.pdf* >>>> >>>> (take their set L to consist of {0,...,O} ... and set A_1 = A_2), and >>>> the references given therein. Anyway that paper should clue you in as to >>>> the right keyphrases to use in hunting down related theorems if you want >>>> to. >>>> >>>> You are right that it's a nontrivial combinatorial problem >>>> >>>> -- Ben >>>> >>>> >>>> On Thu, Oct 16, 2008 at 11:08 AM, Ed Porter <[EMAIL PROTECTED]> >>>> wrote: >>>> >>>> >>>> Eric, >>>> >>>> Actually I am looking for a function A =f(N,S,O). >>>> >>>> If one leaves out the O, and merely wants to find the number of >>>> subcombinations of size S that can be formed from a population of size >>>> N, >>>> just apply the standard formula for combinations. But adding the >>>> limitation >>>> that none of the combinations in A is allowed to overlap by more than O >>>> with >>>> any other combination in A makes things much more complex, and way >>>> beyond my >>>> understanding. >>>> >>>> Ed Porter >>>> >>>> >>>> -----Original Message----- >>>> From: Eric Burton [mailto:[EMAIL PROTECTED] >>>> Sent: Wednesday, October 15, 2008 8:05 PM >>>> To: [email protected] >>>> Subject: Re: [agi] Who is smart enough to answer this question? >>>> >>>> >Is anybody on this list smart and/or knowledgeable enough to come up >>>> with a >>>> >formula for the following (I am not): >>>> >>>> I don't think I'm the person to answer this for you. But I do have >>>> some insights. >>>> >>>> >>>> >Given N neural net nodes, what is the number A of unique node >>>> assemblies >>>> >(i.e., separate subsets of N) of size S that can have less than O >>>> >overlapping nodes, with the population of any other such node assembly >>>> >similarly selected from the N nodes to have the same size S and less >>>> than >>>> >the same O overlapping nodes with any other such node assembly. >>>> >>>> Good question. Let's call the function that returns A for N, "f(N)", >>>> and continue. >>>> >>>> >>>> >For example, if you have 1 billion nodes (N = 1G), how many cell >>>> assemblies >>>> >(A) of size 10,000 (S=10K) will have less than 5,000 nodes (0 = 5K) in >>>> >common with the population of any other node assembly. >>>> >>>> So at this point we are seeking f(1000000000), which I'm going to go >>>> ahead and call P. >>>> >>>> >>>> >Its easy to figure out how many unique cell assemblies drawn from a >>>> >population of N nodes that can have a size S, but I haven't a clue, >>>> other >>>> >than by computational exploration to figure out how many will each have >>>> less >>>> >than a given level of overlap with any other unique cell assemblies. >>>> >>>> This is why we have f. >>>> >>>> >>>> >And for anyone who knows how to solve the above, if possible, could you >>>> also >>>> >please also tell me, once you have close to A node assemblies selected >>>> that >>>> >have less than O overlap, how can you rapidly determine the population >>>> of a >>>> >new node assembly that has less than O overlap? >>>> >>>> For my purposes, f is a black box. You'd have to delve into its >>>> internals to answer this yourself. >>>> >>>> >>>> >This is not just an meaningless math problem. >>>> >A lot of people believe the human brain uses cell assemblies to >>>> represent >>>> >nodes in a representation of semantic knowledge. Such cell assemblies >>>> >create problems with current computer hardware because they tend to >>>> require >>>> >very high internal bandwidth, but in future architectures this problem >>>> may >>>> >not exist, and if the number of cell assemblies that can be created >>>> with a >>>> >sufficiently low cross-talk is large relative to the number of nodes, >>>> the >>>> >use of cell assemblies can allow for redundancy, high representational >>>> >capacity, and gradual degrading of memories over time to make room for >>>> more >>>> >memories. >>>> >>>> Hence, I suppose, the value of your inquiry. If I'm visualizing the >>>> issue correctly you'd like a function that returns the number of >>>> unique neural nets with a number of nodes either less or greater than >>>> n and interconnects either less or greater than i... I'll reiterate >>>> that I'm not a student of mathematics and aren't qualified to address >>>> the details of the problem. But surely this function is what we've >>>> been calling f. I'd welcome corrections and clarifications. I may not >>>> understand the question. >>>> >>>> Eric B >>>> >>>> >>>> ------------------------------------------- >>>> agi >>>> Archives: https://www.listbox.com/member/archive/303/=now >>>> RSS Feed: https://www.listbox.com/member/archive/rss/303/ >>>> Modify Your Subscription: >>>> >>>> https://www.listbox.com/member/?&; >>>> >>>> Powered by Listbox: http://www.listbox.com >>>> >>>> >>>> >>>> ------------------------------------------- >>>> agi >>>> Archives: https://www.listbox.com/member/archive/303/=now >>>> RSS Feed: https://www.listbox.com/member/archive/rss/303/ >>>> >>>> Modify Your Subscription: https://www.listbox.com/member/?&; >>>> >>>> >>>> Powered by Listbox: http://www.listbox.com >>>> >>>> >>>> >>>> >>>> -- >>>> Ben Goertzel, PhD >>>> CEO, Novamente LLC and Biomind LLC >>>> Director of Research, SIAI >>>> [EMAIL PROTECTED] >>>> >>>> "Nothing will ever be attempted if all possible objections must be first >>>> overcome " - Dr Samuel Johnson >>>> ------------------------------ >>>> >>>> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >>>> <https://www.listbox.com/member/archive/rss/303/>| >>>> Modify<https://www.listbox.com/member/?&;>Your Subscription >>>> >>>> <http://www.listbox.com> >>>> >>>> >>>> ------------------------------ >>>> >>>> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >>>> <https://www.listbox.com/member/archive/rss/303/>| >>>> Modify<https://www.listbox.com/member/?&;>Your Subscription >>>> >>>> <http://www.listbox.com> >>>> >>>> >>>> >>>> >>>> -- >>>> Ben Goertzel, PhD >>>> CEO, Novamente LLC and Biomind LLC >>>> Director of Research, SIAI >>>> [EMAIL PROTECTED] >>>> >>>> "Nothing will ever be attempted if all possible objections must be first >>>> overcome " - Dr Samuel Johnson >>>> >>>> ------------------------------ >>>> >>>> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >>>> <https://www.listbox.com/member/archive/rss/303/>| >>>> Modify<https://www.listbox.com/member/?&;>Your Subscription >>>> >>>> <http://www.listbox.com> >>>> >>>> >>>> ------------------------------ >>>> *agi* | Archives <https://www.listbox.com/member/archive/303/=now> >>>> <https://www.listbox.com/member/archive/rss/303/> | >>>> Modify<https://www.listbox.com/member/?&;>Your Subscription >>>> <http://www.listbox.com> >>>> >>> >>> >>> >>> -- >>> Ben Goertzel, PhD >>> CEO, Novamente LLC and Biomind LLC >>> Director of Research, SIAI >>> [EMAIL PROTECTED] >>> >>> "Nothing will ever be attempted if all possible objections must be first >>> overcome " - Dr Samuel Johnson >>> >>> >>> >> >> >> -- >> Ben Goertzel, PhD >> CEO, Novamente LLC and Biomind LLC >> Director of Research, SIAI >> [EMAIL PROTECTED] >> >> "Nothing will ever be attempted if all possible objections must be first >> overcome " - Dr Samuel Johnson >> >> >> > > > -- > Ben Goertzel, PhD > CEO, Novamente LLC and Biomind LLC > Director of Research, SIAI > [EMAIL PROTECTED] > > "Nothing will ever be attempted if all possible objections must be first > overcome " - Dr Samuel Johnson > > > -- Ben Goertzel, PhD CEO, Novamente LLC and Biomind LLC Director of Research, SIAI [EMAIL PROTECTED] "Nothing will ever be attempted if all possible objections must be first overcome " - Dr Samuel Johnson ------------------------------------------- agi Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: https://www.listbox.com/member/?member_id=8660244&id_secret=117534816-b15a34 Powered by Listbox: http://www.listbox.com
