On Sat, Sep 9, 2023, 5:57 PM mm ee <[email protected]> wrote: > There is no reason to believe that 1 bit or 1 synaptic connection > corresponds to a single pattern of a memory >
Not one synapse, one neuron. Human memory is associative. Synapses represent associations between concepts, at least in the connectionist model that makes neural networks easy to understand. But connectionism doesn't have a mechanism for learning new concepts and adding neurons. We solve the problem by having neurons represent linear combinations of concepts and synapses represent linear combinations of associations. A rule of thumb for programming neural networks is to use on the order of 1 synapse or weight or parameter per bit of compressed training data. Too small and you forget. Too big and you over fit. GPT3 uses 175B parameters to train on 500B tokens of text, suggesting a compression ratio of 0.1 bits per character. A human level language model in theory should need 1B parameters to train on 1 GB of text at 1 bpc. ChatGPT knows far more than any human could remember. And compression ratio gets better as the training set gets bigger. All of human knowledge is 10^19 characters compressing to 10^17 bits at 0.01 bpc because 99% of what you know is shared or written down (why it costs 1% of lifetime income to replace an employee). The mystery is why does the brain need 6 x 10^14 synapses to store 10^9 bits of long term memories? Maybe because neurons are slow so you make multiple copies of bits to move them closer to where they are needed. Like a server farm stores 1M copies of Linux on disk, RAM, cache, and registers. Or your body has 10^13 copies of your DNA and still has to make multiple copies of a gene to mRNA before transcribing it. So if we can optimize LLM storage by using faster components, maybe we can do the same for vision at video speeds. We know that we can only store visual information at 5 to 10 bits per second, same as language. We figured out language by abandoning symbolic reasoning and training semantics before grammar. Maybe we can solve vision by modeling a fovea and eye movements. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tc1bcda5fdb4147f4-M2f3394ad2074b3b5f297a8db Delivery options: https://agi.topicbox.com/groups/agi/subscription
