Am Di, 14. Mär 2023, um 13:48, schrieb John Clark: > On Tue, Mar 14, 2023 at 7:31 AM Telmo Menezes <te...@telmomenezes.net> wrote: > >>> > One of the authors of the article says "It’s interesting that the >>> > computer-science field is converging onto what evolution has discovered", >>> > he said that because it turns out that 41% of the fly brain's neurons are >>> > in recurrent loops that provide feedback to other neurons that are >>> > upstream of the data processing path, and that's just what we see in >>> > modern AIs like ChatGPT. >> >> *> I do not think this is true. ChatGPT is a fine-tuned Large Language Model >> (LLM), and LLMs use a transformer architecture, which is deep but purely >> feed-forward, and uses attention heads. The attention mechanism was the big >> breakthrough back in 2017, that finally enabled the training of such big >> models:* > > I was under the impression that transformers are superior to recurrent neural > networks because recurrent processing of data was not necessary with > transformers so more paralyzation is possible than with recursive neural > networks; it can analyze an entire sentence at once and doesn't need to do so > word by word. So Transformers learn faster and need less trading data.
It is true that transformers are faster for the reason you say, but the vanishing gradient problem was definitely an issue. Right before transformers, the dominant architecture was LSTM, which was recurrent but designed in such a way as to deal with the vanishing gradient: https://en.wikipedia.org/wiki/Long_short-term_memory Memory is the obvious way to deal with context, but like you say transformers consider the entire sentence (or more) all at once. Attention heads allow for parallel learning to focus on several aspects of the sentence at the same time, and then combining them at higher and higher layers of abstraction. I do not think that any of this has any impact on the size of the training corpus required. > >> *> My intuition is that if we are going to successfully imitate biology we >> must model the various neurotransmitters.* > > That is not my intuition. I see nothing sacred in hormones, I agree that there is nothing sacred about hormones, the only important thing is that there are several of them, with different binding properties. Current artificial neural networks (ANNs) only have one type of signal between neurons, the activation signal. Our brains can signal different things, importantly using dopamine to regulate learning -- and thus serve as a building block for a decentralized, emergent learning algorithm that clearly can deal with recursive connections with no problem. With recursive connections a NN becomes Turing complete. I would be extremely surprised if Turing completeness turns out to not be a requirement for AGI. > I don't see the slightest reason why they or any neurotransmitter would be > especially difficult to simulate through computation, because chemical > messengers are not a sign of sophisticated design on nature's part, rather > it's an example of Evolution's bungling. If you need to inhibit a nearby > neuron there are better ways of sending that signal then launching a GABA > molecule like a message in a bottle thrown into the sea and waiting ages for > it to diffuse to its random target. Of course, they are easy to simulate. Another question is if they are easy to simulate at the speed that we can perform gradient descent using contemporary GPU architectures. Of course, this is just a technical problem, not a fundamental one. What is more fundamental (and apparently hard) is to know *what* to simulate, so that a powerful learning algorithm emerges from such local interactions. Neuroscience provides us with a wealth of information about the biological reality of our brains, but what to abstract from this to create the master learning algorithm that we crave is perhaps the crux of the matter. Maybe it will take an Einstein level of intellect to achieve this breakthrough. > I'm not interested in brain chemicals, only in the information they contain, > if somebody wants information to get transmitted from one place to another > as fast and reliablely as possible, nobody would send smoke signals if they > had a fiber optic cable. The information content in each molecular message > must be tiny, just a few bits because only about 60 neurotransmitters such as > acetylcholine, norepinephrine and GABA are known, even if the true number is > 100 times greater (or a million times for that matter) the information > content of each signal must be tiny. Also, for the long range stuff, exactly > which neuron receives the signal can not be specified because it relies on a > random process, diffusion. The fact that it's slow as molasses in February > does not add to its charm. I completely agree, I am not fetishizing the wetware. Silicon is much faster. Telmo > If your job is delivering packages and all the packages are very small, and > your boss doesn't care who you give them to as long as they're on the correct > continent, and you have until the next ice age to get the work done, then you > don't have a very difficult profession. Artificial neurons could be made to > communicate as inefficiently as natural ones do by releasing chemical > neurotransmitters if anybody really wanted to, but it would be pointless when > there are much faster, and much more reliable, and much more specific ways of > operating. > > John K Clark See what's on my new list at Extropolis > <https://groups.google.com/g/extropolis> > kuh > > > > > -- > You received this message because you are subscribed to the Google Groups > "Everything List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to everything-list+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/everything-list/CAJPayv089oC%3DAc-DswW5simNfWzQsGAZADjusaWOacE4M6kt9g%40mail.gmail.com > > <https://groups.google.com/d/msgid/everything-list/CAJPayv089oC%3DAc-DswW5simNfWzQsGAZADjusaWOacE4M6kt9g%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to everything-list+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/5076bb8a-99dd-4673-965f-92c6d35edc70%40app.fastmail.com.
