On Mon, Aug 12, 2024 at 7:29 PM YKY (Yan King Yin, 甄景贤) <[email protected]> wrote: > Attached is my presentation PPT with some new materials not in the submitted > paper.
I wonder if you had time to answer your question at the end of the presentation. How does this help AGI? We have the algorithm mostly figured out. A fully connected neural network can simulate an arbitrary number of layers to learn arbitrarily complex features as well as an attention mechanism through mutual inhibition. We established that text prediction is sufficient to pass the Turing test (as well as any possible test for consciousness). The largest language models have 10^12 parameters trained on 10^13 tokens using 10^26 operations at a cost of 10^17 per dollar, on the order of $1 billion. Now it is an engineering problem. We can't make transistors much smaller (2 nm = 18 silicon atoms) to reduce power consumption, but we can optimize the hardware for sparse, low precision vector operations. We can reduce training costs to one operation per parameter per token using one shot learning, like the brain and most text compressors already do. Ultimately it will take nanotechnology, moving atoms instead of electrons, to reduce power consumption for a human brain sized neural network from 1 MW to 20 watts. That technology is still decades away. Meanwhile we have the much larger problem of collecting the training data needed to automate human labor, which is now up to $110 trillion and rising 5% per year. Yet, ChatGPT has been out for almost 2 years without the slightest increase in unemployment. The problem is that you need to collect 10^17 bits of human knowledge to do all the work that people do, and we only have 10^14 bits (15 TB) of text available on the public internet and most of it is already used to train LLMs. AI will profoundly change the world. But when I look at the ads for Meta, Gemini, and Copilot, I think, really? Is this the best we can do with AI? Help kids write fan letters? These are basically toys. Collecting all the knowledge you need to do your job that isn't written down will cost on the order of $100 trillion at the global average wage rate of $5 per hour. Figure one year of human time to train your replacement. It doesn't matter if it is carbon or silicon. You can only output 3 tokens per second. -- -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T04cca5b54df55d05-M802d3df156a65ab8ae5b6fc4 Delivery options: https://agi.topicbox.com/groups/agi/subscription
