I agree with swkane. Intelligence depends on both knowledge and computing power.
Early in the history of alignment research on SL4 and Overcoming Bias (precursors to LessWrong) there was a lot of discussion on how to develop recursively self improving AI in an isolated box and test it for safety before releasing it onto the Internet. The theory was that once humans developed AI that surpassed our own intelligence, then so could it except faster. I pointed out this wouldn't work because an isolated program can gain neither knowledge or computing power. I argued that the correct model would be the internet itself getting smarter. Also, we don't even know if computers have surpassed humans yet. It depends on how you measure intelligence. Computers already surpassed humans in arithmetic and short term memory in the 1950's. But in the Turing test, it would be impossible because the highest possible score is human level. If you believe that, then no further progress in AI is possible. I don't believe that. Global computing capacity is around 10^24 bits of storage and 10^23 bit operations per second, both doubling about every 2 years. The sum of all human brains is 10^25 synapses and 10^26 synapse operations per second. The biosphere stores 10^37 bits as DNA and performs 10^31 amino acid transcription operations per second. Current AI research is on passing the Turing test, or equivalently, on language modeling or text compression, because automating human labor, valued at $1 quadrillion, requires modeling human brains. Once we surpass that threshold, humans will be irrelevant. Once we surpass the second threshold, all DNA based life will be irrelevant. The question is how fast will this happen? We know that the computranium abyss cannot propagate faster than the speed of light. I believe so, even though we have been wrong about physics in the past, because every new discovery, such as atoms, the big bang, quantum mechanics, and relatively have narrowed rather than widened the bounds on computation. We cannot shrink transistors smaller than atoms, so if Moore's law is to continue then it will require nanotechnology, moving atoms instead of electrons. How long will this take? Clock speeds already stalled around 2-3 GHz in 2010. Unlike human level language models, there is no incentive to build the technology for human extinction. Will we build it anyway? What happens when cheap molecular nanoscale printers become widely available? Will hackers start playing around with self replicating nanotechnology and bioengineered viruses? There was no financial incentive to create the first computer viruses either. Will population collapse stop AI? Will humans evolve to reject technology, birth control, and women's rights? Will AI kill us by giving us everything we want? All goal seeking agents seek a state of maximum utility and stay there. Human goals evolved to avoid exactly this state, but that only works when the technology to work around our built in software doesn't exist. -- Matt Mahoney, [email protected] On Fri, May 29, 2026, 3:07 PM Quan Tesla <[email protected]> wrote: > Point taken. I don't know enough to comment on that. > > On Fri, 29 May 2026, 15:34 swkane, <[email protected]> wrote: > >> With current LLM tech, yes, that appears to be the case. But maybe that's >> only because no one yet has figured out how to get LLMs to self improve >> without human AI researchers. And yes, LLM tech is at saturation. What do >> you do training wise after consuming all published works plus the public >> web? >> >> That wasn't my point though. My point was that only a fraction of >> computing (AI, brute force search etc.) is devoted to improving AI or >> computing hardware (the recursive self improvement that underpins a >> computronium abyss). I'm pretty sure, for example, OpenAI's GPUs are being >> used more to generate cute cat pictures than to improve OpenAI's models. >> >> >> On Fri, May 29, 2026, 02:14 Quan Tesla <[email protected]> wrote: >> >>> If you're saying how, at this rate, AI cannot become more intelligent >>> than the sum of the human-produced data it bases the development of its >>> intelligence on, then I'll agree. >>> >>> A likely prediction might be: AI would soon reach the saturation point >>> of its architectural intelligence. >>> >>> On Fri, 29 May 2026, 07:29 swkane, <[email protected]> wrote: >>> >>>> I realized I didn't make a prediction. While it seems that the Earth >>>> and human civilization is on the trajectory of a computronium abyss, there >>>> is a critical point. Most computing today revolves around catering to human >>>> attention, not improving computing substrates or predictive algorithms. So >>>> the Dual Search that is central to an abyss is currently quite weak, but it >>>> trundles on as a relatively small percentage of computing capacity is >>>> devoted to it (e.g. CPU and other hardware research). >>>> >>>> The critical point is when most or close to all computing capacity goes >>>> into the Dual Search. And whether that happens here on Earth ever hinges on >>>> a number of things: >>>> >>>> Is the Dual Search a strong enough attractor for a super intelligence? >>>> Or will an ASI do something else like maybe just shrink down to a much >>>> smaller scale? >>>> >>>> Do humans go extinct before an ASI is launched and gets to the point of >>>> devoting compute capacity to Dual Search? >>>> >>>> Will proposals to launch the Dual Search on a large scale get shot down >>>> because it would be detectable by other civilizations, which could threaten >>>> humans and Earth? >>>> >>>> So, unfortunately, I have no predictions per say, just more questions. >>>> >>>> On Thu, May 28, 2026, 10:06 Matt Mahoney <[email protected]> >>>> wrote: >>>> >>>>> >>>>> On Wed, May 27, 2026, 11:23 AM Quan Tesla <[email protected]> >>>>> wrote: >>>>> >>>>>> Abiogenesis is rare, but not improbable. All is relative. Suppose >>>>>> quantum energy propagated at a factor of 1.8, where is the real potential >>>>>> limit? At quaternion furcation? This sum scales beyond supermassive >>>>>> calculations to observable infinity. >>>>>> >>>>> >>>>> I don't understand what you mean. A group in Cambridge this February >>>>> evolved a self replicating 45 nucleotide RNA strand called QT45 in a bath >>>>> of activated trinucleotide triphosphate (RNA triples) in mildly alkaline >>>>> eutectic ice, replicating both itself and its complement with 94% fidelity >>>>> and 2.1% yield in 72 days. >>>>> https://www.science.org/doi/10.1126/science.adt2760 >>>>> >>>>> This is the critical step in abiogenesis, going from non-life to the >>>>> simplest form of life. The raw ingredients including nucleotides and >>>>> simple >>>>> sugars are produced from lightning and ultraviolet light from Earth's >>>>> early >>>>> atmosphere of hydrogen, methane, ammonia, and carbon dioxide, and also >>>>> have >>>>> been found in meteorites. >>>>> https://en.wikipedia.org/wiki/Abiogenesis >>>>> >>>>> Obviously the rate of abiogenesis is going to be much lower in a soup >>>>> of thousands of random chemicals, including both left and right hand >>>>> versions of chiral molecules that don't appear in biologically derived >>>>> organisms. But if we can calculate the yield, then we will have a good >>>>> idea >>>>> of the size of the universe beyond the event horizon and the bit >>>>> complexity >>>>> of the program that describes it. >>>>> >>>>> -- Matt Mahoney, [email protected] >>>>> >>>>> >>>>> *Artificial General Intelligence List > <https://agi.topicbox.com/latest>* / AGI / see discussions > <https://agi.topicbox.com/groups/agi> + participants > <https://agi.topicbox.com/groups/agi/members> + delivery options > <https://agi.topicbox.com/groups/agi/subscription> Permalink > <https://agi.topicbox.com/groups/agi/T7daa29d46d037f94-Ma5251b7e62dc838d007e4733> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T7daa29d46d037f94-Mca6209f7d2b430a1289b7ddd Delivery options: https://agi.topicbox.com/groups/agi/subscription
