Matt, thanks for the extensive detail. After your response, I've noticed that 
free LLM downloads for notebooks have become available via Github. I was 
impressed by a "doomsday" type machine hosting an LLM, where requests and 
browsing could be done offline in a disaster scenario. Not sure if it would 
hold real agentic value.

More questions:

  1.
What would you say was probably going to emerge from the industry after the 
compression limit was reached and coherent performance still remained a 
challenge?
  2.
What is an accurate definition and description of the learning you're referring 
to. Is it similar to a semantic setup for the machine, or uploading a 
particular domain into its dictionary?
  3.
We're discussing AI in human terms a lot, but how would these models be 
different for purely agentic neural networks?

________________________________
From: Matt Mahoney <[email protected]>
Sent: Wednesday, 18 November 2026 16:39
To: AGI <[email protected]>
Subject: Re: [agi] The Hutter explosion


On Tue, Nov 17, 2026, 9:30 PM Nano Great 
<[email protected]<mailto:[email protected]>> wrote:
Matt, thanks for sharing. Very interesting progress. Any idea on how soon a 
processor-compressor would run as software on an 8 GB notebook computer?

It almost does now. The current Hutter prize entries are required to run in 50 
hours on my laptop on one core and no GPU using 10 GB memory. That is already 
10,000 times faster than the human brain. 1 GB of text is a lifetime of 
learning.

But I think the trend will be faster hardware and more capabilities as it 
always has been, once the temporary hardware price spike is over. But long 
term, Moore's law is over, at least for silicon. You can't make transistors 
smaller than atoms.

There was some discussion on the Hutter prize list about whether the 
transformer architecture models how humans process language. I argued that the 
brain doesn't train using thousands of epochs of back propagation, then freeze 
the weights and use a 1 MB context window to predict words, forgetting and 
replaying the whole conversation before every question. LLMs do this so they 
don't leak information between users. I argued that humans learn continuously 
and that a winning decompressor should be updating the weights continuously in 
a single pass on the partially decoded output.

Otherwise the architecture is pretty similar. Real neural networks are 
recurrent, with lateral (sideways) inhibition producing winner take all states 
(attention) and backwards inhibition to scale inputs logarithmically. In a 
transformer, the recurrence is unrolled to dozens or hundreds of feed forward 
layers, and softmax layers are inserted to implement scaling and attention or 
short term memory.

Current entries train a 6-8M parameter, 5-12 layer transformer for 1-2 GPU 
weeks, quantize the weights to 4 bits, compress them by making the compressed 
size part of the objective function, and then append the 3-4 MB of compressed 
weights to both the compressor and archive before inference with a 2-4K context 
window of 8-14 bit tokens. I argued that online training could remove this 
penalty and allow larger and higher precision modes for better inference, but 
requiring a GPU. Since July, enwik9 went from 110 MB to 89 MB, 20 years of 
progress in 4 months. I believe that 75 MB is possible on LTCB, which does not 
have any time or hardware limits. That is the lower bound of Shannon's 1950 
estimate of the entropy of written English. The Hutter prize requires 50 hours, 
10 GB, 1 core, and no GPU. So far, all of the progress is within these limits.

But how to train online? Marcus Wölk, author of the leading entry, has studied 
childhood language development extensively and argued that humans don't learn 
in a single pass either. In my own research I saw that children learn about 
5000 words per year after having used the word 20 times. Assuming a Zipf 
distribution where the n'th most frequent word has frequentcy 0.1/n and 6 hours 
per day of reading or listening to speech, then that child also heard or read 
the word 2000 times before learning it.

I argued that the transformer weights could be updated after each article. I 
argued that this models the brains of marsupial and placental mammals including 
humans which store about 100K tokens in the hippocampus during the day and then 
replay those chopped up memories during REM sleep to train the cerebral cortex, 
which is otherwise read only. The cortex has a capacity of 10^9 bits, so 90% is 
ultimately forgotten. But an LLM should have no such limitation, even one 
running on a high end PC.
Artificial General Intelligence List<https://agi.topicbox.com/latest> / AGI / 
see discussions<https://agi.topicbox.com/groups/agi> + 
participants<https://agi.topicbox.com/groups/agi/members> + delivery 
options<https://agi.topicbox.com/groups/agi/subscription> 
Permalink<https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mb435b02e8f28b413d4947ded>

------------------------------------------
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mccac775a73d750af01d538cf
Delivery options: https://agi.topicbox.com/groups/agi/subscription

Reply via email to