Thank you Matt for doing this.

Danko


Dr. Danko Nikolić
CEO, Robots Go Mental
www.robotsgomental.com
www.danko-nikolic.com
https://www.linkedin.com/in/danko-nikolic/
-- I wonder, how is the brain able to generate insight? --


On Sun, Nov 15, 2026 at 1:09 AM Matt Mahoney <[email protected]>
wrote:

> My wife and I just returned from 18 days of exploring ancient sites in
> Rome and Tunisia, hiking the canyon walls of Camino del Rey in Spain, and
> running the Porto marathon in Portugal. We were fortunate that our return
> flight from Madrid to Orlando was the day after a nationwide strike halted
> travel in Spain and the day before the FAA grounded all air traffic in the
> US because of a software error.
>
> I didn't bring my laptop so I spent most of the next day updating the
> Large Text Compression Benchmark on my website with a flurry of Hutter
> prize submissions. The rate of progress since July has been astounding.
> From 2006 when I created the benchmark until earlier this year, the record
> improved from 127 MB by durilca (a PPM compressor) to 107 MB set by nncp in
> 2019 using a transformer, an average of 2% gain per year. Now another
> transformer, cmix-neif 2.0 holds the record (pending testing) at 89 MB.
> That is 20 years of progress in 4 months, a 60 fold speedup. Now nncp sits
> at 16'th place and durilca at 31'st among 238 compressors. All of this
> progress is due to AI generated code and (high quality) documentation.
>
> I do not believe in a fast takeoff singularity, but using AI to write AI
> sure looks like one. I just read that Anthropic used 1000 agents on an
> unreleased model to write a complete operating system in 8 days, 1000 times
> faster than a team of humans. I remind you that the cost of human
> intellectual work is $10K per million input tokens and $50K per million
> output tokens. The most advanced models are 1000 times cheaper.
>
> In 1999 I wrote a proposal to NSF for a text compression prize to promote
> AI development. It was rejected, as were 90% of proposals. So I created
> https://mattmahoney.net/dc/text.html in 2006 without prize money until it
> was funded by the Hutter prize. Now I am 71, retired 12 years, and
> supporting this research voluntary because I think that AI is important
>
> One early result was that compression increases logarithmically with
> computing power, as shown in the 2 graphs I made in 2008. That relationship
> still holds. Now the new result is two lineages of transformer based models
> derived from cmix and nncp. Both use offline training by multiple passes of
> back propagation (8-10 GPU days) quantizing 6-8M weights to 4 bits,
> compressing and appending to the compressed file. The cmix lineage uses a
> 12 layer network and the nncp lineage, led by DNA4, uses 5 layers. DNA4
> only compresses to 100 MB but is much faster, 2.5 GPU days to train and 2.5
> GPU hours to model using 14 bit tokens vs 8 bits and 2 CPU days for cmix.
>
> I still believe there are better algorithms than transformers. The brain
> learns by Hebb's rule, which is purely local, not back propagation. Human
> learning is single pass, not offline in dozens or hundreds of passes. An
> online neural network would eliminate the 3-4 MB penalty for storing the
> weights in the decompressor (or double penalty in the Hutter prize for
> storing another copy in the compressor). Once that happens, the number and
> precision of the weights could be increased to improve compression further.
> I believe that 75 MB is possible if hardware limits are removed.
>
> -- Matt Mahoney, [email protected]
>
>
>
>
>
>
>
>
>
>
>
>
> *Artificial General Intelligence List <https://agi.topicbox.com/latest>*
> / AGI / see discussions <https://agi.topicbox.com/groups/agi> +
> participants <https://agi.topicbox.com/groups/agi/members> +
> delivery options <https://agi.topicbox.com/groups/agi/subscription>
> Permalink
> <https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mbd1bc33f7aad1a4d7573352c>
>

------------------------------------------
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-M5303034b9a7b5ab2f5617392
Delivery options: https://agi.topicbox.com/groups/agi/subscription

Reply via email to