My wife and I just returned from 18 days of exploring ancient sites in Rome
and Tunisia, hiking the canyon walls of Camino del Rey in Spain, and
running the Porto marathon in Portugal. We were fortunate that our return
flight from Madrid to Orlando was the day after a nationwide strike halted
travel in Spain and the day before the FAA grounded all air traffic in the
US because of a software error.

I didn't bring my laptop so I spent most of the next day updating the Large
Text Compression Benchmark on my website with a flurry of Hutter prize
submissions. The rate of progress since July has been astounding. From 2006
when I created the benchmark until earlier this year, the record improved
from 127 MB by durilca (a PPM compressor) to 107 MB set by nncp in 2019
using a transformer, an average of 2% gain per year. Now another
transformer, cmix-neif 2.0 holds the record (pending testing) at 89 MB.
That is 20 years of progress in 4 months, a 60 fold speedup. Now nncp sits
at 16'th place and durilca at 31'st among 238 compressors. All of this
progress is due to AI generated code and (high quality) documentation.

I do not believe in a fast takeoff singularity, but using AI to write AI
sure looks like one. I just read that Anthropic used 1000 agents on an
unreleased model to write a complete operating system in 8 days, 1000 times
faster than a team of humans. I remind you that the cost of human
intellectual work is $10K per million input tokens and $50K per million
output tokens. The most advanced models are 1000 times cheaper.

In 1999 I wrote a proposal to NSF for a text compression prize to promote
AI development. It was rejected, as were 90% of proposals. So I created
https://mattmahoney.net/dc/text.html in 2006 without prize money until it
was funded by the Hutter prize. Now I am 71, retired 12 years, and
supporting this research voluntary because I think that AI is important

One early result was that compression increases logarithmically with
computing power, as shown in the 2 graphs I made in 2008. That relationship
still holds. Now the new result is two lineages of transformer based models
derived from cmix and nncp. Both use offline training by multiple passes of
back propagation (8-10 GPU days) quantizing 6-8M weights to 4 bits,
compressing and appending to the compressed file. The cmix lineage uses a
12 layer network and the nncp lineage, led by DNA4, uses 5 layers. DNA4
only compresses to 100 MB but is much faster, 2.5 GPU days to train and 2.5
GPU hours to model using 14 bit tokens vs 8 bits and 2 CPU days for cmix.

I still believe there are better algorithms than transformers. The brain
learns by Hebb's rule, which is purely local, not back propagation. Human
learning is single pass, not offline in dozens or hundreds of passes. An
online neural network would eliminate the 3-4 MB penalty for storing the
weights in the decompressor (or double penalty in the Hutter prize for
storing another copy in the compressor). Once that happens, the number and
precision of the weights could be increased to improve compression further.
I believe that 75 MB is possible if hardware limits are removed.

-- Matt Mahoney, [email protected]

------------------------------------------
Artificial General Intelligence List: AGI
Permalink: 
https://agi.topicbox.com/groups/agi/T2ce81ade71c4115f-Mbd1bc33f7aad1a4d7573352c
Delivery options: https://agi.topicbox.com/groups/agi/subscription

Reply via email to