On Sun, Sep 13, 2026 at 6:48 PM Matt Mahoney <[email protected]> wrote:
> On Sat, Sep 12, 2026 at 9:44 AM James Bowery <[email protected]> wrote: > > From the NiNOR essay: > > > Now we still have a free parameter but is it a choice of Turing > machine? No. It's an integer, the size of which goes up only as the log2 > of the amount of memory required to expand the executable archive. > > > > The point is that if you can reduce the data-dependent prior to an > integer "tape" size, you may have made some progress in defining > algorithmic information. > > I suppose that would work, but all this does is fix the language used > to estimate Kolmogorov complexity. It needs more specification. How do > you represent the description of the gate logic? Shannon's 1938 switching-circuits paper generalized De Morgan theorem <https://www.cs.virginia.edu/~evans/greatworks/shannon38.pdf> <https://www.cs.virginia.edu/~evans/greatworks/shannon38.pdf>: NOR and NAND are not only universal but also equivalent in circuit complexity consequences, so take your pick. Additional choices are available creating a very limited taxonomy of cyclic logics compared to the literally infinite range of UTMs.One can get lost in a sea of choices if one wishes of course but realistically speaking the range of interesting choices is quite limited. For example Turing chose 2 inputs rather than what I prefer which is n-Inputs. In terms of network stability the most interesting choice is 0 gate day in which case a circuit like X=NAND(X,X) is, from within its temporal frame a square wave and from outside its temporal frame a source of random bits. Is it a list of gates > in numerical order, each followed by a list of inputs? How do you > represent the numbers? Can you compress it, like to use macros to > describe repeated logic like adders or registers? > These are all choices captured by the emulator. >> I expect to see even further improvements by growing the transformer > from 6M parameters to maybe 50M. > > > > And that will be interesting in itself for two reasons: 1) Why would > 50M be the optimum? 2) Is there nothing to be learned here regarding the > role evolution plays in establishing priors? of > > From information theory, a neural network should have one parameter > per compressed bit of training data, the minimum needed to reproduce > the data without overfitting. Cite? Are you referring to Shannon's empirical measure of bits per character based on human prediction? > The top LLMs use 5-10 trillion > parameters on 20-30 TB of text, which is in the same ballpark. > That doesn't really answer #1. 50M duplicated between compressor and decompressor is a lot of prior to saddle the compressor with in a 100MB compressed 1GB Wikipedia executable archive. > BTW the LTCB has a new leader. https://mattmahoney.net/dc/text.html > RATA-CMIX is a derivation of fx2-cmix-transformer that follows Hutter > prize limits but is not a submission because the improvement is less > than 1%. It uses the same transformer weights but improves their > compression. It also adds some interesting modeling improvements. > > Also, tufazip is an open source derivation of nncp, which is closed > source and held the top spot for 2 years. > > The top 5 entries all use transformers. I still believe that there are > better algorithms, because we know that the human brain learns in a > single pass. Or maybe it doesn't. I hope that the LTCB or Hutter prize > will either motivate its discovery or explain why the brain needs > 10^14 to 10^15 synapses to represent 10^9 bits of long term memory and > LLMs don't have this limitation. > I think if we closely analyze Vladimir's compressor prior it will reveal something more interesting than other pre-processing approaches thus far involving curriculum design (ordering articles, vocabulary, etc). I suspect it will reveal something like a hierarchical transition grammar specialized for enwik9. The distinction being that such a grammar is a "Zero To One" gain which pays for its duplication in the excutable archive as an "instruction set" for "UTM Choice". > -- > -- Matt Mahoney, [email protected] ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T3f8115622f860785-M0d88c653a816635519af6877 Delivery options: https://agi.topicbox.com/groups/agi/subscription
