I'm assuming 1 but per character compression, so 1 GB of input text is 1B bits, so 1B parameters. enwik9 compression is actually a little better.
A neural network with m neurons and n connections can implement roughly 2^n/m! distinct functions, allowing the m neurons to be permuted to equivalent networks. Taking the log, that's roughly n - m log m bits, or about n where usually n >> m. On Wed, Nov 22, 2023, 3:33 PM James Bowery <jabow...@gmail.com> wrote: > I'm asking because when you say "ideally" this evokes a *recurrent* > neural network that approximates what I've called the NiNOR complexity > <https://agi.topicbox.com/groups/agi/T803f813e57fcb8c4-M11cc58df5c95b3d60c4a089b/theodoric-of-yorks-computer-age-nahhh> > of the corpus: the "ideal" "compressed training data". > > Then you invoke 0.3 bpp as associated with this "ideal" of a "parameter". > This is all in the context of enwik9 where the word "billion" has the unit > "bytes" that may, *somehow*, relate to the occurrence of the word > "billion" in the sense of the sentence in question, which is associated > with the unit "bit". > > See my confusion? > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/Tdc371ce11a040352-M3c722861bab9531dc3fd786b> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/Tdc371ce11a040352-M3dee7509367cde2fecf54328 Delivery options: https://agi.topicbox.com/groups/agi/subscription
